In a recent video posted on the Google Webmaster Central YouTube Channel, Matt Cutts answered the question How does Google determine page speed? In his answer, he said approximately 1 out of 100 searches are affected by the site speed ranking factor.
Cutts said, 1 out of 100 searches done on Google are affected by is site speed “that it would actually change the ranking to a noticeable degree.”
He then added that this translated to about 1 in 1,000 sites are impacted by this overall. So only 0.1% of the web sites out there have site speed issues that are at such a degree where it impacts how well the site ranks.
Here is the video: