I’ve always been a bit uncomfortable with the claims of 250 req/sec for this and 500 req/sec for that. It’s never felt quite….right.  Or real.  After all how many sites truly get that many concurrent requests?  How many visitors a day would you need to get spikes of 500 req/sec hitting your server?

Well it seems the metric is flawed and so sayeth one of the developers of the Google App Engine during one of the IO talks done in May.

In that talk he mentions that people have been trying to run load tests on the GAE that simply throw hundreds of concurrent requests and conclude the test after one second.  Unfortunately this isn’t realisitic and so the optimisations that are built into the system don’t have time to kick in.

I’d say this isn’t just an issue for GAE – its an issue with this benchmarking method full-stop.  Because as he says – its unrealistic.

Take the following example numbers from the slides:

Realistic values
• 50,000 users / day
• 2 pageviews / user
• 100,000 pageviews / day
• 5 requests / pageview
• 500,000 requests / day
• 5.8 requests / second

To some people 50k users may not seem much but irrespective I’d say its certainly a sizable chunk for the majority of readers here.  As you can see if a site is being hit by 50,000 users per day they can expect a realistic hit rate of 5.8 concurrent requests per second.

Let me say that again: A site that gets 50,000 user per day can expect just 5.8 average concurrent requests per second.

Thats a realistic number and it blows out of the water the benchmarking methods that hammer a server for one second with hundreds of requests.  People using those benchmarks have two flaws:

1. If they get a low number they may be unfairly assuming a poor service based on a spurious and unrealistic testing methodology.
2. If they get a high number they’ll assume that their server can take sustained heavy load which is certainly not the case.

What the speaker then recommends is performing more realistic load tests spread over time.  This not only provides a more realistic basis for conclusions but also enables servers to adapt to the increasing load.

You can see the video which is about making production ready apps on Google App Engine by visiting its IO page here.

UPDATE

Digg Effect / Slashdotting

One of the obvious counters to this criticism which is also expressed at the end of this talk is that the Digg Effect or Slashdotting is indeed something that requires high ‘per second’ load testing.  However, thats not the case as these phenomena actually occur over a period of minutes and even hours and not a vertical spike.  The number of concurrent hits is still relatively low and so these hundreds per/sec tests don’t fit here either.