Python Webserver Framework Performance Benchmark

To find out which Python webserver frameworks perform the best, I’m going to benchmark several of them. I will be measuring metrics such as requests per second and latency under different circumstances do that under different situations.

For updates and roadmap see the bottom of this page.

Benchmarking Methodology

Hardware setup

Test-specific components (e.g. CPU/memory) are reported on each test. Most tests use 100% of a single vCPU. This ensures there is plenty of resources available to the application server. This is also fair if all servers are given the same resources during each test. Do note, however, that your results may be very different since in real scenarios the results will be much higher since most deployments don’t use only 100% of a vCPU. The goal of these benchmarks is to get an idea of the performance level and compare results between different frameworks given a level playing field.

Restricting the resources also means restricting the throughput that the framework is able to run through. The upside to this is that the testing tool won’t require massive amounts of resources, and therefore is not resource-restricted and is free to use as many resources as it wants in order not to be the bottleneck. All requests are issued from the same machine hosting the server.

Testing Environment and Tools

All tests are performed inside docker to ensure consistency. You can check the docker environment used by each test by viewing its source code.

Metrics are measured using the wrk tool. You can check the details of the configuration used for each test in the test’s docker-compose in the source code (coming soon). Otherwise, the test would specify any configurations used.

Tests

Simple “Hello World” / Echo application

This simple test runs the application with as little work as possible by simply returning a “Hello World” string with a content type “text/plain” where possible.

Each test is run using 100% of a vCPU and 512MB memory allocated to the test container. Tests are run sequentially and each test is run for 20 seconds.

There’s no surprise that almost all servers perform well in this test given that this is simply a max throughput test without any real application work. Most hover in the 3-6k request per second range. The ridiculously fast servers such as bjoern, meinheld, and fastwsgi (the latter pulling in 214k requests per second) are of course the compiled servers that are written in C so they can pull in these numbers. When these servers are used with a framework such as flask, the framework – which is written in Python – becomes the bottleneck. Although having said that, 62k requests per second by falcon and bjoern is still very high.

The ASGI framework aiohttp performed really well compared to the WSGI-based falcon and flask when paired with gunicorn’s default sync worker (9.1k, 6.5k, and 4.2k respectively). Although the WSGI-based servers pull ahead when paired with optimized servers such as bjoern.

The slowest framework in this test was cherrypy, which handled less than 3k requests per second regardless of the server used.

JSON serialization

This test returns a 5.5KB JSON string with the content type “application/json”.

Each test is run using 100% of a vCPU and 512MB memory allocated to the test container. Tests are run sequentially and each test is run for 20 seconds.

Like in the previous simple test, it is no surprise that most servers perform quite well in this JSON serialization test. Most servers are handling a respectable 2-6k requests per second. Although the JSON is small, all operations are performed in memory since the JSON is preloaded from a file on server start-up. The serialization of the JSON object is a CPU-intensive operation that happens in the Python code, therefore the optimization gains seen in the previous test from the servers written in C (bjoern, fastwsgi, and meinheld) begin to become less exaggerated.

The winner, although not by much is the bjoern server running with pypy which would have optimized the Python code to run more efficiently and was able to handle 18.5k requests per second. The slowest performing framework in this test was cherrypy which handled less than 2k requests per second regardless of the server type.

Simulated CPU Bound

This test returns a “Hello World” string with a content type “text/plain” where possible but also runs a loop that hogs the CPU for around 90-100ms before returning.

Each test is run using 100% of a vCPU and 512MB memory allocated to the test container. Tests are run sequentially and each test is run for 20 seconds.

For this test, perhaps unsurprisingly, most servers performed almost identically since the majority of CPU time is spent in the artificial loop for around 90-100ms for each request. This leads to most servers hovering around the 10-11 request per second mark here.

The exception is when using pypy to run the server. Falcon running with bjoern and pypy ran around 36 times faster than most servers, and the bjoern with pypy ran around 94 times faster than most servers, topping 1000 requests per second.

In the real world, your mileage may vary with pypy depending on your application, but I wanted to highlight that no matter the framework when you have a CPU-bound server request, most framework and server combinations will have very similar results (no magic here I’m afraid).

Simulated IO Bound

This test returns a “Hello World” string with a content type “text/plain” where possible but also sleeps the thread for 100ms before returning (note that the sleep method is not guaranteed to wake the thread up after exactly 100ms).

Each test is run using 100% of a vCPU and 512MB memory allocated to the test container. Tests are run sequentially and each test is run for 20 seconds.

This test is designed to simulate a server request that does not do any local processing on the server itself but makes requests to other servers or resources instead as a middleman. In real-world scenarios, this intermediate server will usually run some lightweight logic to do things such as figure out where to fetch data from, aggregate data once fetched, or any other “stitching” logic required.

This test category is where ASGI frameworks shine since each request can be handled independently of each other, whereas WSGI applications will typically block the request thread while waiting for “external” resources to respond (in this test, the main thread is blocked for the entire 100ms of the mock resource fetch).

The winner in this test was flask with gunicorn and either of gevent and eventlet with 2.9k requests per second. falcon with uvicorn (with or without uvloop) handling 2.6k requests per second is a close runner up. aiohttp with uvloop, gunicorn (with and without uvloop) handled 2.3-2.6k requests per second. muffin with uvicorn handled 2.6k requests.

Using bjoern‘s forking spawns more “workers” to handle more requests simultaneously, but this does not appear as optimized as the ASGI servers.

The slowest servers were the WSGI servers as expected since they are blocked in between handling requests.

Todo

• Add more frameworks and servers, and combinations of frameworks and servers.
• Publish source code
• Explore more parameters to play with such as nuitka compilation options and pypy
• Publish charts for visual comparison

Changelog

• 11/10/2023 – Initial post.

Related posts:

Why Is The First Call To Amazon Lex Slow? Health Checks: The Key to Reliable Applications Everything You Need to Know About Write-Through Caching How to Follow Redirect in Python: A Deep Dive