Benchmarking TileCache, part 1


I have been doing some benchmarking in the wake of my TileCache installation ‘endeavor’ of last week (part 1part 2). In a series – well, probably two – articles, I will try to provide some insight into the performance of the TileCache – Python – Apache ensemble.

Let me lay out first how I went about getting the results below. I used ab on my MacBook Pro to generate the HTTP requests. ab – or ApacheBench – is an extremely easy to use benchmarking tool, and don’t be fooled by the name, it will benchmark your lighttpd / iis whatever-based server just as happily. I sent the requests to my desktop PC at the office, on which I danced the TileCache / mod_python setup dance I described in earlier posts. The connection was from my home through a VPN tunnel to the office, so it’s sort of remote – at least physically. I set ab to run for one minute for each session.

I wrote (ahem) a short shell script to do various ab sessions, with the following variables:

  • using CGI or using mod_python
  • the number of concurrent requests

The former is relevant because I want to show how much faster the mod_python solution actually is. I could not find any good numbers on this specifically related to TileCache. The latter is relevant because a tile server will typically receive many concurrent requests, even from a single client. This specifically applies when TileCache is employed as a caching layer between OpenLayers and a WMS server and OpenLayers is configured to cheat your server and fetch multiple tiles at once, as described here.

So now on to the results. I’ll show you some graphs which are visualizations of ab’s CSV output. They show a devision of the total number of requests in percentiles by time (in ms) needed to serve that percentile of the requests.

Note that the graphs are logarithmic (10logn). I did that to ensure optimal visibility of the results – the differences can be small for most of the percentile range, but will spike at the end of the graph.

c_1.png

With 5 concurrent requests:

c_5.png

…10 concurrent requests…

c_10.png

…50…

c_50.png

…and finally even 100 — which is already quite a heavy load, especially for my poor desktop with a single consumer SATA disk:

c_100.png

I’ll write up my thoughts about these results, as well as ideas for further optimization, in a next post shortly. For now, I leave you to think about this yourselves😉 and – just because I like Excel so much – feed you two more graphs to summarize the mod_python and CGI results:

cgi.png

modpython.png

Still to come:

  • Analysis of the results shown
  • Further optimization techniques (ideas welcome – I’m new at this!)
  • Check the log files – were all requests actually served?
  • Longer running benchmarks, how does TileCache hold up in the long run?

11 thoughts on “Benchmarking TileCache, part 1

  1. These are extremely cool! One comment: I can’t tell, but it looks like you might be fighting with some mod_python issues: The ‘cliff’ looks like the difference between requests that are sent to apache children which have already loaded TileCache instances: Since TileCache caches the configuration information between requests to the same Apache children, I think that any time you have Apache children which haven’t been used for TileCache, they’ll be slower than others.

    What was your n= on these? A higher n= might well smooth out some of the side effects of the mod_python cache.

    Also, the CGI script startup time will be very specific to the number of levels in the config: More layers -> slower. Since mod_python caches startup time, larger config files play a lower long-term affect on the performance.

    Mapnik layers will also slow down your startup time.

    The most important thing for Apache is probably the MaxRequestsPerChild directive, which says how long TileCache config stuff will hang around before it has to be reparsed. A large number (or ‘0’, for infinity) will lead to longer living apache children: if your software has memory leaks, this is a problem, but it will improve performance.

    Another interesting thing to do would be to also test a multi-threaded standalone server on localhost, and see if that works any better. I just checked in threaded support for the standalone http server (-t), and preliminary testing shows that when it works, it works great… but it seems to reliably crash after about 400 requests with c > 10, so I guess that’s not actually worth much😦

  2. Do you know if ab works on windows, or if there’s a similar application for windows, then we could use it to test a few tilecache setups we have running here. We could also do an endurance test on one of those servers.

    Nice graphs BTW😛

  3. Steven: ab comes with a standard Windows Apache httpd installation as well, fortunately.

    Arne: I think I described the steps well enough to be reproducable – if any thing specific is unclear, or lacking, I’d be happy to elaborate.

    Christopher: thanks for your analysis and suggestions! These are extremely useful and I will incorporate them in further benchmarking.

  4. Also note that although this is interesting, unless you’re doing something with apachebench that I don’t know how to do, you’re likely not *really* testing much more than “startup time” comparisons. Any real life comparison needs to have a random selection of tiles: otherwise, you’re always reading the same file on disk, and that probably means that you’re reading it from memory cache, which means that you’re not testing how fast TileCache will perform in the real world…

    … But at the same time, there should be no difference in CGI/mod_python other than startup cost, so as a ‘startup cost’-o-meter for TileCache, these are valuable.

    Another couple options to look into:
    PythonAutoReload defaults to on, should probably be turned off for maximum optimization. PythonOptimize is off by default, should probably be on for maximum optimization. (Thanks to Howard Butler for these suggestions.)

  5. Christopher, thanks for the suggestions. I’ll be sure to take them into account when I take this benchmarking a step further.

    You’re correct in assuming my benchmarks consist of just getting a single tile. It would be interesting to benchmark something closer to real life usage. I will script together a randomizer for the tile request. Interesting.

    The layer used is the one that’s there out of the box. I am very eager to test with a mapnik setup, because – as you well know – mapnik is used extensively in OpenStreetMap, a project in which I am actively involved. What holds me back is setting up mapnik on Windows – that could be quite a challenge, but also an interesting topic for a future blog post🙂. Or does anyone know of a mapnik-on-windows setup guide?

  6. Hi Martijn,

    I had a little chat on benchmarking on #openlayers and I got this url: http://patrick.blog.thus.ch/2008/03/06/perf-tests-around-tilecache-and-apache-caching/ which gives a quick view on the difference between the different options of TC and apache’s own cache. This might be interesting to test as well. Anne benchmarked our apache mod_cache/mapserver setup once and he couldn’t get the CPU or Disk choke before the 100MBit line was filled. So there is a bar to reach.

    On the other hand tschaub did make this very true observation on irc:
    “I’m surprised there’s no discussion of off-lattice requests the whole point (in my mind) of something like tilecache is to be able to specify what the client can and can’t ask for if you want to cache absolutely everything, then it is not surprising that there are faster methods”

    Since mod_cache will cache anything and cache pollution is a problem in EduGIS, maybe even more than cache management. You said that OSM did make some changes to increase the performance of TC or that they created their own version of TC. It might be interesting to test that as well. (sorry for the in-crowd name calling in this comment)

  7. Steven: I intend to set up an environment like the one on http://www.openstreetmap.org and benchmark that against tilecache in the future. The OpenStreetMap tile server used to serve map tiles to http://www.openstreetmap.org doesn’t use TileCache at all. Instead, it uses a combination of a custom Apache module – mod_tile – handling path translations and a rendering daemon – renderd – handling both on-demand and background rendering in separate threads. It is described here. It is supposed to be much, much more efficient than any possible TileCache setup – at least for an environment with a constant huge requirement for tile re-rendering such as the OpenStreetMap server.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s