Comments on: Benchmarking TileCache, part 1 http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/ researching VGI Fri, 06 Nov 2009 13:48:29 +0000 http://wordpress.com/ hourly 1 By: mvexel http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-19 mvexel Tue, 18 Mar 2008 20:56:46 +0000 http://oegeo.wordpress.com/?p=30#comment-19 Steven: I intend to set up an environment like the one on www.openstreetmap.org and benchmark that against tilecache in the future. The OpenStreetMap tile server used to serve map tiles to www.openstreetmap.org doesn't use TileCache at all. Instead, it uses a combination of a custom Apache module - mod_tile - handling path translations and a rendering daemon - renderd - handling both on-demand and background rendering in separate threads. It is described <a href="http://wiki.openstreetmap.org/index.php/Mod_tile" rel="nofollow">here</a>. It is supposed to be much, much more efficient than any possible TileCache setup - at least for an environment with a constant huge requirement for tile re-rendering such as the OpenStreetMap server. Steven: I intend to set up an environment like the one on http://www.openstreetmap.org and benchmark that against tilecache in the future. The OpenStreetMap tile server used to serve map tiles to http://www.openstreetmap.org doesn’t use TileCache at all. Instead, it uses a combination of a custom Apache module – mod_tile – handling path translations and a rendering daemon – renderd – handling both on-demand and background rendering in separate threads. It is described here. It is supposed to be much, much more efficient than any possible TileCache setup – at least for an environment with a constant huge requirement for tile re-rendering such as the OpenStreetMap server.

]]> By: Steven M. Ottens http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-18 Steven M. Ottens Tue, 18 Mar 2008 20:45:16 +0000 http://oegeo.wordpress.com/?p=30#comment-18 Hi Martijn, I had a little chat on benchmarking on #openlayers and I got this url: http://patrick.blog.thus.ch/2008/03/06/perf-tests-around-tilecache-and-apache-caching/ which gives a quick view on the difference between the different options of TC and apache's own cache. This might be interesting to test as well. Anne benchmarked our apache mod_cache/mapserver setup once and he couldn't get the CPU or Disk choke before the 100MBit line was filled. So there is a bar to reach. On the other hand tschaub did make this very true observation on irc: "I'm surprised there's no discussion of off-lattice requests the whole point (in my mind) of something like tilecache is to be able to specify what the client can and can't ask for if you want to cache absolutely everything, then it is not surprising that there are faster methods" Since mod_cache will cache anything and cache pollution is a problem in EduGIS, maybe even more than cache management. You said that OSM did make some changes to increase the performance of TC or that they created their own version of TC. It might be interesting to test that as well. (sorry for the in-crowd name calling in this comment) Hi Martijn,

I had a little chat on benchmarking on #openlayers and I got this url: http://patrick.blog.thus.ch/2008/03/06/perf-tests-around-tilecache-and-apache-caching/ which gives a quick view on the difference between the different options of TC and apache’s own cache. This might be interesting to test as well. Anne benchmarked our apache mod_cache/mapserver setup once and he couldn’t get the CPU or Disk choke before the 100MBit line was filled. So there is a bar to reach.

On the other hand tschaub did make this very true observation on irc:
“I’m surprised there’s no discussion of off-lattice requests the whole point (in my mind) of something like tilecache is to be able to specify what the client can and can’t ask for if you want to cache absolutely everything, then it is not surprising that there are faster methods”

Since mod_cache will cache anything and cache pollution is a problem in EduGIS, maybe even more than cache management. You said that OSM did make some changes to increase the performance of TC or that they created their own version of TC. It might be interesting to test that as well. (sorry for the in-crowd name calling in this comment)

]]> By: mvexel http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-17 mvexel Tue, 18 Mar 2008 17:43:11 +0000 http://oegeo.wordpress.com/?p=30#comment-17 Bart, thanks. I'm still in a 0.4 mindset. Let's see how 0.5 and the OpenStreetMap tileserver toolchain fit together. Bart, thanks. I’m still in a 0.4 mindset. Let’s see how 0.5 and the OpenStreetMap tileserver toolchain fit together.

]]> By: bartvde http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-16 bartvde Tue, 18 Mar 2008 17:24:08 +0000 http://oegeo.wordpress.com/?p=30#comment-16 Martijn, Mapnik now has Windows binaries which makes it quite simple to setup. Martijn, Mapnik now has Windows binaries which makes it quite simple to setup.

]]> By: mvexel http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-15 mvexel Tue, 18 Mar 2008 14:17:07 +0000 http://oegeo.wordpress.com/?p=30#comment-15 Christopher, thanks for the suggestions. I'll be sure to take them into account when I take this benchmarking a step further. You're correct in assuming my benchmarks consist of just getting a single tile. It would be interesting to benchmark something closer to real life usage. I will script together a randomizer for the tile request. Interesting. The layer used is the one that's there out of the box. I am very eager to test with a mapnik setup, because - as you well know - mapnik is used extensively in <a href="http://www.openstreetmap.org/" rel="nofollow">OpenStreetMap</a>, a project in which I am actively involved. What holds me back is setting up mapnik on Windows - that could be quite a challenge, but also an interesting topic for a future blog post :). Or does anyone know of a mapnik-on-windows setup guide? Christopher, thanks for the suggestions. I’ll be sure to take them into account when I take this benchmarking a step further.

You’re correct in assuming my benchmarks consist of just getting a single tile. It would be interesting to benchmark something closer to real life usage. I will script together a randomizer for the tile request. Interesting.

The layer used is the one that’s there out of the box. I am very eager to test with a mapnik setup, because – as you well know – mapnik is used extensively in OpenStreetMap, a project in which I am actively involved. What holds me back is setting up mapnik on Windows – that could be quite a challenge, but also an interesting topic for a future blog post :) . Or does anyone know of a mapnik-on-windows setup guide?

]]> By: Christopher Schmidt http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-14 Christopher Schmidt Tue, 18 Mar 2008 12:54:58 +0000 http://oegeo.wordpress.com/?p=30#comment-14 Also note that although this is interesting, unless you're doing something with apachebench that I don't know how to do, you're likely not *really* testing much more than "startup time" comparisons. Any real life comparison needs to have a random selection of tiles: otherwise, you're always reading the same file on disk, and that probably means that you're reading it from memory cache, which means that you're not testing how fast TileCache will perform in the real world... ... But at the same time, there should be no difference in CGI/mod_python other than startup cost, so as a 'startup cost'-o-meter for TileCache, these are valuable. Another couple options to look into: PythonAutoReload defaults to on, should probably be turned off for maximum optimization. PythonOptimize is off by default, should probably be on for maximum optimization. (Thanks to Howard Butler for these suggestions.) Also note that although this is interesting, unless you’re doing something with apachebench that I don’t know how to do, you’re likely not *really* testing much more than “startup time” comparisons. Any real life comparison needs to have a random selection of tiles: otherwise, you’re always reading the same file on disk, and that probably means that you’re reading it from memory cache, which means that you’re not testing how fast TileCache will perform in the real world…

… But at the same time, there should be no difference in CGI/mod_python other than startup cost, so as a ’startup cost’-o-meter for TileCache, these are valuable.

Another couple options to look into:
PythonAutoReload defaults to on, should probably be turned off for maximum optimization. PythonOptimize is off by default, should probably be on for maximum optimization. (Thanks to Howard Butler for these suggestions.)

]]> By: mvexel http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-13 mvexel Tue, 18 Mar 2008 08:22:21 +0000 http://oegeo.wordpress.com/?p=30#comment-13 Steven: ab comes with a standard Windows Apache httpd installation as well, fortunately. Arne: I think I described the steps well enough to be reproducable - if any thing specific is unclear, or lacking, I'd be happy to elaborate. Christopher: thanks for your analysis and suggestions! These are extremely useful and I will incorporate them in further benchmarking. Steven: ab comes with a standard Windows Apache httpd installation as well, fortunately.

Arne: I think I described the steps well enough to be reproducable – if any thing specific is unclear, or lacking, I’d be happy to elaborate.

Christopher: thanks for your analysis and suggestions! These are extremely useful and I will incorporate them in further benchmarking.

]]> By: Steven M. Ottens http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-12 Steven M. Ottens Tue, 18 Mar 2008 07:58:14 +0000 http://oegeo.wordpress.com/?p=30#comment-12 Do you know if ab works on windows, or if there's a similar application for windows, then we could use it to test a few tilecache setups we have running here. We could also do an endurance test on one of those servers. Nice graphs BTW :P Do you know if ab works on windows, or if there’s a similar application for windows, then we could use it to test a few tilecache setups we have running here. We could also do an endurance test on one of those servers.

Nice graphs BTW :P

]]> By: Arne http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-11 Arne Tue, 18 Mar 2008 02:43:56 +0000 http://oegeo.wordpress.com/?p=30#comment-11 Could you post the steps you took to create these results, so that they're reproducible? Could you post the steps you took to create these results, so that they’re reproducible?

]]> By: Christopher Schmidt http://oegeo.wordpress.com/2008/03/17/benchmarking-tilecache-part-1/#comment-10 Christopher Schmidt Tue, 18 Mar 2008 00:18:17 +0000 http://oegeo.wordpress.com/?p=30#comment-10 These are extremely cool! One comment: I can't tell, but it looks like you might be fighting with some mod_python issues: The 'cliff' looks like the difference between requests that are sent to apache children which have already loaded TileCache instances: Since TileCache caches the configuration information between requests to the same Apache children, I think that any time you have Apache children which haven't been used for TileCache, they'll be slower than others. What was your n= on these? A higher n= might well smooth out some of the side effects of the mod_python cache. Also, the CGI script startup time will be very specific to the number of levels in the config: More layers -> slower. Since mod_python caches startup time, larger config files play a lower long-term affect on the performance. Mapnik layers will also slow down your startup time. The most important thing for Apache is probably the MaxRequestsPerChild directive, which says how long TileCache config stuff will hang around before it has to be reparsed. A large number (or '0', for infinity) will lead to longer living apache children: if your software has memory leaks, this is a problem, but it will improve performance. Another interesting thing to do would be to also test a multi-threaded standalone server on localhost, and see if that works any better. I just checked in threaded support for the standalone http server (-t), and preliminary testing shows that when it works, it works great... but it seems to reliably crash after about 400 requests with c > 10, so I guess that's not actually worth much :( These are extremely cool! One comment: I can’t tell, but it looks like you might be fighting with some mod_python issues: The ‘cliff’ looks like the difference between requests that are sent to apache children which have already loaded TileCache instances: Since TileCache caches the configuration information between requests to the same Apache children, I think that any time you have Apache children which haven’t been used for TileCache, they’ll be slower than others.

What was your n= on these? A higher n= might well smooth out some of the side effects of the mod_python cache.

Also, the CGI script startup time will be very specific to the number of levels in the config: More layers -> slower. Since mod_python caches startup time, larger config files play a lower long-term affect on the performance.

Mapnik layers will also slow down your startup time.

The most important thing for Apache is probably the MaxRequestsPerChild directive, which says how long TileCache config stuff will hang around before it has to be reparsed. A large number (or ‘0′, for infinity) will lead to longer living apache children: if your software has memory leaks, this is a problem, but it will improve performance.

Another interesting thing to do would be to also test a multi-threaded standalone server on localhost, and see if that works any better. I just checked in threaded support for the standalone http server (-t), and preliminary testing shows that when it works, it works great… but it seems to reliably crash after about 400 requests with c > 10, so I guess that’s not actually worth much :(

]]>