Volunteered Geographic Information

Free, Priceless Or Somewhere In Between?

This is the title that has been popping into my head since last summer. I am writing it down because it encompasses in a very general sense the themes that I want to cover in my dissertation, and thus serves me well in trying to guide me while I try to elaborate on them.

I have actually already written some paragraphs elaborating on the themes and ideas that follow, but I want to force myself to touch upon them concisely here.

Volunteered Geographic Information (VGI) is a concept that has not been around for a very long time. Geographic Information has, however: it is what maps are made out of, and what your car navigation device relies on to guide you. Traditionally, Geographic Information is collected, processed and used by professionals, but this no longer holds true: Geographic Information has undergone a process of democratization, both in the usage dimension and in the collection and processing dimension. People are now used to dealing with Geographic Information in different contexts, and have started to pool resources to collectively build repositories of Geographic Information, to facilitate the democratization of the entire ecosystem of Geographic Information.

OpenStreetMap is the most prominent of these efforts, and one in which I have been actively involved since early 2007. Since its conception in 2005, it has grown to a worldwide collaborative effort involving more than 100.000 contributors. In some regions, the maps available from OpenStreetMap are so rich and complete that they are used instead of commercially available map data.

I realize that I need to come up with some examples here, and some numbers that give an indication of how OpenStreetMap has grown, but I am on a train, blissfully disconnected from the internet, so you will just have to bear with me for now. But believe me, it’s getting big fast – at a rate that makes me worried about the validity of any quantitative research results that I might present in the context of this dissertation. But this will have to be dealt with in some future note.

Let us assume for now that OpenStreetMap – there are other VGI efforts around, and they will need to be touched upon as well – is indeed starting to occupy a significant share in the commercial market for Geographic Information. That means the OpenStreetMap data represent a commodity and as such, economic value. As OpenStreetMap data is available at no cost, this value is not quantified in the marketplace, however. This poses intriguing questions:

What is this freely available OpenStreetMap data actually worth?

How do you even begin to measure the value of something that is not subject to the usual economic market mechanisms?

When dealing with value, I believe I cannot omit the concept of quality, especially in this context. Any VGI effort relies on volunteers collecting data in their spare time. While some regions have very active communities, getting together to discuss progress and plan improvements to the map, checking and correcting each other’s contributions, other regions rely on single, isolated individuals contributing to the map – or worse: no-one contributing at all. The resulting picture is one of spotty coverage: very densely mapped regions exist side by side with tersely covered regions. More questions arise!

Is it possible to define the quality of volunteered geographic information in any satisfactory way?

How?

More generally: how do quality and value relate when dealing with geographic information?

I think I cannot proceed from here without looking at real world situations. Economic value is defined in the marketplace where supply and demand meet, and thus cannot be studied without some understanding of how and where this demand arises.

There clearly is a demand for VGI, but where does it originate?

Why would people want to use information that comes with no guarantees of completeness or even factual correctness, and that does not have a consistent quality?

I will need to get to the bottom of this. Apparently it is ‘good enough’ for some! If I’m not careful I will be entering into the domain of psychology. I think I need to stop soon, or I will have covered all domains of modern science and will have defined ample questions to last me three dissertations. But let me just finish this train of thought, and by then I will have arrived in Berlin – one of the best covered cities in OpenStreetMap, by the way; you can even get a detailed map of the zoo!

What drives the decision on the demand side to use volunteered geographic information instead of commercial offerings that do come with a quality label?

I can think of a number of reasons. Firstly, there is a growing number of application domains that do not require extensive, nationwide coverage. The growing domain of location based services are often only relevant in metropolitan areas; consider for example pedestrian and bicycle routing, social networking applications, tourist guide services or restaurant / bar recommendation applications. Even many applications in professional domains operate only within a designated metropolitan area: local police, fire brigades and other public safety professionals operate only within their metro area.

Interestingly, supply and demand sync up really nicely here: in areas where there is likely to be a great demand for high quality – whatever that may mean – geographic information, there is also likely to be a large number of contributors to volunteered geographic information repositories. (This reminds me of my master’s thesis that dealt with the quality of public transportation in rural areas. There was a similar process at play: because of the limited and geographically thinly spread demand, the costs of maintaining a reasonable quality of service had become so high that cuts in service quality had become unavoidable, lowering the demand even further. Both Dutch and German regional governments were struggling to counter this downward spiral, and I did a comparative study on the results of those efforts.)

Secondly, because there is very little restrictions and limitations in terms of how and where you can use the data. Commercial data usage licenses are more often than not restricted to a certain type of application, device or to a limited number of users or devices, and the data can only be used as-is. OpenStreetMap data can be used in almost every context imaginable, and you are free to modify and adapt the data to suit your needs.

Lastly, of course, because it’s free.

I have mixed feelings about this post. It feels unfocused, but I guess that is to be expected. More importantly, I don’t feel comfortable in the domain of economics. Sure, I did my two years of high school accounting and economics, but it did not quite take. It does not particularly interest me, but I feel I need to deal with it anyway. Intuitively, I am drawn to the question of defining and measuring quality. I want to think about how to do that, write tools to analyze OSM data – that part I am really passionate about. It seems like a good moment to talk to Henk and maybe some other people I know that could help and advise me at this junction.

So this is it. This is going to be my dissertation diary. I’m not going to make any commitments as to how often I will write in it; I just read that I should be spending at least 15 minutes every day on my dissertation. Every day for the next four, five, six years! Intriguing at least.

I’m at the very beginning of the process, and my thoughts are really unfocused at this point. In this first entry, I will not go into the theme itself, there will be ample opportunity for that. I would just, for a moment, like to ponder over the implications. At least four years of my life will be dedicated, at least to some degree, to researching and writing about this theme that has yet to unfold.

As I am writing this, I feel that I want to write, I like to explore my thoughts by putting them in writing, although writing in English makes it even harder for my fingers to keep up with my ever-wandering mind.

The first question that springs to mind as I embark on this diary is: should I publish it? Not the dissertation I mean, but these notes? It seems, on the one hand, pointless and vain. Who would want to read about the nitty-gritty details of my struggle towards acquiring a doctorate? Not many, probably, but there might be a reason or two to do it anyway.

Publishing my thoughts might help me overcome a feeling of awkwardness that I frequently have about this project: who am I to think I can do original, creative research? These isolated thoughts, rough outlines of a theme that I might want to pursue, seem so superficial and gratuitous! If I would just go ahead and publish my thoughts and ideas and processes – that would seem to provide some validity to them. An irrational thought maybe, but it works for me.

Publishing these notes may also invoke some sense of urgency. I know I have a tendency to keep thoughts and ideas to myself for too long, thinking they need to mature before they are ready to be shared with the world. This is an inhibition that will seriously slow me down and that I must learn to set aside. It has already happened and I have not even begun to formalize a proposal!

More than a year ago now, Henk Scholten invited me to come to the Vrije Universiteit to discuss the possibilities of him supervising my dissertation. We had a really nice and productive discussion and I felt both flattered and motivated, and told him I would write some ideas I had down for him to ingest. We would have a follow-up meeting soon.
I explored the idea for a while, discussed implications with a couple of colleagues and friends, thought about interesting themes. I think I even wrote some things down, but I did not feel any of them were good or mature enough to even put forward to Henk.

Although the though of doing a dissertation was on my mind now and then over the months that followed, I found myself glad to be distracted by other things to occupy my mind and time. And so time passed, and here we are. I feel that I want to do this more strongly now, for reasons I will explain in a future post. So I am going to write. And explore. It will be beautiful. I can be that naive.

OpenStreetMap is de vrije wereldkaart waaraan iedereen kan bijdragen. De geodata is vrij beschikbaar volgens een Creative Commons-licentie. OpenStreetMap (OSM) bevat allang niet meer alleen straten, maar is uitgegroeid tot een veelzijdige repository van vrij beschikbare geodata. Het is alleen nog niet zo makkelijk om er uit te pakken wat je nodig hebt.
Het standaard exportformaat van OpenStreetMap is een eigen XML-formaat. Dit is met allerlei open source tools, die beschikbaar zijn via de OSM-wiki op http://wiki.openstreetmap.org of de subversion-repository op http://svn.openstreetmap.org/.
Dit artikel illustreert hoe je de actuele Nederlandse gemeentegrenzen uit de live OSM-database haalt en deze importeert in een PostGIS-database.

De database

Het startpunt voor het zoeken naar specifieke informatie in de OSM-database is de Map Features-wikipagina: http://wiki.openstreetmap.org/wiki/Map_Features. Deze pagina bevat een overzicht van alle gebruikte ‘tags’ voor objecten in de database. Gemeentegrenzen vallen onder de Administrative Boundaries: http://wiki.openstreetmap.org/wiki/Key:boundary. Hoewel de op deze pagina bijgehouden tabel met de indeling per land hier niet helemaal specifiek over is – er wordt gesproken van ‘boundaries for cities like Amsterdam but also smaller like Volendam and Lutjebroek’ – vallen de gemeentegrenzen onder admin_level=8. Op dezelfde pagina lezen we dat de modus operandi om administratieve grenzen in OSM te zetten is door gebruik te maken van ‘relations’. (OpenStreetMap kent slechts drie soorten objecten: nodes (punten), ways (lijnen) en relations (relaties tussen groepen van de andere twee types).)

Extractie

We weten nu dat we alle ‘relations’ van het type ‘admin_level=8′ willen hebben. Er zijn verschillende manieren om een dergelijke abstractie uit de live-database te maken. De ene is een actuele dump van het gewenste gebied downloaden (deze zijn beschikbaar via http://downloads.cloudmade.com/ ) en hieruit vervolgens met de command-line tool ‘osmosis’ (http://wiki.openstreetmap.org/wiki/Osmosis ) een selectie maken. Een andere manier is om gebruik te maken van de OSM Extended API (OSMXAPI, spreek uit OSM-Zappy, zie http://wiki.openstreetmap.org/wiki/Osmxapi ). De volgende URL levert dan de gemeentegrenzen op in OSM XML-formaat: www.informationfreeway.org/api/0.5/relation[admin_level=8][bbox=3.35376,50.57484,7.22095,53.51513].

Import

Het resulterende OSM-XML-bestand kun je importeren in een PostGIS-database met behulp van OSM2PGSQL: http://wiki.openstreetmap.org/wiki/Osm2pgsql.
Ervan uitgaande dat je al een spatial database hebt met de naam ‘postgis’ gaat het dan als volgt:

> osm2pgsql -H tm-sr -U postgres -W -d postgis gemeentegrenzen_081118.osm

osm2pgsql SVN version 0.55-20081118 $Rev: 10464 $

Password:
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Mid: Ram, scale=100

Reading in file: gemeentegrenzen_081118.osm
Processing: Node(110k) Way(2k) Relation(0k)
Node stats: total(110573), max(312315964)
Way stats: total(2579), max(28446793)
Relation stats: total(690), max(51805)

Writing way(0k)

Te zien is dat osm2pgsql vier tabellen aanmaakt (als deze al bestaan dan worden ze default leeggemaakt, let op dus!).
We maken ons even niet druk om spatial indexes en bekijken het resultaat:

EPSG:28992, or the Dutch double stereographic RD (RijksDriehoekstelsel) projection, is quite often incompletely or just plain badly defined.

My version of MapServer for Windows (2.2.6 from september last year) states

+proj=stere +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.999908 +x_0=155000 +y_0=463000 +ellps=bessel +units=m +no_defs  no_defs <>

Which yields the following result when a native 28992 dataset is projected onto a Microsoft Virtual Earth (EPSG:900913 or EPSG:3785 as it is now called):

Note that the buildings layer on top of the VE aerial photos is shifted to the north, by about 100 metres.

Spatialreference.org has a slightly different take on EPSG:28992:

+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +units=m +no_defs

which yields an almost identical result:

These projection strings are both incomplete, because they do not take into account the datum shift that is used in the RD projection and can be approximated using the ‘towgs84′ parameter in PROJ4.

The one and only right PROJ4 projection string is

+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.999908 +x_0=155000 +y_0=463000 +ellps=bessel +units=m +towgs84=565.2369,50.0087,465.658,-0.406857330322398,0.350732676542563,-1.8703473836068,4.0812 +no_defs <>

Links

• Explanation of the towgs84 parameter on this page
• Some discussion about the RD datum shift on the PROJ.4 mailing list
• A non-technical discourse on datum shift and coordinate systems in Dutch.
• The Dutch national survey has a website on the RD coordinate system.
• There is also a very Web 0.5 site on the RD system and NAP (Normaal Amsterdams Peil, the Dutch standard sea water level which can be observed in the Amsterdam City Hall)

I came across yet another tile caching implementation, GeoWebCache, through this article on the Google Open Source blog. It integrates nicely with the Geoserver OGC server, which should make it very easy to set up on a Windows box. So let’s try that.First, we need to get the Java SDK (not the runtime environment) from Sun; Geoserver needs it. An older 1.4.x version will do, but if you start a-fresh, you might as well get the Java 6 SDK from here. Download and install it.

Then we grab a fresh copy of Geoserver from their web site. (direct download link for 1.6.3)

UPDATE 1.6.4 is out, as Mike Pumphrey pointed out already in the comments section. Improved support for the GeoWebCache plug-in is among the new features. Read on for the implications for this guide.

While you’re there, download the GeoWebCachce extension as well.

Unzip the GeoWebCache extension, which comes as a zip file.

Install the Geoserver package. It will try and locate the Java SDK and notify you about it. I don’t think installing in the default location is such a nice solution, so I changed this to c:\geoserver\ instead. The data directory then defaults to c:\geoserver\data_dir which I left like that.

When Geoserver is installed, we need to add all the JAR files that comprise the GeoWebCache extension to C:\geoserver\webapps\geoserver\WEB-INF\lib.

UPDATE The following config file tweak is obsolete as of Geoserver 1.6.4. Only if you want to stick with 1.6.3 or earlier, the following paragraph applies:

We need to make a small manual change to one of the Geoserver config files, web.xml in C:\geoserver\webapps\geoserver\WEB-INF\web.xml. This is described in the extension README:

find

<mime-mapping>

BEFORE, add

<servlet-mapping>
<servlet-name>dispatcher</servlet-name>
<url-pattern>/gwc/*</url-pattern>
</servlet-mapping>

Before we start Geoserver, we are going to change the default admin credentials just to be on the safe side. To do this, we open C:\geoserver\data_dir\security\users.properties and change ‘admin=geoserver’ to ‘admin=[yourpassword]‘.

Now we’re ready to start Geoserver and try to call the tile cache. We’re not going to add our own layers just yet; Geoserver comes with some simple sample data. To stret Geoserver, use the Start Menu entry or call

C:\geoserver\bin\startup.bat

from the Run box or a command line.

This will open a Java command line which should remain opened. The output shown could also be useful when you start hacking the Geoserver configuration. Your Windows firewall could complain about Java, in which case you could ‘Unblock’ Java to allow http access from outside your box.

To confirm that it all works, type

http://localhost:8080/geoserver/

in your browser, which should yield your Geoserver Admin home page (no maps there).

Now type

http://localhost:8080/geoserver/gwc/service/ve?layers=topp:states&quadkey=02

which should yield a Microsoft Virtual Earth tile schema compliant tile:

The last thing I want to do right now is to integrate this into a Microsoft Virtual Earth map client. This is quite well described in the GWC documentation and is dead simple anyway. Basically, you take a barebones VE map client template and add a VETileSourceSpecification:

var tileSourceSpec3 = new VETileSourceSpecification(“GeoserverSample”, “http://martijne:8080/geoserver/gwc/service/ve?quadkey=%4&format=image/png&layers=topp:states”);
tileSourceSpec3.NumServers = 1;
tileSourceSpec3.MinZoomLevel = 1;
tileSourceSpec3.MaxZoomLevel = 16;
tileSourceSpec3.Opacity = 0.3;
tileSourceSpec3.ZIndex = 100;
vemap.AddTileLayer(tileSourceSpec3, true);

You only need to change the server part of the URL in the first line to your computer name (or localhost if you’re only going to access the sample from your own machine).

Next steps

The major disadvantage of Geoserver for me is that it cannot do cascading WMS, although cascading WFS was recently implemented. So I’d like to set up GeoWebCache together with UMN MapServer to be able to do this.

When that’s done, I’d like to do some performance benchmarking and compare it to the Metacarta TileCache solution.

Well, certainly not today, and certainly not soon, but the introduction of georeferenced photos on Google Maps this week will certainly rock the online photo communities’ boat. Sure, there are tons of websites overlaying flickr photos on top of a web map, and most are richer than what Google Maps currently offers. Take for example loc.alize.us, a flickr/Google Maps mashup that has been around for a while. It offers tag filtering, user filtering, and a very nice and clean interface. To top it off, it offers a bookmarklet that integrates georeferencing into flickr.com very nicely. I still use it, although Yahoo Maps, the mapper of choice for Flickr’s mapping needs, of course, has had adequate coverage of the Netherlands for some time now.

But still.. It’s not directly ON Google Maps, which is – at least in Western Europe at this time – the ubiquitous web map. The general public will rarely discover any layer of the geographic web beyond Google Maps and Google Earth. ‘So, if I want my photos to show up on the web, I need to be on Panoramio.’ – Panoramio being the photo sharing community that has been showing off on Google Earth for as long as I can remember, and as from now on Google Maps as well. Panoramio was acquired by Google in May, 2007.

No, I don’t expect a mass flux of flickr users towards Panoramio. The latter will see a good number of new members though, and if Google remains as picky about which photos to display within Maps – I’m still confused as to where this leaves Picasa; I guess the user base is not large enough – Panoramio might become a force to be reckoned with in the online photo community universe.

The Dutch like to live in new, modern homes with a garden front and back. This leaves the country with many a suburban jungle like the one depicted below. This also means steady jobs for surveyors with the commercial mapping companies – and many a free weekend spent mapping for a Dutch OpenStreetMap contributor.

So, although the Netherlands got a huge boost in OpenStreetMap when AND donated its dataset for the Netherlands to the community last year, we need to continue our efforts to keep the free world map up to date. And next Saturday is as good as any, so we are heading towards the ‘Saendelft‘ suburbia project just north of Amsterdam.

Jochen Topfs maps comparison web application shows that we are lagging behind the Big G here:

If you’d like to join, find all relevant info on the Wiki (in Dutch). If you just want to see results, check OpenStreetMap in about a week’s time.

The Microsoft Virtual Earth API lets you add your own tile layer to your VE Map. My colleague StevenO wrote about preparing a suitable TileCache setup. This used to work in both 2D and 3D map modes. Recently, Microsoft introduced the latest version of the API, 6.1, along with a major data upgrade and a new version of the 3D control. A step forward in many respects, but the tile layers will not show up in 3D mode anymore. Let’s investigate. UPDATED 080508 11pm, see below

Let me start out by saying that I did not get the tile layers to display again in the new 3D control, so my preliminary conclusion is that this is a bug (read: feature) in the new VE version. But let me at least reiterate what I’ve tried. I’m also touching on some more general VE API topics that crossed my path.

First, I decided to define as many of the optional parameters for the VETileSourceSpecification as possible. Almost all are optional; only TileSource and ID are required and set in the constructor. I defined the Bounds property, first as a simple VELatLongRectangle (supplying only the top left and bottom right coordinates), that made no difference whatsoever.

As a sidenote: having a GIS background, I can’t really get used to the way a bounding box, or a simple VELatLong point for that matter, are defined. I am used to defining bounding boxes by supplying their bottom-left and top-right points, but the VE API turns that all around and has you supply the top-left and bottom-right points. Those points, in turn, are defined as lat-long instead of long-lat. Does MS do this just to annoy us, or what?

The VELatLongRectangle can also be defined using four points, which is relevant for 3D mode, as you might not be looking straight down and the bounding box of your viewport might not be a rectangle at all. So I tried to define the full rectangle for the Bounds property as well, with no discernible effect:

var bounds = [new VELatLongRectangle(new VELatLong(57,3),new VELatLong(50,7),new VELatLong(57,7),new VELatLong(50,3))];

Zoom Levels

Next up is the MinZoom and MaxZoom properties. When not set, the tile layer will just show up in every zoom level, which is considered to be a possible performance issue. I don’t really see why, as the tiles will be generated by a tile caching mechanism in any decent implementation. I can see that you want to restrict the visibility of your tile layer to certain zoom levels for aesthetic reasons, however.

In 3D mode, zooming is continuous, so I was wondering what the GetZoomLevel() method would return. The SDK documentation states: ‘This method may not give the same value in 3D mode as in 2D mode.’, without elaborating. It must have something to do with the viewport bounding box in 3D being determined by three variables (heading, pitch and distance to the earth) instead of just one in 2D (just the distance). In any case, you can’t expect the zoom level to remain constant when switching between 2D and 3D modes. It does however provide you with an indication of scale to base layer visibility on. Also note that in 3D mode, the GetZoomLevel() returns a single value, whereas the control internally seems to use a more dynamic approach to zoom levels. Notice that when you tilt your view, tiles that are closer to the ‘camera’ location are rendered with more detail than tiles that are nearer the horizon.

To get back on track; still no luck displaying our tile layers in 3D mode. Let’s see what options are left. There are a couple more properties in the VETileSourceSpecification: NumServers, Opacity and ZIndex. Experimenting with those did not yield any results though. ZIndex looked somewhat promising, but whereas changing values does affect layer ordering in 2D, in 3D we still see no custom layers.

API versions

The last thing I tried was reverting to an older API version. The test page I was using only worked with versions 5 and 6, although older versions down to 3 are still supported. IIRC, the custom tile layer functionality was implemented in version 4, but there could be other backward compatibility issues that I don’t want to investigate right now. Anyway, switching to v6 and v5 did not yield any results in what I was actually trying to achieve: to get the custom tile layers to show up.

I asked Johannes Kebeck about this issue, following the VE Technical Briefing that was held this morning. For those just starting in VE API development, as well as for developers already somewhat experienced, sessions with Johannes are a really good introduction. Keep an eye on his blog or the MS Events site for future events. If and when Johannes gets back to me, I’ll update this post.

UPDATE

After having looked into the issue with Johannes Kebeck, we pinpointed the problem: the tile store was served up from a nonstandard port, 8080. Microsoft confirms that fetching tiles from a port other than 80 is disabled in the 3D control, for security reasons. We chose the nonstandard port in order to be able to serve both the VE web application and the tilecache from our VE appliance. Because TileCache prefers an Apache environment and VE an IIS server environment, different port numbers seemed the easy way to accomplish this.

There comes a time for every geo-ict professional to have his first encounter with GML. Most of the time, this is not a pretty sight. Until now, I have managed to steer clear from GML when it comes to actually incorporating it into my own software. But today, this day dawned.

So here’s what I’m going to do.  I’m going to generate .NET classes for the GML 3.2.1 schema. When that’s done, I’ll try and actually read a GML 3.2.1 document and do something remotely useful and relevant with it. The second bit will be for a followup post, most probably.

To get started, we’re going to get the combined schemas in one zip file from http://schemas.opengis.net/SCHEMAS_OPENGIS_NET.zip

Then, we unzip those into a directory, say c:\ogc\

We then navigate to C:\ogc\gml\3.2.1 and open a VS2008 command prompt there.

Now we are all set to startto generate .NET classes using xsd.exe

First try is

xsd.exe gml.xsd /classes

This generates a whole bunch of error messages

Some research into the limitations of xsd.exe reveals that the schemaLocation attribute in an import element is ignored by xsd.exe.

from http://msdn2.microsoft.com/en-us/library/ew6ts9yw.aspx:

For the <import> element, the Xsd.exe tool ignores the schemaLocation attribute, expecting imported files instead as additional command-line arguments.

from http://msdn2.microsoft.com/en-us/library/ms950721.aspx

Q: Why doesn’t XSD.exe support the schemaLocation attribute on imports and includes?

A: The W3C XML Schema recommendation describes this attribute as a hint, which can be ignored by processors that can use alternate means to locate schemas. XSD.exe only uses schemas that are specified through the command line to convert schema A.xsd, which imports schema B.xsd.

xsd.exe /c A.xsd B.xsd

So it turns out we need to specify all schemas referenced using import elements need to be supplied as command line arguments to xsd.exe. That means we need to find all schema files referenced to using import elements in gml.xsd and – cascadingly – in all schema files referenced using include elements. This is one hell of a job which took me about an hour of concentrated gazing at XML schemas to complete.

The reward was an xsd.exe command line that actually worked:

C:\ogc\gml\3.2.1>xsd.exe gml.xsd ..\..\xlink/1.0.0\xlinks.xsd ..\..\iso\19139\20
070417\gmd\gmd.xsd ..\..\iso\19139\20070417\gco\gco.xsd ..\..\iso\19139\20070417
\gss\gss.xsd ..\..\iso\19139\20070417\gts\gts.xsd ..\..\iso\19139\20070417\gsr\g
sr.xsd /classes
Microsoft (R) Xml Schemas/DataTypes support utility
[Microsoft (R) .NET Framework, Version 2.0.50727.1432]
Copyright (C) Microsoft Corporation. All rights reserved.
Writing file 'C:\ogc\gml\3.2.1\gsr.cs'.

C:\ogc\gml\3.2.1>

The proof of the pudding is in the screenshot:

The result is a gml.cs C#.NET class file that we shoul be able to use in an XmlSerializer. More in a future post.

I have been doing some benchmarking in the wake of my TileCache installation ‘endeavor’ of last week (part 1 – part 2). In a series – well, probably two – articles, I will try to provide some insight into the performance of the TileCache – Python – Apache ensemble.

Let me lay out first how I went about getting the results below. I used ab on my MacBook Pro to generate the HTTP requests. ab – or ApacheBench – is an extremely easy to use benchmarking tool, and don’t be fooled by the name, it will benchmark your lighttpd / iis whatever-based server just as happily. I sent the requests to my desktop PC at the office, on which I danced the TileCache / mod_python setup dance I described in earlier posts. The connection was from my home through a VPN tunnel to the office, so it’s sort of remote – at least physically. I set ab to run for one minute for each session.

I wrote (ahem) a short shell script to do various ab sessions, with the following variables:

• using CGI or using mod_python
• the number of concurrent requests

The former is relevant because I want to show how much faster the mod_python solution actually is. I could not find any good numbers on this specifically related to TileCache. The latter is relevant because a tile server will typically receive many concurrent requests, even from a single client. This specifically applies when TileCache is employed as a caching layer between OpenLayers and a WMS server and OpenLayers is configured to cheat your server and fetch multiple tiles at once, as described here.

So now on to the results. I’ll show you some graphs which are visualizations of ab’s CSV output. They show a devision of the total number of requests in percentiles by time (in ms) needed to serve that percentile of the requests.

Note that the graphs are logarithmic (10logn). I did that to ensure optimal visibility of the results – the differences can be small for most of the percentile range, but will spike at the end of the graph.

With 5 concurrent requests:

…10 concurrent requests…

…50…

…and finally even 100 — which is already quite a heavy load, especially for my poor desktop with a single consumer SATA disk:

I’ll write up my thoughts about these results, as well as ideas for further optimization, in a next post shortly. For now, I leave you to think about this yourselves and – just because I like Excel so much – feed you two more graphs to summarize the mod_python and CGI results:

Still to come:

• Analysis of the results shown
• Further optimization techniques (ideas welcome – I’m new at this!)
• Check the log files – were all requests actually served?
• Longer running benchmarks, how does TileCache hold up in the long run?