Post-Redaction Repair Tools: Maps and the Remap-A-Tron


Although the OpenStreetMap data license change from CC-BY-SA to ODbL is not formally done, the redaction bot has done its thing. It was activated in mid-July after rigorous testing and months of development to go over the entire planet, square degree by square degree, and delete any tainted node, way and relation or revert it to a untainted state. (An object is tainted, simply put, when it was at some point edited by a contributor who did not agree to the license change and the new contributor terms.)

OpenStreetMap Redaction bot progress map screenshot taken on July 17th as the bot was just progressing to Switzerland and re-running London.

Although less than 1% of all data was removed in the redaction process, some areas are much more strongly affected than others, depending on the amount and previous activitiy of local ‘decliners’ as well as pre-emptive remapping of tainted features. This was made possible by tools that identified potentially tainted objects, such as Simon Poole’s Cleanmap and a JOSM plugin. (These tools have since been discontinued.)

Looking at the situation in the US, what we’re left with after the license change redaction is a map that has issues. Well, let’s say it has more issues now than it had before the redaction. Just looking at the street network, which is the one single most important dimension of OpenStreetMap if you ask me, we don’t have to look very hard to find missing  street segments, entire missing streets, and messy geometry. We had messy geometry before, because of the inherent messiness of TIGER/Line data off of which the US road network is based, but the redaction process left us with some really, ehm, interesting spaghetti in places.

Messy way geometries as a result of the redaction, near Los Angeles, CA.

Missing way segment as a result of the redaction, north of Honolulu, HI

And then there’s the invisible results of the redaction: lost attributes such as speed limits, lane count, direction of flow – but I would suggest we fix the road geometries and topological integrity of the main road network first, so the OpenStreetMap data is routable and the routes make sense. In order to do that effectively, the community needs tools. Fortunately, we have some available already. There’s Toby Murray’s map showing suspicious way segments last touched by the redaction bot. Kai Krueger provided a routing grid page that calculates route times and distances between major US cities, and compares those to reference data to expose potential routing issues. And there’s also the venerable OSM Inspector tool that provides a layer showing objects affected by the redaction process.

I also took a stab at a small set of tools that I hope will be helpful in identifying the remaining issues in the road network for the US: a Redacted / Deleted Ways map, and the Remap-A-Tron.

Redacted / Deleted Ways Map

Not unlike the redaction layer in OSM Inspector, this map duo exposes features affected by the redaction process, with a few notable differences. The main difference is that it has a clear focus on the US road network data. The Redacted Ways Map differentiates between Motorway, Trunk, Primary, Secondary and lower class roads, in an attempt to make it easier for remappers to seek out and prioritize the more important roads.

The Redacted Ways Map showing different road classes in appropriate coloring for easy prioritization of remapping efforts

The Deleted Ways map complements the Redacted Ways map. It only shows the way segments that were completely obliterated by the redaction bot. The focus on the road network means that only ways with a highway tag are visualized here.

The Deleted Ways map differentiates between ways that are likely to already have been remapped, and those that still need to be remapped.

If you look at this map, you will notice that there are two different classes of deleted ways, displaued in red and green respectively. The difference is in remapping status. The red ways are probably not remapped yet, while the green ways are likely to have been remapped already. I make this distinction by periodically comparing the deleted way geometries with newly mapped way geometries in the database. Using the Hausdorff distance algorithm as the main component, I devised an algorithm that reasonably accurately predicts whether a way has already been remapped.

Remap-A-Tron

I came up with another way to leverage the knowledge of remappedness of the OpenStreetMap street network. Building on an idea I already had semi-developed a while ago, I built a web application that serves up one deleted way geometry overlaid onto a current OpenStreetMap basemap. Using a simple menu or handy keyboard shortcuts, you can flag the segment as already remapped, skip it for whatever reason, or load that area into JOSM or Potlatch right away to remap it.

The Remap-A-Tron serves up one non-remapped important way segment at a time for remappers to work on.

The data backend is refreshed every (US) night, so there should not be too much of a lag. Currently, the app serves up only non-remapped important segments: motorways down to tertiary class roads. There are about 1900 segments in this category as of the time of writing. If a handful of people spend an evening with the Remap-A-Tron, this should be down to zero in a matter of days. Once we complete the repairs on the important roads, I can tweak the app to include all other highways (~15K segments not remapped) and ultimately all ways (~52K segments not remapped).

I built this tool with versatility in mind. After the street remapping is done, it could easily be repurposed to identify other issues in OSM that are identifiable on the Mapnik map: self-intersecting ways, road connectivity issues, missing bridge tags, maybe even missing traffic lights? I would love to hear your ideas.

If this turns out to be useful and used, I will try and increase the coverage to the entire world. For that to happen, the Remap-A-Tron will need to find a home that is not the server next to my desk, though..

Happy remapping!

Set up your own routing web application using OpenStreetMap data and open source software


Did I mention that I love OpenStreetMap? It’s not just the map that looks more beautiful and complete every day, but also all the neat things it allows people to do with the raw data which is free for anyone to pick up and use. This is not usually something I engage in myself – suum quique – but it does feel good to get your hands dirty with some open data and open source software once in a while.

My instance of YOURS in action

For me, the reason was that I want a good (looking and functioning) and open bicycle routing application for Amsterdam. There is around 1 million bicycles in this city, and I believe most of those are used every day, but the city lacks a good cycle route planner. There is one attempt, but the app is in flash limiting its usage potential, I find it less than intuitive to use and it’s not open. What I want from a good cycle route planner is

  1. Allowing users to set parameters for their trip: avoid traffic lights, avoid major roads, go through parks if possible, prefer separate bike lanes over shared lanes; this list can grow. There are many reasons why people get on their bikes; sometimes they’re in a hurry, sometimes they want to enjoy the ride, they might be with kids. A good bicycle routing app should appreciate the different route requirements that ensue.
  2. Allowing users to determine what information they see on the map so they can make informed decisions. This may include traffic incidents involving cyclists, car traffic intensity, inclines (sic – we have steep bridges!)
  3. Allowing users to interact with the data and the community. If something is not right or suboptimal in the route returned, offer an easy way to fix the relevant data, or file a bug so someone else can do it. Also: upload GPX tracks which can be used in an aggregated fashion to show preferred routes. Locals to a neighborhood know optimal routes better than cold data, why not leverage that?
  4. Offering a mobile interface so cyclists can do turn-by-turn routing. A dedicated app that collects track points (think TomTom MapShare) while the app is used would be excellent!

So I wanted to get started with this. At first I looked at OpenRouteService, the OpenStreetMap-based routing service. It is an awesome web application that works really well, both for car and for bicycle and pedestrian routing. The web interface is well done. It has one huge drawback though: the code is not open, so there’s no way to use it in a different application.

I asked around for different solutions, and there proved to be an excellent one, as pointed out to me by main contributor Lambertus: YOURS, or Yet Another OpenStreetMap Route Service. There are several other open solutions – I’m currently looking into BBBike and Routino as well – but YOURS proved to be instantly accessible, and Lambertus was available to help ironing out some setup difficulties. It uses gosmore, another open source project from the OpenStreetMap community, as a routing backend. And the best thing is installation is really a breeze. It is quite well explained on the YOURS wiki page, and involves only a few steps:

  • Have a suitable web server environment set up (Linux / Apache / PHP5)
  • Download and compile gosmore (instructions on wiki)
  • Optionally, retrieve a planet file for the region you’re interested in and convert to gosmore binary format (also well documented)
  • Retrieve the YOURS directory from SVN (instructions on wiki) into your apache web directory
  • Set some variables in the YOURS config

And basically you’re done. I had some trouble getting gosmore to work, it turned out that I needed to compile it with a HEADLESS option set in the makefile; gosmore has a UI but my server does not have X11, so gosmore croaks. I am still having some trouble connecting to the (also excellent) OpenStreetMap gazetteer service, nominatim, but that has more to do with my rusty PHP and apache skills than anything else. Basically, I have set up my own routing service for the Netherlands, supporting various transportation mode, using open data and open source software, in less than an hour.  I find that amazing. The first open, community-supported bicycle routing web application for Amsterdam is born!

A big thanks to Lambertus for helping me out. He does awesome things, providing routable map data for Garmin devices to name but one.