OpenStreetMap Chicago Hack Weekend Report


Last weekend I was in Chicago for the second N-American OSM hack weekend of the year. This one, for me and Serge, was entirely dedicated to MapRoulette. We wanted to get the next generation code as close to ready as possible for release – and we got pretty close I think! Here’s a summary of what we did:

  • Looked at user interface challenges with Tom MacWright, the main architect of the iD editor.
  • Finalized the data model and ORM code.
  • Debugged and improved OAuth authentication code.
  • Discussed and finalized the design for the API.
  • Wrote setup instructions for anyone wanting to deploy MapRoulette (which would mainly be other developers wanting to collaborate on the project)
  • Created an EC2 AMI (ami-995032f0) that has MapRoulette and all dependencies ready to go for quick deployment
  • Created a list of tasks and remaining issues.

We have now gotten to a point where we can share the code and hope to attract more developers, which I think overall is a major milestone.

Here is the public GitHub repository: https://github.com/emacsen/maproulette

To reiterate, the main objectives of this next generation are:

  • Make code and database model more transparent to enable more development collaboration
  • Support for regional challenges
  • Support for working on tasks in a defined area
  • Support for difficulty preferences
  • Enable external challenge providers to submit challenge data through an API
  • Enable linking with OSM accounts to better track user activity and leverage OSM account services such as settings storage

Below is an impression of daytime at the hack weekend:

2013-04-28 14.15.25

..and nighttime:

2013-04-26 23.14.35

Looking back on a fantastic OpenStreetMap hack weekend in Toronto


I am on my way back home from my first visit to Toronto, and my first ever visit to Canada – currently nursing a mild hangover with a Minnesota ale on a 4 hour layover in Minneapolis.

All of this is connected to the second OpenStreetMap Hack Weekend in Toronto I was lucky enough to be able to attend. I had an amazing time! Steve (new!), Serge and I were able to make great progress on MapRoulette, which is of course fantastic, and I thoroughly enjoyed hanging out with old friends and new connections. I want to extend a heartfelt thank you! to everyone who participated, but especially to Richard Weait who has been a magnificent host, and Claus Rinner and Michael Morrish of Ryerson University who made sure we had a comfortable hack space.

Besides some Canadian Rye savvy – hence the current hangover – my main takeaway from this weekend is a renewed motivation to bring MapRoulette to the next level. Not only did we gain a new developer (Steve Singer, of PostgreSQL fame), we also discussed very important architectural challenges and made decisions that will allow us to move forward and make MapRoulette more flexible and extensible and easier to maintain. In the near future, it will be much easier to ‘plug in’ a new challenge, and to run more than one challenge in parallel. While we did write  some of the plumbing for this, the true value of the weekend lies not in getting a lot of code written, but in discussing development face-to-face. We now have a well-defined common operational picture, which will prove invaluable going forward.

Perhaps you would like to learn more about MapRoulette development, or the hack weekend in general? Or maybe you are wondering how you can contribute? Join the Mappy Hour tonight, March 11, at 5:30 PDT / 8:30 EDT, where we will talk about these topics among other things.

Get notified about OpenStreetMap changes in your area – on your phone


OpenStreetMap user Zverik built something really cool a little while ago: a map showing you recent OpenStreetMap editing activity in your area: WHODIDIT. We used to have a similar service, OWL or the OpenStreetMap watchlist, but that service has been down for a while and which has been down for a long time but is currently being resurrected by Paweł Paprota and others. In the mean time however, Zverik got tired enough of waiting to build something new. And it works really well – especially now that Simon Legner tweaked it to run even faster. Here’s what I see when I zoom in to my own area:

This is the default view which shows changes within the last week – you can choose a different horizon using a dropdown menu. The color of the cells indicate if the changes are primarily new things (green), changes (yellow) or deletions (red).

What makes WHODIDIT even more useful is that you can get an RSS link for your area of interest, giving you a feed of everything that happens with the OpenStreetMap area you care about. You can use this to just monitor an area for vandalism or irregularities, keep tabs on who is editing in your neck of the woods, or just to feel good about the amount of stuff that gets done.

There’s lots of ways to consume an RSS feed, and one that I particularly like is IFTTT or If This Than That. Their motto is ‘put the internet to work for you’ and that covers what it can do really well. The basic idea is that you connect some input trigger to a process leading to something happening elsewhere on the internet. Example: when the clock strikes midnight on December 31st, post ‘Happy New Year’ on my Facebook wall. But it can do actual useful stuff too. Just have a look at the shared recipes.

I created an IFTTT recipe to grab the OpenStreetMap change feed for an area from WHODUNNIT and post any new items to Pushover, which is a notification app for iOS and Android.

Use this recipe

Now I get the sound of a beer bottle being opened every time someone edits in the Salt Lake region! Yay!

New MapRoulette Challenge: Connectivity Bugs


When I presented the previous MapRoulette challenge, Zorro Ways, at State Of The Map US two weeks ago, I knew that it was not going to last us very long. There were only 2800 or so Zorro ways in the US, and those were fixed within days. MapRoulette has been sitting idle since, showing an annoying ‘more coming soon’ message. This eats at me. I was eager to get the next challenge up and running. I have been working on two fronts for MapRoulette challenges. First, I have been working with Telenav to get a reliable, fresh daily serving of connectivity bugs for the US (more on those in a bit). Because I had already designed some PL/PSQL functions to detect those, it was just a matter of getting some server capacity and scripting a nightly run, and Telenav kindly provided both.

Second, I have been in touch with KeepRight creator and maintainer Harald Kleiner to work on an interface from KeepRight to MapRoulette, and we’re getting close. That is really a super exciting step forward for MapRoulette, because KeepRight detects a huge number of map bugs in OpenStreetMap that are perfect for MapRoulette challenges. So once we work out the kinks, we will be able to go live with MapRoulette powered by KeepRight. This will also expand the scope of MapRoulette to the entire world, instead of the U.S. coverage I have been limited to so far.

Connectivity Bugs

So our current challenge is so fix connectivity bugs. Connectivity bugs are ways that end really close to another way (5 meters or closer). The distances are small enough for this type of map bug to not show up on the rendered map, but they are detrimental to routing. There are a lot of them, for many different reasons. Some of the connectivity bugs are human errors, and they are easy to make. Many of them, at least here in the U.S., are a result of TIGER import. You can see them in OSM Inspector as well as in KeepRight.

Let’s look at an example. Here’s how a connectivity bug would would be served up in MapRoulette:

Connectivity Bug in MapRoulette

Connectivity Bug in MapRoulette

Nothing in the map image points to a connectivity issue here. When I hit ‘EDIT IN JOSM’ (or the handy keyboard shortcut ‘e’), the area is loaded into JOSM. (Of course, Potlatch is also an option.)

Connectivity Bug in JOSM at standard zoom

Connectivity Bug in JOSM at standard zoom

Even in JOSM at the same zoom level, there is no apparent issue here. (Experienced JOSM editors will notice the absence of a link node, but that is easy to overlook.) We have to zoom way, way in to identify the issue, an overshoot:

The same connectivity bug in JOSM, visible when zoomed way in.

The same connectivity bug in JOSM, visible when zoomed way in.

These types of errors are easy to fix. JOSM has a ‘Join node to way’ function (shortcut key is ‘j’ by default) that snaps and joins the selected node to the closest way. This only works when the node is sufficiently quite close to the way, so you may want to move the node closer first. Be careful that you don’t join any other unrelated objects together though, like administrative boundaries. This is the situation after the fix:

Fixed!

Fixed!

Easy enough, huh? There’s around 68,000 of these buggers in the U.S. alone (which is what we cover currently) so let’s get fixing!

>>> To MapRoulette <<<

Oh, anyone care to design a cool & appropriate logo for MapRoulette? Submissions to m@rtijn.org

Binders Full Of TIGER Deserts


The U.S. has binders full of TIGER deserts.

Let me explain. Back in 2007, we imported TIGER/Line data from the U.S. Census into OpenStreetMap. TIGER/Line was and is pretty crappy geodata, never meant to make pretty maps with, let alone do frivolous things like routing. But we did it anyway, because it gave us more or less complete base data for the U.S. to work with. And so we worked. And worked and worked. Fixing spaghetti highways. Reclassifying and arguing about reclassifying. Connecting and disconnecting. And now, four years and change later, we have pretty maps!

Eat that, TIGER! (check it out)

But we don’t all live in Epcot Center (actually, I don’t think any of us do really) and there’s lots of places where we haven’t been taking as good care of the data. Vast expanses of U.S. territory where the majority of the data in OSM is still TIGER as it was imported all those years ago.

The TIGER deserts.

I want those all to go away, but there’s only so many folks mapping here in the U.S., so we may want to prioritize a little bit. So I wanted to look at the TIGER desert phenomenon a little closer. In particular, I wanted to look a the parts of TIGER deserts that cover towns, and even cities. That’s where people actually do live, and that’s where OSM data is most valuable.

So I set out to identify those the only way I know how: using ArcGIS. The thing is: this kind of job has ‘raster analysis’ plastered all over it, and I just don’t know how to do that using FOSS4G tools. So maybe I’ll just explain what I did in ArcGIS, and then you all can chime in with smart comments about how to do this in PostGIS, R, GRASS, QGIS or whatever free software can do this magic. If you don’t care for all that, just scroll to the bottom for the results.

I created a shapefile first with all OSM highways with TIGER tags in Florida using C++ and osmium. (There’s some good example code to get you started if you’re interested.)

Then, with that loaded into ArcMap, I first created a 5km grid with the predominant (in terms of way length) version number as the cell value.

A second grid for the neighborhood way density:

I reclassified the version grid generously – all cells with 1 or 2 a the predominant TIGER way version got true / 1, the rest false / 0. For distinguishing between built-up and boondock cells, a threshold of 1.8 looked good after some tweaking.

And finally some simple map algebra to combine the two variables into the final result grid:

So there we are folks – TIGER deserts and TIGER ghost towns in Florida:

TIGER deserts in Florida

TIGER ghost towns in Florida

Hmm. I hope we can figure out a way to improve this analysis so the situation does not look quite so bleak. GIS does not lie though – these are the 5km cells that have a reasonably high way density and TIGER way versions that are predominantly 1 or 2.

So let me know folks – 1) Is this a good approach for identifying TIGER ghost towns and if not, what is? and 2) how do you do this using FOSS4G tools?

Remap-A-Tron SOTM US Talk Slides


Here are my slides from my Remap-A-Tron talk at the amazing State Of The Map US conference from which I just returned. Without any context this is going to make very little sense, but for those of you who have seen Pulp Fiction more than a couple of times, it may still be worth a few minutes of your time. I will follow up with a more insightful post about this and SOTM soon.

The OpenStreetMap US Chapter – My Outlook For Next Year


It’s been almost a year since I was elected board member to the OpenStreetMap US Chapter. That means the next elections are just around the corner – actually, they are happening this weekend at State Of The Map US. If you’re a Chapter Member, vote! If not, it’s not too late to join.

I decided to run for re-election, and I want to take a moment to briefly reflect on the past year of serving on the board, as well as give my personal outlook for next year.

Reflection

My first year in the US community, and on the US Chapter Board, taught me a few important lessons. First, the US is a big place and for that reason, ‘community’ has a different meaning here than in the Netherlands where I come from. If you want to focus on things like ‘community support’ and ‘community development’, you need to define those things very differently here. More on that later.

Another important lesson I draw from my first year is that big plans are OK, but they need to be broken down into baby steps. The average board member doesn’t have a huge amount of time to invest, and as a result we don’t have the manpower to implement Big Things ourselves.

Finally, I feel that serving on the Chapter board comes without much of the public scrutiny and political constraints that the Foundation board has to deal with. I don’t know how I feel about this – on the one hand, it gives you more freedom to set your own agenda as a board, while on the other hand our mandate as a Chapter board is so flimsy (I was elected with something like a dozen votes) that it makes you think twice about what having a local Chapter actually means.

Outlook – My Manifesto

My most important personal objective in serving on the Chapter board was and remains supporting and growing the community, but I want to redefine that objective based on what I feel ‘community’ means here in the US. I am a strong believer in local communities that meet face to face, know each other, and get together and talk mapping over beers or coffee or shakes. I think there is far too little of that going on in the US, but I think the role of the Chapter Board in growing and building these local communities can only be fairly limited. You need active local mappers to build and sustain them, and we cannot conjure those up out of thin air. What we can do as a board is make these local communities more visible and give them a platform to present themselves and communicate. I want that platform to be our national web site: openstreetmap.us

The other dimension that defines us as the U.S. OpenStreetMap community is the map itself. We all strive to make it better, and this binds us together. I want to work actively to leverage the map, our data that we work together to improve and enhance, as a binding element for our community. The way to do that is to give the community projects to work on. That sounds mundane, but I want to take it beyond a ‘project of the week’. Taking inspiration from the Remap-a-tron that I built and ran, I want to provide the community with tools to work on specific problems. For example, the Remap-a-tron can be repurposed to work on things like connectivity errors, spaghetti ways, and TIGER deserts. I want openstreetmap.us to be the platform to organize these projects and report on their results. If you’re looking for something to map, head over to the web site!

In short, I want to transform openstreetmap.us into much more of a destination for mappers, both hardcore and casual. I want to be able to welcome new mappers by just saying: ‘hey, you want to start doing some useful mapping right away? Have a look at the web site!’.

Are we going to pull this off in the next  year? I don’t know, really. I would love to be able to say that we would, but I can’t even be sure of how much time I will be able to invest in it. There will also be State Of The Map US 2013 to organize, something I want to see happen as well. What I will commit to is investing at least 10 hours a month of my time on working towards these goals, and I believe this is a commitment that every candidate should be prepared to make for the board as a whole to be successful.

Google Maps and OpenStreetMap Data Views – Find The 10 Differences


Google Maps had The Atlantic over for a chat about how they work up their ‘deep map’ from various sources. It’s interesting to read about how Google invests incredible amounts of money and manpower to try and do the best job possible of capturing ground truth without people on the ground.

The article contains some ‘data views’ of Google Maps data in various stages of being worked up. I don’t know if it’s actual screenshots of an editing environment, but regardless, it’s an interesting peek behind the scenes that I had not had before.

This is from the article:

The Google Maps editing environment. Source: The Atlantic

This is about the same area loaded into the OpenStreetMap desktop editor JOSM:

The same area in the OpenStreetMap editor JOSM

Now you can look long and hard to try and make out ten or maybe a hundred differences in the data, but there’s one difference between these two views that reaches much deeper. The data behind Google Maps you will never get to see, let alone touch. The data in OpenStreetMap on the other hand is there for anyone to download, use, make great products out of and, most importantly, edit and improve. That difference marks a cardinal characteristic of the Google Maps platform that the article failed to raise. Consider that itch scratched.

Post-Redaction Repair Tools: Maps and the Remap-A-Tron


Although the OpenStreetMap data license change from CC-BY-SA to ODbL is not formally done, the redaction bot has done its thing. It was activated in mid-July after rigorous testing and months of development to go over the entire planet, square degree by square degree, and delete any tainted node, way and relation or revert it to a untainted state. (An object is tainted, simply put, when it was at some point edited by a contributor who did not agree to the license change and the new contributor terms.)

OpenStreetMap Redaction bot progress map screenshot taken on July 17th as the bot was just progressing to Switzerland and re-running London.

Although less than 1% of all data was removed in the redaction process, some areas are much more strongly affected than others, depending on the amount and previous activitiy of local ‘decliners’ as well as pre-emptive remapping of tainted features. This was made possible by tools that identified potentially tainted objects, such as Simon Poole’s Cleanmap and a JOSM plugin. (These tools have since been discontinued.)

Looking at the situation in the US, what we’re left with after the license change redaction is a map that has issues. Well, let’s say it has more issues now than it had before the redaction. Just looking at the street network, which is the one single most important dimension of OpenStreetMap if you ask me, we don’t have to look very hard to find missing  street segments, entire missing streets, and messy geometry. We had messy geometry before, because of the inherent messiness of TIGER/Line data off of which the US road network is based, but the redaction process left us with some really, ehm, interesting spaghetti in places.

Messy way geometries as a result of the redaction, near Los Angeles, CA.

Missing way segment as a result of the redaction, north of Honolulu, HI

And then there’s the invisible results of the redaction: lost attributes such as speed limits, lane count, direction of flow – but I would suggest we fix the road geometries and topological integrity of the main road network first, so the OpenStreetMap data is routable and the routes make sense. In order to do that effectively, the community needs tools. Fortunately, we have some available already. There’s Toby Murray’s map showing suspicious way segments last touched by the redaction bot. Kai Krueger provided a routing grid page that calculates route times and distances between major US cities, and compares those to reference data to expose potential routing issues. And there’s also the venerable OSM Inspector tool that provides a layer showing objects affected by the redaction process.

I also took a stab at a small set of tools that I hope will be helpful in identifying the remaining issues in the road network for the US: a Redacted / Deleted Ways map, and the Remap-A-Tron.

Redacted / Deleted Ways Map

Not unlike the redaction layer in OSM Inspector, this map duo exposes features affected by the redaction process, with a few notable differences. The main difference is that it has a clear focus on the US road network data. The Redacted Ways Map differentiates between Motorway, Trunk, Primary, Secondary and lower class roads, in an attempt to make it easier for remappers to seek out and prioritize the more important roads.

The Redacted Ways Map showing different road classes in appropriate coloring for easy prioritization of remapping efforts

The Deleted Ways map complements the Redacted Ways map. It only shows the way segments that were completely obliterated by the redaction bot. The focus on the road network means that only ways with a highway tag are visualized here.

The Deleted Ways map differentiates between ways that are likely to already have been remapped, and those that still need to be remapped.

If you look at this map, you will notice that there are two different classes of deleted ways, displaued in red and green respectively. The difference is in remapping status. The red ways are probably not remapped yet, while the green ways are likely to have been remapped already. I make this distinction by periodically comparing the deleted way geometries with newly mapped way geometries in the database. Using the Hausdorff distance algorithm as the main component, I devised an algorithm that reasonably accurately predicts whether a way has already been remapped.

Remap-A-Tron

I came up with another way to leverage the knowledge of remappedness of the OpenStreetMap street network. Building on an idea I already had semi-developed a while ago, I built a web application that serves up one deleted way geometry overlaid onto a current OpenStreetMap basemap. Using a simple menu or handy keyboard shortcuts, you can flag the segment as already remapped, skip it for whatever reason, or load that area into JOSM or Potlatch right away to remap it.

The Remap-A-Tron serves up one non-remapped important way segment at a time for remappers to work on.

The data backend is refreshed every (US) night, so there should not be too much of a lag. Currently, the app serves up only non-remapped important segments: motorways down to tertiary class roads. There are about 1900 segments in this category as of the time of writing. If a handful of people spend an evening with the Remap-A-Tron, this should be down to zero in a matter of days. Once we complete the repairs on the important roads, I can tweak the app to include all other highways (~15K segments not remapped) and ultimately all ways (~52K segments not remapped).

I built this tool with versatility in mind. After the street remapping is done, it could easily be repurposed to identify other issues in OSM that are identifiable on the Mapnik map: self-intersecting ways, road connectivity issues, missing bridge tags, maybe even missing traffic lights? I would love to hear your ideas.

If this turns out to be useful and used, I will try and increase the coverage to the entire world. For that to happen, the Remap-A-Tron will need to find a home that is not the server next to my desk, though..

Happy remapping!

Life After Redaction: Detecting Remapped Ways


There are some pretty awesome tools out there to help with the remapping effort after the redaction bot made its sweep across the OpenStreetMap database. (Does this sound like Latin to you? Read up on the license change and the redaction process here.) Geofabrik’s OSM Inspektor shows all the objects affected by the redaction. It is likely the most comprehensive view of the result of the license change redaction. Numerous other tools are listed on the Remapping wiki page. Most of these tools will show you, in some shape or form, the effects of the redaction process: which nodes, ways and relations have been deleted or reverted to a previous, ‘ODbL Clean’ version of the object.

I want to see if we can take it a step further and determine whether an object has already been remapped. This is useful for monitoring remapping progress as well as determining where to focus efforts when you want to contribute to the remapping effort.

For now, I am going to stick with ways. I think maintaining, or reinstating, a good quality routable road network is an important objective for OSM anyway, and especially at this point in time, when many roads are broken due to redaction.

Let’s start by locating a deleted way here in the US using my own Redaction Affected / Deleted Ways Map. That’s easy enough around severely affected Austin, TX:

 

I am going to use three comparison parameters to determine whether this way is likely to already have been remapped:

  1. The Hausdorff distance between the deleted geometry and any new geometries in that area
  2. The highway type of the deleted and any new geometries in that area
  3. The length of the deleted and any new geometries in that area

For this to work, I will need a table with all the ways deleted by the redaction bot. This is easy enough to compile by looking at the changesets created by the redaction account, but Frederik Ramm was kind enough to send the list of OSM IDs to me, so all I had to do is extract the deleted ways by ID from a pre-redaction database. The comparison can then be run on that table and a ways table from a current planet:

 

It is immediately clear that this way is very likely already remapped if we look at the top candidate, with object ID 172171755. It has a very small Hausdorff distance compared to the deleted way 30760760, it is tagged with the same highway= type, and the lengths are almost identical.

Sure enough, when fire up JOSM and load this area, it is clear that this area has been remapped:

 

(Selected objects are version 1 and created after July 18, 2012).

I need to do some more testing and tweaking on the query, but I will soon integrate this in the Redaction Affected / Deleted Ways Map.