The TIGER versus OpenStreetMap Battle Grid, improved!


I reposted this story on my OpenStreetMap Diary. I think the diaries are a great asset to the OpenStreetMap community and they are not used as much as they should. I will try and post more of my OpenStreetMap-related stories there (as well).

If you follow the blog at openstreetmap.us, you will have heard about the Battle Grid. It is a map that shows you where recent TIGER data is different from OpenStreetMap data. Because TIGER has improved a lot over the years, and has kept up reasonably well with new road construction, a big difference between TIGER and OSM tells us that OSM likely needs some love. Here is how the battle grid looked until today:

Image

Cells with a lot of difference between TIGER and OSM are brighter, and as a simple way of prioritizing the cleanup and update work, I colored the cells that are within a Census CDP orange, and the rest green.

As of today the Battle Grid will look like this, instead:

Image

The brightest cells are still the most different. What is new is a color spectrum ranging from green to red, indicating how many people drive in and through each cell. This is based on Telenav logs. Because lots of people use Telenav apps such as Scout every day, it should be a fair representation of interestingness.

Let me show you a few examples of bright Battle Grid cells to whet your appetites.

Here’s a bright red cell in Greenville, NC:

Image

Look at this! Missing subdivisions, and poorly aligned streets. A mess!

Image

If I weren’t writing this blog post I’d be fixing this…

The green cells are usually no less, ehm, interesting. Here’s one in Saint Louis, MO:

Image

I guess someone had a plan for this area, and the someone with more money / power came along with a different plan, and nobody ever told Census:

Image

What I generally find is that the bright cells on the fringes of urban areas are most gratifying. These usually represent either poorly aligned OSM data, unmapped new subdivisions, or a combo of both.

Speaking of fringes, I think Atlanta has a great visual Battle Grid story:Image

The city itself is well mapped with very few bright cells. (And whatever there was is mostly black, so people have already marked them as done.) The fringes still show a lot of, well, let’s call it mapping potential!

What are your favorite Battle Grid finds? Share them below!

OpenStreetMap Chicago Hack Weekend Report


Last weekend I was in Chicago for the second N-American OSM hack weekend of the year. This one, for me and Serge, was entirely dedicated to MapRoulette. We wanted to get the next generation code as close to ready as possible for release – and we got pretty close I think! Here’s a summary of what we did:

  • Looked at user interface challenges with Tom MacWright, the main architect of the iD editor.
  • Finalized the data model and ORM code.
  • Debugged and improved OAuth authentication code.
  • Discussed and finalized the design for the API.
  • Wrote setup instructions for anyone wanting to deploy MapRoulette (which would mainly be other developers wanting to collaborate on the project)
  • Created an EC2 AMI (ami-995032f0) that has MapRoulette and all dependencies ready to go for quick deployment
  • Created a list of tasks and remaining issues.

We have now gotten to a point where we can share the code and hope to attract more developers, which I think overall is a major milestone.

Here is the public GitHub repository: https://github.com/emacsen/maproulette

To reiterate, the main objectives of this next generation are:

  • Make code and database model more transparent to enable more development collaboration
  • Support for regional challenges
  • Support for working on tasks in a defined area
  • Support for difficulty preferences
  • Enable external challenge providers to submit challenge data through an API
  • Enable linking with OSM accounts to better track user activity and leverage OSM account services such as settings storage

Below is an impression of daytime at the hack weekend:

2013-04-28 14.15.25

..and nighttime:

2013-04-26 23.14.35

New MapRoulette challenge: Lane Counts



MapRoulette is overdue for a new challenge, and we have one that should keep us busy for a while: adding numbers of lanes to the main roads in N-America. This information would be great to have for advanced visualization and transportation analysis purposes. And a lot of it is already there, so let’s just make it complete.

The challenge works as follows: You fire up MapRoulette as usual by going to maproulette.org. You are presented with one OSM way somewhere in North America that does not have a lanes=* tag. (We only care about secondary and higher to keep things useful and manageable, and we’re starting with the motorways and moving down, so you should see only motorways at first.)

Screen Shot 2013-04-06 at 1.26.55 PM

If you want to fix this road, click on ‘Edit in JOSM’ and wait for the way to load in JOSM. Make sure you have JOSM running, otherwise the button won’t work.

Screen Shot 2013-04-06 at 1.30.26 PM

To actually see the number of lanes of this road, you may need to zoom in a little.

Screen Shot 2013-04-06 at 1.32.36 PM

Aha! This road has two lanes. So we go ahead and add that as a tag to the road. Hit Alt-A (Option-A on a Mac) or click ‘Add’ in the Properties pane:

And enter the information in the tag dialog:

Screen Shot 2013-04-06 at 1.49.03 PM

Click OK.

Now upload your change, adding a brief changeset comment in which you should mention you used MapRoulette – for example by adding #maproulette:

Then, return to MapRoulette and indicate that you have fixed the problem!

Caveats

Not all cases are this simple. There’s a few things you need to look out for.

In some cases where the amount of data to load into JOSM would become too big, JOSM will only load the actual way that requires your attention, not the data for the entire area.

Here’s an example:

Screen Shot 2013-04-06 at 2.11.59 PM

If you zoom in and deselect the road, you can see that this road is marked as oneway.

Screen Shot 2013-04-06 at 2.16.46 PM

Because this looks like a dual carriageway road from the aerial imagery, there is probably another OSM way representing the other side of this road. If unsure, confirm by loading more data.

In these cases where only the actual way is loaded, you should also avoid moving or deleting nodes from the way, as these nodes may be shared with other ways you cannot see. Before you make any such changes, load more data.

Also, only enter the number of travel lanes and do not include merge lanes. Here’s an example:

 

Screen Shot 2013-04-11 at 3.58.11 PMThe right lane is a merge lane, and does not count towards the main road lane count. So in this case the way would get lanes=3. You can see that it’s a merge lane by looking at the context (pan around and you will see the ramp) as well as by the difference in lane markings.

 

 

Looking back on a fantastic OpenStreetMap hack weekend in Toronto


I am on my way back home from my first visit to Toronto, and my first ever visit to Canada – currently nursing a mild hangover with a Minnesota ale on a 4 hour layover in Minneapolis.

All of this is connected to the second OpenStreetMap Hack Weekend in Toronto I was lucky enough to be able to attend. I had an amazing time! Steve (new!), Serge and I were able to make great progress on MapRoulette, which is of course fantastic, and I thoroughly enjoyed hanging out with old friends and new connections. I want to extend a heartfelt thank you! to everyone who participated, but especially to Richard Weait who has been a magnificent host, and Claus Rinner and Michael Morrish of Ryerson University who made sure we had a comfortable hack space.

Besides some Canadian Rye savvy – hence the current hangover – my main takeaway from this weekend is a renewed motivation to bring MapRoulette to the next level. Not only did we gain a new developer (Steve Singer, of PostgreSQL fame), we also discussed very important architectural challenges and made decisions that will allow us to move forward and make MapRoulette more flexible and extensible and easier to maintain. In the near future, it will be much easier to ‘plug in’ a new challenge, and to run more than one challenge in parallel. While we did write  some of the plumbing for this, the true value of the weekend lies not in getting a lot of code written, but in discussing development face-to-face. We now have a well-defined common operational picture, which will prove invaluable going forward.

Perhaps you would like to learn more about MapRoulette development, or the hack weekend in general? Or maybe you are wondering how you can contribute? Join the Mappy Hour tonight, March 11, at 5:30 PDT / 8:30 EDT, where we will talk about these topics among other things.

Get notified about OpenStreetMap changes in your area – on your phone


OpenStreetMap user Zverik built something really cool a little while ago: a map showing you recent OpenStreetMap editing activity in your area: WHODIDIT. We used to have a similar service, OWL or the OpenStreetMap watchlist, but that service has been down for a while and which has been down for a long time but is currently being resurrected by Paweł Paprota and others. In the mean time however, Zverik got tired enough of waiting to build something new. And it works really well – especially now that Simon Legner tweaked it to run even faster. Here’s what I see when I zoom in to my own area:

This is the default view which shows changes within the last week – you can choose a different horizon using a dropdown menu. The color of the cells indicate if the changes are primarily new things (green), changes (yellow) or deletions (red).

What makes WHODIDIT even more useful is that you can get an RSS link for your area of interest, giving you a feed of everything that happens with the OpenStreetMap area you care about. You can use this to just monitor an area for vandalism or irregularities, keep tabs on who is editing in your neck of the woods, or just to feel good about the amount of stuff that gets done.

There’s lots of ways to consume an RSS feed, and one that I particularly like is IFTTT or If This Than That. Their motto is ‘put the internet to work for you’ and that covers what it can do really well. The basic idea is that you connect some input trigger to a process leading to something happening elsewhere on the internet. Example: when the clock strikes midnight on December 31st, post ‘Happy New Year’ on my Facebook wall. But it can do actual useful stuff too. Just have a look at the shared recipes.

I created an IFTTT recipe to grab the OpenStreetMap change feed for an area from WHODUNNIT and post any new items to Pushover, which is a notification app for iOS and Android.

Use this recipe

Now I get the sound of a beer bottle being opened every time someone edits in the Salt Lake region! Yay!

OpenStreetMap US Virtual Mappy Hour


The new OpenStreetMap US Chapter board has been using Google+ Hangouts for meetings, and it has been serving us well. Hangouts are like conference calls, but you can see each other and that is a huge advantage over just talking on the phone.

Google offers a more public variation on this theme called Hangouts On Air. Those are basically Hangouts anyone can tune in to via YouTube. Up to 9 folks at a time can hang out, all others get to watch. We (board) thought it would be cool to use this as a new discussion platform for the OpenStreetMap US Community. So we are going to try this out and host our first OpenStreetMap US Virtual Mappy Hour, next Monday, November 12, at 5:30PM Pacific, 8:30PM Eastern.

The link will be posted on the Mappy Hour Google+ Event page on the OpenStreetMap US Chapter Google+ page for details and come hang out (and likely iron out some kinks as we figure this out) next Monday.

As for discussion topics, I would love to talk about how to guide the temporary invasion of armchair
mappers in the upcoming Operation Cowboy. If there’s interest, I could also talk about what the board has been up to. But it’s not a formal meeting, so let’s just see where (and how long) it goes.

For those without google accounts: The YouTube channel is supposedly open to the public. If you want to say something, we can dial you in. Let me know in advance if you want me to do that.
Finally, please be aware that a recording of the hangout will remain available on YouTube for anyone to watch at any time.

See you at the Virtual Mappy Hour!

New MapRoulette Challenge: Connectivity Bugs


When I presented the previous MapRoulette challenge, Zorro Ways, at State Of The Map US two weeks ago, I knew that it was not going to last us very long. There were only 2800 or so Zorro ways in the US, and those were fixed within days. MapRoulette has been sitting idle since, showing an annoying ‘more coming soon’ message. This eats at me. I was eager to get the next challenge up and running. I have been working on two fronts for MapRoulette challenges. First, I have been working with Telenav to get a reliable, fresh daily serving of connectivity bugs for the US (more on those in a bit). Because I had already designed some PL/PSQL functions to detect those, it was just a matter of getting some server capacity and scripting a nightly run, and Telenav kindly provided both.

Second, I have been in touch with KeepRight creator and maintainer Harald Kleiner to work on an interface from KeepRight to MapRoulette, and we’re getting close. That is really a super exciting step forward for MapRoulette, because KeepRight detects a huge number of map bugs in OpenStreetMap that are perfect for MapRoulette challenges. So once we work out the kinks, we will be able to go live with MapRoulette powered by KeepRight. This will also expand the scope of MapRoulette to the entire world, instead of the U.S. coverage I have been limited to so far.

Connectivity Bugs

So our current challenge is so fix connectivity bugs. Connectivity bugs are ways that end really close to another way (5 meters or closer). The distances are small enough for this type of map bug to not show up on the rendered map, but they are detrimental to routing. There are a lot of them, for many different reasons. Some of the connectivity bugs are human errors, and they are easy to make. Many of them, at least here in the U.S., are a result of TIGER import. You can see them in OSM Inspector as well as in KeepRight.

Let’s look at an example. Here’s how a connectivity bug would would be served up in MapRoulette:

Connectivity Bug in MapRoulette

Connectivity Bug in MapRoulette

Nothing in the map image points to a connectivity issue here. When I hit ‘EDIT IN JOSM’ (or the handy keyboard shortcut ‘e’), the area is loaded into JOSM. (Of course, Potlatch is also an option.)

Connectivity Bug in JOSM at standard zoom

Connectivity Bug in JOSM at standard zoom

Even in JOSM at the same zoom level, there is no apparent issue here. (Experienced JOSM editors will notice the absence of a link node, but that is easy to overlook.) We have to zoom way, way in to identify the issue, an overshoot:

The same connectivity bug in JOSM, visible when zoomed way in.

The same connectivity bug in JOSM, visible when zoomed way in.

These types of errors are easy to fix. JOSM has a ‘Join node to way’ function (shortcut key is ‘j’ by default) that snaps and joins the selected node to the closest way. This only works when the node is sufficiently quite close to the way, so you may want to move the node closer first. Be careful that you don’t join any other unrelated objects together though, like administrative boundaries. This is the situation after the fix:

Fixed!

Fixed!

Easy enough, huh? There’s around 68,000 of these buggers in the U.S. alone (which is what we cover currently) so let’s get fixing!

>>> To MapRoulette <<<

Oh, anyone care to design a cool & appropriate logo for MapRoulette? Submissions to m@rtijn.org

Binders Full Of TIGER Deserts


The U.S. has binders full of TIGER deserts.

Let me explain. Back in 2007, we imported TIGER/Line data from the U.S. Census into OpenStreetMap. TIGER/Line was and is pretty crappy geodata, never meant to make pretty maps with, let alone do frivolous things like routing. But we did it anyway, because it gave us more or less complete base data for the U.S. to work with. And so we worked. And worked and worked. Fixing spaghetti highways. Reclassifying and arguing about reclassifying. Connecting and disconnecting. And now, four years and change later, we have pretty maps!

Eat that, TIGER! (check it out)

But we don’t all live in Epcot Center (actually, I don’t think any of us do really) and there’s lots of places where we haven’t been taking as good care of the data. Vast expanses of U.S. territory where the majority of the data in OSM is still TIGER as it was imported all those years ago.

The TIGER deserts.

I want those all to go away, but there’s only so many folks mapping here in the U.S., so we may want to prioritize a little bit. So I wanted to look at the TIGER desert phenomenon a little closer. In particular, I wanted to look a the parts of TIGER deserts that cover towns, and even cities. That’s where people actually do live, and that’s where OSM data is most valuable.

So I set out to identify those the only way I know how: using ArcGIS. The thing is: this kind of job has ‘raster analysis’ plastered all over it, and I just don’t know how to do that using FOSS4G tools. So maybe I’ll just explain what I did in ArcGIS, and then you all can chime in with smart comments about how to do this in PostGIS, R, GRASS, QGIS or whatever free software can do this magic. If you don’t care for all that, just scroll to the bottom for the results.

I created a shapefile first with all OSM highways with TIGER tags in Florida using C++ and osmium. (There’s some good example code to get you started if you’re interested.)

Then, with that loaded into ArcMap, I first created a 5km grid with the predominant (in terms of way length) version number as the cell value.

A second grid for the neighborhood way density:

I reclassified the version grid generously – all cells with 1 or 2 a the predominant TIGER way version got true / 1, the rest false / 0. For distinguishing between built-up and boondock cells, a threshold of 1.8 looked good after some tweaking.

And finally some simple map algebra to combine the two variables into the final result grid:

So there we are folks – TIGER deserts and TIGER ghost towns in Florida:

TIGER deserts in Florida

TIGER ghost towns in Florida

Hmm. I hope we can figure out a way to improve this analysis so the situation does not look quite so bleak. GIS does not lie though – these are the 5km cells that have a reasonably high way density and TIGER way versions that are predominantly 1 or 2.

So let me know folks – 1) Is this a good approach for identifying TIGER ghost towns and if not, what is? and 2) how do you do this using FOSS4G tools?

Un-Zorro-Tron – The 90 Second Video


Folks tell me that the Un-Zorro-Tron is less intuitive than its predecessor the Remap-A-Tron. So here is a 90 second video that shows you what a Zorro way is and what to do about them.

UN-ZORRO-TRON 90 sec instructions from rhodes on Vimeo.

If you don’t have 90 seconds, here’s a 60 second video:

If you don’t have 60 seconds, a picture may tell you more than 1,000 words:

Remap-A-Tron SOTM US Talk Slides


Here are my slides from my Remap-A-Tron talk at the amazing State Of The Map US conference from which I just returned. Without any context this is going to make very little sense, but for those of you who have seen Pulp Fiction more than a couple of times, it may still be worth a few minutes of your time. I will follow up with a more insightful post about this and SOTM soon.