Derick Rethans

In many recent MongoDB related presentations I have used OpenStreetMap data as basis for most of my examples. I wrote a script that imports OpenStreetMap (OSM) nodes, ways and to a lesser extend relations into MongoDB with a specific and optimal schema. I have written about this briefly before in Indexing Freeform-Tagged Data, but now MongoDB received numerous updates to geospatial indexes I find it warrants a new article.

In MongoDB 2.2 and before, the index type that MongoDB used was the 2d type, which was basically a two dimensional flat earth coordinate system index—with some spherical features built on top. MongoDB 2.4 has a new index type: 2sphere. Instead of just being able to index points described by x/y coordinates (longitude/latitude) it now has support for indexing points, line strings and polygons as defined by GeoJSON.

I have modified my import script to use those new types, and also added support for very simple multi-polygons that OpenStreetMap records through its relation tag. The script also creates an indexes on { l: '2dsphere' } (the GeoJson object), { ts: 1 } (the tags), and { ty: 1 } (the type).

The structure it converts a node from OpenStreetMap to looks like:

{
        "_id" : "n26486695",
        "ty" : NumberLong(1),
        "l" : {
                "type" : "Point",
                "coordinates" : [
                        -0.1580359,
                        51.4500055
                ]
        },
        "ts" : [
                "addr:housenumber=97",
                "addr:postcode=SW12 8NX",
                "addr:street=Nightingale Lane",
                "amenity=pub",
                "name=The Nightingale",
                "operator=Youngs",
                "source:name=photograph",
                "toilets=yes",
                "toilets:access=customers",
                "website=http://www.youngs.co.uk/pub-detail.asp?PubID=430"
        ],
        "m" : {
                "v" : NumberLong(5),
                "cs" : NumberLong(11229430),
                "uid" : NumberLong(652021),
                "ts" : NumberLong(1333911628)
        }
}

There are several sections that make up the document:

• _id: Is a combination of n and the OSM node id.

• ty: Is the type. For nodes, this is always 1.

• l: The point's location in GeoJson format. An OSM point is translated to a GeoJson Point feature with an array describing the longitude and latitude.

• ts: Are the tags that describe the node. Each tag is stored as a concatenation of its key and its value. This creates both a smaller index and it still allows for exact tag/value matches as well as matching against specfic keys through a regular expression match. For example, we could find the above document with:

  db.poiConcat.find( { ts: 'name=The Nightingale' } );

  And the index would also be used when we look for all amenities:

  db.poiConcat.find( { ts: /^amenity=/ } );

• m: Contains meta information that describes the node. The following fields are currently present:

  • v: The object's version. This is the version number of the version that was found in the imported file. There is always just one version per object.

  • cs: The changeset ID in which this object was last updated.

  • uid: The user ID of the OpenStreetMap contributor who uploaded the latest version.

  • ts: The Unix timestamp of when this object was last updated.

OpenStreetMap ways are stored in two different types. Unclosed ways are stored as GeoJSON LineStrings (think roads):

{
        "_id" : "w2423886",
        "ty" : NumberLong(2),
        "l" : {
                "type" : "LineString",
                "coordinates" : [
                        [ -0.1044769, 51.508462 ],
                        [ -0.1044093, 51.5106306 ],
                        [ -0.1044139, 51.5107814 ],
                        [ -0.104427, 51.5108453 ],
                        [ -0.1044459, 51.5109208 ],
                        [ -0.1045131, 51.5110686 ]
                ]
        },
…

And closed ways are stored as GeoJSON Polygons (think buildings and parks):

{
    "_id" : "w24257746",
    "ty" : NumberLong(2),
    "l" : {
        "type" : "Polygon",
        "coordinates" : [
            [
                [ -0.0745133, 51.560977 ],
                [ -0.0742252, 51.5609742 ],
                [ -0.0742308, 51.5606721 ],
                [ -0.0745217, 51.5606721 ],
                [ -0.0745133, 51.560977 ]
            ]
        ]
    },
    "ts" : [
        "amenity=park",
        "leisure=park",
        "name=Kynaston Gardens"
    ],
    "m" : {
        "v" : NumberLong(1),
        "cs" : NumberLong(357805),
        "uid" : NumberLong(5139),
        "ts" : NumberLong(1210169336)
    }
}

Both ways and areas (closed ways) will have a ty value of 2, as they both come from a way primitive as stored in OpenStreetMap.

The script is available on GitHub as part of the 3angle repository. The latest version is at https://raw.github.com/derickr/3angle/master/import-data.php and it also requires https://raw.github.com/derickr/3angle/master/classes.php for some GeoJSON helper classes and https://raw.github.com/derickr/3angle/master/config.php where you can set the database name and collection name (in my case, demo and poiConcat).

Map data © OpenStreetMap contributors (terms).

Yesterday, Google announced that they have made Google Map Maker available in the United Kingdom. Like OpenStreetMap it allows everybody to update and add things to the map. But there is one big difference: With MapMaker you don't get access to the data.

Fellow OpenStreetMapper, Richard Fairhurst, describes this as Crowd-serfing:

Crowd-serfing, n.: when a large corp uses crowd-sourced volunteering for its own financial gain, without giving back. See: @googlemapmaker.

I will never understand why people do work for a commercial company without getting any real benefit back: http://www.bbc.co.uk/news/business-21226623

Unfortunately, today's BBC coverage on the availability of Google MapMaker in the UK read more like a manual on MapMaker than a nicely unbiased piece on crowd-sourced maps. Only after one of the OpenStreetMappers reached out to the journalist that wrote the piece, they added some background on OpenStreetMap:

"The biggest problem with Google Map Maker is that anything people contribute may appear on Google's map, but only Google can get at the underlying data to be able to do anything else with it," said Chris Hill.

"If someone includes a Google map on their web site to show where their business is they may also be showing where their competitors are and they can't change that."

Sure, it's nice to have some roads on the Google Map, but you will never even have full access back to your data, unlike OpenStreetMap where you can download and work with all the data. Even nicer is that often, OpenStreetMap still has better maps than GoogleMaps - for example, have a look at this in North Korea: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=125.74677&lat=39.01863&zoom=14 . And closer to home in the United Kingdom, compare some of the hiking trails in the Peak District: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=-1.99876&lat=53.17827&zoom=15 . Even in places like London, the accuracy of the locations of addresses and points-of-interest (POIs) is often a lot better, as OpenStreetMap doesn't use web site scraping and post-code-centroid locations to place POIs: http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=-0.12405&lat=51.50862&zoom=18 . OpenStreetMap mostly relies on surveys, done by individuals (like you!) to verify things are actually there, aided a little by the availability of Bing Maps as background imagery.

One of the things that most people forget, is the terms and conditions that commercial entities state. An except from MapMaker's reads:

"You give Google a perpetual, irrevocable, worldwide, royalty-free, and non-exclusive licence to reproduce, adapt, modify, translate, publish, publicly perform, publicly display, distribute, and create derivative works of the user submission."

Note that it never mentions that you can do anything with the data yourself…

If you are contributing time and knowledge, why not allow yourself to benefit from it as well? I realise that OpenStreetMap might not be as accessible, and the map-tiles on their web site aren't the prettiest, but the real benefit is in the access to the raw data that makes up the images in the map. The OpenStreetMap wiki also has an article on this.

Access to the data allows you to do so many more things. From creating your own fancy map-styles, creating "washable, wearable, all-weather maps designed for the real outdoors" such as SplashMaps, to powering web-sites that show accessibility. If you don't like the way map data is rendered, you can produce something in your own style, just like Nike did with this campaign.

Also, there is nothing better if some of your handiwork shows up in a "best of OSM" poster :-) I would never spend my time adding data to Google's Maps. Instead, I prefer to contribute to OpenStreetMap and do awesome things with the data. In the meanwhile, OpenStreetMap continues to support humanitarian relief efforts in Mali and many other places.

If you're near London, come and join us this summer! (Other places will also run mapping parties).

I have always wanted to see Aurora Borealis (Northern Lights) and I never managed to see it the five years I lived in Norway. The Aurora is caused by the collision of energetic particles coming from the Sun with the atmosphere. Their typical green colour comes from the interaction of the particles with Oxygen atoms in the atmosphere. The auroral mechanism is better explained on Wikipedia. Because the particles are charged, they are directed towards the magnetic poles of the Earth.

Aurora gets more intense, and is visible on lower latitudes, when a geomagnetic storm is in progress. One of their causes is a coronal mass ejection (CME) on the Sun which sends huge quantities of matter and electromagnetic radiation out into space. When CMEs are directed towards Earth they cause a geomagnetic storm which increases the change of Aurora Borealis to occur.

The solar cycle, which is eleven years is a cycle in which the Sun's activity goes from very little activity to a lot of activity and back to very little. During solar maximum there is a lot higher chance of a CME to occur. Originally the most recent solar maximum was forecasted to be in 2010 or 2011 but more recent predictions expect it to be this autumn. For some unexpected reason, Auroras are strongest around the vernal and autumnal equinoxes.

With it being so close to both the vernal equinox and solar maximum we set off to Iceland in the hope to be able to see them. The original plan was to seek them out on the second night of our trip (March 18th) but just before we left I noticed through the SpaceWeather site that the Sun produced a solar flare as well as an Earth-bound CME. Instead of trying our luck the 2nd night we decided to book an extra excursion as chances where very high that we would see the Aurora on our first night (March 17th). The Aurora Buddy application that I had installed on my phone had constantly been warning me about high activity all through Sunday after all. The image above shows the extend of auroral activity on the night of the 17th. Just when we got on the bus to travel to some darker skies the Aurora already showed up in the sky visible through Reykjavik's city lights...

When we got to the viewing location, the sky was fully alight as you can see in this timelapse:

This timelapse shows the auroras over a three minute period with a picture taken every 5 seconds. At a frame rate of 3 frames per second this is sped up 15 times. The show lasted until about midnight, when we headed back to Reykjavik. Some more still photos are available on flickr.

The Golden Circle

Of course, Iceland has much more to offer than just the occasional show of Auroras and the next morning we set off on the Golden Circle—a trip past Iceland's touristic highlights. Our first stop was Þingvellir, the site of Alþingi, the first Icelandic parliament that was founded in 930. It is situated in a rift valley that marks the crest of the Mid-Atlantic Ridge and is part of Þingvellir National Park.

After Þingvellir we proceeded to the geyser Strokkur (Icelandic for "churn"). It's quite spectacular to see a whole lot of water being launched in the air every 4 to 8 minutes.

Our next stop was Gullfoss, a waterfall in the river Hvítá. I had visited Gullfoss in summer many many years ago and this trip's experience was quite different. It was so increadible windy and cold that we could hardly make it to view the waterfall. However, with the ice surrounding it it was quite beautiful:

Smoke and Wind

The last day of the trip consisted of exploring the Rejkjanes peninsula which is a large geothermic area. We visited the Kerið crater, the geothermal fields Seltún and Gunnuhver, the cliffs near Reykjanestá and the "Bridge between two continents". The colours of some of the landscapes were beautiful, but there was a strong cold wind almost everywhere which made us want to go to the car very fast most of the time. Something to re-explore in summer I suppose.

Blue Lagoon

After a good night's dinner, drinks and rest we spend the last morning of our trip with our bottoms in the Blue Lagoon to relax. A perfect ending to a quick, but gorgeous trip in Iceland. We'll be back!

For further photos and timelapses, please see my flickr set.