mobile and gis dev notes » google

Nominatim (Reverse Geocoder) via PL/PYTHON for RubyOnRails Mapping App

rupert — Fri, 06 Jan 2012 04:16:17 +0000

Even google maps enterprise have restrictions on their geocoding/reverse-geocoding services, 100k if my memory serves me correctly. So, I have to rollout our own service to allow millions of lonlats for reverse geocoding. Have a look at Nominatim, yes it’s opensource. If you need to get it up and running, have a read of my nominatim installation via homebrew on OSX.

The nominatim www interface which spits out xml/json depending on the format parameter is done in php.

Anyway, I wanted to expose/use this webservice for our Rails3 app. It will also be good if we don’t use the nominatim webservice all the time if the lonlat was already requested–caching.

Python-Nominatim https://github.com/rdeguzman/python-nominatim

This project was forked from Austin’s Gabels python-nominatim. I added the ability to pass a base_url to the classes and added reverse_geocode.py. So assuming you have Python installed, you can do a reverse geocode like this…

from nominatim import ReverseGeocoder
client = ReverseGeocoder("http://127.0.0.1/nominatim/reverse.php?format=json")
response = client.geocode(-37.856206, 145.233980)
 
print response['full_address']
#Amesbury Avenue, Wantirna, City of Knox, 3152, Australia

PL/PYTHON
Now we wrap this python code via PL/PYTHON so Postgres can call it. Checkout setup.sql

CREATE PROCEDURAL LANGUAGE 'plpythonu' HANDLER plpython_call_handler;
 
CREATE OR REPLACE FUNCTION reverse_geocode(geocoding_url text, latitude float, longitude float) RETURNS
  text
  AS
  $$
    import nominatim
    client = nominatim.ReverseGeocoder(geocoding_url)
    response = client.geocode(latitude, longitude)
    RETURN response['full_address']
  $$
  LANGUAGE 'plpythonu';

With the snippet above, we can now call this with a regular SELECT statement…

SELECT reverse_geocode('http://127.0.0.1/nominatim/reverse.php?format=json', -37.856206, 145.233980); 
                       reverse_geocode
  ----------------------------------------------------------
   Amesbury Avenue, Wantirna, City of Knox, 3152, Australia
  (1 row)

Rails ActiveRecord

    create_table :locations do |t|
      t.float    :latitude
      t.float    :longitude
      t.text     :address
    end

In AR, I created a location model above and exposed a reverse_geocode method below

class Location < ActiveRecord::Base
 
  def self.reverse_geocode(geocode_url, lat, lon)
    sql_string = "SELECT reverse_geocode('#{geocode_url}', #{lat}, #{lon}) as address, #{lat} as latitude, #{lon} as longitude"
    loc_array = self.find_by_sql sql_string
    loc_array[0]
  end
 
end

So now, in one of my models, I could simply do..

class ActiveSession < ActiveRecord::Base
...
  def location_address
    if self.has_gps?
      loc = Location.reverse_geocode('http://path/to/reverse.php?format=json', self.gps_latitude, self.gps_longitude)
      loc.address
    else
      nil
    end
  end
...
end

In the view, we can simple call model.location_address to retrieve the location details. Below is a code snippet which creates a google marker and adds the location details in the infoWindow.

<% location = active_session.location_address %>
 
var latlong = new google.maps.LatLng(<%= active_session.gps_latitude %>, <%= active_session.gps_longitude %>);
 
var content = '';
content = content + '<%= escape_javascript location %>
';
content = content + '<%= active_session.gps_longitude %>,<%= active_session.gps_latitude %>
';

Caching
Our last step is to improve performance via caching. I have opted to do this from the PL/PYTHON end but using a Rails activerecord model/table. This way, the Rails activerecord has no idea that it is cached when it calls model.location_address. Below, I wrap the new reverse_geocode PL/PYTHON function in a rails migration.

class CreateFunctionReverseGeocoder < ActiveRecord::Migration
  ActiveRecord::Base.connection.schema_search_path = "public"
 
  def self.up
    execute 'CREATE OR REPLACE FUNCTION reverse_geocode(geocoding_url text, latitude float, longitude float) RETURNS
      text
      AS
      $$
        plan = plpy.prepare("SELECT address FROM locations WHERE latitude = $1 AND longitude = $2", [ "float", "float" ])
        rv = plpy.execute(plan, [ latitude, longitude ], 1)
 
        if rv.nrows() > 0:
          result = rv[0]["address"]
        else:
          import nominatim
          client = nominatim.ReverseGeocoder(geocoding_url)
          response = client.geocode(latitude, longitude)
          result = response["full_address"]
          insert_plan = plpy.prepare("INSERT INTO locations(latitude, longitude, address) VALUES($1, $2, $3)", ["float", "float", "text"])
          plpy.execute(insert_plan, [ latitude, longitude, result ])
 
        return result
      $$
      language plpythonu;'
  end
 
  def self.down
    execute 'DROP FUNCTION IF EXISTS reverse_geocode(text, double precision, double precision);'
  end
end

Benchmarks
I plotted 1000 records on my MBP (old core2duo early 2009 4GB RAM). Initial launch takes 108 seconds to load, ~ 2 minutes? But subsequent requests loads < 2 secs.

For 1000 records:
Completed 200 OK in 110478ms (Views: 1608.8ms | ActiveRecord: 108674.6ms)
Completed 200 OK in 1744ms (Views: 1110.7ms | ActiveRecord: 443.3ms)

Below is an architecture diagram of how the systems talk to each other. The locations cache is inside the geo_app_development db. Ofcourse, the nominatim database (gazetteer_au) is separate from our domain so it goes into a different db/server whereever.

Using GDALWARP to reproject raster that will fit with Google Satellite

rupert — Tue, 29 Jul 2008 12:48:21 +0000

Just a couple of notes to onself using gdal: Use gdalwarp to reproject your GeoTIFF files! I wanted to use my own satellite images acquired from GeoEye, however, on some areas I wanted to use google sat images as well since I don’t have the coverage. In order to do so, I need to reproject the sat images to 900913. Note you need to specify this in your epsg file in my previous post.

rupert:beijing_900913_satellite rupert$ gdalinfo Mosaic_RGB.tif
Driver: GTiff/GeoTIFF
Files: Mosaic_RGB.tif
Size is 4248, 4553
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235630016,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (116.291476140000000,40.025198500000002)
Pixel Size = (0.000046860000000,-0.000035970000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_XRESOLUTION=100
  TIFFTAG_YRESOLUTION=100
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 116.2914761,  40.0251985) (116d17'29.31"E, 40d 1'30.71"N)
Lower Left  ( 116.2914761,  39.8614271) (116d17'29.31"E, 39d51'41.14"N)
Upper Right ( 116.4905374,  40.0251985) (116d29'25.93"E, 40d 1'30.71"N)
Lower Right ( 116.4905374,  39.8614271) (116d29'25.93"E, 39d51'41.14"N)
Center      ( 116.3910068,  39.9433128) (116d23'27.62"E, 39d56'35.93"N)
Band 1 Block=4248x1 Type=Byte, ColorInterp=Red
Band 2 Block=4248x1 Type=Byte, ColorInterp=Green
Band 3 Block=4248x1 Type=Byte, ColorInterp=Blue

rupert:beijing_900913_satellite rupert$ gdalwarp -s_srs epsg:4326 -t_srs epsg:900913 Mosaic_RGB.tif sat_4m_rgb.tif
Creating output file that is 4245P x 4556L.
Processing input file Mosaic_RGB.tif.
0...10...20...30...40...50...60...70...80...90...100 - done.

rupert:beijing_900913_satellite rupert$ gdalinfo sat_4m_rgb.tif
Driver: GTiff/GeoTIFF
Files: sat_4m_rgb.tif
Size is 4245, 4556
Coordinate System is:
PROJCS["Google Maps Global Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.2572235630016,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]
Origin = (12945507.907502911984921,4869604.732793668285012)
Pixel Size = (5.219801430503303,-5.219801430503303)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (12945507.908, 4869604.733) (116d17'29.31"E, 40d12'53.10"N)
Lower Left  (12945507.908, 4845823.317) (116d17'29.31"E, 40d 3'2.78"N)
Upper Right (12967665.965, 4869604.733) (116d29'25.89"E, 40d12'53.10"N)
Lower Right (12967665.965, 4845823.317) (116d29'25.89"E, 40d 3'2.78"N)
Center      (12956586.936, 4857714.025) (116d23'27.60"E, 40d 7'58.12"N)
Band 1 Block=4245x1 Type=Byte, ColorInterp=Red
Band 2 Block=4245x1 Type=Byte, ColorInterp=Green
Band 3 Block=4245x1 Type=Byte, ColorInterp=Blue

Google Reader Links to your Blogroll

rupert — Thu, 14 Feb 2008 11:21:21 +0000

It is so eeeassy to publish blog links in your WordPress Blog from Google Reader using an OPML file. OPML is a file type that is widely used to distribute lists of RSS/newsfeeds. You could also grab other feeds and see it in Google Reader.

1. Login to your Google Reader with your Google Account.
2. Click on Settings.
3. Then Import/Export.
4. You will see “Export your subscriptions as an OPML file.”
5. Import to WordPress.

Reviving an old web map with Google Maps via OpenLayers

rupert — Tue, 22 Jan 2008 01:10:09 +0000

An old coworker and I worked on a travel portal for the Philippines called travelsite.ph about 4 years ago. We are now given a task of reviving the old web application and even adding mapping functionalities. Back then, the application was using ColdFusion 4.5 and MySQL 3.

Fingers crossed we dropped the app in a ColdFusion 6/7/8 environment with no changes at all. The app still works! Awesome.. how CF really progressed through the years with backward compatibility. The only changes we made was removing the registration/sign up for a quick demo. I just laughed at the oddities and the no brainer features (pertaining to security) that I made when I was starting out.

The database was also intact and have UTM coordinates. We dropped it to a Debian mysql 5 and works flawlessly since its MyISAM. I had the coordinates exported to lon/lat, so I could directly inject it to OpenLayers/Google. After two hours of fiddling around, I got mapping embedded.. hehe.. courtesy of OpenLayers ofcourse.

Here’s a quick reminder to myself…

A. Google WGS 84 Example.

	window.onload = function() {
		var options = {
					projection: "EPSG:4326",
					numZoomLevels: 19,
					maxExtent: new OpenLayers.Bounds(120.8774, 14.3684, 121.1628, 14.7931)
 
				};
 
		// avoid pink tiles
		OpenLayers.IMAGE_RELOAD_ATTEMPTS = 3;
		OpenLayers.Util.onImageLoadErrorColor = "transparent";
		map = new OpenLayers.Map('mapdiv',options);
		sat_wms = new OpenLayers.Layer.Google(
					"Layer",
					{type: G_SATELLITE_MAP}
		);
 
		map.addLayer(sat_wms);
 
		map.addControl(new OpenLayers.Control.MousePosition();
		map.addControl(new OpenLayers.Control.LayerSwitcher());
		map.addControl(new OpenLayers.Control.Scale());
 
		var center = new OpenLayers.LonLat(121.06504,14.65495);
		map.setCenter(center, 16);
	}

B. Google Mercator Projection

		window.onload = function() {
			var options = {
						projection: "EPSG:900913",
						units: "m",
						maxResolution: 156543.0339,
						numZoomLevels: 19,
						maxExtent: new OpenLayers.Bounds(12823075.86334, 4800551.12375, 13101918.14248, 5021301.26141)
					};
 
			// avoid pink tiles
			OpenLayers.IMAGE_RELOAD_ATTEMPTS = 3;
			OpenLayers.Util.onImageLoadErrorColor = "transparent";
			map = new OpenLayers.Map('mapdiv',options);
			sat_wms = new OpenLayers.Layer.Google(
						"Layer",
						{type: G_SATELLITE_MAP,'sphericalMercator': true}
			);
			map.addLayer(sat_wms);
 
			// start custom layer here
			var layer_obj = new OpenLayers.Layer.WMS(
				"Beijing",
				"http://127.0.0.1/cgi-bin/mapserv",
				{
					layers: 'beijing_all',
					map: '/home/map/beijing/new/beijing_google.map',
					format: 'AGG',
					'transparent': 'true'
				}
			);
			layer_obj.setIsBaseLayer(false);
			layer_obj.setVisibility(true);	
 
			map.addLayer(layer_obj);
 
			map.addControl(new OpenLayers.Control.MousePosition());
			map.addControl(new OpenLayers.Control.LayerSwitcher());
			map.addControl(new OpenLayers.Control.Scale());
			var center = new OpenLayers.LonLat(12956625.68367, 4852316.90682);
			map.setCenter(center, 18);
		}

What’s the difference between both snippets? Obviously projection is one. Since most of my point data is in lon/lat, then the WGS84 example is good if I don’t want to overlay custom precise data. Remember the x shift problem in Google with Openlayers. The Google Mercator example is used when I want to overlay more custom data, particularly polygons/line that needs to fit on Google Layers. For more details, please see my previous blog post.

OpenLayers + Google Spherical Mercator Example

rupert — Sat, 22 Dec 2007 11:42:18 +0000

I’ve been a dormant user of OpenLayers for months (4 months?) now and it was a surprise that the svn trunk had huge differences from what I remember OL (2.4/5?) to be. One of the cool features that and the OpenLayers community contributed was the Google Speherical Mercator hack. Below is a quick step tutorial on how I was able to overlay a custom WMS to Google (set as the baselayer). For this tutorial, I want to overlay a road layer on top of Google.

1. We need to convert our data to Google Projection (Spatial Reference System: 900913). This applies to whatever kind of data (mine is vector stored both in Mapinfo and PostGis) we have. For PostGis, we need to:

INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid, proj4text, srtext) VALUES ( 900913, 
 
'spatialreference.org', 900913, '+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 
 
+x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs', 'PROJCS
 
["unnamed",GEOGCS["unnamed ellipse",DATUM["unknown",SPHEROID["unnamed",6378137,0]],PRIMEM
 
["Greenwich",0],UNIT["degree",0.0174532925199433]],PROJECTION["Mercator_2SP"],PARAMETER
 
["standard_parallel_1",0],PARAMETER["central_meridian",0],PARAMETER
 
["false_easting",0],PARAMETER["false_northing",0],UNIT["Meter",1],EXTENSION
 
["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 
 
+units=m +nadgrids=@null +wktext  +no_defs"]]');
 
SELECT AddGeometryColumn('public','roads','the_geom_google',900913,'LINESTRING',2); 
 
UPDATE roads SET the_geom_google = Transform(the_geom, 900913);

2. MapFile Settings courtesy of SpatialReference: Google Projection

WEB
    #Other Web Config Settings goes here...
    "wms_srs"              "EPSG:900913"
END
 
PROJECTION
    "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"
END

3. By ensuring that Mapserver has the new 900913 projection, problems such as “msWMSLoadGetMapParams(): WMS server error. Invalid SRS given : SRS must be valid for all requested layers.” or “msProcessProjection(): Projection library error. no options found in ‘init’ file” will be avoided.

cd /ms4w/proj/nad/
gvim epsg
# GCS Voirol 1875 Degree
<104304> +proj=longlat +a=6378249.2 +b=6356514.999904194  no_defs <>
# GCS Voirol Unifie 1960 Degree
<104305> +proj=longlat +ellps=clrk80  no_defs <>
# Google Spherical Mercator
. . .
<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs

4. Below is an example WMS Request. Note: “SRS=EPSG 900913″ is added; TRANSPARENT=true not TRANSPARENT=on; Check your BBOX settings for the correct extent.

http://127.0.0.1/cgi-bin/mapserv?
LAYERS=beijing_all
&MAP=%2Fhome%2Fmap%2Fbeijing%2Fnew%2Fbeijing_google.map
&FORMAT=AGG
&TRANSPARENT=true
&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap
&STYLES=
&EXCEPTIONS=application%2Fvnd.ogc.se_inimage
&SRS=EPSG%3A900913
&BBOX=12956687.788758555,4852222.554861524,12956993.536871642,4852528.30297461
&WIDTH=256&HEIGHT=256

5. Requesting the WMS from OpenLayers.

	var options = {
			projection: "EPSG:900913",
			units: "m",
			//maxResolution: 156543.0339,
			numZoomLevels: 18,
			maxExtent: new OpenLayers.Bounds(12823075.86334, 4800551.12375, 13101918.14248, 021301.26141)
	};
 
	// avoid pink tiles
	OpenLayers.IMAGE_RELOAD_ATTEMPTS = 3;
	OpenLayers.Util.onImageLoadErrorColor = "transparent";
	map = new OpenLayers.Map('mapdiv',options);
	sat_wms = new OpenLayers.Layer.Google(
				"Layer",
				{type: G_SATELLITE_MAP,'sphericalMercator': true}
	);
	map.addLayer(sat_wms);
 
	// start custom layer here
	var layer_obj = new OpenLayers.Layer.WMS(
		"Beijing",
		"http://127.0.0.1/cgi-bin/mapserv",
		{
			layers: 'beijing_all',
			map: '/home/map/beijing/new/beijing_google.map',
			format: 'AGG',
			'transparent': 'true'
		}
	);
	layer_obj.setIsBaseLayer(false);
	layer_obj.setVisibility(true);	
 
        map.addLayer(layer_obj);
	map.addControl(new OpenLayers.Control.MousePosition());
	map.addControl(new OpenLayers.Control.LayerSwitcher());
	map.addControl(new OpenLayers.Control.Scale());
 
	var center = new OpenLayers.LonLat(12956625.68367, 4852316.90682);
	map.setCenter(center, 17);

Downloading Google Videos

rupert — Sat, 11 Aug 2007 15:29:22 +0000

Having troubles downloading google videos in china. I found this excellent post, http://www.isaacmao.com/meta/2006/07/how-to-view-google-video-locally-in.html”.

Trick here, is for your browser to use anonymous access. This is where TOR comes to the picture.

Follow the steps above in setting up TOR.
You should see two (2) icons on your system tray–[1] Privoxy and [2]Tor.
Go to D:\Vidalia Bundle\Torbutton\ and drag torbutton-1.0.4-fx+tb.xpi to Firefox to install the plugin. Restart Firefox.
To test your settings: Go to tor detector.

TOR-disabled:
219.142.136.4 (4.136.142.219.broad.bj.bj.dynamic.163data.com.cn)

TOR:
88.84.142.165 (v29465.1blu.de)

Now we can view google video from China.
To download the video, view source the file after the flash video has initialized
Look for: “googleplayer.swf?&videoUrl“, copy and paste the URL until you see “&messagesUrl\u003“
Now fire up your PL of choice and unescape the string


 

#URLDecode(temp)#

Once downloaded, you can view it from google’s video player. GoogleVideoPlayerSetup.exe

OpenLayers and Google Layer Zoom fix

rupert — Tue, 20 Feb 2007 11:32:18 +0000

I have finally fixed the disappearing layer on the layer switch when we try to overlay the layer on top of Google. I edited Google.js to reflect the ff:

 
    MAX_ZOOM_LEVEL: 22,
 
    /** Hardcode these resolutions so that they are more closely
     *   tied with the standard wms projection
     *
     * @final @type Array(float) */
/*    RESOLUTIONS: [1.40625,0.703125,0.3515625,0.17578125,0.087890625,0.0439453125,0.02197265625,0.010986328125,0.0054931640625,0.00274658203125,0.001373291015625,0.0006866455078125,0.00034332275390625,0.000171661376953125,0.0000858306884765625,0.00004291534423828125],*/
 
RESOLUTIONS:[1.40625,0.703125,0.3515625,0.17578125,0.087890625,0.0439453125,0.02197265625,0.010986328125,0.0054931640625,0.00274658203125,0.001373291015625,0.0006866455078125,0.00034332275390625,0.000171661376953125,0.0000858306884765625,0.00004291534423828125,0.0000214576721191140625,0.000107288360595703125,0.0000053644182978515625,0.000002682209014892578125,0.0000013411045074462890625,0.00000067055225372314453125],