mobile and gis dev notes » google

Nominatim (Reverse Geocoder) via PL/PYTHON for RubyOnRails Mapping App

rupert — Fri, 06 Jan 2012 04:16:17 +0000

Even google maps enterprise have restrictions on their geocoding/reverse-geocoding services, 100k if my memory serves me correctly. So, I have to rollout our own service to allow millions of lonlats for reverse geocoding. Have a look at Nominatim, yes it’s opensource. If you need to get it up and running, have a read of my nominatim installation via homebrew on OSX.

The nominatim www interface which spits out xml/json depending on the format parameter is done in php.

Anyway, I wanted to expose/use this webservice for our Rails3 app. It will also be good if we don’t use the nominatim webservice all the time if the lonlat was already requested–caching.

Python-Nominatim https://github.com/rdeguzman/python-nominatim

This project was forked from Austin’s Gabels python-nominatim. I added the ability to pass a base_url to the classes and added reverse_geocode.py. So assuming you have Python installed, you can do a reverse geocode like this…

from nominatim import ReverseGeocoder
client = ReverseGeocoder("http://127.0.0.1/nominatim/reverse.php?format=json")
response = client.geocode(-37.856206, 145.233980)
 
print response['full_address']
#Amesbury Avenue, Wantirna, City of Knox, 3152, Australia

PL/PYTHON
Now we wrap this python code via PL/PYTHON so Postgres can call it. Checkout setup.sql

CREATE PROCEDURAL LANGUAGE 'plpythonu' HANDLER plpython_call_handler;
 
CREATE OR REPLACE FUNCTION reverse_geocode(geocoding_url text, latitude float, longitude float) RETURNS
  text
  AS
  $$
    import nominatim
    client = nominatim.ReverseGeocoder(geocoding_url)
    response = client.geocode(latitude, longitude)
    RETURN response['full_address']
  $$
  LANGUAGE 'plpythonu';

With the snippet above, we can now call this with a regular SELECT statement…

SELECT reverse_geocode('http://127.0.0.1/nominatim/reverse.php?format=json', -37.856206, 145.233980); 
                       reverse_geocode
  ----------------------------------------------------------
   Amesbury Avenue, Wantirna, City of Knox, 3152, Australia
  (1 row)

Rails ActiveRecord

    create_table :locations do |t|
      t.float    :latitude
      t.float    :longitude
      t.text     :address
    end

In AR, I created a location model above and exposed a reverse_geocode method below

class Location < ActiveRecord::Base
 
  def self.reverse_geocode(geocode_url, lat, lon)
    sql_string = "SELECT reverse_geocode('#{geocode_url}', #{lat}, #{lon}) as address, #{lat} as latitude, #{lon} as longitude"
    loc_array = self.find_by_sql sql_string
    loc_array[0]
  end
 
end

So now, in one of my models, I could simply do..

class ActiveSession < ActiveRecord::Base
...
  def location_address
    if self.has_gps?
      loc = Location.reverse_geocode('http://path/to/reverse.php?format=json', self.gps_latitude, self.gps_longitude)
      loc.address
    else
      nil
    end
  end
...
end

In the view, we can simple call model.location_address to retrieve the location details. Below is a code snippet which creates a google marker and adds the location details in the infoWindow.

<% location = active_session.location_address %>
 
var latlong = new google.maps.LatLng(<%= active_session.gps_latitude %>, <%= active_session.gps_longitude %>);
 
var content = '';
content = content + '<%= escape_javascript location %>
';
content = content + '<%= active_session.gps_longitude %>,<%= active_session.gps_latitude %>
';

Caching
Our last step is to improve performance via caching. I have opted to do this from the PL/PYTHON end but using a Rails activerecord model/table. This way, the Rails activerecord has no idea that it is cached when it calls model.location_address. Below, I wrap the new reverse_geocode PL/PYTHON function in a rails migration.

class CreateFunctionReverseGeocoder < ActiveRecord::Migration
  ActiveRecord::Base.connection.schema_search_path = "public"
 
  def self.up
    execute 'CREATE OR REPLACE FUNCTION reverse_geocode(geocoding_url text, latitude float, longitude float) RETURNS
      text
      AS
      $$
        plan = plpy.prepare("SELECT address FROM locations WHERE latitude = $1 AND longitude = $2", [ "float", "float" ])
        rv = plpy.execute(plan, [ latitude, longitude ], 1)
 
        if rv.nrows() > 0:
          result = rv[0]["address"]
        else:
          import nominatim
          client = nominatim.ReverseGeocoder(geocoding_url)
          response = client.geocode(latitude, longitude)
          result = response["full_address"]
          insert_plan = plpy.prepare("INSERT INTO locations(latitude, longitude, address) VALUES($1, $2, $3)", ["float", "float", "text"])
          plpy.execute(insert_plan, [ latitude, longitude, result ])
 
        return result
      $$
      language plpythonu;'
  end
 
  def self.down
    execute 'DROP FUNCTION IF EXISTS reverse_geocode(text, double precision, double precision);'
  end
end

Benchmarks
I plotted 1000 records on my MBP (old core2duo early 2009 4GB RAM). Initial launch takes 108 seconds to load, ~ 2 minutes? But subsequent requests loads < 2 secs.

For 1000 records:
Completed 200 OK in 110478ms (Views: 1608.8ms | ActiveRecord: 108674.6ms)
Completed 200 OK in 1744ms (Views: 1110.7ms | ActiveRecord: 443.3ms)

Below is an architecture diagram of how the systems talk to each other. The locations cache is inside the geo_app_development db. Ofcourse, the nominatim database (gazetteer_au) is separate from our domain so it goes into a different db/server whereever.

OpenLayers + Google Spherical Mercator Example

rupert — Sat, 22 Dec 2007 11:42:18 +0000

I’ve been a dormant user of OpenLayers for months (4 months?) now and it was a surprise that the svn trunk had huge differences from what I remember OL (2.4/5?) to be. One of the cool features that and the OpenLayers community contributed was the Google Speherical Mercator hack. Below is a quick step tutorial on how I was able to overlay a custom WMS to Google (set as the baselayer). For this tutorial, I want to overlay a road layer on top of Google.

1. We need to convert our data to Google Projection (Spatial Reference System: 900913). This applies to whatever kind of data (mine is vector stored both in Mapinfo and PostGis) we have. For PostGis, we need to:

INSERT INTO spatial_ref_sys (srid, auth_name, auth_srid, proj4text, srtext) VALUES ( 900913, 
 
'spatialreference.org', 900913, '+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 
 
+x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs', 'PROJCS
 
["unnamed",GEOGCS["unnamed ellipse",DATUM["unknown",SPHEROID["unnamed",6378137,0]],PRIMEM
 
["Greenwich",0],UNIT["degree",0.0174532925199433]],PROJECTION["Mercator_2SP"],PARAMETER
 
["standard_parallel_1",0],PARAMETER["central_meridian",0],PARAMETER
 
["false_easting",0],PARAMETER["false_northing",0],UNIT["Meter",1],EXTENSION
 
["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 
 
+units=m +nadgrids=@null +wktext  +no_defs"]]');
 
SELECT AddGeometryColumn('public','roads','the_geom_google',900913,'LINESTRING',2); 
 
UPDATE roads SET the_geom_google = Transform(the_geom, 900913);

2. MapFile Settings courtesy of SpatialReference: Google Projection

WEB
    #Other Web Config Settings goes here...
    "wms_srs"              "EPSG:900913"
END
 
PROJECTION
    "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"
END

3. By ensuring that Mapserver has the new 900913 projection, problems such as “msWMSLoadGetMapParams(): WMS server error. Invalid SRS given : SRS must be valid for all requested layers.” or “msProcessProjection(): Projection library error. no options found in ‘init’ file” will be avoided.

cd /ms4w/proj/nad/
gvim epsg
# GCS Voirol 1875 Degree
<104304> +proj=longlat +a=6378249.2 +b=6356514.999904194  no_defs <>
# GCS Voirol Unifie 1960 Degree
<104305> +proj=longlat +ellps=clrk80  no_defs <>
# Google Spherical Mercator
. . .
<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs

4. Below is an example WMS Request. Note: “SRS=EPSG 900913″ is added; TRANSPARENT=true not TRANSPARENT=on; Check your BBOX settings for the correct extent.

http://127.0.0.1/cgi-bin/mapserv?
LAYERS=beijing_all
&MAP=%2Fhome%2Fmap%2Fbeijing%2Fnew%2Fbeijing_google.map
&FORMAT=AGG
&TRANSPARENT=true
&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap
&STYLES=
&EXCEPTIONS=application%2Fvnd.ogc.se_inimage
&SRS=EPSG%3A900913
&BBOX=12956687.788758555,4852222.554861524,12956993.536871642,4852528.30297461
&WIDTH=256&HEIGHT=256

5. Requesting the WMS from OpenLayers.

	var options = {
			projection: "EPSG:900913",
			units: "m",
			//maxResolution: 156543.0339,
			numZoomLevels: 18,
			maxExtent: new OpenLayers.Bounds(12823075.86334, 4800551.12375, 13101918.14248, 021301.26141)
	};
 
	// avoid pink tiles
	OpenLayers.IMAGE_RELOAD_ATTEMPTS = 3;
	OpenLayers.Util.onImageLoadErrorColor = "transparent";
	map = new OpenLayers.Map('mapdiv',options);
	sat_wms = new OpenLayers.Layer.Google(
				"Layer",
				{type: G_SATELLITE_MAP,'sphericalMercator': true}
	);
	map.addLayer(sat_wms);
 
	// start custom layer here
	var layer_obj = new OpenLayers.Layer.WMS(
		"Beijing",
		"http://127.0.0.1/cgi-bin/mapserv",
		{
			layers: 'beijing_all',
			map: '/home/map/beijing/new/beijing_google.map',
			format: 'AGG',
			'transparent': 'true'
		}
	);
	layer_obj.setIsBaseLayer(false);
	layer_obj.setVisibility(true);	
 
        map.addLayer(layer_obj);
	map.addControl(new OpenLayers.Control.MousePosition());
	map.addControl(new OpenLayers.Control.LayerSwitcher());
	map.addControl(new OpenLayers.Control.Scale());
 
	var center = new OpenLayers.LonLat(12956625.68367, 4852316.90682);
	map.setCenter(center, 17);

Downloading Google Videos

rupert — Sat, 11 Aug 2007 15:29:22 +0000

Having troubles downloading google videos in china. I found this excellent post, http://www.isaacmao.com/meta/2006/07/how-to-view-google-video-locally-in.html”.

Trick here, is for your browser to use anonymous access. This is where TOR comes to the picture.

Follow the steps above in setting up TOR.
You should see two (2) icons on your system tray–[1] Privoxy and [2]Tor.
Go to D:\Vidalia Bundle\Torbutton\ and drag torbutton-1.0.4-fx+tb.xpi to Firefox to install the plugin. Restart Firefox.
To test your settings: Go to tor detector.

TOR-disabled:
219.142.136.4 (4.136.142.219.broad.bj.bj.dynamic.163data.com.cn)

TOR:
88.84.142.165 (v29465.1blu.de)

Now we can view google video from China.
To download the video, view source the file after the flash video has initialized
Look for: “googleplayer.swf?&videoUrl“, copy and paste the URL until you see “&messagesUrl\u003“
Now fire up your PL of choice and unescape the string


 

#URLDecode(temp)#

Once downloaded, you can view it from google’s video player. GoogleVideoPlayerSetup.exe