mobile and gis dev notes » postgis

freebsd + postgresql-server + plpython + postgis

rupert — Wed, 16 Nov 2011 01:41:06 +0000

1. Install prerequisites

% cd /usr/ports/textproc/libxml2
% make
% make install

2. Install python

% cd /usr/ports/lang/python26
% make config #opens up blue terminal which allows to choose options
% make
% make install

3. Install

% cd /usr/ports/databases/postgresql90-server
% make
% make install clean

4. Initialize

% vim /etc/rc.conf
postgresql_enable=YES
postgresql_data=/var/db/pgsql
% mkdir /var/db/pgsql
% chown -Rf pgsql:pgsql /var/db/pgsql
% /usr/local/etc/rc.d/postgresql initdb -E utf8

5. Configuration
Allow incoming connections and set the default timezone to ‘UTC’

% vim /var/db/pgsql/postgresql.conf
listen_addresses = '*'     # what IP address(es) to listen on;
 
timezone = 'UTC' #not necessary

% vim /var/db/pgsql/pg_hba.conf
# your network
host    all             all             192.168.10.0/24          trust

6. Start/Stop

% /usr/local/etc/rc.d/postgresql start
% telnet 127.0.0.1 5432

7. Create user. Login as root then switch to pgsql user

% su - pgsql
[root@rupert ~]# su - pgsql
$ psql -d postgres
psql (9.0.6)
Type "help" for help.
 
postgres=# CREATE ROLE rupert WITH LOGIN PASSWORD '**********' SUPERUSER INHERIT CREATEDB CREATEROLE;
CREATE ROLE
postgres=#

8. Install plpython

% /usr/ports/databases/postgresql-plpython
% make
% make install clean

To test if plpython is working properly, we create a testdb and loadup plpythonu and call a plpython function.

[root@rupert ~]# createdb -U rupert testdb
[root@rupert ~]# psql -d testdb -U rupert
psql (9.0.6)
Type "help" for help.
 
testdb=# CREATE PROCEDURAL LANGUAGE 'plpythonu' HANDLER plpython_call_handler;
NOTICE:  using pg_pltemplate information instead of CREATE LANGUAGE parameters
CREATE LANGUAGE
testdb=# create or replace function pyver() returns text as
testdb-# $$
testdb$# import sys
testdb$# return sys.version
testdb$# $$ language 'plpythonu';
CREATE FUNCTION
testdb=# select pyver();
                   pyver                    
--------------------------------------------
 2.6.7 (r267:88850, Feb  6 2012, 13:10:39) +
 [GCC 4.2.1 20070831 patched [FreeBSD]]
(1 row)
 
testdb=#

9. Install postgis

% cd /usr/ports/databases/postgis
% make
% make install

Note that ports should install proj and geos.

10. Create template_postgis

% cd /usr/local/share/postgis
% su - pgsql
$ createdb -E utf8 template_postgis
$ psql -d template_postgis -f postgis.sql
$ psql -d template_postgis -f spatial_ref_sys.sql

11. Install adminpack module

This will eliminate “servers instrument error” when pgAdmin loads up postgres (default) db

% cd /usr/ports/databases/postgresql90-contrib
% make
% make install
% cd /usr/local/share/postgresql/contrib
% su - pgsql
$ psql -U pgsql -d postgres -f adminpack.sql

Nominatim + homebrew on OSX + OSM data + PHP = open sourced reverse geocoder

rupert — Mon, 14 Nov 2011 05:45:20 +0000

This installation guide (at the time of writing) was tested on SVN trunk of OSM2PGSQL and running on latest/stable Postgres/Postgis versions on OSX via homebrew.

1. Summary

OSX Snow Leopard
 
OSM2PGSQL: 
Head http://svn.openstreetmap.org/applications/utils/export/osm2pgsql Revision: 27034
Last Changed Author: frederik
Last Changed Rev: 27030
Last Changed Date: 2011-11-09 10:57:49 +1100 (Wed, 09 Nov 2011)
 
POSTGRES: 9.0.4
 
POSTGIS: "POSTGIS="1.5.3" GEOS="3.3.1-CAPI-1.7.1" PROJ="Rel. 4.7.1, 23 September 2009" LIBXML="2.7.3" USE_STATS" # SELECT POSTGIS_FULL_VERSION();
 
PHP: 5.3.8 (cli) (built: Nov 14 2011 14:41:52) #php -v

2. Installation
Most of the software is installed via homebrew.

# Install homebrew
% /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"
 
# Install postgresql
% brew install postgresql
% initdb -E utf8 -D /usr/local/var/postgres
% cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/
% launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist
% psql -d postgres -f /usr/local/Cellar/postgresql/9.0.4/share/postgresql/contrib/adminpack.sql
 
# Install postgis
% brew install proj
% brew install geos
% brew install postgis
 
# Create template_postgis_osm
% createdb -E utf8 template_postgis_osm
% psql -d template_postgis_osm -f "/usr/local/Cellar/postgresql/9.0.4/share/postgresql/contrib/pg_trgm.sql"
% psql -d template_postgis_osm -f /usr/local/Cellar/postgis/1.5.3/share/postgis/postgis.sql
% psql -d template_postgis_osm -f /usr/local/Cellar/postgis/1.5.3/share/postgis/spatial_ref_sys.sql
 
# Install osm2pgsql. Can skip this.
# % brew install osm2pgsql

For detail instructions on installing Postgres/Postgis via Homebrew, read this homebrew + postgresql9.0.4 + postgis.1.5.3 + proj4 + geos3.3.1 + osm2pgsql. If you are having problems installing GEOS, then read that link as it shows you how to upgrade GEOS to 3.3.1.

OSM2PGSQL needs GEOS as well. Note that brew only install the osm2pgsql binary. Don’t worry, we will compile this via source later.

3. More Installation.

We need to get PHP installed to run gazetteer http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/gazetteer/website/

# Install PHP
% brew install php --with-mysql --with-pgsql --with-apache
 
# Hookup with Apache
# Edit httpd.conf to LoadModule
 
# Install PEAR DB
% pear install db

4. OSM2PGSQL
Read this wiki: http://wiki.openstreetmap.org/wiki/Osm2pgsql. Well, we eventually need the whole OSM2PGSQL source as it contains the website (gazetteer).

% svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql osm2pgsql
% cd osm2pgsql
% ./autogen.sh
% ./configure
% make
 
# At this point there should be an osm2pgsql binary.

We need to compile gazetteer for gazetteer.so which is used by gazetteer-functions.sql

gis/osm2pgsql/gazetteer% make clean
gis/osm2pgsql/gazetteer% make 
gis/osm2pgsql/gazetteer% make install
test -z "/usr/local/lib/osm2pgsql" || .././install-sh -c -d "/usr/local/lib/osm2pgsql"
 /bin/sh ../libtool --mode=install /usr/bin/install -c  'gazetteer.la' '/usr/local/lib/osm2pgsql/gazetteer.la'
libtool: install: /usr/bin/install -c .libs/gazetteer.so /usr/local/lib/osm2pgsql/gazetteer.so
libtool: install: /usr/bin/install -c .libs/gazetteer.lai /usr/local/lib/osm2pgsql/gazetteer.la
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/local/lib/osm2pgsql
 
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `DYLD_LIBRARY_PATH' environment variable
     during execution
 
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
test -z "/usr/local/share/gazetteer" || .././install-sh -c -d "/usr/local/share/gazetteer"
 /usr/bin/install -c -m 644 'extract_countrynames.sql' '/usr/local/share/gazetteer/extract_countrynames.sql'
 /usr/bin/install -c -m 644 'gazetteer-index.sql' '/usr/local/share/gazetteer/gazetteer-index.sql'
 /usr/bin/install -c -m 644 'gazetteer-loaddata.sql' '/usr/local/share/gazetteer/gazetteer-loaddata.sql'
 /usr/bin/install -c -m 644 'gazetteer-tables.sql' '/usr/local/share/gazetteer/gazetteer-tables.sql'
 /usr/bin/install -c -m 644 'import_country_name.sql' '/usr/local/share/gazetteer/import_country_name.sql'
 /usr/bin/install -c -m 644 'import_country_osm_grid.sql' '/usr/local/share/gazetteer/import_country_osm_grid.sql'
 /usr/bin/install -c -m 644 'import_gb_postcodearea.sql' '/usr/local/share/gazetteer/import_gb_postcodearea.sql'
 /usr/bin/install -c -m 644 'import_gb_postcode.sql' '/usr/local/share/gazetteer/import_gb_postcode.sql'
 /usr/bin/install -c -m 644 'import_specialwords.sql' '/usr/local/share/gazetteer/import_specialwords.sql'
 /usr/bin/install -c -m 644 'import_us_statecounty.sql' '/usr/local/share/gazetteer/import_us_statecounty.sql'
 /usr/bin/install -c -m 644 'import_us_state.sql' '/usr/local/share/gazetteer/import_us_state.sql'
 /usr/bin/install -c -m 644 'import_worldboundaries.sql' '/usr/local/share/gazetteer/import_worldboundaries.sql'
 /usr/bin/install -c -m 644 'gazetteer-functions.sql' '/usr/local/share/gazetteer/gazetteer-functions.sql'

5. Download Data
You can get some regional OSM data from cloudmade. http://downloads.cloudmade.com/oceania/australia_and_new_zealand/australia/victoria

I suggest you download a regional extract prior to downloading/testing with the whole planet-osm. If you don’t believe me that it will take long, you can read http://wiki.openstreetmap.org/wiki/Nominatim/Installation

6. Load and Index Data
Basically, this is the summary of commands taken from http://wiki.openstreetmap.org/wiki/Nominatim/Installation At the time of writing this, I had issues such as “planet_osm_ways” (and several tables) does not exist. So I did a pg_dump and restored the tables afterwards. Be very careful with using the script below, you can comment the indexing part just to speed up on loading and see if you have errors, etc.

DATABASE_NAME=gazetteer_vic
OSM2PGSQL_HOME=/Users/rupert/projects/gis/osm2pgsql
SOURCE_DATA=/Users/rupert/Desktop/australia/victoria.osm
DUMP_DIR=/Users/rupert/Desktop/pg_dumps/streetlookup
 
dropdb $DATABASE_NAME 
#dropuser www-data
 
createdb $DATABASE_NAME -E UTF8 -T template_postgis_osm
createuser -SDR www-data
 
# This will create the planet_osm_ways, etc
$OSM2PGSQL_HOME/osm2pgsql --create --latlong --database $DATABASE_NAME --username rupert --slim --prefix planet_osm --cache 2048 $SOURCE_DATA
 
pg_dump --host 127.0.0.1 --port 5432 --username rupert --format custom --file "$DUMP_DIR/planet_osm_ways.backup" --table public.planet_osm_ways $DATABASE_NAME
pg_dump --host 127.0.0.1 --port 5432 --username rupert --format custom --file "$DUMP_DIR/planet_osm_nodes.backup" --table public.planet_osm_nodes $DATABASE_NAME
pg_dump --host 127.0.0.1 --port 5432 --username rupert --format custom --file "$DUMP_DIR/planet_osm_rels.backup" --table public.planet_osm_rels $DATABASE_NAME
 
# This will create the place table
$OSM2PGSQL_HOME/osm2pgsql --latlong -O gazetteer --database $DATABASE_NAME --username rupert --slim --prefix planet_osm --cache 2048 $SOURCE_DATA
 
pg_restore --host 127.0.0.1 --port 5432 --username rupert --dbname $DATABASE_NAME "$DUMP_DIR/planet_osm_ways.backup"
pg_restore --host 127.0.0.1 --port 5432 --username rupert --dbname $DATABASE_NAME "$DUMP_DIR/planet_osm_nodes.backup"
pg_restore --host 127.0.0.1 --port 5432 --username rupert --dbname $DATABASE_NAME "$DUMP_DIR/planet_osm_rels.backup"
 
rm -Rf $DUMP_DIR/*.backup
 
psql -d $DATABASE_NAME -q -f import_country_osm_grid.sql
psql -d $DATABASE_NAME -q -f import_worldboundaries.sql
psql -d $DATABASE_NAME -q -f import_country_name.sql
psql -d $DATABASE_NAME -q -f import_gb_postcode.sql
psql -d $DATABASE_NAME -q -f import_gb_postcodearea.sql
psql -d $DATABASE_NAME -q -f import_us_state.sql
psql -d $DATABASE_NAME -q -f import_us_statecounty.sql
 
psql -d $DATABASE_NAME -f gazetteer-functions.sql
 
psql -d $DATABASE_NAME -f gazetteer-tables.sql
 
psql -d $DATABASE_NAME -f gazetteer-functions.sql
 
psql -d $DATABASE_NAME -f gazetteer-loaddata.sql
 
#Indexing
psql -d $DATABASE_NAME -f gazetteer-index.sql

Save this as run.sh in /Users/rupert/projects/gis/osm2pgsql/gazetteer

Where do you run this?

% cd /Users/rupert/projects/gis/osm2pgsql/gazetteer
% sh run.sh

6. Test
If you are successful, you should have a “placex” table. Now that we have a postgis database running, you can now run spatial statements thru pgadmin. See the guts of reverse.php

SELECT * 
FROM placex
WHERE ST_DWithin( ST_SetSRID(ST_Point(145.234377, -37.856320),4326), geometry, 0.0001)
AND ST_GeometryType(geometry) NOT IN ('ST_Polygon', 'ST_MultiPolygon')

This one took only 21 ms.

7. Website

Make sure www-data have permissions to the tables. Rememeber to replace gazetteer_vic with your DATABASE_NAME.

for tbl in `psql -qAt -c "select tablename from pg_tables where schemaname = 'public';" gazetteer_vic` ; do psql -c "alter table $tbl owner to \"www-data\"" gazetteer_vic; done

Assuming you have PHP and PEAR DB installed. Then update the data connection settings found in http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/gazetteer/website/.htlib/settings.php

Run the same query but using reverse.php.

http://127.0.0.1/nominatim/reverse.php?format=xml&lat=-37.856320&lon=145.234377&zoom=18&addressdetails=1

osx + homebrew + postgresql9.0.4 + postgis.1.5.3 + proj4 + geos3.3.1 + osm2pgsql

rupert — Mon, 14 Nov 2011 02:02:21 +0000

For the impatient

% /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"
% brew install postgresql
% initdb -E utf8 -D /usr/local/var/postgres
% cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/
% launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist
% psql -d postgres -f /usr/local/Cellar/postgresql/9.0.4/share/postgresql/contrib/adminpack.sql
% brew install proj
% brew install geos %you should really read below before running this
% brew install postgis
% createdb -E utf8 template_postgis
% psql -d template_postgis -f /usr/local/Cellar/postgis/1.5.3/share/postgis/postgis.sql
% psql -d template_postgis -f /usr/local/Cellar/postgis/1.5.3/share/postgis/spatial_ref_sys.sql

1. Install homebrew

%open https://github.com/mxcl/homebrew
%open https://github.com/mxcl/homebrew/wiki/Installation
%/usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

Install python first if you need PL/Python in postgres. The –framework option tells Homebrew to compile a Framework-style Python build, rather than a UNIX-style build. Read this INSTALLATION GUIDE.

%brew install python --framework

2. Install Postgres
Note that postgres installation will pick up the latest Python Configuration, so it is important to ensure that we want the specific PYTHON interpreter to compile with our postgresql installation. You can turn the flag -v to enable verbose installation. Watch for the configuration log output for PYTHON to check if it loads the PYTHON environment that we expect

%PYTHON=/usr/local//bin/python brew install postgresql -v
....
checking for python... /usr/local/bin/python
checking for Python distutils module... yes
checking Python configuration directory... /usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config
...
If builds of PostgreSQL 9 are failing and you have version 8.x installed,
you may need to remove the previous version first. See:
  https://github.com/mxcl/homebrew/issues/issue/2510
 
To build plpython against a specific Python, set PYTHON prior to brewing:
  PYTHON=/usr/local/bin/python  brew install postgresql
See:
  http://www.postgresql.org/docs/9.0/static/install-procedure.html
 
 
If this is your first install, create a database with:
  initdb /usr/local/var/postgres
 
If this is your first install, automatically load on login with:
  mkdir -p ~/Library/LaunchAgents
  cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/
  launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist
 
If this is an upgrade and you already have the org.postgresql.postgres.plist loaded:
  launchctl unload -w ~/Library/LaunchAgents/org.postgresql.postgres.plist
  cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/
  launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist
 
Or start manually with:
  pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
 
And stop with:
  pg_ctl -D /usr/local/var/postgres stop -s -m fast
 
 
Some machines may require provisioning of shared memory:
  http://www.postgresql.org/docs/current/static/kernel-resources.html%SYSVIPC
 
If you want to install the postgres gem, including ARCHFLAGS is recommended:
    env ARCHFLAGS="-arch x86_64" gem install pg
 
To install gems without sudo, see the Homebrew wiki.
==> Summary
/usr/local/Cellar/postgresql/9.0.4: 2577 files, 35M, built in 3.1 minutes
brew install postgresql  188.73s user 62.38s system 106% cpu 3:55.06 total

3. Postgres Post Installation. Initialize DB.

% initdb -E utf8 -D /usr/local/var/postgres
The files belonging to this database system will be owned by user "rupert".
This user must also own the server process.
 
The database cluster will be initialized with locale en_AU.UTF-8.
The default text search configuration will be set to "english".
 
creating directory /usr/local/var/postgres ... ok
creating subdirectories ... ok
selecting default max_connections ... 20
selecting default shared_buffers ... 2400kB
creating configuration files ... ok
creating template1 database in /usr/local/var/postgres/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
 
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.
 
Success. You can now start the database server using:
 
    postgres -D /usr/local/var/postgres
or
    pg_ctl -D /usr/local/var/postgres -l logfile start

4. Postgres Startup

% cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/
% launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist
%telnet 127.0.0.1 5432
Trying 127.0.0.1...
Connected to rupert-mbp.
Escape character is '^]'.
% psql -d postgres -f /usr/local/Cellar/postgresql/9.0.4/share/postgresql/contrib/adminpack.sql

UPDATE:
I have now disabled the automatic startup of Postgres as it provides me more control especially during development. Extract Postgres9.1.2.tar.gz to /Library/StartupItems.

To start/stop

/Library/StartupItems/Postgres9.1.2/Postgres9.1.2 start

5. Postgis
Note this will install dependencies, PROJ4 and GEOS. At the time of writing this, we have a problem with GEOS. Need to update GEOS formula to 3.3.1. For more info read:

% open https://github.com/mxcl/homebrew/issues/8151
% open https://gist.github.com/1306088
% brew edit geos

% brew install proj
% brew install geos
% brew install postgis
% createdb -E utf8 template_postgis
% psql -d template_postgis -f /usr/local/Cellar/postgis/1.5.3/share/postgis/postgis.sql
% psql -d template_postgis -f /usr/local/Cellar/postgis/1.5.3/share/postgis/spatial_ref_sys.sql

6. osm2pgsql

~% brew install osm2pgsql
==> Checking out http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
==> ./autogen.sh
==> ./configure
==> make
/usr/local/Cellar/osm2pgsql/HEAD: 6 files, 328K, built in 70 seconds

7. Check for loaded/configured PLPYTHON environment for POSTGRES
If you get import module errors, you should check your PYTHON version and if the module is loaded in your python’s site-packages. The function below will provide which version of PYTHON is compiled with your POSTGRESQL.

CREATE PROCEDURAL LANGUAGE
'plpythonu' HANDLER plpython_call_handler;
 
CREATE OR REPLACE FUNCTION pyver() returns text AS
$$
import sys
RETURN sys.version
$$ LANGUAGE 'plpythonu';
 
SELECT pyver();
"2.7.2 (default, Dec 30 2011, 09:25:27) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00)]"

Installing Postgres8.3,8.4,9.0, Postgis1.3.3,1.4.1,2.0, pgRouting on OSX (updated)

rupert — Tue, 15 Feb 2011 03:31:08 +0000

(Updated):Postgres9.0.3 Postgis2.0svn

Before you go any further. Dump data before migrating and installing. Warning: Most of the packages listed below is installed by source.

1. Download the current postgres source. http://www.postgresql.org/ftp/source/

8.3:
$./configure --with-prefix=/usr/local/pgsql --with-python
 
8.4:
$./configure --prefix=/usr/local/pgsql --with-python
 
$make
$sudo make install

2. Don’t delete the postgres folder. You might need this later on for future compilations. See pgadmin3 admin pack below.

3. Add a postgres user from System Preferences > Accounts

4. Initialize the data directory

$mkdir /usr/local/pgsql/data
$chown postgres /usr/local/pgsql/data/
$su - postgres
$/usr/local/pgsql/bin/initdb -E utf8 -D /usr/local/pgsql/data

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.
The default text search configuration will be set to "english".

fixing permissions on existing directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 20
selecting default shared_buffers/max_fsm_pages ... 2400kB/20000
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

Success. You can now start the database server using:

/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data
or
/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start

4. Install Geos , PROJ4, and Postgis

GEOS:

- install geos with the Kyngchaos installer

- installing by source

svn checkout http://svn.osgeo.org/geos/trunk trunk
cd trunk/
sh autogen.sh
./configure --prefix=/usr/local/geos --enable-python
make
make install
sudo make install

PROJ4:

svn co http://svn.osgeo.org/metacrs/proj/trunk/proj/ proj
cd proj
sh autogen.sh
./configure --prefix=/usr/local/proj4
make
sudo make install

POSTGIS:

8.3:
$./configure --with-pgsql=/usr/local/pgsql/bin/pg_config --with-geos=/usr/local/bin/geos-config
 
8.4 using Kyngchaos GEOS Framework:
$./configure --with-pgsql=/usr/local/pgsql/bin/pg_config --with-geosconfig=/Library/Frameworks/GEOS.framework/unix/bin/geos-config
 
9.0:using Kyngchaos GEOS + PROJ framework:(Update Feb 15, 2011)
./configure --with-pgsql=/usr/local/pgsql/bin/pg_config --with-geosconfig=/Library/Frameworks/GEOS.framework/unix/bin/geos-config --with-projdir=/Library/Frameworks/PROJ.framework/unix/ --with-gettext=/usr/local/share/gettext/
 
Note: libintl.h error. You need gettext
gcc -g -O2  -fno-common -DPIC  -Wall -Wmissing-prototypes  -DUSE_NLS -DLOCALEDIR=\"/usr/local/pgsql/share/locale\" -Iyes/include  -c shp2pgsql-core.c
In file included from shp2pgsql-core.c:15:
shp2pgsql-core.h:17:21: error: libintl.h: No such file or di
 
9.0.4: using src from GEOS + PROJ
./configure --with-pgconfig=/usr/local/pgsql/bin/pg_config --with-geosconfig=/usr/local/geos/bin/geos-config --with-projdir=/usr/local/proj4
 
$make
$sudo make install

If everthing is successful, you should see files in /usr/local/pgsql/share/contrib/.

cd /usr/local/pgsql/share/contrib/postgis-2.0
- legacy.sql
- postgis.sql
- postgis_upgrade_20_minor.sql
- spatial_ref_sys.sql
- uninstall_legacy.sql
- uninstall_postgis.sql

5. Starting postgres on boot. Download postgresstartup.tar.gz then extract to your /Library/StartupItems

6. Creating the database

createdb -E utf8 template_postgis
createlang plpgsql template_postgis
 
8.3:
psql -d template_postgis -f /usr/local/pgsql/share/lwpostgis.sql
psql -d template_postgis -f /usr/local/pgsql/share/spatial_ref_sys.sql
 
8.4:
psql -d template_postgis -f /usr/local/pgsql/share/contrib/postgis.sql 
psql -d template_postgis -f /usr/local/pgsql/share/contrib/spatial_ref_sys.sql
 
9.0 + postgis2.0:
psql -d template_postgis -f /usr/local/pgsql/share/contrib/postgis-2.0/postgis.sql 
psql -d template_postgis -f /usr/local/pgsql/share/contrib/postgis-2.0/spatial_ref_sys.sql

7. Install and download pgAdmin3 for MacOS X

http://www.postgresql.org/ftp/pgadmin3/release/

Update Feb 15, 2011: Use pgadmin1.12 to connect with postgres9. Other versions (1.8.4, 1.6.3 don’t work)

8. Startup pgadmin3. You will notice there is a window stating…

Server instrumentation
The server lacks instrumentation functions.
pgAdmin III uses some support functions that are not available by default in all PostgreSQL versions. These enable some tasks that make life easier when dealing with log files and configuration files.

9. Compile the adminpack. Go to $postgresql_install_directory/contrib/adminpack

$ make
gcc -no-cpp-precomp -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv  -bundle -multiply_defined suppress  adminpack.o  -L../../src/port -bundle_loader ../../src/backend/postgres  -o libadminpack.0.0.so
rm -f libadminpack.0.so
ln -s libadminpack.0.0.so libadminpack.0.so
rm -f libadminpack.so
ln -s libadminpack.0.0.so libadminpack.so
 
rupert:adminpack rupert$ sudo make install
Password:
mkdir /usr/local/pgsql/share/contrib
/bin/sh ../../config/install-sh -c -m 644 ./uninstall_adminpack.sql '/usr/local/pgsql/share/contrib'
/bin/sh ../../config/install-sh -c -m 644 adminpack.sql '/usr/local/pgsql/share/contrib'
/bin/sh ../../config/install-sh -c -m 755  libadminpack.0.0.so '/usr/local/pgsql/lib/adminpack.so'

10. Load the adminpack.sql into your maintenance and template database

$ psql -U postgres -d postgres -f /usr/local/pgsql/share/contrib/adminpack.sql 
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
...

11. Disconnect and Reconnect from pgAdmin3. You shouldn’t see the window again.

Continue only if you want pgRouting

12. Essentially we would need Boost Graph Library (BGL) a.k.a boost, Genetic Algorithm Utility Library (or, GAUL for short), and GEOS (which we installed earlier when we installed postgis).

13. The easiest way to install boost is using MacPorts.

$ sudo port install boost-jam
--->  Fetching boost-jam
--->  Attempting to fetch boost-jam-3.1.16.tgz from http://downloads.sourceforge.net/boost
--->  Verifying checksum(s) for boost-jam
--->  Extracting boost-jam
--->  Configuring boost-jam
--->  Building boost-jam
--->  Staging boost-jam into destroot
--->  Installing boost-jam 3.1.16_0
--->  Activating boost-jam 3.1.16_0
--->  Cleaning boost-jam
--->  Fetching gmake
--->  Attempting to fetch make-3.81.tar.bz2 from http://ftp.gnu.org/gnu/make
--->  Verifying checksum(s) for gmake
--->  Extracting gmake
--->  Configuring gmake
--->  Building gmake with target all
--->  Staging gmake into destroot
--->  Installing gmake 3.81_0
--->  Activating gmake 3.81_0
--->  Cleaning gmake
--->  Fetching boost
--->  Attempting to fetch boost_1_34_1.tar.bz2 from http://downloads.sourceforge.net/boost
...
$ sudo port install boost

Note: I encountered an error when I directly installed “sudo port install boost” on my first run. A quick workaround is to install boost-jam, then install boost afterwards. For more details:

Error: Target org.macports.build returned: shell command ” cd “/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_boost/work/boost_1_34_1″ && gmake all ” returned error 2 Command output: Makefile:2: *** missing separator. Stop.

14. You can check if boost was successfully installed by…

$ sudo port -v installed boost boost-jam
The following ports are currently installed:
  boost @1.34.1_3+darwin_9 (active)
  boost-jam @3.1.16_0 (active)

15. For GAUL, we don’t need slang base on http://pgrouting.postlbs.org/wiki/1.x/InstallationManual.

$ ./configure --enable-slang=no
$ make
$ make install

16. Ok, so this is the heartbreaker. I was able to get pass cmake on pgRouting on version 1.02 however, I received “undefined symbols” when linking the librouting.dylib

Linking CXX shared library ../../lib/librouting.dylib
Undefined symbols:
  "_errcode", referenced from:
      _shortest_path in dijkstra.o
      _shortest_path_astar in astar.o
      _shortest_path_shooting_star in shooting_star.o

For the complete error details, see pgrouting_problem.txt

17. Thanks to www.kyngchaos.com, we can install http://www.kyngchaos.com/files/software/unixport/pgRouting-1.0.1-4.dmg binary from http://www.kyngchaos.com/wiki/software:postgres

18. Now we can load the pgRouting functions to our template database

psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_core.sql 
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_core_wrappers.sql 
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_dd.sql
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_dd_wrappers.sql 
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_dd_tsp.sql
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_tsp.sql
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_tsp_wrappers.sql

References:

http://developer.apple.com/internet/opensource/postgres.html

http://www2.russbrooks.com:8080/2007/11/4/install-postgresql-on-mac-os-x-10-5-leopard

Migrating postgres/postgis data (updated)

rupert — Tue, 15 Feb 2011 00:51:56 +0000

One of my hurdles recently is migrating data from my Debian Lenny Desktop Box (Postgres8.2.7/Postgis1.3.1) to my new MacBookPro Leopard(Postgres8.3.1/Postgis1.3.3). I found it out the hard way by inspecting the dump files manually.

1. pg_dump is your friend.

pg_dump --help

2. I strongly suggest if you have a big dump file (mine is 500MB) to split the schema from the data.

Add “-s” to create the schema:

pg_dump -C -s -h 127.0.0.1 -U lbs -W beijing > beijing_20080507_schema.sql

Add “-a” to dump the data only:

pg_dump -a -h 127.0.0.1 -U lbs -W beijing > beijing_20080507_schema.sql

3. pg_dump from an 8.2 would have statically linked liblwgeom to /usr/lib/postgresql/8.2/liblwgeom. You should change that to whereever your liblwgeom resides, mine is on /usr/local/pgsql/lib/liblwgeom. Just do a simple search and replace using vim on your file_schema.sql

4. After editing the schema, we can now restore the structure of the database. Check for errors and manually update the schema if needs be.

psql -h 127.0.0.1 -U beijing_4326 < beijing_20080507_schema.sql

5. Ok, so now we have the structure ready, we can also check this from pgAdmin3. Have a good look on the functions and table structures if they are fully restored.

6. Let’s load the data.

psql -h 127.0.0.1 -U beijing_4326 < beijing_20080507_data.sql

Postgres PostGIS CheatSheet v2

rupert — Thu, 11 Nov 2010 02:29:12 +0000

This is a quick-command list of Postres. If you want detailed instructions, please visit the Postgres Manual.

How do I know the version of Postgis?

SELECT POSTGIS_FULL_VERSION();

How do I Show all databases?
1. Using “psql -l”

2. Using

postgres=# \l
List of databases
Name       |  Owner   | Encoding
------------------+----------+----------
postgis          | postgres | UTF8
postgres         | postgres | UTF8
template0        | postgres | UTF8
template1        | postgres | UTF8
template_postgis | postgres | UTF8
(5 rows)

Note: Do not drop template databases if not necessary.

How do I run a script from the prompt?

psql -d cybersoftbj -u user -f myfile.sql

Its very useful in reloading user-defined functions.

How do I create a user/role?

CREATE ROLE lbs WITH LOGIN PASSWORD 'mypassword' SUPERUSER INHERIT CREATEDB CREATEROLE;

How do I change the password for a user/role?

ALTER ROLE lbs PASSWORD 'mynewpassword';

How to provide/restrict access privileges to tables?

GRANT SELECT ON TABLE TABLE TO user;
REVOKE SELECT ON TABLE TABLE FROM user;

How to dump database in a text file?

pg_dump -U lbs -d cybersoftbjv1 -h 127.0.0.1 -W &gt; cybersoftbjv1.sql

How to dump database cleanly?

% pg_dump -c  -d -E UTF8 -h 127.0.0.1 -U lbs -W platform_v1 &gt; platform_v1.sql

How to rename a database?

ALTER DATABASE beijing_app RENAME TO beijing_app_20080801;

How to update using two tables?

UPDATE road_for_update u
SET the_geom = r.the_geom
FROM roads r
WHERE r.rd_id = u.rd_id;

DROP TABLE

DROP TABLE IF EXISTS "my_table";

How to change a column type with Cast?

ALTER TABLE roads ALTER COLUMN class_new TYPE integer USING class_new::integer;

How to add a geometry column to a table?

SELECT AddGeometryColumn('public', 'poi', 'the_geom', 4326, 'POINT', 2)

Changing column names with spaces?

ALTER TABLE class_aroundme RENAME "level 1" TO level_1;

Setting kernel shmmax for postgres

% sysctl -w kernel.shmmax=134217728

Note: For permanent changes see /etc/sysctl.cfg

11. How to backup table(s) from pg_dump?

% pg_dump poi_beijing -t class -t poi_class -f $BACKUPDIR/test_$MYDATE.sql

Change integer primary key to serial

CREATE SEQUENCE seq_job_id INCREMENT 1 MINVALUE 1000 MAXVALUE 2147483648 START 1000 CACHE 1;
 
ALTER TABLE job ALTER COLUMN job_id SET DEFAULT NEXTVAL('seq_job_id'::regclass);
 
SELECT * FROM job
 
SELECT NEXTVAL('seq_job_id')

Date and Time Function Helper: date_add

CREATE OR REPLACE FUNCTION date_add(diffType Character Varying(15), incrementValue bigint, inputDateTime timestamp without time zone) RETURNS timestamp AS $$
DECLARE
   YEAR_CONST Char(15) := 'year';
   MONTH_CONST Char(15) := 'month';
   DAY_CONST Char(15) := 'day';
   HOUR_CONST Char(15) := 'hour';
   MIN_CONST Char(15) := 'minute';
   SEC_CONST Char(15) := 'second';
 
   dateTemp timestamp without time zone;
   intervals interval;
BEGIN
   IF lower($1) = lower(YEAR_CONST) THEN
       SELECT cast(cast(incrementvalue AS character varying) || ' year' AS interval) INTO intervals;
   ELSEIF lower($1) = lower(MONTH_CONST) THEN
       SELECT cast(cast(incrementvalue AS character varying) || ' months' AS interval) INTO intervals;
   ELSEIF lower($1) = lower(DAY_CONST) THEN
       SELECT cast(cast(incrementvalue AS character varying) || ' day' AS interval) INTO intervals;
   ELSEIF lower($1) = lower(HOUR_CONST) THEN
       SELECT cast(cast(incrementvalue AS character varying) || ' hours' AS interval) INTO intervals;
   ELSEIF lower($1) = lower(MIN_CONST) THEN
       SELECT cast(cast(incrementvalue AS character varying) || ' minutes' AS interval) INTO intervals;
   ELSEIF lower($1) = lower(SEC_CONST) THEN
       SELECT cast(cast(incrementvalue AS character varying) || ' seconds' AS interval) INTO intervals;            
   END IF;
 
   dateTemp:= inputDateTime + intervals;
 
   RETURN dateTemp;
END;
$$ LANGUAGE plpgsql;

How to set the current timezone in postgres?

# Session based only
SET time zone 'utc';
SELECT  current_setting('TIMEZONE');
# Permanent
# Edit /usr/local/var/postgres/postgresql.conf (#postgres installed via homebrew)
timezone = 'UTC'

Date/Time Functions

SELECT current_setting('TIMEZONE'); 
--"Australia/Victoria"
 
SELECT Now(), 
timezone('UTC', now()), 
EXTRACT(EPOCH FROM CURRENT_TIMESTAMP(0)),
to_timestamp(EXTRACT(EPOCH FROM CURRENT_TIMESTAMP(0))),
to_timestamp(EXTRACT(EPOCH FROM CURRENT_TIMESTAMP(0)))::TIMESTAMP
 
--"2011-11-14 09:29:14.249427+11" 
--"2011-11-13 22:29:14.249427"
--1321223354
--"2011-11-14 09:29:14+11"
--"2011-11-14 09:29:14"

How to specify the id of a sequence to prevent SQL Error: PGRES_FATAL_ERROR:ERROR: duplicate key value violates unique constraint “gps_histories_pkey”?
This happens when the maximum number of records in gps_histories is not in sync with the sequence id.

SELECT SETVAL('dfms_4000.gps_histories_id_seq', (SELECT MAX(id) FROM dfms_4000.gps_histories)+1)

Transform 900913 to 4326

rupert — Wed, 03 Sep 2008 14:37:18 +0000

Stumbled upon an error in reprojecting data from 900913 (Google Speherical Mercator) to 4326 (WGS84). I’m recalling this from my head now, the error was something related to “NAD sth”. Workaround was to convert it as follows:

1. 900913
2. 32650 (UTM for your area)
3. 4326

I’ll get the SQL from my ofcmate and paste it here…

Postgres PostGIS CheatSheet v2

rupert — Fri, 29 Aug 2008 05:10:32 +0000

This is a quick-command list of Postres. If you want detailed instructions, please visit the Postgres Manual.

How do I Show all databases?
1. Using “psql -l”

2. Using

postgres=# \l
List of databases
Name       |  Owner   | Encoding
------------------+----------+----------
postgis          | postgres | UTF8
postgres         | postgres | UTF8
template0        | postgres | UTF8
template1        | postgres | UTF8
template_postgis | postgres | UTF8
(5 rows)

Note: Do not drop template databases if not necessary.

How do I run a script from the prompt?
psql -d cybersoftbj -u user -f myfile.sql

Its very useful in reloading user-defined functions.

How do I create a user/role?
CREATE ROLE lbs WITH LOGIN PASSWORD 'mypassword' SUPERUSER INHERIT CREATEDB CREATEROLE;

How do I change the password for a user/role?
ALTER ROLE lbs PASSWORD 'mynewpassword';

How to provide/restrict access privileges to tables?
GRANT SELECT ON TABLE table TO user; REVOKE SELECT ON TABLE table FROM user;

How to dump database in a text file?
pg_dump -U lbs -d cybersoftbjv1 -h 127.0.0.1 -W > cybersoftbjv1.sql

How to dump database cleanly?
pg_dump -c -d -E UTF8 -h 127.0.0.1 -U lbs -W platform_v1 > platform_v1.sql

How to rename a database?
ALTER DATABASE beijing_app RENAME TO beijing_app_20080801;

How to update using two tables?
UPDATE road_for_update u SET the_geom = r.the_geom FROM roads r WHERE r.rd_id = u.rd_id;

How to change a column type with Cast?
ALTER TABLE roads ALTER COLUMN class_new TYPE integer USING class_new::integer;

How to add a geometry column to a table?
EXAMPLE: SELECT AddGeometryColumn(‘public’, ‘poi’, ‘the_geom’, 4326, ‘POINT’, 2)

Changing column names with spaces?
ALTER TABLE class_aroundme RENAME "level 1" TO level_1;

Setting kernel shmmax for postgres
sysctl -w kernel.shmmax=134217728
Note: For permanent changes see /etc/sysctl.cfg11. How to backup table(s) from pg_dump?
pg_dump poi_beijing -t class -t poi_class -f $BACKUPDIR/test_$MYDATE.sql

Postgres PostGis PostInstall

rupert — Fri, 06 Jun 2008 23:34:36 +0000

1. Edit pg_hba.conf

# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
host    all         all         192.168.1.0 255.255.255.0       md5
# IPv6 local connections:
#host    all         all         ::1/128               trust

2. Edit postgres.conf

55
56 listen_addresses = '*' # what IP address(es) to listen on;
57 # comma-separated list of addresses;
58 # defaults to 'localhost', '*' = all
59 # (change requires restart)

4. For CentOS5.1, create symbolic links:
ln -s /usr/local/lib/libproj.so.0 /usr/lib/libproj.so.0
ln -s /usr/local/lib/libgeos_c.so.1 /usr/lib/libgeos_c.so.1
ldconfig
/etc/init.d/postgresql stop
/etc/init.d/postgresql start

3. Postgis Post Install

createdb -E utf8 template_routing
createlang plpgsql template_routing
psql -d template_routing -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql 
psql -d template_routing -f /usr/share/postgresql-8.3-postgis/spatial_ref_sys.sql 
psql -d template_routing -f /usr/share/postlbs/routing_core.sql 
psql -d template_routing -f /usr/share/postlbs/routing_core_wrappers.sql

Installing Postgresql, Postgis, pgRouting on Debian

rupert — Sun, 25 May 2008 03:55:57 +0000

Operating System: Debian sid

Versions:

postgres 8.3.1
postgis 1.3.3
pgRouting1.02

1. Install base system and ssh

#vi /etc/apt/sources.list to include
deb http://debian.cn99.com/debian etch main
deb-src http://debian.cn99.com/debian etch main
#apt-get update
#apt-get upgrade libc6

2. Install the required packages for postgres8.3 and postgis1.3.3

#apt-get install sudo nmap telnet
#apt-get install python2.5 python2.5-dev python-setuptools
#apt-get install g++
#apt-get install build-essential cmake ibboost-graph-dev
#apt-get install libreadline5 libreadline5-dev
#apt-get install zlib-bin zlib1g-dev
#apt-get install libkrb5-dev
#apt-get install libcurl3
#apt-get install libssl-dev
#apt-get install postgresql-8.3
#apt-get install postgresql-8.3-postgis
#apt-get install postgresql-server-dev-8.3

3. Installing pgRouting

# tar -zxvf pgRouting-1.02.tgz
# cmake .
# make 
# make install

Installing Postgres8.3,8.4, Postgis1.3.3,1.4.1, pgRouting on MacOSX Leopard

rupert — Thu, 01 May 2008 08:36:57 +0000

References:

http://developer.apple.com/internet/opensource/postgres.html

http://www2.russbrooks.com:8080/2007/11/4/install-postgresql-on-mac-os-x-10-5-leopard

Warning: Most of the packages listed below is installed by source.

1. Download the current postgres source.

8.3:
$./configure --with-prefix=/usr/local/pgsql --with-python
 
8.4:
$./configure --prefix=/usr/local/pgsql --with-python
 
$make
$sudo make install

2. Don’t delete the postgres folder. You might need this later on for future compilations. See pgadmin3 admin pack below.

3. Add a postgres user from System Preferences > Accounts

4. Initialize the data directory

$mkdir /usr/local/pgsql/data
$chown postgres /usr/local/pgsql/data/
$su - postgres
$/usr/local/pgsql/bin/initdb -E utf8 -D /usr/local/pgsql/data

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.
The default text search configuration will be set to "english".

fixing permissions on existing directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 20
selecting default shared_buffers/max_fsm_pages ... 2400kB/20000
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

Success. You can now start the database server using:

/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data
or
/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start

4. Install Postgis
- install geos with the standard ./configure -> make -> make install
- then install postgis

8.3:
$./configure --with-pgsql=/usr/local/pgsql/bin/pg_config --with-geos=/usr/local/bin/geos-config
 
8.4 using Kyngchaos GEOS Framework:
$./configure --with-pgsql=/usr/local/pgsql/bin/pg_config --with-geosconfig=/Library/Frameworks/GEOS.framework/unix/bin/geos-config
 
$make
$sudo make install

5. Starting postgres on boot. Download postgresstartup.tar.gz then extract to your /Library/StartupItems

6. Creating the database

createdb -E utf8 template_postgis
createlang plpgsql template_postgis
 
8.3:
psql -d template_postgis -f /usr/local/pgsql/share/lwpostgis.sql
psql -d template_postgis -f /usr/local/pgsql/share/spatial_ref_sys.sql
 
8.4:
psql -d template_postgis -f /usr/local/pgsql/share/contrib/postgis.sql 
psql -d template_postgis -f /usr/local/pgsql/share/contrib/spatial_ref_sys.sql

7. Install and download pgAdmin3 for MacOS X

http://www.postgresql.org/ftp/pgadmin3/release/

8. Startup pgadmin3. You will notice there is a window stating…

Server instrumentation
The server lacks instrumentation functions.
pgAdmin III uses some support functions that are not available by default in all PostgreSQL versions. These enable some tasks that make life easier when dealing with log files and configuration files.

9. Compile the adminpack. Go to $postgresql_install_directory/contrib/adminpack

$ make
gcc -no-cpp-precomp -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv  -bundle -multiply_defined suppress  adminpack.o  -L../../src/port -bundle_loader ../../src/backend/postgres  -o libadminpack.0.0.so
rm -f libadminpack.0.so
ln -s libadminpack.0.0.so libadminpack.0.so
rm -f libadminpack.so
ln -s libadminpack.0.0.so libadminpack.so
 
rupert:adminpack rupert$ sudo make install
Password:
mkdir /usr/local/pgsql/share/contrib
/bin/sh ../../config/install-sh -c -m 644 ./uninstall_adminpack.sql '/usr/local/pgsql/share/contrib'
/bin/sh ../../config/install-sh -c -m 644 adminpack.sql '/usr/local/pgsql/share/contrib'
/bin/sh ../../config/install-sh -c -m 755  libadminpack.0.0.so '/usr/local/pgsql/lib/adminpack.so'

10. Load the adminpack.sql into your maintenance and template database

$ psql -U postgres -d postgres -f /usr/local/pgsql/share/contrib/adminpack.sql 
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
...

11. Disconnect and Reconnect from pgAdmin3. You shouldn’t see the window again.

Continue only if you want pgRouting

12. Essentially we would need Boost Graph Library (BGL) a.k.a boost, Genetic Algorithm Utility Library (or, GAUL for short), and GEOS (which we installed earlier when we installed postgis).

13. The easiest way to install boost is using MacPorts.

$ sudo port install boost-jam
--->  Fetching boost-jam
--->  Attempting to fetch boost-jam-3.1.16.tgz from http://downloads.sourceforge.net/boost
--->  Verifying checksum(s) for boost-jam
--->  Extracting boost-jam
--->  Configuring boost-jam
--->  Building boost-jam
--->  Staging boost-jam into destroot
--->  Installing boost-jam 3.1.16_0
--->  Activating boost-jam 3.1.16_0
--->  Cleaning boost-jam
--->  Fetching gmake
--->  Attempting to fetch make-3.81.tar.bz2 from http://ftp.gnu.org/gnu/make
--->  Verifying checksum(s) for gmake
--->  Extracting gmake
--->  Configuring gmake
--->  Building gmake with target all
--->  Staging gmake into destroot
--->  Installing gmake 3.81_0
--->  Activating gmake 3.81_0
--->  Cleaning gmake
--->  Fetching boost
--->  Attempting to fetch boost_1_34_1.tar.bz2 from http://downloads.sourceforge.net/boost
...
$ sudo port install boost

Note: I encountered an error when I directly installed “sudo port install boost” on my first run. A quick workaround is to install boost-jam, then install boost afterwards. For more details:

Error: Target org.macports.build returned: shell command ” cd “/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_devel_boost/work/boost_1_34_1″ && gmake all ” returned error 2 Command output: Makefile:2: *** missing separator. Stop.

14. You can check if boost was successfully installed by…

$ sudo port -v installed boost boost-jam
The following ports are currently installed:
  boost @1.34.1_3+darwin_9 (active)
  boost-jam @3.1.16_0 (active)

15. For GAUL, we don’t need slang base on http://pgrouting.postlbs.org/wiki/1.x/InstallationManual.

$ ./configure --enable-slang=no
$ make
$ make install

16. Ok, so this is the heartbreaker. I was able to get pass cmake on pgRouting on version 1.02 however, I received “undefined symbols” when linking the librouting.dylib

Linking CXX shared library ../../lib/librouting.dylib
Undefined symbols:
  "_errcode", referenced from:
      _shortest_path in dijkstra.o
      _shortest_path_astar in astar.o
      _shortest_path_shooting_star in shooting_star.o

For the complete error details, see pgrouting_problem.txt

17. Thanks to www.kyngchaos.com, we can install http://www.kyngchaos.com/files/software/unixport/pgRouting-1.0.1-4.dmg binary from http://www.kyngchaos.com/wiki/software:postgres

18. Now we can load the pgRouting functions to our template database

psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_core.sql 
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_core_wrappers.sql 
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_dd.sql
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_dd_wrappers.sql 
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_dd_tsp.sql
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_tsp.sql
psql -U postgres -d template_postgis -f /usr/local/pgsql/share/contrib/routing_tsp_wrappers.sql

Installing PostGres8.3 and Postgis1.3.2 on Windows

rupert — Tue, 04 Mar 2008 00:15:05 +0000

It seems very nice for the new PostGres8.3 installer to bundle up third party libraries installation with the new admin pack feature. Using this feature, you can install PostGis, nPgsql, etc. The installation was painless and smooth this time. I have to backup all my data though using pg_dump.

To lessen the learning curve on installing PostGres + PostGis together, I made a short flash movie here. There is no audio in the tutorial. The idea was to introduce PostGres to my Chinese staff. I have to remove PostGreSQL 8.2 first before installing 8.3.

Making GeoJSON outputs from PostGIS

rupert — Sat, 23 Feb 2008 02:08:17 +0000

I have an internet/web application sitting on top of OpenLayers, ExtJS, ColdFusion and PostGreSQL/PostGIS. I can query the geometry from PostGIS thru ColdFusion without any problems. If you are in ColdFusion 7, then you need a postgresql driver installed, but I do suggest you use ColdFusion 8 as the driver is built-in.

To display the geometry, I typically use AsGML output from PostGIS and supply that path to OpenLayers. Although it works as expected, I need to invest in two (2) http calls to the server:

1st Request: Call search.cfm and get the attributes (name, address, tel_no, etc…). Afterwards, display the results on an Ext.Window.
2nd Request: Display the geometries on OpenLayers using GML.

We have decided in our organisation not to write out the javascript statements on AJAX calls but instead retrieve them as Data Objects (either XML or GeoJSON). Putting in javascript statements on AJAX calls seems harder to debug. GeoJSON is highly preferred for internet mapping applications since it has a unique ability to see the javascript objects quickly on the browser, making debugging easier.

I wrote a simple cfc below to be able to parse the geometry from PostGIS using AsText(geometry). The function accepts a geometry as string and returns a structure containing the type and the coordinates.

<cfcomponent displayname="GeoJSON" hint="I am a GeoJSON Component. I parse a geometry and return it as JSON" output="false">
 
	<cffunction name="init" access="public" output="false">
		<cfreturn>
	cfreturn>	
 
	cffunction><cffunction name="format" returntype="Struct" output="false">
		<cfargument name="geomtext" type="String" required="true">
		<cfset>
		cfset><cfset>
		cfset><cfset geometry="StructNew()">
		cfset><cfset>
		cfset><cfset>
		cfset><cfset>
		cfset><cfset>
		cfset><cfset>
 
		cfset><cfset geometry.type="Mid(arguments.geomtext,">
		cfset><cfset>
 
		cfset><cfset geomreplacedbrackets="Replace(arguments.geomtext,">
		cfset><cfset geomreplacedbrackets="Replace(geomReplacedBrackets,">
 
		<cfloop from="1" to="#Len(geomReplacedBrackets)#" index="i">
			<cfset curchar="Mid(geomReplacedBrackets,">
			<cfif>
				<cfset geometry.coordinates="geometry.coordinates">
	    		cfset><cfset secchar="Mid(geomReplacedBrackets,">
				<cfif> 
					<cfset posi="i"> 
					cfset><cfset posj="find(']',">
					cfset><cfset tempgeom="Mid(geomReplacedBrackets,">
					cfset><cfset i="posJ"> 
 
					cfset><cfset>
 
					<cfloop list="#tempGeom#" index="tempGeomIndex" delimiters=",">
						<cfset tempvalue="[">
						cfset><cfset geomreplaced="ListAppend(geomReplaced,">
					cfset>cfloop>cfset>
 
					<cfset geometry.coordinates="geometry.coordinates">
				cfset>cfif>cfset>cfif>cfset>cfloop>
			<cfelseif>
				<cfset geometry.coordinates="geometry.coordinates">
			cfset>cfelseif>cfset>
 
		<cfset geometry.coordinates="Replace(geometry.coordinates,">
 
		<cfif>
			<cfset geometry.coordinates="Replace(geometry.coordinates,">
			cfset><cfset geometry.coordinates="Replace(geometry.coordinates,">
		cfset>
 
		<cfreturn>
	cfreturn>cfif>cfset>
 
cfargument>cffunction>cfcomponent>

OGR Quick Reference

rupert — Mon, 09 Jul 2007 18:22:19 +0000

Here is a list of the most widely used OGR commands I use..

OGR2OGR
1. POSTGRES -> MAPINFO

$ ogr2ogr -f "Mapinfo File" busline_buffer10m.tab PG:"host=localhost user=postgres dbname=cybersoftbj" -sql "select * from table_name" -a_srs WGS84 -nln layer_name -nlt MULTIPOLYGON

2. MAPINFO -> POSTGRES

ogr2ogr -f "PostgreSQL" PG:"host=127.0.0.1 user=rupert dbname=australia password=*****" AUS_ROAD.TAB -nln AUS_ROAD -a_srs EPSG:4269 -t_srs EPSG:3857 -skip-failures
 
ogr2ogr -f "PostgreSQL" PG:"host=myhost user=myloginname dbname=mydbname password=mypassword" mytabfile.tab -nln newtablename -select columnName

Note: If you have Chinese characters, might as well do MAPINFO -> SHAPE -> POSTGRES
http://222.128.19.19/wordpress/?p=108

3. SHAPE -> POSTGRES

shp2pgsql -W "gbk" -s 4326 lbjrdnt_small_polyline roads &gt; roads.sql

4. POSTGRES -> SHAPE

pgsql2shp -h 127.0.0.1 -u lbs -P tracking -f roads.shp databasename tablename

4. MAPINFO TO ORACLE

ogr2ogr -f OCI OCI:username/password@orcl C:\path_to_tabfile\EMPLOYEES.TAB -nln employees

Note: This assumes you already have Oracle 10g Client installed and “orcl” is defined as an instance in tnsnames.ora. OGR2OGR automatically updates USER_SDO_GEOM_METADATA and creates a spatial index.

5. MAPINFO to MAPINFO but different projection. From EPSG:4326 to EPSG:3857

 ogr2ogr -f "MapInfo File" BaseMaps_3857/AUS_CITIES_3857.TAB BaseMaps/AUS_CITIES.TAB -a_srs "EPSG:4326" -t_srs "EPSG:3857"

Buffering points

rupert — Sat, 07 Jul 2007 16:52:43 +0000

Here is a post in buffering points and combining them using GeomUnion. Works in GEOS3.0.0, so note that your POSTGIS installation in windows contains a lower version of GEOS.

DROP FUNCTION combineGeometry();
 
CREATE FUNCTION combineGeometry() RETURNS Void AS'
DECLARE
	--define datatypes here
	updateCount integer DEFAULT 0;
	geom_record RECORD;
	geom_current RECORD;
	geom_final RECORD;
	mytotal integer DEFAULT 0;
 
BEGIN
 
	--get the total
	SELECT count(*) as mycount FROM busstopv1_buffer_20_temp INTO mytotal;
 
	-- 1,2,3,4,5
	FOR geom_record IN SELECT * FROM busstopv1_buffer_20 LOOP
 
		--get the current geom of the record
		SELECT *
		FROM 		busstopv1_buffer_20_temp b
		WHERE 		b.gid = geom_record.gid INTO geom_current;
 
		RAISE INFO ''RECORD GID: %'', geom_record.gid;
		RAISE INFO ''===========TOTAL COUNT: %=============='', mytotal;
 
		--find the intersection of the current geom with other spatial entities
		--and loop through that. For each loop, update the geom.
 
		IF geom_current IS NULL THEN
			CONTINUE;
		END IF;
		FOR geom_final IN
			SELECT GeomUnion(b0.the_geom, geom_current.the_geom ) AS geom_current_union, b0.gid
			FROM busstopv1_buffer_20_temp b0
			WHERE b0.gid <> geom_current.gid
			AND Intersects(b0.the_geom, geom_current.the_geom) = ''t''
			ORDER BY gid ASC LOOP
 
			--geom_current.gid = geom_record.gid
			UPDATE busstopv1_buffer_20_temp
			SET the_geom = GeomUnion( the_geom, geom_final.geom_current_union )
			WHERE gid = geom_current.gid;
			RAISE INFO ''UPDATED GID:%'', geom_current.gid;
 
			DELETE FROM busstopv1_buffer_20_temp WHERE gid = geom_final.gid;
			RAISE INFO ''DELETED GID:%'', geom_final.gid;
 
			mytotal = mytotal - 1;
 
		END LOOP;
 
	END LOOP;
END;'
LANGUAGE plpgsql;
 
SELECT combineGeometry();

Next step.. trying out CGAL.

Buffering Lines

rupert — Wed, 04 Jul 2007 15:15:24 +0000

Here is a sample script in using buffer in Postgis. I buffered the line by 10 and 20 meters. Take note that I have to transform the geometry to the corresponding EPSG, so I could specify “meters”.

DROP TABLE busline_buffer1;
DELETE FROM geometry_columns WHERE f_table_name = 'busline_buffer1';
 
CREATE TABLE busline_buffer1( gid serial, CONSTRAINT pk_buffer1 PRIMARY KEY(gid));
SELECT  AddGeometryColumn('public', 'busline_buffer1', 'the_geom', 4326, 'POLYGON',2);
 
// This would display what the output of the geometry IS...
SELECT AsText( transform( ST_BUFFER( transform(v1.the_geom, 32650), 10 ), 4326 ) ) FROM buslinev1 v1;
 
INSERT INTO busline_buffer1(the_geom) SELECT transform( ST_BUFFER( transform(v1.the_geom, 32650), 10 ), 4326 ) FROM buslinev1 v1;
 
SELECT gid, AsText(the_geom) FROM busline_buffer1;
========================================
DROP TABLE busline_buffer2;
DELETE FROM geometry_columns WHERE f_table_name = 'busline_buffer2';
 
CREATE TABLE busline_buffer2( gid serial, CONSTRAINT pk_buffer2 PRIMARY KEY(gid));
SELECT  AddGeometryColumn('public', 'busline_buffer2', 'the_geom', 4326, 'POLYGON',2);
 
SELECT AsText( transform( ST_BUFFER( transform(v1.the_geom, 32650), 20 ), 4326 ) ) FROM buslinev1 v1;
 
INSERT INTO busline_buffer2(the_geom) SELECT transform( ST_BUFFER( transform(v1.the_geom, 32650), 20 ), 4326 ) FROM buslinev1 v1;
 
SELECT gid, AsText(the_geom) FROM busline_buffer2;

PostgresSQL / PostGIS CheatSheat

rupert — Sun, 01 Jul 2007 14:05:58 +0000

This is a quick-command list of Postres. If you want detailed instructions, please visit the Postgres Manual.

Note: Do not drop template databases if not necessary.

2. How do I run a script from the prompt?
\i
OR
psql -d cybersoftbj -u user < myfile.sql

Its very usuful in reloading user-defined functions.

3. How do I create a user?
CREATE ROLE lbs WITH LOGIN PASSWORD 'tracking' SUPERUSER INHERIT CREATEDB CREATEROLE;

4. How to dump database in a text file?
pg_dump -U lbs -d cybersoftbjv1 -h 127.0.0.1 -W > cybersoftbjv1.sql

4. How to dump database cleanly?
pg_dump -c -d -E UTF8 -h 127.0.0.1 -U lbs -W platform_v1 > platform_v1.sql

5. How to update using two tables?
UPDATE road_for_update u SET the_geom = r.the_geom FROM roads r WHERE r.rd_id = u.rd_id;

6. How to change a column type with Cast?
ALTER TABLE roads ALTER COLUMN class_new TYPE integer USING class_new::integer;

7. How to provide/restrict access privileges to tables?
GRANT SELECT ON TABLE table TO user; REVOKE SELECT ON TABLE table FROM user;

8. How to add a geometry column to a table?
SYNTAX: AddGeometryColumn(

,
, , , )
EXAMPLE: SELECT AddGeometryColumn(‘public’, ‘poi’, ‘the_geom’, 4326, ‘POINT’, 2)
9. Changing column names with spaces?
ALTER TABLE class_aroundme RENAME "level 1" TO level_1; 10. Setting kernel shmmax for postgres
sysctl -w kernel.shmmax=134217728
Note: For permanent changes see /etc/sysctl.cfg11. How to backup table(s) from pg_dump?
pg_dump poi_beijing -t class -t poi_class -f $BACKUPDIR/test_$MYDATE.sql

Preparing routing data for pgRouting

rupert — Tue, 10 Apr 2007 09:23:35 +0000

1. It is important that you already have postgres, postgis, pgRouting installed in your machine.

A. The schema. Below is the sample schema that is a derivative of the kanagawa sample data from pgRouting. Take note of the source and target nodes, as well as the length and the node coordinates (x1,y1; x2,y2) of the line.

cybersoftbj=# \dt
List of relations
 Schema |       Name       | Type  |  Owner
--------+------------------+-------+----------
 public | geometry_columns | TABLE | postgres
 public | roads            | TABLE | postgres
 public | spatial_ref_sys  | TABLE | postgres
(3 rows)

cybersoftbj=# \d roads
TABLE "public.roads"
COLUMN   |          Type          |                      Modifiers
------------+------------------------+-----------------------------------------------------
gid        | integer                | NOT NULL DEFAULT NEXTVAL('roads_gid_seq'::regclass)
rd_id      | bigint
yutu_id    | integer
block_id   | bigint
heirarchy  | character varying(5)
cn_name    | character varying(75)
py_name    | character varying(100)
source     | bigint
target     | bigint
x1         | numeric
y1         | numeric
x2         | numeric
y2         | numeric
costlength | numeric
the_geom   | geometry

A. Extracting the coordinates of the line segments from Mapinfo.

1. I have to format the data structure as follows…

…here is the roads table after weeding out some unnecessary columns…

… adding the source,target,x1,y1,x2,y2,costlength…

2. Using ObjectGeography. Download the MapBasic Reference

ObjectGeography( object, attribute )

ObjectGeography( object, “1″ ) <– gives you the beginning x coord of the point

3. Export the tab file to a shape file for ArchMap.

Loading Mapinfo table to PostGis

rupert — Wed, 04 Apr 2007 21:25:06 +0000

AFAIK, there are only two ways to load data to PostGIS:
1. Using Insert statements
2. Using Utilities.

Utilities include shp2pgsql which is found in “C:\Program Files\PostgreSQL\8.2\bin\” or “/usr/local/pgsql/bin”. To load Mapinfo table, I have used the OGR utilities from FWTools for Windows. and used gdal1.3.2 (since it contains ogr) for Unix.

OGR2OGR CheatSheet should be a good kickstart for basic understanding. For the impatient..

ogr2ogr -f "PostGreSQL" -nlt LINESTRING -a_srs "EPSG:4326" PG:"host=localhost user=username password=mypassword dbname=mydb mytab.TAB -select columnName

ogr2ogr -f "PostgreSQL" PG:"host=myhost user=myloginname dbname=mydbname password=mypassword" mytabfile.tab -nln newtablename -select columnName

Bear in mind that you should select out the columns from your mapfile (mine is chinese) especially if you have a diffent encoding in your column which matches your database (postgres). You might get a “Terminating translation prematurely after failed translation of layer [layername]” error. Since Mapinfo stores the text to ASCII, my workaround is to export the tabfile to a UTF-8 textfile then upload it to PostGres. Hoping the primary ids would match to make the necessary updates…

An alternative to load Chinese text from Mapinfo to PostGIS is the ff:
1. In Mapinfo use the Universal Translator to export the table into a shape file.
2. Once you have the shape file, you can directly use the shp2pgsql.

shp2pgsql -W “gbk” -s 4326 lbjrdnt_small_polyline roads > roads.sql

Creating a Spatial Table in PostGIS

rupert — Wed, 04 Apr 2007 20:06:43 +0000

Creating the spatial table from scratch…

[postgres@rupert-linux ~]$ su - postgres
[postgres@rupert-linux ~]$ createdb _E UTF-8 template_postgis
CREATE DATABASE
[postgres@rupert-linux ~]$ createlang plpgsql template_postgis
[postgres@rupert-linux ~]$ psql -f /usr/local/pgsql/share/lwpostgis.sql -d template_postgis
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
COMMIT
postgres@rupert-linux ~]$ psql -f /usr/local/pgsql/share/spatial_ref_sys.sql -d template_postgis
[postgres@rupert-linux ~]$ psql

Alternatively you could also create your spatial table from a template…

     14 postgres=# CREATE DATABASE gistest TEMPLATE=template_postgis;
     15 CREATE DATABASE

I have also added the routing functions using routing.sql and routing_postgis.sql

[postgres@rupert-linux ~]$ psql -f /usr/local/pgsql/share/contrib/routing.sql -d template_postgis
[postgres@rupert-linux ~]$ psql -f /usr/local/pgsql/share/contrib/routing_postgis.sql -d template_postgis

FYI…

The spatial_ref_sys table contains the ESPG codes and projections. Currently it has 3162 records.

gistest=# \d spatial_ref_sys;
 srid      | integer                 | NOT NULL
 auth_name | character varying(256)  |
 auth_srid | integer                 |
 srtext    | character varying(2048) |
 proj4text | character varying(2048) |

The geometry_columns contains the geometry type of the table you just created..

gistest=# \d geometry_columns;
 f_table_catalog   | character varying(256) | NOT NULL
 f_table_schema    | character varying(256) | NOT NULL
 f_table_name      | character varying(256) | NOT NULL
 f_geometry_column | character varying(256) | NOT NULL
 coord_dimension   | integer                | NOT NULL
 srid              | integer                | NOT NULL
 type              | character varying(30)  | NOT NULL