mobile and gis dev notes

Im trying out ways to create a topology of edges and vertices for a road network. Currently, there are three ways I know of:

1. Using ArcGIS build coverage line.
– This includes the use of ArcGIS. Exporting the feature into tics, arcs and nodes, then afterwards assembling them all together. Its functional but have not fully tested the quality of the road topology. Also, assembling them back together through the spatial objects comparison will take some time. It would be better *If* ArcGIS could create the nodes wrt to the base table.

2. Using PostGres, postlbs functions.
– Use of SELECT ASSIGN_VERTEX_ID(‘table_name’, double_precision_distance. So far this bails on me on my first test on my win-xp laptop. We can test this on Linux if the response is the same.

3. Using Mapinfo Basic Scripts provided by J.
– Haven’t gone indepth with these for now, but looking forward to it.

4. Using Grass
– As documented in pgrouting.postlbs.org site’s Topology Creation

Summary
One of the upcoming power features of Mapserver 5.0 is its AGG Rendering engine. Above is an example image of the AGG rendering capabilities of Mapserver. Below is a list of my experiences with Mapserer 5.0 AGG

• Slower than GD ofcourse, but cartographic quality is really impressive…
• You need to have SUPPORT=AGG to be able to use AGG. So far, I was not able to compile mapserver source with AGG, however using mapserver-beta-3 for linux and ms4w betas worked fine.
• Layer Transparency. I was not able to get transparency for my roads overlaying on top of my basemap. Tbonfort from IRC suggested I use RGBA for the outputformat, but it still breaks. Hope Im not missing anything important here…

The Mapfile
Currently, the mapfile ( map_all_agg.map ) contains 2,587 lines! The lengthy mapfile consists of the ff: districts or political boundaries, water bodies, greens or open spaces, roads, subway lines, subway stops. The roads is classified into 12 categories ( Expressway, Class-1, Class-2, Class-3, Class-4, Major Road, Main, Secondary, Minor, CommunityRoad(w/name), CommunityRoad, ParkRoads). Class1-4 is a classification for Roads outside the 5th Ring Road of Beijing. While Major Roads – Minor Roads is a classification inside the 5th Ring. The roads were drawn as a line layer and not as a polygon.

How to achieve the overlapping of road intersections?
The technique is to seperate the road boundary layer from the actual road line. A mixture of widths and color management was employed to achieve the effect. Essentially, we started with the ff:

1. road boundary layer
2. road layer

Further , trial and error resulted in refining the road boundary layer with different minscales and maxscales. The same was adapted with the road layer. Download map_all_agg.txt. My next iteration, would be to refactor some of the road layers if possible.

A couple of notes to myself, we could speed things up by trying out the ff:

• Using SHPTREE
• Simplifying the geometry as you change scales

Testing Methodology
1. Use Firefox and install firebug.
2. Download YSlow from Yahoo.
3. View load times using “net” tab

Using SHPTREE
1. The utility specifies indexes as it slices the shapefile into quadrants as specified from its definition. To run shptree, do the ff:

$ shptree roads.shp
creating index of new LSB format


2. It would generate a roads.qix. No changes need to be made with the mapfile. Mapserver loads the shapefile without the extension.

3. Testing… Significant speed improvements were made as shown below:

• Before: 29.44 secs
• After: 5.91 secs

Here’s a quick overview of how I upgraded my custom MS4W binded with ColdFusion..

1. Extract the new ms4w_new.zip to E:\ms4w_new

2. diff E:\ms4w_new\Apache\conf\httpd.conf E:\ms4w\Apache\conf\httpd.conf

If the changes is minimal, then proceed.

3. Uninstall ColdFusion connector from Apache.

D:\CFusionMX\bin\connectors\Remove_ALL_connectors.bat


4. Uninstall Apache from services.msc. Run

E:\ms4w\apache-uninstall.bat


5. Move

mv E:\ms4w E:\ms4w_old
mv E:\ms4w_new E:\ms4w


6. Install Apache as a windows services to services.msc

E:\ms4w\apache-install.bat


7. Bind ColdFusion again to Apache

D:\CFusionMX\bin\connectors\Apache_connector.bat


By default, we can install mapserver using etch stable from a debian mirror using apt-get. Final results were almost close as expected, however, I need mapserver with TIFF support to be able to display my satellite images. Thus, recompilation is necessary. Below is the binary version difference for both installation procedures.

Nevertheless, it is worth noting that ‘apt-get’ significantly boost the installation of mapserver. I highly suggest to install mapserver using apt-get and use source compilation only if necessary.

a) mapserver installed using apt-get


rupert-debian:/usr/lib/cgi-bin# ./mapserv -v
MapServer version 4.10.2 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG OUTPUT=WBMP OUTPUT=SVG SUPPORTS=PROJ SUPPORTS=FREETYPE SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=THREADS SUPPORTS=GEOS INPUT=TIFF INPUT=EPPL7 INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE DEBUG=MSDEBUG


Installation instructions for a:

1. apt-get install g++ libgd2-xpm libgd2-xpm-dev libcurl3 zlib1g-dev libgd-dev libtiff4 libtiff4-dev  libapache2-mod-python python-imaging  
 
2. apt-get install php5-cgi php5-common php5-curl php5-dev php5-gd php5-mysql php5-pgsql php5-sqlite php5-xsl libapache2-mod-php5
 
3. apt-get install python2.5 python2.5-dev python2.5-examples
 
4. apt-get install mysql-client-5.0 mysql-server-5.0 mytop mysql-admin mysql-admin-common libmysqlclient15-dev libdbd-mysql-perl libdbd-mysql-ruby mtop
 
5. apt-get install postgresql-common postgresql-8.2 postgresql-client-8.2 postgresql-contrib-8.2 postgresql-8.2-plr postgresql-8.2-plruby postgresql-8.2-postgis postgresql-plperl-8.2 postgresql-plpython-8.2 postgresql-pltcl-8.2 libpq4 
 
6.  apt-get install gdal-bin cgi-mapserver mapserver-bin mapserver-doc perl-mapscript php5-mapscript python-mapscript

b. mapserver compiled


rupert-debian:/usr/lib/cgi-bin# ./mapserv_default -v
MapServer version 4.10.0 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG OUTPUT=WBMP OUTPUT=SVG SUPPORTS=PROJ SUPPORTS=FREETYPE SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=THREADS SUPPORTS=GEOS INPUT=EPPL7 INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE DEBUG=MSDEBUG


Installation instructions for b:

#tar -zxvf geos-3.0.0rc4
#./configure
#make
#make install
 
#modify /etc/apt/sources.list and uncomment testing
deb http://mirrors.geekbone.org/debian testing main
deb-src  http://mirrors.geekbone.org/debian testing main
 
deb http://mirrors.geekbone.org/debian etch main
deb-src  http://mirrors.geekbone.org/debian etch main
 
deb http://security.debian.org/ etch/updates main contrib
deb-src http://security.debian.org/ etch/updates main contrib
 
#apt-get update
 
#apt-get install postgresql-8.2-postgis
 
#tar -zxvf mapserver-4.10.2.tar.gz
#./configure \
--with-jpeg \
--with-gd \
--with-freetype \
--with-png \
--with-ogr \
--with-proj \
--with-gdal \
--with-httpd=/usr/sbin/apache2 \
--with-tiff \
--with-wfs \
--with-wcs \
--with-threads \
--with-wmsclient \
--with-wfsclient \
--with-geos \
--with-postgis \
--enable-debug
#make
#make install
#cp -Rf mapserv /usr/local/cgi-bin/

Tested on: CentOS 5.0, 4.1
vim /etc/ld.so.conf
include /usr/lib64
include /usr/local/lib
include /usr/lib

Prerequisites:
rpm -ivh $CENTOS/bzip2-devel-1.0.3-3.x86_64.rpm
rpm -ivh $CENTOS/libidn-0.6.5-1.1.x86_64.rpm
rpm -ivh $CENTOS/curl-7.15.5-2.el5.x86_64.rpm
rpm -ivh $CENTOS/pkgconfig-0.21-1.fc6.x86_64.rpm
rpm -ivh $CENTOS/libidn-devel-0.6.5-1.1.x86_64.rpm
rpm -ivh $CENTOS/e2fsprogs-devel-1.39-10.el5.x86_64.rpm
rpm -ivh $CENTOS/krb5-devel-1.6.1-17.el5.x86_64.rpm
rpm -ivh $CENTOS/zlib-devel-1.2.3-3.x86_64.rpm
rpm -ivh $CENTOS/openssl-devel-0.9.8b-8.3.el5_0.2.x86_64.rpm
rpm -ivh $CENTOS/flex-2.5.4a-41.fc6.x86_64.rpm
rpm -ivh $CENTOS/libstdc++-devel-4.1.2-14.el5.x86_64.rpm
rpm -ivh $CENTOS/libxml2-devel-2.6.26-2.1.2.x86_64.rpm
rpm -ivh $CENTOS/libxslt-devel-1.1.17-2.x86_64.rpm
rpm -ivh $CENTOS/nmap-4.11-1.1.x86_64.rpm
rpm -ivh $CENTOS/swig-1.3.29-2.el5.x86_64.rpm
rpm -ivh $CENTOS/apr-1.2.7-11.x86_64.rpm
rpm -ivh $CENTOS/apr-util-1.2.7-6.x86_64.rpm
rpm -ivh $CENTOS/neon-0.25.5-5.1.x86_64.rpm
rpm -ivh $CENTOS/perl-URI-1.35-3.noarch.rpm
rpm -ivh $CENTOS/subversion-1.4.2-2.el5.x86_64.rpm
rpm -ivh $CENTOS/libtool-ltdl-1.5.22-6.1.x86_64.rpm
rpm -ivh $CENTOS/libtool-1.5.22-6.1.x86_64.rpm
rpm -ivh $CENTOS/libtool-ltdl-devel-1.5.22-6.1.x86_64.rpm
rpm -ivh $CENTOS/guile-1.8.0-8.20060831cvs.x86_64.rpm
rpm -ivh $CENTOS/libX11-devel-1.0.3-8.0.1.el5.x86_64.rpm $CENTOS/libXau-devel-1.0.1-3.1.x86_64.rpm $CENTOS/xorg-x11-proto-devel-7.1-9.el5.centos.x86_64.rpm $CENTOS/mesa-libGL-devel-6.5.1-7.5.el5.x86_64.rpm $CENTOS/libXdmcp-devel-1.0.1-2.1.x86_64.rpm
rpm -ivh $CENTOS/libjpeg-devel-6b-37.x86_64.rpm
rpm -ivh $CENTOS/libpng-1.2.10-7.0.2.x86_64.rpm
rpm -ivh $CENTOS/freetype-2.2.1-19.el5.x86_64.rpm
rpm -ivh $CENTOS/freetype-devel-2.2.1-19.el5.x86_64.rpm
rpm -ivh $CENTOS/gd-devel-2.0.33-9.3.fc6.x86_64.rpm $CENTOS/fontconfig-devel-2.4.1-6.el5.x86_64.rpm $CENTOS/libXpm-devel-3.5.5-3.x86_64.rpm $CENTOS/libpng-devel-1.2.10-7.0.2.x86_64.rpm $CENTOS/gd-2.0.33-9.3.fc6.x86_64.rpm

1. Install proj4
-tar -zxvf proj-4.4.9.tar.gz
-./configure
-make
-make install

2. Install geos
-bzip2 -d geos-3.0.0.tar.bz2
-tar -xvf geos-3.0.0.tar
-./configure –enable-python
-make
-make install

3. install postgres and postgis
rpm -ivh postgresql-libs-8.3.1-1PGDG.rhel5.x86_64.rpm
rpm -ivh postgresql-8.3.1-1PGDG.rhel5.x86_64.rpm
rpm -ivh postgresql-devel-8.3.1-1PGDG.rhel5.x86_64.rpm
rpm -ivh postgresql-server-8.3.1-1PGDG.rhel5.x86_64.rpm
rpm -ivh postgresql-contrib-8.3.1-1PGDG.rhel5.x86_64.rpm
rpm -ivh postgresql-plperl-8.3.1-1PGDG.rhel5.x86_64.rpm
rpm -ivh postgresql-plpython-8.3.1-1PGDG.rhel5.x86_64.rpm

# tar -zxvf postgis-1.3.3.tar.gz
# cd postgis-1.3.3
# ./configure –with-pgsql=/usr/bin/pg_config
# make
# make install

If you need to put the postgres data for mapserver, then:
initdb -E utf8 -D /var/lib/pgsql/data

3. install gdal
./configure –with-png –with-libtiff –with-static-proj4=/usr/local/bin –with-python –with-perl –with-geos –with-pg=/usr/bin/pg_config

Note: If there is an error regarding libxpat.so because of 64 bit libraries, then edit GDALmake.opt and change /usr/lib to /usr/lib64
http://www.nabble.com/GDAL-v1.5.1-compile-error-on-RHEL5.-td17428872.html

GDAL is now configured for i686-pc-linux-gnu

Installation directory: /usr/local
C compiler: gcc -O2
C++ compiler: g++ -O2

LIBTOOL support: yes

LIBZ support: external
GRASS support: no
CFITSIO support: no
PCRaster support: internal
NETCDF support: no
LIBPNG support: external
LIBTIFF support: external
LIBGEOTIFF support: internal
LIBJPEG support: external
LIBGIF support: external
OGDI support: no
HDF4 support: no
HDF5 support: no
KAKADU support: no
JASPER support: no
ECW support: no
MrSID support: no
CURL support (wcs): yes
POSTGRESQL support: yes
MySQL support: yes
XERCES support: no
ODBC support: no
PGEO support: no
OCI support: no
SDE support: no
DODS support: no
SQLite support: no
DWGdirect support no
PANORAMA GIS support: no
INFORMIX DataBlade support:no
GEOS support: yes

Statically link PROJ.4: yes

Traditional Python: yes
NG SWIG Bindings: perl

enable OGR building: yes

make
make install

4. Remove any existing apache from rpm then install apache2 by source.
rpm -e httpd-2.2.3-11.el5.centos gnome-user-share-0.10-6.el5.x86_64
./configure –prefix=/usr/local/apache2 –enable-rewrite –enable-so –with-mpm=prefork
make
make install

5. rpm -ivh alsa-lib-devel-1.0.14-1.rc4.el5.x86_64.rpm esound-0.2.36-3.x86_64.rpm esound-devel-0.2.36-3.x86_64.rpm audiofile-0.2.6-5.x86_64.rpm mesa-libGLU-6.5.1-7.5.el5.x86_64.rpm mesa-libGLU-devel-6.5.1-7.5.el5.x86_64.rpm libXext-1.0.1-2.1.x86_64.rpm libXext-devel-1.0.1-2.1.x86_64.rpm libXrandr-devel-1.1.1-3.1.x86_64.rpm libXrender-devel-0.9.1-3.1.x86_64.rpm libXt-devel-1.0.2-3.1.fc6.x86_64.rpm audiofile-devel-0.2.6-5.x86_64.rpm libSM-devel-1.0.1-3.1.x86_64.rpm libICE-devel-1.0.1-2.1.x86_64.rpm

6. rpm -ivh SDL-1.2.10-8.el5.x86_64.rpm SDL-devel-1.2.10-8.el5.x86_64.rpm

7. Dont use agg-2.4! PLEASE read this post from mapserver trac ticket. Instead, download the packages form http://dag.wieers.com/rpm/packages/agg/

Note: For 64-bit packages:
- http://dag.wieers.com/rpm/packages/agg/agg-2.5-1.el5.rf.x86_64.rpm
- http://dag.wieers.com/rpm/packages/agg/agg-devel-2.5-1.el5.rf.x86_64.rpm

8. install mapserver. This assumes you have PHP, APACHE, POSTGRES, POSTGIS,
MYSQL already installed.

./configure \
–with-agg \
–with-jpeg \
–with-gd \
–with-freetype \
–with-png \
–with-ogr \
–with-proj \
–with-gdal \
–with-httpd=/usr/local/apache2/bin/httpd \
–with-tiff \
–with-wfs \
–with-wcs \
–with-sos \
–with-wmsclient \
–with-wfsclient \
–with-geos=/usr/local/bin/geos-config \
–with-gdal=/usr/local/bin/gdal-config \
–with-postgis=/usr/bin/pg_config

-make
-make install

9. Post Install Considerations:
ln -s /usr/local/lib/libproj.so.0 /usr/lib/libproj.so.0
ln -s /usr/local/lib/libgdal.so.1 /usr/lib/libgdal.so.1
ln -s /usr/local/lib/libgeos_c.so.1 /usr/lib/libgeos_c.so.1
ldconfig

10. cp -rf mapserv /usr/local/apache2/cgi-bin/
check mapserv
./mapserv -v
MapServer version 5.0.3 OUTPUT=GIF OUTPUT=PNG OUTPUT=JPEG OUTPUT=WBMP OUTPUT=SVG SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=SOS_SERVER SUPPORTS=GEOS INPUT=EPPL7 INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE

11. cp the ff:
cp legend scalebar shp2img shp2pdf shptree shptreetst shptreevis sortshp
tile4ms /usr/local/apache2/cgi-bin/

12. cp $mapserver_install_dir/mapscript/php3/php_mapscript.so /usr/local/apache2/modules/

######################################
11. edit httpd.conf
setenv LD_LIBRARY_PATH /usr/local/lib

12. vi /etc/ld.so.conf
Add the following lines:
/usr/local/include
/usr/local/lib
run:
13 ldconfig