mobile and gis dev notes » oracle

iPhone Note #8: Exporting Oracle to SQLite3

Rupert — Sun, 16 Aug 2009 12:31:09 +0000

Since I’ve been working most of my time with iPhone Dev for the last couple of months, I thought it will be worthwile to post how to import files to SQLite3. In a nutshell, use MesaSQLite’s IMPORT function. It will save you a lot of time. Now it is up to you how to export into a CSV or TAB delimited file. For Oracle, I used SQLDeveloper. For other databases, such as MySQL or Postgres, I’ve used Navicat.

1. Using SQLDeveloper -> Right Click on the Table from the left pane -> Export

2. Choose TEXT since we are going to use that when importing using MesaSQLite. Note, SQLDeveloper does not export CLOB columns. You need to change your column from CLOB to VARCHAR. A workaround is to add a column and then update that column aferwards.

ALTER TABLE poi ADD en_desc VARCHAR(2000);
UPDATE poi SET en_desc = en_short_desc;

3. In MesaSQLite, FILE -> IMPORT -> CSV/TAB

4. Choose the table which you want to import.

5. Choose your exported TAB delimited file from SQLDeveloper.

6. Afterwards, map the corresponding fields to your database. Note, that I added an extra column “dummy” in the file because it seems I can only match n-1 columns, thus the extra column.

Two Oracle Homes in one Machine

rupert — Thu, 11 Jun 2009 01:29:12 +0000

I’ve been toggling back and forth two different oracle homes by editing the environment variables from the Control Panel in Windows which is a pain in the ***. What I did was, to create two different command prompt shells with different oracle home environments. I got this idea after a glimpse from FWTools.

1. Create a Command Prompt Shortcut and drag it to your oracle directory, i.e, E:\oracle\Oracle Shell Local

2. Edit the target as:

C:\WINDOWS\system32\cmd.exe /K "E:\oracle\setlocal.bat"

3. Create “setlocal.bat”

@echo off
SET ORACLE_HOME=E:\oracle\product\10.2.0\db_2
set PATH=E:\oracle\product\10.2.0\db_2\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem

Do the same for your remote oracle but with a different bat file.
1. Repeat steps 1 and 2 but name it “Oracle Shell Remote” and “E:\oracle\setremote.bat”

2. Create “setremote.bat”

@echo off
SET ORACLE_HOME=E:\oracle\product\10.2.0\client_1
set PATH=E:\oracle\product\10.2.0\client_1\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem

Note: This assumes that you have installed an Oracle Client on E:\oracle\product\10.2.0\client_1

Ok, now let’s test. I know that I have different record count for a table in my local and remote
1. Fire up “Oracle Shell Local”. Run sqlplus
>sqlplus username/password@instance_name_defined_in_local_tnsnames.ora

Note: E:\oracle\product\10.2.0\db_2\NETWORK\ADMIN\tnsnames.ora

2. Fire u p “Oracle Shell Remote”. Run sqlplus
>sqlplus username/password@instance_name_defined_in_remote_tnsnames.ora

Note: E:\oracle\product\10.2.0\client_1\NETWORK\ADMIN\tnsnames.ora

So what? Well its very useful when doing export and imports. Say I want to export a table from my local and import it to my remote oracle. I’ll just fire up two shells, issue an export in my “Oracle Shell Local” and run an import command in my “Oracle Shell Remote”

Rails Note #13: RubyonRails + Oracle on Linux (i386 / x64)

rupert — Thu, 11 Dec 2008 00:53:48 +0000

In summary, install Oracle Instant Client and try to run sqlplus. If sqlplus connects to the oracle sid then go ahead and install the rails adapters for oracle. What is important to note here, is to install the oracle-instantclient for the architecture of your machine.. I have tested this on Debian Lenny (i386) and CentOS5 (x64)

1. Download from http://www.oracle.com/technology/software/tech/oci/instantclient/

a. oracle-instantclient-basic-10.2.0.4-1.i386
b. oracle-instantclient-devel-10.2.0.4-1.i386
c. oracle-instantclient-sqlplus-10.2.0.4-1.i386

2. Unzip everything to /opt/oracle/instantclient . You should have something like the ff:

[root@csapp1 instantclient]# ls -la
total 102704
drwxr-xr-x 3 root root     4096 Dec 10 21:54 .
drwxr-xr-x 3 root root     4096 Dec 10 22:03 ..
-rw-r--r-- 1 root root      228 Dec 10 21:52 BASIC_README
-r--r--r-- 1 root root  1609607 Dec 10 21:52 classes12.jar
-rwxr-xr-x 1 root root    67542 Dec 10 21:52 genezi
-r--r--r-- 1 root root     1525 Dec 10 21:52 glogin.sql
lrwxrwxrwx 1 root root       17 Dec 10 21:54 libclntsh.so -> libclntsh.so.10.1
-rwxr-xr-x 1 root root 21038613 Dec 10 21:52 libclntsh.so.10.1
-r-xr-xr-x 1 root root  3796601 Dec 10 21:52 libnnz10.so
-rwxr-xr-x 1 root root  1664116 Dec 10 21:52 libocci.so.10.1
-rwxr-xr-x 1 root root 72674185 Dec 10 21:52 libociei.so
-r-xr-xr-x 1 root root   138033 Dec 10 21:52 libocijdbc10.so
-r-xr-xr-x 1 root root  1435561 Dec 10 21:52 libsqlplusic.so
-r-xr-xr-x 1 root root   997409 Dec 10 21:52 libsqlplus.so
-r--r--r-- 1 root root  1555682 Dec 10 21:52 ojdbc14.jar
drwxr-xr-x 4 root root     4096 Dec 10 21:52 sdk
-r-xr-xr-x 1 root root     7773 Dec 10 21:52 sqlplus
-rw-r--r-- 1 root root      232 Dec 10 21:52 SQLPLUS_README
-rw-r--r-- 1 root root      516 Dec 10 21:53 tnsnames.ora
[root@csapp1 instantclient]#

3. Make a symbolic link for libclntsh.so.10.1 as shown above.

4. Create the Oracle Environment variables

export ORACLE_HOME=/opt/oracle/instantclient
export TNS_ADMIN=$ORACLE_HOME
export LD_LIBRARY_PATH=$ORACLE_HOME:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$ORACLE_HOME
export PATH=$PATH:$ORACLE_HOME

5. At this point, you should have oracle-instantclient properly installed. You can test by running sqlplus.

[root@csapp1 instantclient]# sqlplus
 
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Dec 11 11:47:40 2008
 
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

NOTE: Sometimes you will get a SEGMENTATION FAULT. If so, try to open a new shell with the environment variables loaded and do an sqlplus in a directory which is not /opt/oracle/instantclient.

6. Install the oracle adapter for rails

7. gem install ruby-oci8

8. gem install oracle_enhanced-adapter –source=”http://gems.rubyonrails.org/”

activerecord-oracle-adapter (1.0.0.9250)
activerecord-oracle_enhanced-adapter (1.1.8)

NOTE: Try to look for the latest gems, the source above is at the time of this writing so it might change.

9. Test using irb

[root@csapp1 instantclient]# irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'oci8'
=> true
irb(main):003:0>

Rails Note #12: Oracle on Intel Mac

rupert — Fri, 05 Dec 2008 07:52:28 +0000

1. Read http://www.foliosus.com/2008/05/05/connecting-ruby-on-rails-to-oracle-on-an-intel-mac-in-leopard-take-2/

2. Install Oracle Instant Client on Mac.

a. Instant Client Package – Basic: All files required to run OCI, OCCI, and JDBC-OCI applications
- instantclient-basic-macosx-10.2.0.4.0.zip (34,020,719 bytes)

b. *Instant Client Package – SDK: Additional header files and an example makefile for developing Oracle applications with Instant Client
instantclient-sdk-macosx-10.2.0.4.0.zip (603,493 bytes)

OR download the whole bundle (10.2.0.4.zip) with sqlplus installed from my installers.

3. Put this on your sudo vim ~/.bash_profile.

export ORACLE_HOME=/Library/Oracle/instantclient/10.2.0.4
export TNS_ADMIN=$ORACLE_HOME
export LD_LIBRARY_PATH=$ORACLE_HOME
export DYLD_LIBRARY_PATH=$ORACLE_HOME
export PATH=$PATH:$ORACLE_HOME

4. Make a symbolic link

cd /Library/Oracle/instantclient/10.2.0.4
ln -s libclntsh.dylib.10.1 libclntsh.dylib

5. Go to /Library/Oracle/instantclient/10.2.0.4 and edit tnsnames.ora. Point the Oracle SID to the IP where you installed Oracle.

ORCL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.155)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl)
    )   
  )
 
 
ORCL_2_11 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.11)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl)
    )   
  )

6. Install the oracle-adapter for rails

sudo gem install activerecord-oracle-adapter --source http://gems.rubyonrails.org

7. In your database.yml file

development:
  adapter: oracle
  database: orcl
  username: youzhu_mobile_dev
  password: your_password

or browse the contents of a sample rails project youzhumobile.tar.gz

8. If you ever encounter an encoding problem, then we need to set the NLS_LANG environment variable before running script/server.

# export NLS_LANG=American_America.UTF8
# script/server

or I prefer setting it in the environment.rb

Rails::Initializer.run do |config|
  ENV['NLS_LANG']='American_America.UTF8'
  # Settings in config/environments/* take precedence over those specified here.

Note: If you don’t know your database encoding, then read this post.

Oracle Spatial Tutorial Screencasts

rupert — Tue, 11 Nov 2008 11:01:53 +0000

This is the first time I ever did a screencast and it was a lot of fun! Please do forgive me as my voice was horrible. I used iShowU and keyCaster. Google it!

I hope you guys enjoyed it..as much as I did..

oracle_spatial_1.mov
oracle_spatial_2.mov

How to make Oracle start on boot

rupert — Mon, 22 Sep 2008 12:57:45 +0000

1. Login as root and edit /etc/oratab to reflect “Y”

orcl:/opt/oracle/product/11.1.0/db_1:Y

2. oracle startup script:

#!/bin/bash
#
# oracle Init file for starting and stopping
# Oracle Database. Script is valid for 10g and 11g versions.
#
# chkconfig: 35 80 30
# description: Oracle Database startup script
 
# Source function library.
 
. /etc/rc.d/init.d/functions
 
ORACLE_OWNER="oracle"
ORACLE_HOME="/opt/oracle/product/11.1.0/db_1"
 
case "$1" in
start)
echo -n $"Starting Oracle DB:"
su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/dbstart $ORACLE_HOME"
su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/lsnrctl start"
echo "OK"
;;
stop)
echo -n $"Stopping Oracle DB:"
su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/dbshut $ORACLE_HOME"
su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/lsnrctl stop"
echo "OK"
;;
*)
echo $"Usage: $0 {start|stop}"
esac

3. Oracle EMCTL

#!/bin/bash
#
# oraemctl Starting and stopping Oracle Enterprise Manager Database Control.
# Script is valid for 10g and 11g versions.
#
# chkconfig: 35 80 30
# description: Enterprise Manager DB Control startup script
 
# Source function library.
 
. /etc/rc.d/init.d/functions
 
ORACLE_OWNER="oracle"
ORACLE_HOME="/opt/oracle/product/11.1.0/db_1"
 
case "$1" in
start)
echo -n $"Starting Oracle EM DB Console:"
su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/emctl start dbconsole"
echo "OK"
;;
stop)
echo -n $"Stopping Oracle EM DB Console:"
su - $ORACLE_OWNER -c "$ORACLE_HOME/bin/emctl stop dbconsole"
echo "OK"
;;
*)
echo $"Usage: $0 {start|stop}"
esac

4. chkconfig –add /etc/init.d/oracle
5. chkconfig –add /etc/init.d/oraemctl

Oracle Tuning Tools

rupert — Wed, 03 Sep 2008 03:11:36 +0000

EXPLAIN PLAN

1. Create the PLAN_TABLE
SQL> $ORACLE_HOME/rdbms/admin/utlxplan.sql

2. Run EXPLAIN PLAN for an SQL
SQL> EXPLAIN PLAN
2 SET STATEMENT_ID = ‘example’ FOR
3 SELECT ename,dname
4 FROM emp inner join dept
5 ON ( emp.deptno = dept.deptno )
6 /
Explained.

3. DISPLAY Results
SQL> $ORACLE_HOME/rdbms/admin/utlxpls.sql

AUTOTRACE is a SQL*Plus facility that may be enabled in your database. To get this up and running you need to have administration rights to the system and perform the following steps:

* Log into SQL*Plus as SYSDBA

* Run the script $ORACLE_HOME/sqlplus/admin/plustrce

* Grant PLUSTRACE to SPATIAL (or to specific users/roles)

To Use:

SQL> SET autotrace ON
 
SQL> SELECT ename,dname
  2    FROM emp  INNER JOIN dept
  3      ON ( emp.deptno = dept.deptno )
  4  /
 
ENAME      DNAME
---------- --------------
SMITH      RESEARCH
ALLEN      SALES
WARD       SALES
JONES      RESEARCH
MARTIN     SALES
BLAKE      SALES
CLARK      ACCOUNTING
SCOTT      RESEARCH
KING       ACCOUNTING
TURNER     SALES
ADAMS      RESEARCH
JAMES      SALES
FORD       RESEARCH
MILLER     ACCOUNTING
 
14 rows selected.
 
Execution Plan
----------------------------------------------------------
   0      SELECT STATEMENT Optimizer=CHOOSE (Cost=3 Card=14 Bytes=252)
   1    0   HASH JOIN (Cost=3 Card=14 Bytes=252)
   2    1     TABLE ACCESS (FULL) OF 'DEPT' (Cost=1 Card=4 Bytes=44)
   3    1     TABLE ACCESS (FULL) OF 'EMP' (Cost=1 Card=14 Bytes=98)
 
Statistics
----------------------------------------------------------
          0  recursive calls
          4  db block gets
          3  consistent gets
          0  physical reads
          0  redo size
       1132  bytes sent via SQL*Net TO client
        503  bytes received via SQL*Net FROM client
          2  SQL*Net roundtrips TO/FROM client
          0  sorts (memory)
          0  sorts (disk)
         14  rows processed

Courtesy of Beginning Oracle Programming by Sean Dillon et al

Tutorials Updated for Oracle

rupert — Tue, 02 Sep 2008 10:57:29 +0000

It was a gruesome day.. I have an oracle10g on my MacBookPro running on VMWare (WindowsXPSP3, let’s call this orawin). I have another setup box (CentOS5 + oracle11g = oralin) just this morning. I imported the tables from my orawin10g to my oralin11g, created the indexes for the tables in oralin11g. I ran my spatial test query on orawin10(0.692192571 secs); I ran the same query on oralin11g (14.89 secs). I viewed the execution plans and both are the same!

Eventually, whatever works.. I have to drop the oracle11g and install Oracle10g instead. I have updated the Tutorials Page to include this, as well as the Oracle CheatSheet.

Oracle SQL CheatSheet

rupert — Wed, 27 Aug 2008 12:15:18 +0000

References:
Comparison of different SQL implementations

Oracle DataTypes

Part 1: Oracle Misc Information

*Oracle Services Running on Windows?

* How to create a user?

CREATE USER "APPDEV" PROFILE "DEFAULT" IDENTIFIED BY "*******" DEFAULT TABLESPACE "USERS" TEMPORARY TABLESPACE "TEMP" ACCOUNT UNLOCK
GRANT UNLIMITED TABLESPACE TO "APPDEV";
GRANT "CONNECT" TO "APPDEV";
GRANT "RESOURCE" TO "APPDEV";

* How to load data in bulk?

SQLLDR username/password CONTROL=filename DATA=filename

Note:
1. You can specify the CHARACTERSET UTF8 in your control file for multilingual databases.

LOAD DATA
CHARACTERSET UTF8
INFILE *
REPLACE INTO TABLE LOADER_TEST
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS (
USR_ID         INTEGER EXTERNAL,
USR_NAME       CHAR(50),
USR_LNK_NAME   CHAR(50),
USR_LNK_ORDER  INTEGER EXTERNAL
)

2. Sometimes SQLLDR does not display anything when loading. Be sure to issue a “commit” command before doing anything.

* How to use EXPORT / IMPORT?

EXP scott/tiger@instance FILE=geo_entities.dmp TABLES=geo_entities
 
IMP sysman/password FILE=/path/TO/geo_entities.dmp FROMUSE=scott TOUSER=appdev IGNORE=Y INDEXES=N TABLES=geo_entities

Note: Delete records first before importing

* What is the database encoding of my database?

SELECT * FROM V$NLS_PARAMETERS

Part 2: Oracle SQL

* ADD COLUMN in a TABLE

ALTER TABLE table_name ADD (column_name  NUMBER);

* CHANGE COLUMN NAME in a TABLE

ALTER TABLE geo_entities MODIFY meta_name varchar(255)

* DROP COLUMN NAME in a TABLE

ALTER TABLE table_name DROP COLUMN column_name;

* ADD PRIMARY KEY CONSTRAINT on a COLUMN

ALTER TABLE table_name ADD CONSTRAINT table_name_col_pk PRIMARY KEY(column_name);

* Select Top N rows

SELECT * FROM TABLE_NAME WHERE ROWNUM <= 100

* CREATING AN AUTO INCREMENT COLUMN

CREATE SEQUENCE seq_table_name_pk INCREMENT BY 1 START WITH 10000;
 
INSERT INTO table_name (pid, en_name) 
SELECT seq_table_name_pk.NEXTVAL, en_name FROM other_table

* Concatenating Strings

SELECT concat('hello', 'rupert') FROM DUAL;
SELECT 'hello' || 'rupert' FROM DUAL;

* UPDATE TABLE

UPDATE poi_temp pt
SET geom = (SELECT geom FROM poi_app WHERE poi_id = pt.poi_id)
WHERE EXISTS (SELECT 1 FROM poi_app WHERE poi_id = pt.poi_id)
 
UPDATE poi_app pa
SET (long_900913, lat_900913) = (SELECT pg.long_900913, pg.lat_900913 FROM poi_app_900913 pg WHERE pa.poi_id = pg.poi_id)
WHERE EXISTS (SELECT 1 FROM poi_app_900913 pg WHERE pa.poi_id = pg.poi_id)

Part 3: Oracle Spatial

* What is SDO_GEOMETRY?

SDO_GEOMETRY{
    SDO_GTYPE, - GeometryType: D00T
    D: Dimension (2: 2d, 3: 3d, 4:4d)
        T:GeometryType(
        0 - unknown
        1 - point
        2 - line
        3 - polygon
        4 - collection
        5 - multipoint
        6 - multiline
        7 - multipolygon
    )
    SDO_SRID,
    SDO_POINT, - NULL for line, polygon, etc.
    SDO_ELEM_INFO,
    SDO_ORDINATES
}

* How to know the geometry type?

THE_GEOM(SDO_GTYPE, SDO_SRID, SDO_POINT(X, Y, Z), SDO_ELEM_INFO, SDO_ORDINATES)
--------------------------------------------------------------------------------
SDO_GEOMETRY(3, 4326, NULL, SDO_ELEM_INFO_ARRAY(1, 3, 1), SDO_ORDINATE_ARRAY(116
.281578, 39.854501, 116.281491, 39.853828, 116.281236, 39.853181, 116.280821, ....39
.855174, 116.281578, 39.854501))
 
THE_GEOM(SDO_GTYPE, SDO_SRID, SDO_POINT(X, Y, Z), SDO_ELEM_INFO, SDO_ORDINATES)
--------------------------------------------------------------------------------
 
SQL> SELECT g.the_geom.sdo_gtype FROM geo_entities g WHERE rownum <= 1;
 
THE_GEOM.SDO_GTYPE
------------------
3

* How to create a spatial index?

INSERT INTO USER_SDO_GEOM_METADATA VALUES
(
    'table_name', -- TABLE_NAME
    'geom', -- COLUMN_NAME
    SDO_DIM_ARRAY -- DIMINFO attribute for storing dimension bounds, tolerance
    (
        SDO_DIM_ELEMENT
        (
            'LONGITUDE', -- DIMENSION NAME for first dimension
            -180, -- SDO_LB for the dimension
            180, -- SDO_UB for the dimension
            0.5 -- Tolerance of 0.5 meters
        ),
        SDO_DIM_ELEMENT
        (
            'LATITUDE', -- DIMENSION NAME for second dimension
            -90, -- SDO_LB for the dimension
            90, -- SDO_UB for the dimension
            0.5 -- Tolerance of 0.5 meters
        )
    ),
    4326 -- SRID value for specifying a geodetic coordinate system
);
 
CREATE INDEX idx_poi_app_geom ON poi_app(geom) INDEXTYPE IS MDSYS.SPATIAL_INDEX;
 
CREATE INDEX idx_poi_app_geom ON poi_app(geom)
INDEXTYPE IS MDSYS.SPATIAL_INDEX PARAMETERS('LAYER_GTYPE=POINT')

* How to VALIDATE a geometry?

SELECT SDO_GEOM.VALIDATE_GEOMETRY_WITH_CONTEXT(GEOM, 0.000000005) FROM table_name

* SPATIAL INTERSECTION

SELECT pc.postcode, au.authority_name, au.state, au.authority_type
FROM econ_authorities_valid_temp au, econ_postcodes pc
WHERE pc.POSTCODE = 3128
	  AND SDO_ANYINTERACT(
	  				  pc.GEOM,
	  				  au.GEOM
	  ) = 'TRUE

Note:

1. Both geometries should be in the same projection.

* Get POINTS in a POLYGON using VERTICES

SELECT SDO_UTIL.GETVERTICES( GEOM ) FROM ECON_AUTHORITIES WHERE AUTHORITYID = 90009

Using SDO_WITHIN_GEOM

rupert — Wed, 27 Aug 2008 11:00:21 +0000

Experiment 1: Two POIs

SELECT 
  p1.poi_id, 
  p1.en_name, 
  p1.py_fullpoiadd, 
  p1.py_rdname, 
  p2.poi_id, 
  p2.en_name, 
  p2.py_fullpoiadd, 
  p2.py_rdname,
  SDO_GEOM.SDO_DISTANCE(p1.geom, p2.geom, 0.5) AS dist
FROM poi_test p1, poi_test p2
WHERE 
	Upper(p1.en_name) LIKE '%PARKSON%'
AND	Upper(p2.en_name) LIKE '%KFC%'
AND SDO_WITHIN_DISTANCE(p1.geom, p2.geom, 'DISTANCE=500 UNIT=METER' )='TRUE'
ORDER BY dist

Experiment 2: Road + Two POIs

SELECT
  p1.poi_id, 
  p1.en_name, 
  p1.py_fullpoiadd, 
  p1.py_rdname, 
  p2.poi_id, 
  p2.en_name, 
  p2.py_fullpoiadd, 
  p2.py_rdname,
  SDO_GEOM.SDO_DISTANCE(p1.geom, p2.geom, 0.5) AS dist
FROM poi_app p1, poi_app p2, geo_entities g
WHERE 
	Upper(p1.en_name) LIKE '%BAR BLU%'
AND	Upper(p2.en_name) LIKE '%KOKOMO%'
AND     Upper(g.meta_name) LIKE '%SANLITUN%'
AND SDO_WITHIN_DISTANCE(p1.geom, p2.geom, 'DISTANCE=500 UNIT=METER' )='TRUE'
AND SDO_WITHIN_DISTANCE(p1.geom, g.the_geom, 'DISTANCE=500 UNIT=METER' )='TRUE'
AND SDO_WITHIN_DISTANCE(p2.geom, g.the_geom, 'DISTANCE=500 UNIT=METER' )='TRUE'

Notes:
1. Significant improvement when Sorting is removed.
2. SDO_WITHIN_DISTANCE vs NN? The first finds the nearest geometry within a given distance while NN finds the nearest geometry regardless of the distance. NN could be costly when unused properly.

Oracle Text Notes

rupert — Tue, 26 Aug 2008 14:33:34 +0000

Just a couple of notes from studying Oracle Text…

1. What is the default index?
Its CONTEXT. The text column can be of type CLOB, BLOB, BFILE, VARCHAR2, or CHAR.

CREATE INDEX idx_ft_meta_en_name ON poi_app(ft_meta_en_name) INDEXTYPE IS CTXSYS.CONTEXT;

2. When you perform inserts or updates on the base table, you must explicitly synchronize the index with CTX_DDL.SYNC_INDEX.

SQL> EXEC CTX_DDL.SYNC_INDEX(‘idx_docs’, ’2M’);

3. CONTAINS Phrase Queries

If multiple words are contained in a query expression, separated only by blank spaces (no operators), the string of words is considered a phrase and Oracle Text searches for the entire string during a query.

4. Logical Operators

5. Some sample SQL queries:

-- Simple Query
SELECT SCORE(1) AS myscore, en_name, en_visname, py_name FROM poi_app WHERE CONTAINS(ft_meta_en_name, 'grammy center', 1) > 0 ORDER BY myscore DESC;
 
SELECT SCORE(1) AS myscore, en_name, en_visname, py_name FROM poi_app WHERE CONTAINS(ft_meta_en_name, 'cybersoft', 1) > 0;
 
-- Query Rewrite
SELECT en_name, en_visname, py_name
FROM poi_app
WHERE CONTAINS (ft_meta_en_name,
'
 international hotel boya

transform((TOKENS, "{", "}", "AND"))
transform((TOKENS, "{", "}", "ACCUM"))



')>0;
 
-- Query 'About'
SELECT en_name, en_visname, py_name, score(1)
FROM poi_app
WHERE CONTAINS(ft_meta_en_name, 'about(italian restaurants)', 1) > 0
ORDER BY SCORE(1) DESC;
 
-- Query logical
SELECT en_name, en_visname, py_name, score(1)
FROM poi_app
WHERE CONTAINS(ft_meta_en_name, 'beijing, international, hotel', 1) > 0
ORDER BY SCORE(1) DESC;

Migrating Postgres / Postgis Tables to Oracle Spatial

rupert — Tue, 19 Aug 2008 07:26:10 +0000

Step 1: Installing and configuring

1. On Windows, make sure that Windows > Settings > Control Panel > Regional Settings
- Chinese (PRC)

2. Install Oracle10g
- make sure you have spatial installed

SQL> SELECT COMP_NAME, STATUS
FROM DBA_REGISTRY
WHERE COMP_NAME = 'Spatial';

3. Create a multilingual database using dbca. Make sure setting is set to UTF8. I installed the sample schema for learning purposes. Read this previous post for more information.

4. We can add a user for our spatial database or just enable scott/tiger from the Enterprise Manager.

On Administration > Users > “Search for SCOTT” > Hit EDIT > UNLOCK the status.

Step 2: Get Oracle Client Tools

1. On the same WindowsXPSP3 machine that I installed Oracle. There is already SQL*Plus. However I always go for the native which is CLI based. So on a command prompt, sqlplus still works. Also note that I can toggle to my previous commands using “Arrow Up”. If we have the Oracle Database installed on a different machine, we need to download the ff:

* Oracle SQL Developer (Java GUI-Based)

http://www.oracle.com/technology/software/products/sql/index.html

* Oracle 10g Client for Windows (10201_client_win32 1.zip 453MB)

* Oracle 10g Client for MacOS (Oracle_10204Client_MAC_X86.zip 189MB) – rarely used.

* Oracle Instant Client for MacOS (instantclient-basic-macosx-10.2.0.4.0.zip 32MB) – need to try this.

Step 3: Creating the Table

1. POI Table

CREATE TABLE poi
(
  poi_id number(10),
  cn_name varchar2(255),
  py_name varchar2(255),
  en_name varchar2(255),
  en_visname varchar2(255),
  cn_fullpoiadd varchar2(255),
  en_fullpoiadd varchar2(255),
  py_fullpoiadd varchar2(255),
  cn_rdname varchar2(255),
  py_rdname varchar2(255),
  cn_address_no varchar2(255),
  py_address_no varchar2(255),
  cn_address_other varchar2(255),
  py_address_other varchar2(255),
  postal varchar2(20),
  tel_no varchar2(255),
  fax_no varchar2(255),
  email varchar2(255),
  web_url varchar2(255),
  operating_hours  varchar2(4000),
  cards_accepted  varchar2(4000),
  cust_capacity  varchar2(4000),
  park_space  varchar2(4000),
  longitude number(20,8),
  latitude number(20,8),
  CONSTRAINT "poi_pkey" PRIMARY KEY (poi_id)
);

Step 4: Exporting to TextFile with “|”

Export the table into a textfile. Run the SQL statement below on Navicat. Afterwards, run the “Export Wizard” and specify “|” as the delimiter.

SELECT
  poi_id,
  cn_name,
  py_name,
  en_name,
  en_visname,
  cn_fullpoiadd,
  en_fullpoiadd,
  py_fullpoiadd,
  cn_rdname,
  py_rdname,
  cn_address_no,
  py_address_no,
  cn_address_other,
  py_address_other,
  postal,
  tel_no,
  fax_no,
  email,
  web_url,
  operating_hours,
  cards_accepted,
  cust_capacity,
  park_space,
  longitude,
  latitude
FROM poi

Find out how many lines were exported using “wc -l file-name”. If there are more lines than actual records then most likely there are ‘\n’ (newlines / carriage returns) on the exported file.

Step 5: Use SQLLOADER to bulkload the data

SQLLDR scott/tiger CONTROL=poi_full.ctl DATA=data_navicat_cn_all2.dat

Since we have point data in longitude, latitude columns. It is very easy to populate the SDO_GEOMETRY with these columns.

Step 6: Creating Point Geometries from Long/Lat Columns

CREATE TABLE poi_app AS SELECT * FROM poi WHERE latitude > 0 AND longitude > 0;
 
ALTER TABLE poi_app ADD the_geom SDO_GEOMETRY;
 
UPDATE poi_app 
SET the_geom = SDO_GEOMETRY( 2001, NULL, SDO_POINT_TYPE(longitude, latitude, NULL), NULL, NULL);
 
INSERT INTO USER_SDO_GEOM_METADATA VALUES
(
	'poi_app', -- TABLE_NAME
	'the_geom', -- COLUMN_NAME
	SDO_DIM_ARRAY -- DIMINFO attribute for storing dimension bounds, tolerance
	(
		SDO_DIM_ELEMENT
		(
			'LONGITUDE', -- DIMENSION NAME for first dimension
			-180, -- SDO_LB for the dimension
			180, -- SDO_UB for the dimension
			0.5 -- Tolerance of 0.5 meters
		),
		SDO_DIM_ELEMENT
		(
			'LATITUDE', -- DIMENSION NAME for second dimension
			-90, -- SDO_LB for the dimension
			90, -- SDO_UB for the dimension
			0.5 -- Tolerance of 0.5 meters
		)
	),
	4326 -- SRID value for specifying a geodetic coordinate system
);
 
CREATE INDEX idx_poi_app_the_geom ON poi_app(the_geom) INDEXTYPE IS MDSYS.SPATIAL_INDEX;

Once the index is created, Oracle creates a SEQUENCE(MDRS_C796$) and a TABLE(MDRT_C796$). Please do not delete this table. I accidentally deleted this table and I have to recreate the indexes–well it was not that easy. I have to DROP INDEX, insert it into USER_SDO_GEOM_METADATA, then CREATE INDEX.

Note that it is important to include the geometry to be indexed in the USER_SDO_GEOM_METADATA table. This should happen before creating the index or receive an error like…

Step 7: Creating Geometries from ESRI Shapefiles

* PGSQL2SHP – Use this to convert from a postgres table to an ESRI Shapefile. Note that if you have mixed geometries in a single column, then you need output different shapefiles for each geometry. For example, I have a “geo_entities” table which contains polygons, multipolygons, and multilinestrings. I was able to come up with three (3) different shapefiles for each geometry. Then afterwards load the individual shapefiles using SDO2SHP below.

pgsql2shp -f geo_entities_2006 -h 127.0.0.1 -u lbs -P ******* -g the_geom beijing_app "SELECT gid,cn_name,py_name,en_name,entity_type,geom_type,meta_name,cn_district,the_geom FROM geo_entities WHERE GeometryType(the_geom) = 'MULTILINESTRING'"

* Ensure that you download SDO2SHP

shp2sdo shp\geo_entities_2007 geo_entities_2007 -g the_geom -x (-180,180) -y (-90,90) -s 4326 
-t 0.5 -v

The output of SHP2SDO are three (3) files:

a. CTL – control file containing “LOAD DATA…”
b. SQL – contains “CREATE TABLE…”
c. DAT – Data with “#”

Oracle: Creating a MultiLingual Database

rupert — Tue, 19 Aug 2008 06:47:29 +0000

Notes on to create a multilingual database in UTF8 in Oracle10g.

1. Run the Oracle Database Configuration Assistant (DBCA)

2. Choose Create Database

3. Choose “General Purpose”.

4. Then specify the GLOBAL DATABASE NAME or ORACLE SID (orcl4)

5. Click Next on Management Options.

6. On “Database Credentials”, use the same passwords so we would not forget them for now…

7. On “Storage Options”. Default options is FILE STORAGE. Next.

8. On “Database File Locations”, hit Next as well.

9. On “Recover Options”, same goes here.. Next.

10. On “Database Content”, install the sample schemas for practice.

11. On “Initialization Parameters”, click on the “Character Sets” tab. Note that we chose the 2nd option which is “Use Unicode” (ALT32UTF8) in order to support chinese. On the National Character Set, please choose “UTF8″ (the 2nd option in the drop-down list”) also.

12. Hit Next until we create the database.

13. To check you should have the Oracle Services installed in your Services.

Installing Oracle11g x64 on CentOS5

rupert — Sat, 26 Jul 2008 04:36:25 +0000

FOR EDITING.. This is just a couple of notes from my Oracle testing a few weeks back that I forgot to post..

A. Installing

1. Check for rpm packages:

rpm -q binutils compat-db control-center gcc gcc-c++ glibc glibc-common gnome-libs libstdc++ libstdc++-devel make pdksh sysstat xscreensaver libaio openmotif21 elfutils-libelf libaio-devel libgcc

NOTE: For 10g, you need libXp.so.6 installed. It seems the oracle installer is messed up with some i386 modules pointing to x86? Anyway, I was able to resolve the problem by installing the libXp.i386.rpm version.

2. install the necessary rpm packages:

rpm -ivh compat-db-4.2.52-5.1.x86_64.rpm 
rpm -ivh libgnome-2.16.0-6.el5.x86_64.rpm 
rpm -ivh sysstat-7.0.0-3.el5.x86_64.rpm 
rpm -ivh libaio-devel-0.3.106-3.2.x86_64.rpm 
rpm -ivh lm_sensors-2.10.0-3.1.x86_64.rpm
rpm -ivh beecrypt-devel-4.1.2-10.1.1.x86_64.rpm elfutils-devel-0.125-3.el5.x86_64.rpm net-snmp-devel-5.3.1-19.el5.x86_64.rpm elfutils-devel-static-0.125-3.el5.x86_64.rpm net-snmp-5.3.1-19.el5.x86_64.rpm net-snmp-libs-5.3.1-19.el5.x86_64.rpm

3. vim /etc/profile

ORACLE_BASE=/opt/oracle/product
ORACLE_SID=csdbora
ORACLE_HOME=/opt/oracle/product/11.1.0/db_1
 
export ORACLE_BASE ORACLE_SID ORACLE_HOME
 
PATH=$PATH:$ORACLE_HOME/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib

4. Create oracle Account

groupadd oinstall
groupadd dba
useradd -m -g oinstall -G dba oracle
id oracle

uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)

5. Run modify_kernel.sh

cat >> /etc/sysctl.conf << EOF
kernel.shmall = 2097152
kernel.shmmax = 1073741824 
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1025 65000
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_max = 262144
EOF

6. vim /etc/sysctl.conf and comment kernel.shmall and kernel.shmmax because if
you don’t then we will have double results

7. /sbin/sysctl -p

8. Create Directories:

mkdir -p /u01
mkdir -p /opt/oracle
mkdir -p /opt/oracle/product
chown -R oracle:oinstall /u01
chown -R oracle:oinstall /opt/oracle
chmod -R 755 /u01
chmod -R 755 /opt/oracle
chmod -R 755 /opt/oracle/product

9. Run sh -v modify_shelllimits.sh

10. Add the ff lines in /etc/pam.d/login
session required /lib/security/pam_limits.so
session required pam_limits.so

11. cp -rf /home/installers/linux.x64_11gR1_database.zip /opt/oracle

12. chown oracle:oinstall /opt/oracle/linux.x64_11gR1_database.zip

13. Clost all terminals and logout as root from desktop.

14. Login as oracle

15. Follow tutorial until installing oracle
Installing Oracle 11g on CentOS under VMWare on a Macbook.doc

To extract: cpio -idmv < 10201_database_linux_x86_64.cpio

vim /etc/redhat-release
Redhat 4

16. Run the 2 scripts after the dialog:

[root@cs5ora11g db1]# sh root.sh
Running Oracle 11g root.sh script…

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /opt/oracle/product/11.1.0/db1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin …
Copying oraenv to /usr/local/bin …
Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

17. Open your firewall to 1158

18. Go to https://192.168.1.155:1158/em

- Login as SYSMAN:[password] (nickname+wife)

19. Install Oracle Instant Client for your distro [MacOS]

http://www.oracle.com/technology/software/tech/oci/instantclient/index.html

a. Add exports to runsqlplus.sh
b. copy tnsnames.ora from centos to mac
c. add in firewall: 1521
d. sh runsqlplus.sh

20. do sqlplus
Enter user-name: SYSMAN@cybersof
Enter password:

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>

You are now connected successfully.

21. Check for oracle processes: ps -ef | grep ora

SQLPLUS DBINSTANCE:
oraclecybersof (LOCAL=NO)
 
 
ORACLE LISTENER:
/opt/oracle/product/11.1.0/db1/bin/tnslsnr LISTENER -inherit
 
ORACLE ENTERPRISE MANAGER:
oracle   14487     1  0 15:37 ?        00:00:00
/opt/oracle/product/11.1.0/db1/perl/bin/perl
/opt/oracle/product/11.1.0/db1/bin/emwd.pl dbconsole
/opt/oracle/product/11.1.0/db1/cs5ora11g_csmlcc/sysman/log/emdb.nohup
oracle   14511 14487  2 15:37 ?        00:00:36
/opt/oracle/product/11.1.0/db1/jdk/bin/java -server -Xmx512M
-XX:MaxPermSize=1024M -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40
-DORACLE_HOME=/opt/oracle/product/11.1.0/db1
-Doracle.home=/opt/oracle/product/11.1.0/db1/oc4j
-Doracle.oc4j.localhome=/opt/oracle/product/11.1.0/db1/cs5ora11g_csmlcc/sysman
-DEMSTATE=/opt/oracle/product/11.1.0/db1/cs5ora11g_csmlcc
-Doracle.j2ee.dont.use.memory.archive=true
-Djava.protocol.handler.pkgs=HTTPClient
-Doracle.security.jazn.config=/opt/oracle/product/11.1.0/db1/oc4j/j2ee/OC4J_DBConsole_cs5ora11g_csmlcc/config/jazn.xml
-Djava.security.policy=/opt/oracle/product/11.1.0/db1/oc4j/j2ee/OC4J_DBConsole_cs5ora11g_csmlcc/config/java2.policy
-Djavax.net.ssl.KeyStore=/opt/oracle/product/11.1.0/db1/sysman/config/OCMTrustedCerts.txt-Djava.security.properties=/opt/oracle/product/11.1.0/db1/oc4j/j2ee/home/config/jazn.security.props
-DEMDROOT=/opt/oracle/product/11.1.0/db1/cs5ora11g_csmlcc
-Dsysman.md5password=true -Drepapi.oracle.home=/opt/oracle/product/11.1.0/db1
-Ddisable.checkForUpdate=true
-Doracle.sysman.ccr.ocmSDK.websvc.keystore=/opt/oracle/product/11.1.0/db1/jlib/emocmclnt.ks
-Dice.pilots.html4.ignoreNonGenericFonts=true -Djava.awt.headless=true -jar
/opt/oracle/product/11.1.0/db1/oc4j/j2ee/home/oc4j.jar -config
/opt/oracle/product/11.1.0/db1/oc4j/j2ee/OC4J_DBConsole_cs5ora11g_csmlcc/config/server.xml

22. Set Oracle Environment

# su - oracle
# vim ~/.bash_profile
ORACLE_BASE=/opt/oracle/product
ORACLE_SID=cybersof
ORACLE_HOME=/opt/oracle/product/11.1.0/db1
export ORACLE_BASE ORACLE_SID ORACLE_HOME
 
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib

23. Shutdown and Startup of Oracle

$ . oraenv
<enter your SID>
 
$ sqlplus "/ as sysdba"
SQL> startup
SQL> exit
 
$ lsnrctl
LSNRCTL> start
LSNRCTL> exit
 
$ cd $ORACLE_HOME/bin
./emctl start dbconsole

B. Creating Users

1. Creating admin user:

http://download.oracle.com/docs/cd/B28359_01/server.111/b28301/admqs_administer_users.htm#CHDBDBGI

em > Setup > Administrators > Create

2. Create appdev user:
em > Server > Roles

http://download.oracle.com/docs/cd/B28359_01/server.111/b28301/admqs_administer_users.htm

CREATE USER "APPDEV" PROFILE "DEFAULT" IDENTIFIED BY "*******" DEFAULT TABLESPACE "USERS" TEMPORARY TABLESPACE "TEMP" ACCOUNT UNLOCK
GRANT UNLIMITED TABLESPACE TO "APPDEV";
GRANT "CONNECT" TO "APPDEV";
GRANT "RESOURCE" TO "APPDEV";

Part II. Installing Oracle10g on CentOS5

1. Choose Advanced Installation”

2. If its just network prerequisite.. ignore it. It’s not a big deal.

3. ORACLE_HOME: /opt/oracle/product/10.2.0/db_1

4. ORACLE_SID: orcl, CHARSET: utf8

5. EM Installed…

6. ISQLPLUS

7. Run root.sh

output…

8. Other settings

9. Oracle Environment Variables