Bookmark and Share

Friday, December 18, 2009

Oracle 11gR2, GoldenGate and Koala Jump Start

It was some time ago now, but Oracle GoldenGate statement of direction made me sad. Sometimes, I still wish the person who wrote it is somehow as far away as possible from the database server development team. But I know that's not the case. The sentence that is killing me for sure is:
"Given the strategic nature of Oracle GoldenGate, Oracle Streams will continue to be supported, but will not be actively enhanced. Rather, the best elements of Oracle Streams will be evaluated for inclusion with Oracle GoldenGate."

Some Background

Even today, a few weeks after I've read GoldenGate documentation, tested the product and I've listed GoldenGate benefits for myself (openness, simplicity, completeness, ...). I'm still getting some trouble swallowing I will miss some of the unique features of Streams.

Not that I'm intolerant or I'm stuck to the "good old time". I mean, I know there are bad and good things about Streams. During the past years, I've spent quite some energy(*) on leveraging what is so unique in it and I believe most of its advantages come from its "built-inside" architecture. To a point where it could be very difficult for Oracle to provide Streams strengths to GoldenGate because their architectures are so different.

You may wonder what I am talking about. Well, I'm talking about:
  • Streams in-memory end-to-end propagation that provides an extremely scalable framework without the need to log changes in another separate file (trail)
  • The reliability of the staging (the database) and propagation (buffered queues).
  • Streams lighweight footprint that relies (most of the time) on internal object ids, can easily be offloaded to a separate server without any database copy and doesn't access original data at all
  • The ability Streams has to treat set of changes without "deferring" the constraints
  • The flexibility of the framework that is running inside the database with hooks at every stage you can completely personalize and extend to your needs
  • The openness for developers that can directly use the APIs to provide advanced features to their applications by leveraging their Oracle knowledge
  • The omnipotence and ubiquity of the solution that is already installed with Oracle 11g, work with RAC no change, is extended by the Change Data Capture feature and easy to share with the outside with AQ or XStream.
  • The maturity, ease and simplicity it has gained over the years of hard work by the development team and you can leverage by a simple SQL script
  • The cost model that comes for free with Oracle Database Enterprise Edition and with a limited set of features with 11g Standard Edition
At some point, I was hoping XStream would be the missing link between Streams and GoldenGate so that GoldenGate could leverage the power of Oracle Database 11g. That would for sure have been my preferred scenario and the fact XStream is licenced with GoldenGate made me hope for a while. But the reading of the statement of direction suggests a very different end.

Now Oracle GoldenGate is very likely to, if not succeed, do a better job than Streams to bring real-time data integration to the mass ;-). Nevertheless, I have no clue how Oracle could, in a timely manner, provide Streams strengths to it. Interesting challenge for the product managers and development team... The future will tell us!

Koala jump start

As you can guess, time has come for all of us to learn more about Oracle GoldenGate. Since my laptop runs Karmic Koala, I've given it a try on Ubuntu 9.10. It's not supported but did not face any issue so far. I've downloaded and installed GoldenGate from Oracle E-Delivery and I've choosen the following distribution:
  • Select a product Pack: "Oracle Fusion Middleware"
  • Platform: "Linux x86"
  • Description: "Oracle GoldenGate on Oracle Media Pack for Linux x86"
  • Name: "Oracle GoldenGate V10.4.0.x for Oracle 11g on RedHat 5.0"
To perform the installation, I've just unzip/untar the file in a directory and I've set the environment variable so that I can access my Oracle Database:
. oraenv
ORACLE_SID = [WHITE] ? BLACK
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle
That done, you should be ready to use Oracle GoldenGate; run the command line interface like below:
./ggsci 

Oracle GoldenGate Command Interpreter for Oracle
Version 10.4.0.19 Build 002
Linux, x86, 32bit (optimized), Oracle 11 on Sep 29 2009 08:50:50

Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.


exit

Prepare the database

There are a few settings to check on the database. It must be in archivelog mode and GoldenGate must be able to access the archivelogs and redologs:
sqlplus / as sysdba

archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /u01/app/oracle/oradata/BLACK/archivelogs
Oldest online log sequence 1
Next log sequence to archive 3
Current log sequence 3
I've set the NLS_LANG so that it matches the one from my database:
sqlplus / as sysdba

select parameter, value
from nls_database_parameters
where parameter in ('NLS_LANGUAGE',
'NLS_TERRITORY',
'NLS_CHARACTERSET',
'NLS_LENGTH_SEMANTICS');

parameter VALUE
-------------------- -------------
NLS_LANGUAGE AMERICAN
NLS_TERRITORY AMERICA
NLS_CHARACTERSET WE8MSWIN1252
NLS_LENGTH_SEMANTICS BYTE


exit

export NLS_LANG=AMERICAN_AMERICA.WE8MSWIN1252
The configuration has to rely on a user with a high level of privileges to perform several operations, like extracting the content of an UDT or a LOBs with flashback queries. It should also be able to set supplemental log groups to the tables that are part of the replication:
sqlplus / as sysdba

create user gg
identified by gg
default tablespace users
temporary tablespace temp;

grant create session, resource, dba to gg;
The database must also have the minimal supplemental logging enabled:
alter database add supplemental log data;

alter system switch logfile;
In my case, that was it. Obviously, based on what you want to do, your database may require more changes. It could require some tables to audit the DDL or to store checkpoints to be created. To get the complete list of what needs to be done, check GoldenGate's documentation and more specifically the "Oracle Installation and Setup Guide".

A Demo Schema

I did not try to setup anything advanced. I've just replicated scott.dept in a demo schema. To make the example even more simple, I've considered nobody was accessing the table and I did not pay any attention to pending transactions or out-of-sync instantiation. I've just created a table named demo.dept and loaded it with scott.dept's data:
create user demo identified by demo
default tablespace users
temporary tablespace temp
quota unlimited on users;

grant connect, resource to demo;

create table demo.dept
( deptno number(2,0),
dname varchar2(14),
loc varchar2(13),
constraint pk_dept primary key(deptno))
tablespace users;

insert into demo.dept
select * from scott.dept;

commit;

Configure GoldenGate Process Manager

Once the database configured, refer to the "Administration Guide" to continue and get more details about GoldenGate setup. First, it requires some directories to store its configuration, logs and trail files. You can choose an alternate location for them but that wasn't really my concern either:
./ggsci 

create subdirs

Creating subdirectories under current directory /gg

Parameter files /gg/dirprm: created
Report files /gg/dirrpt: created
Checkpoint files /gg/dirchk: created
Process status files /gg/dirpcs: created
SQL script files /gg/dirsql: created
Database definitions files /gg/dirdef: created
Extract data files /gg/dirdat: created
Temporary files /gg/dirtmp: created
Veridata files /gg/dirver: created
Veridata Lock files /gg/dirver/lock: created
Veridata Out-Of-Sync files /gg/dirver/oos: created
Veridata Out-Of-Sync XML files /gg/dirver/oosxml: created
Veridata Parameter files /gg/dirver/params: created
Veridata Report files /gg/dirver/report: created
Veridata Status files /gg/dirver/status: created
Veridata Trace files /gg/dirver/trace: created
Stdout files /gg/dirout: created
Once done, I've edited the Manager configuration file named MGR to set the port parameter and I've started it:
edit params mgr

view params mgr
port 7809

start manager

status manager
Manager is running (IP port arkzoyd.7809).

Source Table Supplemental Log Group

Like Streams (there is no secret!), GoldenGate needs to be able to identify rows to apply captured changes. It provides some generic tools to enable and check additional logging to the tables from ggsci:
dblogin userid gg, password gg
Successfully logged into database.

add trandata scott.dept
Logging of supplemental redo data enabled for table SCOTT.DEPT.

info trandata scott.dept
Logging of supplemental redo log data is enabled for table SCOTT.DEPT

Parameter Files

I've named the extract scott. It captures changes made to the SCOTT.DEPT table and send them to the remote trail file that, in my case, is managed by the same manager. I've named the replicat demo. The parameter files for scott and demo looks like the ones below:
edit params scott

view params scott

extract scott
userid gg, password gg
rmthost localhost mgrport 7809
rmttrail SC
table SCOTT.DEPT;


edit params demo

view params demo

replicat demo
assumetargetdefs
userid gg, password gg
map SCOTT.DEPT, target DEMO.DEPT;
Note:
With Oracle, you have to use double-quote to manage case-sensitive table names. However, that's not the case with all the database engines. As a result, depending on the parameter, GoldenGate may or may not differentiate strings with different cases. To avoid any issue, I use uppercase for the parameter values, unless I want to specifically use a different case.

Extract and Replicat

Once the parameter files defined, I've added the extract, the replicat and the trail files from ggsci:
add extract scott, tranlog, begin now
EXTRACT added.

add rmttrail SC, extract scott
RMTTRAIL added.

add replicat demo, exttrail SC, nodbcheckpoint, begin now
REPLICAT added.
And I've started them both:
start er *
Sending START request to MANAGER ...
EXTRACT SCOTT starting

Sending START request to MANAGER ...
REPLICAT DEMO starting


info all

Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING SCOTT 00:00:00 00:00:02
REPLICAT RUNNING DEMO 00:00:00 00:00:08


exit

Are Changes Replicated?

Once the setup completed, I've tested the replication with the script below:
sqlplus / as sysdba

update scott.dept
set dname='OPERATIONS2'
where deptno=40;

commit;

select dname from demo.dept
where deptno=40;

DNAME
--------------
OPERATIONS2


update scott.dept
set dname='OPERATIONS'
where deptno=40;

commit;

select dname from demo.dept
where deptno=40;

DNAME
--------------
OPERATIONS


exit

Configuration Cleanup

Obviously that's just a start. To avoid any issue with my next tests, I've cleaned up my configuration, once happy with it:
./ggsci

stop er *
Sending STOP request to EXTRACT SCOTT ...
Request processed.

Sending STOP request to REPLICAT DEMO ...
Request processed.


delete er *
Are you sure you want to delete all groups? y
Deleted EXTRACT SCOTT.
Deleted REPLICAT DEMO.


stop manager
Manager process is required by other GGS processes.
Are you sure you want to stop it (y/n)? y

Sending STOP request to MANAGER ...
Request processed.
Manager stopped.


exit
And I've dropped the demo and gg users:
sqlplus / as sysdba

drop user gg cascade;
drop user demo cascade;

exit
(*) I know at least one person that considers I'm a useless clown and the time I spend on Streams should be treated less mercifully. I admit it easily. I'm aware I'm not close to one percent of his knowledge about what stays an awesome technology. Anyway, good or bad, I've spent some energy on digging into Streams internals and I've always tried to listen to positive and negative feedback.

9 comments:

  1. Hi,

    Thanks for posting this. I have been wanting to have a look at Golden Gate for a little while and your post provided enough of a quick start guide to get me moving. However, I did hit one issue following your example.

    It appears to me that:

    replicat demo
    assumetargetdefs
    userid gg, password gg
    map SCOTT.EMP, table DEMO.DEPT;

    Should read:

    replicat demo
    assumetargetdefs
    userid gg, password gg
    map SCOTT.DEPT, target DEMO.DEPT;

    So, SCOTT.DEPT rather than SCOTT.EMP and "target" rather than "table".

    Sorry if I've missed something obvious, but I was only able to get things working if I made the changes above.

    Thanks again for the post.

    Martin

    ReplyDelete
  2. You're right. Thank you. I'll correct it

    ReplyDelete
  3. Martin,

    Like I said, you are totally right. I've changed the settings in the main post. Thank you

    Grégory

    ReplyDelete
  4. Hi i am facing issue at dblogin itself,please assit me as i am new to golden gate

    ReplyDelete
  5. Ravi,

    If you want to get some real help give some details about what your problem is; do you get an error message? At what step? What is your configuration? What do you already know? etc

    In addition, this blog is, I'm afraid to say, not the right tool to get the best help. I have 2 readers including my mum. I may be able to help you but you'll get way more chances with forums.oracle.com or support.oracle.com that have tens of thousand of people involved.

    Best Regards,

    ReplyDelete
  6. Hey man
    Congratulations!
    The Best Paper!

    Willian Frasson
    wfrasson@gmail.com

    ReplyDelete
  7. I have a streams vs. Golden Gate smackdown here:

    http://cglendenningoracle.blogspot.com/2010/02/streams-vs-golden-gate.html

    - Craig

    ReplyDelete
  8. We just completed a total evaluation of Golden Gate and Streams. At $17K per processor, it was a no brainer. The Golden Gate sales folks were *always* pushing "heterogeneity", even though we told them numerous times we just wanted active/active between two Enterprise Edition Oracle databases. We then tested Streams, and it has met every need we have. Until they push Streams off a cliff, it is what we will use.

    ReplyDelete
  9. I prepared a GoldenGate online test. Who wants to give it a shot?
    http://vgrigorian.com/gg/gg_quiz/gg_quiz.htm

    ReplyDelete