Oracle GoldenGate

Technical Deep Dive

Y V Ravi Kumar
Oracle ACE
Oracle Certified Master (OCM)
Oracle ACE Spotlight – June 2016

New York Oracle User

Group (NYOUG)
7th Dec 2016

Patterns 1
 Y V RAVI KUMAR
 Oracle Certified Master (OCM) – May 2009
 Oracle ACE – May 2015
 Oracle ACE Spotlight for the month - Jun 2016

ABOUT ME
 “Community Expert” in DELL’s Toad World
 “Expert” in Oracle Technology Network (OTN) community

Oracle Speaker @ ORACLE CERTIFICATIONS

Oracle Database 10g: Certified Master (10g OCM)
 Oracle Technology Network (OTN)
Oracle Database 10g & 11g: Administering RAC Certified Expert
 New York Oracle User Group (NYOUG)
Oracle Database 11g: Performance Tuning Certified Expert
 Independent Oracle User Group (IOUG) Oracle Exadata 11g Essentials
 Sangam (Largest Oracle Event in India) Oracle Golden Gate 10 Essentials
 All India Oracle User Group (AIOUG) Oracle Database 11g: SQL Tuning Certified Expert
Oracle 9i & 10g: Managing Oracle on Linux Certified Expert
Author of 70+ articles Oracle Certified Professional (OCP) – Oracle 12c, 11g, 10g, 9i and 8i
 Oracle Technology Network (OTN) SUN Certified – Solaris System Administrator in SUN Solaris 9
 Toad World - Connected-Driven Innovation
 OTech Magazine
 All things ORACLE from Redgate
 UKOUG Library

CO-FOUNDER OF

Patterns 2
ORACLE GOLDENGATE 11g/12c
Introduction

Patterns 3
Quick History In Replication

Oracle Advance
Change Data Capture (CDC) Oracle Streams Oracle GoldenGate
Replication

• Synchronous
• Asynchronous

• Multi-Master Replication
• MV Replication

• Capture Messages
• Staging Messages
• Consumption
• Heterogeneous
• Multiple Architectures
• Multiple Use Cases

Patterns 4
Database Replication Options

Data Guard
Active Data Guard

InfoSphere Change
Data Capture for
Oracle Replication

Patterns 5
INTRODUCTION

Oracle Golden Gate provides low-impact capture, routing, transformation, and

delivery of database transactions across heterogeneous environments in near-
real time.

Oracle Golden Gate enables the exchange and manipulation of data at the
transaction level among multiple, heterogeneous platforms across the
enterprise.

Oracle Golden Gate moves committed transactions from redo logs and maintains
transaction integrity with sub-second latency

Patterns 6
Oracle GoldenGate – Supported Platforms

Choice can be made from 2 types of Oracle GoldenGate (OGG):

Supported Databases Supported Operating Systems

AIX, z/OS, iSeries, z/Linux

Always check the latest Certification Matrix

Patterns 7
ORACLE GOLDENGATE

Topologies

Patterns 8
ORACLE GOLDENGATE TOPOLOGIES

BIDIRECTIONAL STANDBY DB PEER-TO-PEER LOAD BALANCING, MULTI

UNIDIRECTIONAL QUERY
OR ACTIVE-ACTIVE FOR OFFLOADING
HIGH-AVAILABILITY MASTER

Patterns 9
ORACLE GOLDEN GATE TOPOLOGIES

BROADCAST DATA DISTRIBUTION

CASCADING DATAMARTS

INTEGRATION / CONSOLIDATION DATA

WAREHOUSE

Patterns 10
ORACLE GOLDEN GATE TOPOLOGIES
BIDIRECTIONAL STANDBY
UNIDIRECTIONAL QUERY DB OR ACTIVE-ACTIVE FOR PEER-TO-PEER LOAD
OFFLOADING HIGH-AVAILABILITY BALANCING,
MULTIMASTER

INTEGRATION / CONSOLIDATION
BROADCAST DATA
DATA WAREHOUSE CASCADING MARTS
DISTRIBUTION

Patterns 11
ORACLE GOLDENGATE 12c

STANDBY
(OPEN & HIGH AVAILABILITY
ACTIVE)

OLTP SYSTEMS NEW DB / OS ZERO DOWN-TIME UPGRADES &

/HW MIGRATIONS

REPORTING
ORACLE DATABASE
LIVE REPORTING
GOLDEN
GATE
ODS . ETL EDW .
OPERATIONAL
BUSINESS
INTELLIGENCE
ETL
EDW .

ETL
ETL TRANSACTIONAL
DATA INTEGRATION
MESSAGE BUS ETL

Patterns 12
Benefits Of Oracle GoldenGate
 High Availability (Standby Database)
 Load balancing
 Application Specific HA

 Zero Down-Time Upgrades and Migrations

 Consolidations
 Database Migrations / Upgrades
 Platform Migrations (e.g. HP-UX to Linux)
 Database Upgrades
 Homogeneous & Heterogeneous
 Application Upgrades (e.g. Siebel 7 -> Siebel 8)
 Application Releases (special use cases)

 Live Reporting (Reporting Database)

 Off load reports to a separate system
 Operational Business Intelligence
 Transactional Data Integration

Patterns 13
Benefits Of Oracle GoldenGate

 Oracle GG is a middleware product designed to work in a heterogeneous environment

with different databases.
 Oracle GG moves only committed data across platforms where as Oracle database,
which writes committed and uncommitted changes to the redo logs.
 Moves across a TCP/IP network and does not require Oracle Net.
 Oracle GG will not provide automatic failover like Oracle DG.
 Oracle GG uses its own Commit Sequence Number (CSN) to identify a transaction which
based on the Oracle Database SCN (System Change Number).
 Complete data recoverability via trail files.

Patterns 14
Oracle Golden Gate Solutions for Oracle Database

Migrate from non-Oracle databases to Oracle 12c

Upgrade Oracle Database versions 8i, 9i, 10g or 11g to 12c

Upgrade or migrate the database server or operating system

Perform database maintenance

Patterns 15
Oracle GoldenGate Solutions for Oracle Database

 Eliminate Down-Time During Oracle Database Upgrades

 Eliminate Unplanned Down-Time With Active Data Guard
 Improve Production System Performance
 Real-Time Reporting from a Lower-Cost System
 Increase ROI On Existing Servers and Synchronize Global Data
 Capture can be offloaded from the source DB to an intermediate host by copying
the redo logs

Patterns 16
ORACLE GOLDENGATE
Components and Architecture

Patterns 17
ORACLE GOLDEN GATE LOGICAL ARCHITECTURE

INITIAL LOAD
DATA SOURCE EXTRACT REPLICAT
FOR INITIAL
LOAD: SOURCE
TABLES

NETWORK
MANAGER
MANAGER

DATA SOURCE
CHANGE EXTRACT COLLECTOR
SYNCHRONIZA REPLICAT
-TION:
TRANSACTION
LOG TRAIL OR FILE
(OPTIONAL)

DATA CHANGE SYNCHRONIZATION

PUMP

Patterns 18
Oracle GoldenGate 12c – Source and Target
SERVER: Golden Gate 1 (Source)

DATABASE: ORACLE GoldenGate

Extract Trail Files
AP HR
GoldenGate Writes
SCHEMAS

Writes Online Redo HR schema File grows until

AR WH extract process
database Reads changes records are
LGWR (HREXT)
to processed
SYS Process Changes to ---
Log Files redo
only reads redo

Reads
redo log

Files
Trail
log
files files
for HR schema
Writes to
GoldenGate collectors
data pump
process

SERVER: Golden Gate 2 (Target)

GoldenGate
SCHEMAS

SYS GoldenGate
Data Pump
Changes written to
GoldenGate Collector
writes
HR Replicat (RPHR01) process
HR Schema process
Trail files
Reads Trail grow until Writes to
Files data Trail Files
DATABASE: ORACLE processed

Patterns 19
Oracle GoldenGate 12c – Source and Target
SERVER: GoldenGate 1 (Source)
Extract
DATABASE: ORACLE GoldenGate
Trail Files
AP HR
GoldenGate Writes
Writes
SCHEMAS

Online Redo Reads HR schema File grows until

AR WH database extract process
changes records are
LGWR (HREXT)
to processed
SYS Process Changes to redo ---
redo log Log Files log only reads redo Reads
files files for HR schema Trail
Files
Writes to
GoldenGate collectors
data pump
process

SERVER: GoldenGate 2 (Target)

GoldenGate
Data Pump
SCHEMAS

SYS GoldenGate
GoldenGate writes
Changes written to Collector
HR Replicat (RPHR01) process
HR Schema process
Trail files
Reads Trail grow Writes to
Files until Trail Files
DATABASE: ORACLE data
processed

Patterns 20
Oracle GoldenGate 12c – Source and Target
SERVER: Golden Gate 1 (Source)

DATABASE: ORACLE GoldenGate

Extract Trail Files
AP HR
GoldenGate Writes
SCHEMAS

Writes Online Redo HR schema File grows until

AR WH extract process
database Reads changes records are
LGWR (HREXT)
to processed
SYS Process Changes to ---
Log Files redo
only reads redo

Reads
redo log

Files
Trail
log
files files
for HR schema
Writes to
GoldenGate collectors
data pump
process

SERVER: Golden Gate 2 (Target)

GoldenGate
SCHEMAS

SYS GoldenGate
Data Pump
Changes written to
GoldenGate Collector
writes
HR Replicat (RPHR01) process
HR Schema process
Trail files
Reads Trail grow until Writes to
Files data Trail Files
DATABASE: ORACLE processed

Patterns 21
Checkpoints

Patterns 22
CHECKPOINTS - CAPTURE
Start of Oldest
BEGIN : TX1
Open (Uncommited) _________
Transaction _________ CAPTURE
INSERT : TX1 _________
_________
_________
CHECK
BEGIN : TX2
______
POINT

CURRENT WRITE POSITION

CURRENT READ POSITION
UPDATE : TX1

INSERT : TX2

COMMIT : TX2

BEGIN : TX3

INSERT : TX3

BEGIN : TX4

COMMIT : TX3

DELETE : TX4

SOURCE DATABASE CAPTURE COMMIT ORDERED SOURCE TRAIL

CAPTURE CHECKPOINT

Patterns 23
CHECKPOINTS - CAPTURE - PUMP

_________
_________ PUMP BEGIN : TX2
_________
_________
_________
CHECK INSERT : TX2
______
POINT
COMMIT : TX2

CURRENT WRITE POSITION

CURRENT READ POSITION
BEGIN : TX3

INSERT : TX3

COMMIT : TX3

PUMP COMMIT ORDERED TARGET TRAIL

PUMP CHECKPOINT

Patterns 24
CHECKPOINTS - CAPTURE - PUMP - DELIVERY

_________
_________
_________
PUMP
_________
_________
CHECK
______
POINT

CURRENT READ POSITION

DELIVERY
TARGET DATABASE

DELIVERY CHECKPOINT

Patterns 25
CHECKPOINTS - CAPTURE - PUMP - DELIVERY

Patterns 26
ELIMINATE DOWN-TIME DURING ORACLE DATABASE UPGRADES

ZERO DOWN-TIME DATABASE UPGRADES

APPLICATION
SWITCH OVER

REAL-TIME UPDATES

CAPTURE DELIVERY
ORACLE ORACLE
8i / 9i / 10g 11g
Route (LAN/WAN/Web/IP)

CAPTURE
DELIVERY POST-SWITCHOVER
DATA FLOW

COMPARE & VERIFY

Patterns 27
ELIMINATE UNPLANNED DOWN-TIME WITH ACTIVE DATA GUARD

DISASTER RECOVERY & DATA PROTECTION

APPLICATION
SWITCH OVER

REAL-TIME UPDATES

CAPTURE DELIVERY

Route (LAN/WAN/Web/IP) STANDBY SERVER

POST-SWITCHOVER
COULD BE USED FOR
DATA FLOW CAPTURE REPORTING,
DELIVERY
QUERING, TESTING….
STANDBY
SOURCE

Patterns 28
IMPROVE PRODUCTION SYSTEM PERFORMANCE & LOWER COSTS

QUERY OFFLOADING

TRANSACTION
READ-ONLY
PROCESSING
ACTIVITY
ACTIVITY
APPLICATION

REAL-TIME UPDATES

CAPTURE DELIVERY

Route (LAN/WAN/Web/IP)

ORACLE REPLICA
LEGACY PRODUCTION
OLTP

Patterns 29
ORACLE GOLDENGATE FOR OPERATIONAL REPORTING

OPERATIONAL REPORTING

OLTP REPORTING
APPLICATION APPLICATION
Sddsgsdggdsgfsfsf Sddsgsdggdsgfsfsf
ddsjg ddsjg
dggdgjkddgdkjddd dggdgjkddgdkjddd
n dnjnd dsgnjn
sdgnjngj
ngjndjdnsg
jdjnjdsg njn dnj
REAL-TIME UPDATES n dnjnd dsgnjn
sdgnjngj
ngjndjdnsg
jdjnjdsg njn dnj
djgn jnjgnjdg d djgn jnjgnjdg d
Dgjdjgjk d jsdgbjj Dgjdjgjk d jsdgbjj
sdj jbdsg jbsdg sdj jbdsg jbsdg
jbdsg sbdgbsdjb jbdsg sbdgbsdjb
bsdg jsdbgsdgb bsdg jsdbgsdgb
bkbgdg bkbgdg

DELIVERY
kjbdgjkbgsjk

CAPTURE
kjbdgjkbgsjk
dbgjbg jksg jbjdjdj dbgjbg jksg jbjdjdj

TRAIL FILE
CAPTURE Route (LAN/WAN/Web/IP) TRAIL FILE
DELIVERY

Route (LAN/WAN/Web/IP)

REPORTING
PRODUCTION INSTANCE
DATABASE

Patterns 30
INCREASE ROI ON EXISTING SERVERS & SYNCHRONIZE GLOBAL DATA

ACTIVE - ACTIVE

APPLICATION APPLICATION

REAL-TIME UPDATES

DELIVERY
CAPTURE

DELIVERY CAPTURE
Route (LAN/WAN/Web/IP)
SOURCE & TARGET DB SOURCE & TARGET DB

Patterns 31
ORACLE GOLDENGATE – THE WAY IT WORKS

Extract
Committed transactions are captured (and can be filtered) as they occur by reading the transaction logs.

Trail Stages and Queues Data for Routing

Pump Distributes Data for Routing to Target(s)

Route Data is Compressed, Encrypted for Routing to Target(s)

Replicat applies data with transaction integrity
EXTRACT REPLICAT
CAPTURE
PUMP

TRAIL FILES TRAIL FILES

REPLICAT EXTRACT
DELIVERY
SOURCE • LAN PUMP TARGET
ORACLE & • WAN ORACLE &
• INTERNET
NON-ORACLE • OVER TCP / IP NON-ORACLE
DATABASE DATABASE
← BI-DIRECTIONAL →
Patterns 32
OPTIMIZED FOR ORACLE 12c
SOURCE ORACLE 12C MULTITENANT CONTAINER DATABASE TARGET ORACLE 12C MULTITENANT CONTAINER DATABASE

DELIVERY DELIVERY DELIVERY

CAPTURE

PUMP

TRAIL FILES
PUMP
TRAIL FILES
PUMP TRAIL FILES
TRAIL FILES
Patterns 33
UPDATE DISASTER RECOVERY SITE – IN ONE OPERATION

PRODUCTION DISASTER RECOVERY

SERVER SERVER

Patterns 34
MANAGE MANY DATABASES AS ONE
ONE STANDBY DATABASE COVERS ALL PLUGABLE DATABASES

12.1 12.1 12.1 12.1

ERP CRM DW HCM

PRODUCTION CONTAINER DATABASE

12.1 12.1 12.1 12.1

ERP CRM DW HCM

STANDBY CONTAINER DATABASE

Patterns 35
ORACLE GOLDENGATE
Performance Tuning

Patterns 36
Extract Flavours
Integrated Extract
 Is an Oracle GoldenGate Extract for Oracle databases (Database Release 11.2.0.3 and later)
 Is multithreaded & Supports more data types
 Relies on Oracle’s internal log parsing and processing implementation
 Supports downstream topologies
 Is new with version 11.2.1.0.0
 Works with Logminer
Register Extract (capture) with database/logminer required
Example: GGSCI> register extract [ name ] database container [ (PDB) ]
Classic Extract
 Is traditional REDO log-based extract for Oracle
 Works for all DB platforms and versions
Patterns 37
DDL Setup Scripts – Source and Target

Classic Extract
 Oracle Database 11.2.0.3.0 Or earlier, or are running classic mode then the scripts need to be run from SQL *Plus as sysdba
 Scripts to be executed
 @marker_setup.sql
 @ddl_setup.sql
 @role_setup.sql
 Grant role to user
 @ddl_enable.sql
 DDL Trigger must be enabled

Integrated Extract
 No need to run the scripts
 Requires Oracle Database 11.2.0.4.0 or later
 DDL Trigger must be disabled for Integrated mode DDL replication

Patterns 38
Identify Extract – Classic OR Integrated

GGSCI (ggnode1.oracle.com) 1> stop extract eorcl

GGSCI (ggnode1.oracle.com) 2> register extract eorcl database
GGSCI (ggnode1.oracle.com) 3> info eorcl upgrade

ERROR: Extract EORCL is not ready to be upgraded because recovery

SCN 1755571 has not reached SCN 1755605.

GGSCI (ggnode1.oracle.com) 4> alter extract eorcl tranlog begin now

GGSCI (ggnode1.oracle.com) 5> start extract eorcl
GGSCI (ggnode1.oracle.com) 6> info extract eorcl
GGSCI (ggnode1.oracle.com) 7> stop extract eorcl
GGSCI (ggnode1.oracle.com) 8> info eorcl upgrade
GGSCI (ggnode1.oracle.com) 9> alter extract eorcl, UPGRADE
INTEGRATED TRANLOG
GGSCI (ggnode1.oracle.com) 10> start extract eorcl
GGSCI (ggnode1.oracle.com) 11> info extract eorcl

Patterns 39
Oracle GoldenGate Performance Areas

Source Database Target Database

Trail Trail
Capture Pump
Files Files Delivery

GoldenGate
GG tools: LAG, REPORTCOUNT GoldenGate

DATABASE - AWR, ASH, UTL_SPADV, TRACE Database

HOST - MPSTAT, VMSTAT, IOSTAT, STRACE, TOP HOST

Patterns 40
Install the UTL_SPADV Package (Integrated Extract and Integrated Replicat)
 UTL_SPADV PL/SQL package provides subprograms to collect and analyze statistics for the LogMiner server processes.
The statistics help identify any current areas of contention such as CPU or I/O.

 Oracle Streams Performance Advisor (SPADV) enables monitoring of the integrated GoldenGate server
processes which are used by integrated Extract and integrated Replicat, and provides information about how these
processes are performing.

 SPADV statistics are collected and analyzed using the UTL_SPADV package.

 UTL_SPADV package, as the Oracle GoldenGate administrator user on the source database.

To install SPADV, execute the following steps:

a. Grant the following privileges to a designated Oracle GoldenGate administrator database user:
SQL> exec DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE( -
'<db user name>');

b. Connect to the database with the user name that was granted permissions in Step a.

c. Run the utlspadv.sql script. For example:

SQL> @$ORACLE_HOME/rdbms/admin/utlspadv.sql

Patterns 41
Gather Statistics using the UTL_SPADV Package
 Oracle recommends that you gather statistics for a 30-60 minute time period during which you are troubleshooting performance.
 It is also recommended to gather statistics during a 30-60 minute time period where performance is good, serving as a baseline comparison.

To gather statistics every 15 seconds, run the following SQL*Plus command as the Oracle GoldenGate
administrator:
SQL> exec UTL_SPADV.START_MONITORING(interval=>15);

To stop statistics gathering, run the following command:

SQL> exec UTL_SPADV.STOP_MONITORING;

Run the following commands to determine if the monitoring job is currently running:

SET SERVEROUTPUT ON
DECLARE
is_mon BOOLEAN;
BEGIN
is_mon := UTL_SPADV.IS_MONITORING(
job_name => 'STREAMS$_MONITORING_JOB',
client_name => NULL);
IF is_mon=TRUE THEN
DBMS_OUTPUT.PUT_LINE('The monitoring job is running.');
ELSE
DBMS_OUTPUT.PUT_LINE('No monitoring job was found.');
END IF;
END;
/

Patterns 42
Generating Report - UTL_SPADV Package
 It is also possible to create a static report of SPADV statistics after monitoring for a period of time. The report can be
generated in text form much like the display of real-time statistics.

 To generate a text report, from SQL*Plus as the Oracle GoldenGate administrator, execute the following:

spool /tmp/spadv.txt

begin
utl_spadv.show_stats(path_stat_table=>'STREAMS$_PA_SHOW_PATH_STAT',
bgn_run_id=> 1,
end_run_id=> 9999,
show_legend=> TRUE);
end;

After the reports have been generated, Oracle recommends purging the SPADV statistics using the
following
command:

SQL> exec UTL_SPADV.STOP_MONITORING(PURGE=>TRUE);

Patterns 43
Identifying the Bottlenecks

LAN /
EXTRACT WAN REPLICAT
INTERNET

SOURCE SOURCE TRAIL PUMP TARGET TRAIL REPLICAT TARGET

DATABASE DATABASE

MANAGER MANAGER ▲
If the (latency) in
REPLICAT is acceptable
then All is Well
●●●●● ●●●●● ●●●●●

Throughput
If the lag in Statistics are also
If the lag useful in identifying potential bottlenecks
If the lag in
← Monitoring
EXTRACT, Start Lag – Working
in PUMP,from the Target back to the REPLICAT,
Source check
Here! check for for Lag in PUMP
Lag in
EXTRACT
Patterns 44
Q&A

yenugulavenkata.ravikumar

[email protected]

@yvrk1973

http://yvrk1973.blogspot.in

http://in.linkedin.com/pub/yv-ravikumar-oracle-certified-master-ocm/14/13/a50
Patterns 45