GoldenGate Overview
* GoldenGate is a software platform that enables exchange and manipulation of data at the transaction level among multiple, heterogeneous platforms across the enterprise.
* Supports multiple OS platforms (Solaris, Linux, HP, Win) and databases (Oracle, DB2, MySql to name a few).
Supported Topologies
* Unidirectional: Reporting instance
* Bi-directional: Instant failover “Active-Active”
* Peer-to-peer: Load balancing, HA
* Broadcast: Data distribution
* Consolidation: Dataware house/mart/store
* Cascading: Scalability, DB tiering
Components
* Extract
* Data dump
* Replicat
* Trails or extract files
* Checkpoints
* Manager
* Collector
Extract Process
* Runs on source system.
* Provides data capture mechanism:
– Captures all changes that are made to objects configured for synchronization.
– Sends only committed transactions to the trail for propagation to the target system.
* Multiple Extract processes can operate on different objects at the same time.
* Can be configured for
– Initial loads
– Change synchronization
Extraction Types
* Log-based Extraction:
– Extract component obtains data from database transaction logs such as Oracle redo logs.
* VAM-based Extraction:
– VAM stands for Vendor Access Module. It is a communication layer that passes data changes and transaction metadata to the Extract process.
– Database vendor provides VAM to extract data changes and pass it to Extract.
Data Pump
* A data pump is a secondary Extract group within the source GoldenGate configuration.
* A data pump reads source system local trail and sends the data across the network to a remote trail on the target.
* Benefits:
– Adds storage flexibility
– Isolate primary Extract process from TCP/IP activity.
* Can be configured for:
– Online processing.
– Batch processing.
– Pass-through mode.
– To perform data filtering, mapping, and conversion.
Replicat Process
* Runs on target system.
* Reads extracted data changes that are specified in Replicat configuration
* Replicates data changes to target database in the same order as source db.
* Can be configured in
– Initial loads
– Change synchronization
Replicat Configurations
* Initial loads
* Change synchronization
Trail Files
* A trail file temporarily stores data changes on local disk (local trail) or remote disk (remote trail).
* Trail files are generated by Extract process.
* Trail files are read by Data Pump and Replicat processes.
* Data is read/write in large blocks to improve I/O.
* Uses universal data format.
– File header record (Version 10.0 and after): contains info about trail file itself.
– Header area: contains info about transaction environment.
– Data area: contains actual data being extracted.
Trail File Writing Mode
* Append mode (Version 10.0 and after)
* Overwrite mode
Trail File Maintenance
* Default file size: 10 MB
* File name format: same two characters followed by a unique, six-digit serial number (000000 – 999999). For example, ‘tr000001’.
* Default storage directory: ‘dirdat’.
Extract Files
* Similar to trail files but without checkpoints.
* Used for one-time runs such as initial loads and batch processing.
Checkpoints
* Store the current read/write positions of a process to disk for recovery purposes.
* Prevent redundant processing.
* Provide fault tolerance against system, network, and GoldenGate process crashes.
* Read checkpoints are always synchronized with write checkpoints.
* Maintained by checkpoint files in the ‘dirchk’ sub-directory.
Managers
* Monitor/restart GG processes.
* Issue threshold reports.
* Maintain trail files and logs.
* Allocate data storage space.
* Report errors and events.
* Receive and route user requests from user interface.
Collector
* Background process runs in the target system.
* Accepts extracted db changes that are sent over the TCP/IP network and writes them to trail/extract files.
* Dynamic collectors:
– Started by Manager.
– One Extract per dynamic collector.
* Static collectors:
– Started manually.
– Multiple Extract processes can share one static collector (one to one preferred).
Extract/Collector Pair
* Collector process stops when Extract stops.
* Normally, Extract initiates Collector processes. But Collector can be configured to initiate Extract.
Processing Modes
Online process
* Runs continuously until stopped by a user
* Maintains recovery checkpoints in the trail
– Processing can resume after interruptions
* Used to continuously extract and replicate transactional/DDL changes
A Batch Run
* Aka, Special Run
* Extracts or replicates changes within known begin and end points
* No checkpoints maintained (rerun if failures)
A Task
* A special type of batch run.
* Used for certain initial load methods.
* Extract communicates directly with Replicat over TCP/IP.
Groups
Processing Group
* A process (either Extract or Replicat)
* Process parameter file
* Checkpoint file
* Any other files associated with the process
Group Names
* Eight characters long following OS file name conventions.
* Case *insensitive*
* Do *not* end in numbers (can begin with numbers)
Commit Sequence Number or CSN
* Uniquely identifies a particular point in time in which a transaction commits to the database.
* Same as Oracle SCN (System Change Number)
Configure Manager
C:\prog\gg104>ggsci Oracle GoldenGate Command Interpreter for Oracle Version 10.4.0.19 Build 002 Windows (optimized), Oracle 10 on Sep 18 2009 15:54:55 Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved. GGSCI (USRES-MING-LJ2) 1> edit params mgr GGSCI (USRES-MING-LJ2) 6> start manager Manager started. GGSCI (USRES-MING-LJ2) 7> stop manager Manager is already stopped.
mgr.prm
PORT 7809 PURGEOLDEXTRACTS
Global parameters
Stored in GLOBALS file
GGSCI (USRES-MING-LJ2) 13> EDIT PARAMS ./GLOBALS
Runtime Parameters
* Runtime global parameters
* Object specific parameters
Commands
Manager Commands
GGSCI (USRES-MING-LJ2) 1> info manager Manager is DOWN! GGSCI (USRES-MING-LJ2) 2> status manager Manager is DOWN! GGSCI (USRES-MING-LJ2) 3> refresh manager ERROR: Manager not currently running.
References
* gg_wux_admin_v104.pdf
* gg_wux_ref_v104.pdf
* http://gavinsoorma.com/