{"id":1493,"date":"2010-07-14T14:56:06","date_gmt":"2010-07-14T19:56:06","guid":{"rendered":"http:\/\/jianmingli.com\/wp\/?p=1493"},"modified":"2011-02-16T11:15:50","modified_gmt":"2011-02-16T16:15:50","slug":"data-synchronization-with-oracle-goldengate","status":"publish","type":"post","link":"https:\/\/jianmingli.com\/wp\/?p=1493","title":{"rendered":"Data Synchronization with Oracle GoldenGate"},"content":{"rendered":"

GoldenGate Overview<\/h2><\/span>\n

* GoldenGate is a software platform that enables exchange and manipulation of data at the transaction level<\/strong> among multiple, heterogeneous platforms across the enterprise.
\n* Supports multiple OS platforms (Solaris, Linux, HP, Win) and databases (Oracle, DB2, MySql to name a few). <\/p>\n

Supported Topologies<\/h3><\/span>\n

* Unidirectional: Reporting instance
\n* Bi-directional: Instant failover “Active-Active”
\n* Peer-to-peer: Load balancing, HA
\n* Broadcast: Data distribution
\n* Consolidation: Dataware house\/mart\/store
\n* Cascading: Scalability, DB tiering<\/p>\n

Components<\/h2><\/span>\n

* Extract
\n* Data dump
\n* Replicat
\n* Trails or extract files
\n* Checkpoints
\n* Manager
\n* Collector<\/p>\n

Extract Process<\/h2><\/span>\n

* Runs on source system.
\n* Provides data capture mechanism:
\n– Captures all changes that are made to objects configured for synchronization.
\n– Sends only committed transactions to the trail for propagation to the target system.
\n* Multiple Extract processes can operate on different objects at the same time.
\n* Can be configured for
\n– Initial loads
\n– Change synchronization<\/p>\n

Extraction Types<\/h3><\/span>\n

* Log-based Extraction:
\n– Extract component obtains data from database transaction logs such as Oracle redo logs.
\n* VAM-based Extraction:
\n– VAM stands for Vendor Access Module. It is a communication layer that passes data changes and transaction metadata to the Extract process.
\n– Database vendor provides VAM to extract data changes and pass it to Extract.<\/p>\n

Data Pump<\/h3><\/span>\n

* A data pump is a secondary Extract group within the source GoldenGate configuration.
\n* A data pump reads source system local trail and sends the data across the network to a remote trail on the target.
\n* Benefits:
\n– Adds storage flexibility
\n– Isolate primary Extract process from TCP\/IP activity.
\n* Can be configured for:
\n– Online processing.
\n– Batch processing.
\n– Pass-through mode.
\n– To perform data filtering, mapping, and conversion.<\/p>\n

Replicat Process<\/h2><\/span>\n

* Runs on target system.
\n* Reads extracted data changes that are specified in Replicat configuration
\n* Replicates data changes to target database in the same order as source db.
\n* Can be configured in
\n– Initial loads
\n– Change synchronization<\/p>\n

Replicat Configurations<\/h3><\/span>\n

* Initial loads
\n* Change synchronization<\/p>\n

Trail Files<\/h2><\/span>\n

* A trail file temporarily stores data changes on local disk (local trail) or remote disk (remote trail).
\n* Trail files are generated by Extract process.
\n* Trail files are read by Data Pump and Replicat processes.
\n* Data is read\/write in large blocks to improve I\/O.
\n* Uses universal data format.
\n– File header record (Version 10.0 and after): contains info about trail file itself.
\n– Header area: contains info about transaction environment.
\n– Data area: contains actual data being extracted.<\/p>\n

Trail File Writing Mode<\/h3><\/span>\n

* Append mode (Version 10.0 and after)
\n* Overwrite mode<\/p>\n

Trail File Maintenance<\/h3><\/span>\n

* Default file size: 10 MB
\n* File name format: same two characters followed by a unique, six-digit serial number (000000 – 999999). For example, ‘tr000001’.
\n* Default storage directory: ‘dirdat’.<\/p>\n

Extract Files<\/h2><\/span>\n

* Similar to trail files but without checkpoints.
\n* Used for one-time runs such as initial loads and batch processing.<\/p>\n

Checkpoints<\/h2><\/span>\n

* Store the current read\/write positions of a process to disk for recovery purposes.
\n* Prevent redundant processing.
\n* Provide fault tolerance against system, network, and GoldenGate process crashes.
\n* Read checkpoints are always synchronized with write checkpoints.
\n* Maintained by checkpoint files in the ‘dirchk’ sub-directory.<\/p>\n

Managers<\/h2><\/span>\n

* Monitor\/restart GG processes.
\n* Issue threshold reports.
\n* Maintain trail files and logs.
\n* Allocate data storage space.
\n* Report errors and events.
\n* Receive and route user requests from user interface.<\/p>\n

Collector<\/h2><\/span>\n

* Background process runs in the target system.
\n* Accepts extracted db changes that are sent over the TCP\/IP network and writes them to trail\/extract files.
\n* Dynamic collectors:
\n– Started by Manager.
\n– One Extract per dynamic collector.
\n* Static collectors:
\n– Started manually.
\n– Multiple Extract processes can share one static collector (one to one preferred).<\/p>\n

Extract\/Collector Pair<\/h3><\/span>\n

* Collector process stops when Extract stops.
\n* Normally, Extract initiates Collector processes. But Collector can be configured to initiate Extract.<\/p>\n

Processing Modes<\/h2><\/span>\n

Online process<\/h3><\/span>\n

* Runs continuously until stopped by a user
\n* Maintains recovery checkpoints in the trail
\n– Processing can resume after interruptions
\n* Used to continuously extract and replicate transactional\/DDL changes<\/p>\n

A Batch Run<\/h3><\/span>\n

* Aka, Special Run
\n* Extracts or replicates changes within known begin and end points
\n* No checkpoints maintained (rerun if failures)<\/p>\n

A Task<\/h3><\/span>\n

* A special type of batch run.
\n* Used for certain initial load methods.
\n* Extract communicates directly <\/strong>with Replicat over TCP\/IP.<\/p>\n

Groups<\/h2><\/span>\n

Processing Group<\/h3><\/span>\n

* A process (either Extract or Replicat)
\n* Process parameter file
\n* Checkpoint file
\n* Any other files associated with the process<\/p>\n

Group Names<\/h3><\/span>\n

* Eight characters long following OS file name conventions.
\n* Case *insensitive*
\n* Do *not* end in numbers (can begin with numbers)<\/p>\n

Commit Sequence Number or CSN<\/h2><\/span>\n

* Uniquely identifies a particular point in time in which a transaction commits to the database.
\n* Same as Oracle SCN (System Change Number)<\/p>\n

Configure Manager<\/h2><\/span>\n
\r\nC:\\prog\\gg104>ggsci\r\n\r\nOracle GoldenGate Command Interpreter for Oracle\r\nVersion 10.4.0.19 Build 002\r\nWindows (optimized), Oracle 10 on Sep 18 2009 15:54:55\r\n\r\nCopyright (C) 1995, 2009, Oracle and\/or its affiliates.  All rights reserved.\r\n\r\nGGSCI (USRES-MING-LJ2) 1> edit params mgr\r\n\r\nGGSCI (USRES-MING-LJ2) 6> start manager\r\nManager started.\r\n\r\nGGSCI (USRES-MING-LJ2) 7> stop manager\r\nManager is already stopped.\r\n<\/pre>\n
mgr.prm<\/h3><\/span>\n
\r\nPORT 7809\r\nPURGEOLDEXTRACTS\r\n<\/pre>\n
Global parameters<\/h3><\/span>\n
Stored in GLOBALS file<\/p>\n
\r\nGGSCI (USRES-MING-LJ2) 13> EDIT PARAMS .\/GLOBALS\r\n<\/pre>\n
Runtime Parameters<\/h3><\/span>\n
* Runtime global parameters
\n* Object specific parameters<\/p>\n
Commands<\/h1><\/span>\n
Manager Commands<\/h2><\/span>\n
\r\nGGSCI (USRES-MING-LJ2) 1> info manager\r\nManager is DOWN!\r\n\r\nGGSCI (USRES-MING-LJ2) 2> status manager\r\nManager is DOWN!\r\n\r\nGGSCI (USRES-MING-LJ2) 3> refresh manager\r\nERROR: Manager not currently running.\r\n\r\n<\/pre>\n
References<\/h1><\/span>\n
* gg_wux_admin_v104.pdf
\n* gg_wux_ref_v104.pdf
\n* http:\/\/gavinsoorma.com\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
GoldenGate Overview * GoldenGate is a software platform that enables exchange and manipulation of data at the transaction level among multiple, heterogeneous platforms across the enterprise. * Supports multiple OS platforms (Solaris, Linux, HP, Win) and databases (Oracle, DB2, MySql … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[88],"tags":[],"class_list":["post-1493","post","type-post","status-publish","format-standard","hentry","category-goldengate"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8cRUO-o5","_links":{"self":[{"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=\/wp\/v2\/posts\/1493","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1493"}],"version-history":[{"count":4,"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=\/wp\/v2\/posts\/1493\/revisions"}],"predecessor-version":[{"id":1951,"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=\/wp\/v2\/posts\/1493\/revisions\/1951"}],"wp:attachment":[{"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1493"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1493"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jianmingli.com\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1493"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}