Bi Directional Replication Tasks

Summary

This session explores bi-directional replication with Qlik Replicate, focusing on setup, key configurations, and important features like loopback prevention. It covers various replication types, including uni-directional, multi-directional, and cascading, with a deep dive into the bi-directional process. Key steps include initial synchronization, setting up tasks for data replication from both A to B and B to A, and ensuring transactional mode is used. The session also addresses limitations, such as lack of conflict resolution, and provides a demo of the entire process, demonstrating how loopback prevention functions to avoid infinite replication loops.

Takeaways

• 😀 Bi-directional replication in Qlik Replicate allows data synchronization from A to B and B to A.
• 😀 Initial synchronization requires a Full Load task from A to B and, if needed, from B to A for tables unique to B.
• 😀 Bi-directional tasks are defined by setting the replication profile as Bi-Directional, and CDC (Change Data Capture) tasks are used with full load disabled.
• 😀 Loopback Prevention ensures changes are not replicated in an endless loop by using special tables that track updates.
• 😀 Both bi-directional tasks must have opposite Loopback Prevention settings to avoid circular replication.
• 😀 Transactional Apply mode is required for bi-directional tasks, as it records changes with accurate timestamps and ensures correct transaction order.
• 😀 Bi-directional tasks don't support conflict resolution, requiring full load tasks if conflicts arise to reset synchronization.
• 😀 Bi-directional tasks are not compatible with transformations and filters, which can only be used with Full Load tasks.
• 😀 When defining bi-directional tasks, each endpoint must be set as both a source and target with different database names for proper configuration.
• 😀 In the demonstration, changes on endpoint A were replicated to B and vice versa, with loopback prevention successfully preventing reverse replication.
• 😀 A common issue in bi-directional tasks is ensuring that loopback prevention works correctly to avoid circular replication when updates happen on both ends.

Q & A

What are the different types of replication mentioned in the script?

The types of replication mentioned are: Uni-directional (A to B), Bi-directional (A to B and B to A), Multi-directional (multiple endpoints replicating to each other), Many to One (multiple sources to a single target), Broadcast replication (one source to multiple targets), and Cascading replication (A to B, then B to multiple targets).

How is the initial synchronization between the two endpoints achieved in bi-directional replication?

Initial synchronization is done by running a Full Load task from A to B. If there are tables only existing on B, a Full Load task must also replicate those tables from B to A.

What is the significance of setting the replication profile as Bi-Directional?

Setting the replication profile as Bi-Directional ensures that the task will be a Change Data Capture (CDC) task, which is only concerned with capturing changes (no full load). This enables data replication from A to B and vice versa.

What role does the 'Loopback Prevention' setting play in bi-directional replication?

Loopback Prevention prevents changes from being replicated in an endless loop. It ensures that once a change is replicated to the opposite endpoint, it does not get picked up again and sent back, creating a loop.

Why is it important to configure the 'Loopback Prevention' schemas on both bi-directional tasks?

It is important because the 'Loopback Prevention' schemas must be defined to prevent endless replication loops. The settings should be opposites on both tasks to ensure that changes are not replicated back into the same task.

Why must bi-directional tasks be set to Transactional Apply mode and not Batch Apply mode?

Bi-directional tasks must be in Transactional Apply mode to ensure that changes are applied in the correct order and with the latest timestamp. This mode maintains transaction integrity and ensures no conflicts arise in the replication process.

What happens when a conflict occurs in bi-directional replication?

In the event of a conflict, the bi-directional replication tasks must be stopped. After that, Full Load tasks must be run on both sides to reset the synchronization point, and then the bi-directional tasks can be restarted.

What is the limitation regarding the use of transformations and filters in bi-directional replication?

A limitation of bi-directional replication is that transformations and filters cannot be used in conjunction with Full Load tasks, and CDC is only supported in Transactional mode.

How does the 'Loopback Prevention' work during a change on one of the endpoints?

When a change is made on one endpoint (e.g., BD-MySQL 1), the change is replicated to the other endpoint (e.g., BD-MySQL 4). However, the loopback prevention ensures that the change is not replicated back to the originating endpoint, effectively preventing a loop.

What must be done after the Full Load task and bi-directional tasks have been defined and the loopback prevention schema is set?

Once the Full Load tasks and bi-directional tasks are defined, and the loopback prevention schema is set, the bi-directional tasks should be started. Changes from both endpoints will then be captured and replicated, provided there are no conflicts.