Technical Lineage in CDGC

Summary

In this informative video, K Sh, a senior solution architect at Informatica, explains the concept of technical lineage in CDGC (Cloud Data Governance and Catalog). He outlines its significance in understanding data flow between integrated systems, ensuring data accuracy, and identifying potential issues. The video includes a practical demonstration of building technical lineage, showcasing a data movement scenario from Snowflake to a file system. By scanning source and target systems and performing connection assignments, viewers learn how to visualize data dependencies and enhance data governance practices effectively.

Takeaways

• 😀 Technical lineage in CDGC represents the connectivity between technical objects such as tables, views, and fields.
• 😀 Understanding data flow is crucial when integrating multiple systems, such as Snowflake and Oracle.
• 😀 Technical lineage ensures data accuracy, consistency, and integrity throughout the data lifecycle.
• 😀 It helps identify potential issues, optimize performance, and comply with regulatory requirements.
• 😀 Scanning source and target applications is the first step in building technical lineage in CDGC.
• 😀 Connection assignments are critical for linking integrated objects in data mappings.
• 😀 Demonstrating lineage involves showing how data moves through different states, from raw to live.
• 😀 The connection assignment process can be performed at both the database and schema levels.
• 😀 Data observability features can be enabled to monitor data behavior and identify anomalies.
• 😀 The lineage provides insights for impact analysis during system design enhancements.

Q & A

What is technical lineage in the context of CDGC?

Technical lineage represents the connectivity between technical objects such as tables and views, illustrating how data flows between these components.

Why is technical lineage important for data integration?

It helps in understanding the data flow between integrated systems, ensuring data accuracy, consistency, and integrity throughout the data lifecycle.

What are the key benefits of using technical lineage?

Key benefits include the ability to identify potential issues, optimize performance, ensure compliance with regulatory requirements, and manage data quality effectively.

What are the three main steps to build technical lineage in CDGC?

The three steps are: scanning source and target applications, scanning integration systems, and performing connection assignments to link these objects.

Can you provide an example of a data flow scenario discussed in the presentation?

An example discussed is the movement of employee information data from Snowflake to a file system, transitioning from a staging table to a landing table before reaching the file system.

How does the scanning process work in building technical lineage?

Scanning involves identifying and cataloging source and target systems as well as integration mappings to establish a comprehensive overview of data flows.

What role do connection assignments play in technical lineage?

Connection assignments link the scanned objects to their respective sources and targets, allowing for a clear visualization of data movement and dependencies.

What challenges does technical lineage address in data governance?

It addresses challenges such as identifying bottlenecks in data flow, validating data quality, and facilitating impact analysis when changes to systems are needed.

How can users validate the accuracy of the lineage visualizations?

Users can confirm the lineage by checking the original objects within the system and ensuring the data flow representation aligns with expected outcomes.

What additional feature was introduced to enhance data lineage analysis?

The presentation mentions a feature called data observability, which helps track data behavior and identify anomalies throughout different data lifecycle stages.