"Data readiness" is a Myth: Reliable AI with an Agentic Semantic Layer — Anushrut Gupta, PromptQL

Summary

In this talk, Anushut from PromQL challenges the myth of 'perfect data' for AI deployment, highlighting the messy, inconsistent, and evolving nature of real-world enterprise data. He demonstrates how traditional semantic layers, knowledge graphs, and manual standardization efforts often fail to capture business-specific nuances and tribal knowledge. PromQL offers a solution: an AI-powered, agentic semantic layer that learns and adapts like a human analyst, interpreting complex business rules, correcting itself, and improving over time. By decoupling planning from execution, PromQL ensures accurate, explainable, and self-improving AI that can reliably handle enterprise data from day zero to advanced usage.

Takeaways

• 😀 Data readiness is often a myth in AI deployment, as achieving perfectly clean, annotated data is almost impossible.
• 😀 AI systems face challenges when working with messy data, such as inconsistent naming conventions, missing values, and old column names.
• 😀 Past solutions like MDM and semantic layers are not enough to address data inconsistencies, as business data domains change frequently.
• 😀 AI systems struggle to understand domain-specific terms like 'GM', 'conversion', or 'active customer', which vary across departments and teams.
• 😀 Traditional analysts would use their deep business knowledge to navigate messy data and provide reliable answers; AI systems currently lack this capability.
• 😀 The solution lies in creating AI that learns over time, much like an analyst who gets better with experience and feedback.
• 😀 PromQL uses a domain-specific language to query data in a deterministic way, preventing LLMs from hallucinating answers.
• 😀 A strong AI solution should have the ability to improve over time, learning from user interactions to refine its understanding of the business domain.
• 😀 PromQL, as a tool, allows AI to query data reliably, even if the data is messy or incomplete, by generating query plans and executing them in a controlled environment.
• 😀 Continuous feedback from users helps PromQL’s AI improve its understanding of business terminology and data relationships, creating a more accurate and reliable AI over time.
• 😀 By learning from user corrections, PromQL’s AI continuously enhances the semantic layer, ensuring better data interpretation and more accurate results in future queries.

Q & A

What is the main topic of Anushut's presentation?

The main topic is how data readiness is a myth, and how AI can still work reliably with messy, imperfect data. Anushut emphasizes that businesses shouldn't wait for perfect data before deploying AI, but instead use AI that adapts and learns from the data over time.

Why does Anushut argue that data readiness is a myth?

Anushut argues that achieving perfectly structured and annotated data is an unrealistic goal. Data is often messy and inconsistent, with issues like unclear naming conventions and missing values. Businesses spend a lot of time cleaning data, but perfect data will never exist.

What is PromQL, and how does it help in dealing with messy data?

PromQL is a domain-specific language designed to help AI retrieve, compute, and aggregate data. It acts like a smart analyst, learning from mistakes and continuously improving. PromQL allows AI to work with messy data by creating plans for data queries that are executed deterministically, preventing issues like hallucination in AI outputs.

What does Anushut mean by 'AI needs to speak your business's language'?

Anushut highlights that AI systems typically lack domain-specific knowledge. For example, a term like 'GM' could mean different things in different contexts (Gross Margin in finance vs. General Manager in HR). AI needs to understand and learn the specific language and context of the business it operates in.

What traditional role do analysts play, and how is this related to AI?

Traditionally, analysts use their domain expertise and tribal knowledge to solve business problems, ensuring the answers are accurate and reliable. AI, in contrast, struggles to understand the nuances of business language, which is why PromQL aims to make AI more like a human analyst that learns over time.

How does PromQL's AI improve over time?

PromQL's AI improves by learning from user feedback and corrections. When the AI makes a mistake or doesn't understand something, users can guide it, and it adapts its understanding. Over time, the AI gains domain-specific knowledge, mapping relationships and definitions, thus increasing its accuracy.

What is the role of the semantic layer in PromQL?

The semantic layer in PromQL serves as a bridge between the AI and the business's specific domain knowledge. Unlike traditional semantic layers that rely on predefined rules, PromQL's semantic layer is dynamic and self-improving, allowing AI to adapt its understanding of the business context as it interacts with users.

Why is a deterministic runtime important in PromQL's design?

A deterministic runtime is crucial because it ensures that once the AI generates a query plan, it is executed reliably and without hallucinations. If the AI were responsible for generating the final answer, there would be risks of errors or incorrect results. PromQL decouples the AI's planning phase from the actual execution to maintain reliability.

Can you explain the example with the AI querying the top customers by revenue?

In the example, PromQL initially misunderstood the data and did not retrieve the correct results because it missed the correct status values (paid and pending). However, after identifying the mistake, PromQL corrected itself and delivered the accurate top five customers based on the correct status. This showcases how the AI learns from its errors in real-time.

What is the significance of 'Day Zero' and 'Day X' in the context of PromQL's AI?

Day Zero refers to when the AI first starts interacting with the business data. It knows little about the business and may make mistakes. Day X refers to a later stage, where the AI has learned and improved through continuous feedback, becoming highly accurate in answering complex business questions.