The Rise and Fall of the Vector DB category: Jo Kristian Bergum (ex-Chief Scientist, Vespa)

Summary

In this insightful discussion, industry experts dive into the evolution of Retrieval-Augmented Generation (RAG) and its implications for data retrieval and knowledge graph creation. Topics include advancements in context windows, the limitations and potential of knowledge graphs, and the emergence of specialized embedding models for domains like legal and healthcare. The conversation also highlights the growing importance of visual language models and the challenges in building domain-specific models. The experts explore the future of AI-driven data processing and the role of platforms like Ax in fostering meaningful AI community engagement.

Takeaways

• 😀 Context windows have significantly expanded, from 4K tokens to potentially millions, making it easier to process large datasets directly within the model without needing external vector databases in some cases.
• 😀 Retrieval-augmented generation (RAG) remains important for large datasets that exceed the model's context window, but the increase in context size reduces the need for external databases in certain scenarios.
• 😀 Graph databases are excellent for traversing relationships, but the real challenge lies in building the knowledge graph itself, which is necessary before any retrieval system can function effectively.
• 😀 The speaker warns against overly associating technologies like graph databases with specific use cases, suggesting that alternatives like search engines can achieve similar goals in some cases.
• 😀 Knowledge graphs have historically been a difficult area, but LLMs can now more easily generate entities and relationships (triplets), making them more practical for certain applications.
• 😀 Domain-specific embedding models, such as those for legal, financial, or health-related documents, are becoming increasingly important, and the speaker hopes for more innovation in this space.
• 😀 The use of visual language models (e.g., screenshots instead of OCR) could significantly improve the accuracy of embeddings, offering a richer representation with fewer processing steps.
• 😀 The acquisition of Voyage by Nvidia highlights the tough business environment for domain-specific embedding models, as companies need to manage API-based services, compute costs, and customer willingness to pay.
• 😀 The speaker expresses hope for better general embedding models, especially in the context of domain-specific use cases like legal and financial documents.
• 😀 The AI community on platforms like X (formerly Twitter) offers high-signal interactions, fostering meaningful connections, and serving as an important space for professional engagement.
• 😀 Despite challenges in the embedding model market, the speaker remains optimistic about the potential for continued innovation, especially with domain-specific models that offer specialized solutions.

Q & A

What is the significance of the transition from 4K to 10 million token context windows in language models?

The transition from 4K or 8K to 10 million token context windows significantly improves a model's ability to process larger datasets in a single pass. This allows the model to handle more information at once without needing complex retrieval systems in some cases, making the process faster and more efficient.

Why is retrieval-augmented generation (RAG) still relevant despite the increase in context windows?

Despite larger context windows, retrieval remains important for efficiently handling vast amounts of information. For example, large datasets, like a 170,000 document set that exceeds millions of tokens, are impractical to load fully for each query. RAG systems help optimize how relevant data is retrieved and integrated for specific tasks.

What is the core challenge with knowledge graphs and how does it impact their use in AI?

The main challenge with knowledge graphs is building them—specifically creating the entities and relationships necessary for the graph. Although these can be useful for knowledge representation, generating the entity-triplets that make up the graph has traditionally been a bottleneck, though large language models are making this process easier.

How does the speaker view the debate between graph-based and vector-based RAG models?

The speaker suggests that while graph-based systems might offer advantages in some cases, the key issue lies in the technology itself. The importance of graph databases is often overstated, as simpler search engines can perform similar functions without the need for specific graph technologies. It's about choosing the right tool for the problem at hand.

Can graph RAG replace vector RAG systems completely?

No, graph RAG systems cannot replace vector RAG systems entirely. Both have their strengths depending on the context. While graph-based systems might be more effective in certain scenarios, the hybrid or combination of both could be more beneficial for specific use cases.

What role do LLMs (Large Language Models) play in the creation of knowledge graphs?

LLMs help streamline the process of creating knowledge graphs by automating the generation of entity-triplets, which has traditionally been a bottleneck. These models can now assist in extracting relationships between entities much more efficiently than previous methods.

Why is there a growing interest in domain-specific embedding models?

Domain-specific embedding models are gaining interest because they can provide much richer and more accurate representations of data in specialized fields, such as law, finance, or healthcare. These models can encode the specific nuances of these domains, offering a better starting point than general-purpose text embeddings.

What innovation is the speaker hoping to see in the embedding model space?

The speaker is hoping to see more domain-specific embedding models, particularly those that can handle complex documents like PDFs in fields such as legal, finance, and healthcare. Additionally, the use of visual language models to generate richer embeddings from screenshots, bypassing OCR, is an area the speaker wants to see grow.

What concerns does the speaker have about the business model for companies like Voyage in the embedding space?

The speaker is concerned that the business model for companies like Voyage, which focus on domain-specific embedding models, may be challenging. These companies need to balance the high computational costs with customer demand for API-based services, and there's uncertainty about whether customers are willing to pay for such specialized services.

What does the speaker think about the AI community's shift to platforms like Axe?

The speaker praises Axe (formerly Twitter) for its high signal-to-noise ratio, making it a great platform for networking and connecting with other AI professionals. Despite efforts to grow on platforms like LinkedIn or YouTube, the speaker acknowledges that Axe provides a more focused and vibrant AI community where important discussions and connections are happening.