Multi Agent AI and Network Knowledge Graphs for Change — Ola Mabadeje, Cisco

Summary

Ola Mabad from Cisco’s OutShift group presents a product-focused overview of applying AI to network operations. The team addresses production failures during change management by integrating a natural language interface, a multi-agent system, and a network knowledge graph digital twin. Agents perform impact assessments, generate test plans, and execute tests using real-time network data, while results are fed back into ITSM systems like ServiceNow. The solution leverages open standards, scalable frameworks, and layered knowledge representation to optimize workflows, reduce errors, and enhance customer value. Early MVP results highlight the importance of the knowledge graph and agent framework for practical, measurable outcomes.

Takeaways

• 😀 AI-powered network change management can reduce failures compared to traditional rule-based automation.
• 😀 The solution involves three main components: natural language interface, multi-agent system, and a network knowledge graph with a digital twin.
• 😀 A unified network knowledge graph is critical for semantic search, operational flexibility, and multi-vendor support.
• 😀 The knowledge graph can ingest multiple formats, such as JSON, YANG, telemetry, and configuration files, with OpenConfig as a unified schema.
• 😀 ArangoDB was selected for its flexibility in supporting multi-model graph architecture, though Neo4j is still being considered for some use cases.
• 😀 An open-source agent framework based on industry standards aims to make agents interoperable and composable.
• 😀 Five types of agents are used in the application, each with specific roles, such as planning, orchestration, query handling, and test execution.
• 😀 In a network change scenario, agents autonomously handle tasks like impact assessment, test plan generation, and configuration deployment.
• 😀 The solution integrates with ITSM tools like ServiceNow, attaching assessment, tests, and results directly to tickets.
• 😀 Evaluation of the system focuses on extrinsic metrics (real customer value) and is still in the MVP stage, with scalability being a key priority.

Q & A

What is the role of the Outshift group at Cisco?

The Outshift group at Cisco is an incubation team focused on exploring emerging technologies, including AI and quantum networking. Their goal is to use these technologies to accelerate the roadmaps of Cisco's traditional business units.

How does Cisco's team approach product development?

The team starts by identifying customer problems, then works backwards to design solutions. They go through an incubation phase that includes customer interviews, prototype testing, A/B testing, and delivering a minimum viable product (MVP). Once they achieve product-market fit, the product is integrated into Cisco's core business units.

What specific problem did Cisco aim to address with AI in network management?

The team aimed to address the problem of failures in production during change management. They explored how AI could be used to reduce these failures and streamline the process of managing network changes.

What are the three key components of the AI-based solution described in the presentation?

The solution consists of a natural language interface for network operations teams, a multi-agent system that handles specific tasks, and a network knowledge graph (digital twin) that represents the actual production network.

What challenges did the team face when creating a representation of the network using knowledge graphs?

The main challenge was dealing with the variety of data formats and devices in the network environment. Different vendors' devices output data in various formats, such as YANG and JSON, making it difficult to create a unified representation of the network.

What were the performance and flexibility requirements for the knowledge graph?

The knowledge graph needed to support multimodal flexibility, allowing it to understand key-value pairs, JSON files, and relationships across network entities. It also required high performance, so engineers could query the graph for node information instantly, regardless of node location. Operational flexibility was also crucial, allowing for schema consolidation into one framework.

Why was ArangoDB chosen over Neo4j for this project?

ArangoDB was chosen because of its suitability for the use cases Cisco was working on, especially in the security space, where a recommendation system was needed. While Neo4j was considered, ArangoDB was ultimately preferred for the project's current requirements, though Neo4j is still being explored for future use cases.

What is the concept of a 'digital twin' in this context?

A digital twin, in this context, refers to a virtual replica of the production network. It includes a knowledge graph along with a set of tools for testing and analysis, enabling predictive actions based on the network's real-time data.

How does the system's agent framework work?

The system uses a set of agents that interact with each other and the knowledge graph. For example, the query agent interacts with the knowledge graph to retrieve data, while other agents handle tasks like impact assessment, test planning, and test execution. These agents are built to work together seamlessly, leveraging an open standards framework for integration.

What was the issue with the initial approach to querying the knowledge graph, and how was it resolved?

Initially, the team attempted to use a reasoning loop-based query system (RAG) for querying the knowledge graph, but this approach was slow and consumed too many tokens. To resolve this, they fine-tuned the query agent with specific schema information and example queries, resulting in a significant reduction in both the number of tokens used and query response time.

What was demonstrated in the live demo of the system?

The demo showcased how network engineers could use the system to manage firewall rule changes. It highlighted how agents, using natural language interfaces and interactions with an ITSM tool like ServiceNow, conduct impact assessments, create test plans, execute tests, and generate reports, all while interacting with the network's digital twin.

What evaluation metrics is Cisco using to assess the success of this AI system?

Cisco is using extrinsic metrics that map back to the customer's use case, focusing on the effectiveness of the knowledge graph and the open agent framework. These metrics help assess the value delivered to customers, although the system is still in the MVP stage and further evaluation is ongoing.