Iterating on LLM apps at scale Learnings from Discord: Ian Webster

Summary

In this talk, a Discord developer shares key lessons learned while working on the launch of Clyde AI, a chatbot deployed to over 200 million users. The presentation covers crucial topics such as risk management, security challenges, and the importance of evals in AI product development. Key takeaways include the value of simple, efficient evals, the importance of pre-deployment risk assessment, and the necessity of red teaming to safeguard against harmful misuse. The speaker emphasizes the need for developer-friendly tools and robust observability to ensure the safe and successful integration of LLMs at scale.

Takeaways

• 😀 Evals (evaluations) are crucial for systematically assessing and mitigating risks in LLM products, allowing teams to understand model behavior and reduce harm before deployment.
• 😀 Keeping evals simple and deterministic (like unit tests) was key to scaling LLMs at Discord, making them fast, easy to run, and effective at detecting problems.
• 😀 A common risk in deploying LLMs at scale is encountering harmful or offensive content generated by models, which is why a robust predeployment risk assessment process is essential.
• 😀 Discord faced significant challenges with ensuring LLMs like Clyde AI did not generate harmful or illegal content, emphasizing the need for strong moderation and safety protocols.
• 😀 Developers should treat evals as an integral part of their workflow, running dozens of tests per day as part of a continuous improvement process, integrating them directly into CI/CD pipelines.
• 😀 Simple, context-specific evals (e.g., checking if a response begins with a lowercase letter to promote casual chat tone) can often achieve more than complex metrics with minimal effort.
• 😀 Observability tools like DataDog helped Discord track LLM performance metrics alongside product metrics, ensuring smooth integration and early identification of issues.
• 😀 The feedback loop for improving LLMs should ideally incorporate live data, but Discord had to rely on internal testing and external feedback (e.g., from users and social media) to refine their models.
• 😀 Red teaming—deliberately testing LLMs with adversarial inputs (e.g., harmful, illegal queries)—was vital to exposing vulnerabilities and ensuring safety before deployment.
• 😀 Prompt management and versioning (using tools like GitHub) were essential for ensuring consistency and preventing issues when switching between different models or updates.
• 😀 While live filtering is important, predeployment risk assessments and red teaming are far more effective at preventing issues with LLMs, especially when dealing with harmful or illegal content.

Q & A

What was the primary challenge faced when launching Clyde AI on Discord?

The primary challenge was not the technical aspects like models or fine-tuning, but ensuring the safety and security of the product. The team had to prevent harmful outcomes, such as teaching dangerous behaviors or promoting harassment, while getting legal and policy stakeholders comfortable with the launch.

What are 'evals' and why are they important in the context of launching LLMs at Discord?

'Evals' are systematic tests used to measure and characterize the behavior of a system based on specific inputs and outputs. They are crucial in ensuring that LLMs meet product requirements while minimizing risks, such as harmful behavior or failures, especially at scale.

Can you give an example of a simple and effective eval used at Discord?

One example was evaluating casual chat tone by checking whether the output began with a lowercase letter. This simple check was more than 80% effective in encouraging the desired casual tone with minimal effort, demonstrating the power of basic, deterministic evals.

What is the philosophy behind building an eval culture at Discord?

The philosophy was to treat evals as simple, deterministic tests, similar to unit tests in traditional software development. This approach meant that developers could easily run evals locally, without relying on external systems, allowing for rapid iteration and feedback.

How did Discord ensure its LLMs performed well at scale, especially when it came to moderation?

Discord used a combination of pre-deployment risk assessments, red-teaming, and production-level testing to identify and mitigate harmful behavior. This included testing responses to various adversarial inputs, ensuring models would reject inappropriate requests or be safeguarded against abuse.

What role did 'red-teaming' play in Discord's approach to LLM safety?

Red-teaming was essential for uncovering vulnerabilities and failure modes in the LLMs. Discord simulated real-world abuse scenarios, such as harmful queries, to test how the models would respond, and this helped the team refine safeguards and improve the system's resilience against misuse.

What is the risk of vendor lock-in with LLM prompts, and how did Discord address it?

Vendor lock-in occurs when prompts are tailored specifically to one model, making it difficult to switch to a different one. Discord addressed this by ensuring that their evals and prompts were flexible enough to work with various models, avoiding the dependency on a single vendor's format or style.

Why does Discord recommend keeping evals simple and deterministic?

Simple, deterministic evals are easier to implement, faster to run, and more reliable. By breaking down complex systems into small, manageable tests, Discord was able to quickly identify issues and iterate on solutions, which was especially important when scaling LLMs across millions of users.

How did Discord handle observability for its LLMs, and why was this approach effective?

Discord used an existing observability tool, DataDog, to track metrics related to LLM performance. This approach was effective because it integrated seamlessly with their existing infrastructure, avoiding the need for specialized tools and enabling them to monitor the models alongside other product data.

What were some of the challenges Discord faced when scaling up its LLMs for 200 million users?

One of the major challenges was ensuring the models could handle edge cases at scale. With 200 million users, even rare failures or harmful outputs could occur frequently. Discord focused on preemptively identifying and mitigating risks, using tools like evals and red-teaming to safeguard the models.