What We Learned Deploying AI within Bloomberg’s Engineering Organization – Lei Zhang, Bloomberg

Summary

In this presentation, Lei, the leader of Bloomberg's Technology Infrastructure Department, discusses the company's innovative use of AI in software engineering. With over 9,000 engineers, Bloomberg integrates AI to enhance developer productivity, improve system stability, and streamline software processes. Lei emphasizes the challenges and successes of deploying AI tools like uplift agents and instance response systems. He also highlights Bloomberg’s commitment to training, collaboration, and evolving its codebase. Ultimately, the focus is on how AI is reshaping software engineering and driving cost-effective, high-quality development.

Takeaways

• 😀 Bloomberg's technology infrastructure department focuses on global infrastructure, data centers, connectivity, developer productivity, and reliability solutions.
• 😀 Bloomberg employs over 9,000 engineers, with a significant portion focused on AI research and engineering to enhance their flagship product, the Bloomberg Terminal.
• 😀 Bloomberg has one of the largest private networks in the world and a massive JavaScript codebase to support thousands of functions across various domains like finance, trading, and news.
• 😀 AI for coding was introduced over two years ago at Bloomberg to improve developer productivity and software stability, with teams quickly adopting AI tooling.
• 😀 The main challenge with AI for coding is understanding which AI tools best support engineering tasks, as the space is crowded with numerous offerings.
• 😀 Early use of AI tooling showed improved productivity for small tasks, but longer-term impact was less clear, especially beyond proof-of-concept work.
• 😀 AI has both significant benefits (speed, unbiased error-checking) and challenges (complexity in large codebases, risk of slowed review processes).
• 😀 A key initiative involves using AI-powered uplift agents to automatically suggest and apply patches across the codebase, though verification and testing remain complex.
• 😀 Bloomberg aims to strike a balance between freedom and responsibility in developing software, offering a 'golden path' for teams to adopt best practices while maintaining stability.
• 😀 Bloomberg has integrated AI coding tools into their well-established training program, equipping engineers with the skills to use AI effectively in their workflows.
• 😀 One of the challenges in AI adoption at Bloomberg is that individual contributors are quicker to adopt new tools than leadership, which can slow the broader organizational change.

Q & A

What is the primary focus of Lei's team at Bloomberg?

Lei leads the department of technology infrastructure at Bloomberg, which focuses on global infrastructure, including data centers, connectivity, developer productivity, and reliability solutions such as telemetry and incident response.

What is the core product of Bloomberg that the team is building software for?

The core product is the Bloomberg Terminal, which provides tools and data to empower users to make crucial financial decisions in their jobs.

What unique features does Bloomberg’s technology infrastructure possess?

Bloomberg has one of the largest private networks in the world and one of the largest JavaScript codebases globally, supporting a broad range of applications and functions.

How has AI been integrated into Bloomberg’s coding practices?

Bloomberg began integrating AI into their coding practices over two years ago, initially experimenting with various AI tools to boost productivity and stability. This includes tools for automating repetitive coding tasks and improving software engineering workflows.

What were some challenges faced when deploying AI solutions for coding?

Challenges included ensuring deterministic verification of code patches, managing the increased number of open pull requests, and ensuring the right objectives were achieved with AI tools rather than focusing solely on the method of implementation.

What is the concept of 'uplift agents' and how is it used?

'Uplift agents' are tools that scan the codebase to identify applicable patches and automatically propose pull requests with explanations of the changes, helping to streamline code maintenance and improve efficiency.

How does AI help in incident response at Bloomberg?

AI-powered incident response agents quickly scan codebases, telemetry systems, feature flags, and other systems, providing unbiased troubleshooting insights and speeding up the process of identifying and resolving issues.

What is the 'pay path' concept introduced by Bloomberg?

The 'pay path' is a streamlined platform that guides teams on which AI models to use for specific tasks, enables easy experimentation, and reduces friction in deploying AI tools across the company.

How does Bloomberg's AI adoption impact software engineering decision-making?

AI adoption has altered the cost structure of software engineering by making certain tasks cheaper and more efficient to perform, thus allowing engineers to focus on higher-quality software design and core principles.

What role do training programs play in AI adoption at Bloomberg?

Bloomberg has integrated AI for coding into its extensive training programs, helping new engineers adopt these tools quickly. This approach has fostered organic adoption across teams, with newer employees championing AI practices and challenging senior colleagues to innovate.