GitHub Copilot: the AI pair programmer for today and tomorrow

The presentation at GitHub introduces GitHub Copilot, an AI-powered coding assistant that has been in development for nearly three years. Ryan J. Selva, VP of Product at GitHub, discusses the research-led approach focusing on code quality, developer happiness, and productivity. Accenture, with a team of around 125,000 software engineers, collaborates with GitHub on a study to measure the real-world impact of AI-assisted software delivery. The study reveals a 35% acceptance rate of AI suggestions, an 88% retention rate after review, a 45% increase in build success rate, and a 15% increase in completed pull requests. Developers found Copilot extremely useful, with 50% considering it to be highly beneficial. The talk also covers the future of Copilot, including custom models for personalized experiences, the ability for organizations to create their own models, and the extensibility of the platform through plugins, aiming to transform software development and elevate developer productivity.

Takeaways

• 🚀 **GitHub Copilot's Impact**: The integration of GitHub Copilot has shown significant benefits in developer productivity, with a 15% increase in completed pull requests and a 45% increase in build success rates over six months.
• 🤖 **AI-Assisted Software Delivery**: Accenture's research indicates that AI assistance, particularly with GitHub Copilot, is transforming daily work routines for developers, leading to time savings and improved code quality.
• 🧠 **Research-Led Development**: GitHub Copilot's development has been guided by research into code quality, developer happiness, productivity, and economic impact, aiming to understand real-world application beyond controlled environments.
• 🔍 **Customization and Fine-Tuning**: GitHub is focusing on allowing organizations to fine-tune Copilot models to align with their specific coding styles, preferences, and legacy code bases, including support for older languages like COBOL and Fortran.
• 📚 **Knowledge Integration**: Copilot is being expanded to integrate with internal documentation and guidelines, enabling developers to access organization-specific knowledge directly from the AI suggestions.
• 🔗 **Extensibility and Plugins**: The platform is opening up for extensibility, allowing third-party integrations through plugins, which can enhance Copilot's capabilities and keep developers within their workflow.
• 🌐 **Multi-Model Approach**: GitHub is moving towards a multi-model approach where different models can be used for different tasks or preferences, offering a more personalized experience.
• 📈 **Economic and Productivity Gains**: Early adopters of GitHub Copilot, including large engineering teams, have reported substantial economic and productivity gains, with developers saving up to 20% of their time on certain tasks.
• 📝 **Documentation and Best Practices**: GitHub is providing guidelines and best practices for integrators to ensure the safety, security, and privacy of the plugins and the data they handle.
• 🔧 **Developer Feedback**: The team behind GitHub Copilot is actively incorporating developer feedback to continuously improve the tool, reflecting a user-centric approach to product development.
• 🌟 **Community Involvement**: GitHub is inviting the developer community to participate in the evolution of Copilot through the partner program, emphasizing collaboration and shared success.

Q & A

What is the role of Ryan J. Selva at GitHub?

Ryan J. Selva is the VP of Product at GitHub, where he is responsible for GitHub Copilot.

What is GitHub Copilot?

GitHub Copilot is an AI pair programmer tool designed to assist developers in building and creating code more efficiently.

How long has GitHub been working with GitHub Copilot?

GitHub has been working with GitHub Copilot for nearly three years.

What is the focus of the research led approach that GitHub Copilot has taken?

The research led approach focuses on code quality, developer happiness and productivity, and even the economic impact of using AI in software development.

What was the purpose of the high quant study conducted with external companies?

The high quant study aimed to understand the real-world impact of large language models like GitHub Copilot on engineering teams, especially in messy and complex environments.

How many software engineers does Accenture have?

Accenture has approximately 125,000 software engineers.

What was the overall acceptance rate of suggestions from GitHub Copilot in the study?

The overall acceptance rate of suggestions from GitHub Copilot was about 35%.

What was the increase in the success rate of builds when developers used GitHub Copilot?

There was about a 45% increase in the success rate of builds when developers used GitHub Copilot.

How many more pull requests were completed by developers using GitHub Copilot?

Developers using GitHub Copilot completed about 15% more pull requests.

What was the feedback from developers who participated in the study regarding GitHub Copilot's usefulness?

100% of the developers who responded to the survey found GitHub Copilot useful, with 50% stating it was extremely useful.

What are the future plans for GitHub Copilot?

GitHub plans to roll out Copilot to all of their developers and continue partnering with external companies to offer code solutions to clients, while also focusing on integrating Copilot into the rest of the software delivery lifecycle.

Outlines

00:00

🎉 Introduction and GitHub Copilot Overview

Ryan J. Selva, VP of Product at GitHub, welcomes the audience and discusses GitHub Copilot, emphasizing its status as a pioneering large language model product. He highlights the research-led approach focusing on code quality, developer happiness, productivity, and economic impact. Selva also mentions a study conducted in collaboration with external companies to understand the real-world impact of Copilot on engineering teams and introduces Dan Shockey from Accenture as a research partner.

05:01

🔍 Accenture's Research on GitHub Copilot

Dan Shockey from Accenture explains his company's role in digital transformation and the complexity of their development environment with a diverse team of 125,000 software engineers. Accenture's interest in AI-assisted software delivery led to their collaboration with GitHub on research regarding Copilot. The study involved enabling Copilot for hundreds of internal developers and observing the results. The findings showed a 20% time savings, increased build success rates, and a 15% increase in completed pull requests, all of which contributed to higher developer satisfaction.

10:02

📈 GitHub Copilot's Impact on Developer Productivity

The research findings are discussed in detail, revealing that 450 developers using Copilot accepted suggestions at a rate of 35%, retained 88% of the code after review, and showed a 45% increase in build success rates. Additionally, there was a significant increase in the number of builds and pull requests completed, leading to the conclusion that Copilot enhances both developer productivity and code quality. Survey results from developers were positive, with 50% finding Copilot extremely useful.

15:05

🚀 Future Outlook for GitHub Copilot

Looking forward, Accenture plans to roll out Copilot to all developers and continue the partnership with GitHub. Personal excitement is expressed about the potential for Copilot to extend beyond the code editor into the entire software delivery lifecycle. The discussion transitions to developer experience, custom models, and extensibility, with an introduction of Harald from the VS Code team to discuss these topics further.

20:07

💡 Enhancing Developer Experience with GitHub Copilot Chat

Harald, a product manager from the VS Code team, demonstrates GitHub Copilot Chat's integration into the development environment. He showcases how Copilot Chat can be customized to fit a developer's workflow, including moving it to the side, using it in the editor area, or through a Quick Chat command. The demonstration highlights how Copilot Chat uses a hybrid approach to provide relevant code snippets, including GitHub's knowledge graph, a local index, and VS Code's language intelligence.

25:09

🧩 Customizing GitHub Copilot with Multi-Model Approach

The use of GPT-4 in chat conversations is discussed, alongside other models like Jupiter 3.5 Turbo, to achieve a balance of intelligence and speed. The multi-model approach is a result of continuous experimentation. The benefits of inline chat for developers are highlighted, as it allows for direct interaction with AI within the code. Commands like 'fix' are introduced to streamline common tasks and improve the development flow.

30:12

🌟 Custom Models and GitHub Copilot's Future

The focus shifts to custom models, with a discussion on how personalization can be achieved through fine-tuning large language models to suit individual team preferences and styles. The process of fine-tuning is explored, including using repository data, developer feedback, and reinforcement learning. The potential for domain-specific models tailored to different teams is also mentioned, with AMD's experience being highlighted as a case study.

35:14

🔧 Fine-Tuning GitHub Copilot for Specialized Languages

Alex from AMD discusses the challenges hardware developers face with languages like Verilog and the benefits of fine-tuning Copilot for these lower-level languages. The process of identifying representative code and working with GitHub to create a custom model is detailed. The results showed that the fine-tuned model produced code more aligned with AMD's style and was preferred by their engineers, even for languages like C that were not initially supported.

40:17

🌐 Expanding GitHub Copilot's Extensibility

The concept of extensibility in the GitHub Copilot ecosystem is introduced, with a vision for a marketplace of plugins that integrate with third-party developer tools and services. The potential for AI-based economy and the transformation of software development is emphasized. The invitation for developers to join the GitHub Copilot partner program to shape the future of Copilot is extended.

45:21

🤝 Building the Future of GitHub Copilot Together

The presentation concludes with a call to action for the developer community to join the GitHub Copilot partner program and contribute to the future of Copilot. The importance of collaboration and feedback from the community is stressed as key to the ongoing development and improvement of Copilot. The team behind Copilot is thanked for their hard work, and appreciation is expressed for the community's engagement and feedback.

Keywords

GitHub Copilot

GitHub Copilot is an AI-powered code generation tool developed by GitHub. It assists developers by providing suggestions for code as they write, making the coding process more efficient. In the video, Ryan J. Selva, the VP of Product at GitHub, discusses the impact of GitHub Copilot on developer productivity and code quality.

Research-led

The term 'research-led' refers to an approach where decisions and developments are driven by research findings. In the context of the video, GitHub Copilot is described as being research-led, which means its features and improvements are based on research about code quality, developer happiness, productivity, and economic impact.

Large Language Models

Large Language Models (LLMs) are AI models that are trained on vast amounts of text data and can generate human-like text. In the video, the discussion revolves around the use of LLMs in GitHub Copilot to assist with coding, emphasizing their role in enhancing the software development process.

Economic Impact

Economic impact refers to the effect that a particular innovation or change has on the economy. In the video, the economic impact is mentioned in relation to the research conducted on GitHub Copilot, highlighting how the tool can potentially lead to cost savings and value delivery for organizations.

Productivity Gains

Productivity gains refer to improvements in the efficiency of work or the output produced within a given time frame. The video discusses how developers using GitHub Copilot have reported saving time on their tasks, leading to significant productivity gains.

Code Quality

Code quality is a measure of the efficiency, readability, maintainability, and overall effectiveness of the source code of a software application. The video emphasizes that GitHub Copilot not only increases developer productivity but also leads to improvements in code quality.

Custom Models

Custom models in the context of the video refer to the ability to tailor AI models to specific coding styles, preferences, and requirements of individual developers or teams. This customization can lead to more relevant and useful code suggestions from GitHub Copilot.

Extensibility

Extensibility is the ability of a system to be expanded or modified. In the video, extensibility is discussed in relation to GitHub Copilot, where the platform is being expanded to allow for integration with third-party developer tools and services, creating a more comprehensive development ecosystem.

Developer Experience

Developer experience encompasses the tools, processes, and practices that make up a developer's workflow. The video focuses on how GitHub Copilot enhances developer experience by integrating AI assistance directly into the coding process, making it more efficient and enjoyable.

Pull Requests

Pull requests are a feature in version control systems that allow developers to propose changes to a project's codebase. In the video, it is mentioned that developers using GitHub Copilot complete more pull requests, indicating an increase in the volume of work being contributed to projects.

AI Economy

The term 'AI economy' refers to the ecosystem of AI tools, services, and marketplaces that are emerging as a result of AI technology advancements. The video discusses the vision of creating an AI-based economy with GitHub Copilot, where developers can integrate and develop plugins to further customize and enhance their coding experience.

Highlights

GitHub Copilot is a pioneering large language model product that assists developers in building and creating code.

GitHub Copilot has been research-led, focusing on code quality, developer happiness, productivity, and economic impact.

A high-quant study with hundreds of developers was conducted to measure the real-world impact of GitHub Copilot on engineering teams.

Accenture, with approximately 125,000 software engineers, partnered with GitHub to understand the practical implications of AI-assisted software delivery.

Developers using GitHub Copilot reported saving about 20% of their time on unit test cases alone.

450 internal developers at Accenture were observed using GitHub Copilot, leading to a 45% increase in successful builds.

The study found that developers accepted suggestions from Copilot at a rate of 35%, with an 88% retention rate after review.

There was a 15% increase in the number of pull requests completed by developers using GitHub Copilot.

Surveyed developers found GitHub Copilot useful, with 50% considering it extremely useful in their daily work.

GitHub Copilot is being rolled out to all developers at Accenture and will be offered to clients to accelerate market adoption.

The future of GitHub Copilot includes expanding beyond the code editor into the entire software delivery lifecycle.

Custom models for GitHub Copilot are being developed to personalize the AI's suggestions to individual teams' preferences and coding styles.

AMD has partnered with GitHub to fine-tune Copilot models for lower-level languages and hardware description languages.

Custom fine-tuned models have shown to produce code more aligned with AMD's engineering style and preferences.

GitHub Copilot's extensibility allows for the integration of third-party developer tools and services, creating a more powerful development ecosystem.

The extensibility of GitHub Copilot includes the ability for users to bring their own data and for integrators to build plugins for a more personalized experience.

GitHub is committed to safety, security, and privacy, and will provide guidelines and review processes for plugins to ensure quality and trust.