From metrics to impact: Turn GitHub Copilot data into business value

Summary

This presentation explores the vital role of metrics in measuring AI adoption and impact, particularly through GitHub Copilot. The speaker highlights the use of AI metrics dashboards to track adoption, proficiency, efficiency, and ROI across various company roles—from CTOs to engineering managers. With real-life examples from Mona Inc., the session showcases how organizations can use metrics to drive strategic decisions, boost productivity, and demonstrate the tangible business value of AI. The session also introduces upcoming features, including expanded data windows and advanced impact metrics to further optimize AI utilization in software development.

Takeaways

• 😀 Metrics are a critical tool for connecting AI innovation to business outcomes and measuring AI's impact on organizations.
• 😀 GitHub's Copilot Metrics Dashboard offers enterprise-level usage and adoption metrics to help track AI's adoption across organizations.
• 😀 The metrics journey starts with adoption and usage, evolving into proficiency, efficiency, and finally, business impact metrics.
• 😀 Organizations can use the Copilot Metrics Dashboard to track adoption, active users, and usage trends at both organizational and individual levels.
• 😀 The CTO uses metrics to assess AI transformation progress and ensure AI adoption is progressing in their organization, with data-backed baselines.
• 😀 The VP of Engineering looks at metrics to understand how AI affects daily workflows, focusing on active users, pull requests, and onboarding trends.
• 😀 Engineering managers use user-level data to make day-to-day decisions, identify power users, and provide targeted support to improve adoption.
• 😀 Copilot metrics help organizations understand where AI adds value by measuring engagement with features like chat, agent, and inline code suggestions.
• 😀 Billing utilization metrics allow leaders to connect AI spend to business value, assessing the ROI of AI tools in terms of productivity and cost savings.
• 😀 The roadmap for GitHub Copilot includes rolling out more granular data, integrating AI usage into broader analytics, and expanding impact metrics.
• 😀 The journey from adoption to impact requires continuous iteration and feedback, ensuring AI tools evolve in line with organizational needs and goals.

Q & A

What is the primary purpose of the Copilot Metrics Dashboard?

The Copilot Metrics Dashboard helps organizations track AI adoption, usage, and business impact by providing detailed insights into how AI tools like GitHub Copilot are being used, their effectiveness, and their contribution to business outcomes.

How does GitHub plan to expand the Copilot Metrics Dashboard over time?

GitHub plans to enhance the dashboard by adding more metrics such as lines of code, deeper AI impact analysis, longer data trends, and integrations with other tools, as well as rolling out features like analytics agents and data warehouse connectors.

How are different roles within an organization using the metrics dashboard?

CTOs use the dashboard to measure AI transformation and adoption, VPs of Engineering focus on developer productivity and AI's impact on daily work, CIOs track cost and ROI, and Engineering Managers use granular metrics to assess team performance and enablement.

What kind of data can enterprise admins access with the Copilot Metrics Dashboard?

Enterprise admins can access usage and adoption metrics across their organization, including active user trends, agent adoption rates, billing analytics, and user-level reports through the dashboard or API.

What are the three phases of metrics tracking as described in the presentation?

The three phases are: Adoption and Usage metrics (showing who is using Copilot and how), Proficiency and Efficiency metrics (measuring AI effectiveness and workflow transformation), and Business and Impact metrics (tracking ROI, velocity, and quality).

What insights can a VP of Engineering gain from the dashboard?

The VP of Engineering can monitor active user trends, track how Copilot affects daily engineering work, compare usage patterns with throughput, and assess whether higher usage leads to higher productivity and better code quality.

What does the dashboard reveal about the success of Copilot within an organization?

The dashboard provides insights into adoption rates, active user trends, feature usage, and workflow efficiencies, helping organizations understand how well Copilot is integrated into their workflows and identify areas for improvement.

What role does the Copilot Metrics Dashboard play in tracking return on investment (ROI)?

The dashboard helps track ROI by connecting usage metrics (like lines of code generated) to financial outcomes, such as cost savings and productivity gains, allowing CIOs to quantify AI's business value.

What is the significance of lines of code metrics in the Copilot Metrics Dashboard?

Lines of code metrics help organizations track how much code is being generated, deleted, or modified using AI, which can then be translated into productivity gains and cost savings, offering a concrete way to measure the value of AI tools.

How does GitHub plan to increase the value of the Copilot Metrics Dashboard in the future?

GitHub plans to expand the dashboard’s capabilities by incorporating new metrics, such as non-IDE sources and code review metrics, as well as offering extended data windows and enhanced integration with external analytics tools.