AI requires evolution of developer metrics

Developer platform GitHub has surveyed 500 (US-based) developers to gain a better understanding of how new AI tools are impacting their overall working experience. It found that wide-scale adoption is already having a measurable impact on productivity and collaboration but will call for changes in performance measurement.

AI’s impact on DevEx is driving adoption

The research found that 92% of developers are already using AI coding tools at work. 70% reported that AI coding tools will offer them an advantage at work, with numerous reported benefits driving adoption:

• 81% say AI coding tools will help increase collaboration within their teams and organisations, with security reviews, planning and pair programming identified as significant tasks to benefit.
• 57% of developers believe AI coding tools help them improve their coding language skills, enabling them to upskill.
• 53% of developers believe AI coding tools will make them more productive, particularly by automating parts of their workflow.
• 51% of developers believe that AI coding tools will give them more time to focus on solution design, helping them to stay creative.
• 41% of developers believe that AI coding tools can help with preventing burnout, by helping to reduce cognitive effort and freeing up mental capacity and time. In fact, in previous research GitHub conducted, 87% of developers reported that the AI coding tool GitHub Copilot helped them preserve mental effort while completing more repetitive tasks.

Traditional performance metrics need to evolve in the age of AI

One-third of developers report that their managers currently assess their performance based on the volume of code they produce (33%). The other top reported metrics for performance are largely output-based: 40% are measured by code quality, 34% by speed of completion, 34% by production incidents and 33% by number of bugs resolved.

However, with the increasing use of AI tools in software development — which often contributes to code volume — engineering leaders will need to ask whether measuring code volume is still the best way to measure productivity and output.

According to GitHub’s research, developers say AI coding tools can help them meet existing performance standards with improved code quality, faster outputs and fewer production-level incidents. They also believe that these metrics should be used to measure their performance beyond code quantity.

• 35% of developers believe they should be measured on collaboration and communication.
• 34% of developers believe they should be measured by code quality.
• 34% of developers believe they should be measured by number of bugs or issues resolved.
• 34% of developers believe they should be measured by time to complete a task.
• 32% of developers believe they should be measured by test coverage.
• 32% of developers believe they should be measured by velocity and quality of feature delivery.
   

“Developers today do more than just write and ship code — they’re expected to navigate a number of tools, environments and technologies, including the new frontier of generative artificial intelligence (AI) coding tools,” said Inbal Shani, Chief Product Officer, GitHub.

“But the most important thing for developers isn’t story points or the speed of deployments — it’s the developer experience, which determines how efficiently and productively developers can exceed standards, enter a flow state and drive impact.

“Ultimately, the way to innovate at scale is to empower developers by improving their productivity, increasing their satisfaction and enabling them to do their best work — every day. After all, there can be no progress without developers who are empowered to drive impact.”

Image credit: iStock.com/Viorika