There is an odd tension among software engineering leaders when it comes to thinking about productivity, according to Rob Zuber, CTO of CI/CD platform provider CircleCI. They have to weigh the joy of creating products that companies can use against the need to make sure that it’s done in a way that best benefits their own businesses.
“Most engineering leaders grew up as engineers and value the personal reward and fulfillment of being really productive, because nobody likes to fight through toil and wrestle with tools and all those sorts of things,” Zuber told The New Stack. “They like to deliver product. Whether they’re really into scaling backend systems or putting frontends in front of customers, whatever that might be, that’s what engineers really value.”
But when it comes to being measured, there’s “sort of a curious, allergic reaction” from engineers, who feel like they’re always being judged, he said. Organizations need to solve for that, because engineering leaders and other executives have to wrestle with the tug and pull of gauging what they’re getting from an efficiency perspective vs. whether they’re moving forward fast enough relative to their peers.
Adding ROI to the Mix
That’s a reason why San Francisco-based CircleCI is taking an expanded approach to its annual State of Software Delivery report, with the sixth edition released Tuesday. The report still looks at the key metrics used for defining performance – duration, throughput, mean time to recovery (MTTR), and success rate – but the vendor also is measuring the ROI organizations derive from them, a key measuring tool for business leaders and stakeholders.
That also becomes an important metric as AI permeates almost every tier of software development, just as its foothold in every other aspect of IT and business grows.
“Certainly, it’s having an impact on our ability to deliver in some way and [it’s important] knowing, are we being successful? Are we keeping up with the competition?” Zuber said. “Those sorts of things are on the minds of engineering leaders, and we want to give them as much information as we can to help them understand where they are and where they can focus their energy.”
The report is based on the vendor’s analysis of almost 15 million workflows of teams building software on CircleCI’s platform. It also explores what such advanced technologies as automation in CI/CD, Infrastructure as Code (IaC), and AI mean for delivering software.
CircleCI metrics
Speed Is Key
Top-level findings were that the top 25% of performers were continuing to separate themselves from the rest of the pack, in large part due to speed. For example, they shipped updates three times as fast as teams in the bottom 25%, giving them a market advantage in development velocity.
They saved millions in annual development costs, again due to speed: they completed critical workflows five times faster than lower-performing units – which freed resources for strategic initiatives – and debugged products in minutes rather than days, freeing up more time for developers.
“The parts that have been consistent over the years but are still really important to us [is] that moving quickly wins the marketplace and moving quickly is dependent on great systems, processes, [and] approaches,” he said.
This is where those advanced technologies – particularly AI and automation – come in. They’re ramping up the speed of software delivery, and those organizations at the top of the list are the ones adapting to the rapidly evolving nature of engineering and delivering value to their users.
“Historically, we always had to choose between speed and quality, and it’s finally set in that going fast is actually something that drives quality, because you just can’t do it unless you’re really good at quality,” Zuber said.
Other Metrics Factor In
However, he cautioned that it’s not the only gauge. A development team can churn out products quickly, but that will do no good if the product is faulty or if the organization is slow to make fixes. That’s where other metrics matter, from workflow duration to recovery speed to success rate. The aim of the report is to give developers and team leaders details they can sort through in a nuanced way.
“It allows people to see a little more of themselves and that, ‘Oh, this blend of metrics looks a little bit like us. What does that say?’ We’re really great at going quick until … something breaks and then it takes us a really long time to fix it,” the CTO said. “‘Is that because our systems are complex? Does it say something about our culture?’ It could be a number of different things. At that point, it really is on teams and leaders to ask the questions. Data is the beginning of a conversation, not the end of a conversation.”
RecurShip and ROI
That data includes the ROI numbers. In the report, the researchers quantify it through a fictional company called “RecurShip,” complete with 500 developers distributed around the world who are responsible for three commits a week – such as updates and optimization – and who are paid $180,000, or $1.50 per developer minute.
Duration – the time from when a workflow is triggered until all steps are complete – is an important metric. The data in the report indicates that the median duration time is 2 minutes and 43 seconds, with 25% of teams completing their workflows in less than 38 seconds. The other 75% completes it in 8 minutes or less. The fastest times could be the result of lighter workflows with fewer validation steps or other factors. However, some teams had duration times of 25 minutes or more.
Looking at RecurShip, optimizing workflow and reducing duration time from 20 minutes to 10 minutes, the company would recover 750,000 minutes of developer time a year, which at $1.50 per minute would translate into $1.1 million in annual productivity gains.
Another example is throughput, which measures the average number of workflow runs on a project per day. In the projects running on CircleCI, the median throughput is 1.64 runs a day, with 25% of developers reaching 2.7. Among the 20 most productive organizations, the daily throughput reaches 3,762, creating a “delta between average and top performers [that] suggests significant untapped potential in most software teams,” the report says.
At RecurShip, adding 25 engineers with the job of removing friction from development pipelines to its 500 developers (running 300 workloads per day, or 0.6 per developer) would boost throughput to 394 daily workflows, or 0.75 per developer per day.
The company would see a 25% improvement in productivity per developer based on a 5% increase in personnel and sees productivity gains that are equal to adding 156 full-time developers. This investment in optimization and the improvement in developer experience delivers $28.4 million back to the company in productivity gains.
AI Will Bring Promise, Change
The need for speed will only increase as AI and automation become more commonplace. Engineers and developers know how the new tooling used for building software will change how they operate, creating a lot of uncertainty. Given that, the job for engineering leaders is less about predicting the AI future and more about creating teams that can adjust to the changes when they come.
“I need to be able to adapt to that very quickly, so it helps us think about how we build, how we ship the kinds of systems that we build,” Zuber said. “Everyone is facing that right now.
“It’s much more interesting to think about how would I prepare myself for a set of possible outcomes and be prepared to adapt and adjust, because those are the folks that are really going to succeed. Succeeding in uncertainty, its agility, and if you’re not thinking about how we deliver software effectively and how we collect that feedback, then you’re not really setting yourself up for that level of change.”
Strategies, Big and Small
In the meantime, CircleCI mapped out steps that companies can take now to become faster and more efficient in developing and delivering software. Smaller companies need to build resilient pipelines that can run autonomously when their teams are working on other tasks to reduce the burden of debugging software by investing in automated testing.
For midsize companies, the job is keeping up the quick recovery times, standardizing processes, and replicating across the organization the practices used by high-throughput teams. Larger organizations need to streamline change management and approval flows and balance build speed optimization with processes that scale across business units, the report’s authors wrote.
(Team) Size Matters
In addition, all companies should be aware of the size of their teams.
“Smaller teams tend to move quickly with less coordination overhead, while larger teams must navigate dependencies, standardization, and process complexity as they scale,” they wrote. “Understanding these trade-offs is key to optimizing development velocity and reliability.”
Recommended strategies include creating autonomous development teams of five to 10 engineers, while companies scaling beyond 100 developers can use standardized tools and processes to keep their fast MTTR.
Companies with 50 to 100 engineers tend to hit throughput and other barriers when they scale to this size and then get their groove back as they grow beyond those numbers, the CTO said. Given that, these companies need to invest in automation to help drive higher throughput and push past what CircleCi calls “complexity barriers” common at this size.
The post The ROI of Speed: How Fast Code Delivery Saves Millions appeared first on The New Stack.