Everyone says AI is making developers dramatically faster.
But the most important study published this year suggests the reality is far more nuanced.
The METR Study
Researchers at METR ran one of the first serious randomized controlled trials on AI coding productivity, using experienced open-source developers working in their real codebases with modern AI tools like Cursor and Claude.
The result?
Developers using AI were actually 19% slower on average.
Even more interesting:
- The developers felt significantly faster
- Experts predicted large productivity gains
- Reality was the opposite
At first glance, this sounds like an argument against AI coding tools. It isn't. It's an argument against using them incorrectly.
What the Study Actually Exposes
The study exposed something engineering leaders are starting to notice internally: AI compresses execution cost, not thinking cost.
And the highest-value engineering work is still dominated by:
- Architecture
- Tradeoffs
- Debugging ambiguity
- System understanding
- Long-horizon maintainability
- Coordination complexity
AI is genuinely good at the parts of engineering where the answer is mostly mechanical. It is still working out how to help with the parts where the answer requires holding a system in your head.
Where AI Helps and Where It Struggles
Where AI did help:
- Boilerplate
- Repetitive implementation
- Test generation
- Documentation
- Scaffolding
- Unfamiliar syntax
- Iteration speed
Where it struggled:
- Large mature codebases
- Hidden dependencies
- Context-heavy systems
- Nuanced refactors
- Architectural reasoning
The pattern lines up with what we increasingly see across engineering organizations. AI is not slower or faster as a blanket statement. It is faster for some tasks, slower for others, and the mix matters.
How the Best Developers Actually Use It
The best developers don't use AI as autopilot. They use it as:
- Acceleration
- Exploration
- Draft generation
- Implementation leverage
The difference matters. Autopilot users let the tool drive and inherit whatever it produces. Leverage users stay in the loop, edit, redirect, and verify. Same tool. Very different outcomes.
The Real Bottleneck Is Shifting
Engineering performance is not about typing speed. It's about reducing cognitive load while preserving judgment quality.
That's also why measuring engineering productivity through output metrics alone becomes increasingly misleading in the AI era. As AI lowers implementation friction, the real bottleneck shifts toward:
- Decision quality
- Clarity
- System design
- Prioritization
- Review overhead
- Collaboration effectiveness
If developers feel faster but outcomes don't always improve, how do you actually measure whether AI tooling is creating value?
Old Metrics Don't Survive the Transition
Most organizations still rely on simplistic productivity signals:
- Lines of code
- Ticket throughput
- Pull request counts
- Activity metrics
But AI changes the meaning of those metrics completely. When implementation becomes cheap, "more output" stops being a signal of more value. It just means the cheap part got cheaper.
As implementation becomes cheaper, the real differentiators become:
- Decision quality
- Review burden
- System complexity
- Collaboration effectiveness
- Architectural consistency
- Delivery predictability
The Real Engineering Edge
The organizations that benefit most from AI won't necessarily be the ones generating the most code. They'll be the ones that best understand how AI impacts engineering performance across teams, workflows, and delivery outcomes.
That's the real engineering edge. Not whether your developers feel faster — but whether you can tell, with evidence, when the speed is real and when it's just a feeling.
