In summer 2025, the research organisation METR published a study that unsettled the tech industry. It is a textbook case for why companies should measure AI effects instead of asking about them.
The experiment
From February to June 2025, 16 experienced developersS worked on 246 real tasksS in large software projects they knew well. Each task was assigned at random: with AI tools, mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet, or without. A classic randomised controlled trial.
Beforehand, the developers expected a 24 %S speed-up. Measured, they took 19 % longerS with AI. And after the experiment they estimated that AI had made them about 20 % fasterS.
What it means and what it does not
The study has clear limits: a small sample, a wide confidence interval and a very particular setting, namely professionals in familiar, complex codebases. METR published new data in February 2026 and itself describes the 2025 results as out of date. The tools have changed a lot since.
The core lesson still holds: perceived and measured productivity can be far apart. The Harvard Business School and Boston Consulting Group study shows the other side. On tasks inside the AI frontier, the gains were large. Which of your team's tasks belong where can only be answered by measuring.
The workflow: a mini-experiment for your team
- 01StartPick 10 recurring tasks
- 02SystemAssign at random: with AI or without
- 03PersonDo the work and time it
- 04PersonHave quality rated blind
- 05CheckClear difference?no: run another round or keep the task without AI
- 06ResultRecord the decision per task type
Your measurement sheet could look like this:
| Task | With AI? | Minutes | Quality 1–5 (blind) | Rework |
|---|---|---|---|---|
| Draft a proposal | yes | |||
| Draft a proposal | no | |||
| Monthly report | yes |
Limits and risks
- Observer effect: People who are measured work differently. Measure over several weeks.
- Small samples: Ten tasks show a direction, not proof.
- Learning curve: The first weeks with a new tool are slower. Plan for it.
How to start
- Pick ten tasks that recur regularly and are of similar size.
- Let a random generator decide which are done with AI.
- Have results rated by someone who does not know how they were made.
- Decide per task type, not across the board.