Independent Research study
METR RCT: experienced open-source developers took 19% longer with early-2025 AI tools
16 experienced developers worked on 246 real issues in large, mature repos they knew well, with each issue randomly assigned to AI-allowed or AI-disallowed (mainly Cursor Pro with Claude 3.5/3.7 Sonnet). Developers expected a 24% speed-up and afterwards believed they'd been 20% faster, but were measured 19% slower.
The numbers (as reported)
| Developers | 16 |
|---|---|
| Tasks | 246 |
| Measured effect | 19% longer with AI |
| Expected speed-up (before) | 24% |
| Believed speed-up (after) | 20% |
| Repo size (average) | 22K+ stars, 1M+ lines |
Facts
- Company / subject
- METR (16 open-source developers)
- Tools
- Cursor, Claude 3.5/3.7 Sonnet
- Codebase
- brownfield
- Work type
- Research study
- Stack
- —
- Published
- 2025-07-10
- Last checked
- 2026-10-17
Caveats
METR now marks these results as out of date; see its February 2026 update.