Instrument before you blame
Before deciding AI is helping or hurting, measure what is actually happening in your loop.
When AI-assisted output feels off, the instinct is to blame the tool, or the model, or the engineer. Usually the honest answer is that nobody is measuring the loop closely enough to know.
Baseline first. How long work actually takes to move through the loop, how much of it gets reworked, how confident the team is in code nobody read line by line. Once those are visible, the right fix is usually obvious, and it is rarely the one you would have guessed.
We point it at ourselves first
We run OpenTelemetry over our own Claude Code usage before we ever point it at a client. Per-tool execution time, token breakdown, cost per session, and loop detection for when the agent gets stuck going in circles. A week of our own sessions reduces to something like this:
loop.cycle_time 4.2h ▲ from 3.1h baseline
review.wait 2.6h ← where the work actually queues
rework_rate 31% kept after review: 69%
tokens / session 128k
cost / session $2.14What it taught us is that our instincts about where time goes are wrong about half the time. The stage that felt slow was fine. The stage we never thought about was quietly eating the afternoon. You cannot see that without the instrumentation, and once you can see it, the fix often takes an hour.
Count the thing that survives, not the thing that flatters
Most dashboards measure the output that is easy to count. The number that actually predicts delivery is the one almost nobody tracks: how much of the agent’s work you keep.
| Counts that flatter | Signals that survive |
|---|---|
| Commits, lines, PRs opened | Cycle time, measured per stage |
| ”We shipped more this sprint” | Rework rate: how much you kept |
| Tool adoption percentage | Variance between engineers |
Rework is hard to fake and hard to flatter. A team can double its commit count and feel productive while its rework rate climbs and its actual delivery stays flat. Code that gets written, reviewed, half-merged, then unpicked two days later because it was subtly wrong is worse than no code at all: it consumed the cheap stage and the expensive one.
If you measure one new thing this quarter, measure how much of what the agents help produce you actually keep.
Blame is cheap, and it is usually aimed at the wrong target. The tool is rarely the problem. The context you give it, the stage where work actually queues, the habits that vary wildly between engineers: those are the problems, and they stay invisible until you instrument them. Most teams are guessing, confidently, and calling it a strategy. Put a number on what is happening, then decide.