How to measure AI ROI for engineering teams: metrics finance trusts

To measure AI ROI for an engineering team, compare what AI tools cost with changes in merged, working output against a baseline, and check that quality and stability did not get worse. This guide covers which metrics to track, how to calculate cost per merged pull request, where to get the data and what to report to finance.

How do you measure AI ROI for engineering teams?

You measure AI ROI for an engineering team by comparing the full cost of AI tools with measurable changes in delivered work, against a baseline taken before rollout. In practice that means tracking three layers together:

1. Cost: subscriptions, API usage and setup time, expressed per developer and per merged pull request.
2. Outcomes: throughput (merged PRs, deployments), speed (cycle time) and stability (change failure rate, incidents, reverts).
3. Adoption: who uses the tools, how often and for what, so you can explain why outcomes moved.

ROI is a positive answer to one question: did we ship more working software, or the same software with less effort, for less than the tools cost, without making production less stable? Each metric below exists to answer part of that question.

Why is AI ROI so hard to measure?

AI ROI is hard to measure because the easy numbers are misleading and the meaningful ones move slowly. Three traps catch most teams.

People overestimate their own speedup. In METR's 2025 randomized trial, 16 experienced open-source developers took 19% longer on real issues when AI tools were allowed, yet afterward estimated they had been about 20% faster (METR, July 2025). METR's 2026 follow-up with newer tools suggests a speedup, but METR itself calls the new data an unreliable signal because many developers refused to work without AI (METR, February 2026). Surveys are useful for finding friction, not for computing ROI.

Output metrics inflate. Lines of code and number of PRs rise easily with AI. More code is also more code to review and maintain, so volume alone says little about value.

Speed can cost stability. The 2025 DORA report found that 90% of respondents use AI at work and more than 80% believe it increased their productivity. It also found a positive relationship between AI adoption and software delivery throughput, alongside a continuing negative relationship with delivery stability (Google Cloud blog). DORA's summary is that AI acts as an amplifier of an organization's existing strengths and weaknesses (DORA).

The consequence: never report a speed metric without a stability metric next to it.

Which metrics should you track for AI ROI?

Track a small set of metrics across cost, outcomes, quality and adoption, and resist adding more. This table is a workable starting set.

Pick one or two per layer for your first report. A team that tracks forty metrics usually explains none of them.

How do you calculate cost per merged pull request?

Cost per merged pull request is total AI spend for a period divided by the number of merged PRs that AI contributed to in that period. It turns a monthly bill into a unit cost you can compare across teams, tools and months.

cost per AI-assisted merged PR =    (subscriptions + API usage + review tooling for the period)    / merged PRs with AI assistance in the period

A worked example with illustrative numbers (replace them with yours):

On its own, $5 per PR means little. It becomes useful in three comparisons:

• Against engineering time. If the median AI-assisted PR saves even 15 minutes of an engineer's time, compare that time at your loaded hourly cost with $5. Estimate time saved from cycle-time changes, not from self-reports.
• Over time. A rising cost per PR with flat output means usage grew without results, often from runaway agent sessions or oversized context.
• Across teams. Large gaps point to teams with better workflows to copy, or to work that suits AI poorly.

Also compute cost per merged PR including review time. If AI-assisted PRs take twice as long to review, that time belongs in the cost.

Where do you get the data?

Combine three sources: your AI vendor's analytics and billing, your code host, and your incident and deploy data. Most of it is available without new tooling.

AI tool analytics

For Claude Code, Anthropic provides two dashboards (analytics docs):

• Team and Enterprise plans (claude.ai/analytics/claude-code): lines of code accepted, suggestion accept rate, daily active users and sessions, plus contribution metrics such as merged PRs with Claude Code and PRs per user, once you connect GitHub. Contribution metrics are in public beta.
• API customers (Claude Console): accepted lines, accept rate, activity and daily spend, with per-user spend in a team insights table. Console spend figures are estimates; your billing page has actual costs.

When GitHub contribution metrics are on, merged PRs that contain Claude Code-assisted lines get a claude-code-assisted label. Anthropic describes these metrics as deliberately conservative: code that developers rewrote by more than 20% is not attributed. You can count labeled PRs with the GitHub CLI:

gh pr list --repo your-org/your-repo --state merged --label claude-code-assisted \  --search "merged:2026-09-01..2026-09-30" --limit 1000 --json number --jq length

OpenTelemetry for per-team detail

For per-user or per-team cost, export Claude Code's OpenTelemetry metrics to your observability stack (monitoring docs). Available metrics include claude_code.cost.usage (USD), claude_code.token.usage, claude_code.pull_request.count, claude_code.commit.count, claude_code.lines_of_code.count, claude_code.session.count and claude_code.active_time.total.

export CLAUDE_CODE_ENABLE_TELEMETRY=1export OTEL_METRICS_EXPORTER=otlpexport OTEL_EXPORTER_OTLP_PROTOCOL=grpcexport OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317

Cost figures from telemetry are estimates; reconcile them with invoices monthly.

Delivery and stability data

Pull merged PRs, cycle time and deployment frequency from your code host and CI, and change failure rate from your incident tracker. If you already track DORA metrics, use the same definitions before and after rollout so the comparison is fair.

If your agents run on hosted boxes, some of this comes built in. Orkio Insights, for example, shows real spend against API-equivalent usage per project, per model and per merged pull request, with project budgets and alerts.

How do you set a baseline and run a fair comparison?

Set a baseline from the period before rollout, change one thing at a time, and compare over enough weeks to smooth out noise. A clean comparison is what makes finance trust the result.

1. Collect 8 to 12 weeks of baseline data for the metrics you chose: merged PRs per developer, cycle time, change failure rate, review time.
2. Define the cohort. Ideally, roll out to some teams first and keep similar teams as a comparison group for one quarter.
3. Allow a ramp-up of two to four weeks and exclude it from the comparison. Teams get slower before they get faster while workflows change.
4. Hold definitions fixed. Same PR size rules, same deploy definition, same incident severity scale.
5. Note confounders. Hiring, reorganizations, release freezes and holidays all move these metrics. Write them down as they happen.
6. Compare ranges, not points. Report "cycle time fell 10 to 20% across teams" rather than "AI saved 17.3%".
7. Run a survey at the end to explain the numbers: where AI helped, where review load grew, what blocked adoption.

Treat the pilot as part of a broader rollout plan with policy and training; our AI adoption playbook covers that side.

What should you report to finance?

Report one page: what you spent, what changed, what it means per unit, and what you plan next. Finance teams want a clear unit cost, a credible comparison and a clear list of caveats.

Period: Q3 2026 (weeks 3-12 of rollout), 4 teams, 38 engineersSpend- Subscriptions: $X  | API and agent usage: $Y  | Total: $Z- Spend per engineer per month: $AOutcomes vs baseline (Q2, same teams)- Merged PRs per engineer per week: B -> C (range across teams: ...)- Median cycle time: D days -> E days- Change failure rate: F% -> G% (stability check)- Share of merged PRs with AI assistance: H%Unit economics- Cost per AI-assisted merged PR: $I- Estimated engineering hours saved (from cycle-time change): J-K hours/monthCaveats- Two teams hired during the period; attribution metrics undercount AI use.Next quarter- Expand to N teams, add budget alerts per project, target cost per PR <= $L.

Keep the language plain. "We spent $14,000 and shipped about 15% more merged changes with the same stability" lands better than a dashboard screenshot.

What mistakes make AI ROI numbers misleading?

Most misleading ROI reports come from a short list of errors.

Controlling the cost side is often the fastest ROI win. Our guides to reducing LLM API costs and Claude Code usage limits cover caching, model choice and plan sizing.

How to start measuring AI ROI this quarter

You can have a credible first number within a month. Work through this list.

1. Choose five metrics: spend per engineer, cost per AI-assisted merged PR, merged PRs per engineer, cycle time and change failure rate.
2. Pull 8 to 12 weeks of baseline data for those metrics now, before any further rollout.
3. Turn on your AI vendor's analytics and, if you can, GitHub attribution or OpenTelemetry export.
4. Set a monthly budget and alert per team or project.
5. Run the comparison for one quarter, noting confounders as they happen.
6. Send finance the one-page report, with ranges and caveats, and a plan for next quarter.

Frequently asked questions