# Co-Design AI4SE Efficiency Metrics with Your Team: From Fuzzy “% Gains” to Comparable Measurement

When experienced engineers gather, the conversation quickly slides into “how much faster are you with AI?” Someone says 100%, someone says 300%, and someone blurts out 1200%. The numbers sound exciting, but the denominator is often empty: is the speedup in writing code, getting through review, shipping to production, or the whole value stream from request to something users can see? Even the people who say these numbers sometimes cannot explain how they were calculated.

Careful listeners easily feel anxious in these exchanges: looking end to end, their own teams have not seen the leap in the stories. The problem is often not “you are not using AI hard enough,” but fuzzy definitions of efficiency gains—without a falsifiable yardstick, what remains is emotion and comparison.

So instead of arguing who improved more, open the topic: dig in with the team on how to define metrics for AI R&D efficiency gains, and how to implement them. The panorama of DORA, SPACE, and DevEx is already covered on this site (see From DORA to DevEx and DORA 2025: AI Is an Amplifier). This article is a workshop script: how to run the meeting, how to co-design, how to compare, and which numbers to start with.

One discipline up front—we will return to it throughout: efficiency comparisons that do not control variables are meaningless; comparability comes before percentages.

1. Take the “efficiency story” apart

Oral “% faster” claims usually fail in three ways.

1. Undefined denominator: personal coding time, iteration throughput, or lead time for changes under roughly stable failure rates? Mixing them blocks retrospection.
2. Speed without stability: faster coding that makes releases more fragile is not necessarily a win at organization level. Recent Google DORA research keeps stressing that AI is primarily an amplifier—it amplifies the strengths of high-performing organizations and the weaknesses of thin pipelines and thin quality guardrails. A felt sense of “writing at warp speed” does not substitute for delivery outcomes.
3. Cross-condition bragging: comparing “% gains” across different teams, demand difficulty, iteration length, or Definition of Done is scientifically invalid.

So the first goal of the meeting is not another inspiring percentage. It is a shared measurement language the team can sign: outcome metrics first, collectible and computable, comparable before/after under controlled conditions. Activity metrics (lines generated, commit count, PR count) are easy to distort in the AI era—see Developer Productivity in the AI4SE Era. This article deprioritizes them unless a group can prove same-definition comparison and resist gaming.

2. Measurement literacy session: align language first

Bring end-to-end roles into one room: product/BA, development, QA, architecture (ops/security if needed). Aim for 60–90 minutes of terminology only—do not conclude “how much we improved.” For staffing and convergence discipline, borrow from An AI4SE Workshop Is Not a Training Class; this session’s topic is specifically metrics.

2.1 What DORA’s four key metrics are about

The four outcomes crystallized in Accelerate (Forsgren, Humble, Kim) and advanced by DORA are good guardrails: look at speed and stability together—never speed alone.

Metric	Common meaning	Use in AI4SE discussions
Deployment Frequency	How often meaningful changes reach production	Blocks “write fast, ship never”
Lead Time for Changes	Time from code commit to running in production	Aligns pipeline and release capability (narrow definition)
Change Failure Rate	Share of changes that degrade service and need remediation	Quality floor under speed
Time to Restore / MTTR	Time to restore usable service after failure	The other half of stability

Say out loud: DORA’s Lead Time for Changes is not lean Lead Time from “customer request to delivered value.” Confusing the two is the top reason later numbers refuse to reconcile.

2.2 Lean Lead Time and Cycle Time

• Lead Time (lean): request enters the system (or customer places the order) → usable value delivered. Includes waiting and processing.
• Cycle Time: work actually starts (e.g. enters In Progress) → Done; or elapsed time between any two SDLC nodes you choose. Better for diagnosing which segment is stuck.

Toyota / Lean Software and DevOps (DORA) start “Lead Time” in different places. Neither is “wrong”—the rulers differ. After the session, post a one-page terminology map: freeze start/end points for every term on this team; forbid free drift in later discussion.

2.3 Bridging to AI coding efficiency

Agents mostly compress the Inner Loop (understanding, drafting, local edits). That does not automatically shorten lean Lead Time or raise iteration throughput—review, integration tests, environments, and release gates may become the new bottleneck. The DORA 2025 reminder: watch throughput and keep watching stability. The literacy session’s output is not a KPI list; it is that everyone starts describing flow and risk in the same language, paving the way for brainstorming.

3. Co-design in groups: diverge in pairs, funnel with TLs

3.1 Staffing and cadence

We use a fixed structure so later “experiential evaluation” can lock comparison conditions:

• 2 large groups (so they can challenge each other’s definitions)
• 2 small groups inside each large group
• 2 people paired in each small group

Four layers of collaboration:

1. Pair diverge: the two write “metric cards” together—definitions must be computable and data collectible.
2. Merge inside the large group: the two pairs review each other, resolve ambiguity, merge same-name-different-meaning items → large-group candidate set.
3. Cross-check the two large groups: swap candidates; pick the 1–2 definitions with the largest disagreement and align on the spot (usually start/end points).
4. TL funnel: metrics owner + TLs merge, drop what cannot be collected, fill must-have gaps, lock a team metric set (prefer 3–6, fewer is better), and write down formulas, rules, and worked examples.

3.2 Facilitation prompt (projector-ready)

Project this as-is (Superpowers Brainstorming or an equivalent co-design flow also works):

I want to explore and lock metrics that quantify how AI coding assistants improve development efficiency. Metrics may include—but are not limited to—outcome measures such as Lead Time and Cycle Time.

Help identify the KPIs best suited to measure AI coding efficiency gains, and discuss calculation methods, data collection, and how to ensure measurement accuracy. Requirements:

1. Quantitative metrics that can show efficiency gains to management
2. Definition, formula, and measurement method for each metric
3. Feasible collection (Git, CI/CD, code review tools, etc.)
4. How to build baselines and run comparisons
5. Factors that hurt accuracy, and countermeasures
6. Prefer objective, automatable metrics; avoid purely subjective judgment

3.3 Minimum fields on a “metric card”

Field	What to make explicit
Name and business question	Which management/team question this number answers
Precise definition	Start/end timestamps; which work items are in/out
Formula and aggregation	By story / by iteration / per capita; sum vs percentiles
Data sources and owner	Board, Git, CI, review tools, bills; who maintains
Baseline and comparison window	How to build the pre-AI baseline; how long the window is
Error and exclusion rules	Holidays, cross-team waits, hotfixes, whether learning cost counts

3.4 Hard constraints (write into meeting rules)

• Outcomes over activity; every speed metric needs at least one quality/stability companion.
• Any proposal that cannot say how pre- and post-AI will be compared under the same definition is downgraded or sent back in the funnel.
• Done looks like written formulas + worked examples for every selected metric—not enthusiasm.

4. Five-step experiential evaluation: metrics first, then “how much better”

4.1 Comparability before percentages (hard rule)

Efficiency comparisons that do not fix variables are not “controversial”—they are invalid. Only changes observed under the same conditions—before vs after introducing AI (or another intervention)—are meaningful. Recommended “good rulers” (nail at least one; ideally stack them):

• Same team: people and role structure stay put;
• Fixed iteration: same iteration length and cadence;
• Fixed demand scope: same backlog pool or an equivalent complexity subset.

Call these anti-patterns out loud: cross-team % comparisons; stitching an “efficiency story” across different demand difficulty, schedules, or DoD. When reporting up, show the comparison design (what was controlled) before the numbers; management should bounce any “% gain” that lacks that design.

4.2 Prefer percentiles in statistics

For duration metrics such as Lead Time / Cycle Time, default to reporting p50 and p85; means are secondary. Averages are dragged around by a few stuck stories and hide how most work actually flows. Throughput still uses story points completed in the period, or per-capita points in the same iteration.

4.3 The five steps

Step	What to do	Output
1. Fix evaluation groups	Lock the two large groups / pairs and roles; avoid swapping people mid-window	Evaluation protocol: who, which window
2. Familiar backlog + classic estimation	A backlog both groups know well; estimate with classic (non-AI) programming only	Backlog baseline (total points and distribution)
3. Expert judgment for pre-AI	Backfill throughput, segment Cycle Times, etc. under locked definitions	Pre-AI baseline table (with p50/p85)
4. Expert judgment for post-AI	Same definitions and backlog assumptions for post-AI values	Post-AI comparison table
5. Analyze change	Relative change, confidence, footnotes for confounders	Reportable before/after pack

Classic estimation unifies the complexity yardstick first: AI changes the felt cost of “the same story points,” and if estimation itself is rewritten by AI, the baseline drifts—consistent with this site’s warning that Story Points get shaky in the AI era.

Honest boundary: expert judgment is fast diagnosis, not production telemetry. Once the protocol works, collect from Git, board transitions, CI/CD, review tools, and bills to calibrate the judgment tables.

Anti-cheat list (footnotes): giving the model all credit when only the tool changed and process did not; windows so short they only cover the honeymoon; counting learning/standards work as “slower” without a footnote; quietly relaxing DoD then celebrating Cycle Time drops.

5. Starter metric set

On top of comparison discipline and percentiles, start with four families (trim as needed). Each still uses the metric-card format.

5.1 Team throughput (efficiency)

• Definition: total story points completed in a fixed period (same iteration length); or per-capita points in that iteration.
• Example formulas:
  • $Throughput_{iter} = \sum$ story points Done and in scope for the iteration
  • $Throughput_{per\ capita} = Throughput_{iter} / N$ ($N$ = evaluation group size)
• Comparison: same team, same iteration shape, same classic estimation yardstick, pre- vs post-AI.
• Data: board Done lists, iteration completion records; points frozen by the evaluation protocol.
• Error: scope creep, Done drift, cross-iteration leftovers—freeze exclusions.

5.2 Multi-segment Cycle Time (efficiency, by story)

• Method: freeze the start timestamp definition (e.g. entered In Progress); combine ends at important SDLC nodes into several Cycle Times—e.g. In Progress→Ready for Review, Review start→merge, merge→production.
• Calculation: duration per story, then p50 / p85 over the story set; compare before/after.
• Data: board transitions, PR open/merge times, successful CI deploy times.
• Value: answers which segment AI actually sped up and where the bottleneck moved—more actionable than a vague “overall 200%.”
• Error: stale board states, batching, external waits—cards need a “pause the clock” rule.

5.3 Quality and stability (DORA-aligned)

Prefer Change Failure Rate and MTTR / Time to Restore (lightly adapted). Report speed with quality: rising throughput with worsening failure rate must not be sold as one-dimensional success. Data from release logs, incidents, rollback/hotfix tags; define “failed change” before counting.

5.4 Token / incremental AI cost (cost)

AI adds spend vs baseline: tokens/API, seat subscriptions, private inference, etc.

• Definition: total incremental AI-related cost in the window (vs “no AI” or pre-pilot bills).
• Coarse efficiency: e.g. incremental cost / throughput in period, or incremental cost per unit p50 Cycle Time improvement—good enough for ROI talk; accounting precision can wait.
• Data: cloud bills, gateway usage, seat licenses.
• Decision signal: cost up with no clear gain in throughput or stability is a stop-loss / practice-change signal—not an embarrassment to hide.

Prefer objective, automatable sources: bills and pipeline timestamps usually beat surveys for “how much faster”; surveys explain why, and should not be the sole main evidence of gains.

6. Close: co-design the ruler before the percentage

An action chain you can run:

1. Run the terminology session; post the terminology map.
2. Co-design in 2 large groups × 2 pairs; TLs funnel to 3–6 team metrics with written formulas.
3. Sign the comparison protocol: same team, fixed iteration, fixed demand scope; durations default to p50/p85.
4. Run one five-step experiential evaluation; build a reportable before/after pack (design first, then numbers).
5. Calibrate with automated data; place quality and token cost beside speed.

Closing line: comparability before percentages; co-designed definitions before reported numbers. Fuzzy efficiency stories manufacture anxiety; comparable measurement language manufactures improvement.

Related reading

• Developer Productivity in the AI4SE Era: Why Traditional Metrics Are Failing
• From DORA to DevEx: A Panorama of AI4SE Measurement Frameworks
• DORA 2025: AI Is an Amplifier, Not a Silver Bullet
• An AI4SE Workshop Is Not a Training Class: Staffing, Cadence, and Funnel Convergence

References

• Forsgren, Humble, Kim. Accelerate: The Science of Lean Software and DevOps.
• DORA. State of AI-assisted Software Development (2025) / DORA AI Capabilities Model.
• Lead Time and Cycle Time in lean / Lean Software Development (used alongside DORA’s narrower “Lead Time for Changes”).