[measurement] | 2026-08-05 | 10 min

# Three Anchors for Efficiency Comparison: Comparability, Separability, Honesty

After AI enters software engineering, efficiency comparisons often become apples-to-oranges. This article proposes three anchors—comparability (fixed backlog / team / window), separability (freeze the complexity baseline), and honesty (expectation, not fake measured ROI)—plus reading rules for throughput, cycle time (speed framing), and paired aggregation, with common anti-patterns.

[dev-productivity][measurement][cycle-time][throughput][ai4se]

When people ask “how much faster are you with AI?”, they often expect an answer like swapping in a faster machine: same pipeline, higher output. Reality is messier. AI changes how work is organized—how finely demand is split, where coding ends and verification begins, what humans vs machines own. So the real question is often not “did we do the same thing faster?”, but “is the faster thing still the same thing?”

Traditional productivity measures—LOC, function points, story points per person-day—assume that work-unit definitions and completion paths stay roughly stable across teams and over time. When AI becomes an active participant in the delivery chain, that assumption wobbles. Spec-Driven Development pulls specification forward and drives generation and verification from structured specs; the grain and flow of its delivery units do not naturally align with classic story delivery. If you compare on new-unit counts or generated code volume, the two sides may not even share a measurement space—classic apples-to-oranges.

This site already has a workshop script for co-designing efficiency metrics—how to run the meeting, what to start measuring. This article covers the other half: which design decisions decide whether a before/after number is believable. Every “should we compare this way?” choice can be pressed back into three anchors—comparability, separability, honesty.

1. Why measurement is hard: three structural difficulties

First, measurement spaces may not align. Before counted by story, After by a finer delivery slice—distribution shapes are not comparable. Activity metrics (lines generated, commits, PR counts) distort even more easily in the AI era; see why traditional developer-productivity metrics fail.

Second, outcomes depend heavily on process maturity. The same spec-driven flow can differ by multiples between “just getting started” and “high consensus.” Without naming that precondition, you cannot tell whether a number measures capability ceiling or a snapshot on the learning curve.

Third, confounders abound. Different backlog complexity, team parallelism, and iteration window length all move throughput and completion probability. If variables are not pinned, differences cannot be attributed to the method.

The three anchors face these three walls: fix external conditions for a fair baseline, freeze a complexity yardstick so spaces align, and locate results as a committable expectation under high consensus—a directional, auditable conclusion that later telemetry can validate, not a magic percentage from one meeting.

2. Comparability: pin three variables

Between Before and After, the only thing that should change is the development method. At minimum, lock three external conditions:

Fixed condition	Convention	Why it cannot drift
Product Backlog	Same item list	Different complexity mix → capacity gaps cannot be blamed on method
Team size	Fixed headcount and role mix	Parallelism drives throughput
Iteration window	Same length (e.g. two weeks)	Longer windows raise completion probability by themselves

Some alternatives try to freeze “code complexity” or function points. The problem: those quantities may themselves shift systematically under a spec-driven mode—generated structure need not match handwritten form—and introduce new incomparability. Fixing three external conditions is the cleanest contrast design you can usually achieve in a workshop or expert-judgment setting.

Reporting discipline is simple: show what you controlled before you show the number. Efficiency % without a contrast design should be sent back. This also echoes DORA’s long-standing reminder—throughput belongs with stability; a lone speed figure is easy to game. See DORA’s software delivery performance metrics and our DORA / SPACE / DevEx panorama.

3. Separability: freeze the complexity baseline

This is the most critical—and most misunderstood—decision.

In the sizing phase, groups independently estimate story points on the same backlog and freeze the points within the group. Before and After share that scale. After must not re-estimate because the delivery style changed.

Why? Story points aim at inherent demand complexity—they should stay as independent of implementation method as possible. If After re-points, you cannot tell whether “gains” came from a better method or from a shifted estimating lens. Freezing SP forcibly separates “how complex is this?” from “how fast did we do it?”

It is not a perfect separation. Spec-driven practice may make some patterned work “actually simpler.” Allowing re-estimation costs more: both sides lose a shared complexity baseline, and the gain figure forever hangs on “faster, or redefined what fast means?” Under engineering constraints, freezing the baseline is the best available approximation of separability—aligned with the “classic-mode sizing” discipline in the sister article.

4. Honesty: expectation, not fake measured ROI

Expert consensus framing and experiential assessment produce a committable expectation, not ROI already proven in the delivery system. That distinction must run through every external statement.

Both extremes are dangerous. One end is “optimistic probe”—numbers as best-case possibilities that management cannot plan against. The other is “landed ROI”—selling estimates as measured fact. Over-promise; when later telemetry disagrees, you damage both the method’s credibility and the team’s trust.

A committable expectation means: given high consensus on process, tools, and role split, this magnitude of gain is a reasonable target—neither so conservative it is useless, nor so aggressive it cannot be met. Later, validate with board transitions, Git, CI/CD, and similar telemetry. If measured results diverge sharply, ask whether adoption lagged or the consensus assumption never held. Expectation → measurement → validation is the posture a measurement system should have.

5. The metric model: throughput and cycle time

5.1 Throughput

A field-friendly primary report is $\Sigma$ story points completable in the observation iteration. With team size and window fixed, person-days are roughly constant, so $\Sigma$ SP is monotonically equivalent to points per person-day and can be converted when needed. Prefer $\Sigma$ SP on the whiteboard: people see “how many points finished in the iteration” at a glance.

Completion must align with the cycle-time end state. Count a story toward throughput only when it reaches the agreed done state (e.g. testing completed, or a semantic equivalent)—so you never credit capacity for work that has not finished verification.

5.2 Cycle Time

Report at least two variants (names can be localized):

Name	Start	End
CT1 (coding-side)	Dev picks up / first enters development	Last commit linked to that story
CT2 (delivery-side)	Same	Testing completed (or agreed Done)

A production-release variant is often dominated by environments and rollout policy. In expert framing those factors add more noise than signal; leave them for later telemetry rather than force them into the first contrast.

Lock statistical grain to the story. Whether Before or After, compute cycle-time percentiles on stories. If After organizes delivery in finer change slices, fold back to story grain before percentile math—otherwise Before-by-story and After-by-slice are not comparable.

Prefer percentiles over means. For duration metrics, percentiles resist stuck outliers. When workshop estimates are coarse, P75 often better reflects “the upper edge most stories can hit”; reporting P50 alongside is fine. The sister article defaults to p50 and p85—same principle: read real flow, not a mean dragged by extremes.

5.3 Report cycle-time “% gain” in speed framing

Throughput gain uses Before as the denominator—intuitive:

\Delta TP = \frac{TP^{A} - TP^{B}}{TP^{B}} \times 100\%

If you still use duration reduction $(B-A)/B$ but call it “efficiency gain,” you systematically understate capability change. Four days to two is a 50% shorter duration; work completable in the same window is closer to double. Invert cycle time to speed $\mathrm{Speed}=1/\mathrm{CT}$ , then compute lift relative to Before, so CT and throughput share the same “capability got stronger” semantic space:

\Delta CT^{\text{gain}} = \frac{CT^{B} - CT^{A}}{CT^{A}} = \left(\frac{CT^{B}}{CT^{A}} - 1\right) \times 100\%

One pair of numbers, three namings—do not mix them:

Framing	Formula	4 days → 2 days
Duration reduction (vs Before)	$(B-A)/B$	50%
% gain (speed, recommended primary)	$(B-A)/A$	100%
Speed multiple	$B/A$	2×

Prefer saying “X% gain / Y× faster,” and add “Z% shorter duration” only when needed. Same fact, different denominators.

Illustrative single-group numbers (not measured results):

Metric	Before	After	Gain Δ (primary)
Throughput $\Sigma$ SP	20	30	+50%
CT1 P75 (days)	8	4	+100% (2×; 50% shorter)
CT2 P75 (days)	10	6	+66.7% (40% shorter)

6. How to read the conclusion: paired Δ, then aggregate

Compute relative change within each group first. The same group estimates Before and After, sharing judgment of the backlog and team capability; within-group $\Delta$ is more stable than absolute levels—the basic requirement of a paired design.

Across groups, use median + IQR, not a mean that fakes precision. With small $n$ , means are hypersensitive to extremes; expert judgment also carries well-documented optimism bias. Median plus interquartile range says what management needs: “where most consensus sits, and how wide the disagreement is.”

Three “don’ts”:

Don’t average all Befores, average all Afters, then compute one grand Δ—that breaks pairing and inflates bias.
Don’t pool stories across groups and recompute a global percentile—SP scales may differ; the mixed percentile has unclear meaning.
Don’t treat “50% shorter” and “100% gain” as two facts—same pair of numbers; keep framings distinct when speaking.

Default: do not drop groups because numbers look bad. Only when a facilitator confirms fixed conditions were broken (wrong backlog, changed team size, After re-pointed, wrong grain) mark the group disqualified with a reason; keep the detail row. That is a hard honesty rule.

7. Anti-pattern cheat sheet

Anti-pattern	Anchor broken	Why it hurts
Re-estimate SP After	Separability	“Re-pointing” disguised as gain; baseline polluted
Different statistical grain Before vs After	Comparability	Distribution shapes not comparable
Team or window inconsistent	Comparability	Difference cannot be attributed to method
Average levels first, then one grand Δ	Separability	Breaks pairing; inflates bias
Single % gain with no disagreement band	Honesty	Fake precision
Call duration reduction “gain” without naming the denominator	Honesty	Misleads management on magnitude
Write framed estimates as landed measured ROI	Honesty	Over-promise; burns trust

Close

Efficiency comparison is not “throw a percentage, then invent a story.” Reverse the order: pin comparable conditions, separate complexity from speed, then honestly label whether the number is expectation or measurement. The three anchors are criteria, not decoration—whenever you hesitate about how to report, press the choice against them.

Line to keep: Comparability before percentages; separability guards the baseline; honesty guards trust. Fuzzy efficiency narratives manufacture anxiety; comparable measurement language manufactures improvement.

To co-design a team metric set and starter recipes in a meeting, continue with the sister piece: Co-Design AI4SE Efficiency Metrics with Your Team.

References

DORA. Software delivery performance metrics.
Lead Time and Cycle Time in lean / Lean Software Development (used alongside DORA’s narrower “Lead Time for Changes”).