The Disaster of Old Metrics

Old MetricWhy It Fails Under AI4SE
Lines of codeAgents can generate huge volumes of code — that doesn’t mean value
Story pointsAgents change completion speed — relative estimation breaks down
PR countEncourages splitting into tiny PRs to pad numbers — ignores review burden
Commit frequencyAgents auto-commit — no longer a signal of human initiative

Adapting the SPACE Framework for AI4SE

DimensionWhat AI4SE Should Focus On
SatisfactionDeveloper satisfaction with agent collaboration
PerformanceEnd-to-end outcomes (not coding speed)
ActivityVolume of HITL review + decision participation
CommunicationSpec quality + review feedback frequency
EfficiencyEnd-to-end flow efficiency from spec to merge

The Core Shift

From “how much was produced” to “how much was verified”

  • Don’t look at how much code the agent generated — look at how much passed independent review
  • Don’t look at how fast PRs merged — look at how many regressions followed
  • Don’t look at time spent using tools — look at the closed-loop cycle time from spec to delivery

References

  • SPACE Framework (Nicole Forsgren et al.)
  • DevEx: Developer Experience Framework
  • DORA 2025 AI Capabilities Model