What not to measure

The most important discipline in this lesson: don't measure AI initiatives by model accuracy, number of users, or number of prompts, none of those are financial outcomes, and all three can look great while EBITDA hasn't moved at all.

What to measure instead

Measure instead: revenue retained (did the account-health model actually reduce churn, in the cohorts it touched?), cost removed (did support automation actually cut headcount cost or contractor spend, not just ticket volume?), capacity released (did automation free people for higher-value work, distinct from "we didn't need to hire," which is a real but easy-to-miss form of value), conversion uplift, and EBITDA impact, the number every one of these should ultimately roll up into, tying straight back to Lesson 10.

Where possible, measure with a real treatment vs. control split, did the accounts that got the new account-health scoring actually churn less than a comparable group that didn't, rather than just observing that overall churn improved sometime after launch, which could have several other causes entirely.

Treatment vs. control: comparing an outcome between a group that received an intervention and a comparable group that didn't, so the intervention's actual effect can be isolated from everything else changing at the same time.
Illustrative quarterly logo churn, treatment vs. control cohort of comparable accounts.

Checkpoint

  • Measure by financial outcome: (revenue retained, cost removed, capacity released, EBITDA impact), never by model or usage metrics alone.
  • A treatment-vs-control comparison: where possible, isolates the initiative’s real effect from everything else moving at the same time.

If anything here still feels unclear, ask before moving to Lesson 35.