Guides
Part of Model innovation: what to keep and what to drop
Which model innovation metrics separate real progress from noisy cohorts
Model innovation metrics for a change in progress: leading indicators on new cohorts, the mix problem that hides them, and what a transition period cannot prove.
Measuring a business model change is harder than measuring anything else a company does, because the thing being measured is also changing what gets measured. Most transition reporting is therefore confident and wrong.
These are the measures that survive the problem, and the honest limits on them. All figures are invented placeholders.
What to take away
- Read new cohorts, not the whole book. The book is a mixture of two models and moves for reasons that have nothing to do with the change.
- Every transition measure needs a stated period and a stated population, written before the change shipped.
- Some of what you want to know cannot be established from a live transition. Say which parts, rather than producing a number for them.
Read cohorts, not totals
The total is a blend. In month three of a transition, revenue includes old-structure customers, new-structure customers, customers who moved, and customers who left because they were asked to move. A total that rises tells you nothing about which of those four is responsible.
One Total, Four Groups
Blended total
- Who is counted
- Old, new, moved, left
- Reported from
- Blended month three
- Own size
- Hidden in blend
- Own realized price
- Hidden in blend
- Own retention
- Hidden in blend
- What a rise proves
- Nothing about cause
New-structure cohort
- Who is counted
- New-structure only
- Reported from
- First day separately
- Own size
- Reported
- Own realized price
- Reported
- Own retention
- Reported
- What a rise proves
- Cohort progress
Report the new-structure cohort separately from the first day. Give it its own size, realized price, and retention.
Group records by when they arose, not when reported. That is ordinary managerial accounting practice, set out in MIT's financial and managerial accounting course. It will be small and noisy for a while. That is honest. A large confident number built on a blend is not.
The measures worth pre-registering
| Measure | Why it moves first | What it cannot say |
|---|---|---|
| Realized price on newly signed accounts | A structure change lands here before anywhere else | Whether existing customers will accept it |
| Discount distribution, not its average | The interesting movement is in the tail | Whether the discounts were justified |
| Win rate on new-structure quotes | Buyer reaction is fast | Whether the losses were price or timing |
| Support contacts about billing | The clearest signal that a structure is confusing | How many people were confused and said nothing |
| Retention within the moved group | Whether the migration itself caused loss | Anything at all in the first quarter |
Write each with a direction and a period before the change ships. Measures chosen afterward are chosen, consciously or not, for what they show.
Pre-Register These Five Measures
- Realized price on newly signed accounts
- Discount distribution, not its average
- Win rate on new-structure quotes
- Support contacts about billing
- Retention within the moved group
The arithmetic to have ready
Two calculations turn a transition argument into a decision.
If the change is toward recurring revenue, compute what the loss rate implies. A 3% monthly churn leaves 69% of a cohort after twelve months and an average life of 33 months. On a book of 1,000,000, an annual gross churn of 20% requires 200,000 of new revenue a year before any growth exists at all.
If the change is toward volume at a lower price, compute the required increase. At 50% gross margin, a 10% discount needs 25% more units to hold gross profit, and a 20% discount needs 67% more.
At 30% margin, 15% off needs double. Put those numbers in the room before the structure is agreed, not after. Name the layer being changed. That is the discipline in pricing architecture.
What a live transition cannot prove
It is not a controlled experiment. Your customer mix moved, the season moved, sales effort moved, and a competitor did something you did not observe. Any of those alone can produce the result you were hoping for.
Name confounders in writing at the start, accept a directional answer, and record what remains unknown. A transition report claiming precision it cannot have is worse than one saying the direction looks right and here is what would change our minds.
Treating an observed pattern as a hypothesis, not a finding, is the standard caution in MIT's data mining course.
Two things to watch that are not metrics
How the sales team describes the new structure. If three people describe it three ways, the structure is too complicated and no measure will tell you that.
What the exception list is doing. A structure that accumulates exceptions faster than it accumulates customers has already failed, whatever the revenue line says.
What the numbers imply about the shape of the business belongs in business model types, the cost side in unit economics, and the decision about whether the change was the right kind at all in model innovation.
Common questions
How long before any of this is readable?
A full sales cycle plus one renewal cohort, at minimum. Reading a structure change in the first month measures how quickly the sales team adapted, which is a real thing and a different question.
Our new cohort is tiny. Should we report it?
Report the count rather than the rate. A conversion rate on nineteen deals is a decoration, and presenting it as a percentage invites a decision it cannot support.
Should we tell the board the result is uncertain?
Yes, with the confounders named. A board told the truth about uncertainty at month three is a board that will let you continue at month nine. One given false precision will not.
What if the measures conflict?
Then say which one you pre-registered as the primary and read that, and treat the others as context. Choosing the winner afterward is how a company convinces itself of something it did not learn.
