Outgrace, home
StrategyHow-to guide

How to test a publishing model before committing capital

Test a new title, format or subscription tier for a fraction of its launch cost. Set the stop line first, count payments rather than interest, and agree the decision in advance.

A new title, a new print format or a subscription tier usually reaches the board as a launch plan, with the cost of paper, staff and marketing already counted. The question of whether readers will pay for it is left until later, when the money has gone. There is a cheaper route. Test the bet before you fund the thing, and keep the test small enough that losing it costs a meeting rather than a write-off.

Before you start

Start by admitting what you do not yet know. The sources available for this guide do not cover reader economics, title or format testing, or subscription pricing, so the method below is editorial judgement built on the general logic of controlled comparison. Take it to your own finance lead and test it against your numbers before you rely on it. Nothing here replaces a sourced study of publishing economics, and we have not invented one.

Before any test is approved, four things should be true:

  • A single decision, and a date on which it will be taken.
  • The most you will spend before that date, written down and approved by someone who will not run the test.
  • A reader action that involves money, such as a first payment, a pre-order or a renewal, rather than a click or a form.
  • A named person who owns the result and has the authority to say no to the launch.

The one body of published work we could cite directly concerns founders, not publishers. An NBER working paper by Bailey, Fehder, Floyd, Hochberg and Lee asks whether instruction helps founders decide when to walk away, and it reports a multi-year, multi-site field experiment with a randomised design (NBER, Working Paper 34755). An NBER Bulletin summarising the work says training could help entrepreneurs recognise the moment to stop putting time and cash into a project. It adds that well-run training might end some young companies earlier while raising results at others. The subjects differ from a magazine team, but the lesson carries over: a good test is allowed to end the project.

  1. Step 1

    Write the bet in one sentence

    Name the reader, the thing they will do and the money that follows from it. 'Readers of the regional property title will pay an annual fee for the monthly data pack' is a bet you can test. 'Let's see whether subscriptions work' is a hope, and a hope has no stop line. What usually goes wrong is that the team writes three bets into one sentence and then cannot say which one failed.

  2. Step 2

    Set the stop line before the first reader arrives

    Decide the figure that means stop and the figure that means go, and write both down with the review date beside them. Teams that set these after seeing early results tend to find reasons the disappointing figure does not count, and the argument then becomes about definitions rather than readers. A test that ends a weak idea in a fortnight has done its job, and the stop line should make that outcome easy to accept.

  3. Step 3

    Pick the cheapest action that involves money

    A pre-order for a single issue, a founding subscription at a limited price for a small number of places, or a short print run sold in advance will tell you more than a survey. A waiting list with no price on it tells you that people are curious, which is useful but different. Many teams count those sign-ups as evidence of demand. Treat them as a marketing result, record them separately, and do not let them carry the decision.

  4. Step 4

    Split the audience and change one thing

    If you want to compare two prices or two formats, show each version to a separate group chosen at random, and change only that one thing. Two prices and two headlines at once make it impossible to say which caused the difference. Keep the page, the offer and the timing identical across the two groups, so that the only difference a reader sees is the one you meant to test.

  5. Step 5

    Size the sample with a specialist

    A small test can reveal a large difference and still miss a modest one. Ask a statistician, or your finance lead, to work out how many readers you need before you launch to see a difference you would act on. If that number is larger than your list, the test is not cheap after all. Choose a different action, or a decision small enough that the list can answer it.

  6. Step 6

    Measure the second payment, not the first

    A first purchase can reflect curiosity, a discount or a friend's recommendation. A renewal, or a second order from the same reader, is closer to the economics you will have to live with. Decide in advance which of these counts and over what period. What usually goes wrong is that the team reports first-month sales with enthusiasm and then discovers, a quarter later, that the readers did not come back.

  7. Step 7

    Release money in stages

    Do not fund the full launch on the strength of one test. Release the next tranche of spend only when the second measurement clears the bar you set in step two, and keep a written record of what was spent at each stage. The record matters as much as the result. It shows the next proposal's sponsors what the last one cost to learn, and it stops a failed idea from returning under a new name.

Businesses in the sample of the Census Bureau's Business Trends and Outlook Survey
1.2 million
Approximate cases in each of the six panels of the same Census Bureau survey
200,000
Interval at which each panel reports over one year, per the Census Bureau
12 weeks

Mistakes that spoil a test

The commonest mistake is to test the wrong thing: a reader's interest in a topic rather than their willingness to pay a stated price. Interest is cheap. Ask for a card number and the answers change. A second mistake is to run the test long enough to be interesting but not long enough to be decisive, so that the result arrives after the budget is spent. The figures above describe a large federal survey, not a publisher, but they show a sensible design choice: a fixed sample, a fixed cycle and a known schedule of reports. A test should borrow that discipline even when it is far smaller.

A third mistake is to let the person who wants the launch also decide whether it has failed. The sponsor is the least neutral reader of the result, and in our experience the review goes better when someone who did not argue for the title reads the figures first. A related error is treating a pilot as the market. A test run to a friendly list of loyal subscribers measures loyalty. Say so in the review, and mark the result as weaker evidence than a test run to strangers who had no reason to be kind.

How much should a test cost?

There is no correct figure, but the ceiling should be set against the cost of being wrong, not against the cost of the full launch. Write the number down before you start, and make the person who owns the result sign it. If the sum would strain the budget for a year, the test is too big, and the first fix is to cut its scope rather than to hope the readers are generous.

Can a survey of readers replace a paid test?

A survey can help you choose what to test, but it should not settle the question. People tend to say they would pay more freely than they do, and in our experience a stated price is not a charged one. The Census Bureau's Business Trends and Outlook Survey also collects what firms expect to happen next to what they are experiencing now, which is a reminder that expectations and actions are separate measures. Use the survey to shape the test, then rely on payments to judge it.

Do I need to randomise the audience?

Not always, but when you compare two versions, randomising who sees which is the simplest way to stop differences in timing or audience from passing for the effect you wanted. A launch in one month set against a launch in the next will mix the offer with the season. If you cannot split the audience, say so plainly in the review and treat the comparison as weaker evidence.

What if the test says stop?

Stop, and record why. The NBER Bulletin's summary suggests that effective training could end some young companies sooner, which is the purpose of a test that works. A publisher that closes a weak title after a short, well-measured trial has used the money well. Keep the notes, because the next proposal will look similar, and the record shows which early signals misled you.

Pick one proposal that is waiting for capital and write its bet in one sentence. Set the stop line and the review date, and ask your finance lead to price the cheapest action that involves a payment. Ask an accountant or lawyer to review the test if it involves subscription billing terms or the tax treatment of digital sales, because those answers depend on your jurisdiction and are not covered here. Then run the test before the board sees the launch budget.

Newsletter

The Briefing

One email a week: the leaders we profiled, the ideas worth stealing, and nothing else.

Weekly, free, one click to leave. Read our privacy policy.