The thing that kills an automation is visible before it starts.

In 1954 Paul Meehl compared trained clinicians making judgements against simple statistical rules doing the same job, and found the rules won.¹ The finding has been replicated for seventy years. It has also been resisted for seventy years, and the resistance is the interesting part: it is not that the professionals doubt the evidence. It is that being outperformed by a short list feels like an insult, and the list is free, which somehow makes it worse.

The same result turns up in surgery, where a checklist that any nurse can read outperforms the memory of people who have done the operation four hundred times.² Nobody involved got worse at surgery. They just stopped relying on remembering.

Expect to feel some of that resistance reading what follows. The questions are simple and you know all the answers already, which is exactly what makes them worth writing down.

Eight questions

Answer each honestly. There is no scoring page, no email box, and nothing to submit. If you want to score it, give yourself a point for each clear yes and read the section after.

1. Can you say, in one sentence, what finishes?Not “improve operations.” A thing that either happened or did not: the invoice reached the customer, the reminder went, the stock figure updated. If you cannot name the finished thing, nobody can build it and nobody can tell you later whether it worked.
2. Does the information already exist as data, or only on paper and in heads? Software cannot read a notebook or a memory. If the answer is paper, the first project is not automation, it is writing things down, and that is a smaller and cheaper project that somebody should tell you about.
3. Is there one place that is right when two disagree? If the sheet and the billing system disagree about yesterday, is there an agreed answer to which one wins? Where there is no agreed answer, automation does not resolve the disagreement. It industrialises it.
4. Will you switch the old way off? On a date. In writing. If not, you are running a demonstration, and it will quietly end in week three.
5. Is there one person whose week gets worse if it stops? Not you, and not the vendor. If nobody feels it break, nobody will notice it break.
6. Do you know what one outcome is worth in rupees? One recovered booking, one avoided error, one hour returned. Every decision after this is a comparison against that number, and without it you will be comparing against a feeling.
7. Does it still work if the price of the tool doubles? Rate cards move, and the one under most of these builds moved twice in eighteen months. A flow that only works at today’s price belongs to somebody else’s pricing committee.
8. Have you run the four numbers? Fully loaded monthly cost of the person, the fraction of the role it really takes, monthly running cost, build cost. If the payback is past about a year, the honest answer is often a person with a better tool.

Notice that seven of the eight are about your business and one is about the software. That ratio is the finding, and it is why buying a different tool so rarely changes the outcome.

Reading your own answers

This part is judgement, not a scoring key, and I am not going to dress it up as one.

• Question 2 answered “paper” means stop. Nothing further on the list matters yet. The project is to get the information into a form that exists, and it costs a fraction of what you were about to spend.
• Question 4 answered “no” means stop. Everything else can be perfect and it will still end in week three.
• Questions 6 and 8 unanswered means you are not ready to hear a quote, because you have nothing to compare it against.
• Mostly clear yeses means the risk in your project is ordinary technical risk, which is the kind worth paying somebody to take.

What would make this an instrument, and why it isn’t one yet

There is a version of this paper that says “businesses scoring below five never shipped.” It would be a better paper to read and a much better thing to put in a sales deck. We are not writing it, because we have not done the work that would make it true.

Doing that work means, specifically:

1. Scoring a set of finished projects on these eight questions as they stood at the start, from records made at the time, not from memory after the fact. Scoring from memory guarantees the answer.
2. Recording each project’s real outcome on a definition fixed before scoring — still running at twelve months, say.
3. Having the scoring done by somebody who does not know the outcomes.
4. Reporting the correlation whatever it is, including the case where these eight questions turn out to predict nothing, and reporting how many projects were in the set so you can judge the weight of it.

Until that is published, treat the list as a considered checklist from people who have watched this go wrong, which is a useful thing but is not the same as a tested one. When we do publish it, hold us to the four steps above — they are on this page so that you can.