The One Mistake Rule

Epistemic Status: The Bed of Procrustes

Related to (Said Achmiz at Less Wrong): The Real Rules Have No Exceptions

If a model gives a definitely wrong answer anywhere, it is useless everywhere. 

This principle is doubtless ancient, and has doubtless gone by many names with many different formulations.

All models are wrong. That does not make them useless. What makes them useless is when they are giving answers that you know are definitely wrong. You need to fix that, if only by having the model more often spit out “I don’t know.”

As an example of saying “I don’t know” that I’m taking from the comments, if you want to use Newtonian Physics, you need to be aware that it will give wrong answers in relativistic situations, and therefore slightly wrong answers in other places, and introduce the relevant error bars.

Of course, a wrong prediction of what is probably going to happen is not definitely wrong, in this sense. An obviously wrong probability is definitely wrong no matter the outcome.

The origin of this particular version of this principle was when me and a partner were, as part of an ongoing campaign of wagering, attempting to model the outcomes of sporting events.

He is the expert on sports. I am the expert on creating models and banging on databases and spreadsheets. My specialty was assuming the most liquid sports betting market odds were mostly accurate, and extrapolating what that implied elsewhere.

First we would talk and he would explain how things worked. Then I would look at the data lots of different ways and create a spreadsheet that modeled things. Then, he would vary the inputs to that spreadsheet until he got it to give him a wrong answer, or at least one that seemed wrong to him.

Then he’d point out the wrong answer and explain why it was definitely wrong. I could either argue that the answer was right and change his mind, or I could accept that it was wrong and go back and fix the model. Then the cycle repeated until he couldn’t find a wrong answer.

Until this cycle stopped, we did not use the new model for anything at all, anywhere, no matter what. If a new wrong answer was found, we stopped using the model in question until we resolved the problem.

Two big reasons:

If we did use the model, even if it was only wrong in this one place, then the one place it was wrong would be the one place we would disagree with the market. Fools and their money would be soon parted.

Also, if the model was obviously wrong here, there’s no reason to trust anything else the model says, either. Fix your model.

This included tail risk style events that were extremely unlikely. If you can’t predict the probability of such events in a reasonable way, even if those outliers won’t somehow bankrupt you directly, you’re going to get the overall distributions wrong.

This also includes the change in predictions between different states of the world. If your model predictably doesn’t agree with itself over time, or changes its answer based on things that can’t plausibly matter much, then it’s wrong. Period. Fix it.

You should be deeply embarrassed if your model outputs an obviously wrong or obviously time-inconsistent answer even in a hypothetical situation. You should be even more embarrassed if it gives such an answer to the actual situation.

The cycle isn’t bad. It’s good. It’s an excellent way to improve your model: Build one, show it to someone, they point out a mistake, you figure out how it happened and fix it, repeat. And in the meantime, you can still use the model’s answers to help supplement your intuitions, as a sanity check or very rough approximation, or as a jumping off point. But until the cycle is over, don’t pretend you have anything more than that.



This entry was posted in Rationality. Bookmark the permalink.

14 Responses to The One Mistake Rule

  1. This proves too much, as stated; it implies that Newtonian Physics is useless everywhere because it gives definitely wrong answers about light.

    Definitely wrong models that still narrow uncertainty a lot and have pretty-good predictions in important domains are still good rules of thumb in those domains. They are, however, false or incomplete as explanations.

    • TheZvi says:

      Yeah, I wrote this too quickly and was a little imprecise. You’re OK in that situation because you know exactly where you fail and can define that part as outside the model.

    • TheZvi says:

      Added a paragraph early to clarify how I handle exactly that situation.

      A definitely wrong model that narrows uncertainty that you treat as exactly correct is really bad.

      A definitely wrong model that you then modify to include uncertainty around its answers is no longer wrong!

  2. >> If a model gives a definitely wrong answer anywhere, it is useless everywhere.

    Except if it needs to be used right now to make important decisions and it’s the best model we have. See:

  3. Pingback: Seemingly Popular Covid-19 Model is Obvious Nonsense | Don't Worry About the Vase

  4. Pingback: Evaluating Predictions in Hindsight | Don't Worry About the Vase

  5. Quixote says:

    A lot of modeling use cases are asymmetrical. For example, if you are modeling the question “will a bridge collapse?” It is somewhat more costly if the model tells you that a bridge might collapse and you add additional support that wasn’t needed, but it is catastrophic if a bridge collapses and an entire train crashes into the river. If the true answer to ‘Will the bridge collapse?” is “ennh, maybe, but probably not” you wan the model to say, “this bridge will certainly collapse; add more support.”

    There are a lot of use cases like this, in fact I think cases like spread betting where a model needs to be right in both directions are actually the anomalous case for real world mode use.

    • TheZvi says:

      I don’t think you want that. I think you want “It might collapse, add more support.” If every case of might is treated as definitely will some bad things happen. But certainly it’s the smaller mistake to make, and it’s fine to say “might” in some places where the right answer is “won’t” provided you know you might be doing that.

      • Lawrence Chan says:

        I’m interested in precedent for this sentiment, which I’ve acquired primarily by hanging out with rationalists. The only thing that comes to mind is the common law doctrine of “falsus in uno, falsus in omnibus” – a witness who testifies falsely about one matter is not at all credible to testify about any other matter.

      • Lawrence Chan says:

        Whoops, meant to reply to the main thread but accidentally replied here.

        You write that “This principle is doubtless ancient, and has doubtless gone by many names with many different formulations.” – do you know of any other names/formulations?

  6. Pingback: My Covid-19 Thinking: 4/17 | Don't Worry About the Vase

  7. Pingback: 2020 Election: Prediction Markets versus Polling/Modeling Assessment and Postmortem | Don't Worry About the Vase

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s