Most A/B Tests Are a Way to Look Busy
David Selva · · 3 min read
Email & SMS Marketing, Self-Storage, Message Testing
If you can't say what you would do differently depending on the result, you aren't running a test. You're collecting trivia.
Ask somebody what they're testing and you'll usually hear: subject line A against subject line B.
Ask what they'll do if B wins by four percent and the conversation stops. That pause is the tell. If the result doesn't change a decision, it isn't a test. It's trivia with a spreadsheet attached.
The question that kills most tests
Before you set anything up, finish this sentence: if version B wins, I will ______.
If the honest ending is "use version B in this one email," the test is worth close to nothing. You will have spent two weeks learning a fact about a message you'll never send again.
If the ending is "change how we word every rate increase notice from now on," you have a real test, because the answer transfers to everything that comes after it.
What's worth testing at a storage facility
Self-storage is a good place to do this properly, because the volume is there and the decisions repeat. The same handful of messages fire off the same triggers, hundreds of times a month.
The three that pay
- The abandoned inquiry. Somebody priced up a 10x10 and didn't reserve. What you send an hour later is the highest-value message you own.
- The rate increase notice. This decides whether a tenant shrugs or starts pricing up the competitor down the road. Wording and notice period are both live variables, and you send it constantly.
- The past-due sequence. Tone here has a direct dollar value, and most facilities have never tried an alternative to whatever somebody wrote once in 2019.
Not worth it: emoji in subject lines, shifting send time by fifteen minutes, or picking between two nearly identical headlines. Those get tested because they're easy to set up, not because anybody would act on the answer.
Sample size, without the lecture
You don't need statistics training for this. You need one rule: if a difference wouldn't survive being wrong, don't act on it.
Forty people saw each version and one converted better — that's noise. Decide on the back of a handful of conversions and you'll be changing your messaging back next quarter when the noise falls the other way.
Run it long enough that the result is boring and obvious, or don't run it at all.
The test I'd run first
The abandoned inquiry, and not on wording. On timing.
An hour versus the following morning is a real question with an answer that applies to every inquiry you receive from now on. That's the shape a good test has: one variable, a result you'd act on permanently, and enough volume behind it to believe.
Pick one test you'd act on
Write down the message you send most often. Underneath it, write what you would change permanently depending on how a test came out.
If you can't finish that second line, choose a different message. That is the entire filter, and it will spare you most of what passes for optimization.