The Law of Small Numbers
Advertising is a unique mix of art and science. (We previously wrote a deep dive on common pitfalls around measurement in our 5 part Adventures in Significance series.) Phrased in a different manner, advertising is the craft of persuasion measured.
The measurement aspect of advertising can be difficult because statistics can be surprisingly unintuitive. This is where common sense can get people into sticky situations when it comes to data analysis. More importantly,. why it’s paramount to build the muscle around proper statistical rigor.
Here are two phrases that apply directly to advertising. They showcase the importance of having a large enough sample.
- Large Samples are more precise than small samples
This makes intuitive sense. Now for an equivalent statement phrased differently.
- Small samples yield extreme results more often than large samples do
These appear like different statements and yet ultimately they mean the same thing. There is a famous case detailed in *Thinking Fast and Slow that caused the Gates Foundation to even read the data wrong!
Many researchers have sought the secret of successful education by identifying the most successful schools in the hope of discovering what distinguishes them from others. One of the conclusions of this research is that the most successful schools, on average, are small. In a survey of 1,662 schools in Pennsylvania, for instance, 6 of the top 50 were small, which is an overrepresentation by a factor of 4. These data encouraged the Gates Foundation to make a substantial investment in the creation of small schools, sometimes by splitting large schools into smaller units...Unfortunately, the causal analysis is pointless because the facts are wrong...The truth is that small schools are not better on average; they are simply more variable.
This Law of Small Numbers as Daniel Kahneman named it in Thinking Fast and Slow applies quickly to any analysis in advertising. In short, you can expect more extreme results due to good and bad luck to occur in the ads with less spend than those with enough spend.
Having a keen grasp of when “enough” has occurred is highly important to any type of successful marketing effort. As we discussed in our Adventures in Significance series, A:A testing can be a great exercise to understand how an unknown system shows variance.
*If you are a fan of non-fiction and have not read this book, it is well worth the read or **listen.
**Quick Tip on listening to books, put your tax dollars to work! Libby by Overdrive allows you to borrow ebooks and audiobooks from your local library for “free”. Audible is great but Libby is a fantastic resource.
--
Thanks to Jon Tyson for sharing their work on Unsplash!