Contrary to popular belief, A/B testing was not invented for design or marketing. “A/B test” is just a colloquial way of saying “controlled experiment”, and those have been around for centuries.
You may be thinking, what does that have to do with my ability to conduct one?
Well, the key to any successful (read: valid) controlled experiment is the scientific method.
Unfortunately, A/B tests don’t always follow the scientific method. A true controlled experiment must:
- Define an explicit hypothesis
- Maintain fundamental control over the test environment
- Define and separate dependent and independent variables
- Identify and eliminate confounding variables
- Be reproducible
At Codal, our UX design teams recently experimented with a number of ways to apply atomic design towards controlled A/B testing.
Based on concepts borrowed from chemistry, atomic design describes the complex (and simple) relationships between micro- and macro-design elements.
In this article, I will discuss this unexplored use case for atomic design, and show you how to use it for the most scientifically driven A/B tests around.
1. Use Atomic Design to Define an Element-Level Hypothesis
Atomic Design makes it astoundingly easy to get very, very specific with your hypotheses. If you’re unfamiliar with the way in which atomic design is structured, you can think of it like this.
Above: atomic design representation by Brad Frost
Think of any site you visit frequently, or even one you’ve worked on. There are tiny elements, and there are massive elements. Often, tiny elements fit into (and make up) massive elements. In fact, this is true in most cases.
In conjunction with the above image, let’s define the variable types used in A/B tests.
- Atoms: these are the absolute smallest elements on a page. Think icons, buttons, text fields, etc.
- Molecules: these are complete elements; they consist of multiple atoms. Think search bars, option toggles, drop-down menus, etc.
- Organisms: these are complete structures; they consist of multiple molecules. Think navigation bars, blog grids, sidebars etc.
- Templates: these are essentially wireframes; collections of implemented organisms.
- Pages: these are pixel-perfect, themed implementations of templates.
When constructing a testable hypothesis, the biggest challenge facing an A/B tester is simply to craft an explicit hypothesis. The explicitness of a hypothesis depends entirely on the size of the variable being tested.
Here’s what a bad hypothesis looks like for a hypothetical non-profit site:
“Adding more information for visitors on the landing page will increase the number of donors.”
Now, let’s say you see a conversion rate increase of 10% at a confidence interval of 95%.
The hypothesis, however, is poorly defined, non explicit, and difficult to prove or reproduce. What specific information helped increase the number of donors? The increased conversion rate can hardly be attributed to one reproducible element.
The issue is very simple: a lack of explicit variables (independent, dependent, confounding, etc) in their hypothesis. As a result, the test was ambiguous, and thus, unscientific and far from reproducible.
So, how do you use atomic methodology to pick appropriate variables?
2. Atomic Design Breeds Well-defined Variables
Due to the nature of atomic design, designers can focus on the effects of components of all sizes and functions. Whether they’re small, or large, you have strong (and documented) control over every variable that you test, because atomic design categorizes site elements by their size.
The variable types listed above do not share identical levels of impact on the results of A/B tests, nor are they equally weighted in terms of scientific accuracy. When conducting an A/B test, the smaller the independent variable, the more scientific the test due to less dependencies.
There’s a simple explanation: larger variable types, like organisms, consist of multiple smaller variables—this means that you don’t know which part of the organism actually had an effect on the observed data.
Use this knowledge as you develop variables for your A/B test hypotheses.
Always break your independent and dependent variables down to their smallest sizes. For the most part, your dependent variables will be conversions, so it’s hard to break them down. But when it comes to your independent variables, the smaller the better.
There are many ways to illustrate this concept, but Codal’s UX designers like to use the navigation bar to explain the scientific validity of each variable type.
Above: Call Potential nav bar – one of Codal’s projects.
If a UX designer were to conduct an A/B test on the above navigation bar, they may be tempted to make a blanket statement like this: increasing the visibility of the search bar will increase its usage.
The subsequent A/B test might consist of switching out the search bar for a different style or color, and measuring the impact.
But there is a danger in doing so.
Which specific aspect of the search is being measured for impact? It’d be difficult to answer that question. Is it the color of the bar? The placement of the text? The search prompt? It’s too easy to make a blanket change to an element, without doing so in an incremental fashion.
A better way to A/B test the search bar would be to follow this process:
1. Identify the conversion path you want to test
2. Figure out which elements are related to it. Start with larger elements, and work your way down to smaller and smaller components until you have a list of different “atoms” you’d like to test
3. Test each atom individually, and systematically to eliminate confounding variables
4. Make sure nothing outside of the selected conversion path changes while you conduct these tests – that is the mark of a truly scientific study
So, in the example above, rather than change the entire search bar, one ought to make a list of sub-elements (atoms) worth investigating, and test them individually. Such a list might look like this:
1. The search icon (color)
2. The text field (size; color)
3. Placement (center of header)
You could make an endless list if you tried. Regardless of length, keep in mind that each change to any particular element (color, size, etc) must be tested individually.
The point is to use the list above to make a good educated guess with your hypothesis, Sometimes, you’ll have multiple hypotheses. Often, the results from one test will spur a new hypothesis for another.
Such is the nature of science.