1. 100 Days of UX
  2. A/B Testing

A/B Testing

A/B Testing is an optimization technique for comparing two different versions of a design to find out which gets closer to a specified business goal or metric. It is a way to quickly discover which is better based on an educated guess.

A/B Testing

A/B Testing is Day 1 of 100 Days of UX, an exploratory effort to survey the 100 methods of design outlined in Martin and Hannington's Universal Methods of Design. For 100 consecutive days, I learn one new method a day and write about it.

Before an A/B test is conducted, a business goal is specified. For example, increasing the click-through rate on an ad.

Next, an educated guess is made about which single aspect of the current design can be altered to achieve the goal. For instance, making the ad part of the content as opposed to a banner because users tend to unconsciously skip such ads due to banner blindness.

After the change is made, both the existing design (in some cases "A") and the altered one ("B") are randomly shown to live users.

A and B are then assessed to see which performs better at getting to the objective.

Usually, with A/B Testing, the aspects of the design that are altered are color, size, and labeling (when dealing with buttons, calls to action, etc); tone, length, and font size (when dealing with text); and size and placement (when dealing with images). In addition to these, many other aspects of the design can also be tinkered with like the number of form elements on a page, its layout, etc.

Benefits

  • It captures actual user behavior on the live product
  • Small differences between the versions are highlighted because a significant number of people are able to use both
  • It is cheap since not a lot of human effort is needed to run one and complex analytics are not needed to see the effects of the difference between the versions (if any effects are manifest, that is) since it's simply comparing the numbers between them.

Limitations

  • There has to be one clear goal since it only works when an educated guess has already been made as to what the solution could be.
  • A/B Testing is only useful for measuring things that are easily measurable by a computer on an already fully implemented system/design.
  • It does not reveal why users behave differently between the two versions, only that they do.
  • And, it only provides data on the element presently being tested. There might be other potentially worse usability issues with the product, but A/B Testing is incapable of revealing them.

In the design process, A/B Testing is a qualitative evaluative method that lets us know what exactly do to to improve (or not worsen) a currently existing product. To find out why users behave differently to A and B versions, a different design method that gets user feedback would be necessary. Over the next 99 days, those methods will be discovered!


References
  1. Martin, B., & Hanington, B. (2012). Universal Methods of Design: 100 ways to research complex problems, develop innovative ideas & Design effective solutions. Rockport
  2. Nielsen, J. (2005, August 15). Putting A/B Testing in Its Place. Retrieved June 03, 2017


Full List of 100 Methods Day 2 - AEIOU