A recent discussion on a user experience forum I participate in turned to the topic of A/B testing. I really enjoyed the conversation so I wanted to reiterate some of the points I made, and expand on it a little bit as well. It’s not my goal to define A/B testing here but to share my opinion on its use. I believe that even though A/B testing can be extremely valuable to help identify the best iteration of a site or a particular page, it should never be used in isolation.
Since A/B testing is relatively cheap to do and the results are so compelling, companies are in danger of adopting a “test and learn” culture where pages are just A/B tested with no additional user input. That would be the wrong way to go. A/B testing shouldn’t be used on its own to make decisions, it should always be used in conjunction with other research methods — both qualitative (such as usability testing, ethnography) and quantitative (such as desirability studies).