The dangers of “test and learn”

A recent discussion on a user experience forum I participate in turned to the topic of A/B testing.  I really enjoyed the conversation so I wanted to reiterate some of the points I made, and expand on it a little bit as well.  It’s not my goal to define A/B testing here but to share my opinion on its use.  I believe that even though A/B testing can be extremely valuable to help identify the best iteration of a site or a particular page, it should never be used in isolation.

Since A/B testing is relatively cheap to do and the results are so compelling, companies are in danger of adopting a “test and learn” culture where pages are just A/B tested with no additional user input.  That would be the wrong way to go.  A/B testing shouldn’t be used on its own to make decisions, it should always be used in conjunction with other research methods — both qualitative (such as usability testing, ethnography) and quantitative (such as desirability studies).



  1. 1
  2. ...
  3. 112
  4. 113
  5. 114
  6. 115
  7. 116
  8. 117