The traditional way of building software focused on requirements, driven by a competitive view of the market, and responded to perceived needs with additional features and functions. Our new way of developing products focuses on people first: on spending lots of time with them in order to build an almost intuitive sense for their emotional journey. We design with that journey and those emotions in mind, and as a result, we can produce great products that people love.
—Jon Kolko, Understanding our product design strategy and ecosystem.
Posts tagged “product discovery”
Why everyone should do usability testing (even you)
I vividly remember the first usability test I attended. I was a brand new employee at eBay, and I walked into a dark observation room with no idea what to expect. I came out of that room 60 minutes later with the strangest mix of emotions—heartbroken that our product clearly had usability issues that made users incredibly frustrated, but also relieved and excited that we now had the information we needed to fix those issues. I became a usability testing convert for life, and have been making it a part of my product design process ever since.
I’m deeply passionate about this methodology and how it makes us better designers (and improves the experiences of our users), so I don’t think it should be something that we reserve only for the “highly trained” to do. Usability testing is something all of us should do as a regular part of our design process. But that doesn’t mean it’s straight-forward—there are many pitfalls and ways to generate bad data with usability testing. So I wanted to write a brief introduction to the methodology and why it’s so important, as a foundation for people who haven’t had training in the method but would like to make it part of their process1.
So, let’s start at the beginning.
Usability testing is a very powerful (and shamefully underused) user research methodology—when it is used correctly. In fact, usability testing is probably the only method that can be relied on to consistently produce measurable improvements to the usability of a product. Bruce Tognazzini once said:
Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain.
But that all depends on the all-important “when it is used correctly” caveat. To make sure we do that, we need to understand when to do usability testing, and what to use it for.
When to do usability testing
To answer the “when” question I need to bring back the three buckets of user research I first discussed in Making It Right:
- Exploratory Research is used before a product is designed, to uncover unmet user needs and make it easier to get to product-market fit. Ethnography and contextual inquiries are the most-used methods in this bucket.
- Design Research helps to develop and refine product ideas that come out of the user needs analysis. Methods include traditional usability testing, RITE testing (rapid iterative testing and evaluation), and even quantitative methods like eye tracking.
- Assessment research helps us figure out if the changes we’ve made actually improved the product, or if we’re just spinning our wheels for nothing.
Usability testing is best used during the design research phase of a product. Ideally you’ll have an interactive prototype or some other lightweight interface to work with. It needs to be detailed enough to make sense to a user, but not so detailed that you’re reluctant to make changes based on feedback. Of course, you can also do usability testing on an existing live product, as long as the team has an appetite to make changes based on the insights that come back.
Usability testing shines during the design research phase since it plays on its strengths as a way to uncover the issues with an existing product or prototype. Trying to shoehorn usability testing into one of the other user research phases leads to trouble, since the nature of the data you get from it simply won’t help you make good decisions (i.e., don’t use it to try to decide what products to build, or if something you built objectively improved user satisfaction).
What to use usability testing for
This leads into what usability testing is good at: refining a product. It’s not good at finding out what to build (unless it’s combined with an ethnographic component). To put a finer point on what usability testing is most useful for, here’s a much-simplified diagram to put it in context with some other research methods.
We use methods like analytics and surveys to understand what happens in the product. We use analytics to figure what users do, and we use surveys or other interview techniques to figure out what they say about the experience. The problem is that this doesn’t help us understand why something happens, and without that information we won’t be able to fix any of the problems we come across. That’s where usability testing comes in.
What makes usability testing so perfect for understanding the issues with an interface is that it is an observational research method. It’s not about asking people what they think about an interface. It’s about showing them an interface, giving them tasks to do in that interface, and then watching them as they go through those tasks. We can ask them questions about the experience, but that’s just to provide context.
At its core, usability testing means that we observe users as they make their way around an interface, and use that data to understand what issues we need to fix. So, for example, if we see in our analytics that there is a large drop-off in our checkout flow, usability testing can help us figure out why that drop-off happens, and how to fix it.
Haters gonna stop hating
I’ve seen usability testing abused in several ways that have ended up giving it a bad name in some circles. Here are some guidelines to keep in mind to help break through those prejudices.
First, don’t confuse usability testing with focus groups. They are very different methodologies, and they are certainly not interchangeable. Focus groups are good for some marketing purposes—to understand brand sentiment or positioning. It is a terrible way to get feedback on an interface. Usability testing is so good at what it does precisely because it is a 1:1 methodology. There is no groupthink, no way to get influenced by other users. In that sense, it is controlled environment. Focus groups are anything but.
Second, remember the golden rule that usability testing is about observation. It can’t tell you which interface people like more than another, so don’t try to use it to settle those disputes. It’s the wrong question to answer anyway. It doesn’t matter what users like—it matters what they can use effectively to accomplish their goals. So usability testing is not “lightweight A/B testing”, as I’ve heard it described. It is meant to be part of an ongoing iterative design process with the goal of improving the product incrementally.
Finally, remember that you don’t need to be in a large organization or have tons of money to do usability testing. This is a methodology that scales really well. For startups who just have an afternoon to get some feedback, you can take some paper prototypes to a coffee shop. For large companies who need to convince a bunch of stakeholders to make changes, you can run a series of formal usability testing sessions. Whatever works—and don’t be mistaken, every little bit helps.
Go and make it so
I want to end this introduction with a small call to arms. Usability testing is an inherently uncomfortable methodology because it assumes and embraces the fact that your product isn’t perfect. That’s a difficult thing to make peace with—especially as a designer. But taking that position is the only way your product is going to get better. You can’t fix something that you don’t think is broken. Clayton Christensen made a similar point in The Innovator’s Dilemma. He calls this mindset “discovery-based planning”:
Discovery-based planning suggests that managers assume that forecasts are wrong, rather than right, and that the strategy they have chosen to pursue may likewise be wrong.
Investing and managing under such assumptions drives managers to develop plans for learning what needs to be known, a much more effective way to confront disruptive technologies successfully.
Or to repeat one of my other favorite quotes: “Design like you’re right, listen like you’re wrong.” Usability testing gives us a proven process to understand what we got wrong so we can get more of it right. That makes it a methodology we should all invest in more.
- Who knows, maybe I’ll turn it into a longer, very practical series if there’s interest. ↩
Why it's more difficult to prioritize features than problems
Daniel Zacarias’s Moving from Solutions to Problems is a must-read for all product managers, and anyone who’s involved in product prioritization. Daniel’s main thesis is that prioritizing problems results in much better products than prioritizing features—and I wholeheartedly agree with him. He addresses many issues with focusing on features, but the one that really resonated with me is that it’s much harder to prioritize features:
Products and features are versions of a solution to a problem. What this means is that by thinking in terms of the former, the problem they’re solving gets more difficult to grasp. Either because it’s a non obvious problem, or the product/feature are poor solutions for it.
In practical terms, this makes it much harder to prioritize a list of features than a list of problems. There are added layers of indirection that make us evaluate priorities in a different way. It gets difficult to determine the intent and expected impact from a feature. On the other hand, a problem (“low number of transactions”) can more easily lead to an objective (“increasing number of transactions per customer per month by 30%”).
Quote: Tomer Sharon on the importance of understanding a problem
The question “How do people currently solve a problem?” is critical, because deeply understanding a problem can go a long way toward solving it with a product, feature, or service. Falling in love with a problem happens through observing it happen in a relevant context, where the problem is occurring to people in your target audience.
—Tomer Sharon, Validating Product Ideas
The ethics of "empowering" users
Katherine Benjamin wrote a fantastic essay on designing for user empowerment, and what that really means. She asks, specifically in the context of digital health, When are we empowering users, and when are we just being lazy?
The World Bank talks about empowerment in terms of two things. Firstly, they talk about enhancing an individual’s capacity to make choices. They then talk about leveraging those choices into desired actions or outcomes. […]
When we think about things like wearable devices that enable people to actualise the “quantified-self”, we are usually realising just the ability of someone to self-monitor. In other words, we can make it possible for people to take better care of themselves by developing new technologies that support self-care. However, these innovations will only help those who are genuinely interested in taking greater control of their health. This type of self-determination with regard to health is a necessary pre-condition for successful adoption of digital health solutions.
Unfortunately, all too often, in the digital health industry, we get lazy and speak as though technology itself can create that individual level of empowerment. This fails to consider the inherent power dynamics between providers and users of health services, and the role this dynamic plays in facilitating agency among the users of health services.
When we design to empower users we can’t just think about giving people the information they need to act. We also need to help them develop the desire to act on that information.
Why projects fail if they neglect research
Erika Hall’s The Secret Cost of Research is a great explanation of why research is essential for products to succeed:
The reason design projects that neglect research fail isn’t because of a lack of knowledge. It’s because of a lack of shared knowledge. Creating something of any complexity generally requires several different people with different backgrounds and different priorities to collaborate on a goal. If you don’t go through an initial research process with your team, if you just get down to designing without examining your assumptions, you may think your individual views line up much more than they do. Poorly distributed knowledge is barely more useful than no knowledge at all.
I can definitely attest to this. It’s exactly why I’m such a big fan of Product Discovery:
This phase always—without fail—produces insights the team finds incredibly valuable. Startups gain clarity about what to say “yes” and “no” to in their product, and large corporations learn how to go beyond customer-centricity buzzwords and discover which benefits they should be selling to their users. As just one of many examples, I was once in a workshop that revealed the executives had a completely different vision for the company than the designers and developers. It was an awkward two hours, but in the end they agreed on the tough but correct decision to suspend their e-commerce plans until some of the content areas on the site had been sorted out. It’s great to see a statement of purpose emerge from these sessions—one that finally gets an organization to agree on what the product’s focus should be.
It’s pretty remarkable how different people’s visions for a product can be, especially since there’s such a simple two-step program to fix that:
- Understand user needs and business goals.
- Talk about it, together.
The real purpose of a Minimum Viable Product
In Jim Brikman’s post A Minimum Viable Product Is Not a Product, It’s a Process he explains the real purpose of following an MVP approach:
An MVP is a process that you repeat over and over again: Identify your riskiest assumption, find the smallest possible experiment to test that assumption, and use the results of the experiment to course correct.
When you build a product, you make many assumptions. You assume you know what users are looking for, how the design should work, what marketing strategy to use, what architecture will work most efficiently, which monetization strategy will make it sustainable, and which laws and regulations you have to comply with. No matter how good you are, some of your assumptions will be wrong. The problem is, you don’t know which ones.
And the crux:
The only way to find that out—the only way to test your assumptions—is to put your product in front of real users as quickly as possible. And when you do, you will often find that you have to go back to the drawing board. In fact, you’ll have to go back to the drawing board not just once, but over and over again.
This is one of the major problems with MVP-thinking as I see it in many organizations today. For many product managers MVP is unfortunately not a learning process, but just a fancy way of saying “v1”. And as we all know, the biggest lie in the enterprise world is “v1”. There’s never a v2. So many hide behind the “MVP” moniker to obscure the fact that they just want to get a thing out the door and move on. We’d do well to remind our teams that the real purpose of an MVP is to learn, not to complete.
How to decide on prototyping fidelity
Ryan Singer shares some tips about prototyping tradeoffs in The Fidelity Curve: How to weigh the costs and benefits of creating UI mockups:
The purpose of making sketches and mockups before coding is to gain confidence in what we plan to do. I’m trying to remove risk from the decision to build something by somehow “previewing” it in a cheaper form. There’s a trade-off here. The higher the fidelity of the mockup, the more confidence it gives me. But the longer it takes to create that mockup, the more time I’ve wasted on an intermediate step before building the real thing.
I like to look at that trade-off economically. Each method reduces risk by letting me preview the outcome at lower fidelity, at the cost of time spent on it. The cost/benefit of each type of mockup is going to vary depending on the fidelity of the simulation and the work involved in building the real thing.
He goes on to provide some solid guidelines for when to go with paper & pencil vs. interactive or higher fidelity mock-ups.
Using ethnography to build better products
Craig Mod’s essay on doing design ethnography in Myanmar is so far my favorite piece of writing of the year. In The Facebook-Loving Farmers of Myanmar he shares some notes about the team’s visits and interviews:
There is a phrase repeated over and over again during my time in Myanmar: From no power to solar, from no banks to digital currencies, from no computers and no internet to capable smartphones with fast 3G connections. It is the mantra of consultants working in these emergent economies. And these emergent economies have one colossal advantage over the entrenched and techno-gluttonous west: There is little incumbency.
There is, however, instability—in government and currency. It’s one of the reasons why a country like Myanmar is just now getting these connections, these devices. The instability significantly increases risk for outside investors and companies. But the residual effect of that instability is a lack of incumbency and traditional infrastructure. And so there is no incumbent electric giant monopolizing rural areas to fight against solar, there is no incumbent bank which will lobby against bitcoin, there are no expectations about how a computer should work, how a digital book should feel. There is only hunger and curiosity. And so there is a wild and distinct freedom to the feeling of working in places like this. It is what intoxicates these consultants. You have seen and lived within a future, and believe—must believe—you can help bring some better version of it to light here. A place like Myanmar is a wireless mulligan. A chance to get things right in a way that we couldn’t or can’t now in our incumbent laden latticeworks back home.
It’s a long article, and it should be. There’s so much insight here, just from spending a few days with people observing, listening, understanding. I don’t understand why this truth is so hard for some product leaders to understand:
A common mistake in building products is to base them on assumptions around how a technology might be adopted. The goal of in-field interviewing in design ethnography is to undermine these assumptions, to be able to design tools and products aligned with actual observed use cases and needs.
Just imagine how different the world would be—and what incredible products we’d be able to build—if we always took the time to understand users and their needs in this way first.
Hiding stories on Facebook: intent vs. usage
From Will Oremus’s insightful story How Facebook’s news feed algorithm works:
Facebook’s data scientists were aware that a small proportion of users—5 percent—were doing 85 percent of the hiding. When Facebook dug deeper, it found that a small subset of those 5 percent were hiding almost every story they saw—even ones they had liked and commented on. For these “superhiders,” it turned out, hiding a story didn’t mean they disliked it; it was simply their way of marking the post “read,” like archiving a message in Gmail.
This reminds me of a story I read a while back about how tons of people flagged and reported news stories about Lance Armstrong for “drug abuse”. This is why qualitative research is so incredibly important. Analytics can never tell us the whole story.