I want to tell you a quick story about running. And I know it’s a little weird to make everyday life stories about product design but I can’t help myself, so I’m going to do that at the end. Let’s be cool about it, ok?

So.

I hate running. I’ve always hated running. A couple of weeks ago my friend Jason came over for a visit and started asking me about it. He wouldn’t let up. Bastard kept pressing. “Why do you hate running so much?” Over and over and over. Eventually we narrowed it down to breathing. I’ve always felt uncomfortably out of breath when I run. This is when my wife joined the party, and feeding off each other, they kept at it.

What felt like hours later, through their incessant questioning, we figured out that I’ve been breathing ridiculously wrong during my runs for two decades. For reasons that I am unable to explain, I’ve always timed my in-and-out breathing with every third step. Not second. Not fourth. THIRD. The result is that I’ve basically been hyperventilating through my runs for the better part of 20 years.

Anyway, they coached me for about five minutes on how to breathe. Turns out — and this is going to blow your mind — your body knows when it has to breathe. All you have to do is stop trying to force some kind of rhythm. Just focus on getting as much oxygen into your lungs as you possibly can, and then letting it out slowly. Your body will find its own rhythm.

The next day I tried out this revolutionary breathing mechanism. Not only was it enjoyable, I ran faster and further than usual. Last weekend, not even two weeks after they accosted me, I went for a 10k run. It was my fastest 10k ever, and more importantly, I enjoyed it.

I keep switching back and forth between being really pissed that I’ve been doing it wrong for so long, and ecstatic that I’ve been able to figure out the issue. I’m still blown away that a change in breathing can have such an immediate and enormous impact. Mostly, I’m thankful for my friend and my wife, who persisted way past the point of annoyance, and gave me a new lease on running (and, not to be dramatic, but life too).

Here’s where we get to the product design bit. This experience taught me another lesson, something that’s already deeply ingrained but I still forget it sometimes:

You cannot come up with a good solution before you understand the problem completely.

I tried that shortcut with the running thing. I switched up the cadence, or tried to just “break through the wall”. It wasn’t until someone pushed me to get to the cause of it all that we were able to find the right solution. So I resolved to be just a little more annoying at work. All it really comes down to is asking Why? a few more times than I think I should. I know that the answers will help us understand what we’re trying to accomplish, and building a better product for our customers.

Today I’m lucky enough to attend Industry Conf in Newcastle, and also do a talk on user research in large enterprises. Industry is a fantastic conference to speak at, and it’s run by one of the best and nicest people in our community—Gavin Elliot. I can’t say enough good things about the conference organization, the venue, and the quality & usefulness of the talks we’re exposed to here.

The slides for my talk, entitled User Research Challenges and Solutions for the Enterprise, are embedded below. For a more detailed write-up you can download a free e-book I made with the folks at UX Pin, called Practical User Research for Enterprise UX.

I want to thank Gavin for giving me the opportunity to speak at Industry this year. I had a blast!

Alex Maughan voices some things I’ve been thinking about as well in a great post on user journey mapping. After one particular project he observed this:

This user flow may seem an obvious solution, but it wasn’t obvious to the company hired to design the app before I got involved, which had up to that point resulted in some very limiting and confusing design flows. I don’t say this to blow my own horn. The point I’m making is had they explored the designs with some rough, high-level journey sketching upfront, they probably would have realised this themselves.

I’ve become increasingly convinced that there is no complicated design problem that can’t be solved with a few hours of collective thinking and some boxes and arrows. Yeah, I get some eye-rolls for my boxes-and-arrows reliance sometimes—especially because the outcome sometimes feels just too simple. But I’m with Alex on this one—these things are only simple in retrospect.

I recently worked on a project where we had to figure out the migration paths of 6 different products, and how they will all work together. We were talking for a long time, but we didn’t really get anywhere until I started drawing it out on the whiteboard. The end result was deceptively simple (some parts blurred out because it’s roadmap stuff):

“That’s it?” we all said of what came to be known as the Tron diagram. Yep, we realized together: that’s it. But it took a few hours of discussion and drawing to get us to the “that’s it” state. So this is me adding my voice to Alex’s in saying: never underestimate the power of sketching flows and user journeys to help us solve difficult problems.

I worked with the wonderful folks at UXPin to write a short e-book on how to overcome some of the challenges of doing user research in large organizations. From the introduction:

Once a company grows over a certain size, the internal politics and number of people involved in every decision increase so much that it becomes virtually impossible to stay focused on fulfilling user needs and business goals. Instead, the focus turns inward to the opinions and whims of individuals inside the company. Add the complexity of designing B2B products to the mix and, well, things go bad very quickly.

When an abundance of stakeholders are involved in a product, user research is the only way to focus a whole team on the real needs and goals required for success. It’s also the only way to get people out of the habit of thinking “Well, I want this, so everyone else must want it too”—a view that I find much more common in enterprises than in smaller organizations.

If that sounds familiar to you, you’ll hopefully find the e-book useful. I discuss why it’s often so hard to get support for user research in enterprises. Then I provide some advice on how to sell the value of user research. Finally, I offer some practical tips for addressing the subtle differences of conducting research in larger organizations with users who aren’t buyers.

You can download the (free) e-book here: Practical User Research for Enterprise UX.

I vividly remember the first usability test I attended. I was a brand new employee at eBay, and I walked into a dark observation room with no idea what to expect. I came out of that room 60 minutes later with the strangest mix of emotions—heartbroken that our product clearly had usability issues that made users incredibly frustrated, but also relieved and excited that we now had the information we needed to fix those issues. I became a usability testing convert for life, and have been making it a part of my product design process ever since.

I’m deeply passionate about this methodology and how it makes us better designers (and improves the experiences of our users), so I don’t think it should be something that we reserve only for the “highly trained” to do. Usability testing is something all of us should do as a regular part of our design process. But that doesn’t mean it’s straight-forward—there are many pitfalls and ways to generate bad data with usability testing. So I wanted to write a brief introduction to the methodology and why it’s so important, as a foundation for people who haven’t had training in the method but would like to make it part of their process1.

So, let’s start at the beginning.

Usability testing is a very powerful (and shamefully underused) user research methodology—when it is used correctly. In fact, usability testing is probably the only method that can be relied on to consistently produce measurable improvements to the usability of a product. Bruce Tognazzini once said:

Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain.

But that all depends on the all-important “when it is used correctly” caveat. To make sure we do that, we need to understand when to do usability testing, and what to use it for.

When to do usability testing

To answer the “when” question I need to bring back the three buckets of user research I first discussed in Making It Right:

• Exploratory Research is used before a product is designed, to uncover unmet user needs and make it easier to get to product-market fit. Ethnography and contextual inquiries are the most-used methods in this bucket.
• Design Research helps to develop and refine product ideas that come out of the user needs analysis. Methods include traditional usability testing, RITE testing (rapid iterative testing and evaluation), and even quantitative methods like eye tracking.
• Assessment research helps us figure out if the changes we’ve made actually improved the product, or if we’re just spinning our wheels for nothing.

Usability testing is best used during the design research phase of a product. Ideally you’ll have an interactive prototype or some other lightweight interface to work with. It needs to be detailed enough to make sense to a user, but not so detailed that you’re reluctant to make changes based on feedback. Of course, you can also do usability testing on an existing live product, as long as the team has an appetite to make changes based on the insights that come back.

Usability testing shines during the design research phase since it plays on its strengths as a way to uncover the issues with an existing product or prototype. Trying to shoehorn usability testing into one of the other user research phases leads to trouble, since the nature of the data you get from it simply won’t help you make good decisions (i.e., don’t use it to try to decide what products to build, or if something you built objectively improved user satisfaction).

What to use usability testing for

This leads into what usability testing is good at: refining a product. It’s not good at finding out what to build (unless it’s combined with an ethnographic component). To put a finer point on what usability testing is most useful for, here’s a much-simplified diagram to put it in context with some other research methods.

We use methods like analytics and surveys to understand what happens in the product. We use analytics to figure what users do, and we use surveys or other interview techniques to figure out what they say about the experience. The problem is that this doesn’t help us understand why something happens, and without that information we won’t be able to fix any of the problems we come across. That’s where usability testing comes in.

What makes usability testing so perfect for understanding the issues with an interface is that it is an observational research method. It’s not about asking people what they think about an interface. It’s about showing them an interface, giving them tasks to do in that interface, and then watching them as they go through those tasks. We can ask them questions about the experience, but that’s just to provide context.

At its core, usability testing means that we observe users as they make their way around an interface, and use that data to understand what issues we need to fix. So, for example, if we see in our analytics that there is a large drop-off in our checkout flow, usability testing can help us figure out why that drop-off happens, and how to fix it.

Haters gonna stop hating

I’ve seen usability testing abused in several ways that have ended up giving it a bad name in some circles. Here are some guidelines to keep in mind to help break through those prejudices.

First, don’t confuse usability testing with focus groups. They are very different methodologies, and they are certainly not interchangeable. Focus groups are good for some marketing purposes—to understand brand sentiment or positioning. It is a terrible way to get feedback on an interface. Usability testing is so good at what it does precisely because it is a 1:1 methodology. There is no groupthink, no way to get influenced by other users. In that sense, it is controlled environment. Focus groups are anything but.

Second, remember the golden rule that usability testing is about observation. It can’t tell you which interface people like more than another, so don’t try to use it to settle those disputes. It’s the wrong question to answer anyway. It doesn’t matter what users like—it matters what they can use effectively to accomplish their goals. So usability testing is not “lightweight A/B testing”, as I’ve heard it described. It is meant to be part of an ongoing iterative design process with the goal of improving the product incrementally.

Finally, remember that you don’t need to be in a large organization or have tons of money to do usability testing. This is a methodology that scales really well. For startups who just have an afternoon to get some feedback, you can take some paper prototypes to a coffee shop. For large companies who need to convince a bunch of stakeholders to make changes, you can run a series of formal usability testing sessions. Whatever works—and don’t be mistaken, every little bit helps.

Go and make it so

I want to end this introduction with a small call to arms. Usability testing is an inherently uncomfortable methodology because it assumes and embraces the fact that your product isn’t perfect. That’s a difficult thing to make peace with—especially as a designer. But taking that position is the only way your product is going to get better. You can’t fix something that you don’t think is broken. Clayton Christensen made a similar point in The Innovator’s Dilemma. He calls this mindset “discovery-based planning”:

Discovery-based planning suggests that managers assume that forecasts are wrong, rather than right, and that the strategy they have chosen to pursue may likewise be wrong.

Investing and managing under such assumptions drives managers to develop plans for learning what needs to be known, a much more effective way to confront disruptive technologies successfully.

Or to repeat one of my other favorite quotes: “Design like you’re right, listen like you’re wrong.” Usability testing gives us a proven process to understand what we got wrong so we can get more of it right. That makes it a methodology we should all invest in more.

---

1. Who knows, maybe I’ll turn it into a longer, very practical series if there’s interest. ↩