Back in 2012 I wrote the following about a blind spot I’ve noticed in Agile development:

Problem solving involves not just iteration, but also lots of variation. This often requires time to get it wrong a few times, which doesn’t fit comfortably with the concept of release dates. See, the problem with integrating Agile and UX is not that designers want to hang on to “slow and heavy documents,” “big upfront design”, or whatever you want to call it. The problem is that each iteration further solidifies the chosen path, and there is no time to stop and ask if you’re going in the right direction.

All of that came flooding back when I read Jeff Patton’s Common Agile Practice Isn’t for Startups, in which he puts a slightly different spin on the issue that Agile is not very good at helping us figure out what to build. His solution is a product discovery process (something that’s obviously near and dear to my heart as well). He places the discovery process in the context of a different kind of velocity than is usually measured in Agile—trying to learn as much as possible about customers and the product:

There’s something very different about this process loop: the primary measure of progress during discovery isn’t delivery velocity, it’s learning velocity. And sadly, we can’t measure it in features or stories completed. And, even worse, we can’t plan two weeks of it in detail because what we learn today can and should change what we do tomorrow.

He goes on to describe a Nordstrom process:

Notice the Nordstrom Lab still uses time-boxes, 1 week in this case. But, they didn’t start the time-box by predicting how much they’d deliver, but with learning goals in mind. Then they iterated around the build-measure-learn loop as fast as they could.

The post is hard to quote from, so really, just go ahead and read it. It’s a very interesting approach to making discovery part of a regular Agile process.

If I link to a listicle, it has to be good enough to overcome my unnaturally strong negative feelings about those types of article. Alas, Mike Fishbein’s 4 steps to make experimentation a repeatable process in your business is that good. It’s a great overview of the importance of hypothesis testing over “requirements gathering” in product management:

Most new products fail, and most frequently because they do not meet user needs. Running experiments helps product managers validate customer demand for a product concept earlier in the product lifecycle.

By running experiments instead of launching a minimum viable product, product managers in large organizations can gain more autonomy, limit risk and brand exposure, and gain user insights even earlier in the product lifecycle. With this speed to user insight, product managers become better informed to build successful products.

I also especially liked this bit:

In the next evolution of product management, the product leader’s role shifts from making bold assumptions to fostering a culture that encourages learning in an efficient and effective way.

John Willshire wrote a good post on the difference between fidelity and resolution in design. From Want to improve your design process? Question your fidelity:

For us, fidelity is all about the people axis; how close is this to the real world? That’s the future point, when the product is out in front of lots of people, being used often, at scale. If you want to increase fidelity, then you show whatever you have to more people.

Which leaves the vertical axis, things, to be all about resolution. Resolution is a much more technical description of what we have in front of us, used across many different fields to description the detailed specifications of what the thing involves. It’s been much more useful when you’re using that language around the thing you’re working on.

There are some good illustrations in the post to make the point clear. I think this is a pretty important distinction, since it shows how user feedback can be helpful during each phase of a project.

I’m having a hard time with Alex Schleifer’s (Airbnb’s head of design) proclamations in Why Airbnb’s New Head of Design Believes ‘Design-Led’ Companies Don’t Work. There are a lot of sweeping generalizations in the article, but I’ll focus on one specific part — user-centered design. First, this:

The solution Schleifer and CEO Brian Chesky devised actually deemphasizes the designers. The point, Schleifer says, isn’t to create a “design-led culture,” because that tends to tell anyone who isn’t a designer that their insights take a backseat. It puts the entire organization in the position of having to react to one privileged point of view. Instead, Schleifer wants more people to appreciate what typically lies only within the realm of designers—the user viewpoint.

So far, so good. It makes total sense to bake user empathy into the process and not elevate it to some elite role. Let’s continue:

Thus, every project team at Airbnb now has a project manager whose explicit role is to represent the user, not a particular functional group like engineering or design. “Conflict is a huge and important part of innovation,” says Schleifer. “This structure creates points where different points of view meet, and are either aligned or not.”

Ok, this is good too. This is usually the role the product owner or user researcher or UX designer should be fulfilling (if they’re not, something’s wrong). But if we need to call it something else to avoid stepping on toes, that’s fine. Let’s keep going:

Airbnb’s approach does seem fairly novel, simply because it deals with a problem that bedevils any product company to one degree or another: Designers tend to design for themselves, whether they intend to or not.

I agree with this, and have written about the phenomenon before in Designer Myopia: How To Stop Designing For Ourselves. But then the author goes on to say this:

User research, meanwhile, often has limits. It’ll tell you what’s wrong, but it only rarely leads directly to great products. A true user perspective is something more nuanced, specific, intuitive, and independent.

This is where he loses me. User research is not just usability testing that tells you what’s wrong. It’s also ethnography and contextual inquiry and participatory design and in-depth interviews and a slew of other things that result in exactly what they are trying to accomplish: a user perspective that is “nuanced, specific, intuitive, and independent.” As I point out in that designer myopia article:

We do ethnography to learn, not to confirm our beliefs. By using this method to understand the culture and real needs of our users, we’re able to design better user-centered solutions than would be possible if we relied only on existing UI patterns and some usability testing.

Leaving the office and spending time observing users in their own environments is the best way to understand how a product is really being used in the wild. It’s the most efficient way to get out of your own head.

I would love to know how Airbnb proposes that their project managers get these perspectives without qualitative user research, i.e. direct contact with customers.

Articles like this make for great “you’re doing it wrong” headlines, but they are often so light on detail that you’re just left feeling bad about yourself without knowing why (or how to fix it). So I just want to say, don’t let them get to you. Keep doing what you’re doing. Conduct observational user research in context, triangulate your results, and speak up for user needs. There’s no evidence to suggest that those methods have stopped working.

I really like the approach described in Jen McGinn and Ana Ramírez Chang’s RITE+Krug: A Combination of Usability Test Methods for Agile Design. It’s a dense paper, but worth your time. Here’s a key part (my emphasis added):

Prior to using the RITE+Krug combination, the user research process and results had been divorced from the Agile processes, which resulted in the findings coming too late to be acted on. Because of this issue, we integrated the user research with the rest of the process, as illustrated in Table 1.

The team consists of developers, product managers, user experience designers, visual designers, quality assurance engineers, and a user researcher. The user experience designers and product managers work closely with the development team during the feature sprints—answering questions, giving feedback on progress, and fine tuning the feature as it is implemented. The bug fix sprints give the developers time to focus on product stability.

Meanwhile the product managers, user experience designers, visual designers, and the user researcher work on preparing the small set of features that will be implemented in the next iteration (see Table 1). This work includes feature selection, design, user testing, and redesigns. The whole team (including developers) gives feedback on the feature specification and design before it is ready to be implemented. Like others, our design team stays an iteration ahead of the development team. Like Patton recommends, we iterate the UI before it ever reaches development, thereby turning what is traditionally a validation process into a design process.

And here’s the table:

One of the biggest issues with usability testing and Agile is the complaint that testing slows down the process. This seems like a really good way to alleviate those concerns.

Beth Kowitt’s How Ikea took over the world is a great look inside the Ikea machine. For me, the biggest takeaway is how research and prototyping drive everything Ikea does. On research:

One way Ikea researchers get around this is by taking a firsthand look themselves. The company frequently does home visits and—in a practice that blends research with reality TV—will even send an anthropologist to live in a volunteer’s abode. Ikea recently put up cameras in people’s homes in Stockholm, Milan, New York, and Shenzhen, China, to better understand how people use their sofas. What did they learn? “They do all kinds of things except sitting and watching TV,” Ydholm says. The Ikea sleuths found that in Shenzhen, most of the subjects sat on the floor using the sofas as a backrest. “I can tell you seriously we for sure have not designed our sofas according to people sitting on the floor and using a sofa like that,” says Ydholm.

And on prototyping:

Products under development go through rapid prototyping in the pattern shop to provide a sense of what they will actually look like in the flesh—or at least in plastic. On a recent visit, one of the four 3-D printers was outputting a toilet brush. Apparently this is one of the more normal items. “We have a lot of strange things,” says Henrik Holmberg, who manages the department. “That is very good that we can do it in our own shop rather than spreading the crazy ideas externally.” One of the oddest things he’s ever worked on was a lamp made from the same material as egg cartons. “I thought that was very crazy,” he says, “but we proved the technique was possible.”

Great article on the power of user-centered design.

Erika Hall speaks so much truth in her post On Surveys:

If you are treating a survey like a quantitative input, you can only ask questions that the respondents can be relied on to count. You must be honest about the type of data you are able to collect, or don’t bother.

My first role at eBay, years ago, was as a quantitative user researcher1. We ran surveys to measure satisfaction with different areas of the product over time. If that period taught me anything, it’s that surveys are extremely useful when combined with analytics as well as qualitative user research (triangulation), and pretty useless when looked at in isolation. There just isn’t enough context by itself.

---

1. One of my early experiences at eBay was getting to work one morning and discovering that Peter Merholz wrote a scathing blog post about a survey I was running. This was my second month on the job, so I was pretty sure I was going to get fired. The worst part of it was that he didn’t have the full context, so his criticism wasn’t even valid. We were doing a controlled experiment where each group saw only one of the images in the survey, and the “likelihood to purchase” question was just a decoy as an introduction. We weren’t trying to get absolute numbers of likelihood to purchase (that would be ridiculous) — we were comparing responses to different pages to figure out what iconography would be best for ratings (stars, bars, or check marks). Subsequent questions were more specific about the ratings aspect. It went all the way up to our VP of Product and my manager had to write an explanation. I was mortified. I still sometimes wake up in the middle of the night in a cold sweat, screaming “survey!!!!!!!!!” ↩

Another possible origin is the president of Kal Kan Pet Food, who was said to eat a can of his dog food at shareholders’ meetings.

— Eating your own dog food on Wikipedia

In software circles dogfooding is usually encouraged — and for good reason. When companies force themselves to use their own products extensively it can have some great benefits. Since internal users visit even the darkest, most neglected corners of a product, bugs tend to be found and fixed much faster than waiting for reports from end users. Employees are also often the most critical users of their own products, so they are driven to make it better and will often have some fantastic ideas.

However, I’ve started to see a dangerous side effect of dogfooding that can sneak up and do huge damage to a product if we’re not careful. Here’s the problem. If we use our own products all the time we become such overly advanced users that we’re eventually unable to separate our own needs for the product from those of our target market.

Most users of our products don’t eat, sleep, and drink the ins and outs of the features we slave over every day. They dip in and out to buy something, or upload something, or quickly edit a photo. So in a really unfair and ironic turn of events, the more we use our own products the less we’re able to think like our users. It sucks, but it’s unfortunately the way it is.

We all repeat the mantra “I am not the user” to each other, but we tend to forget that the way to solve that particular problem is not to morph into advanced users, but to spend more time with actual users. It’s already so hard for us to put ourselves in the minds and shoes of our users. Dogfooding makes this even harder by lulling us into a false sense of security that maybe — just maybe — we are the user after all.

So, be careful. By all means use the crap out of your product — you wouldn’t be pouring your life into it if you didn’t love it. But once you start demanding features that would make your life easier, stop yourself. Be critical of your own motivations. Look at analytics and figure out if yours is a common use case or something reserved for the 1%. Contact your friendly neighborhood user researcher and ask them if they’ve validated the need for said feature. Hold on to your demand with open hands, and let it go if you realize it’s not something that would improve the product for target users.

In short, keep dogfooding — but don’t assume that all dogs will like the new flavors you’re proposing.

Buzz Usborne’s “God is in the details” is a great reminder that spending just a little time thinking things through a bit more that usual can have a big impact on user experience:

But it’s those little things, the tiny minutia of detail, that ultimately make beautiful products, and beautiful houses.

Unfortunately, when I refer to the “details” in product design, I’m not talking about obvious design things; like colours, drop-shadows or placement. Instead I’m referring to something harder to define; experience and subconscious patterns that help the user feel more at-ease with an interaction. That detail might come in the form of a change in cursor, a “down” style for a button, or a helpful animation.

Whatever form that detail takes, I’ll bet that it wasn’t designed in Photoshop, or included in even the most detailed spec document. It’s the details that fall outside of titles like UX, or UI. It’s interaction detail that can only be found after using a product for real, then dedicating solid design and engineering time to building.

I definitely agree with that last bit as well. I’m working on a prototype for a new app at the moment, and it’s only after we did usability testing with real target users that we discovered some interactions that would make the app a lot easier (and more useful) for users.

I just posted another excerpt from Making It Right to Medium. It’s called How to avoid building products that fail, and it’s all about starting the product development process at the right place:

When it comes to building products, the starting point is — always—needs. Not what we assume would be cool, but what users or the business need to be successful. […] One of the biggest mistakes we can make in product development is jumping to execution before an appropriate planning cycle has been completed, so we need to give planning the attention it deserves.