John Ciancutti explains how Netflix uses data to make product decisions in How We Determine Product Success:

It can be frustrating to be in a product development environment where force of personality or hierarchy determines product outcomes. At Netflix the focus on customer value makes a teachable moment of those times one guesses wrong. My product intuition is vastly better today for the benefit of my mistakes.

It’s a great read on the importance of listening like you’re wrong when you develop products.

This widely linked post from Benedict Evens definitely deserves all the attention it’s been getting. In Glass, Home and solipsism Benedict talks about the fallacy of thinking that customers care as much about your product as you do1:

You can think of people as users or customers — but they’re not yours. They don’t belong to you, and they may barely even care that you exist.

A little bit earlier he discusses Google Glass and says this:

If everyone you know owns a Tesla and is deeply engrossed in new technology, then the idea that there might be social problems with Glass doesn’t come up — everyone’s too busy saying ‘AWESOME!’

This reminded me of what John T. Gourville calls “The 9x Effect” in Eager Sellers and Stony Buyers: Understanding the Psychology of New-Product Adoption (you have to register for a free HBR account to view the article):

There’s a fundamental problem for companies that want consumers to embrace innovations: While developers are already sold on their products and see them as essential, consumers are reluctant to part with what they have. This conflict results in a mismatch of nine to one between what innovators believe consumers want and what consumers truly desire.

This image from the article explains the concept well:

This might explain why products like Color, Facebook Home, and Google Glass appear destined not to do very well in the general market.

---

1. I’ve written about the same thing before in What users really care about. ↩

Jay Cassano did an excellent interview with Nate Bolt, who is in charge of design research at Facebook. My favorite part from Secrets From Facebook’s Mobile UX Testing Team:

We try to never deliver any reports ever, if possible. Reports can’t attend meetings and they can’t argue in favor of their findings. They die in the wastebasket immediately. So we’ll bring up some data in a session, we brainstorm on a whiteboard, absorb some of the human patterns of the people that are using this stuff, and then incorporate that in our next build. That’s the goal.

Since I’m currently on the client side of design, reports are still very much a part of what we do. We often have to convince many more people than just the immediate project team about the changes we’re recommending. However, whenever possible we discuss our findings in a workshop environment with clients first, so that we’re all aligned on what the main usability issues are. This approach sometimes negates the need for a report entirely, which is a great outcome — everyone just does what needs to be done to make the product better.

Baruch Sachs makes some good points in Part 1 of a series called Practicing Great UX Consulting:

You are not there to educate people. You are there to advise your client and guide the creation of an amazing user experience. You are the expert; that’s why they brought you in. Collaboration and openness are key here. People need to feel invested, not put upon.

I agree that collaboration is key — the problem comes when collaboration gets confused with consensus. Consensus cultures often produce watered down, unexciting products. Products where endless rounds of give-and-take have worn down the original idea to a shadow of what it once was. Consensus cultures also wear down the teams working on the product, because no one really gets what they want, they just get some of it.

Collaboration is different. In collaboration cultures people understand that even though everyone gets a voice, not everyone gets to decide. People are able to voice their opinions, argue passionately for how they believe things should be done, and try to negotiate compromises. But it certainly doesn’t mean that everyone has to agree with every decision. And that’s why the client/agency trust relationship is so crucial.

One great collaboration technique that works well with clients is called Design Studio. Jared Spool has a great write-up of how to conduct a Design Studio worskhop, and it’s also worth reading Paul Boag’s thoughts in Never wireframe alone.

Rebekah Cox wrote a great post discussing the difference between product Demo Modes (in-store displays, on-stage demos that work without a glitch, picture-perfect product intros) and what she calls Reality Mode:

Reality mode takes time, iteration, data and user research. It takes honestly using what you’ve created and putting it through its paces. It takes asking yourself “is it useful?” and honestly answering. The result may not align with conventional wisdom, you may have to sacrifice that clever hook everyone comments about (but is ultimately useless or worse) and the demo may be boring or nonexistent. But by using your product in the real world and thinking about its true utility and value, you may end up with an enduring product where people are delighted through consistently delivered value instead of just a cool demo.

It reminds me of Marco Arment’s assessment that Facebook Home is “designed for optimal input and failed to consider real-world usage.” This is a very real problem in product development. It takes courage to look your darling project in the eye, find it wanting, and admit to it.

When I wrote The hype, benefits, and dangers of Big Data a few months ago I thought it would be my only post about Big Data, and then I’d move on. But 2013 appears to be the year of Big Data, so you can’t turn a corner on the web without bumping into an article about it. Looking at Google Trends, it’s clear that interest is at an all-time high:

So I wanted to point out just a few more articles that range from calling for a more tempered approach to Big Data to an all-out assault on its value and validity. Let’s start with the juicy one…

In A More Thoughtful but No More Convincing View of Big Data Stephen Few reviews the book Big Data: A Revolution That Will Transform How We Live, Work, and Think and uses it as a way to articulate his distaste with the whole thing:

Data exists in a potentially infinite supply. Given this fact, wouldn’t it be wise to determine with great care what we collect, store, retain, and mine for value? To the extent that more people are turning to data for help these days, learning to depend on evidence rather than intuition alone to inform their decisions, should we accept the Big Data campaign as helpful? We can turn people on to data without claiming that something miraculous has changed in the data landscape over the last few years. […]

As data continues to increase in volume, velocity, and variety as it has since the advent of the computer, its potential for wise use increases as well, but only if we refine our ability to separate the signals from the noise. More does not trump better. Without the right data and skills, more will only bury us.

It’s a long article, but very detailed and highly recommended as a well-reasoned counter-argument to the Big Data movement. Others are a little more pragmatic, suggesting that we improve on the promise of Big Data rather than destroy it. In Coffee & Empathy: Why data without a soul is meaningless1 Om Malik states:

What will it take to build emotive-and-empathic data experiences? Less data science and more data art — which, in other words, means that data wranglers have to develop correlations between data much like the human brain finds context. It is actually not about building the fanciest machine, but instead about the ability to ask the human questions. It is not about just being data informed, but being data aware and data intelligent.

It’s important to take this further and say the soul Om talks about needs to come from qualitative methods like ethnography. That’s why I like Dave McColgin’s point in his article How Will Big Data Change Design Research?:

In our field of designing products and experiences, the ‘why’ stays at the center of our process and creativity. Many designers work mostly on new products and services for which there may not yet be reliable data available. […] While Big Data can inform designers on how to improve once they put something out there, it is design research that provides principled guidance towards good solutions all along the way. Big Data can’t help us do that right now.

Tricia Wang’s Big Data Needs Thick Data is another excellent plea for ethnographers to get involved in the Big Data movement, to produce what she calls “Thick Data”:

Big Data produces so much information that it needs something more to bridge and/or reveal knowledge gaps. That’s why ethnographic work holds such enormous value in the era of Big Data. […]

Big Data reveals insights with a particular range of data points, while Thick Data reveals the social context of and connections between data points. Big Data delivers numbers; thick data delivers stories. Big data relies on machine learning; thick data relies on human learning.

And finally, Martin U. Müller and Marcel Rosenbach look at some of the scarier implications of Big Data in Living by the Numbers: Big Data Knows What Your Future Holds:

Is it truly desirable for cultural assets like TV series or music albums to be tailored to our predicted tastes by means of data-driven analyses? What happens to creativity, intuition and the element of surprise in this totally calculated world?

Internet philosopher Evgeny Morozov warns of an impending “tyranny of algorithms” and is fundamentally critical of the ideology behind many current Big Data applications. Morozov argues that because formulas are increasingly being used in finance and, as in the case of Predictive Policing, in police work, they should be regularly reviewed by independent, qualified auditors — if only to prevent discrimination and abuses of power.

I personally think there is the same value in data that there has always been, and that the Big Data movement isn’t so much about the size of the data sets, but the ability to extract more of that inherent value (signal) from the noise. But an algorithm will only take you so far. As always, knowing what and how much is not very useful without knowing why. And Big Data will never be able to tell us why…

---

1. What’s that? You think I’ll just automatically link to any article with the word “coffee” in the headline? I resent that accusation, sir or madam! ↩

UX deliverables had a rocky year so far. I feel particularly bad for the humble wireframe, which took some serious knocks over the past few months. There’s also a growing skepticism about the value of Personas. The Persona thing made me particularly uneasy because I’ve always been a huge fan, and we still start most of our projects with a workshop to define Personas and User Journeys.

That unease led me to introspection, which is a good thing, because it made me step back and revisit why we use Personas, and how we use them on a very practical level to design better products. The problem is, I came up short… I realized that though Personas are extremely useful to help clients figure out who their target market is, and understand those users better, they’re often not very useful once we go into the Interaction Design phase of the project1.

In contrast, the User Journey map that we create at the beginning of every project remains open in a tab until everything is done and dusted. I cannot overstate the usefulness of user journey mapping as a UX method. And then there’s the content plan — another essential part of the puzzle that we always create before the design phase starts. Once we’ve done a version of the Information Architecture, the content plan maps what kind of content needs to go on each page. But these are all separate documents, and you can only reference so many PDFs on any given day before it gets terribly distracting.

I realized that one of the problems with Personas is that it takes extra work to turn those user insights into artefacts that are useful for design. And that led me to the realization that there is probably a better way to group all these disparate UX deliverables together to help us create better products.

I decided to test my theory, so on a project we recently started, our User Journey map became more than just a journey with touchpoints, emotions, takeaways, etc. It also became a representation of the Information Architecture and the content plan, with our Personas (needs, goals, scenarios) serving as the starting point for everything — sort of like the glue that ties it all together.

The project is still very much in progress, so I can’t show the full end result yet, but here’s a slightly blurry snapshot of one section of the journey:

This document is a summary of everything we need to know to design the best possible product for users. It has the following elements:

• Unique selling points to keep us focused on what the site needs to communicate at all times. This comes straight from the Persona needs and goals.
• Journey stages and model to remind us how the product fit into people’s lives, and what the primary calls to action need to be throughout the site.
• Questions that our target Personas are likely to ask in each phase of the journey, to focus the type of content we serve on each page. In an e-commerce context, these are questions like, “Can I trust this retailer?” or “When will my stuff arrive?”
• Takeaways and key principles to summarise the above sections (which primarily act as problem definitions/requirements) and document how that translates into the design decisions and solutions we need to keep in mind throughout the design process.
• Content plan that maps each phase of the journey with the questions our Personas will ask during that phase, and what it means for the specific content that needs to go on each page. We get very specific here — nothing gets on the page unless it’s in the content plan. And if we can’t identify a Persona that would find the content useful, it just doesn’t go on the list.

Even though the Personas aren’t explicitly referenced on this document, we extract the key points from each and turn those into information that is actually useful for design — namely the content they are most likely to be interested in. The Persona step is essential to help us get to this point, so we can’t skip it, but we don’t need to show faces and names and stories on the User Journey map to make that information useful.

So, in the spirit of “getting out of the deliverables business”, this expanded User Journey map becomes the only document we use to guide us throughout the design process. You can think of this as the UX Strategy document. It incorporates Persona-based user needs and business goals with site structure and content planning in a way that really works for us. It also places content at the centre of the design process, which makes it easier to follow mobile first and responsive design strategies.

I’m sure it’s not perfect, but so far this has been an extremely useful artefact for us.

---

1. Des Traynor wrote a good article about this. ↩

Kate Crawford wrote a very good critique of Big Data methods in The Hidden Biases in Big Data:

Data and data sets are not objective; they are creations of human design. We give numbers their voice, draw inferences from them, and define their meaning through our interpretations. Hidden biases in both the collection and analysis stages present considerable risks, and are as important to the big-data equation as the numbers themselves.

Kate uses some interesting examples from Hurrican Sandy and the City of Boston to make her argument, and ends with the conclusion that is a common plea among qualitative researchers:

We know that data insights can be found at multiple levels of granularity, and by combining methods such as ethnography with analytics, or conducting semi-structured interviews paired with information retrieval techniques, we can add depth to the data we collect. We get a much richer sense of the world when we ask people the why and the how not just the “how many”.

The ideal situation for any qualitative research project is for the facilitator to rely on someone else to take notes. That way the facilitator can focus all their attention on the participant. This holds true for contextual inquiries, in-depth interviews (IDIs), as well as usability tests. However, sometimes it’s just not possible to have a separate note-taker on a project. In those cases the interviewer has to take their own notes — but that can be distracting and terribly inefficient.

So, what’s the best way to be your own note-taker?

I’ve seen interviewers taking their own notes in a variety of ways, but the inherent flaws in each approach has always made me uncomfortable. Some interviewers use their laptops to make notes while the interview is going on. This is very efficient (there’s no transcribing afterwards), but the clicking of the keyboard can be distracting and off-putting to the participant.

Others print out their interview scripts, and leave blank spaces for writing notes about each question. The problem here is that scripts are fluid. You sometimes skip over questions, while other times you go off on an important tangent that isn’t covered in the script. So you tend to end up with empty spaces and cramped notes, all spanning multiple pages. Not ideal.

I am currently working on a project to make it easier for talk radio producers to do their jobs. As a first phase we’re doing a bunch of IDIs with producers — and I have to take my own notes. So I decided to try a new approach, and I like it so far.

I started the project with a long, free-form interview with one of the project leads to develop a generic user journey for producers. I looked for common elements that remain constant regardless of the process each producer might use to perform their tasks, and used that to build a basic journey model for talk radio production. It’s not a full-on journey map, just a list of steps that all producers have to complete when they put on a show.

I then created an A3 sheet (6.5 x 11.7 in) for each interview, consisting of the participant’s name, time slot, and the headings for each of the steps in the journey. While conducting the interview I filled out any insights that came out for each of the steps — as we worked our way through the script. Here’s an example of what a sheet looked like after an interview:

I discovered that this approach has several advantages over other note-taking methods I’ve tried:

• It’s script-agnostic. The interview questions address each of the steps in the journey, but I don’t have to stick to it religiously — it’s ok to jump around and make notes in a different column if needed.
• Everything is on one page. This not only makes note-taking more efficient, but it’s also going to make the analysis phase easier. I’ll be able to lay out the sheets on a table and see all the data in one place as I start the affinity diagram process.
• It makes me a better listener. I was worried that the note-taking would be distracting to participants, but I found the opposite to be true. By taking notes while we talk (and looking them in the eyes when I’m not writing), participants could tell that I’m really listening to them — not just pretending. And this made for much better interviews.

I’m sure this method of note-taking isn’t perfect, but I’m quite happy with how it turned out. Please let me know if you have any suggestions to improved this process — or if you have a different method that works well for you.

Figuring out what to read (and what to believe) about Big Data is becoming a Big Data problem in and of itself1. I wrote The hype, benefits, and dangers of Big Data a while ago to give an overview of what’s out there, but there are two more interesting articles from the last week that I’d like to highlight as well.

First, on the HBR blog Jake Porway talks about Big Data and social entrepreneurship and makes the point that You Can’t Just Hack Your Way to Social Change:

Any data scientist worth their salary will tell you that you should start with a question, NOT the data. Unfortunately, data hackathons often lack clear problem definitions. Most companies think that if you can just get hackers, pizza, and data together in a room, magic will happen. This is the same as if Habitat for Humanity gathered its volunteers around a pile of wood and said, “Have at it!” By the end of the day you’d be left with half of a sunroom with 14 outlets in it.

And on Wired, Does ‘Big Data’ Mean the Demise of the Expert — And Intuition? is a very interesting excerpt from Viktor Mayer-Schönberger and Kenneth Cukier’s new book on the topic:

In the same spirit, the biggest impact of big data will be that data-driven decisions are poised to augment or overrule human judgment.

The subject-area expert, the substantive specialist, will lose some of his or her luster compared with the statistician and data analyst, who are unfettered by the old ways of doing things and let the data speak. This new cadre will rely on correlations without prejudgments and prejudice. To be sure, subject-area experts won’t die out, but their supremacy will ebb. From now on, they must share the podium with the big-data geeks, just as princely causation must share the limelight with humble correlation.

It seems like an obvious conclusion, but everything I’ve read so far about Big Data confirms that if we think cutting the “messiness” of human decision-making out of data analysis will result in better decisions, we’re sorely mistaken.

---

1. Sorry, I didn’t get much sleep last night, so even though I know this isn’t a particularly funny joke, I just can’t help myself. ↩