Menu

A technical guide to mobile usability testing

I wrote a guest post on mobile usability testing for my friends at Unboxed Consulting. It’s something I’ve mentioned briefly here on Elezea before, but in this post I go quite deep on the ins and outs of setting up a mobile usability lab. From A technical guide to mobile usability testing:

Setting aside the details of recruiting, script writing, and interviewing, from a technical perspective doing usability testing on desktop web applications is pretty simple, thanks to software like Morae and Silverback. There is, however, no straight-forward, single solution for doing usability testing on mobile devices. I recently went through the process of setting up our own mobile usability testing process at Jive, so I thought I’d share some of what we learned about the components of a good setup.

User testing and long-term product planning

Steve Barnett makes some great points on long-term planning in Plans, Details, Dates, and The Future1. I especially like the point about how user research fits into planning:

Before development starts on a new bit of work, you should be building prototypes and doing user testing with them. This always results in some changes to the plan, and often results in rather large charges. You can’t plan what these changes will be: you don’t know until you’ve done your user testing.


  1. And bless his heart for using an Oxford comma. 

The problem with surveys

Erika Hall speaks so much truth in her post On Surveys:

If you are treating a survey like a quantitative input, you can only ask questions that the respondents can be relied on to count. You must be honest about the type of data you are able to collect, or don’t bother.

My first role at eBay, years ago, was as a quantitative user researcher1. We ran surveys to measure satisfaction with different areas of the product over time. If that period taught me anything, it’s that surveys are extremely useful when combined with analytics as well as qualitative user research (triangulation), and pretty useless when looked at in isolation. There just isn’t enough context by itself.


  1. One of my early experiences at eBay was getting to work one morning and discovering that Peter Merholz wrote a scathing blog post about a survey I was running. This was my second month on the job, so I was pretty sure I was going to get fired. The worst part of it was that he didn’t have the full context, so his criticism wasn’t even valid. We were doing a controlled experiment where each group saw only one of the images in the survey, and the “likelihood to purchase” question was just a decoy as an introduction. We weren’t trying to get absolute numbers of likelihood to purchase (that would be ridiculous) — we were comparing responses to different pages to figure out what iconography would be best for ratings (stars, bars, or check marks). Subsequent questions were more specific about the ratings aspect. It went all the way up to our VP of Product and my manager had to write an explanation. I was mortified. I still sometimes wake up in the middle of the night in a cold sweat, screaming “survey!!!!!!!!!” 

A framework for empathy in design

The Paradox of Empathy is such a great post by Scott Jenson that I pretty much want to quote the whole thing. But I’ll stick with just this one gem, and encourage you to read it in full. It is a fantastic exploration of empathy in design, and includes a framework for making empathy part of our everyday work in a very practical way:

Designers will be the first to admit that not every empathic observation leads to a miraculous insight. However, it’s called “Design Thinking” for a reason: it’s how we process and explore, taking a complex problem and breaking it down before we build it back up. Product managers seem to expect a designer to walk up to a product, say something brilliant, and drop the mic. Experienced designers deeply understand a simple fact: design isn’t a deliverable, it’s a process. A process paved with dozens of small empathic observations that lead you, slowly, iteratively to a better product.

The problem for us designers is that our fellow teammates don’t always think this way and unfortunately, we as a community don’t reflect on this difference. It’s ironic that designers are passionate about how a product interacts with people but not how they themselves interact with their team.

Twitter text shots, and what design wants

I’ve been spending a lot of time thinking about how product design decisions aren’t neutral. The way we design a product has a direct effect on how people use it. This is obvious, but I think we often forget the real implication: Design wants something from its users. And we are the architects of those wants. We have a direct impact on user behavior, and we need to recognize the weight of that responsibility.

Let’s look at some recent product changes on Twitter as an example.

In October 2013, Twitter introduced more visual tweets, with photo previews within the timeline. It’s almost hard to imagine now, but before they introduced this you had to click on a link before you could see a photo.

This change had an immediate effect on how people used the product. I’m sure some of it was intentional — people began to tweet a lot more photos. Some of it was probably not intentional but still made sense: social media marketers caught on to the fact that if they attach a photo to an article tweet, they’ll get more attention since the tweet will take up more screen real estate. A little annoying, but ok, so far so good.

But then there was what must have been a fairly unexpected behavior change. People started to use screen shots of text to bypass Twitter’s 140 character limit. At first, only a few people did it. But then publishers and marketers started to notice, and it took off:

Some have even come up with guidelines for the best way to stand out in these “textshots”:

Following an exchange with MG Siegler a while back, I settled on a specific textshot style: sans-serif text with a sepia background pulled from Pocket. The idea of using the app’s sepia theme for these came from MG, who noticed that yellow screenshots had more contrast in Twitter’s native apps.

I’ll admit, the temptation to do this is strong. Earlier this week I used a textshot and it became my most retweeted tweet ever:

Nice, right? Win!

But wait… Let’s step back for a moment and have a look at the metrics on that tweet of mine:

Tweet activity

Almost 25,000 impressions, with a 0.9% click-through rate. That’s worse than a crappy banner ad, and it’s the sum-total of the amount of traffic I sent to Scott’s excellent post. As Derek Thompson points out in The Unbearable Lightness of Tweeting:

Is the social web just a matrix of empty shares, of hollow generosity? As Chartbeat CEO Tony Haile once said, there is “effectively no correlation between social shares and people actually reading.” People read without sharing, but just as often, perhaps, they share without reading. […]

There used to be a vague sense that Twitter drives traffic, and traffic drives renown (or fame, or pride, or whatever word defines the psychic benefit of public recognition). Instead, the truth is that Twitter can drive one sort of renown (there are some people who are Twitter-famous), and traffic affords a different psychic currency. But they are nearly independent variables.

All of this culminated in Medium’s Text Shots announcement yesterday:

Text shots

So there you have it. There is now a very real chance that most of our Twitter timelines will become nothing but screenshots of Medium articles that no one reads. That doesn’t help Medium, it doesn’t help authors, and it frankly doesn’t help us to experience and learn, which is kind of the point of reading. This trend does help Twitter, though. Quoting from The Unbearable Lightness of Tweeting again:

In the last month, I’ve created nearly 2 million impressions for Twitter. Whether that is good for my Twitter persona and my pride is a qualitative question whose answer resides outside the bounds of an analytics dashboard. But it is quantitatively not a good deal for The Atlantic. Something I already suspected has now been made crystal clear: 99 percent of my work on Twitter belongs to Twitter.

Twitter is a business, and impressions are how they make money, so this isn’t inherently evil or wrong. But Twitter is, if nothing else, not what we think it is. Not to get too curmudgeonly about “early Twitter”, but there was something amazing about the 140 character limit. Something about the constraint that brought out people’s creativity. And because it was all text, timelines were easy to scan. Now, all of that is different.

Putting all my personal feelings about this trend (and its implications on traffic and reading) aside, it’s time I get to the point. This fundamental change in the way Twitter is used can all be traced back to a single, fairly simple design decision back in 2013: expanding photos natively in the timeline. Without that change, none of this would have happened.

As designers we can’t possibly know how all the ways our decisions will affect behavior in a product. But we have to, at the very least, recognize that design has an opinion, and that it wants people to behave a certain way. I like the way Jared Spool phrases this:

Over the last year, we’ve started explaining design as “the rendering of intent.” The designer imagines an outcome and puts forth activities to make that outcome real.

We have a responsibility to do our best to ensure design wants things that are good for users as well as the business. We have to think ahead as much as possible, because what design wants is up to us. And once it wants the wrong things, it might be too late to change.

Don’t just design features, design systems

Rune Madsen wrote a really interesting post on how our design methods need to change when we work in software (as opposed to print). He explains in the post On meta-design and algorithmic design systems:

So what is meta-design? In a traditional design practice, the designer works directly on a design product. Be it a logo, website, or a set of posters, the designer is the instrument to produce the final artifact. A meta-designer works to distill this instrumentation into a design system, often written in software, that can create the final artifact. Instead of drawing it manually, she is programming the system to draw it. These systems can then be used within different contexts to generate a range of design products without much effort.

I’ll add my vote for the need to spend much more effort on design systems (like Atomic Design) upfront, to standardize (and eventually speed up) later development.

The managing/making dance

In my latest column for A List Apart I discuss our obsession with “managerial tracks” in career development, and propose something different. From Managing and Making: It Doesn’t Have to Be One or the Other:

I think we need a career system that encourages people to oscillate between individual contributor roles and manager roles. Maybe we provide “manager sabbaticals” where a manager becomes an individual contributor on a team for six to nine months. Maybe when a manager goes on vacation, an individual contributor takes on their role for a period of time (or for the duration of an entire project). I don’t know exactly what this looks like yet, but I think it’s important for us to figure it out.

I explain why in the article. I’d also like to point out that my original title for the piece was “Managing and Making: It Doesn’t Have To Be A Dirty Dance”, and it featured this photo:

Dirty Dancing Croc

I’m really glad my editor talked me out of it. It’s not good to make jokes that only people who grew up in the 80s will appreciate.

It’s a good joke, though.

Bots and the law

Kashmir Hill asks an interesting question: Who do we blame when a robot threatens to kill people?

Last week, police showed up at the home of Amsterdam Web developer Jeffry van der Goot because a Twitter account under van der Goot’s control had tweeted, according to the Guardian, “I seriously want to kill people.” But the menacing tweet wasn’t written by van der Goot; it was written by a robot.

He goes on:

Bots will be bots. They won’t know if they’re doing something wrong unless we program them to realize it, and it’s impossible to program them to recognize all possible wrong and illegal behavior. So we’ve got challenges ahead. In the short term, [Clément Hertling, a Paris-based university student who wrote the software that powered the bot] suggested Twitter — and any other platforms bots might live on — could solve the offensive speech problem by allowing bots to self-identify in an obvious way as bots. “That would allow people (law enforcement included) to ignore what they say when it becomes problematic.”

This issue only gets scarier as the question expands to wondering what happens when we put self-driving cars in morally ambiguous situations.

More

  1. 1
  2. ...
  3. 82
  4. 83
  5. 84
  6. 85
  7. 86
  8. ...
  9. 201