Something’s been bothering me about Twitter’s version of the Retweet.  A lot has been said about the pros and cons of the feature, but here’s my main problem with it:

You can’t easily see when you’ve been retweeted, and by who.

Twitter Retweets don’t show up in your stream as @ Mentions, so the only way to see when you’ve been retweeted, and by who, is by going to Twitter.com, clicking on “Retweets” in the right nav, and then clicking on the “Your tweets, retweeted” tab.  That’s just too many clicks.  Some iPhone apps like Echofon and Tweetie support the Twitter Retweet, but they don’t show you who retweeted you.

The problem with this is that it reduces Twitter’s sense of community.  I often like communicating with those who retweet me, and this takes away that ability (unless you go through a lot of work on Twitter.com).

There are, of course, other issues with the Twitter Retweet function, like:

• No ability to add your own comments (but this is what the “/via @” syntax is for, so that’s probably ok)
• Diluting the value of retweets because some people use Twitter’s Retweet feature, and others use the traditional “RT @” syntax
• Weird and confusing syntax when someone uses Twitter’s Retweet function to retweet a “RT @” tweet.
• Tweetdeck, Echofon, Tweetie… they all handle Twitter Retweets differently, so it makes for a confusing UI.  For example, if I want to unfollow someone who Retweeted something, I can’t do that from within the tweet-level functions in Tweetdeck.

This might sound like I’m nitpicking, but it’s not my intention.  I applaud Twitter’s initiative to embrace the Retweet function.  And I think ever since Doug Bowman joined the Twitter design team, they have made Twitter.com a lot more useful with some great features.

But I do think this Retweet thing isn’t quite working yet.  I think having Twitter Retweets show up in your @ Mentions would solve a big part of this issue.  So, Doug - can you make that happen please!?

I should probably get up, walk around, and have a cup of coffee before I write this post.  Or maybe a little righteous anger over something small is good for the soul?  I’ll go with the latter…  I recently ordered a 2010 calender from Runner’s World.  A few days ago I received the calendar, along with the invoice.  Their payment is handled through a company called Rodale.  I just went to pay my invoice at www.rodalequickpay.com, and the experience left me frustrated, and incidentally still in debt to Runner’s World.

I know this shouldn’t bother me that much, but let me walk you through the experience, and then make a couple of observations.

The Rodale order process

I typed in www.rodalequickpay.com (yes, the irony of the “quickpay” part of the URL is pretty thick), and arrived at this screen:

Immediately, this bugs me.  I just want to pay an invoice, I don’t want to have to create an account.  There’s also so much wrong about this design:

• There are two calls to action, and the affordance is all wrong.  The first text you see is “If you are a new visitor…”, but the “Create New Login” button is too far away, making it look like you should log in if you are a new visitor.
• The “Login” button… first of all, it’s “Log in” (action), not “Login” (noun), but let’s ignore that pet peeve for now.  The button looks different from the top button, and is also much smaller, resulting in a pretty confusing experience on this first screen.
• I don’t understand the text “Please use this site to pay orders in full” at the bottom.  Not sure why it’s needed, and not sure why it’s not at the top of the screen.  Who is going to read that far down?

But ok, since I am a new visitor, I decided to create a new Login:

Ok, this is where things get really out of control.

• I’m not even going to begin to talk about the copy.  “eMail” in one spot, “E-Mail” in another?  And “Thank you!”?  But I digress.
• The first big problem here is that account creation and invoice detail information happen on the same screen.  I should enter my account level information first, and then move on to my transaction level information.  Especially considering that…
• …It is extremely difficult to find your account number and invoice number on the paper statement.  First, the microcopy about where to find it is not useful at all.  There is no “appropriate button” to click, and the clickable text “On My Invoice” and “On My Statement” don’t look like links and actually don’t go anywhere when you click on them.  (Read this post at Polon, and this one by Joshua Porter about the importance of writing good microcopy on forms)

So anyway, I start typing in random numbers from my paper invoice just to see if I can get somewhere, and this is the error message I get:

Ok, now we’re getting somewhere.  Don’t know why it’s a browser error, but fine.  So I know I’m looking for an Account number of >10 characters and an Order number of >12 characters.  Turns out that’s not entirely accurate though.  The form doesn’t let you enter more than 10 or 12 characters depending on the field.  So those numbers should actually be exactly 10 and 12 characters.  Why doesn’t the error message say that?  “Can not be less than”?  But hey, we’re making progress.  Off I go to look for those numbers.  It appears I got my Account number right, because next I got this error message:

Ok, now we have in-line messaging, not browser error messaging.  But whatever.  I verified the crap out of that number, but I couldn’t get past this screen.  It actually makes me sad because I’m sure the payment page would have been a real treat to write about.  I tried to call the toll free number but no one’s there, so as of this moment I still owe Runner’s World $21.75.  I’m really sorry guys, I will pay you as soon as you let me.

The point I’m trying to make

So here’s the problem.  The Rodale website was put together to accept payment.  This is how they make money.  But there was absolutely 0 thought put into the user experience, so I was simply unable to pay them.  And look, I know it’s much easier to take a design apart than it is to create a good one, I get that.  But UX design is becoming a mature field now.  It’s 2010.  Shouldn’t we be able to get rid of designs like this, and demand something better? It’s not rocket science, it’s a methodical thought processes to design a good experience.  Form design is difficult to get right, but it doesn’t have to look like Apple, it just has to get you through the process without friction.

What do you think?  Is it too early to rise up in anger against designs like this?  If not, what can we do to “spread the UX”, so to speak?

As a Product Manager, I understand the Minimum Viable Product (MVP) concept, decisions to de-scope rather than delay, etc.  But too often MVP’s go out into the wild missing that all-important middle “V”, so you end up with, well, minimum products.

An example I came across recently is the sign-in process on Comcast.com.  First, a little background.  Comcast recently deployed a product they call mySIGN-IN.  According to their FAQ page:

mySIGN-IN is a unified sign-in system that lets you use your existing email address and password to access participating Comcast sites. When you sign in to any participating Comcast site, you’ll be conveniently signed in to the other sites that you use.

That all seems well and good, but the actual sign-in experience shows what happens when features go out without proper integration.  The sign-in process now happens on two separate pages:

Step 1: Enter email address

Step 2: Enter password

Two things struck me immediately about this experience:

• There is no reason to split the sign-in process into two screens.
• The visual design of the two pages are completely different.

Wanting to give Comcast the benefit of the doubt, I started looking into this a little more (because, you know, what else am I going to do on a Sunday afternoon apart from listening to Amos Lee on vinyl).  I thought that maybe this was an acquisition, and they are just taking some time on the integration.  But no, as far as I can tell, mySIGN-IN is not an acquisition — it is an internally developed product.  So I think this is what happened:

• A separate Comcast division designed and developed the mySIGN-IN feature
• The different Comcast properties started implementing the feature onto their sign-in pages
• Due to technical reasons, the pages had to be split for Comcast.com
• There was no UX oversight to ensure design consistency (or no resources available to make necessary changes)

Now, it does appear that someone at Comcast realized that this is not an ideal experience, and decided that explaining the changes to users is in order.  It’s a noble idea, but as we know, most users don’t read anything that’s not inside or next to a text box.  Either way, here are some of the tool tips that were added:

That first tool tip really bugs me:

Due to some recent security improvements, we now require you to enter your user name and password in two separate steps.

That just doesn’t seem right to me.  Due to “security improvements”?  I may be missing something from a security perspective, but I just don’t see why the sign-in information can’t be passed through securely without splitting up the pages.

What this means for product managers

I don’t mean to pick on Comcast.  This type of thing is very common, and I’m sure I’ve made similar decisions in the past that results in a user experience that’s not ideal.  But I do think this example can teach us a few things about product management:

1. Product owners (those responsible for individual features) need Product Strategists to ensure UX consistency (see this article from Pragmatic Marketing about different Product Management roles).  mySIGN-IN was clearly design in a vacuum, which could have been ok if there was someone who made sure the user experience stayed consistent across properties.
2. Don’t leave out the “V” in the MVP.  I believe that Comcast didn’t launch a minimum viable product.  Splitting the login pages into two screens is unnecessary and confusing to users.  The MVP might be an incomplete product, but it should never feel incomplete to users.  Users shouldn’t be able to notice that something is missing.  There is clearly something missing here.
3. Tool tips won’t solve everything.  If I had a penny for every time I heard the phrase “We’ll just add some content to explain that to users…”  As a general rule, if you need a tool tip which links to an FAQ page to explain something to users, your design is probably not intuitive enough.  It cannot be stressed enough that users really don’t pay attention to a lot of text.  The average user sees a form field, and starts typing.  Your user experience should support that behavior, not try to change it.

I have no doubt that Comcast had the best of intentions here, and that mySIGN-in is probably a cool feature.  But without proper product management, even inherently cool features can become frustrating user experiences.  Let’s be the users’ champions when it comes to launching new features.

When Netflix first came out with their movie recommendations, I thought it was a great idea. I started rating movies I’d seen — good and bad — confident that the brain behind it all will do its magic and recommend some hidden movie gems that will, you know, change my life. Well, I’m still waiting for those movies. And to be honest, I’ve become a little bit frustrated with the whole thing.

Describing the latest example I encountered will reveal how much I liked a movie that I probably have no business liking, but I’m willing to sacrifice a little bit of my reputation in the name of science, or whatever this is…

The first problem I encountered is a pure UI issue, and has to do with how Netflix shows the star movie ratings on their pages. As an example, this is what I see for the movie August Rush in my queue:

You would assume that the customer average rating is just over the 3-mark, right? Well, looking at it closer, it turns out that Netflix shows you a rating they call “Our best guess” (3.4 in this case), instead of showing you the customer average (4.1 in this case):

Here’s the problem. I loved this movie. I’m giving it 4 stars. But since Netflix doesn’t know that I have a soft spot for modern musicals (despite how highly I rated the movie “Once”), the “Netflix brain” didn’t think I would like this movie as much as the average customer.

This is a problem you see often on sites where the UI does not give proper user feedback about what it’s showing you.  It took me a few weeks to realize they’re showing me “Our best guess” in search results, and not the true customer average. Now I have to mouse over to see the true average every time. Why? Because I don’t trust the brain any more. (By the way, this is just one example, but as I’ve looked into it more, I realized it’s a systemic problem for me — Netflix’s best guess is rarely in line with my tastes).

Incidentally, on Amazon.com, the average user rating is 4.5 out of 5 stars. Pretty good. So this is the problem then. There is such a wide range of tastes out there that it’s hard to know who to trust. This is the problem Netflix is trying to solve — let’s look at “users like you” and then show you that average instead of the overall average. You’re therefore initially more inclined to believe the “best guess” rating provided by Netflix, than the average consensus provided by all users. It’s a good idea, but the implementation doesn’t seem to be there yet.  (The discussion about the validity of 5-star ratings in general is a separate and very interesting discussion).

I say all this to make a simple point — it appears that the collective wisdom of all users does a better job of predicting if I will like a movie than the recommendation engine provided by Netflix. The question is whether it would ever be possible for recommendation engines to get to know you well enough based on your preferences. Maybe if it takes into account not only your movie interests, but also music, books, online activity, etc.? Yes it sounds creepy, but how else would Netflix know how much I like strange modern musicals?

As designers and user experience practitioners, most of us can look at web interfaces and immediately tell the good ones from the bad ones.  The good ones are usually an indication of execution built on a collaborative and equal effort between different groups and stakeholders.  The bad ones usually point to one of two things:

• The site was chopped up and different teams owned different parts of the same pages without a clear plan for holistic design; or
• Somewhere along the line relationships turned sour, decisions got escalated, and one of the groups/stakeholders won a contentious argument about the design of the product.

I came across one such example of poor execution today while browsing the MSN Mobile web site on my iPhone.  Here is what I saw:

Notice that there are (or appears to be) four editable form fields on this screen:

1. The Safari address bar
2. The Safari Google search bar
3. The small MSN search box
4. The big Bing search box

There should be little confusion with the first two fields — they are part of the Safari browser and clearly not part of the mobile web content.  However, the other two search bars present several problems.  Notice how both have the little magnifying glass search icon, indicating that you should be able to use both to search.  The questions users will ask themselves are, which search box should I use?  Do I want to search MSN or Bing?  What’s the difference between these two search boxes?

I’m sure most of you have also figured out what’s actually going on here — the Bing search box is not a search box at all, it is a clickable advertisement (pre-filled with “Miley Cyrus” for some reason, but let’s leave that out of the discussion for now).  It’s also clear from what we know about affordance that the user is encouraged to use the Bing search box due to its prominence, size, and iconography that’s consistent with search behavior across the web.  I think we can all agree that this is just bad usability, plain and simple, but I also started thinking about how something like this could happen.  I can think of two scenarios:

• Informational content and advertising are owned by different groups. It is very likely that the advertising team were given the top banner placement on this page, with not much oversight about what they put in there.  They likely have their own designers who design for the advertising team, with no need or desire to think about the context of the entire page.
• Revenue goals trump good usability.  Another possibility is that the designers are fully aware of this issue, but that Bing, possibly a different business unit than MSN Mobile, has strong revenue goals that they have to meet for their advertising campaigns.

Or it could, of course, be a combination of these two scenarios.  I think we can learn some very important design lessons from this seemingly innocuous usability flaw:

• Never design in isolation.  This is such a simple principle, but it is still so remarkably easy to guess a company’s organizational structure just by looking at their web site.  Siloed design is one of the easiest design problems to fix, but it does take some courage: strong product management, a sincere desire to collaborate across business units, and an executive mandate to make it happen. All the MSN team had to do was get the designers/PMs together and design a Bing ad that fits with the page structure and doesn’t cannibalize search queries that should go through the MSN search box.
• Aggressive revenue goals are not an excuse for bad design. As a user experience product manager, I am a realist and completely in favor of feature prioritization and pushing for meeting aggressive revenue goals.  But revenue-generating features should never be implemented at the expense of the usability of your web site. Too often we see examples of poor implementation of an interface because the team couldn’t find a way to reconcile the business goals with proper user-centered design.  I believe the MSN Mobile example is such an occasion.

But wait, there’s more!  Unfortunately, the MSN Mobile example then goes from bad to worse.  Clicking anywhere in the Bing ad brings up this page:

There’s just not much you can say about that.  At this point you’ve lost customers who could have done their searches through MSN Mobile.  Game over, everybody loses…

I recently attended a brand presentation where the video below was shown. It’s pretty funny, and also a perfect example of how interactive products and consumer-generated content should fundamentally change our traditional views of customer loyalty. Loyalty in our current environment is fostered through repeated great (user) experiences, not just through advertising and coupons.

But even though I like the general point the video is trying to make, I think it stops a little short of the real issue. It is saying that we should listen to our customers better. But that’s not enough — we need to understand customers in ways they don’t even understand themselves, and then build experiences that meet unmet (and sometimes unconscious) needs through repeated, positive experiences that deepen the customer-company relationship.

Uncovering these needs happens not just through “Voice of the Customer” research programs, but also through more contextual research efforts like ethnography and contextual inquiries (combined with validating quantitative research). I believe this is where traditional Market Research programs like NPS (Net Promoter Score) only tell a part of the full brand loyalty story (albeit an important part, for sure).  There is evidence that the tide is turning on this topic as the field of HCI (Human-Computer Interaction) becomes more mainstream and user experience research techniques become more accessible.

There is a powerful synergy in discovering how to deepen true customer loyalty through collaborative efforts between Market Research and User Experience Research, and we need to bring these two disciplines closer together (this view is also very much in line with the thinking described in the excellent Adaptive Path essay The Long Wow).

I’ve been working on a project where we’re trying to come up with a way to establish a visual design “clutter index.”  The goal is to see if there is some threshold beyond which web page clutter impacts business metrics like conversion and click-through rates.  The challenges are widespread of course, and mainly focused on the following 3 areas:

• The definition and measurement of clutter.  There are a variety of ways to measure clutter on pages, ranging from the completely objective (e.g., % of white space on a page) to completely subjective (e.g., how do users rate the page on a clean vs. cluttered scale).
• The definition of conversion.  Since some pages on an e-commerce web site are revenue-generating, and others aren’t, an important question is how you define conversion.  For revenue-generating pages (e.g., pages with a “checkout now” button) this is easy — “Did the page result in a sale?”  For other pages, like product information pages, this measure won’t work, so some other measure of engagement with the page becomes necessary.
• Controlling for other influencing factors.  In conjunction with the first two points comes the problem of causality vs. correlation.  Assuming you have your definitions of clutter and conversion nailed down, how can you be sure any changes you see in conversion is caused by clutter (causal relationship), and not some other factor you are not accounting for (there’s correlation but no causal relationship).

The way to go about it is to take as many measurements of clutter as you can, feed them into a statistical model with a variety of conversion metrics, and see what comes out.  You also have to find a way to account for other influencing factors so that you can control for that in your model.  Easy, right?  Ok, so there are a lot of open issues, but they’re definitely not insurmountable.  I also believe it’s a worthy pursuit, the hypothesis being that there are clear business reasons for keeping designs and interfaces simple.

And apparently we’re not the only ones thinking about this…  Ruth Rosenholtz and her colleagues at MIT recently wrote a paper (Measuring Visual Clutter) where they seem to have developed what they call a “clutter detector” for a variety of interfaces, mostly offline (desk clutter, map clutter, etc.).  They describe some of their challenges in doing this as follows:

The fact that one person’s clutter is the next person’s organized workspace makes it hard to come up with a universal measure of clutter. Rosenholtz and colleagues modeled what makes items in a display harder or easier to pick out. They used this model, which incorporates data on color, contrast and orientation, to come up with a software tool to measure visual clutter.

On the issue of subjective measures of clutter:

Although there was a fair bit of disagreement among the people being tested about what constituted clutter, when the researchers compared results from their clutter measure to those of their human subjects, they found a good correlation.

I’m still digesting the paper, but it’s a fascinating read so definitely check it out.  Thoughts on how to approach this for e-commerce web pages are also more than welcome!

A recent discussion on a user experience forum I participate in turned to the topic of A/B testing.  I really enjoyed the conversation so I wanted to reiterate some of the points I made, and expand on it a little bit as well.  It’s not my goal to define A/B testing here but to share my opinion on its use.  I believe that even though A/B testing can be extremely valuable to help identify the best iteration of a site or a particular page, it should never be used in isolation.

Since A/B testing is relatively cheap to do and the results are so compelling, companies are in danger of adopting a “test and learn” culture where pages are just A/B tested with no additional user input.  That would be the wrong way to go.  A/B testing shouldn’t be used on its own to make decisions, it should always be used in conjunction with other research methods — both qualitative (such as usability testing, ethnography) and quantitative (such as desirability studies).

A/B testing is an important method in the research toolkit because it can give you information that usability testing on its own cannot.  The main goal of A/B testing is to see how business metrics move up and down depending on the version of the page — click through rates, checkout rates, purchasing rates, etc.  You can’t see that with usability testing alone.  But as Kohavi et al. point out in their paper Practical Guide to Controlled Experiments on the Web, A/B testing has some major limitations:

• Quantitative Metrics, but No Explanations. It is possible to know which variant is better, and by how much, but not why.  In user studies, for example, behavior is often augmented with users’ comments, and hence usability labs can be used to augment and complement controlled experiments.
• Short term vs. Long Term Effects. Controlled experiments measure effects during the experimentation period, typically a few weeks.   It is wise to look at delayed conversion metrics, where there is a lag from the time a user is exposed to something and take action. These are sometimes called latent conversions.
• Primacy and Newness Effects. These are opposite effects that need to be recognized. If you change the navigation on a web site, experienced users may be less efficient until they get used to the new navigation, thus giving an inherent advantage to the Control. Conversely, when a new design or feature is introduced, some users will investigate it, click everywhere, and thus introduce a “newness” bias.
• Features Must be Implemented. A live controlled experiment needs to expose some users to a Treatment different than the current site (Control). The feature may be a prototype that is being tested against a small portion, or may not cover all edge cases.  Nonetheless, the feature must be implemented and be of sufficient quality to expose users to it.
• Consistency. Users may notice they are getting a different variant than their friends and family. It is also possible that the same user will see multiple variants when using different computers (with different cookies).

As with most things, it is important to use A/B testing responsibly.   Since every research/testing method comes with its own limitations, a combination of methods is the only way to get the full picture and make the right decisions.