I read quite a few excellent UX/PM posts this week, and wanted to make sure you don’t miss out.  So here are some excerpts from my favorite posts of the week.

User Experience Design in the Agile context

In Agile UX and The One Change That Changes Everything, Anders Ramsy writes about how user experience design can be adopted to fit the agile mold a little better.  He calls for less design up-front to basically embrace the MVP approach instead of fighting it:

The first and probably most fundamental change to flow out of starting to build earlier is that of chopping your up-front design phase down to a fraction of what it might be in a traditional model to allow for establishing a foundation of working software, and then evolving the rest of the product on top of that foundation. In other words, we go from Big Design Up Front to Just Enough Design Up Front.

The rest of the post is devoted to how to do that, including thoughts on lighter, conversation-centered documentation, and the importance of collaborative design.

Enough with the “chicken & pig” story

Speaking of Agile, David Bland wrote an impassioned post arguing that Our Divisive Scrum Terminology Needs to be Deprecated:

Scrum teams succeed or fail as a, well, a team.

If the Product Owner is confused about the role or not living up to expectations, it is the ScrumMaster who should be helping them along the way. If the ScrumMaster is failing at coaching up the Product Owner on the framework, then wouldn’t the ScrumMaster be to blame? But wait, since the team has appointed the ScrumMaster, would they not have failed by choosing one who is incompetent?

W’ll just run in circles pointing fingers because there is no easy answer, and using the Product Owner as the scape goat does nothing to help resolve the situation.

Measurement-driven Product Management

The always brilliant Pragmatic Marketing has a post entitled Measurement-Driven Product Management that should make all of us a little uncomfortable.  But good uncomfortable.  Getting better at your job uncomfortable.  Read the post for details on the proposed ways to measure the success of PM, but this is why they make the case for it:

The long term benefit of Product Management becoming measurement-driven is higher team performance, improved predictability and increased credibility. The ultimate benefit is developing the ability to reliably create outstanding products and market breakthroughs.

Can Product Management operate with this high level of maturity, using a reliable measurements and metrics system with more predictable results in a company?

This “holy grail” of product management performance is doable, but often many cultural and process gaps must be addressed first. An organization fosters a measurement-driven culture by reinforcing other aspects of the process, such as tightly coupling rewards, recognition, compensation and promotion to attainment of operational results. Does yours?

Research and Design, sitting in a tree…

In The product of a healthy relationship, Paul Golden discusses the sometimes rocky relationship between researchers and designers:

Hana Thomas of design consultancy Smallfry agrees that while market research can play a crucial role in product design and development, there are dangers. “There can be an over-reliance on market research, which leads to it being used either as a scapegoat for poor decisions or employed too soon in the creative process, stopping ideas in their tracks before they have even had the chance to be realised.”

Thomas refers to the value of ethnographic research to her company’s work in product development and describes studying people in their own environment, under a relevant context, as the “ideal way” to truly unearth latent needs and desires.

According to Reon Brand, the responsive and listening brand that engages its audiences in the creative process as well as in dialogue has a major advantage in our increasingly social-media driven world. However, all research methodologies have their limitations. While consumers can react to what exists and relate back to what they know, some of the designers surveyed by the Design Council felt that consumers were less able to contribute to the development of completely new product or service concepts for the future.

I just became the mayor of someplace you’ve never heard of

On a slightly different note, I found this RWW called Why We Check In: The Reasons People Use Location-Based Social Networks very interesting.  It presents some research on why we use services like Foursquare and Gowalla, and there are definitely some surprises, like using it to keep track of history:

The thing that surprised me most when I asked people why they use location-based social networks is how many of them say they use it primarily to track their own personal history. It’s a lazy diary, people say.  Some people say they use it to help with their expense tracking on business travels.

Happy reading!

When Netflix first came out with their movie recommendations, I thought it was a great idea. I started rating movies I’d seen — good and bad — confident that the brain behind it all will do its magic and recommend some hidden movie gems that will, you know, change my life. Well, I’m still waiting for those movies. And to be honest, I’ve become a little bit frustrated with the whole thing.

Describing the latest example I encountered will reveal how much I liked a movie that I probably have no business liking, but I’m willing to sacrifice a little bit of my reputation in the name of science, or whatever this is…

The first problem I encountered is a pure UI issue, and has to do with how Netflix shows the star movie ratings on their pages. As an example, this is what I see for the movie August Rush in my queue:

You would assume that the customer average rating is just over the 3-mark, right? Well, looking at it closer, it turns out that Netflix shows you a rating they call “Our best guess” (3.4 in this case), instead of showing you the customer average (4.1 in this case):

Here’s the problem. I loved this movie. I’m giving it 4 stars. But since Netflix doesn’t know that I have a soft spot for modern musicals (despite how highly I rated the movie “Once”), the “Netflix brain” didn’t think I would like this movie as much as the average customer.

This is a problem you see often on sites where the UI does not give proper user feedback about what it’s showing you.  It took me a few weeks to realize they’re showing me “Our best guess” in search results, and not the true customer average. Now I have to mouse over to see the true average every time. Why? Because I don’t trust the brain any more. (By the way, this is just one example, but as I’ve looked into it more, I realized it’s a systemic problem for me — Netflix’s best guess is rarely in line with my tastes).

Incidentally, on Amazon.com, the average user rating is 4.5 out of 5 stars. Pretty good. So this is the problem then. There is such a wide range of tastes out there that it’s hard to know who to trust. This is the problem Netflix is trying to solve — let’s look at “users like you” and then show you that average instead of the overall average. You’re therefore initially more inclined to believe the “best guess” rating provided by Netflix, than the average consensus provided by all users. It’s a good idea, but the implementation doesn’t seem to be there yet.  (The discussion about the validity of 5-star ratings in general is a separate and very interesting discussion).

I say all this to make a simple point — it appears that the collective wisdom of all users does a better job of predicting if I will like a movie than the recommendation engine provided by Netflix. The question is whether it would ever be possible for recommendation engines to get to know you well enough based on your preferences. Maybe if it takes into account not only your movie interests, but also music, books, online activity, etc.? Yes it sounds creepy, but how else would Netflix know how much I like strange modern musicals?

With the three recent big stories in Facebookland (the FriendFeed aquisition, real-time search, and now the test launch of Facebook Lite) it doesn’t take a rocket scientist to figure out that Facebook is going hard after Twitter. (Update 1/16/2010: Facebook just rolled out “via” as their version of Twitter’s “retweet”. That, combined with recent changes to their privacy policy to make the platform more open, are two more clear examples of Facebook’s “Become Twitter” strategy)

What is more difficult to understand is why they are doing it.  Maybe it’s a personal vendetta because of the failed acquisition talks?  I just don’t see the business reason for this.  Here’s why I think Facebook should forget about Twitter and focus on making its own platform great:

Different target markets

It is well known that Twitter skews heavily towards younger tech-savvy users, with the rest of the population finding it hard to see the point.  Facebook, on the other hand, is increasingly being used by an older demographic.  The fastest growing demographic on Facebook is women over 55.

Why is all this important?  Because regardless of what Facebook wants to be, the demographic that is settling in on the site for the long haul is different from the Twitter user base — and they have totally different needs and behaviors. At this point, Facebook is too established as a brand to be able to force their product onto the target market they want.  And why would they even want to?  They have access to a much larger user base than Twitter.  Which brings me to my next point…

Always compete on your strengths

The mistake that Facebook is making is that it is trying to be Twitter for a user base that does not want Twitter.  Not convinced?  Go to http://www.brandtags.net and look at the brand clouds of word associations that people make with Facebook and Twitter.  For Facebook, you get words like Communication, People, Stalking.  For Twitter, you get words like Pointless, Stupid, Useless.

Now, of course Twitter is none of those things, but it shows the enormous gap in brand perceptions.  Why would you want to move a powerful people connection platform closer to something with a niche market that a majority of people find useless? There are a bunch of other Twitter statistics coming out lately that prove the Twitter niche factor: 5% of users account for 75% of the activity, 60% of US Twitter users abandon the site after a month, and 24% of all tweets are from bots (ok, that last one is irrelevant to this discussion, but still interesting).  And there’s also this interesting conversation on Mashable that clearly shows the differences between Twitter and Facebook usage.

The bottom line is that Twitter is for information sharing, Facebook is for life sharing.  That is what people are using it for — sharing photos, videos, those annoying pokes and quizzes, keeping in touch with friends all over the globe, lurking on profiles of people you used to know way back when.  That is the strength of Facebook, and that is what they should focus their platform on.

So what should Facebook do?

So here is my advice to Facebook: go where your users are.  Understand how they use the site, what their needs and behaviors are.  Go visit them, talk to them, watch them navigate around, understand why they are there in the first place.  And then enhance your platform to fulfill those needs.  Build new ways to feel closer to the people in your life.  Make it easier to share and discuss media.  Build families-only mini-communities.  Who knows what you can come up with if you just understand your users and build a web site for their needs?

Seriously — let Twitter be Twitter, forget about them and don’t force your users into that kind of experience.  Don’t try to be “status updates for everyone.”  Be a platform that lives up to the value proposition on your home page: “Facebook helps you connect and share with the people in your life.”

I recently attended a brand presentation where the video below was shown. It’s pretty funny, and also a perfect example of how interactive products and consumer-generated content should fundamentally change our traditional views of customer loyalty. Loyalty in our current environment is fostered through repeated great (user) experiences, not just through advertising and coupons.

But even though I like the general point the video is trying to make, I think it stops a little short of the real issue. It is saying that we should listen to our customers better. But that’s not enough — we need to understand customers in ways they don’t even understand themselves, and then build experiences that meet unmet (and sometimes unconscious) needs through repeated, positive experiences that deepen the customer-company relationship.

Uncovering these needs happens not just through “Voice of the Customer” research programs, but also through more contextual research efforts like ethnography and contextual inquiries (combined with validating quantitative research). I believe this is where traditional Market Research programs like NPS (Net Promoter Score) only tell a part of the full brand loyalty story (albeit an important part, for sure).  There is evidence that the tide is turning on this topic as the field of HCI (Human-Computer Interaction) becomes more mainstream and user experience research techniques become more accessible.

There is a powerful synergy in discovering how to deepen true customer loyalty through collaborative efforts between Market Research and User Experience Research, and we need to bring these two disciplines closer together (this view is also very much in line with the thinking described in the excellent Adaptive Path essay The Long Wow).

I’ve been working on a project where we’re trying to come up with a way to establish a visual design “clutter index.”  The goal is to see if there is some threshold beyond which web page clutter impacts business metrics like conversion and click-through rates.  The challenges are widespread of course, and mainly focused on the following 3 areas:

• The definition and measurement of clutter.  There are a variety of ways to measure clutter on pages, ranging from the completely objective (e.g., % of white space on a page) to completely subjective (e.g., how do users rate the page on a clean vs. cluttered scale).
• The definition of conversion.  Since some pages on an e-commerce web site are revenue-generating, and others aren’t, an important question is how you define conversion.  For revenue-generating pages (e.g., pages with a “checkout now” button) this is easy — “Did the page result in a sale?”  For other pages, like product information pages, this measure won’t work, so some other measure of engagement with the page becomes necessary.
• Controlling for other influencing factors.  In conjunction with the first two points comes the problem of causality vs. correlation.  Assuming you have your definitions of clutter and conversion nailed down, how can you be sure any changes you see in conversion is caused by clutter (causal relationship), and not some other factor you are not accounting for (there’s correlation but no causal relationship).

The way to go about it is to take as many measurements of clutter as you can, feed them into a statistical model with a variety of conversion metrics, and see what comes out.  You also have to find a way to account for other influencing factors so that you can control for that in your model.  Easy, right?  Ok, so there are a lot of open issues, but they’re definitely not insurmountable.  I also believe it’s a worthy pursuit, the hypothesis being that there are clear business reasons for keeping designs and interfaces simple.

And apparently we’re not the only ones thinking about this…  Ruth Rosenholtz and her colleagues at MIT recently wrote a paper (Measuring Visual Clutter) where they seem to have developed what they call a “clutter detector” for a variety of interfaces, mostly offline (desk clutter, map clutter, etc.).  They describe some of their challenges in doing this as follows:

The fact that one person’s clutter is the next person’s organized workspace makes it hard to come up with a universal measure of clutter. Rosenholtz and colleagues modeled what makes items in a display harder or easier to pick out. They used this model, which incorporates data on color, contrast and orientation, to come up with a software tool to measure visual clutter.

On the issue of subjective measures of clutter:

Although there was a fair bit of disagreement among the people being tested about what constituted clutter, when the researchers compared results from their clutter measure to those of their human subjects, they found a good correlation.

I’m still digesting the paper, but it’s a fascinating read so definitely check it out.  Thoughts on how to approach this for e-commerce web pages are also more than welcome!

A recent discussion on a user experience forum I participate in turned to the topic of A/B testing.  I really enjoyed the conversation so I wanted to reiterate some of the points I made, and expand on it a little bit as well.  It’s not my goal to define A/B testing here but to share my opinion on its use.  I believe that even though A/B testing can be extremely valuable to help identify the best iteration of a site or a particular page, it should never be used in isolation.

Since A/B testing is relatively cheap to do and the results are so compelling, companies are in danger of adopting a “test and learn” culture where pages are just A/B tested with no additional user input.  That would be the wrong way to go.  A/B testing shouldn’t be used on its own to make decisions, it should always be used in conjunction with other research methods — both qualitative (such as usability testing, ethnography) and quantitative (such as desirability studies).

A/B testing is an important method in the research toolkit because it can give you information that usability testing on its own cannot.  The main goal of A/B testing is to see how business metrics move up and down depending on the version of the page — click through rates, checkout rates, purchasing rates, etc.  You can’t see that with usability testing alone.  But as Kohavi et al. point out in their paper Practical Guide to Controlled Experiments on the Web, A/B testing has some major limitations:

• Quantitative Metrics, but No Explanations. It is possible to know which variant is better, and by how much, but not why.  In user studies, for example, behavior is often augmented with users’ comments, and hence usability labs can be used to augment and complement controlled experiments.
• Short term vs. Long Term Effects. Controlled experiments measure effects during the experimentation period, typically a few weeks.   It is wise to look at delayed conversion metrics, where there is a lag from the time a user is exposed to something and take action. These are sometimes called latent conversions.
• Primacy and Newness Effects. These are opposite effects that need to be recognized. If you change the navigation on a web site, experienced users may be less efficient until they get used to the new navigation, thus giving an inherent advantage to the Control. Conversely, when a new design or feature is introduced, some users will investigate it, click everywhere, and thus introduce a “newness” bias.
• Features Must be Implemented. A live controlled experiment needs to expose some users to a Treatment different than the current site (Control). The feature may be a prototype that is being tested against a small portion, or may not cover all edge cases.  Nonetheless, the feature must be implemented and be of sufficient quality to expose users to it.
• Consistency. Users may notice they are getting a different variant than their friends and family. It is also possible that the same user will see multiple variants when using different computers (with different cookies).

As with most things, it is important to use A/B testing responsibly.   Since every research/testing method comes with its own limitations, a combination of methods is the only way to get the full picture and make the right decisions.