W. Brian Arthur wrote a very interesting article for McKinsey Quarterly called The second economy (h/t to @justinspratt for the link). Registration is required to view the article but it’s worth it.

Much has been written about digitization and technology’s impact on society, but Arthur takes a fresh approach by looking at the digital economy as an unseen layer underneath the physical economy. He starts by defining communication for this (second) economy:

[Processes] are “speaking to” other processes in the digital economy, in a constant conversation among multiple servers and multiple semi-intelligent nodes that are updating things, querying things, checking things off, readjusting things, and eventually connecting back with processes and humans in the physical economy.

You know, like Siri does. In fact, notice how perfectly Siri fits into Arthur’s central thesis about the second economy:

If I were to look for adjectives to describe this second economy, I’d say it is vast, silent, connected, unseen, and autonomous (meaning that human beings may design it but are not directly involved in running it). It is remotely executing and global, always on, and endlessly configurable. It is concurrent””a great computer expression””which means that everything happens in parallel. It is self-configuring, meaning it constantly reconfigures itself on the fly, and increasingly it is also self-organizing, self-architecting, and self-healing.

These last descriptors sound biological””and they are. In fact, I’m beginning to think of this second economy, which is under the surface of the physical economy, as a huge interconnected root system, very much like the root system for aspen trees. For every acre of aspen trees above the ground, ther’s about ten miles of roots underneath, all interconnected with one another, “communicating” with each other.

Arthur makes it clear that he’s not interested in the realm of Sci-Fi and AI. He’s not sharing a completely improbable vision of the future (well, with the exception of driverless cars, depending on how much of a Google believer you are). And even though nothing he describes is brand new, this idea of a silent, interconnected layer underneath the physical one gives us a new lens through which to view the digitization of our lives.

I don’t want to get all “The End Is Near!” on you, but I’m currently reading Sherry Turkle’s Alone Together - Why We Expect More from Technology and Less from Each Other, and Arthur’s article reminded me of her words of caution:

Now demarcations blur as technology accompanies us everywhere, all the time. We are too quick to celebrate the continual presence of a technology that knows no respect for traditional and helpful lines in the sand.

[A] stream of messages makes it impossible to find moments of solitude, time when other people are showing us neither dependency nor affection. In solitude we don’t reject the world but have the space to think our own thoughts. But if your phone is always with you, seeking solitude can look suspiciously like hiding.

Hopefully there will still be places to hide once the second economy has fully established itself.

When Netflix first came out with their movie recommendations, I thought it was a great idea. I started rating movies I’d seen — good and bad — confident that the brain behind it all will do its magic and recommend some hidden movie gems that will, you know, change my life. Well, I’m still waiting for those movies. And to be honest, I’ve become a little bit frustrated with the whole thing.

Describing the latest example I encountered will reveal how much I liked a movie that I probably have no business liking, but I’m willing to sacrifice a little bit of my reputation in the name of science, or whatever this is…

The first problem I encountered is a pure UI issue, and has to do with how Netflix shows the star movie ratings on their pages. As an example, this is what I see for the movie August Rush in my queue:

You would assume that the customer average rating is just over the 3-mark, right? Well, looking at it closer, it turns out that Netflix shows you a rating they call “Our best guess” (3.4 in this case), instead of showing you the customer average (4.1 in this case):

Here’s the problem. I loved this movie. I’m giving it 4 stars. But since Netflix doesn’t know that I have a soft spot for modern musicals (despite how highly I rated the movie “Once”), the “Netflix brain” didn’t think I would like this movie as much as the average customer.

This is a problem you see often on sites where the UI does not give proper user feedback about what it’s showing you.  It took me a few weeks to realize they’re showing me “Our best guess” in search results, and not the true customer average. Now I have to mouse over to see the true average every time. Why? Because I don’t trust the brain any more. (By the way, this is just one example, but as I’ve looked into it more, I realized it’s a systemic problem for me — Netflix’s best guess is rarely in line with my tastes).

Incidentally, on Amazon.com, the average user rating is 4.5 out of 5 stars. Pretty good. So this is the problem then. There is such a wide range of tastes out there that it’s hard to know who to trust. This is the problem Netflix is trying to solve — let’s look at “users like you” and then show you that average instead of the overall average. You’re therefore initially more inclined to believe the “best guess” rating provided by Netflix, than the average consensus provided by all users. It’s a good idea, but the implementation doesn’t seem to be there yet.  (The discussion about the validity of 5-star ratings in general is a separate and very interesting discussion).

I say all this to make a simple point — it appears that the collective wisdom of all users does a better job of predicting if I will like a movie than the recommendation engine provided by Netflix. The question is whether it would ever be possible for recommendation engines to get to know you well enough based on your preferences. Maybe if it takes into account not only your movie interests, but also music, books, online activity, etc.? Yes it sounds creepy, but how else would Netflix know how much I like strange modern musicals?