In How the internet is making us poor Christopher Mims asks a chilling question about what he calls the “hollowing out of the middle class” — the phenomenon where knowledge workers are being replaced by computers:

Like farming and factory work before it, the labors of the mind are being colonized by devices and systems. In the early 1800′s, nine out of ten Americans worked in agriculture—now it’s around 2%. At its peak, about a third of the US population was employed in manufacturing—now it’s less than 10%. How many decades until the figures are similar for the information-processing tasks that typify rich countries’ post-industrial economies?

The article also quotes this thought-provoking statement from Marc Andreessen:

The spread of computers and the Internet will put jobs in two categories: People who tell computers what to do, and people who are told by computers what to do.

It might seem like the usual doom-and-gloom “technology will kill as all” refrain, but the article reviews some very interesting historical (and current) data, so it’s worth checking out.

Figuring out what to read (and what to believe) about Big Data is becoming a Big Data problem in and of itself1. I wrote The hype, benefits, and dangers of Big Data a while ago to give an overview of what’s out there, but there are two more interesting articles from the last week that I’d like to highlight as well.

First, on the HBR blog Jake Porway talks about Big Data and social entrepreneurship and makes the point that You Can’t Just Hack Your Way to Social Change:

Any data scientist worth their salary will tell you that you should start with a question, NOT the data. Unfortunately, data hackathons often lack clear problem definitions. Most companies think that if you can just get hackers, pizza, and data together in a room, magic will happen. This is the same as if Habitat for Humanity gathered its volunteers around a pile of wood and said, “Have at it!” By the end of the day you’d be left with half of a sunroom with 14 outlets in it.

And on Wired, Does ‘Big Data’ Mean the Demise of the Expert — And Intuition? is a very interesting excerpt from Viktor Mayer-Schönberger and Kenneth Cukier’s new book on the topic:

In the same spirit, the biggest impact of big data will be that data-driven decisions are poised to augment or overrule human judgment.

The subject-area expert, the substantive specialist, will lose some of his or her luster compared with the statistician and data analyst, who are unfettered by the old ways of doing things and let the data speak. This new cadre will rely on correlations without prejudgments and prejudice. To be sure, subject-area experts won’t die out, but their supremacy will ebb. From now on, they must share the podium with the big-data geeks, just as princely causation must share the limelight with humble correlation.

It seems like an obvious conclusion, but everything I’ve read so far about Big Data confirms that if we think cutting the “messiness” of human decision-making out of data analysis will result in better decisions, we’re sorely mistaken.

---

1. Sorry, I didn’t get much sleep last night, so even though I know this isn’t a particularly funny joke, I just can’t help myself. ↩

A Readlist of all the articles referenced in this post is available here. Readlists allow you to send all the articles to your Kindle, read them on your iOS device, or download it as an e-book.

Despite the overly alarmist title, Andrew Leonard’s How Netflix is turning viewers into puppets1 is a fascinating article on how Netflix uses Big Data in their programming decisions:

“House of Cards” is one of the first major test cases of this Big Data-driven creative strategy. For almost a year, Netflix executives have told us that their detailed knowledge of Netflix subscriber viewing preferences clinched their decision to license a remake of the popular and critically well regarded 1990 BBC miniseries. Netflix’s data indicated that the same subscribers who loved the original BBC production also gobbled down movies starring Kevin Spacey or directed by David Fincher. Therefore, concluded Netflix executives, a remake of the BBC drama with Spacey and Fincher attached was a no-brainer, to the point that the company committed $100 million for two 13-episode seasons.

The article also asks what this approach means for the creative process, something I’ve written about before in The unnecessary fear of digital perfection, so I won’t rehash that argument here.

What’s interesting to me about the rise in Big Data approaches to decision-making is the high levels of inaccuracy inherent to the analysis process. Of course, this is something we don’t hear about often, but Nassim N. Taleb recently wrote a great opinion piece about it for Wired called Beware the Big Errors of ‘Big Data’, in which he states:

Big-data researchers have the option to stop doing their research once they have the right result. In options language: The researcher gets the “upside” and truth gets the “downside.” It makes him antifragile, that is, capable of benefiting from complexity and uncertainty — and at the expense of others.

But beyond that, big data means anyone can find fake statistical relationships, since the spurious rises to the surface. This is because in large data sets, large deviations are vastly more attributable to variance (or noise) than to information (or signal). It’s a property of sampling: In real life there is no cherry-picking, but on the researcher’s computer, there is. Large deviations are likely to be bogus.

He gets into more detail on the statistical problems with Big Data in the article, and his book Antifragile looks really interesting too.

Since I haven’t written about Big Data before, I also want to reference a few articles on the topic that I enjoyed. Sean Madden gives some interesting real world examples in How Companies Like Amazon Use Big Data To Make You Love Them2. But over on the skeptical side, Stephen Few argues in Big Data, Big Deal that “interest in big data today is a direct result of vendor marketing; it didn’t emerge naturally from the needs of users.” He also makes the point that data has always been big, and that by focusing on the “bigness” of it, we’re missing the point:

A little more and a little faster have always been on our wish list. While information technology has struggled to catch up, mostly by pumping itself up with steroids, it has lost sight of the objective: to better understand the world—at least one’s little part of it (e.g., one’s business)—so we can make it better. Our current fascination with big data has us looking for better steroids to increase our brawn rather than better skills to develop our brains. In the world of analytics, brawn will only get us so far; it is better thinking that will open the door to greater insight.

Alan Mitchell makes a similar point in Big Data, Big Dead End, a case for what he calls Small Data:

But if we look at the really big value gap faced by society nowadays, it’s not the ability to crunch together vast amounts of data, but quite the opposite. It’s the challenge of information logistics: of how to get exactly the right information to, and from, the right people in the right formats at the right time. This is about Very Small Data: discarding or leaving aside the 99.99% of information I don’t need right now so that I can use the 0.01% of information that I do need as quickly and efficiently as possible.

What I think we should take from all of this is that our ability to collect vast amounts of data comes with enormous predictive and analytical upside. But we’d be foolish to think that it makes decision-making easier. Because Big Data does not take away the biggest challenge of data analysis: figuring how to turn data into information, and information into knowledge. In fact, Big Data makes this harder. To quote Nassim again:

I am not saying here that there is no information in big data. There is plenty of information. The problem — the central issue — is that the needle comes in an increasingly larger haystack.

In other words: proceed with caution.

---

1. Link via @mobivangelist ↩

2. It’s interesting that the phrasing of both this headline and the Netflix one implies that companies are using Big Data to persuade us to do things against our will. But I can’t figure out if that’s a real fear, or just clever linkbait. ↩

In Bridging the gap between humans and computers Heather Kelly takes a look at some recent ethnographic research on our relationship with technology. It’s full of interesting stories like this one:

In one experiment, Ju’s group rigged automatic doors to open in different ways: Some would open slowly, then pause before fully opening; others would immediately jerk all the way open. The people walking by the doors assigned them different levels of intelligence, and thought the doors that opened in two steps just seemed smarter.

It turned out that adding the pause gave illusion of forethought, even though it was just an extra programming step. People thought the door was more intelligent because it appeared to think before carrying out an action.

One of my favorite books on this topic is Sherry Turkle’s Alone Together: Why We Expect More from Technology and Less from Each Other. The first half of the book is all about our relationship with high tech “things” — what we find creepy vs. comforting, how different cultures behave differently, etc. Highly recommended.

In Living inside the Machine James Bridle writes about computers and data centres as aesthetic objects. It’s a very interesting idea and a great article. There’s one part in particular that stuck with me. James quotes William Gibson in an interview with the Paris Review from 2011, about his time in Vancouver in the late 70s/early 80s:

The only computers I’d ever seen in those days were things the size of the side of a barn. And then one day, I walked by a bus stop and there was an Apple poster. The poster was a photograph of a businessman’s jacketed, neatly cuffed arm holding a life-size representation of a real-life computer that was not much bigger than a laptop is today. Everyone is going to have one of these, I thought, and everyone is going to want to live inside them.

Everyone is going to have one of these, and everyone is going to want to live inside them. How prophetic…

James sums it up nicely in his article:

We used to posit this space, the network, the notional space, as being elsewhere, the other side of the screen. But increasingly we have these images of the machine as something that surrounds us, that we live inside, within. As something that enfolds us.

At first it’s hard to figure out what the title of Nicholas Carr’s A post on the occasion of Facebook’s billionth member has to do with Facebook. Especially since he hardly even mentions Facebook. It appears to be an essay about boredom and computer intelligence:

We’ll know that computers are really smart when computers start getting bored. If you assign a computer a profoundly tedious task like spotting potential house numbers in video images, and then you come back a couple of hours later and find that the computer is checking its Facebook feed or surfing porn, then you’ll know that artificial intelligence has truly arrived.

But stick with it. It all makes sense once you get to the end and reflect on the words for a couple of hours. Also, full marks to Parampreet Singh for a comment that references Sisyphus, and compares his plight (“to roll an immense boulder up a hill, only to watch it roll back down, and to repeat this action forever”) with our tendency to check our Facebook feeds constantly.

Nicholas Carr wrote an excellent, balanced article on the rise of massive open online courses (MOOCs1) like Coursera and Udacity, and the complex data mining required to make it work. From The Crisis in Higher Education:

The advances in tutoring programs promise to help many college, high-school, and even elementary students master basic concepts. One-on-one instruction has long been known to provide substantial educational benefits, but its high cost has constrained its use, particularly in public schools. It’s likely that if computers are used in place of teachers, many more students will be able to enjoy the benefits of tutoring. According to one recent study of undergraduates taking statistics courses at public universities, the latest of the online tutoring systems seem to produce roughly the same results as face-to-
> face instruction.

This is some really in-depth reporting, and it’s not all sunshine and roses. Nicholas went out of his way to seek out and report on legitimate counterarguments to this movement as well.

---

1. Yes, really. ↩

Bret Victor wrote a very interesting rant a few days ago on the the problem with touch interfaces and the future of Interaction Design. The piece got a lot of attention, so today he followed up with some responses to the questions and comments he received.

I particularly enjoyed his thoughts on the limits of voice control. His argument is that voice is a good way to get information or issue commands (yes, like Siri), but that it’s not very good for creating and understanding:

I have a hard time imagining Monet saying to his canvas, “Give me some water lilies. Make ‘em impressionistic.” Or designing a building by telling all the walls where to go. Most artistic and engineering projects (at least, non-language-based ones) can’t just be described. They exist in space, and we manipulate space with our hands.

It’s obvious, yes, but I think we need to remind ourselves of this. Creating things requires “manipulating space with our hands”, even if that means manipulating words onto a page when they’re stubbornly stuck in space somewhere.[1]

---

1. Sure, some people (like John Siracusa) are able to dictate the first drafts of stuff they write, but I’m pretty sure they’re not editing their work through voice control. Editing (which is the hardest part of writing) requires a keyboard and lots of banging your head on it.  â†©

W. Brian Arthur wrote a very interesting article for McKinsey Quarterly called The second economy (h/t to @justinspratt for the link). Registration is required to view the article but it’s worth it.

Much has been written about digitization and technology’s impact on society, but Arthur takes a fresh approach by looking at the digital economy as an unseen layer underneath the physical economy. He starts by defining communication for this (second) economy:

[Processes] are “speaking to” other processes in the digital economy, in a constant conversation among multiple servers and multiple semi-intelligent nodes that are updating things, querying things, checking things off, readjusting things, and eventually connecting back with processes and humans in the physical economy.

You know, like Siri does. In fact, notice how perfectly Siri fits into Arthur’s central thesis about the second economy:

If I were to look for adjectives to describe this second economy, I’d say it is vast, silent, connected, unseen, and autonomous (meaning that human beings may design it but are not directly involved in running it). It is remotely executing and global, always on, and endlessly configurable. It is concurrent””a great computer expression””which means that everything happens in parallel. It is self-configuring, meaning it constantly reconfigures itself on the fly, and increasingly it is also self-organizing, self-architecting, and self-healing.

These last descriptors sound biological””and they are. In fact, I’m beginning to think of this second economy, which is under the surface of the physical economy, as a huge interconnected root system, very much like the root system for aspen trees. For every acre of aspen trees above the ground, ther’s about ten miles of roots underneath, all interconnected with one another, “communicating” with each other.

Arthur makes it clear that he’s not interested in the realm of Sci-Fi and AI. He’s not sharing a completely improbable vision of the future (well, with the exception of driverless cars, depending on how much of a Google believer you are). And even though nothing he describes is brand new, this idea of a silent, interconnected layer underneath the physical one gives us a new lens through which to view the digitization of our lives.

I don’t want to get all “The End Is Near!” on you, but I’m currently reading Sherry Turkle’s Alone Together - Why We Expect More from Technology and Less from Each Other, and Arthur’s article reminded me of her words of caution:

Now demarcations blur as technology accompanies us everywhere, all the time. We are too quick to celebrate the continual presence of a technology that knows no respect for traditional and helpful lines in the sand.

[A] stream of messages makes it impossible to find moments of solitude, time when other people are showing us neither dependency nor affection. In solitude we don’t reject the world but have the space to think our own thoughts. But if your phone is always with you, seeking solitude can look suspiciously like hiding.

Hopefully there will still be places to hide once the second economy has fully established itself.

When Netflix first came out with their movie recommendations, I thought it was a great idea. I started rating movies I’d seen — good and bad — confident that the brain behind it all will do its magic and recommend some hidden movie gems that will, you know, change my life. Well, I’m still waiting for those movies. And to be honest, I’ve become a little bit frustrated with the whole thing.

Describing the latest example I encountered will reveal how much I liked a movie that I probably have no business liking, but I’m willing to sacrifice a little bit of my reputation in the name of science, or whatever this is…

The first problem I encountered is a pure UI issue, and has to do with how Netflix shows the star movie ratings on their pages. As an example, this is what I see for the movie August Rush in my queue:

You would assume that the customer average rating is just over the 3-mark, right? Well, looking at it closer, it turns out that Netflix shows you a rating they call “Our best guess” (3.4 in this case), instead of showing you the customer average (4.1 in this case):

Here’s the problem. I loved this movie. I’m giving it 4 stars. But since Netflix doesn’t know that I have a soft spot for modern musicals (despite how highly I rated the movie “Once”), the “Netflix brain” didn’t think I would like this movie as much as the average customer.

This is a problem you see often on sites where the UI does not give proper user feedback about what it’s showing you.  It took me a few weeks to realize they’re showing me “Our best guess” in search results, and not the true customer average. Now I have to mouse over to see the true average every time. Why? Because I don’t trust the brain any more. (By the way, this is just one example, but as I’ve looked into it more, I realized it’s a systemic problem for me — Netflix’s best guess is rarely in line with my tastes).

Incidentally, on Amazon.com, the average user rating is 4.5 out of 5 stars. Pretty good. So this is the problem then. There is such a wide range of tastes out there that it’s hard to know who to trust. This is the problem Netflix is trying to solve — let’s look at “users like you” and then show you that average instead of the overall average. You’re therefore initially more inclined to believe the “best guess” rating provided by Netflix, than the average consensus provided by all users. It’s a good idea, but the implementation doesn’t seem to be there yet.  (The discussion about the validity of 5-star ratings in general is a separate and very interesting discussion).

I say all this to make a simple point — it appears that the collective wisdom of all users does a better job of predicting if I will like a movie than the recommendation engine provided by Netflix. The question is whether it would ever be possible for recommendation engines to get to know you well enough based on your preferences. Maybe if it takes into account not only your movie interests, but also music, books, online activity, etc.? Yes it sounds creepy, but how else would Netflix know how much I like strange modern musicals?