Nicholas Carr wrote a really interesting article on the dangers of computer automation. In All Can Be Lost: The Risk of Putting Our Knowledge in the Hands of Machines he weaves together stories about airline crashes and Inuit hunters to make salient points like this:

Psychologists have found that when we work with computers, we often fall victim to two cognitive ailments — complacency and bias — that can undercut our performance and lead to mistakes. Automation complacency occurs when a computer lulls us into a false sense of security. Confident that the machine will work flawlessly and handle any problem that crops up, we allow our attention to drift. We become disengaged from our work, and our awareness of what’s going on around us fades. Automation bias occurs when we place too much faith in the accuracy of the information coming through our monitors. Our trust in the software becomes so strong that we ignore or discount other information sources, including our own eyes and ears. When a computer provides incorrect or insufficient data, we remain oblivious to the error.

Carr goes to great lengths to make the argument that the automation of tasks is slowly robbing us of the ability to learn new skills. It is, in some ways, a more nuanced argument than his famous 2008 article Is Google Making Us Stupid? It’s well worth reading — even if you’re skeptical of this argument, it will definitely make you think.

Yesterday I posted The problem with letting algorithms make most of our decisions, discussing how removing all knowledge obstacles can make us less adept at dealing with challenges. As is often the case, within a few hours of posting that I came across two more articles that addresses the same issues. First, from Kyle Baxter’s very interesting essay On the Philosophy of Google Glass:

Page’s idea — that we would be fundamentally better off if we had immediate access to all of humanity’s information — ignores [how we develop knowledge]. It provides facts, but elides conclusions and connections. What’s worse, it starves us of opportunities to use our skill for critical thinking, and since it is a skill and is therefore something that must be developed and practiced, it starves us of the chance to develop it.

I find that troubling. Glass is not a technology that is designed to amplify our own innate abilities as humans or to make us better as humans, but rather one that acts as a crutch to lean on in place of exercising the very thing that makes us human. I don’t find that exciting. I find that disturbing.

And then, from Smart cities and smart citizens, an editorial in Sustain Magazine (which I’ll reference more over the coming days):

Furthermore, [Dan Hill, CEO of Fabrica] argues that current smart-systems thinking could lead us down a dangerous path towards passive citizens. As citizens — and city leaders — devolve their decision-making and responsibility to technology, their awareness of their environment diminishes in line with their ability to do something about it.

“If you automate too much stuff, people stop thinking about the issues. Yes, it might be more efficient to make the lights go off automatically, but it stops us thinking about it, we’re not engaged — and when we’re disengaged that’s not a good idea. We want people to think about something like carbon. Besides, we can turn the lights off on the way out — it’s entirely possible, we’re quite a smart species potentially!”

I find it fascinating how the Internet sometimes feel like one organism, always thinking and debating the same issues from many different angles. From Google Glass to Architecture to self-driving cars, it seems that currently we’re collectively worried about the impact of smart technologies on our lives.

Image source: Knight Rider’s KITT - My finished replica!

Nicholas Carr asks some serious questions about things like self-driving cars and our increased reliance on algorithms for decision-making in Moral code:

As we begin to have computer-controlled cars, robots, and other machines operating autonomously out in the chaotic human world, situations will inevitably arise in which the software has to choose between a set of bad, even horrible, alternatives. How do you program a computer to choose the lesser of two evils? What are the criteria, and how do you weigh them?

Clive Thompson picks up the thread in a very interesting Wired article called Relying on Algorithms and Bots Can Be Really, Really Dangerous:

The truth is, our tools increasingly guide and shape our behavior or even make decisions on our behalf. A small but growing chorus of writers and scholars think we’re going too far. By taking human decision-making out of the equation, we’re slowly stripping away deliberation—moments where we reflect on the morality of our actions.

But even stepping away from the morality issues, there are some other undesirable side-effects to algorithmic decision-making:

Or as Evan Selinger, a philosopher at Rochester Institute of Technology, puts it, tools that make hard things easy can make us less likely to tolerate things that are hard. Outsourcing our self-control to “digital willpower” has consequences: Use Siri constantly to get instant information and you can erode your ability to be patient in the face of incomplete answers, a crucial civic virtue.

The argument is that smart technology has the potential to strip us of our grit. And that’s a big problem, particularly if you subscribe to what author Paul Tough calls “the character hypothesis”: the notion that noncognitive skills, like persistence, self-control, curiosity, conscientiousness, grit and self-confidence, are more crucial than sheer brainpower to achieving success.

The hypothesis is that character is created by encountering and overcoming difficult situations. Therefore one of the big dangers of algorithms making our decisions for us is that if it removes challenges from our lives, it reduces our ability to develop grit and build character. It’s like an Axiom for our brains.

Update: I came across a couple more articles about these issues. See More on algorithmic decision-making.

When I wrote The hype, benefits, and dangers of Big Data a few months ago I thought it would be my only post about Big Data, and then I’d move on. But 2013 appears to be the year of Big Data, so you can’t turn a corner on the web without bumping into an article about it. Looking at Google Trends, it’s clear that interest is at an all-time high:

So I wanted to point out just a few more articles that range from calling for a more tempered approach to Big Data to an all-out assault on its value and validity. Let’s start with the juicy one…

In A More Thoughtful but No More Convincing View of Big Data Stephen Few reviews the book Big Data: A Revolution That Will Transform How We Live, Work, and Think and uses it as a way to articulate his distaste with the whole thing:

Data exists in a potentially infinite supply. Given this fact, wouldn’t it be wise to determine with great care what we collect, store, retain, and mine for value? To the extent that more people are turning to data for help these days, learning to depend on evidence rather than intuition alone to inform their decisions, should we accept the Big Data campaign as helpful? We can turn people on to data without claiming that something miraculous has changed in the data landscape over the last few years. […]

As data continues to increase in volume, velocity, and variety as it has since the advent of the computer, its potential for wise use increases as well, but only if we refine our ability to separate the signals from the noise. More does not trump better. Without the right data and skills, more will only bury us.

It’s a long article, but very detailed and highly recommended as a well-reasoned counter-argument to the Big Data movement. Others are a little more pragmatic, suggesting that we improve on the promise of Big Data rather than destroy it. In Coffee & Empathy: Why data without a soul is meaningless1 Om Malik states:

What will it take to build emotive-and-empathic data experiences? Less data science and more data art — which, in other words, means that data wranglers have to develop correlations between data much like the human brain finds context. It is actually not about building the fanciest machine, but instead about the ability to ask the human questions. It is not about just being data informed, but being data aware and data intelligent.

It’s important to take this further and say the soul Om talks about needs to come from qualitative methods like ethnography. That’s why I like Dave McColgin’s point in his article How Will Big Data Change Design Research?:

In our field of designing products and experiences, the ‘why’ stays at the center of our process and creativity. Many designers work mostly on new products and services for which there may not yet be reliable data available. […] While Big Data can inform designers on how to improve once they put something out there, it is design research that provides principled guidance towards good solutions all along the way. Big Data can’t help us do that right now.

Tricia Wang’s Big Data Needs Thick Data is another excellent plea for ethnographers to get involved in the Big Data movement, to produce what she calls “Thick Data”:

Big Data produces so much information that it needs something more to bridge and/or reveal knowledge gaps. That’s why ethnographic work holds such enormous value in the era of Big Data. […]

Big Data reveals insights with a particular range of data points, while Thick Data reveals the social context of and connections between data points. Big Data delivers numbers; thick data delivers stories. Big data relies on machine learning; thick data relies on human learning.

And finally, Martin U. Müller and Marcel Rosenbach look at some of the scarier implications of Big Data in Living by the Numbers: Big Data Knows What Your Future Holds:

Is it truly desirable for cultural assets like TV series or music albums to be tailored to our predicted tastes by means of data-driven analyses? What happens to creativity, intuition and the element of surprise in this totally calculated world?

Internet philosopher Evgeny Morozov warns of an impending “tyranny of algorithms” and is fundamentally critical of the ideology behind many current Big Data applications. Morozov argues that because formulas are increasingly being used in finance and, as in the case of Predictive Policing, in police work, they should be regularly reviewed by independent, qualified auditors — if only to prevent discrimination and abuses of power.

I personally think there is the same value in data that there has always been, and that the Big Data movement isn’t so much about the size of the data sets, but the ability to extract more of that inherent value (signal) from the noise. But an algorithm will only take you so far. As always, knowing what and how much is not very useful without knowing why. And Big Data will never be able to tell us why…

---

1. What’s that? You think I’ll just automatically link to any article with the word “coffee” in the headline? I resent that accusation, sir or madam! ↩

John Pavlus in Your Body Does Not Want to Be an Interface:

The assumption driving these kinds of design speculations is that if you embed the interface — the control surface for a technology — into our own bodily envelope, that interface will “disappear”: the technology will cease to be a separate “thing” and simply become part of that envelope.

The trouble is that unlike technology, your body isn’t something you “interface” with in the first place. You’re not a little homunculus “in” your body, “driving” it around, looking out Terminator-style “through” your eyes. Your body isn’t a tool for delivering your experience: it is your experience. Merging the body with a technological control surface doesn’t magically transform the act of manipulating that surface into bodily experience. I’m not a cyborg (yet) so I can’t be sure, but I suspect the effect is more the opposite: alienating you from the direct bodily experiences you already have by turning them into technological interfaces to be manipulated. 

It’s an excellent essay. I especially like the distinction between Ready-at-hand and Present-at-hand technologies, and how our bodies shouldn’t become marionettes to technology.

In his post Why we fear Facebook and why we shouldn’t Paul Jacobson makes an interesting counterpoint to the common refrain that it’s bad to share our personal data with companies:

Conventional wisdom is that if you are not paying for a product, you are the product. That may be true, as a generalisation. I prefer to think it isn’t so much we who are the products on Facebook but rather our preferences and attention. What does that buy us? For starters, it buys us Facebook, Twitter, Google services and more. It also buys us slightly less annoying ads that can be remarkably relevant. It buys advertisers a better chance that we may want to buy their products and services because those products and services may just be what we are looking for at that point in time.

It’s a good question. Is it really that bad to get highly targeted ads in our news feeds? The more targeted the ads are, the more useful they are to us, right? So why is there such pushback against this trend in companies like Google and Facebook to try to find out everything they can about us?

I think there are three main reasons why we need to be wary of letting ad-driven companies know too much about our preferences, even if they just use it to serve us more targeted information and ads.

1. It makes the web smaller

If we only see stuff we’re already interested in, we run the risk of becoming sucked into the Internet’s “filter bubble”, where it’s much harder to discover new information beyond our current knowledge. Maria Popova puts it like this in Are We Becoming Cyborgs?:

The Web by and large is really well designed to help people find more of what they already know they’re looking for, and really poorly designed to help us discover that which we don’t yet know will interest us and hopefully even change the way we understand the world.

When an algorithmic constraint is placed on the information we see, and that constraint is based solely on our current preferences, we will remain safely locked into the world we know. That means that we become less likely to broaden our horizons with new discoveries.

2. It results in heightened confirmation bias

When we’re steeped in information that confirms our existing beliefs (regardless of whether those beliefs are true or not) we not only seek out more of the same information everywhere we go, but we also become incapable of changing our minds even if we eventually are presented with the truth (the denial of Global Warming is a good example of this…). This is called confirmation bias, and Clay Johnson writes about it in the context of media and the Internet in his book The Information Diet:

It’s too high of a cognitive and ego burden to surround ourselves with people that we disagree with. If you’re a Facebook user, try counting up the number of friends you have who share your political beliefs. Unless you’re working hard to do otherwise, it’s likely that you’ve surrounded yourself with people who skew towards your beliefs. Now look beyond political beliefs—how many of your friends share the same economic class as you? […]

Those algorithms are everywhere: our web searches, our online purchases, our advertisements. This network of predictions is what Pariser calls the Filter Bubble in his book by the same name—the network of personalization technology that figures out what you want and keeps feeding you that at the expense of what you don’t want.

So, for example, through its EdgeRank algorithm Facebook figures out what we like and what we believe in, and then shows us stories and ads that confirm those beliefs. It doesn’t care about truth, it cares about engagement — even if that engagement comes at the expense of what is right.

3. It designs our lives for us

This is true for all advertising, but even more so for hyper-targeted advertising: it tries to sell us stuff we don’t necessarily need. Yes, I know we’re tired of hearing how we should all live with less stuff blah blah blah. That’s not necessarily what I’m saying. What I’m saying is that we need to be careful that we don’t become a society built around the needs of corporations. David Cain talks about this in his chilling essay called Your Lifestyle Has Already Been Designed:

We’ve been led into a culture that has been engineered to leave us tired, hungry for indulgence, willing to pay a lot for convenience and entertainment, and most importantly, vaguely dissatisfied with our lives so that we continue wanting things we don’t have. We buy so much because it always seems like something is still missing. […]

The perfect customer is dissatisfied but hopeful, uninterested in serious personal development, highly habituated to the television, working full-time, earning a fair amount, indulging during their free time, and somehow just getting by.

There’s nothing wrong with stuff, of course. But there is something scarily wrong about the way we let our desires be dictated by advertising — especially targeted advertising by companies that know us so well.

What it means…

I don’t think our biggest fears about the data that companies collect about us should revolve around identity theft or the government coming to get us (although, in some regions, that’s certainly legitimate concerns). Our biggest fear should be what Huxley points to in the future he paints in Brave New World: that we will be ruled by what he calls “man’s almost infinite appetite for distractions”. Huxley believed we should fear companies who aim to control us by inflicting pleasure on us, and I think he might have been on to something.

I know that sounds really alarmist. But still, I can’t look at my Facebook news feed and not think about this possible future. That’s why I think we should hold our personal data and preferences just a little bit closer to our hearts.

In How the internet is making us poor Christopher Mims asks a chilling question about what he calls the “hollowing out of the middle class” — the phenomenon where knowledge workers are being replaced by computers:

Like farming and factory work before it, the labors of the mind are being colonized by devices and systems. In the early 1800′s, nine out of ten Americans worked in agriculture—now it’s around 2%. At its peak, about a third of the US population was employed in manufacturing—now it’s less than 10%. How many decades until the figures are similar for the information-processing tasks that typify rich countries’ post-industrial economies?

The article also quotes this thought-provoking statement from Marc Andreessen:

The spread of computers and the Internet will put jobs in two categories: People who tell computers what to do, and people who are told by computers what to do.

It might seem like the usual doom-and-gloom “technology will kill as all” refrain, but the article reviews some very interesting historical (and current) data, so it’s worth checking out.

Figuring out what to read (and what to believe) about Big Data is becoming a Big Data problem in and of itself1. I wrote The hype, benefits, and dangers of Big Data a while ago to give an overview of what’s out there, but there are two more interesting articles from the last week that I’d like to highlight as well.

First, on the HBR blog Jake Porway talks about Big Data and social entrepreneurship and makes the point that You Can’t Just Hack Your Way to Social Change:

Any data scientist worth their salary will tell you that you should start with a question, NOT the data. Unfortunately, data hackathons often lack clear problem definitions. Most companies think that if you can just get hackers, pizza, and data together in a room, magic will happen. This is the same as if Habitat for Humanity gathered its volunteers around a pile of wood and said, “Have at it!” By the end of the day you’d be left with half of a sunroom with 14 outlets in it.

And on Wired, Does ‘Big Data’ Mean the Demise of the Expert — And Intuition? is a very interesting excerpt from Viktor Mayer-Schönberger and Kenneth Cukier’s new book on the topic:

In the same spirit, the biggest impact of big data will be that data-driven decisions are poised to augment or overrule human judgment.

The subject-area expert, the substantive specialist, will lose some of his or her luster compared with the statistician and data analyst, who are unfettered by the old ways of doing things and let the data speak. This new cadre will rely on correlations without prejudgments and prejudice. To be sure, subject-area experts won’t die out, but their supremacy will ebb. From now on, they must share the podium with the big-data geeks, just as princely causation must share the limelight with humble correlation.

It seems like an obvious conclusion, but everything I’ve read so far about Big Data confirms that if we think cutting the “messiness” of human decision-making out of data analysis will result in better decisions, we’re sorely mistaken.

---

1. Sorry, I didn’t get much sleep last night, so even though I know this isn’t a particularly funny joke, I just can’t help myself. ↩

A Readlist of all the articles referenced in this post is available here. Readlists allow you to send all the articles to your Kindle, read them on your iOS device, or download it as an e-book.

Despite the overly alarmist title, Andrew Leonard’s How Netflix is turning viewers into puppets1 is a fascinating article on how Netflix uses Big Data in their programming decisions:

“House of Cards” is one of the first major test cases of this Big Data-driven creative strategy. For almost a year, Netflix executives have told us that their detailed knowledge of Netflix subscriber viewing preferences clinched their decision to license a remake of the popular and critically well regarded 1990 BBC miniseries. Netflix’s data indicated that the same subscribers who loved the original BBC production also gobbled down movies starring Kevin Spacey or directed by David Fincher. Therefore, concluded Netflix executives, a remake of the BBC drama with Spacey and Fincher attached was a no-brainer, to the point that the company committed $100 million for two 13-episode seasons.

The article also asks what this approach means for the creative process, something I’ve written about before in The unnecessary fear of digital perfection, so I won’t rehash that argument here.

What’s interesting to me about the rise in Big Data approaches to decision-making is the high levels of inaccuracy inherent to the analysis process. Of course, this is something we don’t hear about often, but Nassim N. Taleb recently wrote a great opinion piece about it for Wired called Beware the Big Errors of ‘Big Data’, in which he states:

Big-data researchers have the option to stop doing their research once they have the right result. In options language: The researcher gets the “upside” and truth gets the “downside.” It makes him antifragile, that is, capable of benefiting from complexity and uncertainty — and at the expense of others.

But beyond that, big data means anyone can find fake statistical relationships, since the spurious rises to the surface. This is because in large data sets, large deviations are vastly more attributable to variance (or noise) than to information (or signal). It’s a property of sampling: In real life there is no cherry-picking, but on the researcher’s computer, there is. Large deviations are likely to be bogus.

He gets into more detail on the statistical problems with Big Data in the article, and his book Antifragile looks really interesting too.

Since I haven’t written about Big Data before, I also want to reference a few articles on the topic that I enjoyed. Sean Madden gives some interesting real world examples in How Companies Like Amazon Use Big Data To Make You Love Them2. But over on the skeptical side, Stephen Few argues in Big Data, Big Deal that “interest in big data today is a direct result of vendor marketing; it didn’t emerge naturally from the needs of users.” He also makes the point that data has always been big, and that by focusing on the “bigness” of it, we’re missing the point:

A little more and a little faster have always been on our wish list. While information technology has struggled to catch up, mostly by pumping itself up with steroids, it has lost sight of the objective: to better understand the world—at least one’s little part of it (e.g., one’s business)—so we can make it better. Our current fascination with big data has us looking for better steroids to increase our brawn rather than better skills to develop our brains. In the world of analytics, brawn will only get us so far; it is better thinking that will open the door to greater insight.

Alan Mitchell makes a similar point in Big Data, Big Dead End, a case for what he calls Small Data:

But if we look at the really big value gap faced by society nowadays, it’s not the ability to crunch together vast amounts of data, but quite the opposite. It’s the challenge of information logistics: of how to get exactly the right information to, and from, the right people in the right formats at the right time. This is about Very Small Data: discarding or leaving aside the 99.99% of information I don’t need right now so that I can use the 0.01% of information that I do need as quickly and efficiently as possible.

What I think we should take from all of this is that our ability to collect vast amounts of data comes with enormous predictive and analytical upside. But we’d be foolish to think that it makes decision-making easier. Because Big Data does not take away the biggest challenge of data analysis: figuring how to turn data into information, and information into knowledge. In fact, Big Data makes this harder. To quote Nassim again:

I am not saying here that there is no information in big data. There is plenty of information. The problem — the central issue — is that the needle comes in an increasingly larger haystack.

In other words: proceed with caution.

---

1. Link via @mobivangelist ↩

2. It’s interesting that the phrasing of both this headline and the Netflix one implies that companies are using Big Data to persuade us to do things against our will. But I can’t figure out if that’s a real fear, or just clever linkbait. ↩

In Bridging the gap between humans and computers Heather Kelly takes a look at some recent ethnographic research on our relationship with technology. It’s full of interesting stories like this one:

In one experiment, Ju’s group rigged automatic doors to open in different ways: Some would open slowly, then pause before fully opening; others would immediately jerk all the way open. The people walking by the doors assigned them different levels of intelligence, and thought the doors that opened in two steps just seemed smarter.

It turned out that adding the pause gave illusion of forethought, even though it was just an extra programming step. People thought the door was more intelligent because it appeared to think before carrying out an action.

One of my favorite books on this topic is Sherry Turkle’s Alone Together: Why We Expect More from Technology and Less from Each Other. The first half of the book is all about our relationship with high tech “things” — what we find creepy vs. comforting, how different cultures behave differently, etc. Highly recommended.