Conor Friedersdorf talks about the differences between recommendations provided by people and algorithms in Would You Rather Get Tips from an Expert or an Algorithm?

The Amazon.com algorithm is very good at using what you’ve just bought to recommend things that you’ll want to buy, [David Weinberger, a senior researcher at the Berkman Center for Internet and Society] observed, but it can be hard to tell why. Perhaps you’ll be attracted to the content of the recommendation — or perhaps it’s the fact that the cover is also green, or that the print is in Helvetica font. 

In contrast, a skilled librarian is usually going to recommend a book solely because of its intellectual value, without any lurking, contentless variables. The librarian is therefore likelier to send a person in a direction they wouldn’t otherwise have gone in a way that will advance their thinking, education, or aesthetic taste, because they’re not just meeting needs that have already been expressed.

We’re seeing this divide come out in products as well, and some are starting to use their “humanness” as a differentiator. Whereas most music recommendation systems like Pandora, Spotify, and Rdio use algorithmic approaches, Beats touts the power of human curation on their product.

Go Book Yourself is a Tumblr site that publishes curated recommendations for books you might like based on other books you read and liked. Their tag line is Book recommendations by humans, because algorithms are so 1984.

The humans are coming.

In What If Doctors Could Finally Prescribe Behavior Change? Sean Duffy explains why behavior change is so difficult, particularly in healthcare:

Whether it’s for weight loss, smoking cessation, diabetes, or otherwise, the best research shows that meaningful behavior change outcomes require not just guidance from a trusted health professional, but also positive social support, easy-to-digest insights about their condition, a carefully orchestrated timeline, and a process that follows validated behavioral science protocols. That’s hard to squeeze into a phone call. Or a doctor’s visit, for that matter.

The good news is that this research is resulting in a new field called Digital Therapeutics, and despite quite a bit of snake oil out there, some apps are having success:

Another example is Jenna Tregarthen, a PhD candidate in clinical psychology and eating disorder specialist. She rallied a team of engineers, entrepreneurs, and fellow psychologists to develop Recovery Record, a digital therapy that helps patients gain control over their eating disorder by enabling them to self-monitor for destructive thoughts or actions, follow meal plans, achieve behavior goals, and message a therapist instantly when they need support.

In The Dawn of the Age of Artificial Intelligence Erik Brynjolfsson and Andrew McAfee talk about some of the good things that are coming out of the Artificial Intelligence community:

A user of the OrCam system, which was introduced in 2013, clips onto her glasses a combination of a tiny digital camera and speaker that works by conducting sound waves through the bones of the head. If she points her finger at a source of text such as a billboard, package of food, or newspaper article, the computer immediately analyzes the images the camera sends to it, then reads the text to her via the speaker.

There are a few more interesting, feel-good examples in the article.

John Foreman digs into Disney’s MagicBands in his article You don’t want your privacy — Disney and the meat space data race:

Disney World is like a petri dish for advanced analytic techniques because the hotels and parks are all tied together in one large, heavily controlled environment. If you ever wanted to star in The Truman Show, a trip to Disney is the next best thing — it feels like a centrally planned North Korea only with more fun, less torture and the same amount of artifice.

From the mundane to the magical, the fact is there’s probably an engineer behind the scenes at Disney who has thought through it. Disney has industrial engineers that work on everything from optimal food-and-beverage pricing and laundry facility optimization, to attraction performance and wait-time minimization (the vaunted FASTPASS system).

The article is largely a negative look at (legitimate) privacy issues with programs like these, but in Disney’s case, I just think it sounds awesome.

It took me a while to get through Michael Pepi’s The Postmodernity of Big Data. It’s dense, and the premise seemed so far-fetched that I wasn’t sure it would be worth the time investment:

But beyond economic motivations for Big Data’s rise, are there also epistemological ones? Has Big Data come to try to fill the vacuum of certainty left by postmodernism? Does data science address the insecurities of the postmodern thought?

Yes, I know, that sounds like a bit of a stretch. But I’m glad I stuck with it. The essay brings up some really interesting thoughts around the certainty promised by Big Data (even though some view it as nothing more than a clever marketing campaign for something that has been around a long time), and how that might be a response to the relativism of postmodernism:

Though both are projects that address positions about empiricism and meaning making, postmodernism and Big Data are in some senses opposites: Big Data is an empirically grounded quest for truth writ large, accelerated by exponentially expanding computing power. Postmodernism casts doubt on the very idea that reason can unearth an inalienable truth. Whereas Big Data sees a plurality of data points contributing to a singular definition of the individual, postmodernism negates the idea that a single definition of any entity could outweigh its contingent relations. Big Data aims for certainties — sometimes called “analytic insights” — that fly in the face of postmodernist doubt about knowledge. Postmodernism was confined to the faculty lounge and the academic conference, but Big Data has the ability to dictate new rules of behavior and commerce. An e-commerce outfit is almost foolish not to analyze browsing data and algorithmically determine likely future purchases, or as Jaron Lanier put it in Who Owns the Future, “your lack of privacy is someone else’s wealth.”

Consider this your difficult but satisfying weekend reading project.

Alexis Madrigal — who is turning into one of the most interesting journalists of our time — goes deep on Netflix’s 76,897 (often bizarre) micro-genres in How Netflix Reverse Engineered Hollywood:

Netflix has meticulously analyzed and tagged every movie and TV show imaginable. They possess a stockpile of data about Hollywood entertainment that is absolutely unprecedented.

Netflix is putting in a staggering amount of effort on the structured data of their TV shows and movies. And of course, it’s all for one reason — to get to know you better:

They capture dozens of different movie attributes. They even rate the moral status of characters. When these tags are combined with millions of users’ viewing habits, they become Netflix’s competitive advantage. The company’s main goal as a business is to gain and retain subscribers. And the genres that it displays to people are a key part of that strategy. “Members connect with these [genre] rows so well that we measure an increase in member retention by placing the most tailored rows higher on the page instead of lower,” the company revealed in a 2012 blog post. The better Netflix shows that it knows you, the likelier you are to stick around.

And now, they have a terrific advantage in their efforts to produce their own content: Netflix has created a database of American cinematic predilections. The data can’t tell them how to make a TV show, but it can tell them what they should be making. When they create a show like House of Cards, they aren’t guessing at what people want.

What’s interesting is that similar things are happening in other forms of media as well. Spotify and Rdio’s knowledge of our listening data can be used to inform record labels what type of albums they should invest in. And as David Streitfeld reports in As New Services Track Habits, the E-Books Are Reading You, a new crop of companies are helping authors figure out what type of books they should write:

The move to exploit reading data is one aspect of how consumer analytics is making its way into every corner of the culture. Amazon and Barnes & Noble already collect vast amounts of information from their e-readers but keep it proprietary. Now the start-ups — which also include Entitle, a North Carolina-based company — are hoping to profit by telling all.

“We’re going to be pretty open about sharing this data so people can use it to publish better books,” said Trip Adler, Scribd’s chief executive. […]

Scribd is just beginning to analyze the data from its subscribers. Some general insights: The longer a mystery novel is, the more likely readers are to jump to the end to see who done it. People are more likely to finish biographies than business titles, but a chapter of a yoga book is all they need. They speed through romances faster than religious titles, and erotica fastest of all.

All of this raises familiar questions about the loss of serendipity — finding interesting things we’re not looking for. But I still think this is an unnecessary fear.

Nicholas Carr wrote a really interesting article on the dangers of computer automation. In All Can Be Lost: The Risk of Putting Our Knowledge in the Hands of Machines he weaves together stories about airline crashes and Inuit hunters to make salient points like this:

Psychologists have found that when we work with computers, we often fall victim to two cognitive ailments — complacency and bias — that can undercut our performance and lead to mistakes. Automation complacency occurs when a computer lulls us into a false sense of security. Confident that the machine will work flawlessly and handle any problem that crops up, we allow our attention to drift. We become disengaged from our work, and our awareness of what’s going on around us fades. Automation bias occurs when we place too much faith in the accuracy of the information coming through our monitors. Our trust in the software becomes so strong that we ignore or discount other information sources, including our own eyes and ears. When a computer provides incorrect or insufficient data, we remain oblivious to the error.

Carr goes to great lengths to make the argument that the automation of tasks is slowly robbing us of the ability to learn new skills. It is, in some ways, a more nuanced argument than his famous 2008 article Is Google Making Us Stupid? It’s well worth reading — even if you’re skeptical of this argument, it will definitely make you think.

Yesterday I posted The problem with letting algorithms make most of our decisions, discussing how removing all knowledge obstacles can make us less adept at dealing with challenges. As is often the case, within a few hours of posting that I came across two more articles that addresses the same issues. First, from Kyle Baxter’s very interesting essay On the Philosophy of Google Glass:

Page’s idea — that we would be fundamentally better off if we had immediate access to all of humanity’s information — ignores [how we develop knowledge]. It provides facts, but elides conclusions and connections. What’s worse, it starves us of opportunities to use our skill for critical thinking, and since it is a skill and is therefore something that must be developed and practiced, it starves us of the chance to develop it.

I find that troubling. Glass is not a technology that is designed to amplify our own innate abilities as humans or to make us better as humans, but rather one that acts as a crutch to lean on in place of exercising the very thing that makes us human. I don’t find that exciting. I find that disturbing.

And then, from Smart cities and smart citizens, an editorial in Sustain Magazine (which I’ll reference more over the coming days):

Furthermore, [Dan Hill, CEO of Fabrica] argues that current smart-systems thinking could lead us down a dangerous path towards passive citizens. As citizens — and city leaders — devolve their decision-making and responsibility to technology, their awareness of their environment diminishes in line with their ability to do something about it.

“If you automate too much stuff, people stop thinking about the issues. Yes, it might be more efficient to make the lights go off automatically, but it stops us thinking about it, we’re not engaged — and when we’re disengaged that’s not a good idea. We want people to think about something like carbon. Besides, we can turn the lights off on the way out — it’s entirely possible, we’re quite a smart species potentially!”

I find it fascinating how the Internet sometimes feel like one organism, always thinking and debating the same issues from many different angles. From Google Glass to Architecture to self-driving cars, it seems that currently we’re collectively worried about the impact of smart technologies on our lives.

Image source: Knight Rider’s KITT - My finished replica!

Nicholas Carr asks some serious questions about things like self-driving cars and our increased reliance on algorithms for decision-making in Moral code:

As we begin to have computer-controlled cars, robots, and other machines operating autonomously out in the chaotic human world, situations will inevitably arise in which the software has to choose between a set of bad, even horrible, alternatives. How do you program a computer to choose the lesser of two evils? What are the criteria, and how do you weigh them?

Clive Thompson picks up the thread in a very interesting Wired article called Relying on Algorithms and Bots Can Be Really, Really Dangerous:

The truth is, our tools increasingly guide and shape our behavior or even make decisions on our behalf. A small but growing chorus of writers and scholars think we’re going too far. By taking human decision-making out of the equation, we’re slowly stripping away deliberation—moments where we reflect on the morality of our actions.

But even stepping away from the morality issues, there are some other undesirable side-effects to algorithmic decision-making:

Or as Evan Selinger, a philosopher at Rochester Institute of Technology, puts it, tools that make hard things easy can make us less likely to tolerate things that are hard. Outsourcing our self-control to “digital willpower” has consequences: Use Siri constantly to get instant information and you can erode your ability to be patient in the face of incomplete answers, a crucial civic virtue.

The argument is that smart technology has the potential to strip us of our grit. And that’s a big problem, particularly if you subscribe to what author Paul Tough calls “the character hypothesis”: the notion that noncognitive skills, like persistence, self-control, curiosity, conscientiousness, grit and self-confidence, are more crucial than sheer brainpower to achieving success.

The hypothesis is that character is created by encountering and overcoming difficult situations. Therefore one of the big dangers of algorithms making our decisions for us is that if it removes challenges from our lives, it reduces our ability to develop grit and build character. It’s like an Axiom for our brains.

Update: I came across a couple more articles about these issues. See More on algorithmic decision-making.

When I wrote The hype, benefits, and dangers of Big Data a few months ago I thought it would be my only post about Big Data, and then I’d move on. But 2013 appears to be the year of Big Data, so you can’t turn a corner on the web without bumping into an article about it. Looking at Google Trends, it’s clear that interest is at an all-time high:

So I wanted to point out just a few more articles that range from calling for a more tempered approach to Big Data to an all-out assault on its value and validity. Let’s start with the juicy one…

In A More Thoughtful but No More Convincing View of Big Data Stephen Few reviews the book Big Data: A Revolution That Will Transform How We Live, Work, and Think and uses it as a way to articulate his distaste with the whole thing:

Data exists in a potentially infinite supply. Given this fact, wouldn’t it be wise to determine with great care what we collect, store, retain, and mine for value? To the extent that more people are turning to data for help these days, learning to depend on evidence rather than intuition alone to inform their decisions, should we accept the Big Data campaign as helpful? We can turn people on to data without claiming that something miraculous has changed in the data landscape over the last few years. […]

As data continues to increase in volume, velocity, and variety as it has since the advent of the computer, its potential for wise use increases as well, but only if we refine our ability to separate the signals from the noise. More does not trump better. Without the right data and skills, more will only bury us.

It’s a long article, but very detailed and highly recommended as a well-reasoned counter-argument to the Big Data movement. Others are a little more pragmatic, suggesting that we improve on the promise of Big Data rather than destroy it. In Coffee & Empathy: Why data without a soul is meaningless1 Om Malik states:

What will it take to build emotive-and-empathic data experiences? Less data science and more data art — which, in other words, means that data wranglers have to develop correlations between data much like the human brain finds context. It is actually not about building the fanciest machine, but instead about the ability to ask the human questions. It is not about just being data informed, but being data aware and data intelligent.

It’s important to take this further and say the soul Om talks about needs to come from qualitative methods like ethnography. That’s why I like Dave McColgin’s point in his article How Will Big Data Change Design Research?:

In our field of designing products and experiences, the ‘why’ stays at the center of our process and creativity. Many designers work mostly on new products and services for which there may not yet be reliable data available. […] While Big Data can inform designers on how to improve once they put something out there, it is design research that provides principled guidance towards good solutions all along the way. Big Data can’t help us do that right now.

Tricia Wang’s Big Data Needs Thick Data is another excellent plea for ethnographers to get involved in the Big Data movement, to produce what she calls “Thick Data”:

Big Data produces so much information that it needs something more to bridge and/or reveal knowledge gaps. That’s why ethnographic work holds such enormous value in the era of Big Data. […]

Big Data reveals insights with a particular range of data points, while Thick Data reveals the social context of and connections between data points. Big Data delivers numbers; thick data delivers stories. Big data relies on machine learning; thick data relies on human learning.

And finally, Martin U. Müller and Marcel Rosenbach look at some of the scarier implications of Big Data in Living by the Numbers: Big Data Knows What Your Future Holds:

Is it truly desirable for cultural assets like TV series or music albums to be tailored to our predicted tastes by means of data-driven analyses? What happens to creativity, intuition and the element of surprise in this totally calculated world?

Internet philosopher Evgeny Morozov warns of an impending “tyranny of algorithms” and is fundamentally critical of the ideology behind many current Big Data applications. Morozov argues that because formulas are increasingly being used in finance and, as in the case of Predictive Policing, in police work, they should be regularly reviewed by independent, qualified auditors — if only to prevent discrimination and abuses of power.

I personally think there is the same value in data that there has always been, and that the Big Data movement isn’t so much about the size of the data sets, but the ability to extract more of that inherent value (signal) from the noise. But an algorithm will only take you so far. As always, knowing what and how much is not very useful without knowing why. And Big Data will never be able to tell us why…

---

1. What’s that? You think I’ll just automatically link to any article with the word “coffee” in the headline? I resent that accusation, sir or madam! ↩