Tim Harford has an excellent critique of the statistical issues with the “big data” trend in Big data: are we making a big mistake? First, there’s this:

But the “big data” that interests many companies is what we might call “found data”, the digital exhaust of web searches, credit card payments and mobiles pinging the nearest phone mast.

I still love the term “digital exhaust”. I first saw Frank Chimero use it in the context of social media when he said (in a post that’s now gone from the internet):

The less engaged I become with social media, the more it begins to feel like huffing the exhaust of other people’s digital lives.

But back to big data. The big problem (see what I did there?) is that statistical problems don’t just go away when you have more data. In fact, they get worse. For example:

Because found data sets are so messy, it can be hard to figure out what biases lurk inside them – and because they are so large, some analysts seem to have decided the sampling problem isn’t worth worrying about. It is.

The article goes into the detail on this, and I think it’s important for us to recognize the limitations of big data before jumping on the bandwagon.

In How do Smartphones Affect Human Thought? Jenny Davis addresses the recent research behind the “Smartphones are Making Us Stupid” narrative:

[The research] hypothesis implies (though does not state) a research question: How does smartphone usage affect cognitive processes? This is an important question, but one the research was never prepared to answer thoughtfully. Rather, the authors recast this question as a prediction, embedded in a host of assumptions which privilege unmediated thought.

This approach is inherently flawed. It defines cognitive functioning (incorrectly) as a raw internal process, untouched by technology in its purest state. This approach pits the brain against the device, as though tools are foreign intruders upon the natural body. This is simply not the case. Humans [sic] defining characteristic is our need for tools. Our brains literally developed with and through technology. This continues to be true. Brains are highly plastic, and new technologies change how cognition works. Our thought processes are, and always have been, mediated.

The response echoes many of the points Clive Thompson brings up in Smarter Than You Think (my review) — namely that technology can be great augmentations to human thought. It’s not all bad.

Kashmir Hill asks an interesting question: Who do we blame when a robot threatens to kill people?

Last week, police showed up at the home of Amsterdam Web developer Jeffry van der Goot because a Twitter account under van der Goot’s control had tweeted, according to the Guardian, “I seriously want to kill people.” But the menacing tweet wasn’t written by van der Goot; it was written by a robot.

He goes on:

Bots will be bots. They won’t know if they’re doing something wrong unless we program them to realize it, and it’s impossible to program them to recognize all possible wrong and illegal behavior. So we’ve got challenges ahead. In the short term, [Clément Hertling, a Paris-based university student who wrote the software that powered the bot] suggested Twitter — and any other platforms bots might live on — could solve the offensive speech problem by allowing bots to self-identify in an obvious way as bots. “That would allow people (law enforcement included) to ignore what they say when it becomes problematic.”

This issue only gets scarier as the question expands to wondering what happens when we put self-driving cars in morally ambiguous situations.

In The Cathedral of Computation Ian Bogost makes the argument that algorithms have replaced religion for many people:

Here’s an exercise: The next time you hear someone talking about algorithms, replace the term with “God” and ask yourself if the meaning changes. Our supposedly algorithmic culture is not a material phenomenon so much as a devotional one, a supplication made to the computers people have allowed to replace gods in their minds, even as they simultaneously claim that science has made us impervious to religion.

It’s a long article but very much worth reading, especially for the conclusion:

Algorithms aren’t gods. We need not believe that they rule the world in order to admit that they influence it, sometimes profoundly. Let’s bring algorithms down to earth again. Let’s keep the computer around without fetishizing it, without bowing down to it or shrugging away its inevitable power over us, without melting everything down into it as a new name for fate. I don’t want an algorithmic culture, especially if that phrase just euphemizes a corporate, computational theocracy.

But a culture with computers in it? That might be all right.

Jason Z. Resnikoff’s Seeing the Sixties and Seventies Through 2001 and Alien is a wonderful essay about his father’s experiences as a computer scientist growing up in the era of 2001: A Space Odyssey and Alien. Here’s a taste:

My father was so buried in computers that when he saw 2001 he very much liked HAL, the spaceship Discovery’s villainous central computer. To this day, he enjoys quoting the part of the movie where HAL tries to explain away his own mistake—the supposed fault in the AE35 unit—by saying, “This kind of thing has cropped up before, and it has always been due, to human error,” an excuse that more or less sums up my father’s considerably erudite understanding of computers. According to my father’s interpretation of the film, HAL wanted to become something more than he was. Becoming, always and ever becoming, is in my father’s eyes a worthy, nay, a noble way to go through life, always trying finally to be yourself, that most elusive of goals. The mission to Jupiter was a mission to take the next step in evolution, and HAL wanted to be the one to evolve. My father made this sound like a very reasonable desire, one that makes HAL the hero of the movie.

It’s a story about two iconic movies, but also about how we used to see the future. Turns out we won’t be space babies after all.

A couple of articles about work and technology caught my eye this week. First, Claire Cain Miller describes how Technology, Aided by Recession, Is Polarizing the Work World:

[A new working paper from the National Bureau of Economic Research], which analyzed data from the Current Population Survey from 1976 to 2012, illustrates that the recession had a disproportionately large effect on routine jobs, and greatly sped up their loss. That is probably because even if a new technology is cheaper and more efficient than a human laborer, bosses are unlikely to fire employees and replace them with computers when times are good. The recession, however, gave them a motive. And the people who lost those jobs are generally unable to find new ones, said Henry E. Siu, an associate professor at the University of British Columbia and an author of the study.

Now, combine that problem in the mid-paying job market with an issue Thomas B. Edsall pointed out a few weeks ago in The Downward Ramp:

Just one example: the drying up of cognitively demanding jobs is having a cascade effect. College graduates are forced to take jobs beneath their level of educational training, moving into clerical and service positions instead of into finance and high tech.

This cascade eliminates opportunities for those without college degrees who would otherwise fill those service and clerical jobs. These displaced workers are then forced to take even less demanding, less well-paying jobs, in a process that pushes everyone down. At the bottom, the unskilled are pushed out of the job market altogether.

So, college graduates are pushed into mid-paying jobs, and those jobs are being replaced by technology. Not good.

Meanwhile, in opposite world, Louise Aronson writes about The Future of Robot Caregivers (if you’re counting, that’s three for three on the New York Times):

We do not have anywhere near enough human caregivers for the growing number of older Americans.

Zeynep Tufekci’s excessively titled Failing the Third Machine Age: When Robots Come for Grandma is a good critique of that piece:

Let me explain. When people confidently announce that once robots come for our jobs, we’ll find something else to do like we always did, they are drawing from a very short history. The truth is, there’s only been one-and-a-three-quarters of a machine age—we are close to concluding the second one—we are moving into the third one.

And there is probably no fourth one.

Humans have only so many “irreplaceable” skills, and the idea that we’ll just keep outrunning the machines, skill-wise, is a folly.

Put all these pieces together and you get a very scary vision of the future of jobs. The good news — I think — is that job != work.

The future of jobs might be bleak, but the future of work certainly isn’t. Technology might be taking our jobs, but it’s also giving us new ways to be creative. To be entrepreneurs. To work. As programs like Girls Who Code continue to grow, I’m increasingly optimistic about my daughters’ futures. They might not get a “regular” job one day. But my role as a parent is not to prepare them for a job anyway. It’s to foster in them the tenacity and grit to learn how to think big and make things. I’m excited about that.

Gideon Lewis-Kraus discusses The Fasinatng … Frustrating … Fascinating History of Autocorrect. Turns out there’s more to it than meets the eye:

A handful of factors are taken into account to weight the variables: keyboard proximity, phonetic similarity, linguistic context. But it’s essentially a big popularity contest. A Microsoft engineer showed me a slide where somebody was trying to search for the long-named Austrian action star who became governor of California. Schwarzenegger, he explained, “is about 10,000 times more popular in the world than its variants”—Shwaranegar or Scuzzynectar or what have you. Autocorrect has become an index of the most popular way to spell and order certain words.

This article also taught me that swear words are complicated. And I really like the cartoons of various autocorrect errors, especially this one:

The AP is increasingly starting to use software with no human intervention to write basic news stories, but Kevin Roose says that we shouldn’t be alarmed about it. From his article Why Robot Journalism Is Great for Journalists:

Robot assistance may even spur human reporters to do our jobs better. With software producing the equivalent of old-school “clip files” for us, we’ll essentially have full-time research assistants. The information in our stories will be more accurate, since it will come directly from data feeds and not from human copying and pasting, and we’ll have to issue fewer corrections for messing things up. Plus, with our nuts-and-bolts reporting out of the way, we’ll be able to focus on the kinds of stories that educate and entertain readers in a deep way, rather than just dragging simple information from Point A to Point B.

Conor Friedersdorf talks about the differences between recommendations provided by people and algorithms in Would You Rather Get Tips from an Expert or an Algorithm?

The Amazon.com algorithm is very good at using what you’ve just bought to recommend things that you’ll want to buy, [David Weinberger, a senior researcher at the Berkman Center for Internet and Society] observed, but it can be hard to tell why. Perhaps you’ll be attracted to the content of the recommendation — or perhaps it’s the fact that the cover is also green, or that the print is in Helvetica font. 

In contrast, a skilled librarian is usually going to recommend a book solely because of its intellectual value, without any lurking, contentless variables. The librarian is therefore likelier to send a person in a direction they wouldn’t otherwise have gone in a way that will advance their thinking, education, or aesthetic taste, because they’re not just meeting needs that have already been expressed.

We’re seeing this divide come out in products as well, and some are starting to use their “humanness” as a differentiator. Whereas most music recommendation systems like Pandora, Spotify, and Rdio use algorithmic approaches, Beats touts the power of human curation on their product.

Go Book Yourself is a Tumblr site that publishes curated recommendations for books you might like based on other books you read and liked. Their tag line is Book recommendations by humans, because algorithms are so 1984.

The humans are coming.

In What If Doctors Could Finally Prescribe Behavior Change? Sean Duffy explains why behavior change is so difficult, particularly in healthcare:

Whether it’s for weight loss, smoking cessation, diabetes, or otherwise, the best research shows that meaningful behavior change outcomes require not just guidance from a trusted health professional, but also positive social support, easy-to-digest insights about their condition, a carefully orchestrated timeline, and a process that follows validated behavioral science protocols. That’s hard to squeeze into a phone call. Or a doctor’s visit, for that matter.

The good news is that this research is resulting in a new field called Digital Therapeutics, and despite quite a bit of snake oil out there, some apps are having success:

Another example is Jenna Tregarthen, a PhD candidate in clinical psychology and eating disorder specialist. She rallied a team of engineers, entrepreneurs, and fellow psychologists to develop Recovery Record, a digital therapy that helps patients gain control over their eating disorder by enabling them to self-monitor for destructive thoughts or actions, follow meal plans, achieve behavior goals, and message a therapist instantly when they need support.