Menu

Posts tagged “culture”

Don’t blame outages on human error and technical debt; improve the system instead

The FAA outage in January that caused the first nationwide ground stop of all flights in the US since 9/11 is kind of old news now, but there’s one detail that I can’t stop thinking about. In the aftermath of the incident the cause was determined to be a database sync issue:

The F.A.A. said in a statement that the workers had been trying to “correct synchronization” between the main database for the Notice to Air Missions alerts and a backup database when the files were mistakenly deleted, causing the outage that snarled air traffic throughout the day on Jan. 11.

CNN added a little more detail:

A contractor working for the Federal Aviation Administration unintentionally deleted files related to a key pilot safety system, leading to a nationwide ground stop and thousands of delayed and canceled flights last week, the FAA said Thursday.

The FAA determined the issue with the Notice to Air Missions (NOTAM) system occurred when the contractor was “working to correct synchronization between the live primary database and a backup database.”

The unsurprising narrative that came out of the tech world following the incident can basically be summarized as “ha ha, silly contractors!” But that feels like a lazy response to me. I didn’t see anyone ask what I believe is the more important question: How do we improve the system (people, processes, technology) that enables one person to inadvertently take down all air traffic in the US?

Let’s remember that this kind of thing can happen to absolutely anyone. Etsy even hands out a “three-armed sweater” award to the engineer who had the most spectacular mishap in any given year:

Kate’s story is a nail-biter, involving a tiny code change that unexpectedly brought down Etsy.com. All of her coworkers rallied around her to help get the site back online, while offering words of encouragement and reassurance.

So it might be really convenient to blame the FAA outage on “contractor error” and then just keep going. But that’s not going to prevent the next incident from happening.

It is further also tempting to blame the entire issue on “tech debt” and call it a day. And, fair enough, there’s certainly plenty of that going around in FAA systems. Ars Technica has a good overview of some of the major issues and how the FAA wants to fix them. But like all giant “replatforming” projects (this one is called NextGen, because of course it is) things are… not going great:

FAA tech problems were previously described in a March 2021 report by the US Department of Transportation Office of Inspector General. The report discusses the FAA’s Next Generation Air Transportation System (NextGen), “a multibillion dollar infrastructure project aimed at modernizing our Nation’s aging air traffic system to provide safer and more efficient air traffic management.”

“NextGen’s actual and projected benefits have not kept pace with initial projections due to implementation challenges, optimistic assumptions, and other factors,”1 the report said.

But blaming tech debt—and especially blaming individuals—is not going to get us very far. Tech debt will always be there (although I have some thoughts on how to prioritize it), and individual mistakes are not going to go away. What we can do is examine the system that enables, in this case, a database sync to corrupt the primary live db, and figure out how to prevent that from happening in the first place.

Almost 30 years ago Jakob Nielsen published his 10 Usability Heuristics for User Interface Design, and “error prevention” is still as true today as it was then:

Good error messages are important, but the best designs carefully prevent problems from occurring in the first place. Either eliminate error-prone conditions, or check for them and present users with a confirmation option before they commit to the action.

The example I always think of here is how you often seen battery packs shaped in a certain way so that it’s impossible to insert them incorrectly (contrast that with the terrors of trying to insert a USB cable the correct way the first time!).

In a situation like the one the FAA experienced, yes it’s important to acknowledge human error, and talk about the underlying tech issues, but that’s not enough. We have to figure out how to add preventative measures to our systems and pipelines2. To put it another way, they might not be able to replace their battery packs with NextGen solar yet, but they can certainly change the shape of the battery to prevent contractors from blowing up the camera.


  1. My emphasis added because who among us have not heard those words before… ↩

  2. For further reading on what to do after a major incident, check out Will Larson’s Move past incident response to reliability. ↩

Link roundup for March 3, 2023

The African Bricks 3. Mosaic artworks inspired by the culture and beauty of Africa, by Charis Tsevis.

The Cello in Soho Square. I like this description by Michael Lopp of the difference between “dabblers” and “S-tier” people (who are the absolute best at something): “There is an infinite list of exciting things to learn, but the Dabbler knows they have finite time, so they dabble. They get 80% of the juice, and they move on. Respect. S-Tier knows the last 10% of the challenge is the hardest, but it also teaches you the most.”

Physicists Say Aliens May Be Using Black Holes as Quantum Computers. This is fine. “In a recent study, a German-Georgian team of researchers proposed that advanced extraterrestrial civilizations (ETCs) could use black holes as quantum computers. This makes sense from a computing standpoint and offers an explanation for the apparent lack of activity we see when we look at the cosmos.”

Honestly, it’s probably the phones. Don’t dismiss this argument just from the headline, like I almost did. There’s some solid evidence presented here. “If we’re looking for one big ‘silver bullet’ or ‘grand unified theory’ of modern teenage unhappiness, phones are probably the place to start looking.”

Papercraft Models by Rocky Bergen. “Construct the computer from your childhood or build an entire computer museum at home with these paper models, free to download and share. Print, Cut, Score, Fold and Glue.”

In an Uncertain Job Market, How Can Companies Retain Workers? The conventional wisdom that people tend to hunker down when there are layoffs around them might not be accurate: “Layoffs ‘create an environment where people worry it might happen to them next,’ said Laszlo Bock, who was Google’s SVP for people operations. Poorly handled reductions may ‘degrade trust in management as people start hearing rumors of further cuts, and that in turn raises anxiety, which causes more people to quit.’” (NYT gift article)

How the Phonograph Created the 3-Minute Pop Song. I can’t resist a good “technologies people thought would ruin everything” article, and this is another fascinating one: “Plenty of folks worried that records would destroy musical culture. John Philip figured it would demotivate anyone from learning to play an instrument themselves. Why bother, when you could just put on music by a true virtuoso? ‘When music can be heard in the homes without the labor of study,’ he fretted in a 1906 article, ‘it will be simply a question of time when the amateur disappears entirely.’”

The Case for Hanging Out. I love this essay. “Pushed further into isolation by the pandemic, we’re all losing the ability to engage in what I view as the pinnacle of human interaction: sitting around with friends and talking shit.”

Explore. I think it’s probably too late for a viable LinkedIn alternative, but this site would be a great contendor.

The 90s, having time, and always rushing to the next thing

I’m sure every generation writes lots of articles like Freddie deBoer’s It’s So Sad When Old People Romanticize Their Heydays, Also the 90s Were Objectively the Best Time to Be Alive. But hear me out. This is the impassioned, forceful, yet balanced Gen X take I wish I had the skill and wherewithal to write. It is a balm to the nostalgic soul in a way that somehow doesn’t feel like cringey old-person fanfic.

Here he is on the experience of visiting a record store:

When you were there you were Doing Music. Now we’re never doing anything—we’re always getting through something to get to something else to get through, using various time-saving techniques that maximize the amount of time we have to get through things while keeping our attention divided into a thousand things we then get through. When you went to a record store you were intent on music, and sometimes, you’d care enough about a particular artist that you paid for their album, real money, so that the artist got a cut that was more than the .002 cents they get per stream now.

This reminds me of the question Alan Jacobs asks: What exactly are we’re rushing towards with all our 2x listening and cliff notes skim-reading?

My question about all this is: And then? You rush through the writing, the researching, the watching, the listening, you’re done with it, you get it behind you—and what is in front of you? Well, death, for one thing. For the main thing. 

But in the more immediate future: you’re zipping through all these experiences in order to do what, exactly? Listen to another song at double-speed? Produce a bullet-point outline of another post that AI can finish for you?

Maybe the 90s have a thing or two to teach us yet.

You can't stand under my umbrella

In You can’t stand under my umbrella the Raw Signal team makes the case for when it’s not appropriate for managers to be “sh*t umbrellas” for their teams:

When things are steady, and people know the right things to work on, teams are constrained by velocity. We know the course we’re racing, the question is just how fast we can go. In that context, it makes sense for a manager to clear every obstacle out of our way. But during times of significant change, teams are constrained by agility. It’s not that velocity doesn’t matter, it still does. But when everything has changed about the race, we need the ability to steer. A manager who tries to preserve velocity at all costs risks running us into a wall.

They go on to talk about how to Accept, Adapt, and Act in such moments of significant change.

Link roundup for February 19, 2023

Underwater Photographer of the Year—2023 Winners. These are so great.

Impostered. Great post from Mandy Brown about the need to reframe how we think about imposter syndrome. “I’ve started to think less about imposter syndrome (a description of a person’s experience with it) and more about being impostered (a framing that draws attention to the systems and structures that lead people to believe they are imposters). While the former framing remains useful in many contexts, the latter creates space to consider not only the symptoms but the root cause of the phenomena.” [everythingchanges.us]

What’s So Funny? Very good essay about the current state of stand-up comedy, and what makes something funny. “The audience whooping and applauding Roseanne’s ‘anti-woke’ comedy is not reacting with laughter at a previously un-acknowledged truth, but instead expressing approval for the point of view that they already knew they agreed with. This is not the same thing as laughter in response to a joke.” [biblioracle.substack.com]

Why Are You Seeing So Many Bad Digital Ads Now? “Social media advertising, once a niche art practiced by specialist agencies, is now easily available to anyone. Many of them are eschewing targeted ads — placements intended to reach specific audiences, usually at a higher cost — in favor of a cheaper spray-and-pray approach online, hoping to catch the attention of gullible or bored shoppers.” [NYT gift link]

Traffic Lights Need a Fourth Color, Study Says: Here’s Why. Yeah what could possibly go wrong. “For the dawning age of the self-driving car, transportation engineers from North Carolina State University are proposing the addition of a fourth ‘white light’ whose function would be to alert humans to simply ‘follow the car in front of them.’” [popularmechanics.com]

The people who live inside airplanes. Ok I kind of like this idea. “By the end, she had a fully functional home, with over 1,500 square feet of living space, three bedrooms, two bathrooms and even a hot tub — where the cockpit used to be. All for less than $30,000, or about $60,000 in today’s money.” [cnn.com]

Move past incident response to reliability

Here’s an interesting article by Will Larson with advice on how to move past incident response to reliability in our products. Among other things it reminded me to watch out for “incident legalism”:

Incident legalism is when an incident response and analysis program—trying to better drive reliability improvements—becomes focused on compliance and loses empathy for the engineers and teams operating within the program’s processes.

He goes on to propose a more holistic, expanded model for reliability to help teams diagnose their systemic problems—and how to solve them:

Finally, you study the mitigated incidents, determining how to prevent them from recurring, and they become remediated incidents.

How to build human connections in an async workplace

This is a great post by Chase Warrington for the Twist Async newsletter on How to build human connections in an async workplace. They make this really important point about what human connection is actually about on a remote team:

I’ve come to realize that team culture and human connection is primarily built by how you work together—not how you socialize together. […]

The work we do is what actually brings us together. That’s ok (and frankly healthy) to admit. One of the biggest benefits of remote work is that it provides you the opportunity to spend more on the people and things you care about outside of work. Let’s not sabotage that with a bunch of forced and awkward social events for teammates to attend on top of their work duties.

I think we forget this too often. Doing a fun online social activity together doesn’t improve team culture if we haven’t also made sure that actually working together is safe, healthy, and enjoyable.

Link roundup for February 15, 2023

AMEN by Jessica Hilltout. “The aim of AMEN was to shine the light on all those in the shadow of the World Cup, far from the big stadiums and the corporate carnival-nature of the event. To embrace Africa and everything that makes it unique. To speak of the authenticity and sheer ingenuousness of a continent that manages to do so much with so little. To capture people with simple needs and huge hearts. To express football in its purest form.” [jessicahilltout.com]

God Did the World a Favor by Destroying Twitter. I love Paul Ford. “Our smarter, richer betters often preach the idea of a town square, a marketplace of ideas, a centralized hub of discourse and entertainment—and we listen. But when I go back and read Genesis, I hear God saying: ‘My children, I designed your brains to scale to 150 stable relationships. Anything beyond that is overclocking. You should all try Mastodon.’” [wired.com]

SolidGoldMagikarp (plus, prompt generation). This is all super weird and shows that we really have no idea what’s going on with these LLMs. Are we really ready for this stuff to become the backbone of internet search? “We’ll demonstrate a previously undocumented failure mode for GPT-2 and GPT-3 language models, which results in bizarre completions (in some cases explicitly contrary to the purpose of the model), and present the results of our investigation into this phenomenon.” [lesswrong.com]

Hey, Ease Up; A Load-Bearing If-Statement. What happens if, for health reasons, you need to use a fitness tracker to move less? “But if you’re trying to conserve energy, you don’t want to reach that goal. You want to stay under it. Sure, you want to maybe get up and about, I guess? Take a very slow short walk outside? But you are supposed to be resting.” [newsletter.danhon.com]

Scientists Develop Compound That Kills So Efficiently They Named It After Keanu Reeves. “The molecules ‘kill so efficiently that we named them after Keanu Reeves,’ German researcher Sebastian Götze said in a press release, ‘because he, too, is extremely deadly in his roles.’” [futurism.com]

Buy Nothing groups and the culture of free stuff. This deep-dive into the things people post in Buy Nothinggroups (and that actually get picked up!) is quite something. “There is something about free stuff that makes us abandon all rational thought.” [washingtonpost.com, soft paywall]

The mystery of the disappearing vacation day. Why have we stopped taking regular vacations? “Many were on a paid-time-off (PTO) plan that lumps sick days, personal days and vacation days in a single bucket. While workers often appreciate the flexibility of PTO and employers find it easier to administer, such plans can deter taking long vacations by making us feel as if we’re cutting into the PTO we might need in case of sudden illness or tragedy.” [washingtonpost.com, gift article link]

Nope, coffee won’t give you extra energy. It’ll just borrow a bit that you’ll pay for later. Ok listen, don’t come at me with your “facts”, please. “While it feels energizing, this little caffeine intervention is more a loan of the awake feeling, rather than a creation of any new energy.” [theconversation.com]

Mono no aware

What would happen if we look at time through the lens of attachment theory? That’s the question my friend Simon asks in Attachment Styles to Time. I definitely have an “anxious attachment style” with time:

An anxiously attached person to time will try to arrest it: to find comfort again in a space where time felt distant. A coping strategy is to try and keep things the way they were. To hold onto people and places even if you aren’t present anymore.

The framing also reminds me of the Japanese phrase Mono no aware:

Mono no aware (物の哀れ), lit. ‘the pathos of things’, and also translated as ‘an empathy toward things’, or ‘a sensitivity to ephemera’, is a Japanese idiom for the awareness of impermanence (無常, mujō), or transience of things, and both a transient gentle sadness (or wistfulness) at their passing as well as a longer, deeper gentle sadness about this state being the reality of life.

That is also basically what the entire “synthwave” genre is about so if you’d like to hear what that concept sounds like as a song, just make your way over to Los Angeles by The Midnight.

Link roundup for February 11, 2023

Cassettes Are Making a Comeback, But Can Production Keep Up? “After music cassettes died in the late ’90s, National Audio kept busy with cassettes for instructional materials, spoken-word bibles and Library of Congress work until indie bands and labels came calling as early as 2006. ‘Suddenly, we were back in business,’ Stepp says.” I love that story. [billboard.com]

Things I Do Not Like Hearing. I appreciate a well-written personal grievances post. This one—about phrases the author doesn’t like—is bound to become a classic of the genre. “I have never read the words ‘friendly reminder’ and not imagined that person seething, incandescent, smoke blowing out of their ears like a hot kettle, just absolutely furious. I simply don’t believe you. I do not think that you think we are friends or that this interaction is friendly. If you want to fight, we can fight.” [holapapi.substack.com]

Engagement, Attention, Shining a Light. This is a great writing goal: “My goal is for my writing to engage readers on a ‘shared inquiry’ level, where whatever I am saying is not viewed as a declaration that demands agreement, but an exploration attempting to illuminate the subject at hand in a way that encourages the reader to go exploring with a light of their own.” [biblioracle.substack.com]

A library of words. I bet you didn’t think you’d want to read about the real purpose of a Thesaurus today, but you’re going to have to trust me. This post is fantastic. “The purpose of an ordinary Dictionary is to simply explain the meaning of the words. After you look up the word, you are given the idea the word is supposed to convey. The Thesaurus is supposed to work in the opposite direction: you start with an idea, and then you find the words to express it. A dictionary turns words into ideas and a thesaurus turns ideas into words.” [austinkleon.substack.com]

SF’s Market Street Subway Is Running on Floppy Disks. This is quite something. “SFMTA is hardly unique in using them, however. As recently as 2020, British Airways was loading avionics software onto 747s via floppy disk.” I also love that they felt the need to include a photo of a floppy disk in the article. [sfstandard.com]

Latex, severed legs and fake erections: why is a whole new generation obsessed with DVD menus? This is a wonderful homage to the lost art of DVD menus. “Some turn-of-the-century landing pages were so imaginative they cut through into popular consciousness: 2003’s House of 1,000 Corpses featured a murderous clown directly addressing (and mocking) the viewer, while the Harry Potter DVD let viewers choose a wand, cast spells, and solve puzzles to access deleted scenes.” [theguardian.com]

An Imperfect List of Books Like “Tomorrow and Tomorrow and Tomorrow”. This is my favorite book I’ve read in a long time. Good list of others to try. [bookriot.com]