Menu

Posts tagged “technology”

Don’t blame outages on human error and technical debt; improve the system instead

The FAA outage in January that caused the first nationwide ground stop of all flights in the US since 9/11 is kind of old news now, but there’s one detail that I can’t stop thinking about. In the aftermath of the incident the cause was determined to be a database sync issue:

The F.A.A. said in a statement that the workers had been trying to “correct synchronization” between the main database for the Notice to Air Missions alerts and a backup database when the files were mistakenly deleted, causing the outage that snarled air traffic throughout the day on Jan. 11.

CNN added a little more detail:

A contractor working for the Federal Aviation Administration unintentionally deleted files related to a key pilot safety system, leading to a nationwide ground stop and thousands of delayed and canceled flights last week, the FAA said Thursday.

The FAA determined the issue with the Notice to Air Missions (NOTAM) system occurred when the contractor was “working to correct synchronization between the live primary database and a backup database.”

The unsurprising narrative that came out of the tech world following the incident can basically be summarized as “ha ha, silly contractors!” But that feels like a lazy response to me. I didn’t see anyone ask what I believe is the more important question: How do we improve the system (people, processes, technology) that enables one person to inadvertently take down all air traffic in the US?

Let’s remember that this kind of thing can happen to absolutely anyone. Etsy even hands out a “three-armed sweater” award to the engineer who had the most spectacular mishap in any given year:

Kate’s story is a nail-biter, involving a tiny code change that unexpectedly brought down Etsy.com. All of her coworkers rallied around her to help get the site back online, while offering words of encouragement and reassurance.

So it might be really convenient to blame the FAA outage on “contractor error” and then just keep going. But that’s not going to prevent the next incident from happening.

It is further also tempting to blame the entire issue on “tech debt” and call it a day. And, fair enough, there’s certainly plenty of that going around in FAA systems. Ars Technica has a good overview of some of the major issues and how the FAA wants to fix them. But like all giant “replatforming” projects (this one is called NextGen, because of course it is) things are… not going great:

FAA tech problems were previously described in a March 2021 report by the US Department of Transportation Office of Inspector General. The report discusses the FAA’s Next Generation Air Transportation System (NextGen), “a multibillion dollar infrastructure project aimed at modernizing our Nation’s aging air traffic system to provide safer and more efficient air traffic management.”

“NextGen’s actual and projected benefits have not kept pace with initial projections due to implementation challenges, optimistic assumptions, and other factors,”1 the report said.

But blaming tech debt—and especially blaming individuals—is not going to get us very far. Tech debt will always be there (although I have some thoughts on how to prioritize it), and individual mistakes are not going to go away. What we can do is examine the system that enables, in this case, a database sync to corrupt the primary live db, and figure out how to prevent that from happening in the first place.

Almost 30 years ago Jakob Nielsen published his 10 Usability Heuristics for User Interface Design, and “error prevention” is still as true today as it was then:

Good error messages are important, but the best designs carefully prevent problems from occurring in the first place. Either eliminate error-prone conditions, or check for them and present users with a confirmation option before they commit to the action.

The example I always think of here is how you often seen battery packs shaped in a certain way so that it’s impossible to insert them incorrectly (contrast that with the terrors of trying to insert a USB cable the correct way the first time!).

In a situation like the one the FAA experienced, yes it’s important to acknowledge human error, and talk about the underlying tech issues, but that’s not enough. We have to figure out how to add preventative measures to our systems and pipelines2. To put it another way, they might not be able to replace their battery packs with NextGen solar yet, but they can certainly change the shape of the battery to prevent contractors from blowing up the camera.


  1. My emphasis added because who among us have not heard those words before… 

  2. For further reading on what to do after a major incident, check out Will Larson’s Move past incident response to reliability

Link roundup for March 3, 2023

The African Bricks 3. Mosaic artworks inspired by the culture and beauty of Africa, by Charis Tsevis.

The Cello in Soho Square. I like this description by Michael Lopp of the difference between “dabblers” and “S-tier” people (who are the absolute best at something): “There is an infinite list of exciting things to learn, but the Dabbler knows they have finite time, so they dabble. They get 80% of the juice, and they move on. Respect. S-Tier knows the last 10% of the challenge is the hardest, but it also teaches you the most.”

Physicists Say Aliens May Be Using Black Holes as Quantum Computers. This is fine. “In a recent study, a German-Georgian team of researchers proposed that advanced extraterrestrial civilizations (ETCs) could use black holes as quantum computers. This makes sense from a computing standpoint and offers an explanation for the apparent lack of activity we see when we look at the cosmos.”

Honestly, it’s probably the phones. Don’t dismiss this argument just from the headline, like I almost did. There’s some solid evidence presented here. “If we’re looking for one big ‘silver bullet’ or ‘grand unified theory’ of modern teenage unhappiness, phones are probably the place to start looking.”

Papercraft Models by Rocky Bergen. “Construct the computer from your childhood or build an entire computer museum at home with these paper models, free to download and share. Print, Cut, Score, Fold and Glue.”

In an Uncertain Job Market, How Can Companies Retain Workers? The conventional wisdom that people tend to hunker down when there are layoffs around them might not be accurate: “Layoffs ‘create an environment where people worry it might happen to them next,’ said Laszlo Bock, who was Google’s SVP for people operations. Poorly handled reductions may ‘degrade trust in management as people start hearing rumors of further cuts, and that in turn raises anxiety, which causes more people to quit.’” (NYT gift article)

How the Phonograph Created the 3-Minute Pop Song. I can’t resist a good “technologies people thought would ruin everything” article, and this is another fascinating one: “Plenty of folks worried that records would destroy musical culture. John Philip figured it would demotivate anyone from learning to play an instrument themselves. Why bother, when you could just put on music by a true virtuoso? ‘When music can be heard in the homes without the labor of study,’ he fretted in a 1906 article, ‘it will be simply a question of time when the amateur disappears entirely.’”

The Case for Hanging Out. I love this essay. “Pushed further into isolation by the pandemic, we’re all losing the ability to engage in what I view as the pinnacle of human interaction: sitting around with friends and talking shit.”

Explore. I think it’s probably too late for a viable LinkedIn alternative, but this site would be a great contendor.

Link roundup for March 1, 2023

Open Circuits is “a photographic exploration of the beautiful design inside everyday electronics. Its stunning cross-section photography unlocks a hidden world full of elegance, subtle complexity, and wonder.”

Good conversations have lots of doorknobs. This is a fascinating essay about the elements of good conversation and the difference between “takers” who keep things going, “givers” who tend to ask a lot of questions, and how the wrong match-up can cause a conversation to stall. Includes good advice backed up by tons of academic research. This is one to save and revisit often.

Why do modern pop songs have so many credited writers? Some of the examples are wild. “When these cases are settled in favor of the plaintiff, more songwriting credits are added after a song’s release. This is why the number of songwriters listed on Mark Ronson’s “Uptown Funk” has increased over the years. To avoid a Mark-Ronson-style-courtroom-induced headache, artists will sometimes preemptively credit writers of older songs even if the similarity between the older song and their composition is purely coincidental.”

A “Last of Us” Episode 7 musical mystery (light spoilers). I just want to say don’t worry The Last of Us fans, I’m thinking about the important things over here.

The choice is easy. Robin Sloan with a good reminder: “Anyone who adds one of those email newsletter pop-ups to a website demeans them selves and makes the world worse for everyone else.” Reminder that if you are an author using Substack you can turn off “Subscribe prompts on post pages” in Settings.

Quick Review Summary. Ok this seems like an actually good use of OpenAI. Instead of poring over hundreds of reviews of a hotel, copy the Tripadvisor URL of the hotel into this website and it will generate a summary of the general sentiment of the hotel.

Neurodiversity Design System. Great resource. “The NDS is a coherent set of standards and principles that combine neurodiversity and user experience design for Learning Management Systems. Design accessible learning interfaces supporting success and achievement for everyone.”

SoundPrint is an app to “discover quiet places and share them with others.” This looks really useful, especially if you’re a fellow tinnitus sufferer.

Link roundup for February 19, 2023

Underwater Photographer of the Year—2023 Winners. These are so great.

Impostered. Great post from Mandy Brown about the need to reframe how we think about imposter syndrome. “I’ve started to think less about imposter syndrome (a description of a person’s experience with it) and more about being impostered (a framing that draws attention to the systems and structures that lead people to believe they are imposters). While the former framing remains useful in many contexts, the latter creates space to consider not only the symptoms but the root cause of the phenomena.” [everythingchanges.us]

What’s So Funny? Very good essay about the current state of stand-up comedy, and what makes something funny. “The audience whooping and applauding Roseanne’s ‘anti-woke’ comedy is not reacting with laughter at a previously un-acknowledged truth, but instead expressing approval for the point of view that they already knew they agreed with. This is not the same thing as laughter in response to a joke.” [biblioracle.substack.com]

Why Are You Seeing So Many Bad Digital Ads Now? “Social media advertising, once a niche art practiced by specialist agencies, is now easily available to anyone. Many of them are eschewing targeted ads — placements intended to reach specific audiences, usually at a higher cost — in favor of a cheaper spray-and-pray approach online, hoping to catch the attention of gullible or bored shoppers.” [NYT gift link]

Traffic Lights Need a Fourth Color, Study Says: Here’s Why. Yeah what could possibly go wrong. “For the dawning age of the self-driving car, transportation engineers from North Carolina State University are proposing the addition of a fourth ‘white light’ whose function would be to alert humans to simply ‘follow the car in front of them.’” [popularmechanics.com]

The people who live inside airplanes. Ok I kind of like this idea. “By the end, she had a fully functional home, with over 1,500 square feet of living space, three bedrooms, two bathrooms and even a hot tub — where the cockpit used to be. All for less than $30,000, or about $60,000 in today’s money.” [cnn.com]

Move past incident response to reliability

Here’s an interesting article by Will Larson with advice on how to move past incident response to reliability in our products. Among other things it reminded me to watch out for “incident legalism”:

Incident legalism is when an incident response and analysis program—trying to better drive reliability improvements—becomes focused on compliance and loses empathy for the engineers and teams operating within the program’s processes.

He goes on to propose a more holistic, expanded model for reliability to help teams diagnose their systemic problems—and how to solve them:

Finally, you study the mitigated incidents, determining how to prevent them from recurring, and they become remediated incidents.

How the fediverse can help us collaborate better at work

Mehul Kar says he’s not super excited about the “fediverse” in the context of social media. However, he sees a huge need for The Fediverse At Work. The issue? The lack of integration across all the tools we use at work has become incredibly tedious and hard to keep track of:

Sometimes there are Figma design specs, with their own set of comments. And Loom walkthroughs, also with comments and likes. And any number of other things over time. The combinatorial complexity of these tools across these platforms (not to mention emails) can be quite messy to track. It’s really hard to remember where a conversation took place. Coworkers often repeat the same text in multiple places, prefixing with phrases like “Shared this in Notion comment also, but…” or “Just left a review, but high level: …”.

He believes that a decentralized platform for all these tools to effectively talk to each other would be hugely beneficial:

Maybe the protocols that make up the Fediverse can help. What if, instead of sharing a Github Pull Request URL in Slack, your Slack team channel could instead be subscribed to the Github repository. Maybe new Pull Requests are broadcasted to followers, and replies from Slack users to those posts are sent as comments to the Pull Request in addition to being threaded in Slack. Maybe the Notion document is treated the same way. Maybe the Loom walk through is a reply to a Slack thread, and comments on the video appear in Slack. Maybe the Slack thread is a series of comments displayed on a Figma design.

There are more examples in his post. I really hope we can get to this type of philosophy for our work tools. It does sound a little bit like the problem that Luro is trying to solve.

Link roundup for February 15, 2023

AMEN by Jessica Hilltout. “The aim of AMEN was to shine the light on all those in the shadow of the World Cup, far from the big stadiums and the corporate carnival-nature of the event. To embrace Africa and everything that makes it unique. To speak of the authenticity and sheer ingenuousness of a continent that manages to do so much with so little. To capture people with simple needs and huge hearts. To express football in its purest form.” [jessicahilltout.com]

God Did the World a Favor by Destroying Twitter. I love Paul Ford. “Our smarter, richer betters often preach the idea of a town square, a marketplace of ideas, a centralized hub of discourse and entertainment—and we listen. But when I go back and read Genesis, I hear God saying: ‘My children, I designed your brains to scale to 150 stable relationships. Anything beyond that is overclocking. You should all try Mastodon.’” [wired.com]

SolidGoldMagikarp (plus, prompt generation). This is all super weird and shows that we really have no idea what’s going on with these LLMs. Are we really ready for this stuff to become the backbone of internet search? “We’ll demonstrate a previously undocumented failure mode for GPT-2 and GPT-3 language models, which results in bizarre completions (in some cases explicitly contrary to the purpose of the model), and present the results of our investigation into this phenomenon.” [lesswrong.com]

Hey, Ease Up; A Load-Bearing If-Statement. What happens if, for health reasons, you need to use a fitness tracker to move less? “But if you’re trying to conserve energy, you don’t want to reach that goal. You want to stay under it. Sure, you want to maybe get up and about, I guess? Take a very slow short walk outside? But you are supposed to be resting.” [newsletter.danhon.com]

Scientists Develop Compound That Kills So Efficiently They Named It After Keanu Reeves. “The molecules ‘kill so efficiently that we named them after Keanu Reeves,’ German researcher Sebastian Götze said in a press release, ‘because he, too, is extremely deadly in his roles.’” [futurism.com]

Buy Nothing groups and the culture of free stuff. This deep-dive into the things people post in Buy Nothinggroups (and that actually get picked up!) is quite something. “There is something about free stuff that makes us abandon all rational thought.” [washingtonpost.com, soft paywall]

The mystery of the disappearing vacation day. Why have we stopped taking regular vacations? “Many were on a paid-time-off (PTO) plan that lumps sick days, personal days and vacation days in a single bucket. While workers often appreciate the flexibility of PTO and employers find it easier to administer, such plans can deter taking long vacations by making us feel as if we’re cutting into the PTO we might need in case of sudden illness or tragedy.” [washingtonpost.com, gift article link]

Nope, coffee won’t give you extra energy. It’ll just borrow a bit that you’ll pay for later. Ok listen, don’t come at me with your “facts”, please. “While it feels energizing, this little caffeine intervention is more a loan of the awake feeling, rather than a creation of any new energy.” [theconversation.com]

Link roundup for February 11, 2023

Cassettes Are Making a Comeback, But Can Production Keep Up? “After music cassettes died in the late ’90s, National Audio kept busy with cassettes for instructional materials, spoken-word bibles and Library of Congress work until indie bands and labels came calling as early as 2006. ‘Suddenly, we were back in business,’ Stepp says.” I love that story. [billboard.com]

Things I Do Not Like Hearing. I appreciate a well-written personal grievances post. This one—about phrases the author doesn’t like—is bound to become a classic of the genre. “I have never read the words ‘friendly reminder’ and not imagined that person seething, incandescent, smoke blowing out of their ears like a hot kettle, just absolutely furious. I simply don’t believe you. I do not think that you think we are friends or that this interaction is friendly. If you want to fight, we can fight.” [holapapi.substack.com]

Engagement, Attention, Shining a Light. This is a great writing goal: “My goal is for my writing to engage readers on a ‘shared inquiry’ level, where whatever I am saying is not viewed as a declaration that demands agreement, but an exploration attempting to illuminate the subject at hand in a way that encourages the reader to go exploring with a light of their own.” [biblioracle.substack.com]

A library of words. I bet you didn’t think you’d want to read about the real purpose of a Thesaurus today, but you’re going to have to trust me. This post is fantastic. “The purpose of an ordinary Dictionary is to simply explain the meaning of the words. After you look up the word, you are given the idea the word is supposed to convey. The Thesaurus is supposed to work in the opposite direction: you start with an idea, and then you find the words to express it. A dictionary turns words into ideas and a thesaurus turns ideas into words.” [austinkleon.substack.com]

SF’s Market Street Subway Is Running on Floppy Disks. This is quite something. “SFMTA is hardly unique in using them, however. As recently as 2020, British Airways was loading avionics software onto 747s via floppy disk.” I also love that they felt the need to include a photo of a floppy disk in the article. [sfstandard.com]

Latex, severed legs and fake erections: why is a whole new generation obsessed with DVD menus? This is a wonderful homage to the lost art of DVD menus. “Some turn-of-the-century landing pages were so imaginative they cut through into popular consciousness: 2003’s House of 1,000 Corpses featured a murderous clown directly addressing (and mocking) the viewer, while the Harry Potter DVD let viewers choose a wand, cast spells, and solve puzzles to access deleted scenes.” [theguardian.com]

An Imperfect List of Books Like “Tomorrow and Tomorrow and Tomorrow”. This is my favorite book I’ve read in a long time. Good list of others to try. [bookriot.com]

20 Things I've Learned in my 20 Years as a Software Engineer

Old technologies that have stuck around are sharks, not dinosaurs. They solve problems so well that they have survived the rapid changes that occur constantly in the technology world. Don’t bet against these technologies, and replace them only if you have a very good reason. These tools won’t be flashy, and they won’t be exciting, but they will get the job done without a lot of sleepless nights.

—Justin Etheredge, 20 Things I’ve Learned in my 20 Years as a Software Engineer.

Link roundup for February 9, 2023

Lego reveals massively detailed Lord of the Rings Rivendell set. Take my money! [polygon.com]

The Window Trick of Las Vegas Hotels. “In order to make the buildings look smaller, less intimidating and messy, architects have come up with a ‘four or six windows in one’ solution. This means they grouped several windows (usually four or six) together and made them look like one window. This creates the visual effect of ‘shrinking’ the building, of making it more orderly and symmetrical.” [schedium.net]

You have to read this whole article for the full context, but this classification of the different ways we can choose to act online really got me thinking: “I see roughly three typical public stances: boring, lively, or outraged. Either you act boring, so the bandits will ignore you, you act lively, and invite bandit attacks, or you act outraged, and play a bandit yourself. Most big orgs and experts choose boring, and most everyone else who doesn’t pick boring picks bandit, especially on social media. It takes unusual art, allies, and energy, in a word “eliteness”, to survive while choosing lively. And that, my children, is why the world looks so boring.” [overcomingbias.com]

The Junkification of Amazon. Why Does It Feel Like Amazon Is Making Itself Worse? “If you understand Amazon as an aspiring megascale infrastructure company — a provider of systems, services, capacity, and labor — its junkification makes sense. Amazon hasn’t been acting like a store for a while. In its ideal future, selling things to people is everyone else’s problem.” [nymag.com]

People Can’t Stop ‘Spotify Snooping’ on Friends, Exes and Crushes (WSJ paywall, Archive.is link here). “When Ms. Ticoalu looked up what her ex-boyfriend was listening to in November, she saw ‘Glimpse of Us’ by Joji, a song about starting to date again after a relationship ended. Because he played the song so soon after their breakup, it led her to believe the two events were related. ‘It does lead me to overthink a lot,’ Ms. Ticoalu says.” [wsj.com]

New Form of Ice Discovered. “The newly discovered ice is amorphous — that is, its molecules are in a disorganized form, not neatly ordered as they are in ordinary, crystalline ice. Amorphous ice, although rare on Earth, is the main type of ice found in space. That is because, in the colder environment of space, ice does not have enough thermal energy to form crystals.” [scitechdaily.com]

This is such a fun and interesting story by Louie Mantia about his time working as an icon/UI designer at Apple in the early 2010s. [lmnt.me]