Lorin Hochstein shares a characteristically solid systems-thinking take in CrowdStrike: how did we get here?:

Systems reach the current state that they’re in because, in the past, people within the system made rational decisions based on the information they had at the time, and the constraints that they were operating under. The only way to understand how incidents happen is to try and reconstruct the path that the system took to get here, and that means trying to as best as you can to recreate the context that people were operating under when they made those decisions.

The “no root cause” concept is something I’ve been thinking about a lot as I’m working on a particularly complex project at work. Somehow we constantly forget that things usually are the way they are not because of a single “mistake”, but because of a the culmination of a bunch of legitimate reasons.

Systems get the way they are because of decisions made in good faith based on the data available at the time. And the worst thing you can do as a new person coming in to improve things is to hunt for a single “root cause” to fix. That’s just not how software (or people!) work. So take the time to understand Chesterson’s fence. Go ahead and draw boxes and arrows until no one disagrees any more about how the system works. And then figure out which parts can be improved, and in which order.

---

PS. Also see How Complex Systems Fail:

Because overt failure requires multiple faults, there is no isolated ‘cause’ of an accident. There are multiple contributors to accidents. Each of these is necessarily insufficient in itself to create an accident. Only jointly are these causes sufficient to create an accident. Indeed, it is the linking of these causes together that creates the circumstances required for the accident. Thus, no isolation of the ‘root cause’ of an accident is possible. The evaluations based on such reasoning as ‘root cause’ do not reflect a technical understanding of the nature of failure but rather the social, cultural need to blame specific, localized forces or events for outcomes.

This is a fun bit of gaming history:

The thing about games, particularly Japanese games, is that they exist in a symbiotic relationship with manga and anime. When I spoke to Uemura-san, the engineer of the NES, he said ‘Once hardware developed to the point where you could actually draw characters, designers had to figure out what to make. Subconsciously they turned to things they’d absorbed from anime and manga. We were sort of blessed in the sense that foreigners hadn’t seen the things we were basing our ideas on.’

Emma Sarappo has a fascinating review of a new book about the Challenger space disaster (gift article). It is the first global disaster I was old enough to witness and experience in real time, and I’ve never been able to get those images out of my head. This review (and book) shines a horrifying light on the many human failures (mainly due to hubris) that resulted in this tragedy:

These issues—faulty O-rings, foam strikes—were understandable. Theoretically, with study and ingenuity, they were fixable. The problem was not really a lack of technical knowledge. Instead, human fallibility from top to bottom was at issue: a toxic combination of financial stress, managerial pressure, a growing tolerance for risk, and an unwillingness to cause disruption and slow down scheduled launches.

(Side note, tell me that last sentence doesn’t remind you of software development sometimes…)

Also, the astronauts knew what was happening:

The astronaut assisting them into place and finishing final preflight checks “looked down into her face and saw that her Girl Scout pluck had deserted her,” he writes. “In her eyes he saw neither excitement nor anticipation, but recognized only one emotion: terror.” She would fly for 73 seconds before the shuttle broke apart in a fireball and a cloud of smoke. After that gut-wrenching instant, and more seconds of stunned silence, a NASA public-affairs officer would speak the understatement that would become famous: “Obviously a major malfunction.”

Yes, I’m a relentless fanboy of whatever Paul Ford writes, but this is a truly wonderful post about what makes AI so addictive and impossible to look away from. He frames AI as a technology that truly has no shame because “it possesses an absolute willingness to spout foolishness, balanced only by its carefree attitude toward plagiarism.” And so:

By aggregating the world’s knowledge, chomping it into bits with GPUs, and emitting it as multi-gigabyte software that somehow knows what to say next, we’ve made the funniest parody of humanity ever. These models have all of our qualities, bad and good. Helpful, smart, know-it-alls with tendencies to prejudice, spewing statistics and bragging like salesmen at the bar. They mirror the arrogant, repetitive ramblings of our betters, the horrific confidence that keeps driving us over the same cliffs. That arrogance will be sculpted down and smoothed over, but it will have been the most accurate representation of who we truly are to exist so far, a real mirror of our folly, and I will miss it when it goes.

I finally read this very long essay about Rewilding the Internet that’s been making the rounds. It’s about 30 mins of your time and in my opinion it’s time well spent.

It’s about what internet-builders can learn from the field of ecology, where the word “rewilding” has a very specific meaning. It’s essentially about systems thinking, which I know a lot of us care about deeply.

Rewilding the internet is not a nostalgia project for middle-aged nerds who miss IRC and Usenet. For many people across the generations today, platforms like Facebook or TikTok are the internet. They’ve long dwelled in walled gardens they think are the world. Concentrated digital power produces the same symptoms that command and control produces in biological ecosystems; acute distress punctuated by sudden collapses once tipping points are reached. Rewilding is a way to collectively see the counterintuitive truth; today’s internet isn’t too wild, even if it feels like that. It’s simply not wild enough.

In the end, I can’t help but think that though I love these ideas, it’s just… too late. I hope I’m wrong though.

I recently mentioned how I like to draw it until it works when I’m ramping up on a new system. Clint Byrum says it so much better in his post System Diagrams are Performance Caches for Cognitive Load. First, this bit resonated with me because it’s exactly the situation I currently find myself in:

Having joined just a few months ago, I was overwhelmed about 5 minutes into the meeting. The individual words and concepts all made sense. JSON parsing slow. Network transit treacherous. Changing things at the source hard. I got all of those components of the discussion, but through the whole thing I was just barely able to follow the overall system conversation and ask very basic questions to understand what was going on. I came away with a bunch of exploratory personal action items, and a very clear hole in my mental model of the system that needs to be filled.

Clint goes on to use a systems analogy for the individual people that make up a team—people and knowledge as components of caching, computation, and storage. This leads to a perfect explanation for why system diagrams are so important:

A single system diagram is where those primed nodes can push the most relevant bits of their information out of their local brain-caches, and into a high-performance distributed cache from which everyone can read. This will preserve precious cognitive load for those critical low-latency tasks. Of course, all of these caches may be stale. The local in-memory ones are particularly hard to test, but at least the system diagram is observable. Everyone can look at it, and if there are nodes with updates, they can update the cache.

So, prime those caches. Draw it until it works!

This isn’t entirely surprising but it’s a sad state of affairs, and it’s worth highlighting not just how, but also where LLMs are being trained:

Hundreds of thousands of hours of work goes into providing enough feedback to turn an LLM into a useful chatbot, and that means the large AI companies outsource the work to parts of the global south, where anglophonic knowledge workers are cheap to hire.

I know it’s too dismissive to call chatbots “fancy autocomplete” like many do, but we have to remember that this isn’t magic. The words the bots use come from somewhere. And in the case of “delve”…

I said “delve” was overused by ChatGPT compared to the internet at large. But there’s one part of the internet where “delve” is a much more common word: the African web. In Nigeria, “delve” is much more frequently used in business English than it is in England or the US. So the workers training their systems provided examples of input and output that used the same language, eventually ending up with an AI system that writes slightly like an African.