Lorin Hochstein shares a characteristically solid systems-thinking take in CrowdStrike: how did we get here?:

Systems reach the current state that they’re in because, in the past, people within the system made rational decisions based on the information they had at the time, and the constraints that they were operating under. The only way to understand how incidents happen is to try and reconstruct the path that the system took to get here, and that means trying to as best as you can to recreate the context that people were operating under when they made those decisions.

The “no root cause” concept is something I’ve been thinking about a lot as I’m working on a particularly complex project at work. Somehow we constantly forget that things usually are the way they are not because of a single “mistake”, but because of a the culmination of a bunch of legitimate reasons.

Systems get the way they are because of decisions made in good faith based on the data available at the time. And the worst thing you can do as a new person coming in to improve things is to hunt for a single “root cause” to fix. That’s just not how software (or people!) work. So take the time to understand Chesterson’s fence. Go ahead and draw boxes and arrows until no one disagrees any more about how the system works. And then figure out which parts can be improved, and in which order.

---

PS. Also see How Complex Systems Fail:

Because overt failure requires multiple faults, there is no isolated ‘cause’ of an accident. There are multiple contributors to accidents. Each of these is necessarily insufficient in itself to create an accident. Only jointly are these causes sufficient to create an accident. Indeed, it is the linking of these causes together that creates the circumstances required for the accident. Thus, no isolation of the ‘root cause’ of an accident is possible. The evaluations based on such reasoning as ‘root cause’ do not reflect a technical understanding of the nature of failure but rather the social, cultural need to blame specific, localized forces or events for outcomes.

Charity Majors makes some really interesting points about the consequences of “replacing” junior engineers with AI:

By not hiring and training up junior engineers, we are cannibalizing our own future. We need to stop doing that. People act like writing code is the hard part of software. It is not. It never has been, it never will be. Writing code is the easiest part of software engineering, and it’s getting easier by the day. The hard parts are what you do with that code—operating it, understanding it, extending it, and governing it over its entire lifecycle.

Also:

To state the supremely obvious: giving code review feedback to a junior engineer is not like editing generated code. Your effort is worth more when it is invested into someone else’s apprenticeship. It’s an opportunity to pass on the lessons you’ve learned in your own career. Even just the act of framing your feedback to explain and convey your message forces you to think through the problem in a more rigorous way, and has a way of helping you understand the material more deeply. And adding a junior engineer to your team will immediately change team dynamics. It creates an environment where asking questions is normalized and encouraged, where teaching as well as learning is a constant.

Lots more in the full article, it’s well worth a full read.

Emma Sarappo has a fascinating review of a new book about the Challenger space disaster (gift article). It is the first global disaster I was old enough to witness and experience in real time, and I’ve never been able to get those images out of my head. This review (and book) shines a horrifying light on the many human failures (mainly due to hubris) that resulted in this tragedy:

These issues—faulty O-rings, foam strikes—were understandable. Theoretically, with study and ingenuity, they were fixable. The problem was not really a lack of technical knowledge. Instead, human fallibility from top to bottom was at issue: a toxic combination of financial stress, managerial pressure, a growing tolerance for risk, and an unwillingness to cause disruption and slow down scheduled launches.

(Side note, tell me that last sentence doesn’t remind you of software development sometimes…)

Also, the astronauts knew what was happening:

The astronaut assisting them into place and finishing final preflight checks “looked down into her face and saw that her Girl Scout pluck had deserted her,” he writes. “In her eyes he saw neither excitement nor anticipation, but recognized only one emotion: terror.” She would fly for 73 seconds before the shuttle broke apart in a fireball and a cloud of smoke. After that gut-wrenching instant, and more seconds of stunned silence, a NASA public-affairs officer would speak the understatement that would become famous: “Obviously a major malfunction.”

I’ve been pretty bad at sharing what I’m reading over the past couple of weeks, so I wanted to do a quick roundup post of some articles I have found helpful recently. I keep wanting to write longer posts about each, but it’s just not happening due to time constraints, and I need to clear out that queue, so here goes!

Unexpected Anti-Patterns for Engineering Leaders

Insightful post by Will Larson with some great advice for all leaders, including a reminder that it’s not a good idea to extract yourself from the details completely:

New engineering managers are often advised to “step away from the code.” But an extremely high-functioning exec understands the domain they are operating in at some level of detail. As you get too far out of the details, you just become a bureaucrat. Too many well-meaning engineering managers end up as bureaucrats.

The Busy Trap: How to Distinguish Being Busy from Being Productive

This one deserves a thorough read, because it talks about maximizing throughput in systems—and it’s not about making sure everyone on the team is at 100% capacity:

The real bottleneck isn’t in doing the work but in the waiting—queues that turn days of development into weeks of delay. This insight shifts the focus from individual busyness to process efficiency and the smooth flow of work from start to finish. Our findings debunk the myths that more planning, parallel tasks, and pull requests guarantee better outcomes. Instead, they emphasize the need for streamlined processes and effective collaboration to enhance true productivity. Let’s prioritize making each moment count towards faster, more efficient delivery, moving beyond mere activity to meaningful progress.

How to Communicate When Trust Is Low (Without Digging Yourself Into A Deeper Hole)

Lots of things in this post that I found very relevant and helpful. I especially have a… well, “growth edge”… in this area:

To the best of your ability, try to resist reading layers of meaning into textual communication; keep it simple, overcommunicate intent, and ask for clarity. And if someone is asking you for clarity, help them do a better job for you.

And related…

You vs. the forgetting curve

My friend and former colleague Fio with another excellent newsletter edition about why we should be ok with having to repeat ourselves a lot:

And it’s not that other people forget because they are selfish or don’t care—it’s because we plonked a bit of information at the top of their forgetting curve […] Because the forgetting curve exists, being an effective communicator might well require us to share the same information multiple times, at the right intervals, across different channels, without ever assuming that our teams and stakeholders will magically remember everything after the first iteration.

Principles of Engineering Culture at Wealthfront

I love (most of) these principles, but especially this one:

We believe our organization is most healthy when engineers, not management, propose and drive platform improvements. New products and problems are often brought to engineering teams to solve, but then technical leadership of these teams interweave these priorities with necessary infrastructure as part of their platforms’ continued advancement in engineering quality. While it is the team’s job to propose and defend these improvements, it is then management’s job to internally align and clear the path for the improvements to happen. The alternative would be for management to command infrastructure projects that teams then find time to execute. Such management decrees are avoided as they lead to poor trade-offs and unhappy teams.

Clarity for Product Managers: Part 1, Directional Clarity

Great series by Arne Kittler. This quote stood out to me because I’ve been surprised at how many people I’m currently interviewing for roles on my team cannot succinctly describe what value PMs bring to an organization:

Imagine you meet your CFO in the elevator: How well can you briefly and convincingly tell them what you are doing and why, and what the company will gain from it?

The Moral Economy of the Shire

On the non-work side of things, I adore real-world critiques of fictional worlds, like this one:

The implication in both books and movies is that most Hobbits spent their time either farming or enjoying leisure time, but this makes little sense, when one considers what we know about premodern agriculture and what little of life and culture in the Shire.

We have to realize that Bilbo and Frodo were independently wealthy…

Bilbo and Frodo are both gentlemen of leisure because the Baggins family is independently wealthy, and that wealth almost has to come from land ownership, because there isn’t enough industry or trade to sustain it. They can afford to go on adventures and study Elven poetry because they draw their income from tenant farmers renting their land.

I really like this No Wrong Doors approach, and I think we can learn a lot from it in modern knowledge organizations:

Some governmental agencies have started to adopt No Wrong Door policies, which aim to provide help–often health or mental health services–to individuals even if they show up to the wrong agency to request help. The core insight is that the employees at those agencies are far better equipped to navigate their own bureaucracies than an individual who knows nothing about the bureaucracy’s internal function. […]

Something I’ve been thinking about recently is how engineering organizations can adopt a variant of the No Wrong Doors policy to directly connect folks with problems with the right team and information. Then the first contact point becomes a support system for navigating the bureaucracy successfully.

The Slackification of the workplace has, among other things, resulted in too many different places someone might be able to go for help. It’s frustrating to be sent from team to team, with no one really taking the time to understand and assist with the problem. What if we took a “No Wrong Channel” approach instead? I know it takes a bit of extra time, but I think it’s a worthy goal to become “a support system for navigating the bureaucracy successfully” when someone wanders into our team channel with a question that is not necessarily in our direct sphere of influence.

I recently mentioned how I like to draw it until it works when I’m ramping up on a new system. Clint Byrum says it so much better in his post System Diagrams are Performance Caches for Cognitive Load. First, this bit resonated with me because it’s exactly the situation I currently find myself in:

Having joined just a few months ago, I was overwhelmed about 5 minutes into the meeting. The individual words and concepts all made sense. JSON parsing slow. Network transit treacherous. Changing things at the source hard. I got all of those components of the discussion, but through the whole thing I was just barely able to follow the overall system conversation and ask very basic questions to understand what was going on. I came away with a bunch of exploratory personal action items, and a very clear hole in my mental model of the system that needs to be filled.

Clint goes on to use a systems analogy for the individual people that make up a team—people and knowledge as components of caching, computation, and storage. This leads to a perfect explanation for why system diagrams are so important:

A single system diagram is where those primed nodes can push the most relevant bits of their information out of their local brain-caches, and into a high-performance distributed cache from which everyone can read. This will preserve precious cognitive load for those critical low-latency tasks. Of course, all of these caches may be stale. The local in-memory ones are particularly hard to test, but at least the system diagram is observable. Everyone can look at it, and if there are nodes with updates, they can update the cache.

So, prime those caches. Draw it until it works!