I’ve been seeing a lot of this type of sentiment about React recently…

By my reckoning, if you’ve maintained a React codebase for the past decade, you’ve re-written your application at least three times and possibly four. […]

By choosing React, we’ve signed up for a lot of unplanned work. Think of the value we could have produced for our users and company if we weren’t subject to the whims of whatever the cool kids were doing over in React.

My work at Jeli so far has given me a new lens on “incidents”—both in the software world and beyond—that I didn’t have before. These “failures” are everywhere around us. But are they really failures? Or are they ways for us to learn more about the systems we work within, and how to improve them? I think it’s the latter, and My $500M Mars Rover Mistake by Chris Lewicki is another story that showcases that…

The core lesson I’ve drawn from my rover ordeal is best expressed in these words: Let your scars serve you; they are an invaluable learning experience and investment in your capability and resilience.

There’s some good insights in this year’s 2023 State of DevOps Report. It’s well worth skimming through. Things like this aren’t exactly surprising, but it’s nice to have some data around it:

Teams with generative cultures, composed of people who felt included and like they belonged on their team, have 30% higher organizational performance than organizations without a generative culture.

This is an older post from Lorin Hochstein but it’s new to me, and really insightful. It’s about how to best use our knowledge about the past behavior of a software system to figure out where we should invest our time to improve the system—and how the common method of error budgets is generally not a good way to do this:

I’m skeptical about relying on predefined metrics, such as reliability, for getting insight into the risks of the system that could lead to big incidents. Instead, I prefer to focus on signals, which are not predefined metrics but rather some kind of information that has caught your attention that suggests that there’s some aspect of your system that you should dig into a little more.

So basically, vibe-based incident analysis is where it’s at.

This is a great document outlining the communication principles followed by GitHub Engineering. I’d say this is broadly applicable to teams and organizations—not just Engineering. I love this point about making work visible:

Capturing and exposing processes through URLs also helps make your work more visible. So work in the open and proactively share your work to the widest extent practical. As we continue to grow as an organization, points of collaboration will become even more important as we try to reduce redundant work. Avoid hoarding information: Like in any production system, observability is key. And if you make something useful, find a way to make it available so others can benefit from it too.

I’ve always been adamant that Product and Engineering are in a partnership, not a “healthy tension” relationship. So I very much agree with this post:

The problem is in the assumption that Product and Engineering teams inherently have different goals. They don’t. Both teams are responsible for the growth and stability of the company, for revenue and scalability. Neither can succeed without the other. When we assume otherwise, we sell each side short.

Here’s a good reminder that Incidents can’t be prevented, but learned from:

But approaching incidents with a mindset of learning makes it an exciting rather than painful situation. Because you’ll know you’ll never run out of sources for learning. And once you’ve realised what a good source for learning incidents are, it’s maybe even time to take a good look whether shallow incident data like “mean time to detection” and “mean time to resolution” (or the maybe worst offender of all “mean time between failure”) are actually helping your team approach incidents as a learning opportunity or maybe are incentivising an approach that foregoes learning for a better look of those metrics.

Here’s a great post by Camille Fournier about The Product Culture Shift, and how every part of an engineering culture needs to change when product managers are added to traditional software infrastructure organizations.

To start, let’s be clear about one thing: as tempting as it might be, just hiring product managers won’t fix this problem. Even if you could find enough good product managers who want this type of job, which you can’t, product managers are only useful when they are paired with willing engineering teams. If the engineering teams don’t feel a sense of ownership for delivering a great product to their customers, product managers are unlikely to close that gap, and they will more likely turn into glorified backlog groomers than true product leaders.

I love the point Simon Sarris makes here about the importance of reading fiction, and how it’s useful for work purposes as well:

I also tend to stress fiction because I think, especially among my professional peers in the industry of software, that there is too great a fondness for non-fiction. I think this arises from a belief that superior knowledge of the world comes from non-fiction. This thought is attractive to people who build systems, but over-systematizing and seeing systems in everything can be a failure mode. Careful descriptions and summaries miss too much of the world. Hard distinctions make bad philosophy. Reading fiction helps you become an unsystematic thinker, something that is equally valuable but more elided by some engineers. It is easy to maintain an intellectual rigidity. It takes more care to maintain a loose poeticism of thought.

Here’s a good post from Gergely Orosz discussing Interesting Learnings from Outages. It covers internal vs. public postmortems, how investing in reliability can have bumps along the way, and how to make the difficult decision to try and fix something on the spot, or to do a lengthy restore. This point stood out to me:

“Move fast with autonomous teams” often builds up infrastructure debt. Reddit is a fast-moving scaleup where teams move fast, and it sounded like they had autonomy in infrastructure decisions. The wide range of infra configurations caused several outages, and the company is now paying down this “infrastructure debt.” This is not to say that autonomous teams moving fast is a bad thing, but it’s a reminder that this approach introduces tradeoffs that could impact reliability and will eventually have to be paid down, often by dedicated teams.