Menu

Posts tagged “engineering”

Move past incident response to reliability

Here’s an interesting article by Will Larson with advice on how to move past incident response to reliability in our products. Among other things it reminded me to watch out for “incident legalism”:

Incident legalism is when an incident response and analysis program—trying to better drive reliability improvements—becomes focused on compliance and loses empathy for the engineers and teams operating within the program’s processes.

He goes on to propose a more holistic, expanded model for reliability to help teams diagnose their systemic problems—and how to solve them:

Finally, you study the mitigated incidents, determining how to prevent them from recurring, and they become remediated incidents.

How the fediverse can help us collaborate better at work

Mehul Kar says he’s not super excited about the “fediverse” in the context of social media. However, he sees a huge need for The Fediverse At Work. The issue? The lack of integration across all the tools we use at work has become incredibly tedious and hard to keep track of:

Sometimes there are Figma design specs, with their own set of comments. And Loom walkthroughs, also with comments and likes. And any number of other things over time. The combinatorial complexity of these tools across these platforms (not to mention emails) can be quite messy to track. It’s really hard to remember where a conversation took place. Coworkers often repeat the same text in multiple places, prefixing with phrases like “Shared this in Notion comment also, but…” or “Just left a review, but high level: …”.

He believes that a decentralized platform for all these tools to effectively talk to each other would be hugely beneficial:

Maybe the protocols that make up the Fediverse can help. What if, instead of sharing a Github Pull Request URL in Slack, your Slack team channel could instead be subscribed to the Github repository. Maybe new Pull Requests are broadcasted to followers, and replies from Slack users to those posts are sent as comments to the Pull Request in addition to being threaded in Slack. Maybe the Notion document is treated the same way. Maybe the Loom walk through is a reply to a Slack thread, and comments on the video appear in Slack. Maybe the Slack thread is a series of comments displayed on a Figma design.

There are more examples in his post. I really hope we can get to this type of philosophy for our work tools. It does sound a little bit like the problem that Luro is trying to solve.

Principles for building software for developers

Kathy Korevec started a series about her principles for building software/tools for developers. Since I work on Postmark—one such tool—I read the intro post with great interest. The second installment is on the principle she calls You are a chef cooking for chefs:

Developers are masters of building applications, so when you’re building tools and experiences for them, you’re cooking in their kitchen. You can marvel at the delight you bring to the experience because no one can appreciate your hard work more than another developer. Developers can spot inconsistencies, antipatterns, and hurdles a mile away, so you must pay close attention to these details. At the same time, they know the challenges, understand the concerns, appreciate the details, and can provide crucial feedback to make your product even better.

This is one of the main reasons why I love working on developer tools. It’s an audience that can be brutal critics. But for the most part they do that because they care and want to see the product succeed—not because they want to fight just for the sake of it. And because they care, feedback generally have a degree of specificity that is invaluable for troubleshooting, use case discovery, and improving the product.

Anyway, this looks like a fantastic series and I can’t wait to read the rest. You can sign up for Kathy’s newsletter here.

Advice For Engineers, From A Manager

Marco Rogers has been an engineer and manager of engineers for 20 years. In this post he shares some short, practical (but not always easy to follow!) advice for engineers. A few of my favorites:

  • Learn what the true scope of the project needs to be. Back away from “story points” and understand what the project needs to accomplish. More context about the goals will help you negotiate what’s in and what’s out of scope.
  • Collaborate on designs. Designs never have the level of detail that matters. When you run into UX problems, work with people to develop a solution. Don’t just ask for more mocks. Own the details of what you’re building.
  • Don’t just write code. Solve problems. Make sure you understand the value of your work and you talk to people about that. Not just “features”. For example, “this needs to ship by Fall because it’s our big strategic bet for the year.” Tell people how to achieve the strategic goal.

Read the rest of his post for the others.

20 Things I've Learned in my 20 Years as a Software Engineer

Old technologies that have stuck around are sharks, not dinosaurs. They solve problems so well that they have survived the rapid changes that occur constantly in the technology world. Don’t bet against these technologies, and replace them only if you have a very good reason. These tools won’t be flashy, and they won’t be exciting, but they will get the job done without a lot of sleepless nights.

—Justin Etheredge, 20 Things I’ve Learned in my 20 Years as a Software Engineer.

Human systems and the stuff they make

There’s so much goodness in Mandy Brown’s latest management post Made It, but it’s this paragraph that gets to the heart of it:

This is one of the ways that the adage about “maker time” versus “manager time” does harm. Setting aside the fact that it’s not even coherent—managers make decisions and plans and systems and visions all day long!—it also serves this notion that the stuff is what matters and the humans are just an unruly mess that gets in the way of the stuff. This is a self-defeating story, inasmuch as you don’t get good stuff without a system that takes care of the humans.

I have been thinking about human systems a great deal over the past few weeks. It started when I read Donella Meadows’s Dancing With Systems. When I applied that lens to my own work it was impossible not to see it in absolutely everything we do. Take this quote, for instance:

Don’t maximize parts of systems or subsystems while ignoring the whole. As Kenneth Boulding once said, Don’t go to great trouble to optimize something that never should be done at all. Aim to enhance total systems properties, such as creativity, stability, diversity, resilience, and sustainability–whether they are easily measured or not.

There are so many implications of this principle for the way we make software today. Here are a couple I’ve been thinking about in particular.

Quality

When we have quality issues in our product it’s tempting to try to solve those issues with process and overhead. Common ways to do that include instituting additional approval gates before code can be deployed to production environments, or adding “QA periods” where the entire organization is asked to test a feature before it goes live.

These solutions optimize for parts of the system—such as deployments or manual testing—while ignoring the system that is responsible for introducing quality issues into the product in the first place. To quote Deming:

Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product. As Harold F. Dodge said, “You can not inspect quality into a product.”

We have to look at the system as a whole. What kind of automated test coverage do we have? What bottlenecks exist in our CI/CD pipelines? What level of responsibility do we ask of engineers to be stewards of their own code? Do teams have enough time to make sure that quality is built in from the start and not an afterthought? These are the total systems properties questions that have to be addressed in order to fix quality issues.

Team stability

One of the bigger leadership mistakes we make is to think that the humans that make up a social system are interchangeable and can be moved around at will without impacting the system as a whole. You see this especially in engineering teams where “resources” are constantly moved from one project to another—usually to speed something up that is perceived to be behind schedule. But we forget that it is the human system that produces the stuff, not the individuals.

Similar to the discussion about quality, if we perceive a project to be off track we need to understand how the project came to be off track, and then find ways to improve the system to correct that. Did we place unrealistic deadlines on the team without getting their input? Are we asking teams to build features that have not been validated with customers? Do teams have adequate autonomy and ownership over the way they prioritize their work? Do teams have the business context they need to make the best possible decisions?

If we instead go for the shortcut of moving people around, we will never get to the heart of the matter. Inserting additional humans into a system can inadvertently break it—and worse, moving folks from one project to another leaves a gap that is very likely to destabilize the project that they were working on before. Suddenly Customer Success doesn’t have an Engineer to talk to about documentation, or the Marketing Team doesn’t have anyone to fix an issue with a piece of demo software.

Once again, Donella Meadows says this well:

Before you disturb the system in any way, watch how it behaves. If it’s a piece of music or a whitewater rapid or a fluctuation in a commodity price, study its beat. If it’s a social system, watch it work. Learn its history. Ask people who’ve been around a long time to tell you what has happened.

Stable teams that work together for an extended period of time all have a steady beat, and we shouldn’t make changes to the system until we know exactly how that beat works. Here’s Donella again: “Before you charge in to make things better, pay attention to the value of what’s already there.”

And that’s how all this connects to Mandy’s piece that I linked to at the beginning. If we want the stuff to be good, we have to pay attention to the beat of the system of humans that make up an organization. We have to understand how they work, and we have to ask them how they want to work. We have to listen to the people who are beating the drums.

It may sound a little counter-intuitive, but if we optimize for the humans who make up a system, the good stuff will follow.

→ Discuss this post on Mastodon

Link roundup for February 1, 2023

Inside the Globus INK: a mechanical navigation computer for Soviet spaceflight. Dang this thing is beautiful. And what an amazing piece of engineering.

Youth Spies and Curious Elders. Life goals: “Waters is what I call a Curious Elder — someone who manages to retain their curiosity as they age and stays interested in what young people are up to. The curious elder isn’t interested in judging youth, they’re interested in learning from them.”

The Thoughts of a Spiderweb. This is a fascinating article about animal cognition but I am especially blown away by the idea of spiderwebs being “an extension of the spider’s cognitive system” because I’m reading the sci-fi novel Children of Time right now and that is how the spiders communicate in the story.

Audi’s new EV is a luxury SUV with augmented reality that doubles as a pickup. I can’t decide if I love this or hate it.

Prisoners Usually Can’t Have Cell Phones. See How People Use Them Anyway. “Some men use their phones to take online classes, posing as regular students in the free-world, a ruse that only works in the age of Zoom classrooms and online meetings.”

It’s the Coolest Rock Show in Ann Arbor. And Almost Everyone There Is Over 65. “The show always starts at 6:30 p.m. and ends at 9 p.m., in time to get to bed at a reasonable hour.” Sign me up! (NYT Gift Link)

How ‘The Last of Us’ changed gaming, strained relationships and spawned an empire. Probably my favorite read of the week. “‘The Last of Us’ always felt like a mission statement, a game that wanted to prove that big-budget action shooters could not only have a sense of gravitas but could advance the medium in narrative, gameplay and representation.”

Engineering maturity models and the importance of a strong foundation above all else

In his article Engineering Maturity Model Mike Fisher shares how he thinks about the importance of different team capabilities when building software organizations. Despite how some maturity models—such as the Capability Maturity Model (CMM)—have been misused in the past, Mike encourages us to look past the process and focus on the principles. Here’s the important part:

[The layers] aren’t stages in that you’re never really finished with any of them but you do need to have the ones at the base stronger and more developed than the ones further up or else you are certain to run into problems. […]

While I do think of this kind of like a maturity model, they are not stages that one achieves and moves on from. These are areas that one must keep returning to and keep investing in, always from the bottom up. Getting over your skis and investing too much in the top, which is very tempting for startups, is fraught. Too many product development teams without continued investment in the infrastructure or deployment pipeline can slow everyone down, proving Brooks’s Law. The important task for Engineering leaders is to determine when and how much investment gets made into each of these layers.

To put it another way, if the base of your infrastructure and deployment pipeline is shaky, an increased focus on product development is eventually going to bring the whole house down. Click through to the article to see Mike’s full model.

You can't just cancel 76,500 hours of meetings

Hot on the heels of yesterday’s post Meetings for an effective engineering organization, I bring you more meeting content! In You can’t just cancel 76,500 hours of meetings Becky Kane makes some good points about the context of meetings within an async culture:

Reducing meetings is just one piece of creating an async-first culture.

She gives some examples of other pieces that are harder but even more important in having a lasting impact on engagement and productivity:

  • Decentralizing decision-making so people don’t have to wait for permission and deliberation before acting
  • Delineating clear areas of responsibility so people feel individual ownership to move work forward

You can read the post for the other examples, they’re all very good! As with most of these kinds of topics it’s really valuable to think about them not in isolation, but as a system. It’s not about whether meetings are good or bad, it’s about how meetings fit into the culture and system of planning and delivery that the organization operates in.

Becky’s illustration of what “async” really means is a perfect example of this:

Meetings for an effective engineering organization

It seems like the topic of meetings is on everyone’s minds again as we start the year. Will Larson has some good perspective from the engineering org viewpoint:

Some engineers develop a strong point of view that meetings are a waste of their time. There’s good reason for that perspective, as many meetings are quite bad, but it’s also a bit myopic: meetings can also be an exceptionally valuable part of a well-run organization. If you’re getting feedback that any given meeting isn’t helpful, then iterate on it, and consider pausing it for the time being. It may not be useful for your organization yet, but don’t give up on the idea of meetings. Good meetings are the heartbeat for your organization.

He goes on to recommend six core meetings for every organization to start with. The “weekly team meeting” is one I’ve become a fan of as well. Getting the entire team on a call every week has the potential for being a giant time-waster, so getting the purpose right and facilitating it tightly is essential here. For us, the purpose is:

  • See each other’s faces at least once a week. I wasn’t sure if the team would feel like this goal is a waste of time, but it absolutely is not. Since we’re all remote, “let’s just chat for a bit” is such a great way to start the week.
  • Discuss blockers/issues. This is not a status meeting where everyone goes around the room and tells us where they’re at. We have an agenda in Google Docs that anyone can add to. The goal is to bring up any issues that the team is struggling with so that we can all figure out the best way to help.
  • Company updates. This is also the opportunity for the leadership team to make sure the entire team has all the information and context they need to do their work effectively.

There’s one more thing about this that I highly recommend: every meeting is facilitated by a different team member. We have a schedule that we cycle through with a clear guide on what it means to facilitate—and of course, an option to opt out. This keeps the meetings interesting and everyone invested.


Previously, on meetings: