Menu

Posts tagged “agile”

Evals Are the New PRD

Braintrust makes a good case (apologies for the X.com link…) for rethinking how PMs work on AI products: the eval replaces the PRD.

An eval is a structured, repeatable test that answers one question. Does my AI system do the right thing? You define a set of inputs along with expected outputs, run them through your AI system, and score the results using algorithms or AI judges.

The eval becomes both the spec and the acceptance criteria. The directive to engineering:

“Here is the eval. Make this number go up.”

That’s very different to how most teams work today, but I can definitely see the industry moving this way. Product usage generates signals, observability captures them, and evals turn them into improvement targets. The PM’s job is to define what “good” looks like in code and curate the data that reveals what “bad” looks like.

The PM skills that transfer are the same ones that always mattered — discovering needs and opportunities, and making judgment calls about what to build for business value. The difference is that instead of a document that describes the intent, you have a test suite that encodes it.

The invisible 40% of engineering work

Anton Zaides wrote a good post about shadow work in engineering teams. He discovered a senior engineer on his team was spending over 40% of his time on work that didn’t show up anywhere—code reviews, mentoring, ad-hoc support fixes, etc.

This part is important:

The shadow backlog isn’t the problem—in my opinion, that’s probably the work that should have been done in the first place. The solution is to stop doing it under the table and make sure you have space for it. The more people don’t agree with your roadmap because it was decided for them, the more shadow backlog you’ll have.

The shadow backlog is a symptom of a roadmap that doesn’t reflect reality—and that often happens when engineering teams are not involved in planning and prioritization. That is the real fix—making sure everyone understands and is aligned on the roadmap, and making sure this kind of BAU (Business As Usual) work is visible and planned for.

On estimates as navigation, not promises

I’ve been thinking about engineering estimates a lot and need to write about it. But for now, Adam Keys sums it up nicely:

Everyone knows surprises will happen. The estimate should help the team make better decisions when they do, not box them into promises they can’t keep. The best estimates I’ve given weren’t the most accurate—they were the ones that helped teams navigate uncertainty instead of pretending it away.

On estimates as navigation, not promises

Regarding "missed" and "slipped" deadlines

I really like Basecamp’s concept of “Hill Charts”. The gist of it is that each project we work on has two distinct phases: a “figuring it out” phase and a “making it happen” phase:

To quote from their post about it:

First there’s an uphill phase where you figure out your approach. You have a basic idea about the task, but you haven’t figured out what the solution is going to look like or how to solve all the unknowns.

Eventually you reach a point where there aren’t any more unsolved problems. That’s like standing at the top of the hill. You can see clearly all the way down the other side. Then the downhill phase is just about execution.

“Figuring it out” is full of uncertainty, unknowns, and problem solving. “Making it happen” is all about certainty, confidence, and execution.

I bring this up because one of the issues with quarterly planning cycles and committing to due dates on that cadence is that teams are often asked to commit to dates during the “figuring it out” phase. There’s a lot of uncertainty and unknowns, so teams have to make best guesses. The problem with this is that these do not reflect what Cagan calls “high-integrity commitments”:

The key is to understand that the root cause of all this grief about commitments is when these commitments are made. They are made too early. They are made before we know if we can actually deliver on this obligation, and even more important, if what we deliver will actually solve the problem for the customer.

So what inevitably happens is that, for a lot of projects, the date that teams provided “slips”. I believe this language matters, and I think by framing it this way an org tells teams they did something wrong. But the irony is that in the majority of cases, if teams move a date after they reached the “top of the hill”, they are doing the right thing for the business. They are saying that they have now figured out all the unknowns and uncertainties, and they are ready to make a high-integrity commitment. Again, here is Cagan:

So the compromise is simple. The product team asks for a little time to do product discovery before commitments are made, and then after enough product discovery is done to consider the risks, we are willing to commit to dates and deliverables so our colleagues can effectively do their jobs as well.

I think it’s important to encourage behavior that does the right thing for customers and the business. The right thing would not be to try to cut corners to meet a date that was committed to during a “figuring things out” phase. I also don’t believe the right thing is to inherently communicate to teams that moving a date once they reach the top of the hill means they “missed” a commitment.

I think we should be very clear about the nature of commitments when we make them. What is our confidence in our dates? Are we still figuring things out? Or are we at the top of the hill and ready to make a high-integrity commitment? The health of any Product org can be improved if we say that it’s ok to communicate what we think we can accomplish early on in the quarter, and move to high-integrity commitments once teams reach the top of each project’s hill.

What does a date actually mean?

James Stanier has a good argument for why deadline-driven development is so… difficult:

Given that non-technical people don’t understand why software is hard, dates become the stick that they beat you with when you don’t deliver on time. Don’t ask me why, it’s just human behavior. I’m sure you’ve done it when roadworks have taken longer than were specified on the sign, or if a delivery of a package was late. Dates mean something to people, so handle them with care. In fact, perhaps we could do something entirely different instead.

What’s the “something different”?

So, instead, you should take a forecasting approach that follows the uncertainty curve that we outlined above. You start wide, and you taper in. At the beginning of a given project, you might even just have the year that you’re aiming to ship. Then, as you progress, you can start to narrow it down to a quarter, then a month, and finally a specific date.

This is why I will always advocate for time horizon roadmaps.

Quality vs Quantity

Mike Fisher preaches the good stuff in Quality vs Quantity. He talks about the importance of shipping small, incremental releases to customers:

By prioritizing the quantity of opportunities to learn, teams encourage a culture of continuous learning and flexibility. Each iteration becomes a learning opportunity. This method aligns more closely with real-world conditions where customer preferences and market dynamics are constantly evolving. Furthermore, this approach reduces the risks associated with big launches like Windows Vista, where a significant investment is made in a single, large-scale product release. Instead, smaller, more frequent releases allow for adjustments and refinements based on actual user feedback and engagement.

Why using a Now/Next/Later roadmap might be right for you

I was recently asked by a colleague to write up our team’s reasoning for using a Now/Next/Later roadmap to plan our work (instead of quarterly/annual roadmaps with dates). If you already use Now/Next/Later nothing in here will be new to you, but I thought I’d share what I wrote for this internal document in case it’s useful to anyone hoping to make this shift as well.

We use an adapted version of a Now/Next/Later roadmap to plan our work. You can read more about this approach in Introduction to Lean Roadmapping by its creator, Janna Bastow. In short, here are the guiding principles for using this roadmap and why it is effective:

  • Deadline-driven development is fraught with issues that make it a fairly ineffective way to plan delivery work. This includes:
    • Long-term priorities frequently change based on new data and developments, so any planning past a few months out is mostly fiction and rarely happens as planned.
    • Deadlines are often set without input from the delivery teams who are building the product, which makes estimates inaccurate and difficult to attain.
    • Because deadlines are often arbitrary, delivery teams have to make quality tradeoffs to meet the dates, which introduces unnecessary technical debt into the system.
  • Using a Now/Next/Later approach helps delivery teams know what is most important to work on, and what is coming down the road.
    • “Now” means Now–it is literally the work that is in flight. This work should be limited to 1–2 projects per team to ensure effective delivery.
      • Changes to “Now” should only happen in the rarest of occasions so as not to interrupt work in flight.
    • “Next” means anything from 2–8 weeks from now. This is work that is planned and ready to go as soon as a team becomes available. It has been spec’d and scoped, and everyone agrees it’s the next important thing. We limit not only Work In Progress (Now), but also Work in Next, so that there are not too many priorities vying for attention.
      • Changes to “Next” should happen infrequently since the work is planned and the team will be ready to go at any moment.
    • “Later” means anything from 2–6 months from now. This is work we believe is important to be prioritized, but it hasn’t been fully spec’d and scoped yet.
      • Changes to “Later” can–and often does–happen whenever new data becomes available that makes us shift priorities. This is expected and encouraged, until the project moves to “Next” where it gets locked in and fully spec’d.
    • We cheated and added “Much Later”, which lists things that we think will be 6–12 months out. The likelihood of these projects changing are high, but it is good to have a long-term view on what we believe, with the current information we have, will be important for the business and our customers to work one.

We do acknowledge and recognize that delivery dates are important. We prefer to work with high-integrity commitments, which are dates that delivery teams commit to once they have had a chance to properly scope out a project (which sometimes means getting started without a completion date set).

The teams are accountable to these dates because they have been involved in setting them, and though they can change based on unknowns, these changes should be infrequent.

How To Ship Fast

This is a highly opinionated piece (nothing wrong with that!) and I think there’s some nuance needed, but in general I like these principles from How To Ship Fast. This one is definitely a hot take, but in a dream world where everything is perfect I would agree:

Time spent on prioritization is the canary in the coal mine for not being close enough to your customers. The discrete task of prioritization should take almost no time. Only work on Important things. Sort those Important things into “Do now” and “Do next”. If you are only working on Important things, the order does not matter too much (and don’t spend time debating it). If what’s Important is not obvious you are not close enough to your customers.

Ask Teresa: My Leaders Still Want Roadmaps with Timelines—What Should I Do?

Good points here from Teresa Torres about deadline-driven development, especially the need to take change management slowly:

If your stakeholders are insisting you use date-based roadmaps, I wouldn’t engage in the ideological war about deadlines and predictable work. Instead, start with a feature-based roadmap. Give your stakeholders what they are asking for, and over time, you can introduce opportunities and outcomes.

How Process Impacts Your Culture

Josephine Conneely has some excellent thoughts on the feared P-word in How Process Impacts Your Culture. I especially like going back to the purpose of adding process when evaluating what you have in place:

The aim of process in its purest form is to:

  • Facilitate ease of doing work: Design methods for teams to effectively work together, make decisions, and achieve their goals.
  • Reduce risk: Ensure company doesn’t fall foul of legal & compliance obligations or go bankrupt.
  • Ensure consistency and fairness: Aim for all customers and employees to have a similar experience in their interactions with an organisation.