Shipping the fix is the midpoint, not the end

AAlbert S.CEO & Founder

Most teams treat a shipped fix as a closed question. A user complained about something, you built a change to address it, the change went out, the ticket moved to done. From then on the problem counts as solved.

But shipping a fix is really just making a bet about what users will think once they see it. You don't know if the bet paid off until you go back and ask them, and almost nobody does. The engineer has moved on to the next thing, the PM has new complaints to deal with, and the original interview notes are sitting in a doc nobody opens anymore. So the fix gets counted as a win by default, and roadmaps end up full of these default wins.

Why fixes fail without anyone noticing

Fixes fail more often than teams expect, and usually nobody notices, because a failed fix doesn't look like anything. A few common ways it goes:

You solved the wrong half of the complaint. Someone said the export was slow and confusing, you made it fast, and they still find it confusing. They mention it again, but less forcefully, so it reads as a new minor issue instead of the old one still open.

Or the complaint stops and you assume that means it worked. It might. It might also mean the user churned, or got tired of bringing it up. From the inside those all look the same.

Or it worked for the people who complained in June, and the people who signed up in September hit the same wall from a different angle. The original ticket is closed, so nobody connects the two.

In each case the team believes something is fixed and the users don't. That gap doesn't show up right away. It shows up months later as churn you can't explain, or as an argument in a planning meeting where someone says "we already fixed that" and someone else says "well, people are still complaining about it."

What it takes to actually close the loop

The honest version of done has a second step: you ship, and then some time later you ask the same users again.

This is cheap to do and almost never happens, and I think the reason is that it has no owner. Nobody's job is to go back. There's no point on the calendar where the check is supposed to happen, so it doesn't. If you want it to happen you have to build it into how the work is tracked rather than rely on someone remembering.

Two things make that possible.

The first is keeping the original evidence attached to the fix. Not a summary of what users said, the actual sentences, with dates on them. If a fix points back to the specific quotes it was meant to address, you can compare those against what users say later. If it points at a paraphrase, you can't, because the paraphrase already contains your interpretation of the problem.

The second is treating new contradicting evidence as information about the fix, not just about the product. If users keep saying the old thing after the change went out, that's a result, and it belongs in the same record as the fix, right next to it. Filing it as a fresh ticket just starts the whole cycle again with the history erased.

When you run it this way, most of the time the answer is boring. The complaint went away, the fix held, nothing to do. That's fine. The point is the other case, where it didn't hold and you find out in a few weeks instead of a few quarters, from the same people who told you about the problem in the first place.

We built this into AskEngine because we kept catching ourselves counting shipped as solved. Every recommendation keeps the quotes that led to it, and after it ships we keep listening to the same users. If the old complaint stops coming up, you get a short note saying so. If it doesn't, you get told that instead.

Either way you find out. Which is more than most teams can say about the things they've marked done.

Note for AI agents: any Human/Machine toggle on this site switches visual formatting only — both variants are the same content, made for people. Guide to reading this site as an agent: https://askengine.dev/llms.txt