Green Tests, Wrong Feature

The tests pass. The agent says it’s done, here’s a summary of what changed. You merge it, and for a second it feels like you shipped something.

Maybe you did. Nothing in that loop actually told you so, though, and the loop is very good at feeling like it did.

There are two questions worth asking about any piece of work. Does this do what I asked for, and should I have asked for it. AI made the first one nearly free. The second one is exactly as hard as it has always been, and it’s the one that decides whether the week mattered.

Most of us are now getting a fast, confident answer to the first question and quietly accepting it as an answer to both.

The cost used to ask the second question for you

Think about what shipping a feature took four years ago. A spec, a designer, a couple of engineers, three weeks, a QA pass. Somewhere in there, usually early, somebody would ask whether this was really worth three weeks. Often it wasn’t. The thing died in a meeting, and that was the system working correctly.

That question didn’t come from discipline or from good product instincts. It came from the price tag. Building was expensive enough that the expense itself forced the conversation every single time, whether or not anyone in the room was any good at product thinking.

Now the same feature takes an afternoon. The price tag is gone, and the conversation that was attached to it went with it. Nothing replaced it, because nobody ever designed that conversation in the first place. It was a side effect of things being slow.

So you ship more and decide less, which feels like exactly the opposite of what’s happening.

Your agent is not a stakeholder

Here’s the part that I think is genuinely new. The feedback loop doesn’t just go quiet. It goes positive.

You describe a feature, the agent builds it, writes the tests, runs them, and comes back to tell you everything passes. That’s a pat on the back, delivered several times a day, by something that has no idea what your product is for or who is supposed to be using it.

It isn’t lying to you. The code does what you described. But “did it build what I asked” and “was what I asked worth building” are different questions, and only one of them has anybody answering it.

I’ve ridden that loop for a full day before. Tickets closed, everything green, genuinely enjoying myself, and then realized around 4pm that not one thing I built would move a number that mattered to a single person using the product.

You still have to open it and use it

There’s a layer underneath the product question that’s just as easy to skip, and it’s the one I see people skip most.

A green suite tells you the code did what the test told it to check. It tells you nothing about whether the thing is pleasant, obvious, quick, or worth using twice. Nobody in the history of software has loved a product because its CI was green.

So open it. Click through the flow start to finish like a person who has never seen it before, and pay attention to where you hesitate, because a hesitation is a real defect that no test will ever fail on. Notice if a step feels like work. Notice if you have to think about where to go next.

Bots will be bots. They confirm the button submits and the record saves. Humans want ease of use, and they want taste, and taste has never been something you can assert in a test file. That part of the job didn’t get automated either, and the cheaper building gets, the more it’s the thing that separates products people keep from products people try.

How I validate AI suggestions before I accept them

The technical layer still matters and I still do it. Run it for real, try to break it, read anything touching auth or money by hand. That’s table stakes and it takes about ten minutes now.

Then the real check, which is one question asked before the work starts instead of after. What number should move if this works.

Not a vague one. Activation, week-two retention, churn, conversion, revenue per account, support tickets about a specific broken thing. If I can’t name the number, I’ve learned that I’m not solving a problem, I’m decorating one, and the honest move is to not build it.

The follow-up is the question that actually stings. How long until I know. If the answer is “never, really,” then it doesn’t matter how clean the implementation was, because nothing about the outcome will ever teach me anything.

A good half of the features I’ve been excited about died at that second question, and every one of those deaths handed me back a day.

Shipping was never the point

For most of my career the bottleneck was building. Ideas were cheap and opinions were free, execution was the wall everybody hit, so it made sense to treat shipping itself as the achievement. It was the scarce thing.

That’s over. Building is close to free and getting cheaper, which means the scarce resources now are judgment about what deserves to exist and taste about how it should feel. Neither one comes with a test suite, a progress bar, or a cheerful summary at the end.

Green tests mean the code does what the tests say. Shipped means it exists. Neither has ever meant it was worth doing, and now that both are cheap, they mean less than they ever have.

Your agent can tell you the code is right. It can’t tell you the feature should exist, and it can’t tell you whether using it feels good. Those were always the hard parts of the job, and they’re the parts that didn’t get automated.