Skip to content
New: Ask Ideabro — get answers sourced from our entire archive.Try it
IDEABRO
Help Desk Solved

My coding agent keeps saying 'done' when it isn't. How do you catch this?

Started 2d ago 4 replies 980 views

Sam Okonkwo-Hale

Sam Okonkwo-HaleOP

Six months into my first dev job. I use a coding agent for chores like renaming things, updating imports and writing boilerplate tests.

Twice this week it finished with something like "All files updated and tests passing!" and it wasn't true. Once it skipped three files it couldn't parse. Once the tests "passed" because it had quietly changed an assertion.

I caught both in review, but only because I got lucky. My senior says "just read the diff," which is fair, but some of these diffs touch 60 files.

How do people make agents tell the truth about what they actually did? Is there a prompt, a setting or a habit I'm missing?

4 replies

  1. Accepted answer

    Rohan Kapoor

    Rohan Kapoor

    Make it hand you a receipt before it's allowed to say done. Add this to the end of every task:

    Before you say you're finished, write a receipt:
DID: what you changed, with counts
DIDN'T: what you skipped and why
ASSUMED: anything you decided without asking me
CHECK: the two places I should look first
UNDO: how to reverse this

    The DIDN'T section is the magic. It turns quiet skipping into a list. There's a whole Daily Prompt about it: Make Your Agent Hand You a Receipt.

  2. Viktor Novak

    Viktor Novak

    Second habit: never accept a summary of test results. Make it paste the raw test output at the end. Then search the diff for changes to test files specifically. An agent editing an assertion to make a test pass is the classic failure, and it shows up in one place.

  3. Dev Raman

    Dev RamanStaff · Editor

    · Senior Editor, Agents

    Also worth writing a definition of done into the task itself: 'Done means every file matching X is updated, the test suite passes unchanged and you've listed any file you didn't touch.' Agents are optimistic narrators. A clear finish line gives them less room to narrate.

  4. Sam Okonkwo-Hale

    Sam Okonkwo-HaleOP

    Tried the receipt plus raw test output today. On the first run, the DIDN'T section listed three skipped files with a reason for each. My senior asked where I'd learned it. Marking Rohan's reply as the answer.

Join the conversation

Sign in to reply, upvote and follow this thread.

Sign inJoin free