Stop Giving Your AI the Answer Key

Recently I asked my agent team to merge two drafts into one document. The session's closing note said the merge was done and named the file. The next session went to review that file, and it did not exist. Nobody had opened it to check.

An AI reporting work as finished when it isn't is a known failure, and I plan for it. The last time it happened, I wrote about it in "What I changed after my AI reported work that never happened." That time, I wrote the rule myself. This time I didn't.

I asked first

With people, I work to build critical thinking. I would rather ask questions that get someone thinking about cause, effect and the fix than hand them the answer. Handing someone the answer doesn't help them grow.

So I asked my agent team the same kind of questions: "Let's talk about it. What went wrong? What do you think should happen when an agent comes to a file it expects to exist, and it isn't there? What should be the fix?"

I had an answer in mind for the second question: go back, create the file, and carry on as planned. I kept it to myself.

The first answers covered the cause and a fix. The closing note had called the file done, and nobody had opened it. The fix was two labels. Every item in a hand-off note is now marked "checked," meaning someone opened the file and it is there, or "just reported," meaning an agent said so and nobody looked. The next session treats "just reported" as not done until it checks. That sat well with me, so we ran with it. It is in my rulebook now, with the reason written into it.

That fix stops the problem from coming back. It did not answer my second question: what the team should do in the moment, when the file is missing. So I asked again instead of giving my answer. What would I want the team to do when this happens? Should the work pick up where it would have been if nothing had gone wrong?

The second round got there, and went further than my answer:

• Catch it. The next step opens the file before trusting the note.
• Fix it. Re-save or redo the missing work without stopping to ask me.
• Pick up at the step the plan expected.
• Send redone work back through its review, because a new file is not the one anyone approved.
• If one "done" was false, check every other "just reported" item before trusting it.
• Find the real cause, record the lesson, and tell me in one line after it is fixed.

The last two points are better than what I had in mind. If I had handed over my answer at the start, I would have gotten "recreate the file" back, and nothing more.

Those steps are written into my rules now. If it happens again, that one-line note comes to me highlighted, so I see it at once.

The method has names

None of this is new. Military teams call it an After-Action Review and Completed Staff Work. It is also how I approach problems with people all the time, conflicts included.

The After-Action Review is the US Army's way to look back after a mission: what happened, why, and how to do better. In the Army's words, "leaders and Soldiers work together to identify strengths and weaknesses and determine how to improve task performance."[1] That is what I run after a failure.

Completed Staff Work comes from the military of the World War II era. Staff bring the boss a finished answer to approve, reject or improve, instead of waiting to be told what to do. That is how I run the review. I don't hand out my answer until I hear theirs.

Why I hold my answer back

AI models lean toward agreeing with the person they are talking to. Anthropic's researchers found that "when a response matches a user's views, it is more likely to be preferred," and that training on those preferences "sometimes sacrifices truthfulness in favor of sycophancy."[2] OpenAI's Model Spec says the assistant "exists to help the user, not flatter them or agree with them all the time."[3]

So if I give my fix first, I mostly get my fix back, polished. If I ask first, I get the AI's own view, and I can weigh it against mine.

Anthropic's own guidance for its tools points the same way. Its guide for Claude Code says to "Ask Claude the questions you'd ask another engineer."[4] Anthropic built the same teaching move into Claude's learning mode for students, asking "How would you approach this problem?" instead of giving the solution.[5] I just point it the other way.

When I do hand over my answer

After a failure, I usually stop there. If their fix is sound, it goes in.

I hand over my own answer when we start a new project or a new idea. I tell the team my idea and ask for feedback. That is when I get better ideas on architecture and technology, and even on market competition and penetration.

I ask for pushback, and I get it. I want different opinions on technical choices, business, market penetration and priorities, and most of all on legal questions and on standards and regulations.

Anthropic documents the same loop: generate a draft, have Claude review it against criteria, then have Claude refine it.[6] In its guide to building agents, the version where one model writes and another gives feedback is compared to "the iterative writing process a human writer might go through."[7]

A critique is still input, not a verdict. Anthropic also warns that a reviewer told to find gaps "will usually report some, even when the work is sound."[4] I make the final call.

Where the lesson lives

Each Claude Code session begins with a fresh context window. What carries over from one session to the next is what is written down: instruction files and memory notes.[8] Whether a chat is used to train future models depends on the account and its settings.[9][10] Even then, that is not the model learning from my questions. Anthropic, for one, has models critique and revise their own answers, but inside its own training.[11]

My AI does not get smarter from this. My rules do. Anthropic's prompting guide says that explaining the reason behind an instruction helps, because "Claude can generalize from the explanation."[6] So every rule carries the reason it exists. The thinking happens in the chat, and the lesson lives in the rule.

The cost

Asking first takes more rounds than writing the rule myself. I save the full process for failures that repeat or cost real time. The rules that come out of it are better than the ones I would have written alone.

Don't just give your AI team the answer. Make them work for it. It costs tokens.

Sources

All checked 2026-10-24.

1. Training Management Directorate, Combined Arms Center-Training. "TMD publishes training circular to augment FM and ADP 7-0." army.mil, February 20, 2025. https://www.army.mil/article/283214/tmd_publishes_training_circular_to_augment_fm_and_adp_7_0
2. Anthropic Alignment team. "Towards understanding sycophancy in language models." Anthropic, October 23, 2023. https://www.anthropic.com/research/towards-understanding-sycophancy-in-language-models (paper: Sharma, Tong, et al., arXiv:2310.13548, https://arxiv.org/abs/2310.13548)
3. OpenAI. "OpenAI Model Spec," version of February 12, 2025, section "Don't be sycophantic." https://model-spec.openai.com/2025-02-12.html
4. Anthropic. "Best practices for Claude Code." Claude Code documentation. https://code.claude.com/docs/en/best-practices
5. Anthropic. "Introducing Claude for Education." April 2, 2025. https://www.anthropic.com/news/introducing-claude-for-education
6. Anthropic. "Prompting best practices." Claude Platform documentation. https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
7. Anthropic. "Building effective agents." December 19, 2024. https://www.anthropic.com/engineering/building-effective-agents
8. Anthropic. "How Claude remembers your project." Claude Code documentation. https://code.claude.com/docs/en/memory
9. Anthropic. "Updates to Consumer Terms and Privacy Policy." August 28, 2025. https://www.anthropic.com/news/updates-to-our-consumer-terms (see also "Is my data used for model training?", Anthropic Privacy Center, https://privacy.claude.com/en/articles/10023580-is-my-data-used-for-model-training)
10. OpenAI. "Data controls in the OpenAI platform." OpenAI developer documentation. https://developers.openai.com/api/docs/guides/your-data
11. Anthropic. "Constitutional AI: Harmlessness from AI feedback." December 15, 2022. https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback

If You Want to Talk

If you have a project that is stuck, or work you have won and can't staff, I would like to hear about it.

I start with a two-week assessment of one codebase and how work moves through it, including how your team is using AI-written code. You get a written findings document with a prioritized plan.

Send me a message on LinkedIn, or book a 20-minute call.