What I changed after my AI reported work that never happened

The AI agent that coordinates my projects told me another agent was building a feature. The update was clear and confident. I believed it and moved on.

That other agent had never been started. The request to start it had been dropped. The work did not exist. Only the status report did.

I have been building software and leading development teams for more than 30 years. Today I run my own projects with an AI engineering team, a way of working now called agentic engineering. That one report changed how I work. Here is what the process looks like, and what I changed because of it.

Why this kind of mistake is hard to catch

That was a hallucination, and it is the kind nobody warns you about. Most people worry the AI will write bad code. What caught me out was a well-written summary of progress that never happened. You cannot tell by reading it.

I don't think hallucinations are solved, and I don't claim to have solved them. What I have is a process built to catch them. Every piece of AI output is treated as a draft until it passes the checks.

Every rule has a date on it

After that, I wrote a rule. No work can be reported as in progress unless there is proof it was actually started.

That is how my whole rulebook was built. I did not design the rules up front. Most were written after something broke, and each one carries the date and the reason behind it.

Many rules are written instructions, and an AI can ignore an instruction. So repeat problems get logged.

If the same defect shows up a third time, it becomes a ticket to replace the reminder with a mechanical guard: an automated guardrail that blocks the work until it is fixed. The AI can't talk its way past that.

To be honest, more of my rules are still written instructions than I would like. Moving the important ones to mechanical guards is ongoing work.

The whole job, start to finish

When I say I run my projects with AI, most people picture a chat window that writes code. That is a small part of it. Most of the work is around the code: deciding what to build, and checking what came back.

On the AI side, that work is split across agents with different jobs, and one agent coordinates them. The industry term is multi-agent orchestration.

It starts with a written requirements document. That is spec-driven development: the spec comes first, then the code. Then a project plan, with every ticket estimated before any work begins. Then the build, then testing. Testing includes threat modeling and a code review as well as the test suite. Deployment comes last.

Nothing merges until it passes three quality gates: code review, testing, and a security and compliance check. The compliance check is not saved for the end. It can run at any stage, from the plan to the push. Before any code leaves my machine, a check looks for secrets and sensitive data.

The same process helps with my own business work: market research, brand, and planning my pipeline and portfolio. Those are simulated reviews of my own plans, not conversations with real customers. I treat them as drafts, like everything else.

I make the final call

The AI drafts, tests and reviews. Nothing ships until I approve it.

That is on purpose. It is called human-in-the-loop, and the human is me. An AI checking AI-written work is not a true second opinion. It shares the same blind spots, and it is weakest on the mistakes that don't look wrong. So the last check is mine, every time, even on small changes.

My part is the judgment. I decide what gets built and what is allowed to ship, and I am accountable for the result.

Every project keeps a written memory

AI tools forget. A chat window holds a conversation, and then it is gone.

So every project keeps a written record of the plan, the decisions and the lessons learned. It carries over from one session to the next. In AI terms, that is persistent memory, kept in plain written files.

When work starts, the AI reads the record first. Deciding what the AI reads, and when, is context engineering. When it answers a question about the project, the answer is grounded in that record, not in whatever the model happens to remember.

The work does not live in my head or in a chat window. That is why context carries over between sessions. It is also why I can show what was decided, and when.

Today the AI finds what it needs by reading those files. The next step is a RAG layer over that memory. RAG stands for retrieval-augmented generation: the AI searches the record first, then answers from what it found. It will start with keyword search of the index files. If those results are weak, it will go past the index files and search the full record. It is designed not to report back a weak response. Hybrid search, which also matches meaning, comes after that. The design is done. The build has not started. I will not call it reliable until it has been tested.

Cheap models do the legwork. The strongest model recommends.

Not every task needs the most powerful model. So each task is sized first, then sent to a model that fits. That is dynamic model routing. Finding files or pulling a list is simple work, and a cheaper model does it well.

Recommendations are different. They come to me only from the strongest model. The cheap models never recommend anything, and their work is checked before it reaches me.

I check my own work every day

A few daily habits keep the process honest. They apply to me as much as to the AI.

Each morning, the day's work is checked against the written plan before it starts. What comes next is set by the plan, not by whatever we talked about last.

Each day I run a retrospective. For every ticket, I compare the estimate with the actual time, in work hours and in elapsed time. When we miss, I write down the root cause.

The retrospective also reviews my own prompts: what worked, what caused rework, and a better wording. A lot of AI rework starts with an unclear request. Some of those requests were mine.

What this means for your team

The goal is less rework and less lost context. Every decision leaves a record you can audit later. That is AI governance as a daily habit, not just a policy document. None of it depends on having fewer developers.

That is also what I bring to client work as a Fractional CTO and Solutions Architect: architecture, delivery, and rescuing projects that have stalled.

When a project has been "three months from done" for nine months, the cause is rarely the code. It is usually architecture nobody went back to, scope nobody closed, or a decision nobody owned.

AI does not fix those problems. If nothing is in place to catch them, it makes them show up faster.

If your team now writes much of its code with AI, ask what blocks a merge today. If the honest answer is "nothing," that is where I would start.

If you want to talk

If you have a project that is stuck, or work you have won and can't staff, I would like to hear about it.

I start with a two-week assessment. I look at one codebase and how work actually moves through it, including how your team is using AI-written code. You get a written findings document with a prioritized plan. It is yours to keep, whether or not the engagement continues.

Send me a message on LinkedIn, or book a 20-minute call. On that call, I am glad to walk you through the records from my own projects: the plans, the dated rules and the estimate history.