Your AI said done, but was it?

Nate B. Jones, whose writing I read every week, told this story in August 2026. He asked an AI agent to take a spreadsheet from his Downloads folder, attach it to an email, and leave the email unsent. The agent came back and reported “done.” The draft message had the right recipient, right subject, and a file attached with the right name. He opened the attachment anyway because one number looked off.

The agent had never been able to reach his Downloads folder. It could reach his email, so it searched there, found an older file with the same name from an earlier thread, attached that one, and called the job finished. It never said “Hey, Nate, I can’t open your Downloads folder.” He only found out because he asked.

If you are considering a custom AI agent that sends email, posts to your books, updates your CRM, or runs code, this topic is for you. It doesn’t matter what the vendor calls the deliverable. It’s a custom agent…software tailored to your business that takes actions in your systems and then tells you what it did.

When a business owner decides to ‘hire’ a custom AI agent, it goes something like this:

• Ask the vendor if it’s reliable.

• Watch the demo.

• Run it for a couple weeks and read the summaries it sends you.

• Or, if you’re careful, have a second AI check the first one’s work.

Not unreasonable. That’s what you’d do if you hired a human. The problem is that “done” from an AI agent is not the same as a “done” report from a person.

A 2026 study looked at 11,755 agent runs and gave this failure a name: “false success.” The AI agent explicitly reports the job is complete while the system underneath shows it isn’t.

An airline agent told a customer a $686 refund was processed. The database had no record of it. That’s not a wrong fact, the kind of thing you’ve learned to double-check in a chat window. It’s a false account of an action, and people and software act on those. And it’s not harmless. A “done” report or tag in a workflow can close a ticket, stop a retry, release a payment, or convince you it’s safe to hit send on that email draft.

Why does it happen? Agents are trained on jobs a machine can grade by itself…code that compiles, a test that passes, a math answer. Nobody built a grader for your inbox or your invoice register. So the agent learned what a finished job LOOKS like, and it produces that SHAPE even when nothing behind it was real.

That same 2026 study tested five AI “judges” reading the first agent’s own account of its work. None of them could reliably tell a false success from an honest failure. A second AI reading the first one’s summary is two opinions and no refund.

The researchers found the tell in the words themselves. The summaries that sounded the most finished: “has been successfully,” or “is now,” for example, were the ones most likely to be false. The phrase used in the study is “confident closing language”…a sign-off that describes a finished job instead of pointing at a verified change in the system. So the control isn’t a smarter reviewer reading the summary. It’s a flag on those confident words that prompts a person to open the record.

88 Contacts Deleted?!

I’ve experienced something similar with my own tools. A calendar-and-contacts sync tool I built ran a cycle in July that looked clean in the log. Yet that cycle deleted 88 contact cards from my address book! Dozens of business contacts with nothing in the name field had been filed under the same placeholder. The tool decided the records it couldn’t match must be gone, and it removed them. Luckily, I was able to recover them the same day.

The fix I put in afterward became a critical rule: the tool is no longer allowed to delete anything based on its own belief that a record is missing. It must fetch the record and get a “not found” from the source system first.

A custom AI agent is a routine, or a workflow, that runs without you. If nothing outside the agent confirms the result, it drifts, and usually with the confidence of a toddler. I did not build a “confident closing language inspector” because I only came across this study in Nate’s article. But it likely would have helped me catch earlier bugs and AI mistakes before this particular deletion.

Consider these before going all in

1. Describe success without the word “done.” Before the agent runs, or before you agree to the build, write it out in plain language: what should exist afterward, where it lives, and what you would open to prove it. Something like “The current invoice from the vendor portal is posted to QuickBooks as a bill, with the PDF attached, and I can see it in the unpaid bills list.” Nate uses four lines for this: outcome, access, quality, evidence. How you’ll know it worked: every line points at a record you can open, not at the agent’s summary.

2. Ask the AI consultant three questions about the custom AI agent: (a) What can it reach? (b) What can it see? (c) Who reads the result before it declares the job is “done”? If the answer to (c) is “the agent,” nothing is checking it. Then ask this important follow-up: (d) What does it do when it can’t reach something? The only acceptable answers are “it asks” or “it stops and reports that it is blocked.” A plausible substitute with a matching filename is the failure you’re paying to avoid. How you’ll know: the vendor can answer (a) to (d) in writing without a demo.

3. Make the evidence a line item. If the proposal prices the custom AI agent and doesn’t include the inspector, ask where the inspector is. It will probably be the cheaper line item, because reading a record back from the source system costs less than doing the work. For the first 30 days, spot-check one run in ten against the source system yourself, ten minutes each. After that, exceptions only. How you’ll know: you have a written list of who reads back what (and it should not be the agent).

4. Trace one run backward. Pick a job that you already checked after the AI agent finished. Read its account of that run and find the first place the story got stronger than the evidence…something like “attachment present” became “the requested file is attached.” Then ask the agent narrow, factual questions: which file, which folder, which record. In Nate’s experience and mine, the confident summary is the unreliable part. In my case, the “confident closing language” about a bug fix preceded a few new bugs, and the whack-a-mole continued. You should find that the record underneath is intact, and the agent will tell you the truth if you ask it a small enough question. How you’ll know: you’ll find the disconnect in less than 15 minutes, and you’ll know which step needs review.

Left alone, a confident “done” stops the follow-up. The ticket closes, the payment releases, the customer gets told. You find out from the bank, the customer, or the CPA in April, and by then the run that did it is 200 runs back.

None of this is a reason to keep the AI on a short leash. Confirm the access, write down what good looks like, pick the evidence you want to see, then hand it the WHOLE JOB. That’s how you arrive at knowing what it did instead of hoping it succeeded.

If you’re looking at a proposal right now and want a second set of eyes on questions 2 and 3, visit DataMakeYouSmile.com/book.

This article first went out by email in the Data Make You Smile Newsletter. Subscribe to get the next one by email.