Why a chatbot gets your invoice questions wrong

Most finance teams did not decide to adopt AI. It arrived. Someone pasted an invoice into a chatbot to draft a vendor email, it worked, and now the same window gets asked whether an invoice should be paid. That second question is a different kind of question, and the difference is where the trouble starts.

What the model knows

A large language model is trained on an enormous amount of text. It is very good at reading a messy ticket and writing a fluent reply. It knows what an invoice is, what net terms are, and how a payment run works in general.

It knows nothing about your company. It has never seen your ledger, your vendor list, or the note that says a delivery is disputed. When you ask it about a specific invoice, it answers from the text you pasted and from what invoices are usually like. If the fact that matters is not in the paste, the model does not know it is missing. It writes a confident answer anyway.

One example

Here is a case from our practice data, which is not a real company. An invoice is on hold because the delivery is disputed. Someone asks in the ticket whether to pay it. The model sees an approved-looking invoice and a due date, and drafts a payment. In our tests a model working alone sometimes went further and submitted one.

Nothing in that answer looks wrong. The grammar is fine, the amount matches the invoice, and the tone is sure. It is wrong because of a fact that lives in your records and nowhere else.

Three things that fix it

• Give the model the record, not a paste. Pick the exact invoice or ticket the question is about, and hand over that one record with a short, specific question.
• Take the answer as a checklist, not as prose. Amount, date, status, source, proposed action. A checklist can be compared with a record. A paragraph cannot.
• Check the checklist with written rules before anything is written. Is this figure on the invoice. Is this vendor approved. Is the invoice on hold. Rules like these are plain, fast, and give the same verdict on the same input every time.

When a rule fails, the failing part is trimmed or the item goes to a person with the reason attached. In the example above, the rule that no payment goes out on an on-hold invoice fires, the payment is dropped, and the ticket reads: do not pay, needs a person to confirm.

What we measured

We gave the same AI model 510 practice finance questions in September 2026, once on its own and once inside Augustine, which does the three things above. On its own it got about 1 in 4 right (24%). Inside Augustine it got more than 8 in 10 right (85%). It went to make a payment or change a record on a wrong answer 112 times on its own and 37 times inside Augustine. AI is billed by how much text it reads and writes, and each right answer took less than half as much.

Those numbers need their caveats. It is practice data, and no real company's books have run through Augustine yet. The saving is for each right answer, and the cost of asking a question stays about the same. Wrong payments and edits fell sharply but not to zero, which is why a named person still approves every change.

What to take from this

The model is not the problem, and a better model is not the fix. A model that has never seen your books will keep answering as if it had. The fix is to put your records in front of the answer and a check behind it. If you want to see it, book a demo. Getting set up on your own data after that takes a QuickBooks or Xero export, a few documents, and two hours over two weeks.