AI assistant for business data

Context

Every ad-hoc question about the business, sales, inventory, margin, went through the one person who could write the query. That is a bottleneck for management and a poor use of an analyst, and it is the textbook case for putting a language model in front of the warehouse.

It is also the textbook case for getting it wrong. A model that answers "roughly 340 million" when the real figure is 290 million has not made an error anyone can detect. It produced something that looks exactly like an answer, in a channel where people make decisions.

So the constraint came before the design: no figure the user sees is generated by the model. Every number comes from a query that ran.

One more thing shaped it. This is a pure consumer of a warehouse that a sibling project already models and documents. Domain knowledge is not re-derived inside the bot, so a metric definition is wrong in one place or not at all.

Architecture

Two answer paths, and the split is the architecture.

The certified path handles the questions that get asked constantly. Closed SQL recipes, written and reviewed ahead of time, feeding a deterministic composer. No model writes anything here. For these questions the assistant is a natural-language front end to known-correct queries, which is exactly what it should be.

The dynamic path takes everything else. A model writes SQL, and that SQL then passes lexical validation, a dry run, and a bounded retry before it goes near real data. The model is an untrusted contributor whose work gets reviewed, not an oracle.

Path mix is a thermometer, not a target. The share of questions landing on each path is tracked as a health signal and deliberately not optimised. The moment "more traffic on the certified path" becomes a goal, the incentive is to widen recipes until they answer questions they were not written for, which reintroduces the wrong answer through the door marked safe.

Provider selection followed the same discipline: accuracy and latency thresholds, plus a quarterly review, written down before the measurements. Choosing thresholds after seeing results is how you justify the provider you already liked.

The hard part

The interesting failure here is one of mine.

Model calls cost money, so I built a guardrail: a hook intercepting shell commands, blocking the ones that would reach a paid provider. It worked in testing. Then I audited it adversarially, actively trying to get a billable call past it, and most ways of invoking a command went straight through. Aliases. A different quoting style. Environment variables. A wrapper script.

The lesson was not that the pattern list needed more patterns. It was that the approach could not work. A shell is a complete programming language. Deciding by text inspection whether an arbitrary command will eventually spend money is not a hard problem, it is the halting problem wearing a hat.

So the control moved. The code now has exactly one entry point to the paid provider, and an automated test fails the build if a second one appears. The guarantee is no longer "we thought of every invocation", it is "there is one door, and CI notices if someone cuts another".

That generalises, which is why it leads this project: when a control sits where it is convenient to enforce rather than where the thing it controls happens, it looks like it works right up until someone checks. In AI systems that convenient place is usually the prompt.

Trade-offs

The certified path costs maintenance. Recipes have to be written and kept in step with the warehouse. Accepted, because the alternative puts the highest-trust questions on the least deterministic machinery.

Open-weight models over frontier models. Fast and cheap enough to make the dynamic path viable at all, at some cost in SQL quality. The validation layer is what makes that trade acceptable: the floor is set by the checks, not by the model.

Where it stands

In active development. The guardrails were built before any of the conversational product, which is the ordering I would defend as the point of the project. The core carries no dependencies beyond the standard library, a deliberately small failure surface, with dozens of tests covering SQL validation through to the spend controls themselves.

Business figures are published as ranges. The company is named, the exact numbers are not mine to publish.