An AI agent is only as reliable as the data and definitions beneath it
What we saw about AI agents in a hands-on Snowflake lab
Notitia's Adelaide team recently took part in a hands-on Snowflake lab on building AI agents that deliver trustworthy answers.
The exercise started with a common business question: which products are customers returning most, and why?
An AI agent working directly over the raw data answered quickly and confidently. On closer inspection, the answer was wrong. The issue sat in the data underneath it: the source data was counting returns the wrong way.
The fix was in the foundations. Using Snowflake's AI tools, we:
- Traced the issue to its source.
- Set a clear definition of what counts as a return.
- Translated and categorised multilingual customer feedback so it could be measured alongside the numbers.
With those foundations in place, the same agent gave an accurate answer. It was backed by both the data and customer evidence, and it produced a report ready to share.
Why AI agents give plausible but wrong answers
When an AI agent gets a business question wrong, the cause often sits in the data and definitions beneath it rather than in the AI.
Four common causes:
- Undefined business terms. "Return", "active customer" and "revenue" can mean different things to different teams and systems. If the term isn't defined, the agent picks an interpretation, and it may not be yours.
- Data recorded for a different purpose. Operational systems capture events the way the process needs them, not the way analysis needs them. Counting those records at face value can double count, miss events or mix categories.
- Unchecked generated code. Agents often write the query that produces the answer. If nobody reviews that logic, small errors in joins, filters or grouping pass straight through to the result.
- No benchmark to test against. Without a known result to compare with, there's no easy way to tell a correct answer from a convincing one.
A checklist before you trust an AI agent's answer
Before an agent's answers reach decision-makers, work through these five checks.
- Define the business terms the agent will use. Write down what counts as a return, a customer or a sale, and make those definitions available to the agent.
- Check how the source data records events. Confirm whether the raw tables can be counted as they are, or need cleaning first.
- Review the code the agent generates. Have someone who knows the data read the query logic, especially joins, filters and groupings.
- Validate against known results. Test the agent on questions where you already know the answer, such as last quarter's reported figures.
- Ask for the evidence. A trustworthy answer shows the data behind it and, where relevant, what customers actually said.
The Australian Government's data quality checklist for AI points the same way, including checking whether processing the same event twice would skew your results.
Don't leave "real people" out of the picture
Numbers tell you what is happening. Feedback from real human interaction often tells you why.
In the Snowflake lab, the "why" behind product returns lived in free-text feedback written in several languages. On its own, that text was hard to measure. Once it was translated and grouped into categories, it became evidence the agent could count and cite.
This is where AI adds real value before an agent is ever asked a question. Using AI to structure unstructured data, such as reviews, support tickets, survey comments and emails, turns it into something your analytics and your agents can rely on.
Data Quality: Get the foundations right first
The lesson from the lab is simple: an AI agent is only as reliable as the data and definitions beneath it, whatever platform it runs on. Clear business terms, careful review of AI-generated code and validation against known results are what turn a plausible answer into one your organisation can act on.
That's the work we do with clients every day. If you're planning to put AI agents in front of your teams and want confidence in the answers they give, let's have a conversation.
