Testing is a required phase for any system development and deployment process, and this is not different for an AI analytics agent.
An analytics agent does not simply take an input and return an output. It reads the question, determines what is actually being asked, decides which tables and columns it needs, writes SQL, runs it, interprets the results, and may revise its approach if the first result does not look right.
A semantic layer, context definitions or any other knowledge layer you give the agent will narrow what it can query and how. That reduces the surface area for error, but it does not eliminate it. The agent still interprets, selects and sequences, and each of those steps needs to be tested.
Using the agent yourself will expose some of these problems. You ask it questions, notice when an answer looks off, and gradually learn where it struggles. But this approach doesn't scale, since every change means checking all the cases again by hand, and in practice you can't tell whether a newer model performs better, whether a prompt change improved how the agent interprets requests, or whether adding a new tool affected something that already worked.
An evaluation gives you a repeatable way to make those comparisons. The same cases can be rerun after a change, scored using the same criteria, and compared with the previous version, reducing the need to manually recheck the same questions after every update.
An eval is a structured, repeatable test that assesses how well your agent performs across a set of scenarios using defined criteria.
You do not need a dedicated evaluation platform to begin. A list of questions, a way to run them, and a script that checks the responses is already a working eval. More tooling becomes useful as the number of cases grows, the agent begins using more tools, and evaluations need to run often enough that managing them becomes difficult.
Evals can differ both in where they run and in what they are intended to test. Offline evals run the agent against a controlled set of cases, which makes them useful for comparing models, prompts, tools, and other changes before release. Online evals use production interactions and show how the agent performs on the questions users are actually asking, including cases that were not anticipated when the offline dataset was created.
They can also serve different purposes. A capability eval checks whether the agent can handle a class of task, such as answering retention questions, comparing periods, or resolving an ambiguous metric definition. A regression eval checks whether a recent change has broken something the agent could already do. These distinctions overlap because capability and regression tests will often run offline, while online evaluation helps uncover new cases that can later be added to the offline suite.
At its simplest, an eval has three parts: the task, the data, and the scorer.
The task efines what the agent is expected to do and what a good response should contain. For an analytics agent, the main task may be to answer a question correctly against a given dataset, but the final number or table is only one part of what can be evaluated. You may also need to know whether the agent correctly interpreted the request, selected the appropriate data, followed the relevant business definitions, used its tools appropriately, and handled uncertainty sensibly.
Which of those matters will depend on the request. A question with one well-defined answer may mainly need a result check, while a question with an ambiguous metric definition may need the evaluation to consider how the agent handled that ambiguity before writing SQL.
The data is the set of cases the agent is evaluated against. A case will usually contain a question and whatever is needed to judge the response, which may include an expected result, a reference query, a business definition, a grading rubric, an expected tool call, or a set of conditions the answer should satisfy.
Building this set usually requires more work than assembling a list of questions because each case needs enough context for someone or something to determine whether the answer was correct. It also needs to represent the work the agent is expected to handle, rather than only the requests that are easiest to score. The first dataset can remain small, but it should include common questions that reflect reality, important metrics, and cases where using the wrong table, grain, date range, or business definition would still produce a believable answer.
The scorer determines how each response will be evaluated. Different parts of the task may require different scoring methods because some checks can be defined precisely, while others depend on judgment.
Code-based scorers work well for checks that have a clear expected outcome. They can compare the returned result with an expected result, allow for numerical tolerances, check that required columns are present, confirm that the correct date range was used, or verify that certain tables were included or avoided.
The comparison should not usually require the agent to reproduce one exact SQL statement. Two queries may be structured differently and still return the same correct result, so a reference query is more useful for establishing the expected answer or identifying required logic than for enforcing one implementation. Code-based comparisons also need to account for differences that do not affect correctness, such as row order, column order, labels, data types, or rounding.
An LLM judge can be used for parts of the response that are difficult to reduce to a deterministic rule. It can assess whether the agent understood the question, followed a business definition, explained its assumptions, or communicated the result clearly. For that to work, the judge needs the relevant context and a rubric that explains what a good answer should contain.
The judge can still be wrong, especially when it is grading an answer that sounds convincing but rests on incorrect data or logic. Its ratings, therefore, need to be compared with human review before they are treated as reliable, and that comparison should be repeated occasionally as the cases and scoring criteria change.
Human review is slower, so it is usually reserved for important or difficult cases and for checking whether the LLM judge is producing sensible ratings. A reviewer with the right domain knowledge can identify problems that are hard to express as a fixed rule, particularly when the answer depends on how the organisation defines a metric rather than on whether the SQL ran successfully.
User feedback is also useful, but it serves a different purpose. A user can tell you whether an answer was clear, useful, or consistent with what they expected, but that does not establish that the result was correct. A user may accept a wrong number because it looks familiar or reject a correct one because it conflicts with an assumption. Feedback is therefore most useful for identifying interactions that need investigation and cases that may be missing from the evaluation dataset.
A basic evaluation workflow for an analytics agent can be kept to four steps:
Start with questions that represent the work the agent is expected to do and include enough information to evaluate each response. Depending on the case, that may mean providing an expected result, a reference query, a business definition, or a set of conditions the response should meet.
The cases also need to run against a controlled data state because warehouse data changes over time. If the same question produces a different result a week later, the difference may come from new records in the warehouse rather than from a change in the agent. A fixed warehouse snapshot, a controlled test database, or recorded query results can be used, depending on how the system is built and what part of it needs to be tested.
The important point is that the versions being compared should have access to the same underlying data. Without that, a change in score may reflect a change in the warehouse rather than a change in the agent.
Run the dataset against the version of the agent you want to evaluate, while keeping everything else as consistent as possible. When comparing models, keep the prompt, tools, and data fixed. When testing a prompt change, keep the model, tool configuration, and data fixed.
The run should capture the trace and the final response. For an analytics agent, this may include how the request was interpreted, what context was retrieved, which tables were selected, the tool calls made, the SQL generated, the query results, and any errors or retries.
This trace is what allows you to distinguish between different kinds of failure. A wrong answer may result from misunderstanding the request, applying the wrong business definition, selecting the wrong table, writing incorrect SQL, or misreading a correct query result, and each failure requires a different fix.
Apply the scoring method that fits each case. Some cases may only need a comparison between the returned result and an expected result, while others may also check the tables used, the filters applied, the date range, important SQL properties, and how the agent handled assumptions and ambiguity.
To use those scores in a release decision, you need a comparison point. That may be the current production version, a minimum acceptable score, or a rule that important cases must not regress. The rule does not need to be complicated, but it should be decided before the results are used to determine whether a change is ready.
Review the failed cases using the trace to identify where the problem occurred. The agent may have misunderstood the request, selected the wrong data, applied the wrong definition, generated incorrect SQL, or explained the result without making an important assumption clear.
Once the cause is understood, update the agent and rerun the cases. The evaluation case itself should be changed only when its expected result, context, or scoring rule is incomplete or incorrect, since changing a valid case to fit the agent's response would hide the failure rather than resolve it.
Failures found in production, human review, or user feedback can then be turned into new offline cases. When a problem has been fixed, keeping a representative version of it in the suite makes it possible to detect whether a later change reintroduces the same issue.
The first evaluation suite does not need to cover every kind of analytics request.
It needs a representative set of cases, a controlled way to run them, and enough information to understand why a response passed or failed.
As the agent is used, production traces and reviewed failures can be added to the dataset, including business definitions that were missed, questions that exposed the wrong grain or date range, and cases where the final answer looked reasonable despite being based on the wrong data. Over time, this gives you a more useful basis for comparing changes than checking a few familiar questions by hand before each release.
If your team is exploring AI analytics and wants to understand whether your data foundation is ready, our AI Readiness Assessment is a good place to start. Reach us at hello@datacult.com
