Category: Testing

Marc Lelijveld trusts but verifies:

Over the past year, I’ve written quite a bit about Fabric Data Agents. From configuring them and preparing Semantic Models for AI, all the way to deploying Data Agents using the Data Agent SDK. But once you start moving Data Agents beyond experimentation, another question quickly becomes relevant: “How do you actually know whether your Data Agent continues to return the right answer?”

Of course, you can open the Data Agent, enter a prompt and manually validate the response. That works perfectly fine while experimenting with a handful of questions, although it can already become quite time-consuming. Now imagine having to validate 20, 50 or maybe hundreds of questions, and not just once, but every time your data, Semantic Model, Agent Instructions or other parts of the solution change.

There is another aspect to this. With generative AI, we don’t only care about accuracy, but also about determinism. If we ask the same question again, can we rely on the Data Agent to consistently produce the expected outcome? This is where automated testing becomes increasingly important.

Read on to see how you can work toward answers for both.

Jens Vestergaard says don’t trust, do verify:

In two previous posts I went down the path of getting a semantic model ready for AI: descriptions on every measure, an instructions file, the schema tidied up enough that a Fabric data agent has something real to read. That work has a satisfying endpoint. The model looks ready.

Ready is not the same as right.

The kind of evaluation Jens is talking about is fundamental to good business intelligence practices, regardless of whether you throw language models into the mix. Where language models do add complexity is the arbitrary scope of questions, how ambiguous people tend to be when writing, and the stochastic nature of answers. All of that makes the problem harder, though at least it isn’t an entirely different class of problem to solve.