Marc Lelijveld trusts but verifies:
Over the past year, I’ve written quite a bit about Fabric Data Agents. From configuring them and preparing Semantic Models for AI, all the way to deploying Data Agents using the Data Agent SDK. But once you start moving Data Agents beyond experimentation, another question quickly becomes relevant: “How do you actually know whether your Data Agent continues to return the right answer?”
Of course, you can open the Data Agent, enter a prompt and manually validate the response. That works perfectly fine while experimenting with a handful of questions, although it can already become quite time-consuming. Now imagine having to validate 20, 50 or maybe hundreds of questions, and not just once, but every time your data, Semantic Model, Agent Instructions or other parts of the solution change.
There is another aspect to this. With generative AI, we don’t only care about accuracy, but also about determinism. If we ask the same question again, can we rely on the Data Agent to consistently produce the expected outcome? This is where automated testing becomes increasingly important.
Read on to see how you can work toward answers for both.