Press "Enter" to skip to content

Category: Generative AI

Testing AI Agent Skills

Published 2026-11-05 by Kevin Feasel

Eugene Meidinger and Ruben Van de Voorde try things out:

Writing a skill takes time, and skills are context that can help but also hurt the quality of the agent’s work. So how do you know if a skill actually helps? You can test it with evals. An eval is a prompt that you re-run under the same conditions, so that the only variable is what you deliberately change. You use the same model, starting context, and the same files (and wipe leftover memory from previous attempts). Then you compare: before and after an edit, with and without a skill, or two versions of a skill. Agents are non-deterministic, so you have to run the comparison multiple times to get a reliable difference; a few runs per condition is enough to start. Decide how a run is scored before looking at the results: use objective checks where you can (did it run the right command?), and for subjective calls write down beforehand what a good answer must include. Also run in a clean workspace: if the solution is already in another branch, the baseline run isn’t a real test. Some companies run large automated eval suites, but a few saved prompts and some notes are enough to start.

Click through for a testing technique and a warning around relying on a generative model to do the work for you.

Leave a Comment

Testing Fabric Data Agent Accuracy and Determinism

Published 2026-10-29 by Kevin Feasel

Marc Lelijveld trusts but verifies:

Over the past year, I’ve written quite a bit about Fabric Data Agents. From configuring them and preparing Semantic Models for AI, all the way to deploying Data Agents using the Data Agent SDK. But once you start moving Data Agents beyond experimentation, another question quickly becomes relevant: “How do you actually know whether your Data Agent continues to return the right answer?”

Of course, you can open the Data Agent, enter a prompt and manually validate the response. That works perfectly fine while experimenting with a handful of questions, although it can already become quite time-consuming. Now imagine having to validate 20, 50 or maybe hundreds of questions, and not just once, but every time your data, Semantic Model, Agent Instructions or other parts of the solution change.

There is another aspect to this. With generative AI, we don’t only care about accuracy, but also about determinism. If we ask the same question again, can we rely on the Data Agent to consistently produce the expected outcome? This is where automated testing becomes increasingly important.

Read on to see how you can work toward answers for both.

Leave a Comment

DAX Lib MCP Server

Published 2026-10-28 by Kevin Feasel

Marco Russo makes an announcement:

At the end of 2025, we launched DAX Lib, a public repository of DAX user-defined functions. So far, importing a DAX library has required a few manual steps, like clicking Add to Power BI on daxlib.org and then copying and pasting the DAX code into the TMDL View of your model, or using the dedicated tool in Tabular Editor 3. Both work well, but if you already use an AI assistant next to Power BI Desktop, we can now delegate this task to it. The DAX Lib MCP server version 0.1.0 is available now.

Click through if you’d like to snag a copy. It’s only available for Windows at this point, though considering Power BI’s OS requirements, that makes sense.

Leave a Comment

Choosing a Chunking Strategy for RAG Applications

Published 2026-10-22 by Kevin Feasel

Bala Priya C puts together a decision tree:

In this article, you will learn how to choose the right chunking strategy for a retrieval-augmented generation system by working through a structured decision tree tied to the properties of your documents and retrieval task.

Topics we will cover include:

  • What each major chunking strategy assumes about your documents, and when those assumptions hold.
  • A six-question decision tree that maps concrete document and retrieval conditions to specific chunking strategies.
  • The failure signals that indicate a chosen strategy is not working, and the fixes that tend to resolve them.

Click through for the list of strategies and when to include each.

Leave a Comment

Lessons Learned from Troubleshooting with LLMs

Published 2026-10-21 by Kevin Feasel

Grant Fritchey spends a weekend:

Grant Fritchey walks through a real troubleshooting saga – using Wine, a shim DLL, a misconfigured registry entry, and a lot of trial and error – to get a BridgeCom Maverick radio’s Windows CPS running on Linux Mint.

Along the way, repeated dead ends revealed a pattern: AI tools stay locked onto whatever direction you last pointed them, and it’s up to you to redirect, request alternatives, and demand measurement over guesswork.

In this article, Grant shares everything he learned from the experience, including 8 key takeaways to help you avoid similar trouble when using AI in the future.

Grant takes us through the story and shares eight tips for better troubleshooting along the way. Oh, and Wine is notoriously finicky, so I can totally get why it’d take a while to sort out the issues Grant had.

Leave a Comment

Connecting Azure Databricks to a Microsoft Foundry AI Gateway

Published 2026-10-20 by Kevin Feasel

Drew Furgiuele connects the dots:

While some people might look at the different capabilities and think this is an “either/or” scenario, the reality is that most Azure customers are going use (or even need to use) both: developers building apps and agents in Databricks, for example, may want to leverage a model or other agent deployed in Foundry. Meanwhile, developers in Foundry may need to call a Databricks managed MCP service.

These types of calls aren’t all that difficult; in fact, both platforms provide built-in connectors or UI solutions to make this happen: simply provide the service endpoint, pick your authentication method, and you’re pretty much done. However, the introduction of the respective gateways, especially on the Foundry side, can make this just a little more complicated. Let’s look at one example.

Click through for that issue and how Drew was able to work around it.

Leave a Comment

Adding Temporal Reasoning to RAG

Published 2026-10-02 by Kevin Feasel

Ivan Palomares Carrascosa checks the date:

Topics we will cover include:

  • How to extend standard subject-predicate-object triples into time-stamped quadruples stored in a simple temporal graph.
  • How to calculate recency weights with exponential decay and use them to rank conflicting facts as of a given query date.
  • How to tune the half-life parameter and integrate the temporal graph into a deterministic 3-tiered Graph-RAG retrieval pipeline.

This is especially important if you’re searching over the news or other systems that value more recent information over older information.

Comments closed

Using the Azure Resiliency Agent

Published 2026-09-24 by Kevin Feasel

Reitse Eskens gives a copilot a spin:

And that isn’t limited to services like storage, web portals, user management, and data stuff. It’s also location, backups, and disaster recovery. But there’s a catch: you need to make sure this is configured. I always compare the cloud (Azure in my daily work) with a huge box of Lego. You have all the pieces and elements to make something cool, but you have to build it. Even when you automate it, you still need to think about what you want.

Now, before I continue my Azure Fundamentals training, let’s move on to what this post is about: resiliency. Or, how your environment is set up for disaster recovery.

Let’s use the Azure Copilot to guide the process, create the necessary resource and review the advice.

Click through to see how it works. Because it’s in preview right now, we don’t know how much it will cost later. But in the meantime, you can give it a try.

Comments closed

Building Daily Averages in DAX

Published 2026-09-24 by Kevin Feasel

Marco Russo and Alberto Ferrari take generative AI tooling for a spin:

Computing an average daily sales amount is straightforward: you should iterate over the dates, compute sales for each day, and average the results. An AI assistant can generate DAX code that works in just a few seconds; whether it computes the number you need is a different question.

In this article, we ask AI to generate every measure, function, and validation query. We start with a simple average and test it against more demanding scenarios. Our job is to explain the business requirements, challenge the proposed code, and verify the results. Each test reveals a missing detail from the original request.

The prompts below show a conversation with an AI agent. They describe how to get the DAX code from the AI. They are not a verbatim transcript: a different agent might produce different expressions.

This shows the duality of generative AI outcomes: you can end up with a good solution, but unless you know the ins and outs of what you’re doing, you can end up in a sub-optimal situation.

Comments closed

Managing AI Context

Published 2026-09-09 by Kevin Feasel

Eugene Meidinger has a pair of articles. The first one covers the use of Markdown for building AI context:

When performing agentic development, most of your time will be spent either writing prompts or context files in Markdown (.md) files. This is a simple format used for documentation and notetaking, but which is also easy to read or write for humans and agents. You have almost certainly encountered Markdown before, but you may not have known it. Markdown is used by AI agents, in code documentation, and in some chat programs. Even Microsoft Teams supports a subset of Markdown.

Markdown is extremely common in more than just AI development work, so it’s definitely worth knowing.

The other is a guide to understanding and managing AI agent context:

The most important task for you during agentic development is to create, gather, and curate context. Context is anytext (or images) that the model ingests to produce a more relevant and useful response. Providing the right context at the right time helps agents produce better results; it’s about more than just writing markdown files:

Comments closed