Initial Impressions of Data Agents – Ep. 459
Data Agents are quickly becoming a central part of the Fabric story. In this episode, Mike and Tommy share early impressions—where the experience is genuinely useful today, where it’s still hype, and how the new multi-agent orchestration story (Copilot Studio + Data Agents) changes the conversation.
News & Announcements
- Chicago Fabric / Power BI User Group — Crash Course in Fabric (Oct 2)
Main Discussion: First Impressions of Data Agents
Smarter Agents with Data Source Instructions
Microsoft added the ability to provide richer instructions to Data Agents so they return more accurate answers:
Mike and Tommy discuss why this matters: most “bad AI answers” are really “bad context.” Good instructions, examples, and naming conventions dramatically improve results.
Multi-Agent Orchestration with Copilot Studio
They also discuss Microsoft’s multi-agent vision:
The shift is from “one agent answers questions” to “multiple specialized agents coordinate”—which is closer to how real organizations operate.
The Hype Cycle Reality Check
Mike and Tommy reference the Gartner hype cycle as a useful mental model:
Data Agents are exciting, but expectations need calibration:
- Agents work best with strong semantic models and clean metadata
- Early versions require iteration and testing
- The most valuable use cases are narrow, repeatable workflows
Looking Forward
Data Agents will mature fast—especially as tooling improves for instructions, governance, and lifecycle management. The teams that invest in semantic model quality and agent configuration practices now will be positioned to take advantage as orchestration becomes mainstream.
Episode Transcript
Full verbatim transcript — click any timestamp to jump to that moment:
0:03 Good morning and welcome back to the
0:36 Explicit Mentes podcast with Tommy and Mike. Hello everyone and welcome back. Good morning Mike. It’s a beautiful day. The weather in the Midwest is gorgeous now, but that probably means a polar vortex coming in winter. Oh yes, that probably will that’ll probably cause problems later on. We’ve actually had some decent weather. Our fall, our intro to fall has been actually pretty mild for once. , usually we have pretty scorching heat through the month of August and it was actually cool for once and a good amount of rain for us also. Yeah.
1:08 Anyways, let’s jump into some news items here. Oh, before we get into that, let’s just do our quick main topic. Our main topic top topic is our our initial impressions with data agents. So data agents is a item that you can create in the fabric workspaces and it helps it allows you to create a custom experience around talking connecting to and chatting about your data and having it return results for you. So that’s data agents and we’re going to just walk through our initial impressions of what’s going on there and what Microsoft is building and how you might want to use it. All right,
1:41 That being said, now we can do the news. Tommy, give us some news. Well, we got one main item. Guess what? The Chicago Fabric PowerBI user group is coming back in town downtown on October 2nd. I Mike, I am so glad we’re going back on in person. There’s nothing like it. And what we’re talking about is it’s a crash Chicago’s crash course on fabric. I think for a lot of people who listen to us or if you we’re talking a lot about fabric but we also hear from you guys mailbags and all the
2:14 Other social medias that well it would be great to play with it but our organization can’t. I’ve seen videos but I want to ask questions and interact with it. And that’s exactly what we’re going to do. We’re actually going to do a lot of live demos. We’re going to let people drive and actually see what it’s all about and again really take advantage of being in person. And that’s a big thing for me when I’m in person, Mike, is rather than just doing something we could do on a video is what can we take advantage of of the people in there to be able to ask questions, change course a little. So, I’m really excited to go back in person, my friend.
2:49 This will be on a Thursday in October. So, it’s a Thursday and then it will be also at the Microsoft office downtown Chicago. So, just for those of you who are in the area, it’s going to be in the downtown Chicago Microsoft Office, right? the AON building. Aon building right north of Mill Millennium Park on Randolph. So super easy to get to. and we’re doing it at 3 p.m. So this is a cool thing. So we’re keeping it up. A lot of people have set up for us to do that. So it’s a great time also rather than trying to meet at 6 PM. It will be two hours. And please make
3:23 Sure to register on Meetup because this is a big thing for the building. You are not allowed in unless you provide your full name. So, please make sure to do that. Last time we had a good group, but I think the last time was with Maruso, too. Yes. And also the the building, , signing up and things. there’s a little bit of a barrier to entry to make sure that we actually want you to show up cuz a lot of effort for us to go downtown, get into the building, all those things as well. Bring your computers if you if you are going to be coming because we’ll try to get you hands-on time with fabricings.
3:57 Awesome. I cannot wait. That being said, let’s jump into some of these articles. Tommy, , so all right, one more final thought here. Yeah, the link for the meetup is in the description below on the video. So, it’s down below. You can use the meetup link if you’d like. It’s down below in the description of this video, and this will be here for the next couple videos until we actually have the actual event. All right. Same thing if you’re on the podcast listening. If you’re looking on Apple Spotify, the description, there’s a link there as well. Excellent. All right. Jumping main into our main topic. So, data agents, Tommy,
4:29 You found two blog post articles about this. , let’s unpack these and maybe use these articles to give us some context to what is a data agent. Maybe we should start there a little bit and start from the the beginning and then we can start walking into some of our impressions. Yeah. So this is the evolution of AI skill and Mike we’ve talked about copilot a lot but I don’t think we’ve given a lot of due time to what data agents are the impact of them and we’ll ask the question at the end of the podcast is it this considered a gamecher
5:02 There’s two major things that we feel that this is worth an episode right now it’s this idea around data source instructions where per data source I can provide to the data agent what I want it to do how it should interact with the data source in terms of joining with things and what its actions should be. But probably the more significant one here is the orchestration with Microsoft Copilot Studio. Copilot Studio is the one that lives in all of Microsoft business applications. It lives in
5:36 Your Dynamics. It lives in any data source you can imagine. It’s a standalone outside of fabric as well, but again now available utilizing data agents as well. So Mike, these are really two major features that before with the AI skill by itself was like, okay, this is cool, neat, fine. But this is really giving this a more comprehensive feature for not just doing it and creating it, but for most organizations and most users to use it and to utilize it. So this is I I’ve done some work with
6:10 The agents or make preparing your data for co-pilot. So I can so prepare your data for AI inside the semantic model as well different than this which is different than this because this is an this is its own item inside the workspace right but in general I’ve looked at this going when I was doing the prepare your data your semantic model for copilot it was able to get close to a lot of things but again it’s non I’m going to call use the word non-deterministic it it doesn’t actually have like enter
6:43 This, get that out. It was it’s a little bit less consistent on its answers. Now, you can definitely guide it. The prompting and the text you provide in it. So, this is why they talk about the instructions. So, the instructions are what they talk about in the agents is here’s the things the agent must know, how you want it to respond, things about your data set, your semantic model, other things that are in there. when you’re talking about business language or logic that the business user knows, you’re giving it context to that. and then it will use that information to guide its answers.
7:16 But very rarely does it give you like this is the question I’m going to ask. It needs to respond exactly this way every time. And so this is where it gets a little bit weird because it’s it’s it gets you close but sometimes it’s a little bit off. Now let me caveat this. Yeah. Yeah. Yeah. Yeah. The results it gives you back might not be what you want, but typically I’ve seen the numbers are spot-on and it tells you how it got to those answers. So it says I’ve used sum of sales. I’ve used this column and that’s how it resolved the number.
7:48 So it seems to be a bit more clear like I can trust the numbers I think a bit more cuz it’s actually showing me what columns and measures it used to wrote DAC to write DAX and then that DAX was then being used to show you the answer. So from that perspective, I think I can trust the numbers. I feel like a bit better now. But how can you adjust numbers for something you didn’t ask for? Well, you are asking for it. , you’re asking for what is my sum of sales in 2014 compared to 2013 or something? You’re asking for something, but does it give you like a chart that
8:19 Like a graph? I see. I see what you’re saying. Yeah. Those results like it doesn’t always give you charts and graphs all the time. And so, , I actually when I was doing prep data for AI, I had a couple instances where it would return, here’s my result. It said an error. I couldn’t give you anything, but then it showed me I used this measure and I used this datetime and I used this column. And I was like, so you returned a null or something that wasn’t there, but yet you still told me how you got to the answer and yet I still can’t see the number. So, it was a little bit again,
8:51 It is in preview. It gets still new. So I think it’s give it some leeway here but that was one of my initial challenges was it was a little bit more difficult to figure out okay what are you actually going to get out of this with proper answers anyway so I think it’s important you say that Mike because we’re talking about three different products already and it’s important that they are not at all connected together right well it’s similar in concept but it’s different in product but they don’t overlap in terms of the effort or the work that that’s a great distinction right I think probably important at this
9:23 Point to break down what we’re talking about and especially for people listening if you have not used it that they don’t overlap in terms of I think the prompting and the work that you do the product that it is. So you mentioned the AI prep for data and AI which is part of desktop which goes to co-pilot in the standalone in PowerBI or the PowerBI co-pilot. So there’s some features there where I can tell PowerBI the one of the administrator settings is or for the data source like hey only show copilot
9:56 Data sets that I have said they’ve been prepped correct that’s an admin setting yes that’s not and that way you’re you’re covered and it’s not going to just show any random semantic model you have access to it’s only going to show in standalone copilot these are the only semantic models that I’ve prepped they’re certified as prepped think of it that Right. Like I’m acknowledging that this is prepped and then the agents will only use those as data sources. Right. Right. And again that lives in the copilot studio not a data agent not copilot part of the Microsoft platform
10:29 Part of the fabric platform. Then there’s copilot studio which I don’t even have to touch fabric ever. Actually the only connection is say connect to a data agent that’s already created. Copilot Studio allows me to create orchestration automation data agents across all my business data. That’s we call it more soft data. It’s your shareepoint. It’s your dynamics and nothing that we’ve structured in fabric again that lives I can do that as a career never touch fabric in my life. And then finally there’s data agents which are another feature or artifact in
11:03 Microsoft fabric which allows me to do what we’re talking about today or what we’re that overlap where we have the instructions what we wanted to do to a data source we can connect it to copilot studio but the interesting thing Mike what you’re especially why I brought this up is because it’s funny that you’re talking about the instructions you’re given to co-pilot the prep for data that has no correlation or overlap with data data agents, which is weird when you actually think about it. So if I prep data for AI, that does not overlap to data agents at all.
11:36 You are you meaning that data agents are not going to use the information that you provide in the semantic model for this? Yes, correct. That I would agree with. But you are in the data agents experience, you are having a similar experience around you’re adding instructions to the data agent. So I want to be clear like two separate instructions. Yeah. And they’re and it’s like we’re they’re like two separate machines and I’m talking to them independently. It would be like in my family I would talk to my wife in a conversation and then I’d have to repeat
12:09 The same information or different information to one of my children cuz they’re not in the same room, right? It’s like I’m literally talking to two separate things. Exactly. Exactly. And and it’s not like the the data agent is well and at at this point in time it’s not like the data agent can go read the instructions that were given right through the semantic model to understand additional context maybe yet maybe that’ll happen in the future but for now you can’t go access those that information
12:40 Right as these cool days in September for whatever reason we cannot do that so data agents but though have better I would say probably more enhanced capabilities. The point of it is that we can connect to not just a semantic model and prep that, but any data in fabric we can use as a data source. Well, lake houses, warehouses, semantic models we can use as a source to basically prompt and tell the data agent what we want to do. And there’s a lot of potential here because Mike a lot of people I talk to at least when they
13:13 Want co-pilot they’re like this is great that we can do this SharePoint dynamics but what about all this data and fabric right like what can we do with this and that’s really I think the goal of data agents and let’s start there Mike because I want to ask you a few questions because I’ I’ve been I’ve doing a ton with this and I don’t want to say I got issues but there’s some weird things in terms of the application for this and for me I want to start here the application for this in my organization when I I know I always think in the terms of adoption I think of roll out I
13:47 Think of getting people to use this rather than me creating it I want to make sure that people are going to utilize this and make that part of their normal operations why are they being positioned or how do you see them being positioned right now in an organization how do I see it right now let me let me Sure. I’m going to give you rephrase your question and then answer. Yeah. Yeah. Yes. And no. Like how are they being used? I don’t think they are honestly at this point in time. I don’t think organizations are using them because they’re just they’re still in preview.
14:19 They’re very new. Like so I don’t I don’t think organizations are using them right now today actively to to do their their work. So are they used? No. Are they starting to be explored? I think the answer is yes. And what I would hear is I feel like I’m hearing a lot of messaging coming from leadership or organizations are asking BI teams, hey, we’re hearing AI becoming a very important thing in my business intelligence. We’re seeing AI show up in data bricks or in Snowflake or in other experiences. And they’re like, well,
14:51 Where does the where does chat with your data exist inside fabric? And so I think this is Microsoft’s response to some of that is here’s our flavor of a data agent that is more it’s broader than just a semantic model. So one thing that’s a feature of these data agents when you talk about data agents or fabric data agents specifically is when you read some more of the documentation here it says you can add additional context to the data agents directly through like SharePoint or other locations. So like there’s the
15:23 Ability for you to add other data sources outside of just the scope of the semantic model, right? So you have a knowledge library. It’s it’s inside SharePoint. You can give that reference point some information back to the agent and now the agent can use that information to help people answer questions about things. So that might be something else that’s useful here, right? You’re mixing and matching multiple data sources to the agent level. So, I think this is I think organizations are asking for it. I think this is Microsoft’s resolve to
15:54 That. How is it going to finally be used? What is going to really move the needle here? I’m not sure yet, Tommy. I think I think that the for me the jury is still out. I haven’t quite decided if it’s yes, 100% I bought on board. This is going to be what we’re going to do. Or is this more of like we need to wait for Microsoft to rough out sand out the rough edges at this point? I’m not sure yet. So this is like So first you almost caused me to do a spit take spit take when you said organizations aren’t using this because that’s that’s an interesting actually that’s a really
16:27 Probably good take here because you answered more to me what data agents are trying to solve not necessarily what it’s solving now because Mike I would actually agree with you where data agents are feel almost like a beta version of what’s going to happen or what’s going to be there because Let’s let’s not let’s not cut around corners here. I think there’s some major limitations here for it to be adopted. If someone to ask me right now, we want to roll this out and have everyone trust it and use it. I would say we can pilot
16:59 It, but I’m not going farther than that. we’re going to test it out with your data, but there’s a lot of limitations for the developer here in terms of using it effectively or I think using it in the way that AI demands in terms of the intake process, the ingestion, the training and then actually what people are going to ask because again there’s a lot more complex things here than just creating an agent. This is not an if then this that type of scenario and it’s almost being seen that way. When
17:32 I think Asian I think oh automation but you have three major pieces here. You have the instructions I give which is the only thing I can do right now. There’s nothing else I can do to influence this data agent. There’s the data source itself which hopefully we prepped. And then there’s the the known unknown the user query. And these three parts make up a data agent. But I only have control on one of these really. Yeah. And and there’s also think think about the data agent as it relates to like the semantic model, right?
18:04 Semantic models you’re asking questions of a semantic model and then the semantic model is you’re basically asking questions of the semantic model. It’s trying to interpret the tables, the relationships and then potentially write you some DAX that says here’s the DAX I’m going to write you that will then return a table or some information. and then maybe it’ll attempt to build a visual for you. Okay, that’s that’s the semantic model prep your data for AI experience. Now, this agent experience has a little bit more to it. And when you read the docs on what is a fabric agent, it talks about like there’s, , you have
18:38 To parse the question, you have to validate that it understands the question. It knows what to do. Yeah. Then then the the AI is making a decision to say where does this data source come from? And when you again when you go back to the docs, the agent can pull from lakehouse. It can pull from warehouse. It can pull from a PowerBI data set or a KQL data set. So this agent has actually got more scope to the amount of data sources that it has. So if we’re thinking about prep data for AI, that’s only a semantic model. This one’s talking across
19:10 Multiple things. And then if you look further down the documentation, there’s actually three different I guess call it tools, right? It’s it’s almost like they’re building like multiple agents. It’s it’s like an agent for the other agents, right, Tommy? Like I’m going to ask you a question and then it’s going to deep think or think about, okay, Michael asked a question. Should this question be answered directly from the lakehouse tables or should I go right to the data warehouse or is this a KQL database? I don’t know. So, it’s it’s trying to reason its way through that and then it’s using based
19:43 On what you’re asking. It’s doing natural language to SQL for lakehouse and warehouse. a lot of work. It’s doing natural language to DAX for semantic models and then it’s doing natural language to KQL for SQL databases. Now let me this is my experience so far. Natural language to SQL experiences that Microsoft has given me has been really solid. SQL is a very known entity. TSQL might be a little bit sketchy. It might be giving you a bit different answers there cuz it’s a little bit less frequently used than just regular SQL. In my experience, when I’ve talked to
20:16 Agents about SQL, it’s spot on. SQL is very well known. There’s lots of examples. They’ve trained a ton of information on SQL. Awesome. DAX has been dicey for me. Like there hasn’t been enough. I feel like it’s been getting better of writing DAX, but it’s also hasn’t been like as smooth I feel like as doing the SQL experience. Sometimes it does some weird things and I wasn’t I’m like I’m not sure I understand why it did what it did here. , and then again, I don’t use KQL every day, so I really couldn’t have a a phrase on, but again, I would
20:49 Also argue KQL is a very specific language for very specific type of databases. It’s not like just it’s not like writing Python, which like that would be everywhere, right? If you said write me Python, it’ll just be able to burn out Python and be like no problem. So, whenever I’m in a notebook, yeah, and this is over the weekend, Tommy, here we had a little bit of a conversation on Reddit. I don’t know if you saw this this conversation on Reddit, , but someone I think Marco Russo had made an announcement that said, , dataf flows gen 2 is nice, but more and more people are going to start favoring notebooks for a proc code or
21:22 More code solution because of the cost. He was actually ribbing data flows gen 2 on how expensive it was compared to running a notebook. And I believe the PM for data engineering, Mark Cromer, jumped in and said, why do you think that? And I I jumped into this conversation with Marco and said, “Oh, I saw 100% agree like the, , dataf flows gen 2, your days are marked if you don’t start getting more price competitive.” And so Mark Chrome came in and said, “Hey, we’re giving you a nice pretty UI with dataf flows gen 2. What’s , what’s the reasoning? Why
21:56 Wouldn’t you want to pay more for an easy to use interface?” I said, “Look, my perspective is if you’re paying 10 or 20% more for a nice pretty easy to use interface. Let’s have a conversation. Yeah, I’m I’m I’m willing to sacrifice that little bit of extra maybe even 30% more of like a a data engineering job at that time. And then again, my experience here would be is just because I use a UI data to just because I use a UI to generate the code, right? I’m only doing that for like 10, 15, 20 minutes to
22:29 Generate the UI. So, charge me more for when I’m in the UI only. And then if I’m running a scheduled job, don’t keep charging me more to run that job. It feels like it’s I’m penalized every time I run the job between when I run the UI versus when I run the actual job. So I said, when you’re charging me 50% plus, like it’s double the price or how I’ve seen it was dataf flows gen 2 was like double the price of dataf flows gen one which is double the price of a notebook. So you’re like way way above, , 3 4x more cost in and so that’s when I
23:05 Start saying no. That that’s when I’m like not that’s not worth it to me because we’re no longer competing dataf flows gen 2 against talent. We’re competing dataf flows gen 2 against co-pilot and notebooks. And so what I’m finding is customers I’m like hey you’ve got this ultra knowledgeable again we’re going back to agents. I I promise I’m bringing it back to agents. I’m waiting. We have this ultra knowledgeable land this. Yeah. Yeah. Here we’re going to land it here. The landing is co-pilot is there. It’s in my browser. It’s for free. I can use it in Edge. I can go get a custom one. I
23:37 Can use chat GPT. I could ask it, hey, write this M code in Python. Rewrite this. And it just boom does it. Like it. There’s almost no, , it’s it’s almost frictionless. And if someone gave me a slightly better experience inside notebooks with agents, you wouldn’t need anything. , if you gave me the VS Code GitHub like experience inside notebooks, which it’s not quite there yet. We’re not we’re not there yet fully, but I’m with you, Tommy. When you get that, then I have me as a business user
24:10 Who maybe is a little uncomfortable writing code. The fact that I can talk to Cop and say, “Write me a function that does this.” I’ll spit an answer. I can put it in my notebook and test it. It almost works almost 100% of the time. You and I should be able to create a data flow UI that basically writes Python. All those functions are available in Python. So there’s your notebook done. Solid. Okay. Tommy the I 100% agree with you. But my complaint here is so yes 100% agree with you in that standpoint. Actually dataf
24:43 Flows gen 2 missed the mark in my opinion. It should have just written Python and it should just be like here’s the Python transformations that you need. I’m just going to go run that on a very simple Spark cluster either single node or multiple node and it’s already when you have bigger data it automatically would handle it. Sure. But the issue here for me is Tommy the way I read the data flows team they’re continually pushing the price higher because of the UI. So we could build it but it’s the UI issue. Give them a little more credit on that one but
25:15 But it’s the issue of the UI. They’re they’re pushing the price up to run that thing. Even if it wrote Python, I still don’t think we’d get the pricing where we want it to be because dataf flows Gen 2 is priced so much higher than everything else because they want they’re trying to say, “Look, we’ve made this so clicky clicky draggy droppy for you. That’s the price you’re paying and you’re going to pay it every single time you run the job.” And that’s where I’m saying I don’t want to do that anymore. And now that I have agents, I I don’t need I code is so much less scary now with agents or co-pilots.
25:47 Well, I want to I want to touch back on something you said about DAX and SQL. And I think this is an important distinction because this really goes back to two major factors for me. It’s how we’re writing our instructions, but also more importantly the language itself. So think about SQL first. It’s probably like 20 to one the amount of coding examples they have with SQL to DAX in terms of the training data that they have. Sure. Two SQL always outputs a number or an aggregation. Right? If I write join this
26:19 Group by I always get a solid number. But DAX is different because filter context right if we know the our the philosophy of a D or not philosophy but the mechanics of a measure well it’s never calculated actually until some filter context. So that’s an interesting thing there too. So that requires the and it doesn’t say that a lot in the documentation. If you’re writing DAX on semantic model, that’s your data agent. You need to provide it instructions that go through this because for me, I’ve seen a lot of success with prompting and
26:53 Getting a really solid DAX function, but has required a ton of instructions and context on the model even to the point of I’m filtering on this. When I interact with this, this needs to ensure it stays or modifies. when I have when I provided that type of detail, I get really great DAXs. But if I just say, hey, I have two tables joined. Yeah. Then it can get very wonky. Do you have anything else on that? Because there’s something I really want to touch on when we’re talking about the data sources here that we have not touched on.
27:25 So that’s a great thanks for Yeah. lobbing it over to me. I would agree with you, Tommy. That’s the same. Like what comes to mind is am I spending so much time writing the prompt that it would have been just faster for me to like actually just learn DAX and just write it the way I wanted. Right. And and maybe maybe Tommy when you’re talking about this really elaborate prompt that’s doing a lot of things, right? There’s also this concept of a conversation, right? Here’s my two tables. Here’s my
27:57 Relationship. calculate this, right? You give it the simple calculate my sum of sales and whatever and you it will write you a DAX statement, but like with what we all doing, sometimes your your thoughts aren’t fully fleshed out about that particular measure and you need to think more about, wait a minute, that didn’t quite work the way I wanted to to. And so then you have a conversation of a bit more, okay, no, I actually don’t want that. I want you to retain this filter context when I click on this but then let this one be
28:28 Flexible. So I think sometimes too again it’s your knowledge Tommy of like the report and the semantic model is much higher than I think an average user would be and so co-pilot may need to ask some clarifying questions be a bit more like interactive to you to I need a little more info before we get this running. Yeah, here’s a basic measure. It’s just going to calculate this very simple thing. And then Tommy, to your point, the reason you’re getting really good DAX out of it is because you understand the relationship between the visual the DAX you’re trying to write and the
29:03 Filter. Yeah. The filter context and the report context. So there’s this concept here like if we know this, how is the AI or the co-pilot helping us get through that filter context conversation? Right. And maybe there’s a Yeah, correct. Well, again, you and I are you and I aren’t looking for the DAX statement. We’re looking for when a user asks a certain question, how to write that DAX statement. Which goes onto my next point, unless you add anything here because this is this to me is the elephant in the room on on data agents
29:36 And copilot in general. I think a lot of people are getting confused the data that we’re using and training data or lack thereof. And this is what . Any agent, any model, go to hugging face, go to any any place where there’s an AI model that can be utilized. It runs on data. And we say that a lot of times, but the misconception is doesn’t run on random data that but it runs also on training data. That’s why there are different when you say a different model, it’s because it’s been trained on different things.
30:09 And that doesn’t mean it can’t be trained on a semantic model. Like there’s another post about semantic models or what AI needs to use. That’s fine. I agree with that but not training data. And this is the problem I think Mike right now and especially where people are saying hey we have all this data with data agents. You do but you don’t have a good you don’t have a good agent because do what a training data actually looks like? And this is the difference here. It’s more vector-based, but it’s really input and expected output a million times thing. And letting the agent know if we
30:41 Get any types of questions around this, this should be the expected output in text or if it’s an image type model. And all we have available to us is the final data, right? And we don’t have we can’t create a like in a sense its own template to say hey data agent one if you got questions like this this is expected output this is expected data source to use we only have a single prompt to use which is not going to be sufficient for a lot of people’s queries it’s a sufficient for the exact query
31:14 That you give because in a sense when you think about it Mike your prompt is a single record for the data for that data agent as an example of what to do. What we really should be providing it is a training source data to say here’s all the types of queries we may get or questions we may get or questions that we may get. Here’s the output we want. This is what data agents or let me let me back up. This is what agents run on. Mike, when you look at any model, any AI model is uses training
31:48 Data before we get to the real data. And I think there’s a there’s a lot of confusion here. He’s like, “Oh, I can use my data in a data agent. Great. Why isn’t it working?” Because we have not trained it on anything. And Mike, this is the where here I’m seeing the hiccup there. There’s a there’s a knowledge gap between the agent and what we want, right? And in some situations, we’re going to know very clearly what answers we’re looking for.
32:18 Sure. And then there’s a lot of this like choose your own adventure type path stuff. Yeah. And that’s where I feel like I have a harder time getting to the answer of like what is it actually doing? Like how did it get to this answer? so yes, I feel like data agents is more of like here’s a pile of information, right? a lakehouse, a warehouse, a semantic model, and you’re going to present this as a here’s a a collection of data.
32:51 Users, you may now ask questions or interact with it and it will, again, I do like the idea of like it’s it , I I feel like it does a pretty decent job of like, hey, if I give it a warehouse, it’s going to write SQL. If I ask some specific questions around if I’m very careful about what I’m asking for. So I feel like this is let me let me say this way me as the user I still need to learn how to use it correctly. Right? This is a new technology built very in prompting and how to prompt and what’s the questions you
33:22 Should ask and how do I ask a question to it in a way that’s going to get me the answer that I want. If I want a very specific answer I have to write a much more elaborate prompt to get down very closely to the details. But if I want just more like a general answer like what’s the average discount by product ID I’ve got to make sure that me the user is understanding okay does that table exist am I writing the columns in a very in a similar again you don’t have to be spot on this is one of the things that I think is different between this experience and like Q&A yes yes yes I
33:54 In Q&A in Q&A you had to be like exact like if it wasn’t exactly the column name exactly the way you chart yeah like it wasn’t able to figure it out and then Now it’s it’s a bit more conversational. And then you can you you have to start with this and maybe to some degree this is when I look at like the co-pilot home experience as well. Again that’s the one I probably play with the most. That’s I’m relating it to that as well. Start with a generic question. Just get it to return some result and then once you have the result you can say hm interesting. I can see the
34:27 Results of that data. Let’s start talking about, okay, I’m asking for average discount by product. Well, maybe we should start talking about can you make a bin, , bin this information into groupings or, , the average discount should be represented as a percentage. Can you please represent it as a percentage instead of a just a number, right? So then I can more humanly read. So you can be more specific as you go, but again, it’s more conversational, right? You can ask it. I’m programming it with my language, my my text, and I’ I’m me personally, I’m not good at it yet. Like
35:01 I’m still trying to figure out how do I talk to it. The I think the reason why is no one is is because the the other misconception I think is when we think of a data agent, we think it can do all things. That’s not true. But many data agents can do a lot of things. And here’s what by this. We hear data agents, you think, I’m going to create one over this data source. I’m going to create another over multiple data sources. And that’s sounds fine, but it’s one of those if it sounds too easy, it probably is. There’s another approach here, Mike, that I’ve
35:34 Been seeing much more successful is I’m creating, and I don’t say this often to people, but limit yourself. Limit limit your possibilities per data agent. And this is hopefully this gets a little clear here, but I creating data agents that really accomplish specific tasks. I have orchestration available to me. So I can have multiple agents in the same conversation, but I don’t want a single data agent to do a lot of general things. I want a single data agent to do some very specific things. For example, rather
36:06 Than a data agent for a semantic model, Mike, I’ve been testing semantic or like five data agents with the semantic model like, hey, you are the sales agent man. all the things about reps and their quota and that’s it. That’s all they’re going to touch. I have another one that’s going to be based on the budget and the operations. Hey, you’re my date time intelligence agent. Anything that has to do with measures on time intelligence, anything has to do with our month-to- date numbers, you’re going to help us there current. Anything that’s going to be based on my quota or my budget, that’s
36:38 All they’re focused on. And that seems to work a lot better because again, we’re limited right now by a single prompt query. And it’s very, again, the more you add to that prompt, it’s not going to understand all of that. You’re still dealing with the basics of prompt engineering, right? What you show in the beginning is more important. How you word things is very important. And if you have 300 character or 300 lines, well, not everything’s going to be treated with the same priority. So, there’s another approach here where we
37:10 Limit our data agents, but create more of them. And I’m finding that to be more successful because let’s not kid ourselves. We’re still dealing with a product that most companies don’t know how their own AI works. Mike, I think I said this to you, but Anthropic has it’s my favorite white paper I’ve ever read in my life. And it’s Anthropic who creates Claude, and it’s their study on how they think Claude works. Another way of saying that, it’s their best guess on what they created, how it does
37:42 What it does. They don’t even know. It’s their It’s their estimate. It’s their theory, which is mind-boggling. So, we’re still dealing in this world where most It’s like It’s like, “Look, here’s a nice lasagna. I don’t know what’s in it, but here I can get it. Looks like Yeah. It looks like tomatoes.” And you’re like, “There may be a rock in it, but I don’t know.” Yeah. It’s like I made it. It’s like I was part of the process a little. I would you eat it? Like, and so a lot of things are like this. Now it is really cool but I’m going back to the point here is I think there’s a mind shift we need to do when we’re looking at data
38:15 Agents rather than a data agent can solve all things is many data agents can solve a lot of little things and can be very successful that way. Mike let me throw that back to you here. I’m going to again to your point I’m going to I’m going to volley that back to you and your approaches with data agents or your starting point here the methodology that you’ve been trying to take a look at we’ve talked about where it’s like I don’t see this but where are you seeing this be more like okay I see the potential here. Well, I think there’s a little bit of a
38:50 Game we’re playing here, too, because as I’m as I’m building this one, so you could basically you can give the AI agent some very good instructions, right? So, , literally while we’re talking here, I’m I’m I have an agent up. I’m I’m asking some questions. I’ve added a lakehouse table. So, , think about the different layers of your lakehouse and what information is in it, right? So, for example, a lakehouse has tables and columns. That’s basically all the metadata you get for a lakehouse. You can’t add a description to the table name. There are no measures. I’m not building relationships in the lakehouse between different
39:22 Tables. I have to define all those things. So if we’re pulling data from that area, it you have to give it more instructions to help it understand these are how this stuff works inside the lakehouse. Now if we move forward to like semantic models, the semantic model has a lot of metadata in it. Yes. Right. And so if you give if you give the agent the semantic model, the semantic model can have relationships between tables, descriptions of columns, actual measures that calculate something. And us as designers of those things need to add enhanced information
39:54 About this measure calculates this. It’s filtering because of this. These types of things are stationary in the filters. They will be calculated every single time. And while this one so when you get more complex DAX you could just let the AI read the DAX but it also helps to have human based context as to why that measure exists right so as I’m playing with this right now I’m playing with a lakehouse and the lakehouse I defined hey list me the top 10 orders with the highest revenue and then I and because it’s a lakehouse there is no calculation to define
40:25 Revenue I have quantity and price but then I say okay go in and take the quantity times the price that equals revenue. I have to define some math to it and it did it. it actually it actually returned the results. It gave me a SQL statement return the top 10 sum this group by that revenue this. So my experience is when it when the thing is writing SQL it’s pretty dang good. And to your point, Tommy, right, if we’re going after raw tables of data and telling the AI what we want, it’s pretty
40:59 Intelligent to figure out, yeah, from that information, what do I need to produce and get you an answer? Where I think things get a little bit more interesting here, and this is what my point was, is there’s a relationship between the designers, the prodevelopers, and the the AI and how people are going to perceive it. So, I could build a bunch of measures. I could have so for example I could have gone to the same table inside a semantic model made a measure that calculated revenue and then the agent would had to be oh well I could have written the same
41:31 Question but now because I’m in the semantic model I can give it more metadata and context so the user doesn’t have to work as hard right so I feel like in the same way that we’re designing this is maybe going to sound like an analogy but maybe maybe it is maybe it isn’t in the same way I play with the semantic model to build measures and columns and data structures to support the visuals. I’m playing the same game between the lakehouse, the semantic model, the KQL databases to build enough metadata either in the instructions of
42:05 The AI or within the semantic model so the AI can read it and then we can now transition over to okay, now that we have the metadata. So now I’m playing this game between okay what does the agent resolve when I ask this question and then what do I have to supply the agent either in metadata in the semantic model or instructions. So if I if I can’t supply it directly with the semantic model again thinking this is serving two purposes the semantic model serves business users who are trying to create their own reports on top of the semantic model and now I’m also thinking
42:37 About programming for the agent as well. Those two things combined makes it easier for me to write less instructions inside the AI agent. So again, all this to me feels like a balance beam between effort and value. I’m going to keep going back to that. Yeah. No, I I’m curious here because you you mentioned something that you’re beginning to fire the synapses here with the lakehouse. , we have so many more possibilities and I I still Mike to this day and when I was nervous when fabric came out like how is this
43:11 Going to work man have I loved the fact what we can do with lake houses because it opens up so many more doors in terms of storage and what we can function especially to the user and something like this scenario I’m going to propose well theoretically Mike we could add to a lakehouse almost that training data or that metadata as addition columns like hey look to this here in the same table that same raw table for whether examples or what to filter or whatnot where we’re providing AI context to the AI in the lakehouse itself where we almost have
43:45 Those like feeder columns or those true or false that are only meant for AI only meant for the data agent to see that can be part of my prompting like hey these columns are for you if if someone mentions this anything that has a true for the AI sales you So only include that. That’s another possibility here where we can actually feed it with some additional information and utilize a lakehouse. You’re not dealing with semantic model getting loaded with additional columns in lakehouse. So there there is some points that you’re saying here too and I
44:17 Want to touch briefly. I love your point too. There’s effort versus the benefit here. How much effort am I going to be putting in for the prompting? And even if I did this true or false with AI helper columns call them agent helper columns I kind I might write a blog on that. , but Asian helper columns, if I’m putting all this effort into this, well, what’s the value we’re going to get out at this point compared to to your point, someone does the explore in PowerBI, right? Or someone who goes to the metric set in PowerBI. I guess yes, I agree. I will agree with
44:50 That. And I want to add something to that because Yeah, I think I think we I think we should focus on what is the endgame. This Yep. Right. So, go ahead. Yeah, we’re there’s a lot of really interesting flashy tools here. What’s the endgame, right? What is the endgame of letting users answer that question first? And I think that helps you direct a lot of your attention towards should I be spending time in agents or should I be spending time in something else? So, is the endgame to expose and help users understand the tables in a lakehouse? Right? Is that is that the end game? Is
45:24 The endgame for you to allow users to talk to a semantic model and answer real business questions and walk away with an actionable item? Is that the endgame? Is the endgame for the users to talk to a semantic model or the AI through the semantic model and actually produce a report for one time or repeatable reports? Like what what are we doing there? So like I think if you decide the purpose of what you’re trying to put the agent in, I think a lot of organizations are like pilots’s here, agents are here, we need these, how do we use them, how are we
45:57 Using new technology because they want they want the message to their leadership, we’re using AI and it’s adding value. I think it’s a a hot buzzword and people want to figure it out. But I think there’s a pause moment here. I would want to caution you pause and step back and just wait and say why are we putting like I agree you should use agents if they add if they make sense but why are you using them what is the endgame and I think if you just clearly define what is the endgame I think you’ll be able to sift out some of these like hot new features and say well the if the endgame is this
46:29 Repeatable regularly reusable report then that should we should just build that we should spend the effort instead of trying to get this AI to build this. You use the AI to build a report. We just build the report we know we need, get our experts in there and do that. Okay, fine. Now, if there’s something around the idea of we want people to discover things or go start learning SQL or start learning how to go manipulate things, that’s a little bit different story, right? Right. Cuz because if I’m in the I’m in agent mode and I’m asking questions of my lakehouse, it’s actually going to
47:00 Write SQL statements for me. And so the way I learned SQL was sadly enough, I took a class on it eventually, but I started SQL learning it inside Access had an access database and it was so again it goes down to this whole I started learning the graphical interface of like making tables, joining them together, adding relationships and then I was able to pluck and pull columns down to a graphic interface and again it couldn’t do everything. It would do some of the simple operations for SQL but then it would work. So that to me that was very useful. That was how I started learning.
47:33 Yeah, I think agents in some way will be this next level for people to learn how to write SQL again. So and and they don’t have access anymore, but they now they have agents. So hey agent, I want to do this and then here’s the SQL statement to do it. Oh well, I didn’t want that column as a number. I want you to format it as a number. Can you change it? And now it does, oh, here’s a cast statement. Here’s the numeric. And we’ll then format it for you. And then you can take that statement and go to a SQL endpoint or the warehouse directly and run and execute that against the SQL endpoint.
48:06 So I think there’s I think the endgame of what you’re trying to do with these agents is very important here cuz that’ll help you decide if you should spend time building it or not. I’ll just pause there. Does that make sense? Mike, it’s like we’ve done 460 episodes together because I I completely agree here and I’ll take this a little more micro in terms of how I was taking what you said because I was literally going to say with data agents and really like what’s the use here like what what’s the scenario here when we do how would you recommend this in
48:37 Organization and they are very much right now to me what I call conscious workflows there’s not a I would not go into a data agent right now and just start exploring yeah we’ll test something out they work much better in copilot studio and with what I’m really a conscious effort on hey who what’s the persona here this is not I’m not creating a data agent for the organization this is for an in operations an has to go through her normal day-to-day X y and z connect to that data have the co-pilot studio with
49:10 The orchestration but have simple workflows here and it does do a really great job there but they need to have again to your point endgame you’re talking big picture here I’m talking a little more little picture here on if you do want to get started you’re like what I want to explore this you have to have very simple or not simple rather but very conscious workflows of persona and expected a action that they’re trying to do right now that’s probably where we’re at in that playground that’s what the playground and the tools available to us
49:42 Designate but I’ll take it back with what you’re saying here because I I really do love it is right now everyone’s trying to keep up with the AI Jones. That’s just how it is basically thing. So well they have AI and we should be using AI and fabric has AI and it’s great and it’s fine but let’s also again let’s not beat around the bush here. There are some limitations here. If you want to utilize it like the other AI dedicated tools out there which have training data, waiting modeling, vector databases, they have all the
50:14 Other data sources here. We don’t have that in fabric and that’s okay. That’s okay right now because we do have something a data agent is not another version of anthropic or another chat GPT right now. Let’s get that out of our mind. But what it can be is actually very effective. Again, it’s just not you’re not creating chat BT chat GPT in a data agent. And that’s probably not the meant not the use case right now. Can it solve someone’s problem in operation who has
50:46 To input a bunch of information? Great. Yeah. And Mike, I’m gonna let you I’m going to volley back to you because I know where we’re going with this. We’ve talked about this before. What does this remind you of? What does this sound like to you, Mike? Yeah. I said, Tommy, I’m chatting here. You on the side here. , I just threw down an image in our chat between you and I, Tommy. So, just you I think this is where you’re going with this one. I feel like this is we are so the AI like where where where are we in what we call the Gartner hype cycle right new technology
51:19 Comes out there’s this like wave of like excitement of like here’s the new thing this is going to be amazing and then you hear all these people saying this is the first generation we’re early to adopting there’s a mass media hyping that’s going on like everyone’s excited about this you have to get these things it’s work figure it out and then So everyone starts supplying these things and then we have this early adopters. Everyone’s early adopting and then people come in and say well is it really worth the value? Is it you start having some negative press some suppliers start consolidating or failing and then you
51:53 Start having second or third rounds of venture capital and so funding comes in and then you hit this trough of disillusionment. So when you think about the different areas I don’t think I think we’re just beyond the peak of inflated expectations. I think people are starting to get a little bit more real with AI agents. At least I am in some cases is extremely powerful. But I also realize it can’t do everything. It’s not the silver bullet. I don’t go into this thinking, oh, co-pilot and agents are going to be I’m going to delete all my reports and I’m just going to use agents. I’m not sure. That’s not the story here. So, I think
52:26 There’s some alignment to people’s expectations and the effort and time it takes to build the agents themselves. And again, to be honest here, Microsoft has agents in preview. , it’s still a feature that is being refined. It’s still getting its legs. It’s still trying to figure itself out here. So, I think we’re close to the trough of disillusionment where people are saying it adds some value, but it’s not enough value yet. And we’re we need a couple more iterations on the solution to then align, okay, what are people’s
52:59 Expectations and needs to what the agents can start doing. Now, I will agree of the two hype cycles that we’ve had, Q&A was like this really big thing. It was in every demo. I don’t hear anyone talking about Q&A now. Like, it’s it’s done its time. Agents feels like this next wave of Q&A, but at a much better pace and much more useful tooling. So, maybe the maybe maybe I’m looking too too short at this. Maybe my trough of disillusionment is too small. Maybe I should have gone was Q&A visual the first trigger of talk with your data
53:35 Right and we’ve already gone through the trough of disillusionment like we’ve already done it and this co-pilot is now agents now this is the second or third generation evolution on chat with your data and so now we’re getting like okay this does add real value this does seem useful and so we’re actually to the tooling is now shifting a much and maybe I’m again I’m trying to figure out like where we are. There’s probably like little mini hype cycles and there’s like the big hype cycle, right? I think that’s fair because Mike I think I think the biggest misconception again and
54:08 Again I really think is people are expecting that this data agents to be their chat GPT and their their model their AI model and it’s not meant for that and that’s again to me okay I’m trying to champion data agents and I really do see the value here but I don’t want to say hamper your expectations or it’s not as good as you think it’s not that at all. It’s it has its function and it has its and to your point endgame,
54:40 But it’s not the full hey I’m going to open up chat GPT and I’m going to open up my co-pilot data agent and they’re both going to work exactly the same. Not that case. So, as long as you understand that, then I think that actually opens up what data agents can do. A lot of people are trying to prompt it so it’s their own local model that’s running everything and can do everything. , and I guess that sounds like a closing thought here, but I so I’ll probably end with that because this has been I love these conversations for me as I look at the future here with data age unless you had something else.
55:12 Don’t want to close it too early. Yeah. No, I will end on my closing thought as well. I think this is a good time to close as well. So, one thing I think I would say from this conversation unpacking unpacking what we’re doing, I really believe that agents will be very helpful. They will be useful. I think we underestimate the amount of time needed to invest in developing them and getting them to be useful to team members. I think there’s a little bit of an underestimation right now at this point. Now, it’s getting better. , I really would like some like testing framework around
55:44 This. Like, so I’ve done all this instructions. I’ve done a couple questions on my own. I’ve done some hey, when I get this question, return this answer. But I think at some point we need some level of like saying I’m going to have this like it almost feels a little bit like Tommy like when we do even data validation right because the AIS are a little bit undeterministic it would be really nice to have a way of being able to say look I’m going to ask a handful of questions to it and then before I publish it it will go through and will it return the right results or the right answers to me. So that way, , again, thinking through this as well, if I
56:16 Change the AI, if I add different instructions, if I tweak it a little bit, those same five or six or 10 questions that I think are going to be like needed to be outputed, I have to have some level of confidence it’s going to give me the right kinds of answers for users and it’s going to be the right answers. for some reason, that seems like something that needs to exist. Yeah. All this I’ll say is it’s it’s a balancing act. This is a balancing act between value and effort. And I think right now not quite sure I’m sold yet. Excuse me.
56:47 He’s good. Excuse me. so anyways, that being said, I’ve really enjoyed this conversation. I think data agents are a very good evolution here. if I had to look at the bigger spectrum of things when Q&A came out versus what I see data agents doing, I feel like data agents is way more capable, adding a lot more value than what we did initially with just a Q&A visual inside PowerBI desktop. So, if anything, we’ve gotten another round of chat with your data or, , ask questions of your data and get results of your data. I think this
57:19 Whole new experience is very much changing things and I really like it. Again, I’m still also in this learning mode of I still got to figure out how to prompt everything better. And my children and my kids are going to have this. , their whole world is going to be Yeah. get computers, learn how to prompt from like could you imagine Tommy, imagine if you were coming out into high school and your first couple interactions with computers, you can do the things in the computers, but then you start seeing these agent things pop up and you just talk to it and it just does things for granted. You just go, “Oh, that’s a computer thing.”
57:52 Well, and the reason why I’m saying this is then, , back to my very early comment around dataf flows gen two, which is like, , why use that or why not just write notebooks and code, right? Well, if I’m already learning to write English or my my home language, Spanish, Portuguese, whatever, wherever you live, right? If I can write that language directly to a computer, it can understand what I’m trying to do and output a result that’s decent. Like anything, anything I say write a Python, write a Python function, like there’s when when do we just drop all
58:24 Languages? Like when when is the computer programming language just drop? You’re going you’re going big picture here. Yeah. when I see when do we need none of it and you just talk to it and and it will be and maybe it’ll show you but it’ll be like yeah it runs on it but like we really don’t need to know anymore the AI is just so good we just keep prompting it with a couple different ways and it just fixes itself like there’s going to be a day I think and I think it’s coming very soon where writing Python or SQL or KQL will be almost non-existent and we’ll we’ll talk to it in regular terms and
58:58 We’ll just know what to do all the way through. Anyways, let’s wrap it there. Thank you all for listening to the podcast around agents today. I hope you found this informative. , let us know in the comments. It actually really does help the algorithm if you engage with this content either on YouTube or other places. What are you doing with it? Are you using them today? Do you have perspectives on where this is working for you and where it’s not working for you? Please let us know in the comments. Tommy and I do read them and we do comment and we’ll respond back to your comments as well. Check that out as as well. Also, I want to point out if you like this episode and
59:30 You’d like to watch this episode without any advertisements, you can do that because YouTube puts ads on videos and that’s how it happens. So, go check out our videos, become a member. Our lowest member level of membership will let you say visualize this. So, you can also go and watch this episode with no ads. The videos will be posted right after we do them. Also on YouTube as well. With that being said, Tommy, where else can you find the podcast? And there’s more, Mike. You can find us on Apple, Spotify, wherever you get your podcast. Make sure to subscribe and leave a rating. It really does help us out a ton. And share with a friend
60:03 Because we do this for free. Do you have a question, idea, or topic that you want us to talk about in a future episode? Maybe a little more data agents or AI or just PowerBI or anything else? Head over to powerbi.tipsodcast. Leave your name and a great question. And finally, join us live every Tuesday and Thursday, 7:30 a.m. Central, and join the conversation on all of PowerB.tips social media channels. Thank you all so much, and we’ll see you next time. down.
Thank You
Want to catch us live? Join every Tuesday and Thursday at 7:30 AM Central on YouTube and LinkedIn.
Got a question? Head to powerbi.tips/empodcast and submit your topic ideas.
Listen on Spotify, Apple Podcasts, or wherever you get your podcasts.