Self Service Analytics with AI – Ep.539

Anthropic wrote up how they enable self-service data analytics with Claude, and the striking part is how ordinary the foundation is: canonical datasets, dimensional modeling, a semantic layer, and documentation that isn’t stale. Mike and Tommy walk through the article layer by layer, because it makes an argument they’ve been making for years — the semantic model is the heart of this, and agents raise the stakes rather than replacing the work.

News & Announcements

• How Anthropic enables self-service data analytics with Claude — The article driving the whole episode. It describes four layers: data foundations with canonical datasets and enforced governance, sources of truth with a semantic layer prioritized to remove ambiguity, a skills layer of markdown procedural knowledge, and business context. The headline numbers are hard to ignore — roughly 95% of business analytics queries automated at around 95% accuracy, where accuracy never exceeded 21% before skills were added.

Main Discussion

Topic: What Anthropic’s analytics stack confirms about semantic models, metadata, and where humans stay essential

The hosts treat this as external validation rather than news. Anthropic did not start with a clever agent; they started with dimensional modeling and metric definitions, then layered agents on top. Mike’s summary is blunt: without a good semantic model, none of this works, no matter which tool you buy.

• Good data comes first, and the article says so up front. Standard data engineering, data quality practice, dimensional modeling. Mike’s line for it: if the agent can’t figure your model out, humans weren’t going to either.

• Analytics has one right answer; code usually has many. For coding, a plausible implementation is generally a valid one. For analytics, a question like “what were members in Q2” has exactly one correct result — which is why ambiguity in the model is fatal in a way it isn’t for a coding agent.

• The same word meaning two things is still the problem. Both hosts have watched two teams define “leads” differently — usually one filtering something the other doesn’t. Somebody has to own the metric definition, and the other teams have to adopt it. Ten-plus years on, this hasn’t changed.

• Domain models beat one monolith. Build a sales model and an operations model rather than a single enterprise semantic model. Some duplication across domains is acceptable; a user asking a question about their own area shouldn’t have to face the whole estate. Kimball and dimensional warehousing practice carries over unchanged.

• Metadata is a first-class product. Coding agents work well partly because codebases are legible — readmes, docs, structure. Semantic models need the same treatment, including explicit weighting so the agent knows which table “revenue” should resolve against.

• Skills took accuracy from 21% to consistently above 95%. Mike flags this as the number to remember. Business context encoded as skills is what moved Anthropic’s results from unusable to production-grade — which lands squarely on the previous episode’s argument about governing them.

• Mike doesn’t want the agent to answer the question — he wants it to build the report. Rather than agents reasoning over data live and possibly landing somewhere different each time, he’d rather have them produce ten domain-specific reports that answer specific questions consistently.

• Thumbs up/thumbs down is very low signal. Mike would rather agents offered several candidate answers to choose between, the way PowerPoint suggests layouts. Picking among options tells the system far more than a binary rating.

Trust and the overwhelmed user

The failure mode both hosts keep returning to is a huge semantic model and a user who has no idea where to begin — too many tables, too many columns, no obvious starting point. Agents get overwhelmed by the same sprawl, just differently. Their fix is citations: show which model answered, which report it maps to, and where to go for more. If a user has to re-ask the question eighteen times in a different interface each time to feel confident, they’ll stop asking.

The soft data question

Tommy closes on something the article surfaces and Fabric doesn’t really address: decision logs, roadmaps, org structure, meeting notes, PowerPoints, SharePoint documents. It is all genuine business context, none of it is tabular, and the common failure across every layer Anthropic describes is the same — poor or stale documentation. He’s putting it on the topic board for a future episode.

Looking Forward

Read the article, then audit your own semantic layer against it — if two teams still define your core metric differently, that’s the work, not the agent.

Episode Transcript

0:26 Hello Tommy and welcome everyone back to the Explicit Measures podcast. Happy to have everyone back back to being in the in the show again. Good morning. How you doing? doing? Mike, I am doing excellent. It’s good to see your face. It’s good to be here. We are rocking and rolling yet again. The podcast is up and running. We got a great article today. this is an article that I found that was I just read this one. I thought, “Wow, this is so applicable to what we’re trying to do now with Power BI and all these new agentic tools that are showing up here.” So, it’s talking about how Anthropic enables self-service data

0:57 how Anthropic enables self-service data with analytics with Claude. Which is very interesting. So, I’m looking forward to unpacking this solution and where does this fit? Tommy, I’m seeing so many things across the internet around people building reports now with agents and now Rayfin has been out. We were so pro on Rayfin not last week but the week before. We did an entire series on Rayfin projects, how to get started, how to use templates, some tips and tricks on how to use agents with, visual

1:27 to use agents with, visual UI programming for apps. It’s getting better and better every week and I’m actually really enjoying all of the projects. projects. I don’t see anything else being mentioned on LinkedIn anymore other than like like my mainly Rayfin projects. Look what I did. Look what I built., it’s just been all over the place. And I think to the point what we’ll get into the article. again, this is from Anthropic on how they use self-service analytics is you can build all those things, Mike, but there’s a whole other side that you need to account for. How

1:57 side that you need to account for. How do you actually make it work for the business that I cannot wait to get into. Yeah, I totally agree with that one. All right, that being said, let’s jump into any news items, Tommy. Did you find anything that’s interesting and newsworthy? So, there’s actually no news today, Mike. I just want to briefly give some best tips on traveling long distances with family. All right. So, I was on vacation last week. We went to a wedding. We were in New York. we were in Long Island visiting family at a wedding in Jersey. And we drove from Chicago. That’s a 12-hour, 13-hour

2:28 from Chicago. That’s a 12-hour, 13-hour drive. drive. to say, how long is that a drive to get to New York? That’s a 12-hour drive straight if you went straight. Yes. If you went straight. So, to Long Island is 13 and then if that’s not accounting for,, the World Cup, the US Open, and everything else happening in New York City at the time. So, So, So, right now I have a bunch of Icy Hot on my back. I do not like not to drive. I like to drive. and one of the things that we’ve done is you minimize the stops. We have three kids. kids. Yeah. Yeah. But, we had Trader Joe’s and Trader

2:58 But, we had Trader Joe’s and Trader Joe’s is excellent for all the snacks because we just snack on that the whole way. way. It’s not the best diet. But, Tommy, I’ve also Listen, this is funny. This is I’m going to I’m going to out you here, Tommy. Every time I’m not on the podcast with Tommy, there’s like a bag of something at your desk. So, I know I know you’re a snacker on snacks around the office. I have put them in the corner. They’re over here. I know I know there’s snacks somewhere. So, So, So, So, your your secret a little bit is making sure get you get good snacks for your

3:30 sure get you get good snacks for your road trip. Espresso chocolate beans for me. the kids have chips. We have they they have like these little breakfast like pancake things that you that are in bags. So, that makes the the trip go faster and we do like every 4 hours just get out of the car, the car,, get gas. I always get my espresso, let the kids run around, and you can we we can get there in about 13 hours, which is pretty good with kids. Dang, that is like really good. Yeah, I have to admit, Mike, I was disappointed with the pizza.

4:01 disappointed with the pizza. Really? Really? Have you turned into a Chicago pizza person now? Absolutely not. Absolutely not. I just have a standard. A pretentious standard. And well, because we were a little east of where I’m normally in Long Island. So, we’re in a place all all the names are just perfect, like Syosset, Hog Bog. And where we were, we didn’t know a lot of the pizza places. So, a little dis- a little disappointed on that front. We did come home with our 24 bagels, because you do not leave New York without bagels.

4:31 without bagels. Really? Really?, at least, you can, but my family never has. Okay. Okay. Like, I don’t I’m sure it’s totally fine for most people, but yeah, no, New York was good. The trip straight was back, but I know we got got home in that weather storm or whatever it was, the the storm that I’m sure you guys hit you guys. guys. Yep. Yep. Yeah. So, that was a doozy, but I cannot wait to just get into everything. But, Mike, I know you actually do have some news, I believe, around one of our favorites, Kurt Buhler. Well, I I can’t I can’t argue with this

5:02 Well, I I can’t I can’t argue with this one,, Kurt comes out with things like almost at the opportune time, exactly what I’m thinking these things things whenever it just seems to be a neat correspondence between what he comes up with and what I can think about like what’s what should be tested. Right. Kurt has gone through and there’s a there’s a number of ways you can use agents inside Power BI. You can use agents with editing files directly with the agent. You can just throw an agent right at the PBIR files. You can use a PBI

5:33 at the PBIR files. You can use a PBI Power BI Power BI You can use the manipulate JSON if you want directly, or you can use what he has built with Mec- Mark Mexkenco, I think it’s how you say his name. I don’t know how to say it. it. Yeah. Yeah. He and another gentleman have been building this like Power BI CLI solution. And so interestingly, he went through and he tested the performance of,, how did you you if you ask it to build a visual or ask it to build a report, how good is these

6:04 it to build a report, how good is these different tools for using tokens, what model, which one performs the best? These are questions that I in general would like to see. Right. , right now I believe a lot of the challenges, we don’t really know which model does a good job doing what task. task. Mhm. Mhm. And what tooling or harness do you need to get that tool to correct outcome? You’re trying to do what we’re going to call like a min-maxing. You want to maximize the output, meaning getting exactly what you want, but you want to

6:34 exactly what you want, but you want to minimize the amount of tokens and the amount of cost it takes to run that solution. And so what Kurt does here in this post, I’ll put the post here inside the chat window, Kurt goes through and with different models across the same prompt, which model was the most performant and what tooling was the most efficient tooling to get the job done. I think this is incredibly interesting, Tommy, because I think we’re very close or we should be getting into the stage of optimization. Mhm. Mhm. Where, agents can do a lot of

7:05 Where, agents can do a lot of things, but if Tommy spends 2 hours and, 100, 000 tokens and Mike can do it in 30 minutes in 50, 000 tokens, Right. Right. or 2 hours and 20, 000 tokens, like there there’s a there’s value in making it more efficient because this is now costing us real money at this point. Anyways, I thought this article was really interesting. Kurt’s got a really cool infographic, which I’m sure he vibe coded with like a rave project or some real data that was behind it, so absolutely love that as well. but this is a really interesting read and

7:36 this is a really interesting read and article. article. And and Kurt, I know you may or may not be listening to this one as well. I wanted to challenge Kurt and say Kurt, how did you build the infographic? Was that built inside Raven or did you just vibe code it in HTML page? I’m get Based on what I know about Kurt, I’m guessing he just vibe coded it. Yeah. Yeah. Think about this, Tommy. Now when we collect data, because it’s so easy for us to get a vibe coded HTML animated page, you can you can now build a one-off report

8:08 you can now build a one-off report for your blog posts. Right. Right. And he shows you clicking through the thing., this is really pretty impressive. So one thing the output is great, but two, the data in this article is I think it’s also really important as well. And I’ll take a slightly different approach here to what Kurt’s been doing and I think what you’ve been doing, too. Mike, years ago, I think when we started seeing the onset of AI being part of workflows and really before it was part of business intelligence, I had this prediction around the business

8:38 this prediction around the business intelligence skills. And I’ve I’ve talked about these Venn diagrams before. Where we’ve had Power BI and design, Power BI and data engineering. And I’m like, “Well, there’s going to be a Power BI and AI Venn diagram of crossover skills.” skills.” Yep. Yep. What I I think I actually that prediction fell short was we’re now seeing the niches of AI skills with BI. You’re seeing what you’ve been doing with Raven. We’re seeing what Kurt’s doing around with visualizations. We’re seeing the governance and the agentic side for operations. And you’re seeing how this

9:10 operations. And you’re seeing how this all there’s the sub Venn diagrams here. Which I I did not see coming. And I think this is really cool that people are really finding not just hey, I do AI and business intelligence. Mhm. Mhm. That’s way too general to say. And now we’re actually seeing that there are actually almost departments in the organization here around you’re using business intelligence, you’re using a data stack. How are you using AI with it? Well, now there’s really categorical places where people can go to find a niche that they like.

9:45 I’m I’m People keep saying that AI is going to take our jobs. I rapidly see it shifting what we do. Yes. Yes. It’s taking some work off our plate, but it’s adding a whole bunch of net new work that no one knows what it’s about. And I think this is one of these cases of like, the internet came out and we’re like, “Oh, no. The internet’s going to take away all the printing press and no more writing.” Excuse [snorts] me. But here we are. We’re reading more than ever now. We’ve got a bunch of video and

10:15 ever now. We’ve got a bunch of video and the internet’s changed how we do everything. Like we’re just in this stage where things are just changing. It’s just different. It’s just going to shift the work we do into something new. Now everyone’s talking about agentic loops and how we’re not going to be writing prompts anymore and we’ll let the agents figure that out. All that stuff takes orchestration and setup and, infrastructure and and there we’re just shifting the work to something else to say, “Well, how do you build this loop thing? What Do you understand what a loop is?” loop is?” I love how you said we’re shifting our work. I I think to your point a little, if you could just continued what you’ve

10:45 if you could just continued what you’ve been doing, yeah, AI might actually take may take that. But the good news is what we’re seeing and I think just Anthropic’s article just confirms a lot of things we’ve been saying on this podcast somehow. somehow. The fact that what we have to do now is still needs that human input in that design, that strategic side. Now, are Again, are you going to be doing the same work you’re doing 5 years ago? Can AI do that? Yeah. However, to your point, there’s this whole other layer of

11:15 point, there’s this whole other layer of what AI can do to make honestly our jobs, dare I say, a little more valuable. So, Mike, I think it’s time to get into the article unless you want to say anything more encouraging. No, I just wanted to point I just kind No, I just wanted to point I just want to highlight that thing out of want to highlight that thing out right now. I haven’t done any videos

11:29 right now. I haven’t done any videos on it currently, but I’d like to do some videos in the future around using Kurt’s CLI Power BI CLI’s tool so we can compare. I think I think now that we have like three different ways of building apps and data and solutions, now it’s going to turn into well, which way is the most efficient? Which tool should I use to get the best results out of this? It’s Right. Right. It’s just a lot of more things that we need to see tests around. So, I’m actually happy that the community is starting to test things, really push into is this tool effective? Does it get the output value that I

12:00 Does it get the output value that I want? want? So, let’s dive in and Mike, I’m going to preface this article in our discussion today with there comes every once in a while Mike where we have an article or topic that really challenges, provokes and I think really excites me and this article is the first in a long time that to me is one of those groundbreaking ways that we need to think about what we do. Mhm. Mhm. A lot of it I think we’ve talked about, but I think for a lot of people who are

12:30 but I think for a lot of people who are in the data space, in the fabric space and they’re trying to navigate things, this is really a statement, a strong statement about what the future lies for us and where we need to focus. This is a powerful article and again, this is from Anthropic on their blog called how Anthropic enables self-service data analytics with Claude. What they were able to go through in the article is talking about Anthropic’s own journey to use Claude and to use AI to be able to help answer questions, to validate data.

13:02 help answer questions, to validate data. These are the big things we’ve talked about about how do you co-pilot chat? How do you actually get the answers you need? What has their teams built and set up? What are the best practices and what are the foundations, the layers that make it work? We’ve talked about some of these things. I think they have introduced a few other concepts here, but Mike, befo- I want to go through pieces of the article chronologically, but before we do that, you what’s your take overall from what Anthropic put out there and really I

13:34 Anthropic put out there and really I guess does it change your framework or your idea about our role, our job, or, the data space and its relationship to AI? No, it it doesn’t actually. I think in in a lot of ways, Tommy, this really affirms and solidifies a lot more of what we’ve been saying and I think this is also helping me align like this is the most forward agent thinking company in the world. They’re building

14:05 They’re building the agents that everyone is using these days and a lot of people are copying what they’re doing. And, to be honest, Anthropic is also copying a lot of other tools. Any good idea that comes out now out now Right. Right. open claw, mobile remote chat with your agent kind mobile remote chat with your agent thing. Any If you can dream it up, of thing. Any If you can dream it up, Tommy, Tommy, and you put it out there in the internet and some other company thinks it’s valuable, within days, weeks, other companies are copying that idea and incorporating their tool. So, the love the playing field for like what

14:35 love the playing field for like what people are doing is becoming very even, in my opinion. Almost every tool has the features that you need cuz they’re all and if they don’t have it, wait a week. It’ll get there. Right? but, if if you start reading through this article, one of the areas that they point out here at the beginning of the article is talking about data foundations. And one of the things they talk about is standard data engineering, data quality and practice such as dimension modeling. Yes. Yes. All at the very beginning, they’re saying you have to have good data. It

15:05 saying you have to have good data. It has to be understood by the business. You have to have good structure and doing a dimension model, dimension model is the way to go. They even highlight semantic modeling, yeah. yeah. Correct. Correct. And a lot of this article to me points to and that they’re not directly saying that they’re using AI to supplement looking at, go getting at, finding data, but there’s a lot of I would call maybe pillars, foundational pieces that they’re saying these foundational pieces need to be in place.

15:35 foundational pieces need to be in place. Mhm. Mhm. And those foundational pieces aren’t another skill for Claude, it’s the semantic model. It’s relationships between tables. It’s not having redundant data between two different tables. This is what we’ve been building for the last 15 years, Tommy. Yeah. And by the way, how happy do you think the Microsoft team is for changing the name from data model to semantic model that years ago? Remember that people had all that hoopla but like, oh, why they changing the name? Is it a data model now or

16:05 the name? Is it a data model now or semantic model? Because that’s what it was for most of us. They’re high-fiving each other now because they’re like, good. We use the right term. So, we’re right in it. It was the same thing. They just changed the name. the name. And yes, correct, Tommy. And I and I think even the even the semantic models of things is even a little bit maybe like not even fully in company of what a semantic model actually is. Cuz a semantic model is the semantics and the data that’s in it. Yes. Yeah, right, right, right. Right? It’s actually two parts brought together. And so, data model

16:35 brought together. And so, data model makes a bit more sense because it’s like semantics data model. Like it’s it’s like both, semantics and data all rolled together. and I think what Anthropic’s maybe talking about here a bit more is like they they’re just talking a bit more about the semantics model, right? Relationships, tables, column definitions, what does all this stuff mean? it it reduces ambiguity. And I think another thing that stood out to me in this article, Tommy, if the agent can’t figure it out, humans aren’t going to figure it out either. either. Right. Right. You’re already going to

17:05 Right. Right. You’re already going to have the pro- You already have the same problems. Yeah. Yes. Yes. So, I I think this is a good place to go into where Anthropic went through a few tips that theme out the article here. And I want to highlight these and then talk about their main point here about how the difference between data and software or the difference between building applications and then utilizing agents for data. But first, let’s go over what they really what their biggest insights were and I’m going to read these out. Sure. Sure. When analytics accuracy is a context No,

17:36 When analytics accuracy is a context No, why analytics accuracy is a context and a verification problem, not a code generation issue. What are the three failures that caused the most errors? The The agentic analytics stack was built that they addressed those errors. How to measure effectiveness and a basic template to create the majority of these skills. And this really starts with what they called data is really not software. Where coding is an open-ended solution that rewards the model’s creativity. You

18:07 that rewards the model’s creativity. You that rewards the model’s creativity., data has usually one answer and know, data has usually one answer and they broke it up into three different areas: context, generation, verification. And the difference between coding and analytics. For context, codebases are,, well maintained, read me signatures, docstrings, commit history. There’s a structured context before writes any code. The difference with analytics, Mike, is the database has millions of fields, many of them that are poorly documented. There may some notion of revenue in multiple places., we know this. We’ve talked about this for 5 years on

18:38 We’ve talked about this for 5 years on the podcast and 10 years before that. Even with the right field, two different teams can define the same metric differently. We’ve had podcasts on just that topic or just that sentence. Yeah. Yeah. So, and then when it comes to generation for coding, the time to actually write the code, almost any spec can be created. So, a plausible answer is usually a valid one. For generation for analytics, implementing analytics in solution is often trivial trivial. It’s only 5 to 10 lines of code. But usually there’s only one correct answer or one correct

19:09 one correct answer or one correct approach. You can do the same things with coding and finally verification for all code that is produced and compilers, there’s unit tests and they’re fast feedback. For coding, for analytics, there’s no unit test, no compiler, no ground truth. Short of defining preset expectations, baselining,, there’s no deterministic way to confirm the output is right. And that is what I think we’ve struggled with, what a lot of companies have struggled with trying to set up analytics with AI. They’re trying to

19:39 analytics with AI. They’re trying to take this coding application-based approach and realizing that you cannot do that. So, let’s start right there where there’s this concept and entity ambiguity when it comes to all the viable options. Mike, this has been, I think, the holy grail that people have talked about why AI has not worked for analytics so far. I’d agree in a lot of these areas and I think think [snorts] [snorts] what this points out to me, Tommy, is the data here, the the information

20:09 the data here, the the information that’s being presented, right? This is a lot less around using agents to go answer your questions. A lot of this article feels to me use agents to build frameworks on top of existing code software IP that you can run regularly, right? It doesn’t talk It definitely talks about the semantics model, but it doesn’t say have the agent go through and build its own semantic model and store it inside the memory of the agent while it’s doing the task. No, it’s it’s it’s alluding to

20:41 the task. No, it’s it’s it’s alluding to having semantics from people described in a way the agent can understand it. That’s really the That’s the important part here, right? So, there’s a There’s a lot of these things when it starts talking about data foundations and, you talking about data foundations and,, dimensional modeling and how that know, dimensional modeling and how that looks. looks. This is just reaffirming a lot of what I think we’ve been saying for weeks now, which is the agents are going to be really good at helping you build things and then you can go look and review and test them. And then, I also agree here with this statement around testing.

21:11 testing. It’s actually quite difficult to build out a test of data or build a a a test that when you run the DAX statement, it outputs the right numbers. Right. Right. And does it output, that’s that data quality issue is like a whole study field of study in and of in and of itself. itself. Mhm. Mhm. And so, we should be using agents to help us build a framework, a system so we can use data quality, but we need to be able to like

21:42 but we need to be able to like not ask the agent to give us the answer for the actual numbers. And this is a great part here because the three errors that they talked about that usually get inaccurate responses is again that concept entity ambiguity, what columns or fields to use, Yeah. Yeah. the staleness of data, and then the right information may be in the data model, but the data simply can’t find it because there’s so much to look for. And it’s funny, Mike, because one of the things that they mention is,, two different teams can have a different

22:12 two different teams can have a different definition of a metric. That’s part And it’s funny because Mike, literally at this point now, 10 years ago was my seminal moment with data business intelligence that I was building Power BI reports for marketing for leads, and sales had a completely different definition that the whole company had to stop what they were doing at that moment to figure out how do we now have three different dashboards with three different numbers for leads. How did this happen, and what do we do about this? And we’re this is not something that was solved in

22:43 this is not something that was solved in 20 2016. 20 2016. We are All companies are still dealing with this, and I want to get into data governance later on, but there’s a huge part here around to your point, the human involvement here and the role of

22:56 human involvement here and the role of the business, the role of teams owning their data. So, but before we get to that, I think just the three errors around the data staleness, retrieval failure in concept. Okay. Okay. This is nothing new. And I I think just really starting there, there’s nothing new here on whether using this for AI or I can say those same three things five years ago, data staleness, concept, and entity ambiguity, and retrieval failure that I think still dealt with

23:27 failure that I think still dealt with when teams are building reports. Where do you find the data? What is the correct definition? And is the data accurate? That’s literally the same thing, the same song and dance we’ve been doing for 10 years at this point. And And I think as we go into this, does this elevate, let me just ask you question here, does this really elevate now or give a little more importance to the business intelligence teams because it’s the same three things we’ve been crying for the last 10 years?

24:00 I think it not any more than it was before., you just keep doing what you’re doing. You just keep doing what you’re what you’re doing. It’s not going to It’s not It’s not emphasizing any more than it was previously. I I see the the value of the BI team BI team or being able to define the metrics or to find the tables or,, to your point, leads, right? Yeah. Yeah. There’s a broader conversation happening. Does your company allow Will your company allow for the threshold of

24:30 company allow for the threshold of having multiple teams have definite different definitions for the same name, right? Or can you say leads is a general term, it’s not specific enough for what each team needs? Right. Right. And typically, and again, I’m going to step on your toes here a bit, Tommy, based on like what I understand about multiple terms being the same word being used for multiple different definitions. The difference between the two different teams was one team was filtering out additional things where the other team

25:01 additional things where the other team was not or including more things than than it wasn’t. So, a lot of these terms, leads, is just additional filter context that’s changing how you you view that perspective of what a lead really is. And so, each team has a definition of what that is. I think the broader question is, does the organization is the organization okay with that? Can you provide separate names that here’s what a lead is, but for this team, this is a leads without XYZ

25:32 team, this is a leads without XYZ things. Leads with leads without or with this XYZ things for different teams. So, we have to be able to have language that we can say, “Well, when I calculate leads, it means this and when you calculate leads, it means that.” There’s there’s meaning behind what those are what those two definitions are doing. And I think that’s where Anthropic is trying to address some of these issues, which is [snorts] [snorts] by creating a proper data set, the agent can get a better example of if I want revenue by product,

26:02 I want revenue by product, you can put it you it can go find it. Now, I don’t think revenue by product is the question you want to go ask your agent. What I would rather see is like, “Okay, take all products and put revenue against them.” Like let’s see products by revenue. Generate Generate the visual that you want and then the person has to come in and be the test for that. Once you have that test, then you understand what is real and from there, you can then see then see or or build out like data quality and

26:33 or or build out like data quality and then the checking and so that way, you then the checking and so that way,, every time you refresh the data know, every time you refresh the data set, it becomes,, it’s not stale anymore. By refreshing it, you can then verify the numbers are not changing or changing the way you expect them to change over time. That’s the data quality a problem here as well. So, So, I’m not going to say it’s going to make it any more important, Tommy. I’m just going to say, it’s just means we’re just going to keep doing what we’re doing. I’m here. I lost you for a sec. Can you still hear me? I can hear you. Okay. Okay. So, So, back to you. Cool. So,

27:05 Cool. So, I feel like there’s a bit of a chicken or egg thing, Mike, because you’re speaking about the quality testing, right? But there’s this part about the ownership. Should that start beforehand? And there’s a part about data governance here. There’s a part about the ownership. When you actually are setting up these solutions and you’re building the stack, well, should you already have governance in place? And I think for me, I’m looking at this and as we get into the main kind

27:35 and as we get into the main kind and as we get into the main meat of the meat and potatoes of the of meat of the meat and potatoes of the article around this agentic analytic stack, stack, it’s going to be really hard for organizations to build out their agentic analytic solutions or really agentic things over their data. I think without already having some solid governance policies in place around the ownership and accountability. I do not see organizations being able to be successful around doing AI solutions or agents that are running over their data or chatbots

28:06 are running over their data or chatbots or answering questions until they actually have accomplished data governance. And that may be my first hot take today. But just from what I’m hearing you say about, okay, when we do the testing or when we have someone verifying a visual, I feel like you can’t do that until you actually have some governance in place. I There’s a There’s a diagram that comes to mind about this, Tommy, when you when I look at like Microsoft or Fabric or how things are going to be rolled out. They’re there’s different ways to roll out governance for things.

28:37 roll out governance for things. You can do governance in a very flat way, meaning work on all the governance first and then build, right? So, you can work out all your policies around who can have workspaces, what does the data set do? Do all the governance up front, and then once you have what you feel like is defined correctly, then you release Power BI and Fabric into the wild. Right? Right? Another approach would be is, well, we don’t really want to have our hands in all of the governance. We want to, you all of the governance. We want to,, let the business just

29:07 know, let the business just drive. Okay, so here’s the tool, business. Go figure out some things. And then you deal with issues as it comes. So, you basically do a lot more of the here, just go build things that add value to you, Mr. Company and business, cuz we know you’re doing it in Excel anyway. So, here here’s a tool. Go figure it out. And then as you go through, then you’re like, “Ah, shoot. We should have turned off paginated report We should have turned off publish to web. Let’s go do that now.” So, you govern as it happens or govern after the fact. Don’t recommend that approach. I don’t think that’s the way to go. to go. There’s also a hybrid method where you

29:37 There’s also a hybrid method where you do some governance, a little bit of governing, and then you open it up and and let the business build. And then you come back and say, “Let’s review that governance and build some more governance, and go back and let the business build.” So, I like this handshaking approach. I think this is part of the This is in the Power the Fabric adoption roadmap, which is describing how to do governance, how you want to roll this out with your your organization. I see the the two patterns I probably see the most are organizations that spend an inordinate amount of time up front governing and

30:08 amount of time up front governing and then rolling out, and then ones that are doing like a little bit of governance, a little bit of building, a little bit of governance, a little bit of building, and go back and forth. I feel like personally I like the approach of doing some level of governance, roll out some solutions, see what you get, get some feedback, then refine the governance model, keep going back and forth. So, I don’t think you To your point, Tommy, like do you do this without any governance? No. Do you have conversations while you’re building the system? Yes. And I really

30:38 building the system? Yes. And I really like this whole idea of like volleying back and forth between leaders, business teams, business analysts, and the technical side of things and doing a volley back and forth between the both of them because I think that’s really where most of the communication and stuff breaks down. It’s communicating between people, between teams, and does everyone align on the same definition of how data is being presented and built. And I think that’s so essential and I love that you said that because when you work on large projects, especially long

31:08 work on large projects, especially long data governance projects, one thing that you have to say as a big risk is the fact that data staleness. What we started working on when the project began around our data could be rapidly different by the time we try to finish. And I think that volley approach also allows you to be flexible and dynamic as your data changes. Again, this is one of the things that Anthropics talks about from an agentic point of view, but this is also something we’ve talked about for 10 years, which I think is a good lead way into the what this mean around this Anthropics agentic analytics stack, AAA.

31:41 Anthropics agentic analytics stack, AAA. Mhm. Mhm. There are four main areas here that I want to dive into you with and they’re going from the bottom up here around how do you actually build an agentic analytics stack? Data foundations, there are the canonical data sets or semantic models for the single sources of truth, the sources of truth that map questions to actually govern entities. There are skills, so there comes into play. Good thing we’ve talked about that over and over again. again. Yep. Yep. They have and then finally at the end there’s a there’s a validation framework that they have

32:12 validation framework that they have around how they actually evaluate and go through all the things that have been built. And this starts with the data foundations. And again, whoa, right off the beginning they talk about semantic modeling or dimensional modeling as they put it. They say both, but Mike, Mike, our ability to semantic model, our ability to map entities together and provide these core pieces of truth are now are now being not validated because they’ve already been validated, but just

32:43 already been validated, but just reconfirmed, re-emphasized on how important our job is number one whatever we do, whether you build applications in Rayfin or you’re building things using skills or CLI tools, or if you’re doing things in Copilot, Copilot, if you don’t have a good semantic model or you don’t have the that structured and truly planned out, you’re going to fail. You’re not going to see success in a business without this heart and soul of what Power BI and really at this

33:14 of what Power BI and really at this point what data is. So, data foundation is the most essential part to build up any agentic solution. Canonical data sets, enforcing standards, having making sure you have the right artifacts. And the last thing that they said is treat metadata as a first-class product. Yep. Yep. So, there’s a ton here, Misha. What were your takeaways from the data foundations layer of the stack?, I think they they nailed it right here. Their their first comment around their most common failure is the

33:45 around their most common failure is the agent can’t map some concept. Revenue to this particular product. You Revenue to this particular product., it can’t map the correct table, know, it can’t map the correct table, the column, the metric definition. there there was too many candidates for it to actually get the right answer of what the user wanted. Mhm. Mhm. So, their answer to this and then I I like how they even describe the fix that they found was better. The fix is fewer, more heavily governed logical models. Create a small set of single source of the truth data sets

34:15 single source of the truth data sets that are clearly owned. And I think this is one of the things I want to really point out here. Mhm. Mhm. Clearly owned. So, your to your point earlier, Tommy, around defining different metrics across different

34:23 different metrics across different teams, right? That is an alignment of who should own the metrics. Who defines what a lead is. Which team is using that? And then once the definition is created, other teams need to adopt and use that definition through their analysis. That way when I say leads and you say leads, if we’re going to talk if our two teams are going to talk leads together, we have to at least agree on the definition of what a lead is. So, I think this is very important and one of my

34:53 important and one of my projects I did back in the past, which was master data management, it was really difficult to get people that own different master data elements of products. products. This team is finance owns this and sales owns this and engineering owns this. One product you build has many different teams that need to have input and value against that item. And so, yeah, I I thought that was really relevant., it also said on that real quick because ahead. ahead. you mentioned the small set of data of the semantic models, a smaller set of

35:25 the semantic models, a smaller set of many semantic models. I wonder if this changes how we think about building because this has always been a we’ve got a mailboxes about this and this is why I wanted to interrupt you here and ask you, people have asked, do I build a large semantic model for many reports, the gold model thin reports? Does this fly in the face of this a little or does this cause a little conflict here because again, we had a we had an episode, I remember the number about do you build for agents or do you build for humans around semantic modeling?

35:56 around semantic modeling? This would be in favor of I’m going to build for models or LLMs where rather than building a giant semantic model that can build all my reports off for marketing, that the course of action should be much more specific, targeted semantic models that work better with LLMs that I can still build reports off of. Do you So, do you see what I’m saying here? Yes, I do. I would argue the same effect you get out of an agent looking at a model

36:27 model and the same effect as you’re getting users to understand which model to use is incredibly important. the the the number of projects I’ve been on where everyone’s like, “Let’s just build the one monolithic semantic model.” Not really what we want., a lot of times we’re talking more about domain-based modeling. Build a domain around something. So,, this is the sales model. This is the operation model. model. Yes, some of that dupli- the data may be duplicated across those two different domains, but

36:58 domains, but by defining what is in that model, you almost have a pared-down view of this. Now, Now, I’m I’m blurring the line here, Tommy, a little bit between like what’s happening in the technical space of like what we actually have today in tooling from Power BI and Fabric versus what we actually want. Okay? So, I’m going to going to, , Our favorite thing, yeah, the future. Yeah. There’s There’s a dream of things. and yeah, dreams and wishes. I’ve said this on the podcast multiple

37:29 I’ve said this on the podcast multiple times, Tommy. There is There is the concept of the enterprise semantic model. And I’m going to keep throwing this idea out there cuz I think this is really important. Like, if you think about these data domains that I’m talking about here, data domain for sales, data domain for ops, data domain for for, whatever the fulfillment, whatever the whatever the different domains of the business are doing, that data exists as part of the whole enterprise solution around all the tables, all the data, and everything’s going in here. Another point they made in the article here, which I thought was

37:59 in the article here, which I thought was very relevant, is they said, “Having a single source of truth of data sets that is clearly owned is important, and then aggressively deprecating any of the near duplicates, having physical roll-ups and caches reduce costs.” So, grouping or aggregating, or in our world we would say, having many fact tables in a model, and then some of those fact tables being reduced in granularity increases performance, making it cheaper and easier to run, which is a good thing.

38:30 easier to run, which is a good thing. So, you’re balancing that reduced granularity of data in order to get to the answers that you want, but still doing it in a cost-effective way. So, these are all like domain-specific things and cost-balancing optimizations, but at the highest level, there still is the enterprise data set. All the domains, all the tables, there is some related information across all of these elements across the entire stack. And so, when I when I look at this going, this is what we want. And right now,

39:02 this is what we want. And right now, Fabric doesn’t really support the full enterprise semantic model. We we don’t have that level of support. And so, what we what we wind up doing is we wind up breaking these large monolithic semantic models down into domain-specific things. And there’s an underutilized feature, Tommy, that we have here that we don’t use in in semantic models that we could I I wrote it down. Say it. It’s perspectives. I wrote it down. I wrote it down. I It’s like a guessing game.

39:32 It’s like a guessing game. So,, So,, we can have a domain model that has different views of what data and tables are in it. And this is what metrics view was a while ago, but it it didn’t quite get implemented correctly. So, perspectives are incredibly important because I have this,, you could think of a domain a data domain as actually being a perspective of the larger enterprise model. So, you you and when I go back to this anthropic thing, which is there’s this concept of like, okay, I have the raw fact table with all the

40:03 have the raw fact table with all the data in it, non-aggregated in any way. That’s the main table. But, you can use an aggregated form of that table to then build out a page, a report, or be more proficient. But, again, that aggregation table is just a shadow of the original data set, the the original factual table. And so the the question or comment here is how do you stitch all these different kind do you stitch all these different elements? Like how do we how do we of elements? Like how do we how do we manage the entire enterprise semantics? How do we enter how do we manage the

40:33 How do we enter how do we manage the domain specific semantics? And when a user comes in, they don’t probably need the entire model of the enterprise. They’re probably looking for questions specific to their area and data that is specific to just them. So this is where I think it gets really interesting and this is Yeah. Yeah. all the all the traditional practices of Kimball and data warehousing, that’s what we’re talking about. None of this is changing. A lot of people think, oh, we bring agents, we bring Power BI in here, we have to rethink how we did all of our data warehousing.

41:03 how we did all of our data warehousing. No, it’s not. It’s actually very much the same. It’s just done with a different layer of technology. And it’s it’s hilarious that you said this and I love that you said this because when you really read that canonical the canonical data sets paragraph or section, Mhm. Mhm. your mind goes to, well, hold on a sec. A too big of a semantic model, the one a semantic model many reports. For an agent, that’s probably not the best way to use it because it has too much context. context. Yes. Yes. So that the gold model thin reports is

41:34 So that the gold model thin reports is good for humans, bad for agents. But then on the other side of the token, if I’m building out separate semantic models one-to-one now, every semantic model only has a single report, good for agents but bad for humans building and sources of truth. So you introduced what I wrote down and this is why we’ve done 539 episodes together among other reasons, but perspectives for agents. That’s that second layer here where for us to scale and to continue enterprise semantic modeling, which is essential. We cannot

42:06 modeling, which is essential. We cannot go to one-to-one because agents demand it because we’re not going to be able to do our job correctly. So, there’s that second I think there’s this added layer where we need agents to look over the perspectives, not just over the semantic models. So, that allows us to have a single semantic model model perspectives, which are going to be more I guess one perspective one report or one to two reports. Because again, it answers specific and targeted questions, which is also where the agent would look.

42:36 look. I think again, use the fluffy music again because we don’t really have that ability today. But Microsoft, if you’re listening, that’s the dream that needs to become a reality. Yeah. What Anthropic’s introducing. And then I want to touch on So, I love that you said that because I am completely aligned with with you. I want to touch on that treat metadata as a first-class product here. And it talks about coding agents perform well partly because code bases are legible. Readmes,

43:08 because code bases are legible. Readmes, doc strings. Your warehouse can be just as legible, but only if column and table descriptions, canonical metric definitions, grain documentation, lineage and ownership, and model tearing are maintained with the same rigor as the transformations themselves. While it’s not necessarily new insight, good governance provides critical context. And Mike, I am going to toot my own horn here and call something I mentioned years ago when automated insights came out in Power BI. If you recall, there’s a

43:39 Power BI. If you recall, there’s a feature that would look over a page or context and provide insights based on anomalies, trends, or KPIs and wasn’t very helpful. One thing I said on this podcast was we needed the ability to wait and to guide those insights, which we did not have. It would look over the entire model, but it did not provide in a sense of waiting towards specific tables, specific columns. There was no way to do that. And that section here, the treat metadata is a first-class product.

44:09 product. Is that what I was saying? So, I don’t normally do this. I don’t normally say I was right about something because honestly it’s few and far between, but in this case I think this also goes back into they say the treat it as a first-class product. It needs the same rigor. This is the most overlooked because it’s not the most attractive., it’s it’s the one that you don’t necessarily see all the flowers from doing the work. But if you want the agents to work, just like if you have that perspective layer you need to provide the right documentation. You need to write provide

44:39 documentation. You need to write provide the right guidance and waiting. Say, “Hey, if someone says revenue or members give more weight to the,, customer table over the,, location table, whatever the case may be.” be.” You have to guide it that way. This is something that we’ve been saying anyways. And I think this is another part of that data foundation is so important. Some something you’re bringing up here, Tommy, that is really interesting that you you made the comment and I comes to light of mind here. And and again, I’m going to move down here to the next Yeah, yeah.

45:09 Yeah, yeah. area of sources of truth here pretty soon. soon. yeah. Let’s do that. But so something that comes to mind when you when you said that comment, Tommy, which was which was have you ever have you seen I I think Rock was doing this for a bit. other agents may be doing this something similar. There’s just concept of we don’t really So, if the agent is going to pull things out of the model that is non-deterministic, it doesn’t have a consistent [snorts] way of getting the information out of whatever that model looking looking at

45:40 whatever that model looking looking at that semantics model layer. Right. Right. One of the things I’ve been saying for a while that I think would be incredibly useful here, which is I don’t want the agent to give me one answer.

45:51 agent to give me one answer. I feel like agents are used to help me create things. So, if I So, if I’m asking an agent to build me a bar chart with something on it Mhm. Mhm. or I’m looking for an analysis of products over time or I’m looking for I want the agent to build me like six different versions of the same visual or the same item that I can go use. I think this is actually more valuable because the the cost of the agent to build the solution is actually really simple and easy. Right. Right. So why not push the agent to build more of the same

46:22 push the agent to build more of the same solutions? Mhm. Mhm. And and And and one of the things that I that I see a lot on these agent pieces are agents do that a lot of this, which answer did you like? like? Give me a thumbs up or a thumbs down on the answer or response. That’s a very low signal back to the agent or the developers or the harness to say if it was a good answer or not. or not. And And what I saw recently was agents were now I asked it a question

46:52 agents were now I asked it a question and it gave me two answers, A and B. Which answer did you like more? And you pick one of the two answers. Mhm. Mhm. Already you’re forcing the user to make a judgment call around what they wanted and help train the next wave of AI to get a better output or result. And I think there’s something in this that is that is greatly missing in a lot of the Microsoft tooling, which is instead of trying to give me the one

47:23 instead of trying to give me the one right answer, the one right visual, the one built solution. Cuz again, that’s what the ultimate goal is. Instead, AI agents should be used to build seven or eight or 10 different visuals of the same thing. Hey, you asked me this question. Here’s six different ways to think about this information. I used this table, I built it as a line chart, I built it as a bar chart, I built it as this this this. Here’s some text output. Which one of these things did you look at that you like? Right. Right. And what this does is it immediately

47:53 And what this does is it immediately informs the agent and it gives feedback from the user to say which of the solutions should I be presenting? And so the more you do this, the more you actually have real feedback. Okay, I had this input, I had these five outputs. Of the five outputs, this is the best one. And I think that’s information that needs to be generated and brought back to to models, agents, semantic model layers, so that the users can actually get value from it. And this is again, when I use data agents, when I use other chatbots

48:26 data agents, when I use other chatbots or other systems, none of them seem to give you multiple answers to the same question. question. Okay, and this is interesting you say that because I feel like you’re talking a little bit here about the query corpus that Anthropic is talking about. It’s interesting when we’re talking about the sources of truth. Yeah, yeah. Yes and no, because Yeah, give me a little more here. So Well, you’re you’re on the right track there because the query corpus is like is basically every query that’s been run against the model for all time,, , getting a bunch of different queries

48:56 getting a bunch of different queries against the models or or,, solutions. But in here they say, well, we have to understand which of the questions which of the questions resulted with the right SQL statement or output, which one actually provide the right output, right? Which one gave me the right answer. And I think there’s some some inherent inherent feedback loops that are presenting themselves that that we may or may not be using, right? So, for example, Tommy, if a if an agent

49:26 for example, Tommy, if a if an agent gives me the right answer the first time or I work through it with an answer, at the end of that answer, I stop and say, reflect on what you did, update the scale if I was using a scale, right? There’s there’s a sequence of things that I do at the end of that that session session that I complete. I I tie off the conversation, move on. If I if I That’s that’s a signal back to the agent that said that worked. That was the answer. I should not I should never have to click thumbs up or thumbs down. down. Also,

49:57 Also, Wait, are you saying Wait, you’re saying you should not have a yes or no on things? things? No, the So, I think my behavior of how I interact with the output of the agent immediately tells the agent whether or not it was a good answer or not because Tommy,, you’ve done this a number of times. You ask the agent to do something, it did something, and then you say, “No, I didn’t want that.” And then you give it a whole bunch of set of new instructions to do something. Immediately, that’s an that’s a signal that said, “Whatever the output you gave me was wrong.” Right. Right. Right? And so, my other idea here of

50:27 Right? And so, my other idea here of like having multiple things in front of you, having the agent create things, like and I think of it is, the the lousy example I have in my mind here is if I look at PowerPoint, PowerPoint, you can have you can have general slides, and you can go to the right-hand side and say, “Suggest for me templates.” And so, from those suggested templates, there’s like eight templates there, and you just pick the one that you like the best for that particular slide. It may not get it right on the first time. It may be three different versions of, the image on the left and the words on the right. I don’t like that. I want the words on the left

50:57 like that. I want the words on the left and the image on the right. Like, it just gives you different permutations. Well, the fact that I’m opening that window up and presented with those multiple options, when I select the option, I’m automatically giving feedback feedback to say that’s the best option for how I want to present my data. So, that’s what the agent should be doing. The agent should be producing many of the same things and saying, “You asked this question.” And this is I again, this is also what I would say for data agents, this should be implemented immediately. Here’s my data agent. Ask it a question.

51:27 Here’s my data agent. Ask it a question. Great, I found an answer. Here’s the SQL that I ran. Here’s the table I produced. Here’s four or six visuals that you can pick from that would represent this data. Great, you select the visual you want, which one resonates with your story, you pick the visual. Automatically, by picking the visual and giving me some slight choices, it’s informing the agent around what what actually added value from the system. Right? Does that make sense? sense? yeah, it does. But if I’m hearing you right, you’re Are you talking about the building of

51:58 Are you talking about the building of this or the actual production of this? Because I’m going to actually disagree with you if it’s the latter. If I actually deploy this with all these options and choices every time a user prompts or puts a question about their numbers, numbers, one, there could be a it could be an erroneous feedback loop because the users may not know what they’re looking for and then there’s providing input to the agent. And then two, I think you’re giving too many options to the users. What you mentioned to me is part of the

52:28 What you mentioned to me is part of the building and the development of an agent to make sure that when a user asks specific questions, they get a specific answer. answer. And I think this is so essential because again, let’s go back to the first part around coding and analytics. There’s only one answer if I want to say, “What are my members in Q2?” Yes, but that’s not something the agent should know. Like I I I think the the core difference that we’re talking about here, Tommy, is I don’t want an agent to return that result to me. I want the agent to build the report that has that answer on it or

53:00 the report that has that answer on it or go find that answer from a whole library of reports that has that answer on it already. Like that’s that’s what I want the agent to go do. I actually don’t want the agent to go through and say, “I’ve wrote the SQL query. I got this table together. Here’s a one-off solution for you.” Because now it happens every time I want to ask that question, the agent has to like rethink that whole thing again and reason through it again. And we all know like it reasons through it now, it may give me a different answer in 6 months when we got a better model, I get better stuff. Like the answer may be totally different. I don’t like any of that. I

53:31 different. I don’t like any of that. I want the agent to build me the consistent solution that says here’s what we here’s what we’re doing, here’s the options we have. I want to reason with the agent to tell me these things. Also, Tommy, I want to go into a visual and say like I want to chat to that where did this data come from? The agent should be able to say I’m going to read the metadata of this visual, visual, find the columns used and the measures used in here. Here’s the columns and measures used in this visual and now I will go trace back through the system

54:03 will go trace back through the system and say from this visual to the semantic model, from the semantic model to these columns, from these columns to these data definitions and metadata. Like there’s a lot of metadata that’s probably existing that we’re not surfacing in the report the visual per se. se. And we should be able to ask the agent to explain the lineage of thing. And this is actually one of the items they they called out here which was lineage and transformation graph. When the semantic layer doesn’t cover a question, the lineage and table ranking based on a number of references, the agent can then

54:33 number of references, the agent can then reason about upstream models and give you more context. I think that instills trust in users. And so when I look at really large semantic models, that’s the challenge users have. The challenge users have is I have these massive semantic models. I don’t know where to start. I don’t know what columns to pick from. I can’t get my output that I want. It can becomes overwhelming. And in the same way, I think the agent gets overwhelmed as well. It does a better job of like maybe reasoning through and it doesn’t like get overwhelmed in the same way humans humans do. But we get

55:05 same way humans humans do. But we get frustrated. And because it’s just too much there. And they’re not going to keep users are not going to keep going back to try to get the right answer especially if they don’t know how it works, right? you get one or two shots at this. If you don’t get it right the first time, the second time, they’re going to like maybe try again that second time. And then after that, if they can’t get the answer they want after the second time, they’re like this thing’s junk. I’m not going to use it anymore. And then you walk away from the solution. solution. I So it’s funny. I’m reading a book called the greatest story ever sold and it’s about the internet bubble. Okay. Yeah. And as we have this discussion today,

55:36 And as we have this discussion today, I’m seeing so many similarities or crossovers to the problems that I think we’re seeing in the agentic side, Mike. But also too, very much on what you just mentioned. Where or or something we’ve talked about too, even though the barrier to entry around websites back in the day was so low, like for e-commerce, right? It was incredibly low. All you needed you could have a little warehouse like amazon. com did, which was the original name, and a website, and you could have an e-commerce site. However, users were not

56:08 e-commerce site. However, users were not willing to go like you insert a CD and then you can easily take it out, put a new one in. Every website had its own technical skill or challenges that the user had to navigate. So, even if I could build the website and have the e-commerce site ready to go, are users going to be willing to go from Amazon to whatever the next one is, where I have to put my account information in, I have to navigate this new website user interface, and to your point, most people are not willing to do that. So, if you have something where they have to ask the question multiple

56:39 they have to ask the question multiple times in a new co-pilot and a new agent every single time, and the interface and experience is different, they’re not going to do that. And this is so important on the in terms of how you build it, and I think just to your point where you have to have the very structured output. Also, the trust for users, well hey, we got this answer querying, you hey, we got this answer querying,, X, Y, and Z. This is meant to one know, X, Y, and Z. This is meant to one on this report. For more reference, go here. Because you have to have that trust, too. Users are not going to go, I think that number is right. Let me ask it 18 more

57:09 number is right. Let me ask it 18 more times. They’re going to move on, they’re going to go back to the report, they’re not going to utilize that. So, I love that point here, and just on that lineage transformation graph, I think

57:20 lineage transformation graph, I think, the I don’t know the metric goes into, I know where to actually get this aggregation from. one thing that I know we’re getting close in time, there are a bunch of other things in here, Mike, that I think we need to touch on another day. but I want to end here at least with the sources of truth, for me at least. at least. Mhm. Mhm. Around this business context. And I would like to read this out because I think this could be a whole series that we do. And the reason why is one, it’s

57:50 we do. And the reason why is one, it’s incredibly powerful, but hey, what? Let’s give each other flowers here. This is something that you and I have done series around, have con- constantly talked about. And this again goes into the sources of truth. And one of the things that they’ve seen in descending order of trust. Business context. The layer most teams skip. And then again, this is Anthropic speaking here. And the one we underrated the longest. An agent that doesn’t understand your business will answer what the user ask, not what they meant.

58:22 what the user ask, not what they meant. It won’t know that Q2 launch refers to a specific product, that two teams define the same term differently, or that a question is being asked because a board meeting is on Thursday. We pipe in company knowledge graphs consisting of index documentation, roadmaps, decision logs, and our organizational structure, so the agent can resolve ambient references and ask better clarifying questions. So there’s a back and forth there. Yep. Yep. The common failure pattern across all four is the same one for data foundations layer. Poor or stale documentation in bold.

58:54 Poor or stale documentation in bold. Claude is useful for closing the gap, but the curation of ownership are managed by humans. Mhm. Mhm. So this, Mike, is absolutely essential when we talk about that we generate documentation Claude, but the own humans have the definition. Mike. This is talking about here that again, the data alone, the semantic model alone, is not going to solve your problem. Even if you have the bar chart or ask the questions here, how do you actually incorporate the business

59:25 actually incorporate the business context? How do you incorporate the culture into an agent? And this is again people don’t find this attractive because it doesn’t generate revenue and I can’t see it. It’s not a viable or tangible thing people hold as a deliverable. But, you’re setting yourself up for failure to me without emphasizing this or adding this layer in. layer in. Yeah, I’m going to keep leaning on this whole idea of like the agents are there to create. They’re there to help you build. They’re here to help you build the report page. The like the business

59:55 the report page. The like the business user context of what I’m trying to ask for, what this report page means. You agents should be allowing us, Tommy, to go from go from, , three domains, , let me say this way. Agents should be allowing us to go from 20 domains reasoning about what we can do to get down to six domains. Simplify, right? So, the agents should be helping us simplify. Agents should also be helping us figuring out how to do reduce grain on tables to become more efficient, right? But yet, still meaning the same thing, right?

60:25 the same thing, right? Right. Right. That’s another area that would make a lot of sense here. all of this business context and knowledge needs to be stuck in a way that the agents can understand it. And I And again, I’m going to keep going back to to I would rather have the agents building 10 more reports on those those domains that answer very specific questions, right? Rather than rather than having a bunch of agents always asking and trying to reason through the data and potentially getting to the same or not to the same

60:55 getting to the same or not to the same answers at some point. Because those 10 reports can be vetted. You can test them. They’re very deterministic in their output. And I think one thing that’s potentially missing here, Tommy, as you as you mentioned that thing, we don’t have a really good way of adding We have the ability of adding descriptions to reports, but not reports and pages. Like you think about Yeah. Yeah. If you step back from the report layer and say and say if this report has five pages in it, like, does that report have a context

61:25 like, does that report have a context description? What’s the metadata of what’s in this report? Page one is an overview of these kinds of things. It answers these kinds of questions. Bing, bing, bing, bing, bing. Page two is details of this and has these kinds of filters. It allows you to filter based on this, this, and this. It’s designed for digging deep around this type of question. Right? We’re looking for where sales have gone. Which sale What’s our pipeline look like? Right. Right. How healthy is our pipeline? That’s what this page is designed to do.

61:55 this page is designed to do. These are the the business logic questions, reasoning is is at a higher level and we’re not really incorporating that directly into reports. So, somewhere in this is where we have to start giving that that information. And I think that actually leads into our next area, which we’ll probably have to talk about again later, is skills. The skills they talk about is the the the accuracy of what they can return from data from data is very low without the use of skills. And when you incorporate skills around

62:26 And when you incorporate skills around that additional business context, that’s when the skills working with the model and the data and AI agents, that’s when you get really high results. So, they were go they’re touting their tests, their evaluations, when they just sit through an agent at something, it was doing like 21% success. success. But, as soon as they added skills, the numbers changed from 21% to 95 up to 100% on certain domains of information. Wow. Like, that’s a huge improvement. So, the addition of skills again, skills

62:57 So, the addition of skills again, skills are those business logic elements. When you do this, use this. When you do that, use this. use this. That’s the business context that’s being added there. So, yeah, that This makes sense. Tommy, if I just gave you a semantic model with no business context, you could dig through some things, but even even if you were just a genius and could figure out how to make reports and something that looked good, what I expect out of that data set and what you find in it may be two totally different things. And and this is what the skills is aligning to. The skills is shaping and giving that general context

63:29 shaping and giving that general context for both of us so we understand what does this table mean, how is it being aggregated, how did it get here, and what questions we can answer about that data. So, I think this is just emphasizing more of what we should have been doing all along. Well, honestly, Mike, are we psychics? Because I want you to go back to I think we can find it on now that you have an agent that built it. When is the first time we talked about agent skills on the Fabric Power BI podcast, right? Because obviously, you have another podcast around Agentic Solutions, so I imagine skills come up, but you and I

63:59 imagine skills come up, but you and I have been early on touting skills here, and now again, that gets reinforced. before we go because I like I think I agree with you. I think that’s going to be a part two here because I want to make sure it gets the, time it deserves. One thing I want to ask you as we close out here around that business context here, here, I’ve introduced the concept before around our job around soft data and hard data. Hard data is a relational modeling, the things in tabular format,

64:30 modeling, the things in tabular format, but then we have all this soft data in SharePoint and Excel. they talk about,, the Slack meetings, PowerPoints, decision logs, the organizational structure. It’s not in necessarily in Fabric, so to speak, in terms of it’s in a semantic model, but that is still is data. And I think I’m going to put this in our topic board, but I think do we need to spend more time around the soft data in business intelligence now? Around things that are not tabular, things that are

65:00 that are not tabular, things that are not in a semantic model, but yet are going to provide a better enhanced Agentic Solutions. We’re seeing this with Copilot already. I think Claude is emphasizing this. So, I think this is another consideration we need to have for the future. But, do you have any thoughts there or is that for another day? No, I don’t have any other thoughts on that one, but I did I did find our one of our earlier episodes episode 396 is one of the episodes that we have

65:30 is one of the episodes that we have fairly long ago. We start talking a little bit about AI skills. Actually, that episode was very controversial. It was a was a C# scripting in Tabular Editor versus Tim View fight. So, so Tabular Editor C# scripting, which was like one of our early early ones. So, there’s a couple references here to some AI skills that we talked about there. there were some enhanced Microsoft Fabric Copilot and AI skill previews happening around that time. So, some news and announcements there. Anyways, that was interesting. Also,

66:01 Anyways, that was interesting. Also, randomly side side thought here. I went to the to the Power BI Tips website. So, if you want to go search what we said in the past. If you want to that actually How did we find this so quickly? I I don’t have an agent. Sadly, I do not. You can go to powerbi. tips. com. We have a search button. not every episode, but more of I think the last from 300 and forward, I think we have every episode transcribed fully on the website where you can actually go to the website and search specifically words or key phrases and it will talk you bring you

66:32 phrases and it will talk you bring you directly to the point or topic around what we talked about. So, incredibly useful if you want like background or information to back up what you’re looking for. These are things that we’ve talked about. We’ve unpacked it and we we put information around it. Also, side note, when I clicked control F on the website page and search for AI skills, it was really interesting. Copilot wasn’t available and when I searched for it, as I skimmed through it, there was a Copilot option that appeared underneath the finder window on the webpage in the

67:04 the finder window on the webpage in the Edge and said, “Would you like Copilot to find on this website page where the word AI skill was used?” I thought that was really interesting because I have never seen that before. And so now Copilot is even in your search bar on a webpage that will help you summarize things. So anyways, really interesting stuff there. So with all that being said, great topic today. This article is a must-read, and you it’s just one you probably need to read two or three times to really unpack everything that’s in there. It’s very

67:34 everything that’s in there. It’s very dense. So the article’s in the description of this video if you want to check it out. It’s also here in the chat window as well. Make sure you go read this article. I believe this article is extremely accurate, very useful, and I think it’s going to help highlight areas in your business that are weak around analytics or AI or how you’re using AI with your analytics. So very, very useful, super good article. That being said, Tommy, where else can you find the podcast? See if I still remember this. You can find us in Apple, Spotify, wherever you

68:05 find us in Apple, Spotify, wherever you get your podcast. Make sure to subscribe and leave a rating. Helps us out a ton. Do you have a question, idea, or topic that you want us to talk about a future episode, maybe around what we’re talking about today? Head over to powerbi. tips/podcast, leave your name, enter a great question, and finally join us live every Tuesday and Thursday a. m. Central on all Power BI Tips social media channels. Thank you all so much, and we’ll see you next time. Explicit Explicit Measures, pump it up PII. Tommy and Mike got it up the sky. Hands

68:35 Tommy and Mike got it up the sky. Hands to the data last in the mix. Fabric and AI get your feels. Explicit Measures, Measures, drop the beat now. Power BI is king, feel the crowd. Explicit Measures,

Thank You

Want to catch us live? Join every Tuesday and Thursday at 7:30 AM Central on YouTube and LinkedIn.

Got a question? Head to powerbi.tips/empodcast and submit your topic ideas.

Listen on Spotify, Apple Podcasts, or wherever you get your podcasts.