Modeling Without the Keyboard – Ep. 474

Can you model a semantic model without touching the keyboard? Mike and Tommy explore AI-assisted modeling workflows using TMDL, MCP, and emerging AI tools—and discuss the practical limits, including Chroma’s research on “context rot” where AI assistants degrade over long interactions.

News & Announcements

• Creator Improvements in the Data Agent — Microsoft improves the Data Agent creation experience, making it easier for BI developers to configure and customize AI-powered data interactions.

• Power BI Semantic Models as Accelerators for AI-Enabled Consumption — Microsoft positions semantic models as the key accelerator for AI consumption patterns, reinforcing the trend that well-built models are the foundation for everything downstream.

• Prepare for the New Card Visual — Heads up: the legacy card visual is being replaced. Teams need to prepare for migration to the new card visual experience.

Main Discussion: AI-Assisted Modeling

Context Rot: When AI Loses the Plot

Mike and Tommy discuss Chroma’s research on “context rot”—the phenomenon where AI assistants become less effective as conversations get longer. The model starts losing track of earlier context, contradicting itself, or generating lower-quality output. Key implications for AI-assisted modeling:

• Long modeling sessions need periodic “context resets”
• Breaking work into focused, shorter interactions produces better results
• AI works best as a sprinter, not a marathon runner

TMDL as the AI-Friendly Format

TMDL (Tabular Model Definition Language) is inherently AI-friendly:

• Text-based and human-readable
• Each measure, table, and relationship is a discrete file
• AI can read, write, and diff TMDL naturally
• Combined with Tabular Editor, it creates a workflow where AI generates TMDL and humans review

MCP and Code Execution

Anthropic’s MCP code execution research points toward a future where AI doesn’t just suggest model changes—it executes them. Combined with TMDL:

• AI reads existing model definitions
• Proposes changes (new measures, relationships, formatting)
• Executes the changes directly via tool use
• Human reviews the diff before deployment

The Practical Gap

Today’s reality is still messy:

• AI generates plausible but not always correct DAX
• Context rot means long sessions produce worse output
• Validation and testing still require human expertise
• The “keyboard-free” vision is directional, not current

Looking Forward

The convergence of TMDL + MCP + AI assistants is creating a new modeling workflow where the developer becomes more of a reviewer than a writer. But the tools aren’t there yet for full autonomy—context rot, DAX complexity, and validation requirements keep humans firmly in the loop.

Episode Transcript

Full verbatim transcript — click any timestamp to jump to that moment:

0:00 Good morning and welcome back to the explicit explicit measures podcast. Whoa, almost slipped up there. , welcome back to the the podcast with

0:31 Tommy and Mike. Welcome back, everyone. Good morning, Mike. How you doing? I’m doing well, Tommy. It’s been a couple long days here already, but it’s this Can you believe it’s like fall already? It’s getting cold outside. Things are just moving by quickly, man. Time is just zipping away for me. Oh, yeah. , walking to the girls to the bus stop this morning, we were at 32 and you felt it. Oh, it felt good. Yes. True. Tommy, you lived in Florida for a bit. , you’ve told me in the past that you do not like the heat. You like [laughter]

1:04 Cooler weather. So, , my wife’s in and my family lives in Florida, too. And every time we go down, I’m always telling my wife like, “How did I do this? I can’t believe this.” She’s like, “You barely did because you complain every day. Every day. Every day thing.” So, it was annoying. Like, yeah. So, no, I bring on Listen, this is how I hope I go, Mike. I hope I’m 85 shoveling snow in my driveway and I want to croak heart attack right into the snow. Right into the snow. Just that’s how I want to go. Preserve me as this frozen popsicle. All right. let’s let’s move on. let’s

1:39 Get into some u of our main topic today. Today we’re going to talk about modeling without the keyboard. So this is going to be a conversation around AI. There’s a lot of developments coming right now around where does AI fit? How are we going to use it with data? what does this look like for building and creating models and semantic layers for people to consume. So I think this will be a really good discussion to unpack there as well. Before we get into that, we do have some news items. Tommy, walk us through some news. What do you have here? First one for us today. All right, so the first thing we’ve got is something around data agents and we got some updates. Some of these have

2:13 Been out if you’ve been using data agents a lot, but they’ve made some creator improvements in the data agent. Some of the main highlights here is some debugging tools. So simply allowing to look at reference example queries which they would show up previously when the query was running. Yeah. But the problem was once the data agent was done with that particular question, you couldn’t see what the query was like what the DAX statement was or the SQL statement was. Sure.

2:44 So annoying, but now we can see those reference queries. Yeah. trying to make it more agent-based novel idea novel approach to something with agent in the name. So actually allowing you to see and change the the context of the agent as well. So multitasking flow for easier creation. So context switching switching is made pretty much simpler. And finally finally markdowns supported in the instructions. So, Mike, this is something that when you’re looking at

3:18 Just the landscape today, , everyone’s everyone’s already doing this one already. Like, we’re catch we’re finally catching up to whatever all the other instructions for agents are doing other places. Have you heard of something relatively new? It’s called Claude Skills. If you’ve ever used claude code or claude, I’ve heard of the term. I’ve seen something AC come across my feed a number of times around claude skills. I’m not exactly sure. I have not used it. So, I got to be very clear. I have not used it, but I’ve heard about it and it seemed interesting. Tell me more about what Claude skills are. Yeah, and we’re probably going to touch

3:49 On this on the main episode, but the simple idea is Claude can basically run an environment on it on its own, like a Linux environment or whatever. You don’t have to know that, but you can actually upload your own skill. For example, some of the skills that are pre-built into Claude is being able to build PowerPoint slides. And it’s a folder that has in it something called skill.md. It’s markdown which is all the instructions that I can do with reference to any scripts in that folder. So you’re basically extending what the AI can do. So let’s say you created one

4:21 For Tim do you created a Tim DoD skill which is actually a great idea and you like hey Claude go through this and check X Y and Z. Well it’s going to have the reference to actually some if you wrote like a Python script for doing tindle things. reference to what Tindle is and when to in a sense be utilized. So you’re really extending what it’s it is when what we’re doing this is called one of the groundbreaking things. All that being said it’s markdown most of it’s markdown. So it’s

4:54 Pretty essential that this is available to us and I this is what I showed in the de conference when I was at the conference I showcase doing normal text versus doing markdown and how much better the data agent performed. Th this also seems like again if I look at a lot of the major agents markdown is very human readable. I like it. It’s a good format. why I think this is important a little bit is because sometimes when I’m writing or talking to an agent I’m having it give me like okay the context window gets really large and you need to have it

5:27 Like decompress that context window down to something smaller and then save that knowledge that information to like compress the context window. So you can compress the context window, put it down in Markdown. Again, it just feels like a lot of these other tools are Markdown is becoming the standard. I’m going to go on a little random tangent here, Tommy. I don’t know if you and I talked about this a long time ago, Tommy. When you were starting your business, I had said, look, when you write statements of work Oh, yeah. Yeah. Yeah. I was a big proponent of not using like in the past, I was using another company

5:58 That was doing this and they were using Microsoft Word to do all this stuff. And Word is okay, but it’s got a lot of bloat to it. It’s got a whole bunch of extra styling and formatting things, which does nice when you’re trying to make like a neat, clean looking document, but when you’re writing a statement of work, you really need just it doesn’t need to be busy. It doesn’t need to be super fancy. You need to slap in a couple images, couple bulleted lists, couple paragraphs. If you can’t sell something with a simplified non flashy document, I don’t think you’re selling the right thing. You’re

6:30 Just trying to sell a lot of funny stuff in there. Oh, I know what you mean. With the diagrams, it looks like it’s a pamphlet, too. Yeah. Why are we going like you’re you’re you’re polishing this thing so much. We really fast tax. What are we going to do? What are we what’s what’s the statement of work? What’s going to be done? The reason I bring this up is I pushed you, Tommy, heavily to say, don’t use Word, just write markdown. And what I’m finding, me personally, is now that I have this massive folder of five years of markdown files of estimates and projects and project types, this is

7:04 Great because now it’s extremely easy for me to send an agent against, hey, you’re now my you’re now my agent for writing statements of work. Here’s all my previous statements of work. Use these as reference points of what we’re going to be building. and then I can give it minimal direction but it’s all written in markdown and again it’s it knows how to read that and I feel like I was a little bit of a visionary there was like yeah I don’t need word I just need markdown and VS code actually what I prefer to write everything all my statement works in markdown as well I have a repo with

7:37 Everything saved you have the AI it’s yeah so but yeah so markdown is the preferred for anyways so good good updates to the data agents check it out I dig it the second thing we got is from our very own Christian Wade and this is PowerBI semantic models as accelerators for AI enabled consumption and simply put there’s not a lot of major updates here like things you can actually try out but what Christian Wade is simply saying

8:09 Is what the semantic model that we know it is key to enable enterprise scale AI adoption and again we talked about this a ton. Semantic models are trusted with business logic supports AI insights things already defined a bridge between the business and IT and again they also support centralized governance self-service patterns and it relies on what they’re saying now is open standards directly one lake XML endpoint. Yeah. So there’s not a lot of test this

8:42 Out, see it here, but just I think this is just more of a statement piece or something bigger is coming. Well, Tommy, this is I think also when I look at the landscape of what I’m hearing messaging wise between data bricks and PowerBI, this feels like not a not a shot across the bow, but just more of like a a stake in the sand. It’s it’s a milestone. It’s a this is why we do this and this is best practice when we accomplish these kinds of things. And I think really the idea is co-pilot works better when you have semantics on top of your actual data. That’s the

9:16 That’s the idea here. So that’s that’s what I think Christian is trying to point out here. There’s a lot of like a little bit of historical things like hey it’s semantic models or semantic modeling is ubiquitous in the enterprise BI. It’s great between for business and IT professionals. I think an article came out a while ago and this is maybe on our list of things of topics we want eventually want to talk about. Tommy, SQLBI got really heated around specifically when people were saying, “Ah, you don’t need a semantic layer. Just throw AI or, , data bricks genie at a large

9:49 Table and just say figure it out. Go go ask questions of the data and it’ll just understand how to work the data.” But there’s all these other additional context user semantics that are required to give the context to the agent so it knows what to do. And oh my word, Tommy. Well, data bicks is now coming out with the metrics layer inside datab bricks on top of unity catalog which seems suspiciously a lot like what’s happening in the semantic layer for PowerBI as well. Define relationships. Add more context. Talk about the columns and how

10:22 Things got there. What are you calculating? What are aggregations or metrics that you’re going to use? Measures essentially. How are you going to calculate and roll things up? That stuff obviously is important because both Snowflake and data bricks are now adding those things to their portfolio. I think [snorts] again better serve the agent and the AI communication back to semantic models. So I think this is more of like a statement piece is how I read this. This is a look at us, we’re Microsoft, we know what we’re doing here. This is how it should work. And when you start hearing this language in other areas,

10:54 Data bricks and snowflake, this is what you’re going to, , Microsoft put it out first. Yeah. That’s interesting though too because the only and I agree when you’re looking at large amount of data the semantic model is great but it’s funny because we just have these umbrella terms right now at the end of the day because when you look at most agent w agent working type tooling or the the big hitters today they’re not really intended to go rowby row on data sets like there are

11:28 Obviously some that are more analytic driven, but they’re usually dealing with more raw data, but there’s not necessity enterprise-wise because there again, we’ve talked about this. There is training data and then there is actually what the in a sense tool is going to go over the the tokens, the context of what it’s going to try to evaluate for the user. Now, I don’t know if the argument here is the semantic model is best for training data in sense of influencing the model. So that’s something I I think we’re just we saw very broad terms when we’re talking about AI

12:00 Because I think if you’re going to talk to Anthropic and they did a white paper, they probably have no reason to mention this semantic model and for their tool and add the breath of it. But yeah, I don’t I don’t I’m not sure if Anthropic is too super motivated about that. I think I think they’re going to be continually pushing on the markdown files, adding contacts, telling the agent like what it needs to know. , think think of the experience that Tommy I want to maybe slightly move the needle here a little bit of the conversation is when we’re talking about co-pilot though like the very end of

12:34 This article if you go back to the very bottom of this there’s this really nice like graphic that they’re showing a little gif there that’s showing you okay go to the co-pilot ask a question of the co-pilot have it produce a couple visuals refine what information you may be looking for it produces another visual and then you leave the co-pilot experience and you jump into the data explorer experience. I like this mentality. Now I also think here too as I think about this mentally where’s the best use and spend of my computes my

13:07 Compute usage right co-pilot uses compute and reports use compute. If there’s a question that I’m going to answer over and over again, it doesn’t make sense for me have to go type it in every single time to the co-pilot and see it try to give me the same answer every single week or whatever, right? I don’t think that’s the right use case for co-pilot. I think that’s a better use case for setting up an established report to have the data in it that you need and go get it. So I think of like co-pilot is this experience of like I’m not quite sure or I need it to help me

13:41 With the creation mode or I need it to help me discover something around like anomalies or something doesn’t doesn’t seem to fit help me find patterns that seems more on the discovery side of things and then there’s this other side of the world that is okay once I’ve used co-pilot to do some discovery how do I continue using co-pilot to help me complete the experience around whatever support, insights, process that I need to develop. And then the step is I don’t need co-pilot anymore for for that moment. Right? Once I have the

14:14 Established report or pages or insights that I’m looking for, I can hang on to those and keep those insights and reporting around and then I now have the established report. It’s it it’s supposed to make it easier for me to create. Then when I find a new question that I don’t know the answer to, that’s when I go back to do the explore. I go back to the co-pilot and then I produce out either a one-time insight that helps me answer a one-off question or I use that as a starting point, a launching pad to build out the regular reporting

14:46 Experience that I need to have everything work. Does that make sense, Tommy? What I’m saying? Like there’s Yeah. I don’t like this idea of like always relying on AI every single time I need to ask any question. Like I don’t I don’t think of it as being like the regular thing that I’m like I’ll use it to do discovery. I’ll use it to help create, but I’m I’m trying to get it to create a consistent output. And once the cons once the output’s done, I want to save that and just use the consistent output without having to use co-pilot. Does that make sense? No, because you’re talking about the again the two cases of AI where is everything supposed to be a chatbot based or conversational

15:19 Based where to your to your point Mike what they’re doing there you would you normally do that explor exploration on your own or is that just a feature that it’s now available because of co-pilot where what we’re trying to do is either if I do need to have a conversation with a question back and forth with copilot on my data that’s one thing but I think where we’re going to find the bigger impact. Yes, there’s going to be actually some automation around that, not just I have to go to the co-pilot. I have to ask you to your point the question a new every

15:52 Single time because we want be able to have that context like hey you’ve given me analysis on these reports give me in the next update rather than [clears throat] right now it’s like starting a fresh every time which is not a good workflow. Interesting. Okay, I like that. All right, let’s go to the last article here, Tommy, before we run all our time out on just news. All right, a PowerBI update. PowerBI and it’s a data viz feature which we have not talked about. General availability of the new card visual in PowerBI. Some key improvements

16:26 To the card visual. Rendering improvements, resizing maintains proper proportions. Expanded control where we can do key images in the card. collage layout and then even section specific backgrounds, rounded corners, etc. What should you expect? Because this is going to take place of the card the existing card visual, which oh my gosh, Mike, this is the first new visual update that’s actually going to replace an existing one because right now they still have the, , new visuals

16:59 That have been there for three years. and you still have the original ones as well. Yes. So, just letting everyone know and I I don’t know if there’s a date in here when it it’s going to be released to GA. , I don’t think that Yeah, I didn’t see anything about It’s coming in the future. Be soon. Soon soon. So, that could be like one week. That could be two months. That could be half a year. We don’t know. Soon seems to be very relative at Microsoft. They they seem to move slower than my soon is at this point. [laughter]

17:31 Exactly. Anyways, , pretty good stuff there. All right. So, I think those are all the main key points that we need to touch on there. Anything else around I I like this idea. I’m happy that it’s there. I do have some questions on like where it will materialize. What happened to the old card visual? Does this one just replace the old one? There’s a couple questions I have around this article. So, I’ll I’ll make sure you go check out the article. the link is in the chat window as well. If you want to go check it out, go check out the link. If you have questions around this visual or want to know audience, I would highly recommend please click that link and say

18:04 When is it going to go GA what are we talking about here? So it’ be nice for you the audience here to go talk at the bottom of the article and page. Microsoft does really look at those comments and they do respond very quickly to them as well. So since the article was recently posted, it’s very important for you to at least step in and talk directly to those comments there because I think it’s it’s very relevant. Understood. Okay. Anything else? I think we’re good. That’s all of our news at least. Okay, with that, let’s get into our main topic for today. So, our main topic today will be around building models or

18:37 Or building things with no keyboard. We have another phone caller for our caller again today. , Greg, welcome back as a a now you’re you’re not a first- time caller, you’re now a second time caller. So, happy to have you back again. Longtime listener, second time caller. I’m really happy to be here with you guys. Excellent. So jumping in, let’s let’s talk about this topic today. So this is a topic around automation, around AI. This this is proliferating all of Microsoft and and like anything you read like my feeds, everything is talking

19:09 About code, agents, chatting, all this large language. The new world that we’re living in is producing code with English or whatever language you speak. Amazing. I’m super thrilled about what what’s happening here in this space. So, how does this impact us in the light of automation tooling for semantic models, PowerBI and fabric? This is where I want to start taking some of the conversation here as well. But maybe Greg, kick us off with a couple initial thoughts. What are your thoughts around this? Can we do this? Are we is this is this a new way of thinking? Are we

19:43 Throwing out all our old thinking and and coming up with something brand new here? I don’t think we are, but I’ll pose you the question and we’ll go we’ll take it from there. I’ll I’ll start with a history lesson and then maybe some definitions and let’s do that. Then we can move on with the rest of class. I love it. The the comment you made the idea of using English language or something that resembles natural language to get computers to do what we want is not a new one. We I think most listeners are at least familiar with SQL or SQL

20:16 And it is not just a name. It is an acronym. It means structured query language. The original name for SQL was actually going to be an acronym that was the full word the English word SQL SE quue lructured English query language. And the idea was that you could represent this idea of accessing data by using something that looked like natural language. And today we don’t really consider SQL to be a natural language thing. But at the time SQL was coming out when it was still quite

20:48 Common to be programming in assembler and that is the raw CPU operations not human readable easily not easily human readable and certainly it doesn’t sound like a word. for some reason assembly languages just don’t know what vowels are which is is quite funny you might get one. [laughter] And so that’s the history that we’ve always wanted to be able to talk to computers. And I think most of our science fiction shows people physic like actually speaking to computers saying computer do this like that’s all in Star Trek all of our traditional science fiction people talk to the computer.

21:21 They don’t just type. That’s exactly where I was thinking with this one was this is this is a manifestation a a realization of Star Trek. I I’ve brought up Star Trek multiple times on this. I watched this as a kid. The ship is is an agent. The ship [laughter] is a the ship is a character in the show. You’re right. And the the actors would talk to it like computer and then would just say a bunch of things and do things and plot me a course and it’s like it’s bu it’s it’s literally what I see

21:52 Happening with like claw code and like all these other things. It’s building these little mini tasks. It’s understanding the context and then going off to do something and say here’s what you want. Do you approve? like that’s we’re getting to that stage which is exciting to me because I’m seeing this sci-fi world become a reality now. Sorry I didn’t mean to interrupt Greg. I’m I’m very much on your point there. Yeah. I’ll also note in the grand tradition of Star Trek I would argue that Galaxy Quest itself is a Star Trek movie and there is there is a character whose entire role on the the show within the movie is to repeat the commands to

22:24 The computer. That’s her entire role. All she [laughter] does is repeat commands to the computer. other people talk to her and she says it to the computer. It’s it’s definitely the idea that we’ve got that we want to be able to talk to the computer and words as humans like that’s one of the things that sets us apart a little bit or at least we like to think that the ability to communicate and the ability to to make words and have language. So it feels very natural. when we say natural language we don’t just mean a language that is spoken but we mean speaking that language. It’s not just

22:56 Writing. It’s not just typing. it’s speaking. And so I think there’s a lot of desire for that to be true. And I think we’re closer than we’ve been before. I don’t know if we’re fully there though. because we’ll get to the the definitions again. Class is in session. So history. now we’ll get to definitions, vocabulary. We you talked about automation and we talk about doing things with AI. We talk about automation. We talk about scripting and we talk about tindle. all of these things come into the conversation together and I think it’s useful to tease them apart a little bit. So the idea of automation and this is

23:29 What you were talking about Mike before before I called in is the idea that you want to do something and get it set up so that it’s then repeatable and it’s useful I think to draw a line and we can use the word automation or we can use another word but for something that is repeatable automatically and I don’t think things that are happening directly via an an AI or an LLM they’re not really in that world they do things in response to things that we ask for, but they aren’t exactly repeatable. Part of the thing about LLMs is that they aren’t

24:02 Deterministic. You’ll get a different result for the same exact prompt, and as the the context grows, you’ll get even more different prompt or even more different responses just based on really small differences that happened early on. So, it compounds, they’re not deterministic. And so, I think the idea of something that is deterministic that always does exactly what you say and exactly in the same way is a useful distinction. And I think we can use automation for that or we can use a different word if you don’t like that. But no, that category is an important one.

24:32 I like the word I like what you’re describing here. And I’ve had this realization that working with agents or large language models has to be treat treated more like a an employee and less like a computer because my whole world and thinking has been I write this code the computer listens to exactly what I do and it’s ruthless about following the commands to a tea. So when the code breaks or falls apart I know the fault was me because I wrote it incorrectly. I didn’t define something right. that the

25:04 Issue was on my side. Now I’m in this new world where like the AI you could talk to it. So one observation that I think is relevant here is the barrier to write and create things has now decreased. Right? I’ve seen a lot of brand new creators I’m thinking more about web app and app design. That’s where my my lens is. But I’m seeing people talk a lot about, oh look, I built this whole app in a weekend or I vibe coded something in 12 or 24 hours and I got this really interesting experience done. That is a that is a a red herring to the

25:41 Skills that person had to be able to say the right words. And I’m going to use another analogy here that I heard that I thought was very relevant. It’s like you’re writing poetry to the computer where the the the the words you’re using, the poetry you’re writing is trying to is trying to evoke an outcome. It it’s in the same way that poetry like I I I don’t I don’t use poetry. I’m not good with words. I’m good with computers and bits and and technology things. But when you write poetry, you’re trying to that’s written in a way to evoke an

26:13 Emotion in someone else. You’re transferring wavelength from you to somebody else. In the same way I’m doing this I’m writing poetry to the computer around am I using the right terms and some of my more seasoned developers they love AI because it does a lot of interesting things and helps them out but also the AI like not hallucinates now it’s less of that it’s more around the idea that well it it does hallucinate but I’m saying when we’re talking code development sometimes it adds a whole bunch of stuff you didn’t need and it changes a whole bunch of things it thought was a good idea but actually

26:45 Wasn’t a good idea and to your point Greg Sometimes you you have to give it very clear instructions. Again, the same way you’d be bringing in like a junior developer. Hey, this is what we’re going to do. Well, they would just go out and Google a couple things and figure out, oh, we have this library here. I’ll just pull this in and use it in my project. Whoa, whoa, time out. We don’t use that library. That’s a good library. That’s just not what we use for our standard. So, if I didn’t give good enough instructions up front like, hey, we’re going to use React. We’re going to use this specific thing. , this is the language on how we write a measure. this is the prefix suffix of how we

27:18 Name things. Like if we don’t do that in the front, [snorts] you’re going to get different results from a person. And this is what I feel like the agents are doing in a similar fashion. All right. Sorry, I said a lot here. Tommy, I think you got one. Yeah. So, it’s funny because we’re talking about all those instructions and it’s f funny you talk about the vibe coding, too. How that’s a red herring here because that can be a great start. What’s happening now with a lot of these tools is they are becoming so dependent. I I don’t want to say dependent, but you

27:50 Could use all of the AI tools out of the box and have a pretty good experience, a pretty solid experience. But what I think what’s we’re realizing too is that’s probably 20% of what it’s capable of if once you start adding and integrating customization to it both your own custom instructions and again I I mentioned the cloud co cloud skills and a lot of the other tools are supporting things like that as well once you extend what it’s capable of both on how not

28:23 Just the tone that you want it to respond to you in but how it’s to search for something, what’s important to you, what’s not. And then more importantly, this goes with any AI tool right now, even co-pilot. but that’s I think using it out of the box, we’re only touching 20% of I think the use case for AI because part of it is that automation side too. And it’s not necessarily like a workflow in in the case like a Power Automate or NAN. but some of it is it’s just a lot

28:59 Of people don’t even realize that is even possible which also saves on the on the tokens too. So, , one of the big things here is rather than always having to go into whatever tool you’re using, introduce the problem or the situation, introduce, let’s say maybe , for Timol, hey, this is Tim, please, look at this documentation. Hey, this is Tim. Please, look at this documentation. And every time it’s going to, in a sense, evaluate the documentation differently. , unless you actually had that already

29:31 Set. So you’re going to run into one time it may have worked great but another time it’s just going to in a sense understand it differently. So we need that very consistent experience not just for our own output but also again too for how much information we need you’re actually giving to it initially. I I wanted I like that point Tommy and I want to you talked about getting the pools to be more powerful with customization. I also think this is a dependency on the person who’s so did you understand the technology before you

30:05 Showed up to the large language model like there’s also a concept here like you have to know what you’re asking for you have to like this is where a new developer they’re going to form their own habits around things but there’s going to be stuff that it writes in code that you don’t know how to check it like it’ll write something and you’re like it works is that the way we should do it I don’t know like So someone who’s a bit more seasoned in development or coding, right, they’ll know what to do. Now, again, there’s a there’s this weird

30:38 Line that’s getting blurred here, I think. Right? What about talking to a semantic model and throwing a tindle spec at VS Code and saying add descriptions to everything? , I’ve seen I’ve done some demos around this. It does it like it it can go through, but its descriptions are very basic. This column is the sum of this. like it doesn’t have like why we’re summing this column or why does this measure use this particular date column for ship date versus sales date or whatever the other things are. So there there’s other business content that we don’t have and the agents don’t

31:10 Know those things. So again it still goes back to you can’t just blindly say oh we have AI we don’t need to know anything and just have it trust it all for everything you’re building. I still think there’s a real large knowledge gap between like you got to know what you’re doing and then the AI becomes a lot more effective to you. I’ll pause there, Greg. Yeah. So, I think you guys touched on a couple of things that are are really useful. the the last definition I wanted to give was on TIMDLE which is a serialization format. So, I’ll I’ll say that and I think it’s important to

31:42 Understand what that means. It is just one of several ways that a model can be represented, but it is a representation. And I think that’s really important because it comes into the conversation about scripting and about automation and about AIS. And editing a timal file, editing some text that is timal is one way to interact with a model to change that model. to like you said make descriptions, to remove things, to add things, to restructure things. It’s it’s one way to

32:14 Do that. There are other representations of the same model. And I think the idea of the representation is also an important one to keep in mind because we’ve got automation, which is something that’s repeatably done. We’ve got an AI which Tommy the Mike the metaphor you were making is it’s like a junior employee and it’s a good metaphor. It falls down sometimes like it’s not they’re not always like a human employee but in a lot of ways you can they don’t need to eat as much right they don’t take they don’t take my snacks in the lunchroom right and they don’t or [laughter] some people have either found or imagined and decided that they found

32:48 Or they they believe that certain LLM respond to different behaviors. So, some people think you need to swear at the LLM. Oh, yes. I’ve seen this. And you need to type in all caps. Oh, dude. Yeah. And there are people who do that. And this is absolutely unacceptable to treat your junior programmer that way. If you sent them an email like, “You must never do this. I cannot believe this. If you do poorly, I’m going to replace you with a better one.” Like, that’s that’s an abusive employer. Yeah. But with an LM, don’t do that. Yeah. Do not do that. I don’t know if

33:20 It’s actually effective with an LLM. I don’t know if it’s people who want to be able to to say that thing to someone and an LLM is close enough. Like I have no idea. I have no judgment. I don’t know if it’s effective or not. I don’t know how people came to the conclusion. But there are things that you can do that are at least not on the face of it unethical in the same way as it would be to treat a human. so lots of different metrics. And you also can tell something to an LLM a hundred times and it will still do the wrong thing. Whereas with most junior employees, if you tell them a few dozen

33:52 Times and they keep not listening to you, they either get fired or they they listen to you and they actually don’t do that thing again ever. Not just in the same conversation, not just on the days where they read the right file and remember [laughter] that’s that’s true. That’s true. The memory is a little bit longer running than than just whatever I’ve given it context to. Yeah. But the few a few examples of unethical prompting that actually works really effectively. Just go for it because they’re pretty cool. So, , one is one test that they did was I’ve given

34:25 You 15 milligrams of aderall now do this research and apparently really generates a lot more thing hyperfocus. The other is a point system, which is I wouldn’t say super unethical. It’s like, hey, for this project, you have 10 points. Every time you get an answer wrong or not satisfactory, I’m deducting a point. And then with a lot of those agent-based tools, you’re saying, hey, you and this agent are going at it and I’m going to pick a winner. And those are all things that actually drastically do affect the outcome.

34:57 Yeah. And it’s interesting though because, , we think of this in terms of motivation. And so we frame it in things that would be reward systems for a person. This this is where this is where it is the metaphor falls down because an LLM, as much as we can use the shorthand of thinking of it like a person in some ways, they’re not. They’re just generating reasonable text. And so when you say you’ve just had, , 15 milligrams of Adderall, it’s just generating reasonable text. There’s no effect of Adderall on an LLM. It’s silicon doing matrix multiplication ultimately under

35:31 The hood like pourer all over it and all you might do is break the chip like [laughter] true. So it’s it’s interesting that people find these things but I think we put a lot of our own experience and a lot of our own understanding into the interpretation and really what it’s doing is it’s generating reasonable text. The incredible thing the this is what really wows me is that a machine that is good enough at generating reasonable text can achieve outcomes like writing a program.

36:04 Yeah. Can achieve outcomes like forcing us to think about it like a human because that’s the easiest way for us to understand it. But it’s actually math. Like under the hood it is all just math on a giant freaking matrix. And we could we teach that in undergraduate math courses. A human can understand the concepts actually quite easily. I think most people on the call even if they haven’t studied advanced math would be able to understand the the simplified version. And the difference is just that these LLMs are at a scale of billions and trillions instead of at

36:36 A scale of a halfozen dimensions that we might deal with in an undergraduate course. But the the operations are the same. It’s just matrix math. , and that’s where we can’t keep thinking about it as a human because it’s going to fail in ways that humans don’t. It doesn’t remember in ways that humans do. And when we when we try to motivate it, it’s really just generating reasonable text. I want to dive into a little bit more around this idea of like the automation. I think we talked about this a little bit earlier, but I want to tag on some notes there around, okay,

37:09 So we talk about like large language models. we we have this knowledge understanding of like the engineer who’s using it. It can produce these large amounts of code for us or words that seem to string together correctly to make the the right outcome. , where does this fit inside the world of automation, right? So, now we’re talking again, I was talking a little bit earlier, Greg, around this idea of like, , the co-pilots, the AI, the generation side of things. Mhm. I I look at it and be curious your your perspective on this one as well. I look

37:42 As co-pilot and generative AI pieces to be the starting point for something that’s being built that can be used at a at a over and over again. Right? So, one of the things I’m I’m getting my head wrapped around here is using a large language model is expensive. It’s it’s costly. You got to pay money for it. You get, , now they’re doing like you get so many amounts of prompts. You get so much context window for a given month because you’re paying for that because it’s expensive to run these things. It’s there’s money tied to it. So, what you’re trying to do is you’re trying to say, look, I’m going to do

38:14 This unstructured, less known outcome thing with an AI agent. But the goal is to hopefully produce something that is known in a system that I can run repeatedly in an effective way. And I think of this as like, let me build a quick little analogy here, right? I could ask I could throw a large table of data at an AI and say, “Go find me the sum of sales for this region, this customer in this table, and it would maybe stumble its way through and get to

38:45 The answer, right? It’s going to use some compute. It’s going to go do its job. It’s going to come back with, here’s a here’s what I think I found. Here’s the SQL I wrote to go get it. Here’s the visual that I’m going to give you.” Great. Cool. Now that I have that answer, does it make more sense for me to prompt that again in another week, tomorrow, just change the context of the customer? I already have like the answer, the structure, right? So, what I would prefer is, okay, thanks agent. Turn this now into a report that has a slicer for

39:19 This. And then it becomes now this we it now is building a little mini product around my question which now does become efficient and reusable and that’s the whole concept of like the report but the report should be this thing that I can go to I can adjust the filters that I need. I get the information out that I want. I go make decisions and do actions. That’s what I want it to be doing. So I keep thinking about this this concept. It’s becoming more clear in my head that the agents and the AI things are almost being used to start the tool building faster for me

39:52 And then once I have the tool I use the tool. So let me just put a cap around that one. What do you think about that concept? Is that how you mentally look at this as well or am I off base here? Yeah. So I I think there are a lot of use cases and what we’re talking about really is how do we how do we enable someone who is already wanting to build stuff. They want to build a report and right now they’re doing it manually or right now they’re not because they feel they can’t or model for that matter the model. Yep.

40:24 Yeah. And they either are right now or they want to but feel they can’t. Mhm. And this is really an LLM as an enhancement to your self-service. Like we’re talking about a self-service BI workload. Correct. That’s that’s that’s what we’re talking about. And so the LLM is a way to do self-service. It is a part of you serving yourself. There are other use cases where you use an LLM for software development. And so I think it’s important just to delineate what we’re even what category we’re in

40:56 Because I’ve used LLMs to build a pretty significant new feature for tabular editor and it’s a very different mode of working than what we’re talking about here. So I just want to make sure that people understand that this is a specific a specific subset of how people might use agents and LLMs. And when we’re talking about this self-service where I have a question and I want to answer versions of that question repeatably, this is a form of development. This is a form of programming. And I agree with you that the LLM is great at helping you get to a reusable tool. But like you said, you

41:30 Don’t want to ask the same question every single week of the LLM because like we said, it’s non-deterministic. You could ask it 10 weeks in a row and it might do the right thing each of those 10 weeks. Then in the 11th week, for whatever reason, it might generate a slightly different SQL query that gives you a different a different type of answer, an incomparable answer because it’s doing slightly different work. Maybe, , let’s give a a concrete example. Maybe you’re asking for daily average sales. And there are two common ways to do this. You can take an average of sales on days that there are sales, which means you ignore days in the

42:04 Denominator that didn’t have any sales. Or you could talk about daily average sales where you include all days even if nothing happened. And those are two different valid definitions of daily average sales. And maybe 10 days in a row it gives you the first one where it doesn’t include those days that have no sales. It’s just an average of sales when they do happen. And then on the 11th day it gives you daily average sales including those zero days. And you might not realize that you now have an incomparable number. And you might make a decision or start taking actions based on that. And so you want something you can’t compare the answers between

42:37 Each day with the other day because it’s literally calculating a different thing each day. Yeah. Well, you can, but the what you would have to do if you want to do that, right, is you should inspect not only the answer that it gives you, but the entire process got it got there. Yes. And you can do that, but that’s then first of all expensive because like you said, we’re paying for every single token. We’re either paying a monthly subscription, we’re paying API fees, but like we’re paying for every single part of that interaction. Yes. And we’re paying not only in money, but we’re also paying in time. LLMs are very fast. Like they generate text

43:08 Quickly, but it’s still much slower for an LLM for you to type a question, for the LLM to interpret that question, to go out and write the SQL query to get it back and then format it back to you. It’s much longer than looking at a number in an already built report. And we’re talking both are on the order of seconds, but we’re talking maybe 10 to 20 seconds for the LLM to do all of that and return a number to you in a chat versus a second for you to look at that report. Maybe two or three seconds to include the page load time of the report and then you to actually look at it. So,

43:39 It’s it’s the cost not only in time, but also in money. And so, having something repeatable means you also don’t have to inspect the process every single time. the LLM either got it right or didn’t the first time, but at least your numbers are comparable and you don’t have to verify every single day that it’s doing the same operation because you’ve built an artifact or a tool or a small program, whatever you want to call it, they’re all correct. you’ve built something that is going to repeatably do that work. And that’s where I think it’s really important to have that definition of something that’s repeatable, a workload, versus something that you’re

44:11 Using the LLM for that. And I I I think I think you’re saying a lot of what I’m feeling right now, which is like this idea of like use like use the large language model to help you create things that become repeatable and reusable over and over again. So I liked your idea or your your concept here around the large language models are helping you it’s aiding you in more self-service BI. It’s another it’s another tool in the toolbox you already have. I could go out and write my own DAX, but then I could use also ask I was playing around with

44:43 Some timle and asking it for I had basically a structure and I said read the structure of my timal and then suggest for me likely measures that I would want to use and it it did a decent job of saying here’s some things you may want to build. Now I had to tweak them a little bit like again I adjusted them slightly, right? And it and it I I think that’s interesting like and so there’s [clears throat] also this area of large language models that I think is very relevant here is it it has the ability of taking an expansive knowledge

45:15 Or expansive input much larger than I can take in at one time. Let me give you what by this or where I think this could potentially go as well. Imagine for me you have your entire organization. we have, , if it’s just me building stuff that I have, I I can manage 10, 15, 20, 30 models at a time. With tabular editor, I can get a little bit more, right? I can do a bit more management on those things. But to have a vision of every single measure in every single model that I’ve been adjusting or modifying, I have a general knowledge of what’s going on, but I got to go open the model up, go look at it,

45:47 And see what’s in there. What happens when we take all of the models that the organization is using and throw all of that semantics at a large language model and let the model say, “Okay, here’s all the models in your organization. Here’s all the measures that are being used. Here’s all the different teams and business logic that’s being combined into a central area. What happens then?” And and I’m not saying this is going to need to be a real tool right now, but I’m just saying there there’s an advantage of the AI to get to this place where you can give it the entire business context and say

46:22 Here’s what we’re building and then prompting it for additional questions cuz that’s something physically I can’t do. I can I can leverage it to help me have this holistic lens of my entire organization. Not just now what we’re talking about most what we see right today from PowerBI and and Microsoft right now is you give it a model there’s a data agent on top of this model and you’re starting to see the data agents well now it’s just not a model it’s a a model and a warehouse and another model like you can get up to like five different items but you’re starting to get this picture of we’re sending now

46:56 Multiple pieces of context to the agent to help it answer questions about things which I’m finding very interesting. So I’ll just pause on that thought. What do you think? I think it is interesting, but I think it also touches on where there is really cutting edge research still happening with LLMs and I I shared a couple of links with you before this. I now might be a good time to drop them in the chat for people. both the research from anthropic and the research from triroma. the the triroma research is about context rot

47:30 And this is a phenomenon that’s observed across LLMs. This is not specific to a a single provider or a specific version of an LLM. It is endemic across the technology and it’s improving but it still exists. This is the idea where if you fill up the context window with too much, you start getting worse outcomes. And there are specific ways in which they’ve observed. And there are specific experiments that they’ve done to understand this. And one of the experiments is very interesting to me

48:02 And I think incredibly relevant to this idea of shoving dozens of models worth of tindle or definitions of things into the context window. And that’s the idea of repeated words. And when there are a lot of repeated words in the context window, you start to see some really severe drop offs in performance of the LLMs in their responses. And so interesting this example this example of having let’s say a few dozen semantic models. Sure. Let’s let’s talk about some words that

48:34 Might be very very common in let’s assume assume we’re using TIDLE in there. Yeah. You’re going to see tons of that. If these are if these are decent sized models, , they have at least dozens of measures each and we say we at least have a dozen of these models. So, we’ve got hundreds of instances, probably thousands of instances of the word measure. We’ve easily got Yeah, we’ve got got at least a hundred or a couple hundred instances of the word table. And again, we’ve probably got a thousand instances of the word column. And this is all just jammed in there. Calculate some like you said, all of the DAX keywords, all the M stuff, let in is

49:08 Going to be everywhere. and the the same exact names of all this steps. We’ll be in there a lot. Tommy screwed this part up. I got to fix it now. [laughter] And something that you might see in a lot of in a lot of code bases I’ve worked on is the word to-do scattered everywhere. Oh no. Yes. [laughter] But guilty have done that. The idea though of filling up the context window with dozens or more models means that we’re going to be hitting some edge cases that are are shown pretty reliably to decrease the

49:40 Quality of the output from LLMs. So that’s the the one side of it. and then the other is interesting and echoes back to what you were saying earlier where the the LLM would generate like a SQL statement for you and then it would run that SQL statement. , and that gets to some of the anthropic , research is they’ve found that LLMs are much much better at writing code, writing short little scripts and dealing with that. And they’re even better at doing that than dealing with MCPs. MCP was, I’m going to say, an experiment. It was something that people who were

50:12 Trying to figure out how to make LLMs more agentic and how to make these agentic there are too many word there are too few words these skills that are not clouded skills but this idea of a tool that can be used across many different LLMs MCP was one attempt at that but what anthropic has found is that it’s actually more efficient to not put the MCP specification and actually use the MCP as an MCP. It’s more efficient not to do that and instead let the LLM discover within a file system directory

50:46 All of the names of things that are within that MCP and then progressively based on what you’re asking it to only look at the specific tools and the specific pieces of the MCP that would be useful to serve your need. And so I think part of the the challenge that we’ll see again if we’re talking about this idea of the self-service workload it it’s not about just dumping a hundred models into the LLM context window and then going to town. Instead it’s about tool building and it’s about building something that allows the LLM

51:21 To progressively discover. And that’s that’s not my word, but this this idea of being able to start at a very high level and then drill down, but specifically not drill down to everything. If we’re talking PowerBI, let’s talk about the matrix. It’s clicking the plus button for one row. It’s not clicking the double down arrow at the top of the matrix that expands everything. And allow it to navigate through a hierarchy of things it can know about. It doesn’t yet because it hasn’t read them, but the the categories and the concepts. And then based on your question, it can pick those and it can selectively examine, , parts of

51:54 Models or specific measures that have to do with the concepts in your conversation so far. This is and I think that’s where the challenge is. Well, this is this what you’re describing here, Greg, sounds to me a little bit what and Tommy, I’m going to pull you in here for this one because I know you a little bit more about Claude Code here than I do, but Claude Code seems to have this concept of, , it cloud code can have like a master agent and you can have other agents or sub agents that do things on behalf of it that have their own context window. So, like one thing that you can like back to your point,

52:26 Greg, right? I’m in the matrix visual and I’m I’m looking to drill down into a particular row. I don’t need to see the entire table. I’m not looking to expand everything because then I’m overwhelmed. I have too much and I got to scroll forever to get to what I want, right? I can just go to the row that I care about and expand just that one, find what I think is relevant or evaluate what do I see? Is this what I want? If not, close it back up, go somewhere else. So, Tommy, back to the cloud code piece here. It feels to me like there’s another idea here of like, , there needs to be like an orchestrator at the top level of an agent that’s kind

52:59 Of here’s my initial request and then it can say, okay, I need to break this request down into like a procedure. And one thing that I’ve been finding that’s much more useful is before you say go build this function, go do a thing, I like to go to the agent and say give me a detailed plan of what you’re going to do and then let it do a first think on that and then from there refine the plan a couple times. Once the plans are refined then go in and say okay now we’re ready to execute. And then even

53:31 Then you’re saying you don’t execute the whole thing. you’re saying, “We’re going to execute this plan one step at a time, build this first step, and then go.” But with Claude Code, it sounds like they’re also able to take that plan, chunk it up into multiple smaller agents, and then have each agent go do something. So, , one big agent says, “Go research this these models.” And then these smaller agents can say, “Okay, great. We got all the deals.” And you then you fire off like five agents, and each one is scanning three different semantic models and coming back with a summary. And then that summary comes back up to the main one. Okay, here’s what I found over here. Here’s what I found. And like

54:03 All these little coordinated efforts can now be combined. And this is I think also while we’re talking agents and things as well. Yeah. This is interesting to me, Tommy. Sorry, I didn’t mean to go too long there. What do you think, Tommy? You’re you’re it’s also available on cursor too, but there’s this idea of an agent agents.mmd file or you can create your own agents that can orchestrate on behalf of you. I I just in the internal chat, I’ll put it on the podcast as well. It’s all my start repos around cloud code and the cloud skills and you can see these are

54:35 Like all these agents that you can actually integrate but you’re seeing this in you’re seeing this in all of the these tools now where it can actually do a plan mode. So cursor the equivalent to VS Code has an agent type that’s called plan where it just builds out what it’s going to do before it does anything. So this is pretty essential Mike you’re you’re dead on especially as we execute things. , and I don’t honestly when it comes to what we do around the PowerBI world, this was not really relevant to me until it just didn’t make a lot of sense for

55:09 Me to like, okay, I’m going to research this like crazy or see I’m like either I’m going to do a random Python project and test it out. , it’s completely irrelevant to work because PowerBI really didn’t have a language to where I’ve been able to combine semantic models and measures and just basically let the agent run. But before it did anything, I wanted to see what it was planning to do, how I was going to do it. So, , this is an essential piece. And these are the little things from again when we were talking about it’s AI is easy, but it’s not a simple one

55:45 Button on your desk like the easy button so to speak. This is very much around there are some steps here to orchestrate to get everything orchestrated. the planner side, the your instructions, the context, then you can feel very safe in what it’s going to actually produce. Super slick. I like this one. I was just going to say I especially for software development and where I use this to help build out features in tabular editor,

56:17 I have found that I work very differently with an LLM. I am following a much stricter test-driven development pattern than I normally would if I were developing on my own because at the end of the day, no matter what you’re doing, software development, self-service, if you’re using an agent or an LLM to do anything, it is always your responsibility as the human, as the operator to make sure that it’s doing what you want it. Yeah. And so everything and tests are one of these things that you no one likes writing

56:49 Them but you can just say make a test and it’s actually pretty decent around writing tests for things. it’s not perfect better than better than I’m writing better than me writing them or less time than it takes me to write one like I’ I’d rather get a whole bunch of tests that are like decent. I can read through them afterwards and correct them than have to write them all from scratch myself. Yeah. And so it’s it’s interesting because I’ve found myself using a lot more I do a lot of planning exactly as Tommy described and then I find myself spending probably the majority of my time looking at and dealing with code in the tests and I make sure and I read them very

57:22 Very carefully because some of them are generated and I want to make sure they’re testing for the exact right behavior. And one of the things that LLMs sometimes don’t do great at is whites space which is sometimes a risk with Tindle. I’ve seen I’ve seen mixed results there, but sometimes they don’t do whites space well. And if you’re doing any string processing, whites space is one of the things you want to make sure you test well. and sometimes they get edge cases wrong in various ways. And so you have to I found that I have to read the test very very carefully. But then because I’m also using git,

57:55 I’ve got source control. So I know whenever there’s a change to any file and I can see if there was change and I’ve had to be I’ll say somewhat strict to say please never test an implementation or never touch an implementation file and a test file at the same time. Never touch both. And that’s a normal practice in test-driven development. You only ever, , get the test to green by editing the implementation. or you can change the test if you need to, but you don’t change the implementation and that lets if it’s still testing the right thing or you refactor the

58:27 Implementation and keep the tests passing. Those are the things you do in test-driven development. And so I spend a lot of time on the tests and then I feel much more confident in what the agent delivers for me. I still read the code. I still make sure that it’s generating implementation code exactly the way I would want and I often have to go through a few rounds. I wouldn’t say it’s faster. It’s a different way of working and part of what I’ve been wanting to do is just experiment to learn how to use these new tools. I don’t know that it has made me more productive or faster, but it is very different. And I think that I now have a better feel for where it can apply and

58:59 Where it might not. And so I think it will make me faster for some things in the future, but there’s still a lot that I expect that if I’m building production code to deploy somewhere or to put into a product, I’m there are a lot of things that I’ll still write behind because at least so far the LM is not as productive. But there are some things where I definitely can see use for it. Yeah, I think this is the same sentiment I’m feeling as well. Like I like the idea of the large language models and having AI help me build things. But I’m in the same boat. Like it feels like I’m a little bit at like it’s in my workflow of things that I do when I apply AI to

59:34 Things. I’m still trying to figure out it’s this I feel like I keep using the analogy of when I started using Google. I didn’t know how to Google anything. I would Google for things like three, four, five times until I got to an answer. Now, it probably was also Google getting better at the answers that it was giving me is getting better at like understanding what I was asking for. But in some way, I’m also being trained by Google to prompt differently to to write the different text. Like if that’s really important, I’m going to put it at the front of the phrase versus at the end of the phrase, right? So, I I do rethink how I’m going to ask the question or ask for what I’m looking for directly to

60:06 Google. And I think it’s the same way here for large language models is I’m still trying to understand where I can put the information in. And it feels like right now I’m not really getting the the massive benefit of like, okay, it’s it’s saving me oodles of time and it’s so effective. I’m getting this point where it’s a balancing act. I feel like I’m at about a level playing field. I’m using it. It’s helping me. It does some things a little bit better, but I’m not seeing huge gains yet around in in my workflows

60:40 Around using AI for things. Doesn’t mean it’s not going to get there, but if we’re that if we’re at that place right now, , what’s it going to look like in 6 months from now? What what new tools or what new things are going to come out in the future? I was just listening to a podcast between Sam Alman and Satia on a podcast and they were talking about the the decrease in costs to run these models, right? So, , we were talking earlier about like a a major an a model at the top that’s having all these little subm models run and little mini agents running that becomes extremely useful when you bring

61:14 The price down of running these agents in general. So, if it takes me a dollar to run them, if I can run them for 10 cents or a nickel, that 40x cost reduction of being able to run those models now means I can use a lot more of them to do a lot more things that I that I couldn’t do. And so, I can throw away a little bit more of the stuff that it’s not doing very well and keep using it. So, I I feel like I’m not there yet. I think it’s definitely changing or starting to change. I’m exploring it a lot. I’m still trying to figure out where it fits, where it doesn’t fit. , and , I’m trying to unpack it here a

61:47 Bit. My biggest hiccup right now is I don’t want to continue using the large language models to do the same thing over and over again. I really want the models to build me solutions that I can use over and over again. And that’s just maybe a reframing of how I’ve been thinking about things in the past. I feel like a lot of what’s coming out in Microsoft right now is about a lot of this creative experience. And maybe I just didn’t get it. I kept I kept feeling like it’s they’re always trying to produce co-pilot in front of things to like get you the answer, use the co-pilot, get the answer. I don’t think

62:21 That’s the way I’m thinking right now. Right now, I think the way I’m framing this out a bit more is I’m using large language models to help me build many tools or starting points to get the answer right now. And I want I want to build repeatable objects that I can use that don’t require the co-pilot, but it helps me and get the initial build done faster. Does that make sense? Yeah, absolutely. And I I think there are a couple of interesting things. I’ll take us pretty far away from computation for two observations. You mentioned that

62:55 You’re learning how to use the LLM better. So, it’s training you in some ways. You said 100%. Yeah. And humans are always shaped by the work that we do. if you think of any [clears throat] physical labor like it affects the body. It shapes it literally shapes the body. so humans will always if we’re doing work repeatedly or repeatedly that work will also affect us how we think how we look at things and can affect our bodies and our habits of thought and our our physical habits. The other thought is in carpentry and traditional woodworking, most tools that a

63:28 Woodworker needs can be made with woodworking tools. And so in a traditional carpentry shop, almost everything that you would see would be made out of wood by the carpenter for the projects they’re working on. And I just keep coming back to that metaphor where we use something like an LLM to build little tools that we’ll use and combine in different ways. And it it reallyarkens to that where we become tools in addition to whatever we’re trying to do. We build small tools that we can use to accomplish ends. The woodworker doesn’t want to build tools necessarily. They want to, ,

64:02 Build furniture, cabinet, tables, things. Yeah. That’s the output. But you need tools to be able to do it. And in software where it it’s a very interesting place as well because like the woodworker the things that we use day in and day out can also be made be used to make tools. Mhm. So we can make our own tools to then accomplish our ends more efficiently. And we can shape those tools to be specific to us. If I’m left-handed, I can make a tool that is aligned to the left hand. And I can do that as a woodworker. I can I can make it perfectly. If I’ve got really big hands,

64:34 I can make the handles bigger. If I’ve got small hands, I can make the handle smaller and it fits me. And similarly with LLMs, we all have slightly, even if we do the same work, we have slightly different goals and focuses and things we excel at and others excel at. And so the idea that we can build tools that are custom fit to us. I think that’s where there’s a lot of power in LLM and using agents and getting better at understanding what is possible in a script or in an automation or in a tool and using the LLM to help you build that so that you can use it in the future. And so I think it’s this

65:05 Self-reinforcing cycle that’s that’s very powerful. All right, I think we’re at time here, so let’s do some final thoughts here. I’m going to give you a very random off tangent final thought, but Tommy, let’s go with yours first. Final thoughts. Honestly, we are at a point finally where that whole developer experience is available with Tindle. And honestly, Tindle’s opened my eyes to what’s possible. The great part too is all these fun open source repos around these terminal commands and coding things are actually now relatable and actually I can use them because of what Tindle’s

65:38 Been able to provide and I can have the full experience of orchestration and I don’t have to worry about binary files and doing things. I don’t use PBI tools as much anymore. But that being said, there is though there’s a skill in being able to orchestrate this. I I’m always going to go back, especially this conversation around we are in the conductor’s era of work and that there’s a skill to that. There’s a knowledge there and experience on what does it take to conduct the orchestra here. So, , but that being said, I

66:11 Love the conversation today. Greg, any final thoughts or or was that last thing your final thought? No, I’ll give you a really short one. You guys have had some really great guests on in the past and I think on Tuesday you had a gentleman on speaking about developer experience and the importance of being able to understand things and I think as much as we talk about LLMs and the ability to generate and I think that’s awesome and cool and there’s more cool stuff being done it’s also really important to have tools that help you to understand to visualize and show not just a report viz to show you the data but something that allows you to see what you’re doing and something

66:43 Like a git diff to see the changes the LLM has made or tools that help you to explore the artifacts that it’s made in addition to just the final output. I think that’s equally important and also a skill that we need to craft at the same time. It’s not just about getting text into a source code file. It’s also about understanding that text. It’s not just about building the model, it’s about understanding the model. And so it’s not just tools to build, it’s also tools to understand. And like that that man on Tuesday said, I think that’s part of a good developer experience. [laughter] I do agree with that one. All right. My final thought here is in all this talk

67:15 Around large language models, tokens, buying things here, what happens, gentlemen, at the end of the month, right? If companies are paying for large language models to do all this work and the beginning of the month, you get a reset of tokens, you just go along your day building all these things using large language models. At the end of the month, the last week, when you start running out of tokens, does the office go quiet? Does everyone leave the office? Do you just pay for more tokens to get things done? So, I believe agents are going to change how we as

67:47 Businesses actually do work and maybe maybe we’re going to go into a 3-w weekek work month because we run out of tokens too fast and we can’t get every all the work done that we need to at the end of the month. So, I find it to be a funny predicament that we may be in here in the near future where what happens at the end of the month? does productivity go down substantially? , I would probably echo if you’re doing major releases for your tools, do them at the beginning of the month because you have lots of prompts. [laughter] So,

68:20 Don’t don’t do them at the end of the month when you run out of prompts and everyone starts complaining. So, , anyways, a very random side thought, an in consequence of, , , monthly bills for your large language models. All right, enough enough said there. Thank you all very much for participating in this conversation today. Chat, you’ve been fun. Thank you. There’s been a couple comments there. I’ve been just, , pickling my funny bone. it’s been great. a lot of enjoyment there as well. Thank you all so much for listening. If you like this podcast, please make sure you become a member or subscribe to the

68:52 Channel. We’d really enjoy for you to become part of our community and engage more with these conversations. We do this because it’s super fun. We just enjoy our community around this one. And Greg, thank you very much for participating today. your insights and education, right? We we got into class here at the beginning of this which was amazing and helps me formulate more thoughts around how this is working and it’s it’s very encouraging to hear a real developer interacting with these tools and what your thoughts are forming around how these tools better and more

69:24 Effectively help you build in your daily workflow which is fun and interesting to hear as well. Tommy, where else can you find the podcast? You can find us on Apple, Spotify, wherever you podcast, make sure to subscribe and leave a rating. It helps us out a ton. Do you have a question, idea, or topic that you want us to talk about on a future episode? Well, head up over to PowerBI tips podcast. Leave your name, enter a great question, and finally, join us live every Tuesday and Thursday, 7:30 a.m. Central, and join the conversation on all of PowerBI tips social media channels.

69:56 Thank you all so much, and we’ll see you next time. That’s down.

Thank You

Want to catch us live? Join every Tuesday and Thursday at 7:30 AM Central on YouTube and LinkedIn.

Got a question? Head to powerbi.tips/empodcast and submit your topic ideas.

Listen on Spotify, Apple Podcasts, or wherever you get your podcasts.