Fabric Notebook Toolkit - Ep.571

Notebooks are where a lot of Fabric work already happens, and agents are getting better at writing them. Tommy Puglia and Mike Carlo look at the Fabric Notebook Toolkit, a pre-release Python library that lets GitHub Copilot, Codex, or Claude Code run that work from the command line.

News & Announcements

• All About the Power BI / Fabric MCP. The Chicago user group on October 22 is still the MCP night: install the Fabric remote server and the local Power BI semantic-modeling MCP, then use them. Tommy’s point on air is that the same MCP should show up in Claude Desktop, a Claude terminal, Copilot, and VS Code. RSVP with a full name.

• Why I built Skill Vault. Tommy’s post is the origin of Skill Vault, the open-source app he built because skills were drifting. His desktop had 257 skills, his Dell 160, and his Surface 158, and he could not say which copy of a semantic-model skill was current. The app keeps one vault and links it into Claude Code, Cursor, Copilot, and project repos, with a zip path for Claude Desktop and a Notion skills API for machines that do not share a folder. He built the CLI first so an agent can do anything the UI can do. On the episode he describes adopt, push, and the worry that an upstream skill (Fabric’s, or Kurt Buhler’s) changes while a local copy stays frozen.

• A new look and new capabilities in the capacity metrics app (preview). Tommy’s read of the preview is a monitor that can say which jobs ran off the reservation, and when. The old smoothing view told you something was hot without telling a new client what the number meant. He wants the app to be explainable to someone who is not already an expert in capacity metrics, because the first reaction he sees is “what does that mean?”

Main Discussion

Topic: What the Fabric Notebook Toolkit is for, and who should run it

The library is fabric-notebook-toolkit on PyPI, shortened to FNTK on the show. The Microsoft post they read describes end-to-end agentic notebook work from GitHub Copilot CLI, OpenAI Codex, and Claude Code. Mike flags it as pre-release before the news is even finished. The interesting part, for him, is bringing an agent he already pays for, over a CLI, instead of hoping the in-product notebook copilot is the only door.

• A CLI is the harness they already trust. Mike’s comparison is the work he has been doing in VS Code with GitHub Copilot: one agent inside the editor is fine, and a team of specialized agents that can talk to each other is better. Tommy’s Skill Vault writeup makes the same bet. Distill the process in commands first, then put a UI on it. If a desktop app such as Claude, Codex, or VS Code is open, the practical hack they offer is to tell it to install the Python package and then write a skill around the commands.

• The toolkit is a library of notebook actions, not a new notebook product. The shape they describe is command-shaped: say “run a notebook” and the agent calls the notebook command, instead of inventing a Fabric API dance with an app registration and a downloaded .ipynb. Mike had been doing that download-and-edit loop by hand. He reads FNTK as the precursor to a closer loop, where Claude, Copilot, or VS Code can create and run notebooks against the service. He also says the docs need a Learn path. Two methods and a paragraph are not enough to know whether you are supposed to live in the command line.

• Preview means play with it, not hand it out. Tommy’s governance line is awareness for users, and a sandbox for the people who will actually run it. He would not let a random person in the workspace build notebooks this way yet. Mike agrees from the leadership seat: a CTO, CIO, or the BI lead monitors a preview. They do not release it into the wild. General availability and preview are different promises, and the person who owns the platform has to keep them apart.

• Standards have to move into the prompt. Tommy’s client story is a team that asked whether AI was allowed. His answer was yes, with a reason: he can cover more ground than a room of developers, at lower cost, if the engagement includes the tooling. The failure mode he has already seen is code that runs and is not written the way a developer would write it. Their fix is to give the writing agent the team’s standards, and to give a second agent the job of reviewing the output.

• Hiring changes at the edges, not in the core. Mike still wants a data engineer who understands notebooks, Fabric, and why bronze, silver, and gold look the way they do. There is real disagreement right now about what those layers even are. He would still take a meeting with a novice who can explain medallion design, MCP, and FNTK and has not yet run it. Tommy brings back Kurt Buhler’s three Cs from the recent episode: creative, curious, and critical. Critical means asking whether the new feature adds value or only smoke.

• The quick start should say where it runs. Near the end they get stuck on the install path. If the intent is the GitHub Copilot CLI or the Claude CLI, the quick start should be the CLI commands, not a tour of the portal. Mike’s closing read is that Microsoft is finally putting agent experiences on top of Fabric things he already wants to use. Tommy’s is that this is another tool in the hardware store: new, useful, and not a replacement for knowing what you are building.

Looking Forward

Install the preview in a workspace you can throw away, have the agent write a skill for the commands you actually run, and read the output before you trust it.

Episode Transcript

0:00 Explicit Measures pumpin’ up, be it high. Tommy and Mike lighting up the sky.

0:06 Dance to the data, laughs in the mix. Fabric and AI get your fix.

0:11 Explicit Measures, drop the beat now. Pump as kings feel the crowd.

0:17 Explicit Measures. Explicit Measures.

0:22 Drop it loud. Whoo

0:28 Hello everyone, and welcome back to the Explicit Measures podcast with Tommy and Mike. Good morning, Tommy, how are you doing?

0:35 Good morning, Mike. what? Episode 571, not too bad.

0:40 Not too bad. We’re clipping along.

0:42 We’re almost to,, 600. Another even number here.

0:45 I know, I

0:45 You like it? Moving along?

0:46 We were just talking about how fast things are going, and I’m like, I think, didn’t we just celebrate 500?

0:52 Yeah. It’s, it’s,, w- we do roughly around, I guess it would be like 100 episodes a year

0:59 A year

0:59 … ish, just a little over.

1:05 So,, we just broke 500. It’ll be about a year since we get another hundred in the list, so here we go.

1:11 Anyways, awesome. Very cool.

1:16 Good to get going here. We’ve got some interesting topics today.

1:22 So our main topic for today will be around this new thing called the Fabric Notebook Toolkit. So for those of you who have played with it, please let us know in the chat.

1:28 Be,, let us know what you find about it. Do you like it?

1:33 Are you using it?, are you finding this to be somewhat useful? We’d be curious to see your opinion on this one, but this seems like this is gonna be a great,, addition to be able to getting,, notebooks inside Fabric more capable.

1:39 There’s a l- there’s a lot more, more CLI tools, more,,, session management, your code management

1:45 Mm

1:46 … Spark pieces, right?

1:49 So a lot more things here that are gonna be,, more helpful for us as we get into notebooks. And again, I absolutely love notebooks.

1:52 Notebooks are very cool. Really like it.

1:54 I will also note this is a pre-release, so it’s not

1:57 Mm-hmm

1:57 … stable enough yet for production use.

2:02 They’re still trying to refine and clean things up, but that being said, that was what we’ll be talking about today. All right, Tommy, give us some news.

2:08 Mike, we got a few things on the docket today. First, the next Chicago user group is happening next week, next Thursday, the 22nd of October, 3:00 PM, downtown Chicago, and it’s all about,, the Power BI and Fabric MCPs.

2:26 We’re gonna walk through how to install it,, both the local semantic model,, Power BI authoring one, and then also the Fabric Remote MCP. We’re gonna showcase what capabilities it has, and we’re gonna build both using the design of,, building out semantic,, models and measures and relationships, while also

2:45 Mm-hmm

2:45 … managing everything that we have, doing a recon session in Fabric.

2:52 So we’d love to have you. Make sure to RSVP before Wednesday of next week to make sure that you’re in with your full name in your Meetup username, or p- just provide it when you actually sign up, because that’s very important.

3:00 And we’re really excited to keep going on on our monthly cadence.

3:10 You’re gonna enjoy this one. I will make a note around

3:13 Mm

3:13 … some experiences that I’ve been having inside VS Code with GitHub Copilot and trying to get my agents to do what I want inside VS Code.

3:20 It’s been a little bit of a trickier,, I’ll just share with people ‘cause I’m finding it

3:27 Yeah

3:27 … challenging.

3:33 One of the things I’ve been trying to do in VS Code is,, there has been the Power BI modeling MCP server. And so there is the extension on how you build it.

3:39 Mm-hmm.

3:39 I believe there’s also now a plugin that they’re moving towards, and so I’ve— So a couple things. First things first, when I’ve been using VS Code, I’m like, “What happened to the chat window?

3:45 The chat window’s all different now.”

3:53 Well, for whatever reason, when, for some reason, the VS Code has now changed its default interaction, and this is something I didn’t, I wasn’t aware of. When you go into VS Code and you open the chat window, previously it was always set to local mode. Local mode

4:09 Mm

4:09 … had different ways of turning servers on, MCP servers, all kinds of things.

4:17 So it did its own version of how it handled extensions, essentially. Then, I don’t know, last week sometime, they dec- they decided to change the default to not be local anymore. It’s now cloud

4:27 Mm-hmm

4:27 … version of whatever the thing runs as.

4:33 Which is fine, it just now means I’m very confused as to, like, what’s going on. I didn’t understand the UI, the whole chat window experience

4:38 The agent window that they have now.

4:39 Yeah. It changed a lot more now.

4:45 The agent window’s what they’re using to, to use this new,, how to get your plugins working or whatnot. It just looked very different to me and I was like, “What’d they do?

4:51 They changed everything. I can’t find anything now.” Well, I was on the wrong mode.

4:58 So odds are, if you’re using VS Code and you’re playing around with it, you may have seen that it changes it. You just have to change your VS Code back to local only and not cloud mode, and then you’ll get all the original stuff that you were familiar with and you’ve been using for months now, you’ll get it all back.

5:04 So just FYI, I just wanted

5:10 Yeah, for Cursor, by the way

5:11 … I wanted to point that out.

5:13 Yeah, those things have been out for Cursor for a while. But no, I, it isn’t— VS Code’s becoming — I still, I know we had this conversation, and I even looked it up after, that it’s not an IDE, but what are we doing here?

5:23 If it’s not an IDD,, it’s closer to an IDE than it is closer to a text editor or, or whatever they called it, a hypertext editor.

5:35 Yeah,, I

5:37 If you

5:37 I understand your, I understand your point, Tommy

5:39 … Nope, but I

5:41 But an IDE is super bloated, and there’s a lot of extra stuff that comes with it., even just the size of the programs,, I get the fact that there’s extensions.

5:50 Totally.

5:51 But when I look at it, it’s just,, w- I’ll say this. On our social medias, when we started talking about whether or not VS Code was an IDE or not, everyone had an opinion on this one.

6:01 So there’s definitely, this is a hotly debated topic. No one really knows what it is. It’s, it, but

6:07 No

6:07 NoTechnically

6:08 Right

6:08 … it’s a text editor with a lot of extra bells and whistles, but

6:12 Right

6:12 … it feels, to your point, a lot like an IDE.

6:16 Yeah, so but no, we’re really excited for that user group. And yeah, it’s interesting with the…

6:24 We are gonna show a VS Code as well, but I, I do wanna make sure during the user group… And by the way too, if you have anything you wanna share, any wins or things you’ve been working on that are cool, we’re gonna have time for that too.

6:32 We’re gonna show it in Claude Desktop, we’re gonna show it in Claude Terminal, we’re gonna sh- or,,, get a Copilot. We’re gonna show it in VS Code just to show that you can use the MCP anywhere, and I think that’s also an important thing.

6:41 You don’t need to use VS Code to do so, and it works just as fine as long as you have an agent and skills.

6:49 Mm-hmm.

6:49 That being said, it is a lot easier to install in VS Code than it is on any other tooling.

6:57 Yes.

6:57 But that’s- But because there is, like you said, just the extension or really the plugin rather than actually having to go in and,, edit a JSON file.

7:06 Yes. And I’m, a- again, I’m gonna go back to like this,, I watched the doc- the documentary.

7:10 Tommy, will you watch the documentary around VS Code and

7:14 Yeah

7:15 … and did a, a show about that one.

7:23 VS Code traditionally has been very well known. Like, it’s been sh- Up until this point, VS Code has had a clear developer path.

7:31 They knew what they were building, they knew how to build it, they knew what the developers wanted. AI is a brand-new world for developers, and it’s also a brand-new world for the developers of VS Code, and there’s a lot of things here that they’re not sure about yet.

7:39 We’re still trying to figure it out, understand what is happening inside VS Code and what does it actually mean to be a developer with AI moving forward?

7:47 Mm-hmm.

7:48 There’s a lot of stuff changing here, so I think this, we’re entering a new era of VS- The, the developers of VS Code don’t really know what is the right level of experience they should provide to people who are writing code now moving forward with agents, and this is why I think you see all these things shifting and changing and,, as soon as one company comes out with something, another company’s immi- immediately copying it and making it a d- a version of it.

8:02 I will say this: The new version of Cursor and GroqBot, favorite.

8:17 Yeah.

8:17 Like, it’s so capable. It’s automating so many of my tasks., agents and AI are good at single tasks.

8:27 I would say, like, it’s the inside VS Code doing something there to help you build something. But really where it excels is it is doing a way better job when I have a team of them and they’re specialized and they can talk to each other and I can get multiple agents to do a specific thing for me.

8:36 Man, I really like that experience.

8:46 Yeah. And by the way, another reason why it’s closer to an IDE than Notepad or a text editor, I’ll hold my breath until Microsoft releases their documentary on the creation of Notepad.

8:59 Notepad?

9:00 No- Note Editor, Text Editor, the Notepad app on Windows.

9:04 Yeah?

9:05 They’re not gonna make a documentary about that, so I’m gonna hold my breath.

9:08 Oh, I see what you’re saying.

9:09 So just to say

9:09 You’re basically saying like

9:10 … the fact that they made a documentary about VS Code and all the intricacies

9:14 Yeah

9:14 … the documentary for Notepad would be five minutes long., we create a program and you could enter text.

9:18 Ta-da. And that would be it.

9:21 That would be the documentary at this point.

9:25 Yeah, but it’s not as a… Yeah, but it’s not, it’s just not a,, Notepad isn’t, like, impactful for every…

9:31 It’s not used by…, it’s used by a lot of people, but it’s not like billions of users building new things in it. It’s not as

9:36 I know

9:36 … exciting as

9:37 Which sounds

9:37 Yeah

9:37 … like an IDE to me. Yeah, so.

9:40 Okay. Agree to disagree.

9:41 All right, so that’s our first one. Mike, I have a new article on promptingbi.

9:55 com. This is based on our meeting, or not meeting, our live event that we did on SkillVault.

10:08 So in case you’re not aware,, there is a open source application available on GitHub called SkillVault, and the general gist of it is it’s an ability to sync, manage, adopt all agent skills from acro- across the land, manage your own skills, and make sure they’re synced across not just devices, but also harnesses.

10:21 I built this because this was one of the more frustrating things as I was building skills and managing skills, going, “Well, how do I know the skill’s fabric updated? I wanna make sure that all my harnesses are updated., what’s the right version?” Or if I’m creating my own skill or edited it, I wanna make sure it’s the same skill in Claude Desktop as it is in VS Code, as it is in Claude Code.

10:36 Mm-hmm.

10:36 And this is not easy to do. So SkillVault is a way to actually manage,, create a skill repository, so to speak,, look for stale clones if you are adopting it or cloning from, like, skills for fabric, the Microsoft repo, or anywhere else you get your skills.

10:47 You can also use an AI assistant that allows you to push skills to the right harness, find skills that are stale, or even just discover new skills entirely saying… I, what, what did we do for the video, Mike?

10:57 What was it called? HV,, on the demo? The

11:08 Mm-hmm

11:08 … the thing that…

11:10 Yeah. So you’re like, “Hey, I need skills that do X, Y, and Z,” and it’s gonna find the top

11:13 Yes

11:13 … rated skills or the top part.

11:14 Hyperframes, I think is what it

11:15 Hyperframe

11:16 … what we used

11:16 That’s it, Hyperframe.

11:17 What we were trying to… Yeah.

11:17 Wasn’t terribly off. Hyperframe, and it, and the my, the agent that’s created underneath it using skills that are

11:25 Mm-hmm

11:25 … from SkillVault, so it’s a, a redundancy there.

11:33 But it was able to find everything, and then just adopt in the SkillVault and then push it to wherever you need it to. And using that with Notion’s,, skills API, you can push that to Claude Desktop really easily too if you need to.

11:40 So that article’s out on Prompting BI as well, and just basically why I built it, and showcases really how to use it along with the video that we did.

11:51 All right, excellent.

11:52 And then

11:53 So that’s one out there is what… That’s the why I built SkillVault.

11:57 Mm-hmm.

11:58 , collect your skills, get them all into…, it’s like Pokemon now. I feel like I’m, I’m always running around., I feel like for a while, Tommy, when,, early on in the AI space, it was- I literally felt like I was trying just to keep pace of like, “Oh, that’s a good skill.”

12:14 Yeah.

12:14 “Oh, that’s a good skill. Oh, this is interesting.

12:16 Oh look, here’s a library of 150 skills. Oh, here’s 300 skills.”

12:20 How similar are these skills? I don’t even know.

12:28 I just kept running around trying to collect more and more things thinking, like, I would lose them and, and I would be uneffective or inefficient if I didn’t have all these skills attached. So I definitely feel like it was like a,, Pokemon collect-them-all a challenge for me

12:35 Yeah

12:35 … initially to get all the skills together. And then you

12:38 And then you feel there’s almost, like, FOMO for not finding out new skills that are out there. Like, “Oh, how did I not know about this one?” And it could be har- like, overwhelming.

12:47 Well, yes, but I think it’s even more than that. I felt, I feel like it was even more, like, impactful in that,, if you didn’t find that skill, you were wasting tokens. Like it was

12:59 Mm

12:59 … almost like a, an efficiency gain too.

13:03 Like I, I was trying to prompt my AI to do something. It doesn’t quite do what I want.

13:07 I want better performance from it. So if I have these secret skills or learn these skills or give skills

13:11 Mm-hmm

13:11 … to my agents, they’re able to become more capable by default.

13:15 And so I look at this going, “Oh my gosh, this feels like a hack to me,” right?

13:21 And so maybe that’s why I’m going around, like, trying to collect all these things, ‘cause I’m like, if I don’t collect all these little pebbles a- and bre- and put them in my repertoire or don’t have them, I’m gonna lose those efficiencies. I’m gonna lose,, the ability to be,, token-wise based on these other tools.

13:30 So that’s why I was maybe, like, paranoid a bit

13:39 … initially

13:40 Yeah

13:40 … at the beginning.

13:41 Do what you’re describing to me? You’re describing to me…

13:47 Remember when, like, you download a game when they first came online, and then you could add the option to, like, download the content pack or like

13:52 Sure

13:52 … you used to.

13:56 So it had, like, the default adventure, but you’re like, oh, if you want you can actually open up a new road this way, and it’s a whole

13:59 Yes

13:59 … new thing.

14:04 And for me s- I th- for skills, for me you’re, like, installing new applications in terms of, like, what their capabilities are. It’s not just a nice way to do it or a better way.

14:10 Mm-hmm.

14:10 It changes how the harness I am is working, and I think that’s why s- I see them as so powerful. So a- and not just so powerful, but th- there are some of the skills that I have right now that I would not be able to do what I do today without them.

14:20 So it’s, yeah, you’re like you wanna know all the top skills and what they can do, but that’s hard to manage.

14:31 I 100% agree with this. And then,, maybe you use them, maybe you don’t.

14:38 Hmm. Yeah.

14:40 Right? Now, that’s a whole different problem, ‘cause once you have too many skills, you’re like, which one do I use?

14:44 What happens when there’s duplicate skills? Which one fits better with your workflow?

14:48 Right.

14:48 All kinds of problems come along with this one. But anyways, regardless, you still need a skills vault to collect them all, put them together, and then you can figure out what to do with them after you have them.

14:56 So regardless, it’s still a good, a good feature there. All right,, let’s talk about one more other item here, Tommy.

15:05 Yep.

15:05 I think you found an article here around,, there are new capabilities in the Fabric Capac- Capacities Metrics app.

15:13 Yes. So we have

15:14 I wanna not only go there

15:15 Yeah

15:15 … but I…

15:19 Yes, this is what I wanna talk about. I wanna also, Tommy, have you recently gone into the admin portal of Power BI?

15:23 I have.

15:23 And looked at the capacities in there?

15:26 Yes. Is there something that I’m missing now?

15:29 Maybe.

15:31 Okay. The capacity settings, you mean?

15:34 So if you go into your admin tenant settings now

15:37 Mm-hmm

15:37 … I’m just curious if you have, have you been in there recently?

15:39 The capacity settings on your tenant.

15:42 Try… -huh, huh. Oh, yes.

15:45 These

15:45 I see this now.

15:47 Yeah.

15:48 The c

15:48 There’s a

15:48 Are you talking about

15:48 There’s, there’s a s

15:49 … the manage capacities?

15:51 Yeah. Well, there’s a, an option there.

15:56 There is,, y- you need to investigate a capacity s- you can try capacity monitoring inside the admin portal.

16:01 I saw that, mm-hmm.

16:02 And then if you click on that, and in the capacity monitoring pieces, when you try that, you get this really beautiful new grid view of your capacities. So it’s a little bit, it’s, like, a little bit hidden in there, but it’s inside the portal, and it’s slick.

16:23 Oh. what, Mike?

16:25 It’s not just that, but there’s also this OneLake catalog for governance that you can do.

16:29 Yes.

16:30 So the new monitor experience, yeah, I see the new monitor experience. Look at, hey, this is looking a little fancy.

16:37 There’s a little bit more around, like, seeing your health over time and figuring out when your capacity is overrun or when it’s not, or when it’s being throttled or if it’s not., so there’s a lot, there’s a, there’s a lot of really, I think, better messaging

16:52 Mm-hmm

16:52 … and easier to drill into the new capacity pieces.

16:58 So anyways, I just wanna point this out, that there are some new features coming even in the monitoring side of things. So you have,, job runs

17:05 Oh

17:05 … you have capacity

17:06 Mm-hmm

17:06 … management.

17:08 It’s all coming to the admin portal., it’s, it’s really pretty slick. I’m

17:11 Well

17:11 … I’m liking this new

17:12 I like the job runs, yeah.

17:14 Yeah.

17:14 I like the run history too.

17:16 Yep.

17:16 That shows… Yeah, I like just, again, conveniency.

17:18 We were just talking about that offline.

17:21 Yes.

17:21 This makes conveniency. Oh, look, agents are in here too.

17:25 Yep, applications.

17:26 Applications. Yeah.

17:27 Yep, exactly right. So it’s giving you s- like s- more of a central place to manage some more of these, like, critical things,, which I think is very useful here.

17:35 Anyways, all this to say is that I’m liking these changes that I’m starting to see in the new monitoring hub for admins.

17:43 Hmm.

17:44 Pretty cool.

17:44 Have you seen the new OneLake catalog, the governance too? That’s the new thing they’ve been promoting.

17:49 I haven’t played with it as much as I should have, Tommy, but yes

17:52 Okay

17:52 … I am, I’m aware that it’s existing.

17:57 I haven’t, I don’t have enough the time and seat on it to give an opinion about it yet. But it’s getting…

18:02 I think I’ve, I think this is reaffirming my point, which is, look, I don’t wanna go over to Purview to go do stuff, right?

18:09 All, all, all of or most of my data is now going to reside in Fabric anyways. Fabric is becoming the back end-…

18:13 for what I wanna use, period.

18:18 Yeah.

18:18 This is what I wanna do now.

18:20 Dude, very

18:20 So having a OneLake that has, like, a better visibility to all the items that are in my lakehouse and in my reports and all that, that makes a lot more sense to me.

18:30 This is interesting. I like this a lot, and they even have now,, like a Power BI OneLake catalog report to say, like, data freshness, item curation.

18:40 Yes.

18:40 This was definitely… Oh, I love this. I

18:43 Okay

18:43 … and here, here’s where we need to go.

18:47 This is the type of report, Mike, that I wanna s- go in my weekly meetings with my BI team and to make sure that we’re all on the same page.

18:53 Yeah.

18:53 Like, this is what I want on the screen during the agenda. Oh, man.

18:56 Yep.

18:56 That is great.

18:57 If, if nothing else, it’s, it’s at least an awareness part of this, right?

19:00 Yeah.

19:00 “Hey, here’s how we’re doing on performance. Here’s what’s going on.

19:06 We had a couple jobs run off the reservation. Here’s where they are and when they happened.” Like, just be aware what you do impacts the broader part of the organization, and we’re watching this.

19:11 We can help you. Like, reach out if you need assistance, but, like, you just can’t just go build whatever the heck you want

19:17 Right

19:18 … and hope it all works.

19:20 Dude, and that goes along with the new Capacity Metrics app, and

19:24 Yes

19:25 … there’s a few main things here that, and this is, again, on the blog, along with this that

19:30 Mm-hmm

19:30 … just makes me happy.

19:35 I love the governance side. I love the, the, the, our data governance, man.

19:40 But so there’s some main goals here that, and main feature updates that we’re able to see. First, it’s a completely redesigned interface.

19:45 Mm-hmm.

19:45 A better navigation, and different, new metrics focus on helping administrators m-, understand their capacity health. They have, apparently the redesign and the updates were trying to accomplish three main goals: detect issues as they occur, investigate problems for the previous 14 days, and identify capacity risk before in- the, the throttling impacts users.

20:01 There’s a health page to see cross capacity to identify which capacities need attention. These include metrics that are easier to understand, such as indicators of percentages of time spent in throttling.

20:17 So rather than those weird CU ones, you’re like, “I know that’s the most, but I don’t know what the most means,” which a lot of times when you’re dealing with,, new clients, Mike, I’m sure you see this the first time they see the, the report. They’re like, “What does that mean?” They’re like, “All right, how do I explain this?” It’s like, that just means a lot.

20:32 It’s the mo- it has the most of a lot, and that’s the best way to explain it, so yeah, we needs a little more layman terms metrics here. There’s a better compute page with,, several capabilities, a utilization graph, shows consumptions in CUs by seconds instead of percentages.

20:48 So scaling changes m- are easier to understand. A heat map visualization for things that are, are utilized or over-utilized hours, and in a CU consumption table identifies that item’s contributing.

21:05 Mm-hmm.

21:05 There’s a details page, and then also this is,, being released alongside the GA Capacity Metrics app. So this allows customers and organizations to evaluate the new one while keeping the old one.

21:13 They don’t have to upgrade their existing one.

21:20 Yes.

21:20 So Mike, I, I love every single part of this. I love making this easier, I love the design.

21:28 Because again, not everyone has that IT background to look at, or more importantly, is an expert in Fabric capacity metrics.

21:35 Yes.

21:36 Which you had

21:37 This

21:37 … to be back in the day.

21:39 And, and there, they’re, they’re, they’re getting to the point where it’s like we’re actually able to detect, like, where there’s problems. The smoothing thing was good, but the smoothing thing was also difficult to understand.

21:49 Mm-hmm.

21:49 I understand why they were doing it, but also there, it came with a lot of headaches, I think in general

21:54 Yeah

21:54 … of just being able to like, okay, you’re giving me this stuff, things.

22:02 I just don’t understand how to get back to like, okay, what individual item is ripping through my capacity and overusing something? That, that’s the part I really needed, and sometimes you needed this unsmooth amount of, like, what jobs were occurring on my tenant.

22:11 And I think this, we’re getting much closer to better diagnosis of, like, why things are falling apart.

22:19 Yeah.

22:19 Anyways.

22:20 100%. Dude, I’m very excited about this.

22:23 Just again, just make things easier, please. That’s all I’m asking, so awesome.

22:26 Anything else with the Capacity Metrics app?

22:30 No, I think those are the two main updates I think were coming out. Both of those I believe were announced in Barcelona.

22:36 They were touting these things there. They’re now starting to show up in your tenants, so go check them out.

22:42 Go download the nos- most recent version of the Capacity Metrics app. I think you’ll be pleasantly surprised.

22:47 Feels a lot cleaner, feels a lot better, so I can’t tout how much benefit that these new features have been coming out with. Like, it, I just…

22:53 The team has been working hard, I’ll say that.

22:59 Yeah.

22:59 And it, it shows, I think. I think this really shows that they’ve been working hard on this, and it is, they’re doing a really good job of getting this stuff like knocked out.

23:06 So anyways, all that to say is check it out, go look at the new workspace. I think you’re gonna like it.

23:14 All right. So Mike, I think it’s time then, let’s dive into our main topic today, because there’s a lot going on.

23:24 There’s this new CLI tool. This is not necessarily, and I think this is important here, this is not necessarily meant for the,, Pyth- or the n- in your Fabric environment.

23:33 Yes.

23:33 You can run this in your terminal anywhere, right?

23:37 Yep.

23:37 This is a command line tool for Windows or,, believe it w- it work with Linux. But it’s important here because up until now, most of the packages that have been around the notebooks were things like the,, the notebook helper or what was it called?, the no- the Fabric helper library

23:57 Mm-hmm

23:57 … which is basically, but can only be run in a Fabric environment.

24:01 It can only be actually run if you were using VS Code that had the extension

24:05 Yes

24:05 … or in the actual UI of a Fabric notebook.

24:09 Mm-hmm.

24:09 What was been released last week, or we talked about it last week, is this new Python library called Fabric Notebook Toolkit, FNTK is their shortcut for short. And the f- Do, how, how are we gonna refer to this so I don’t have to say the whole thing the whole time?

24:18 ‘Cause even FNTK is a lot.

24:27 , yeah. No, I would just call it,, the, the, the Notebook Toolkit is what I think

24:34 Okay, let’s call it the Notebook… Let’s just call it the toolkit for today. So

24:38 Let’s, yeah. So when we

24:39 I’m already struggling to remember my word. Yeah.

24:41 Right, exactly. So we’re talking about this toolkit.

24:43 We’re gonna, we’re just gonna call it the toolkit.

24:45 Right.

24:45 But it’s actually the Fabric Notebooks Toolkit

24:48 Right

24:48 … that we’re actually gonna be communicating about. Okay.

24:50 So if you skip

24:51 I think that’s good because

24:52 Just go back

24:52 … it’s a lot of things going on here.

24:54 Yeah. So the toolkit here is, one, can be run on any machine that, as long as it can connect to a Fabric capacity, but it doesn’t have to be

25:03 Yes

25:03 … in a Fabric environment. So that’s the first

25:05 Correct

25:05 … distinction.

25:15 Two, it can manage and run notebooks, but also execute code from your Fabric capacity in its own environment, in its own environment. And I really, the third one, Mike, is this library is meant to support end-to-end agen- agentic development of Fabric no-, notebooks

25:26 See

25:26 … possible from GitHub Copilot and Codex and Cloud Code. So it is

25:31 That’s, that’s the part

25:32 … a model agnostic, harness agnostic here.

25:36 That, to me, is really the big part here. So Mike, there’s a lot of things I think unpack with this. Where do you wanna start?

25:43 , well, I th- I think this is a pretty big shift in notebooks just in general.

25:48 Mm-hmm.

25:48 The fact that there’s now a, like, the idea is bring your agent to notebooks, I think this is a pretty big shift in general. Up until this point, we really haven’t had anything really useful other than Copilot and the service maybe to help you out with some notebooks.

26:03 Mm-hmm.

26:03 But I’ve also found like for me personally, like, I don’t really like to use the GitHub Copilot or the, the Copilot that was coming with Fabric to work on notebooks.

26:11 Yeah.

26:11 It wasn’t really able to read the notebook instructions. It didn’t seem to have a really good understanding of, like, when you ran a cell and it failed, what was the failure? Like

26:19 Mm-hmm

26:19 … those are the activities that I wish that the Copilot inside Fabric would use.

26:25 The rub for me is I’m already paying for an agent somewhere else in my environment. So why do I need

26:32 Mm

26:32 … to go pay more CUs to go talk to a notebook when I already have an ample subscription that I’m going to be using anyway?

26:42 So I wanna be able to bring my own agent to, like, the work that I’m doing in notebooks, and this to me feels like a bit more of a release to that, where I’m able to bring whatever agent I already have or I’m paying for, show up with it, and then have the agent talk with this u- using the CLI, which I think many people would argue CLI is extremely powerful.

26:52 Some don’t love it, but some, some others really think this is the way to go. Anyways.

27:02 We know people who do. We know people who swear by it.

27:09 Kurt. So thank you.

27:15 Which and I agree, but, and I think, Mike, this is… I think we need to talk about the fundamental shift here because to me it almost feels like the Fabric Notebook UI, the default way you manage notebooks, is almost becoming optional at this point, which is crazy when you think about that.

27:22 So at least in my opinion here, again, I’ll say that again, the notebook UI, going to app. fabric.

27:28 powerbi. com or whatever it was

27:35 Mm-hmm

27:35 … and you manage notebooks there, okay, you really had no other choice besides VS Code, which was already a lot.

27:45 But now that notebook UI to me almost feels like like again, this is gonna be a hot take or a hot initial take is, it may, at this point it’s like you’re a beginner if you’re going to the notebook UI because you don’t know about these other tooling here.

27:54 Then so I guess let me s- ask you that. What’s your take on, do you think that the UI is optional now?

28:08 I’m gonna, I’m gonna go back to what I think we talked about, Tommy, earlier, which was,, how do you devel- how do you develop apps anymore? Right?

28:16 When my, my traditional method of building an application was like, okay, let’s think about the user, the person involved with building something. I want the user involved., what UI do they need to land on?

28:25 What’s the home screen? What does the educational path look like?

28:34 And just walking the user through screen one, screen two, screen three. So a lot of my initial app design was really centered around how can I bring a user interface in front of the user so that they’re comfortable and they know what to click on, when to click on things to get things done, right?

28:42 That, that was the mentality of which I was building. Tommy, you were just talking about it for Skills Vault.

28:50 You started with the CLI. You started building the agent side of things.

28:59 How would I… What, what workflows would need to be written out in code first?

29:08 And then you wrapped around a UI around it and said, “Okay, well, now, now that I have it working, I can actually move in and out-”

29:16 Yeah

29:17 … “the skills that I need.” You, you basically distilled the process down in code first and then built the UI around it. Now

29:27 Yeah

29:27 … I don’t know if this is

29:28 This is a big point. Yeah

29:30 Right? So the, how I’ve traditionally built apps was focused on the people first and then the process and weaving that together.

29:40 The n- this new world, I believe, is because the agents are so much better at using tools than I am, and the fact that I need to have my agents and me use the tools, you should probably start with the thing that’s common across both surface areas, which is

29:51 Mm-hmm

29:51 … an agent.

29:57 And the interesting thing is it’s easier, Tommy, I think, to build a CLI tool for an agent, right? I can give the agent

30:02 Yeah

30:02 … a single markdown file.

30:03 Well

30:03 Here’s what this project does, and then here’s all the tools you need, and here’s what they do. You, you now know when to use the tools at the right time.

30:15 And, and when you describe it that way, you’re now focusing only on the process, the workflow, giving the agent exactly what it needs to be able to do the job. And I think that’s much more efficient because you can always come back and build a UI later on. But

30:28 Well, and yeah

30:29 … the agent needs to understand what the process is.

30:31 Well, let’s be clear here. Most of the action that are happening that you think are so cool on your machine if you use Claude Cowork, which has now gone away, it’s just Claude now.

30:41 But if you’re running on Claude Desktop or VS Code, their CL, it,, every session has a terminal running in the background, a console that really your harness is operating on.

30:52 Mm-hmm.

30:52 And this is again, whether it’s Notion that doesn’t have a machine, but it’s still executing Python scripts that’s coming from a CLI, right? It’s running from a console, be it Linux or Windows.

31:01 Most of them have like a Linux,, environment. You just don’t know that, and you, most of the times, Mike, you don’t need to know that, right?

31:10 It’s not really essential, but knowing that and knowing that that is the integral part of how these machines look in an Excel file, right? They’re not just reading an Excel file and changing it.

31:20 It’s all through scripts and co-, commands. Also, the, the commands help an agent understand the actions, right?

31:30 Mm-hmm.

31:30 That’s basically what commands are. It is what actions do you wanna take?

31:45 For example, with the toolkit, I almost said the whole thing, it has a bunch of different main commands or domains. Discover, find notebooks, notebooks, lakehouses, and capacities, notebooks, manage notebooks, sessions, and lakehouses, code, submit code, retrieve re- results, and request cancellation,, cells, read and edit notebook cells, monitor, logs, resources, server,, initiation, or,, install or update skills, and inspect the built-in skill content.

32:00 So those actions allow it to, in a sense, run a lot easier because it knows if you say, “I wanna run a notebook,” okay, it’s gonna use the notebook command, right?

32:16 Mm-hmm.

32:16 And it’s like, can we check what’s actually what the data is for this lake house? It knows to use the code command, and it’ll build the code and run those actions.

32:28 So we’re getting to this point now where m- I’m finding, Mike, for the first time, a lot of what Microsoft is building for Fabric and Power BI are very Copi- or A- AI agnostic. I think initially it was like, “We’re building everything for Copilot.” It’s the Copilot UI in the Fabric UI.

32:40 That’s how you were gonna build. That’s the gonna b- And we complained about this., what, I know last year, the beginning of this year, we’re, Copilot’s fine, but I like using my harness.

32:53 And you’re seeing both Power BI Desktop Bridge, Skills for Fabric, and now the FNTK or the toolkit.

33:06 Yeah.

33:06 They’re, you don’t have to use Copilot. You don’t have to use your Copilot capacity to do so.

33:16 And I think it, be clear, you can run this yourself, but if you are using the toolkit, you are using this specifically or 99% of the time with an agent. That’s the intention.

33:25 Oh, yeah. I think that’s the point, yeah.

33:26 Right? Not just our recommendation, that’s the intention of why it was built.

33:30 Yes.

33:31 And I guess with that in mind, are we really moving to this phase where… I, I was on a call last week, and they asked me during the call, it was a prospective call,, and they were like, “Are you gonna use it?

33:51 Would you use AI to help develop?” So I said, “Absolutely.” And I said, ”, if that’s okay with,,, with you if, if we were to engage together or have a partnership,” but, and my reasons were, “Look, I can help you out a heck of a lot more than an individual or team of developers at a lesser cost.” More importantly, I’ve been u- using these A, these AI toolings for now the past three years and trying to make sure how it works and spending a lot of time not being billed, just making sure that I know how these skills and harnesses work.

34:10 You can run this yourself if you’re just getting started. But Mike, the, find those errors and revisions, you gotta know about this stuff.

34:29 So I’ll, I will land the question to you about you as a consultant, or it’s easier even if you as a professional, right, being posed the question with all these tools out there, they say, “Are you gonna use AI?” What’s your, I don’t wanna say how do you justify that, but how do you respond in, I think, the right way?

34:52 ,, if, if a c- if a client, if I walk into a client and they say, “Hey, we wanna work with you,” and they say, “Well, we’re, we don’t allow the use of AI,” immediately it’s like, “No, thanks. I’m not wor-”

35:05 Yeah.

35:05 There’s enough

35:05 Yeah

35:06 … there’s enough, there’s enough companies out there right now that are interested in using AI.

35:15 To me, it’s a non-negotiable at this point. Everyone should be using it and needs to be using it.

35:25 Now, the question I think becomes like, okay, well, whose information or where does that information go?, my recommendation on this one is,, make sure there’s some company policy around the AI that the, the com- the client is providing. And the safest way is the client provides you a license to whatever AI they’re using, right?

35:34 Right.

35:35 , a lot of our clients use GitHub Copilot. Great.

35:45 Easy, easy way to license. Here’s my user ID, here’s my user information.

35:56 You, you create a user account around that, and then anything you use or do with that agent is all now stored, used inside the client’s environment, which I think that’s actually the right approach, right? Companies need to approach this with, we have AI, we’re going to be using it.

36:07 May not be effective, but,, what I will so that— Well, what I will say though is,, the outputs of the AIs, I think there’s a lot of hesitation from companies to let it just build. And,, I continually see, Tommy, I continually see a lot of, every time I see someone saying, “Man, we just built this new app,” or, “We’re building things with AI now.

36:17 Look how fast it’s making us go,” there’s so much messaging on the internet around, well, how do I maintain it? What does support look like?

36:27 And I think that is a concern, but I also want people to be very aware that the ease in which it was used to build the AI to begin with

36:38 Mm-hmm

36:38 … is also the same ease and ability for you to update it and create tests around it, and actually have it become more robust.

36:50 So I think we’ve hit a point, Tommy, where, and, and this is maybe a general concept. Previously, I would use AI, and I would give it some code, and it maybe would write

37:01 Yeah

37:01 … code as good as I was writing, but over time, the code would, like, start to degrade, and we’d get

37:08 Deteriate, yeah

37:08 … repeated functions.

37:19 We’d get,, things that were not designed well., it, it would make it work, but the code wasn’t written, like, in a developer fashioned way. What we’ve been doing internally for us has been developing standards around, okay, well, we have these standards on how we would write code.

37:30 We have principles we use to dev- design those. We’re now giving those principles to the AI that writes the code, and we have a separate AI that reviews the output of the code.

37:41 And so one of the big checks that we’re doing is if anything is this new pull request or any- if any of this new code is coming in that is a copy of existing code, reject the pull request. Go back and say

37:52 Mm

37:52 … “This is something that should be a function.

37:55 You should have made this a component. You should, like, adjust the code

37:58 Right

37:58 … so it actually has, like, if not the same level of standard, but slightly better standards as we go.” And I think we’re getting to the point where if you have good enough instructions and you have a good enough process in place, you can get code that actually retains its quality over time.

38:12 And if you work really hard, I think you’re even making code that’s better every time you touch it or ev- better every time a round of agents is using it. And I think that’s where I’m at right now is I’m

38:25 Do you

38:25 … getting to the point of

38:26 Yeah

38:26 … it’s getting better every time we touch it.

38:28 I,, I, I would feel that’s even being very conservative. Do you remember when you started first using AI around developing?

38:37 Like, I remember the try- doing with notebooks, and at the time, it was much more you would select the code you want changed and say what you would want the update to be.

38:46 Yes.

38:46 It was very micro, right? You’re like, “Change this so I can return this.”

38:50 Yeah.

38:50 And that worked really well, but you couldn’t really productionalize that, nor did you feel comfortable trying to do a whole project that way, right?

38:57 You, you were babysitting the entire time that you were using anything.

38:59 You’re, yeah, you’re babysitting the whole time, and it was much more likely that you would run into an error over a long period of things. Like, I need an,, create an AP-, a connection to this API.

39:12 Okay, now it has that. Now what do I have to do?, there’s still a lot of time, but with the,, really the, I think the rise of skills and the rise of harnesses that can take advantage of them, and al- and also really the rise of MCPs.

39:25 Mm-hmm.

39:25 We’re at this point now where I, and I mentioned this on LinkedIn and Twitter too, Mike, I said I’m, I’m more and more resolved right now that what we’re gonna define as our skills or the agent skills or even the skills that we have

39:41 Mm

39:41 … the user skills should be referred to as conductor skills.

39:46 I read this 2023, and someone said, “We are in, welcome to the conductor’s era.” And I’m gonna keep

39:51 Yeah

39:51 … saying that because

39:52 Yep, I think you’re right

39:52 … the whole, the premise there was you are now the conductor in the orchestra.

40:01 And guess what? That person needs experience and the skill to lead that team.

40:09 And I am more and more resolved that where we’re at now, me as a professional, as an individual, and then you can add consultant and whatever you wanna on top of that, that my ability, you have to trust someone that they know what they’re doing with all these agents.

40:19 Sure.

40:19 Because anyone can tell an agent to build a notebook and a pipeline and use skills.

40:24 Mm-hmm.

40:24 But how do you verify? How do you make sure things are running?

40:27 Do you do know how to do recon? How are your prompts coming in?

40:30 How is that context flowing throughout the duration of a project?

40:34 Mm-hmm.

40:34 To me, that’s where the secret sauce is, and you look at what’s happening with the toolkit, it’s phenomenal, and I think it also opens up a lot more, but you can’t just run it and say, “Okay, I connect,”, like, “I installed FNTK, update something.”

40:52 Yeah.

40:52 Mike, there’s a few things in the toolkit that I really love, and I think the biggest one to hear for me is the code command because this is the first time now outside of the MCP, because it does not require the MCP to run, that you can execute your code in Fabric without having connected to a notebook.

41:06 It’s gonna use a Spark runtime, but it’s not in a notebook.

41:24 , are you sure about that?

41:26 I’m sure about that. Take a look, my friend.

41:33 There is the code,, code run, where you can simply submit code, use the execution ID to return the polling command. And the code run is a workspace I- ID, and you simply put the code, it will return it back in JSON.

41:39 You can get the result of that. It does not have to run in a notebook.

41:48 It doesn’t have to, but wouldn’t you wanna use this in a notebook?

41:51 You could, but what if you wanna do, “Hey, give me the results of a current lake house. What’s the data set?” Right?

41:57 M- And again, I, for me, as I’ve talked about reconning,, my Fabric environment, what a great way to do so.

42:05 But is this, is all the code, like in the— So this is, this is one area that I’m a little bit more leery on in this package.

42:11 Mm.

42:11 So this is a, so to be clear

42:13 Let’s get into this

42:14 … this package we’re talking about is a Fabric notebook package, right?

42:18 So it’s Fabric notebook toolkit, which how are you running this? So, so this is the thing I’m, so two, two parts.

42:22 Let me, let me put some links here.

42:26 I remember where we were with this, yep.

42:27 Yeah. So I, I think it’s some worthy to add some clarity to this one.

42:33 All right, so first things first- There is a Fabric Notebook toolkit that’s there. So I’m gonna click on this link here.

42:38 This is the Fabric Notebook toolkit documentation that we’re aware of, that we’ve been given, right? This is coming from,, Fabric…

42:44 Well, let’s just… I’ll just use the, the acronym here.

42:50 Fabric Notebook toolkit. All right?

42:55 So this is the actual documentation that came out from the Microsoft blog that was the community saying, “Hey, there’s this new thing. The Notebook toolkit is now available.”

43:02 Mm-hmm.

43:02 In the link of that it talks about, okay, agents rely on Spark., this closes the gap. You can now s- use supported large, large language models in a safe way.

43:08 All those things are there. Oops, it said it can’t post the message here.

43:14 Let me try that again. Okay, trying one more time to post the message. All right.

43:20 Why don’t you ask the notebook to do it?

43:22 Right? At the end of this it says, “To learn more, refer to the Fabric Notebook toolkit documentation.” Oh.

43:28 Mm-hmm.

43:28 Interesting. I’d like to learn more about this one.

43:31 When you click that, it takes you directly to the pre-release,

43:37 The pipeline

43:37 … pypi.

43:43 org release of it, and it just basically says… There’s very little instructions.

43:49 It says, “Here’s the Fabric Notebook toolkit, here’s how you install it, and here’s your quick start.” And there’s not much else to it. It’s basically, that’s it., couple of examples, that’s probably it.

43:56 Where do you run this, Tommy? Where’s the instructions of, like, getting started with… Is this all

44:02 Oh, there’s no, no instructions here.

44:04 Yeah. This is where I’m, I’m trying to unpack.

44:09 Like, it needs a little bit more… Like, I understand you built the tool, I understand that it’s out there.

44:14 Are you supposed to run this from the command line? Are, like, what are, what are we trying to do to run or execute this?

44:20 How do I give this to my agent so it can run these things and use… Like, how do I, how am I using this with the agent?

44:25 So that’s where I’m, like, a little bit, like, confused as to, like, where does this package fit in my workflow?

44:31 So naturally there needs to be a skill here. And when I was testing, Mike, was I was in my harness and was like, I want you to…

44:40 Because again, this is the cool thing, by the way, if you’re listening, one of those hacks. If you’re in a, a desktop application such as Claude, Codex, or VS Code, just tell it to install Python package or an application that’s a CLI tool, and then from that you can ask it to develop a skill.

44:49 And that’s what I did where we say, “Hey, install the FNTK. Here’s the link, and tell me what you can do with it.

44:57 What do we need to do to get set up?”

45:06 Mm-hmm.

45:06 And generally speaking, if it doesn’t have great documentation, like this does

45:10 Mm-hmm

45:11 … it can go, “Let’s try this.” All right, we can log in, and once we start trying a few things, then I wanna say, “Let’s start creating a skill around this.” And this is really, that’s usually the best way to start up because you’re right.

45:21 There’s… I would hope, I was hoping that on micro- the Microsoft Learn there would be some documentation. Nada.

45:32 That’s, that’s like what I w-… So maybe that’s my gap in my mo- my knowledge, Tommy.

45:36 There needs to be, like, some Learn documentation of, like, “Here’s how you can use this. These, there’s two methods.

45:41 You can use this in a notebook, or you can just use this straight running Python on your machine.”

45:46 Mm-hmm.

45:46 Or having your agent run Python

45:48 Right

45:48 … on your machine using these commands.

45:51 Like, but to your point, Tommy, there needs to be a paired skill with this

45:55 Right

45:55 … which I think is much more relevant to this whole story, which is, okay, there’s a skill now that lets me manipulate, or the, the skill turns into here’s all the things I would do with the CLI and this Python library that can now go manipulate and do things in this notebook.

46:03 That makes a lot more sense to me now.

46:12 So, and again, I, I hate that’s where the community comes in, right? I’m sure hopefully this will be part of skills for Fabric.

46:23 Just like most of the skills right now with MCP, like, it’s hard to use the MCP without the skills for Fabric. But Mike, let’s go real world here.

46:34 You got your team, and,, actually let’s do the data czar because, like, I think, again, consultants can be different. You’re in an organization right now and you guys are semi-adopted to AI.

46:46 Are you tell-, is this a command for people to do, or is this optional, the FNTK, as we look at what it is right now? Is it like, “Hey, we encourage you to use it,” or is it mandated, or is this for beta testing only right now?

47:02 I don’t know, Tommy. Right.

47:04 Well, one that… So first, first and foremost, it’s in pre-release.

47:08 This is… Tha- thank you. Yep.

47:10 S- so I’m gonna say,, if we’re talking about, like, who needs to be using this one, I think users need to have awareness of this. But I think at the end of the day, it’s in pre-release, it’s something just to play around with.

47:23 Mm-hmm.

47:23 It’s just gonna provide some assistance or help with you.

47:25 Mm-hmm.

47:25 We’re not in a place where this is like, we’re talking this is hot off the gate, just barely getting started.

47:30 It’s alpha.

47:31 It’s probably gonna, it’s gonna probably be shaping and changing and doing things differently,, over and over. So it’s going to be fluid here for these first couple months., so I think it’s a, a, for lack of a better term, I think right now it’s a toy.

47:40 It’s discovery. It’s figuring out does this work well for your workflow.

47:49 I would not rely on it to be there for long-term experiences with notebooks.

47:58 Mm.

47:58 However, it does feel like this is really closing the gap where,, let’s talk about, let’s not talk about this tool specifically, but let’s talk about Fabric and notebooks in general. I don’t want to have to do a lot of work with notebooks.

48:07 I don’t want to have to do a lot of work inside that code with Spark.

48:16 Mm-hmm.

48:17 I want my agents and I to collaborate on them together. So I think what we’re desiring here is an easy way to get my agent, Claude, GitHub Copilot, VS Code, right next to downloading, using, creating notebooks right from this service, right?

48:32 So,, - There’s, there’s a lot of th- the idea… So the point here is in the latter part, why is the toolkit critical for an agentic workflow, right?

48:46 Without your toolkit, you have to write a prompt, generate some code, and then there’s usually a developer in the loop which is copy said code over or go over to this notebook and then do some manual step

49:01 Mm-hmm

49:01 … to get the code into the notebook and do something.

49:15 Inside this toolkit, the new workflow is write a prompt, the agent is able to describe and understand context about your notebook, then look at said notebook, generate some code, find a section of code to make the change, execute, do diagnostics, and then correct its behavior.

49:28 Mm-hmm.

49:28 So we’re actually trying to create more of like a loop for the agent to see more of what’s happening in the notebook.

49:34 Right.

49:34 And I think that’s why this exists, right? This exists so that I’m not using an agent on the side asking it to write a Python function or something in a notebook function and drop it in.

49:43 I actually need the agent to run the notebook and see the output of, of, of that cell, right? “Hey, agent, I’m having trouble on cell three.

49:52 Help me come fix this.” Okay, when I say that, that’s my prompt, it- the agent is then able to use this tool to go get context about

50:01 Right

50:01 … the cell, the error message, inspect, figure out, let me propose some changes, immediately update the cell for you, and then have the things work that way.

50:06 So I think that’s where this is trying to go.

50:12 And I, and I think this is, this is an important part too, because we’ve talked about this with the MCPs where who do you give it to? I still think we’re s- this is an ongoing conversation in terms of is this just your developers?

50:23 Is it just a set of developers? Can your managed self-service side have access to the MCP?

50:33 And I think we’re looking at all these agent tools and that’s great, that’s super cool. However,, we have to be very careful in terms of all the different access that people have.

50:45 Mm-hmm.

50:46 With the toolkit then, to me, this definitely seems that this is more of your developers, right? This is really gonna be my BI team that’s gonna be using this.

50:52 I’m not going to allow,, just a random Joe Schmo to build their notebooks this way. There may be a flow there.

50:58 Oh, no. Oh, no. I, you are

51:05 I don’t know, Tommy

51:05 … if you’re just listening, Mike’s face is g- contorting.

51:08 I’m sighing. I’m, I’m

51:09 Okay

51:10 … what I’m thinking about your comment there, Tommy, like in some, in some directions, yes, I agree with you, right?

51:20 This is for the developer of notebooks, but this is also trying to allow you to prompt with it. And anytime I’m seeing an agent showing up with prompting, the agent showing up with prompting is trying to then democratize that build experience to, like, business users.

51:29 Mm-hmm.

51:29 So I, I do understand your point, Tommy. It is for a developer, but is the intent of this package to open up more developers to notebooks?

51:37 I think the answer is yes. It’s

51:45 Okay

51:45 … it’s making it easier for you, the developer, to work with it.

51:53 But it’s like if I’m reducing the barrier, if I can have agents help me build things inside notebooks, all that’s doing is it’s bringing more novice or n-, an opening of that experience to bring more people to the, to the data engineering space, right?

52:02 And you want that?

52:03 You, you do want that.

52:04 I’m not saying I don’t. Yeah, okay.

52:05 No.

52:05 So

52:07 Right?

52:09 Yes., but here’s the thing. Without without any very…

52:20 This goes back to our raise the floor,, or,, lower the ceiling argument. So yeah, I’m gonna bring people who take someone who is just a Power BI pro, which we know they’re still out there, and we’re like, “You gotta build notebooks now.” Am I giving them the toolkit and saying, “This is your, your introduction to notebooks and what they are and what they’re supposed to do”?

52:30 Are you nodding or are you not sure there? Because

52:42 Is this the introduction for notebooks? No, I, I don’t think it is.

52:46 I don- I don’t think this is the introduction part. I do think this…

52:50 So how do I say this, Tommy? It’s like I’ve just gone to the hardware store

52:54 Oh, boy.

52:56 … and I just found another better drill.

53:03 Like, I’ve been using drills my whole life, right? Regular just drills.

53:10 It has the little knob on the front that tells the,, when it starts ratcheting, right? Simple, right?

53:17 And I just went to the hardware store and I found, oh, there’s this thing called an impact wrench, right? Oh, this is a tool that does this other thing, and it does the normal thing and something else that’s new, right?

53:23 I feel like this is another tool in our toolbox. I just walked into the hardware store and I found a brand-new tool I didn’t know existed but will help me get my work done.

53:30 So that still means I need to understand how to be like the carpenter or builder or the whatever. Like, I still have to have skills around using notebooks already.

53:37 So I’m not going into someone and saying… I’m not bringing a brand-new person to this and saying, “Start here with this tool.” That’s not what I’m trying to say.

53:44 Okay.

53:44 I’m trying to say that this is a tool for developers,, but it’s, it’s only after you have some fundamentals of notebooks, clusters, how they run, what they do, then do you step into like this other world of like, okay, this is a tool that will then help you direct better on top of your notebooks.

54:02 Okay. So let me…

54:10 I’m gonna expand on your analogy, or I’m gonna modify it a bit because to me it’s not so much like you have now upgraded your toolkit., you got something for Father’s Day, which by the way, I’ve never gotten equipment for Father’s Day. My wife knows me very well because it would not be useful to anyone.

54:17 So, but it… I, I think it’s more than just getting a, a better drill or drill kit.

54:25 So you never walk into a, a hardware store and that’s not you, Tommy?

54:28 I walk in there when she tells me to, to get a piece to tell me what bay and aisle it’s in

54:33 … and I walk out, hopefully I got the right piece.

54:35 All right. That sounds good.

54:36 There are good… There are things I’m okay at, that is not one of them., is, is

54:40 Gotcha.

54:41 But I would, I would say the analogy, it would be more akin to I wanna start building something, my neighbor comes over, who can build anything, goes, “What are you trying to do?”, I wanna build a wall downstairs. Like, I can help, and they do basically take over.

54:54 I just go, “Yeah, okay, I can do that.” It’s like, “Well, I do want a sliding door, not a normal door.” I can do that. And they’re basically building it.

55:08 You’re just there watching, going, “That’s so cool. Why are they doing that?

55:21 What’s the frame for? Okay, where is the piece of wood there?” And I think,, because Mike, the, the upgraded,,, idea is nice, but again, I can take someone, like with, with little to no skill in building a wall and ask my neighbor, and I can have it built for me in my house, and I think we’re looking at all this F&TK and the toolkits., again, I, at this point, we could literally get someone on the phone ca- on the phone right now who has never touched notebooks in their life, but saying they need to connect to some random API and put it into a semantic model.

55:34 I can give them this toolkit, and they could probably get some semblance of it, right? Not necessarily validated or, or all the way, but they’re gonna have some artifacts and some data in their lakehouse.

55:48 So that to me is a lot more than just getting a better drill kit.

56:03 I feel like this is a precursor to something else that’s coming that we’re not ready to see yet.

56:09 That’s a good point too.

56:11 Right?, when I look at this, it’s going,, so th- thinking about the things, so I’m, I’m having my agent here on the side like interact with this a little bit on the side here., there is,, there’s no… Again, this is where the documentation I think for me falls short here this.

56:21 You don’t really know what it can do, right?

56:31 Mm-hmm.

56:32 They’re, they’re, for ea- this, this toolkit is designed to make multiple actions with these notebooks: data, notebooks, the source, outputs, and logs. It’s extending the experience of what the agent can see inside a notebook, right?

56:42 Notebooks r- revel in like there’s a data portion of this, right? The, the notebook has a definition.

56:53 The notebook has an output. There might be a log that this notebook is doing or a log of the tool.

57:03 The agents cannot see this stuff. It does- they don’t, they don’t appear.

57:13 And so having something like this in a tool is extending this experience for the agent. And so, again, I’m still, I’m still tr- I’m, I’m where I’m torn, Tommy, on this one is how do we use this effectively?

57:24 I think this is going to push us into a place where we’re gonna do a lot less notebook writing

57:35 Mm-hmm

57:35 … and we’re gonna do a lot more directing around, “I have this table in the b- bronze layer.

57:40 I wanna do this, this, and this to it. Help me run some data quality on this one.

57:44 I need you to develop a notebook that does the work.” And I know, Tommy, we’ve been talking about this a lot on the podcast

57:49 Yeah

57:49 … which is we don’t— agents or agentic experiences shouldn’t be used to get you the answer about the data.

57:55 Agent sessions or agents should be used to build the system that can be p

58:02 Mm-hmm

58:02 … repeatedly run over and over again to get you the answer, right?

58:11 And I feel like this is development on top of that system that’s helping me, helping our teams become more capable to build the system. This is, this is the agent being used to build the machine that makes the machine.

58:20 The mach-, the machine in this example is like I want a data pipeline, right? Bronze, silver, gold.

58:30 That’s what I wanna do. This is giving the agents more,, visibility, assistance, helping to see what’s inside those notebooks so I can describe more things at a high level.

58:39 Because what I had to do in the past, which was, “Hey agent, go use the Fabric API. Here’s an app registration. Go pull down this notebook.”

58:48 All the thing, yeah.

58:49 Right? Load it on my computer, look at what the code is doing, make your changes, and then go publish it, and then I could run it, right?

58:59 So I think this is trying to really adjust or give that agent more capability to do, not have to do all those manual steps all the time. At least that’s how I’m perceiving it.

59:08 Puts that barrier a lot l- lower than it already was.

59:12 Yes.

59:12 Mike, and it’s funny because I was gonna ask you initially, w- would, does this change how you would hire a data engineer,, in 2026, ‘27?

59:20 No, not yet.

59:21 Not yet.

59:22 Not yet.

59:22 But, and I think to your, to your point a little and what, what I’m hearing just from what you just mentioned is this by itself is not like the,, game changer, but w- all these little steps and building blocks and stones that Microsoft is doing around the agentic developer for Fabric, F&TK, the MCPs, the skills that they have, they all work together, right?

59:35 You’re not gonna use one of these by themselves to like, in a sense, complete the, the whole agentic experience. You do need all three., you’re, or at least you do need MCPs, you need skills at this point.

59:48 They are just continuing to bring it forward. And to me, is it the main thing, the future of Fabric development are gonna be things like F&TK?

60:01 Probably not, but I think it’s gonna be an essential part of the ingredients, right? If, if Mike, I know my next article actually.

60:14 It’s gonna be what is the Fabric recipe? What, if I was making a meal, the Fabric agentic meal, what would be in the ingredients, right?

60:27 And I think obviously there’d be a hefty dose of the skills,,, with a large meat of MCPs. And F&TK is just like all the necessary sauces or spices and spices there that are just gonna be just as essential to the final product.

60:40 That’s what we’re seeing more and more of today. I look at this and yes, by itself I would agree with you.

60:53 This is not gonna change how I’d hire someone. But if I met someone who’s like, “I wanna get in data engineering, I’m pretty novice, but man, do I know MCPs, and have I been learn- learning the theory of data engineering, and,, I know about F&TK and I’m really good with,, the harnesses, but I’ve never actually executed it myself.

61:06 But I can tell you everything to know about Medallion.” I wanna, I wanna be serious about having that conversation with them on hiring them, because that’s the elements that they need to know to execute.

61:21 I think, I think in this new era when we’re trying to talk about who we’re hiring and what skills should they know,,, first and foremost for me, I’m making sure I’m hiring individuals that are very curious.,, Kurt gave… And we talked to Kurt Bueller on the podcast recently, and he gave the three Cs that I think are very relevant, right?

61:34 You need to be creative, right? Think about ways creatively.

61:47 You need to be curious,, willing to learn, willing to, again, ‘cause everything is changing so fast, you’re gonna ha- that’s gonna have to be part of a core part of who you are, and you’re gonna have to be very critical, right? Critical meaning does this new feature, function, value add value for me at the end of the day, or is it, or is it just blowin’ smoke?

62:00 So I don’t f- I still think I’m hiring a data engineer that understands notebooks, that understands Fabric, that understands, like, a lot of the core aspects of, like, why we design things the way we design them.

62:13 Mm.

62:13 , there’s a lot of,, I wanna say maybe friction, but maybe it’s not… Maybe friction’s not the right word right now, Tommy.

62:23 There’s a lot of,, opinions, I’ll say it that way. There’s a lot of opinions right now on what does bronze, silver, and gold look like.

62:34 Oh, yeah.

62:34 I think it’s pretty clear in my mind. But how, how do you define that at scale?

62:40 And I think there’s a lot of organizations that are still resistant to moving into a world where they’re doing data contracts and handing off from one team to the next. Like, this is your responsibility.

62:47 I’m gonna give you a Lakehouse table.

62:53 Right.

62:53 What you do with it after the fact, that’s on you, right?, a- and th- that’s, that’s the stuff that I think I look at this going, it’s we have to evaluate where do people sit in these roles.

63:12 Mm.

63:12 And as these tools come out, you have to have the fundamental knowledge. You’re the data engineer.

63:16 You’re going to be using these things. This is one more thing in your toolbox.

63:20 You’ve gotta be curious about it. You’ve gotta figure how it works.

63:24 But then you have to e- evaluate, does this actually fit your better s- your, your larger plan for the organization?

63:28 Mm-hmm.

63:29 Are we g- like, are we even a company, as a company, are we even comfortable letting people build notebooks with agents? I think you should be.

63:38 Yeah.

63:39 This is a step in the right direction to get you closer to that. I think you’re already seeing this in Databricks., Databricks has their Genie capability, and their Genie is basically the GitHub Copilot equivalent, and inside that tool you’re editing and manipulating and working with things with notebooks.

63:54 It, it depends on what you… And I think as we get near the end, I would add on to that, I think you as that data czar or the leadership point of view, you have to be aware of things that are preview, general, available, and you have to be very conscious of that.

64:03 I think you mentioned this just about the,, toolkit. It’s pretty powerful.

64:13 It’s really cool, but it’s alpha. Like, we’re, it’s so early, and I think you, just like we know right now, the whole bigger story around AI in general is it’s moving fast.

64:22 It’s moving faster than some people would like, than most people would like at this point. I think, too, both on even things that are being developed in Fabric as well, you have to be conscious of that because there are capabilities here, we’re not sure how it works, works.

64:32 It looks really cool. It’s really great.

64:41 But I think if I’m a CTO or CIO or a developer BI at my organization, am I gonna release this in the wild right now? I’m gonna be monitoring it.

64:50 I’m gonna have a sandbox for it. But I’m gonna be very careful with that, especially as we have something that’s just in alpha.

65:00 And this is all well and great, but I think part of the responsibility, and I know Kurt was mentioning this too, you have to be responsible around what you actually in a sense allow out there. I think a lot of these things too, once you allow it, you can’t really put it back in the box.

65:09 Again, what we talked about the other day, if MCPs went away right now and your ability to use them with Fabric went away, holy crap, there’s a lot of relearning we’re gonna have to do. And I think we just have to be aware of what we start allowing.

65:19 I don’t wanna end on such a sour note. I love this tool.

65:28 I love where this is going, and I think we’re gonna see such a powerful impact here. So that’s my thoughts, Mike.

65:38 Mike, w- where do you wanna end us?

65:47 I think I wanna end this one on, I, so I liked this, the conceptually I think this is in the right direction. The i- the execution of this one is right now, the way it stands today is poor, right? There’s not clear documentation.

66:01 Mm.

66:01 ,, looking at their document, it’s like they literally have an article and say, “Here, go read it up on PyPi.” That, that’s not documentation for users to understand. Like, I’m reading this documentation, I’m confused.

66:09 Where do I start with it? How do I install it?

66:17 So I have to go into Python. I have to install the, the packaged library.

66:24 But then I believe in your quick start, that’s all CLI code. You should be running this all in your CLI.

66:32 So I’m confused as to, like, the code they g- even the code examples they give you, they’re not really telling you where to run and execute them on inside this code base. So I’m, I’m a little confused here as to, like, what should I be doing with it? Where do I run it?

66:40 Mm.

66:40 And what’s the value of where I’m running things in? So,, if this, if the intent of this is to run this from the GitHub Copilot CLI or the Claude CLI, make it clear.

66:50 Make it clear what we’re trying to do here,, and making sure that you go, “Okay, for Python you must install this library. For the quick start-”

67:00 Right.

67:00 ”… here’s what you’re doing.” And so to me, I look at this going, it feels like they’re mixing Python with s- with,, command line interface code, and in their documentation I’m not clear And so I think, again, it’s alpha.

67:15 This is super hot off the press. So again, very, very new., and I’m hoping— but we’ll see.

67:30 And w- I’m, I’m hoping in the next couple months, Tommy, or maybe even a couple weeks, we’ll see an update to this or an update from an official Microsoft Learn documentation that shows you step-by-step how to use this and where it’s tended to be used. Right now, to me, it’s unclear, and even my agent’s unclear because when I ask my agent to go build something around this one, it’s trying to build a full Python notebook around everything that is, that is the,, the toolkit.

67:45 Cool.

67:46 I actually don’t think that’s the right— I don’t think that’s the right approach.

67:48 Oh.

67:48 I think it’s actually wrong.

67:49 No.

67:50 Like, it— If my agent can’t read the instructions on that information page and know that it is a CLI that we’re trying to build this from, and which commands to run on the CLI versus in the notebook, that’s a problem. So to me, I look at this going, there’s an issue here with this one because the documentation isn’t matching

68:08 Right

68:09 … my understanding.

68:12 And that’s my only feedback on this one. I like the idea of this one.

68:16 This will be a good product eventually. I just need s- a little bit more love on the documentation side. That’s my, my hesitation.

68:20 I love it.

68:21 That’s okay for alpha, though.

68:22 That’s not— what? This— A whole conversation on alpha, not bad. You’re welcome, man.

68:26 Yeah. This, this is the right dir— The— So I will say this, mad kudos for Microsoft to start to recognize,, most of what they’re building nowadays is all around building tools that build the machine

68:41 Right

68:41 … and not requiring your agents to be daily running to answer questions, or daily running to do data transformations, or daily running to do notebook execution that should be just happening through the Fabric system.

68:49 So I think this creator agent experience or the agent developer is really good.

68:59 Yeah.

68:59 Like this— We’re getting— We’re f- we’re finally getting to the place where I feel like Microsoft is listening, and we’re getting real creative experiences with our agents that are on top of all the Fabric things.

69:09 Hundred percent.

69:09 And that I do like.

69:11 Hundred percent. Dude, love it.

69:12 All right. That being said, Tommy, where else can we find the podcast?

69:16 You can find us on Apple, Spotify, wherever you get your podcasts. Make sure to subscribe and leave a rating.

69:20 It helps us out a ton. Do you have a question, idea, or topic that you want us to talk about on a future episode?

69:23 Maybe you’re using the toolkit, FNTK. That’s right.

69:27 And you wanna—, maybe something we wanna discover more. Well, head over to powerbi.

69:31 tips/podcast. Leave your name and a great question.

69:35 And finally, join us live every Tuesday and Thursday, 7:30 a. m.

69:38 Central on all powerbi. tips social media channels.

69:43 Thank you all so much, and we’ll leave you with another earworm. Here we go.

69:46 Oh, boy.

69:46 I used to chase the moment when they stared for hours. Thinking longer glances meant I had the powers.

69:57 Then I learned the quiet wind that changes everything. The dashboard wins when they barely need to cling.

70:08 Barely look, barely look. If the answer finds him, you already took.

70:20 Barely look, barely look. A glance is enough when the truth is booked.

70:31 Barely look, that’s the victory. Information landing with such clarity.

70:42 Barely look. Barely look.

70:53 Nobody wants a maze dressed up as art. They want the signal singing to the heart.

71:04 If they skim and know, you did the sacred part. Beauty is the speed of understanding stars.

71:15 Barely look, barely look. If the answer finds him, you already took.

71:26 Barely look, barely look. A glance is enough when the truth is booked.

71:38 Barely look, that’s the victory. Information landing with such clarity.

71:49 Barely look. Barely look.

72:00 I used to feel ignored when the eyes moved on. Now I feel successful when the need is gone.

72:11 They came for the knowing, not the show I made. Barely look means the message stayed.

72:22 Designed for the skim, designed for the breath. Designed for the person who has nothing left.

72:34 If they barely look and still they understand. You held the whole festival in one soft hand.

72:45 Every pageant river on a desert rise. I built for the glance beneath the open skies.

72:56 Power BI quiet as a winning horse. The best report is one they barely look.

73:07 Barely look, barely look. If the answer finds him, you already took.

73:18 Barely look, barely look. A glance is enough when the truth is booked.

73:29 Barely look, that’s the victory. Information landing with such clarity.

73:40 Barely look. Barely look.

73:52 Barely look. That’s the victory.

74:03 Barely look. That’s the victory

Thank You

Want to catch us live? Join every Tuesday and Thursday at 7:30 AM Central on YouTube and LinkedIn.

Got a question? Head to powerbi.tips/empodcast and submit your topic ideas.

Listen on Spotify, Apple Podcasts, or wherever you get your podcasts.