Team vs. Individual Skills – Ep.538

A skill is a folder of instructions that teaches an agent how your team does something. It also lives on one person’s machine, in plain text, editable by anyone who has it. That combination is the whole problem this episode picks at: the moment more than one person is using skills, you have a distribution question, a version question, and a quality question that the tooling does not yet answer.

News & Announcements

• Agent Skills — The open format behind all of this: a skill is a folder with a SKILL.md holding metadata and instructions, optionally bundling scripts, references, and templates. Agents load them through progressive disclosure — name and description at startup, full instructions only when a task matches — so you can keep many on hand cheaply. Originally developed by Anthropic and released as an open standard, it is now supported across a long list of agent clients.

• Evaluating LLM outputs — Anthropic’s documentation on building evaluations: define measurable success criteria, then grade outputs with exact match, similarity scoring, reference-based metrics, or LLM-based grading. It’s the closest answer to Mike’s open question in this episode — how do you actually prove one version of a skill is better than another instead of guessing?

Main Discussion

Topic: Governing skills across a team when they live on individual machines

Mike opens with the practical shape of the problem. Skills are local. That’s fine for one person, awkward across two machines, and genuinely difficult once a team is involved — especially when Microsoft, GitHub repos, and the wider internet are all producing skills faster than anyone can vet them. Tommy’s contribution is a clean rule for deciding what to govern, and a persistent worry about what governance costs the people using them.

• Tommy’s rule: anything requiring a consistent standard becomes a governed skill. Discovery documents, statements of work, report authoring, model development — if the output has to look the same regardless of who produced it, the skill belongs to the team, not the person.

• This is the theme file problem again. Both hosts have watched an organization roll out a template, then survey the reports a year later and find every one of them different. Skills will drift the same way, for the same reasons, unless someone owns distribution and updates.

• Define the output first, then the skill. Mike’s answer to teams with mixed AI adoption: specify what a finished deliverable looks like and hold everyone to it. Whether a given person gets there with a skill or by hand becomes their business — and the standard survives the fact that adoption is uneven.

• You cannot actually enforce a skill. It’s text on someone’s machine. Tommy presses on this and Mike concedes it outright — short of hashing files, there is no way to guarantee two people are running the same skill. Referencing a shared repo helps; it doesn’t lock anything down.

• Token cost is now a design constraint. A verbose skill burns real money on every run. Mike’s question is whether an egregiously heavy skill is a skill you should be shipping at all — and picking the right model for the task matters just as much. Running Opus to do what a small model handles is driving a Ferrari to the mailbox.

• Nobody can measure skill quality yet. Mike lands on this as the real gap: publish a skill and you need to know its token efficiency and how close it gets to the intended output, across Claude Code, VS Code, Codex, and every other harness. Microsoft has this exact problem with skills-for-fabric right now.

• Turn skills into scripts wherever you can. Mike’s own loop is to build a skill, then ask the agent which parts can become deterministic scripts. He learned this cleaning podcast transcripts — doing it purely with an agent was expensive and inconsistent; the fix was pushing the repeatable work into code and leaving the agent the reasoning.

• Leave room for garage time. Both hosts want governed skills and deliberate space for people to experiment. Tommy’s concern is that fully standardized skills strip out individual contribution; Mike’s counter is that a person still reviews everything at the end, and today’s personal experiment is next quarter’s governed skill.

The all-or-nothing question

Tommy raises the case neither host fully resolves: if your team adopts governed skills, does everyone now have to learn AI? Adoption speeds differ, and not every company has every person on agents. Their working answer is to hold the output constant rather than the method — which they both acknowledge will create tension on a team where some people move much faster than others.

Looking Forward

Pick the one deliverable your team is least consistent about, write down what “done” looks like for it, and only then decide whether a shared skill is the way to get there.

Episode Transcript

0:28 Hello everyone and welcome back to the Explicit Measures podcast with Tommy and Mike. Tommy, good morning. How are you doing? doing?, some people may be wondering why does it look like our first episode on my end? True. That’s because I’m using all the technology for my first episode. Yes. Yes. So, the fun thing Mike about owning your own business among other things is if things break, it’s on you. There is no IT. IT. You are IT. So, Yeah. Yeah. finally, my old PC and again, thank

0:58 finally, my old PC and again, thank you my old company for giving me an amazing machine, but it was old, you amazing machine, but it was old,, and things just happened to know, and things just happened to melt. so many words. Tommy ran it hot and the CPU just can’t handle the heat. run it hot. Yeah, so we’ll be getting that fixed. The good news there’s a backup. The bad news is all the things configured work only on that machine. So, we’re going to make it work. So, sorry Tommy about the machine. That’s quite frustrating. I’ve also been going through some machine issues. I

1:28 I I have recently gone through some computers where the battery I bought a bunch of Razers, Razer laptops, which is like all by gaming laptop cuz we’re going to need like high performance stuff. stuff. The laptops are nice. They’re back when they were like aluminum cases, pretty pretty sharp stuff., I liked them compared to like Windows computers at the time. the time. And And the batteries are all like dying on me. So, they’re all like the belt the batteries are like swelling and the bottom of of the the cases are like getting big. Like yeah, we’re having to go through all the laptops and basically I’m having them like return slowly one

1:59 I’m having them like return slowly one by one back to me. I’m buying a new battery, I’m replacing that. It’s it’s just a mess. The whatever the charging electronics thing on the Razer laptops for the year that we bought them for, I don’t remember what year it was, just not that great. So, anyways, I’ve been slowly going through that as well, changing over a bunch of things. Yeah, I’m I’m a I’ve also realized that as a consultant, tell me what we do. We have to have backup computers. Oh, yeah. Oh, yeah. We’ve got other stuff laying around. So, yes, while I do have like my main

2:30 yes, while I do have like my main desktop, which is what I’m on right now, I have like three other laptops that are nearby just in case like something ever went bad and I could at least get by and do work and,, ,, you have to have that as your backup plan cuz you can’t stop working. Well, people look at my office and they think, “Why do you have so many screens?” So, actually, what? Since we’re having fun here, let me For those who are listening, you’ve never seen it, but if you want to know how many screens I actually have, Look at this. Tommy’s space station. This is what this is.

3:00 This is what this is. Yeah. And then I got eight screens over here or three screens here. So, there’s eight screens in my room, but guess what? If your computer goes down, you still got to communicate with your clients. clients. Yes, correct. tell you IT broken because I’m also IT and HR, which is fun. Have you ever worked in HR? Yeah, in my Come on up with a complaint with HR. Yeah, go ahead. Like to see you try, dude. That’s a joke I made with people from my like, “Oh, you you run a company?” Yeah, like my HR is terrible, but horrible.

3:30 horrible., no, no, no, no, no., but it is fun working on a motherboard because I was trying to configure and I took the GPU out and you’re testing things and it’s and there is something very satisfying if you’ve ever worked on your own custom PC. PC. Yes. Yes. It’s it’s It’s it’s I always As a kid growing up, I was a mechanical engineer by what I did in college. So, tinkering with like electronics, taking things apart, like I always loved taking things apart, and working in computers was always intimidating, but then

4:00 always intimidating, but then you just start doing it, and you’re like, “Oh, that’s not that bad. It’s a lot of plug-and-play stuff.” the only trick is to do a lot of like if you’re building your own PC, which I assume now, Tommy, if you’re going after like a new motherboard, a new CPU thing, those are two pretty expensive items on the computer ring. but Yeah. Yeah. Now Now you’ve You’ve got the case, right? The case is still there, so you can keep a lot of your stuff. And then you didn’t lose your data. I I would assume. I think that’s all still in the Data’s fine, the GPU’s fine. I would have 128 mega

4:30 have 128 mega gigabytes of RAM, which is great. So. Wow. Okay. Yeah. Yeah. So, So, oh, when you get it all back together, we’ll have to look to see what the new PC’s like. one thing as as I recommend recommend to see my wrinkles. It’s going to be so good. It’s going to be so super high definition. SUPER HIGH DEFINITION. AWESOME. All right. Let’s move from there. So, moving from shifting computer around. Actually, this is probably another area that’s pretty interesting here as well. let’s talk about this concept of skills. Particularly skills for agents

5:01 skills. Particularly skills for agents and things. How does this work? I run a team of developers. and Tommy, you work with teams of developers. We have these skills that we can build. How do we share these things? Let’s Let’s talk about our main topic today. So, our main topic today is building skills for teams and how do you share, distribute, handle those skills that you build with your harnesses across your team? What does that look like? So, I think this is an interesting topic right now. Tommy, you’ve built some special stuff around here to help you

5:31 special stuff around here to help you out with some skill stuff as well. maybe we should just open up with the idea of idea of Right. Right. Why are we trying to transport skills between team members? And then maybe we can go into maybe some solutions and how we solve that. Yeah. So, if this is your first time listening, cuz we’ve talked about skills in the past. But again, when we say a skill, we’re not talking like I’m very good at cooking. I have a skill of cooking. We’re talking about in a harness such as Claude or fabric or an IDE as VS Code, you can provide something that’s specialized with a script, with instructions for each agent

6:04 script, with instructions for each agent what are you working on. We’ve talked about Microsoft built up our skills for fabric, which have ones for report design, for semantic model, relationships, which are specified some have scripts. we’ve talked about from our consulting point of view, I have a special skill for my statement of work building and project management that knows my past work and it has resources and things such as all my markdown files, even scripts that will convert markdown to PDF the way I like in the format that I like. That’s a skill. It’s

6:35 format that I like. That’s a skill. It’s a specialized skill. Now, most skills live on the computer usually. usually. there are some systems that you can upload a single skill, but usually you say, “Hey Claude, create a skill.” or you import it. It’s actually local on your machine, which is great for an individual. However, my one, if you have multiple machines, but more importantly, when you’re building skills in an environment with other people, people, if I’m using the Power BI custom, you if I’m using the Power BI custom,, whatever,, X Y and Z know, whatever,, X Y and Z skill for my company and someone else is

7:07 skill for my company and someone else is not, you’re going to have discrepancies. So, today let’s talk about why do we need to think about if I’m working with a team, the importance of team skills rather than the individual skills. What can we do about that? And what are some of the I think approaches we should take? take? Yeah, let’s let’s unpack some of that. So, I think of skills so one concept that we we need to talk about here like

7:37 we need to talk about here like why is the skill exist? How does that influence the work that you do on the day-to-day basis. Yeah. Yeah. Right? So, that I think that’s maybe the one of the things that I think about when I’m talking about skills for our team. team. The harness is the thing that wraps around the agent, the large language model, and you customize the harness with the skills. That’s That’s the business logic. That’s what how to do. That’s your process. That’s what you want the agent to understand. There’s a lot of, customization there. There’s If you think about your company,

8:07 There’s If you think about your company, and again, Tommy, this is weird. I will be saying no, but there’s a lot of parallels between you running a team of agents. So, Tommy, you’re a, HR, but you can have a two or three agents running underneath your purview. Right. Right. and so, you can run like multiple target containers. You can have multiple agents running on one machine. You can You can talk to them individually. They can talk to each other. That’s actually this It’s I’m finding this is very similar to running a real team of engineers to have them talk against each

8:38 engineers to have them talk against each other. Right? Right? So, in the same vein, right? If I have three real people engineers, it’s always about, okay, what are our standards? How do we communicate those? Does everyone understand like the expectation for like code development? Do we have the same definition of like how we build on local host? What do we publish to dev for integration testing or test for integration testing? And then how do we get to production? Does everyone understand the process of what we’re doing? Right? These So, we have

9:08 we’re doing? Right? These So, we have conversations with our team. We have stand-ups. We have regular meetings. We We draw up documents and process and all these things to help everyone get on like the same page. This is how we do work efficiently together as a team. [snorts] Now, I’m Now, I’m I’m bringing this,, this is like what we do today. In the same way, skills are the ability for us to lift our process, our, our documentation, how we want our agents to run run in the different workloads either

9:39 in the different workloads either individually. Tommy, you’re working on a report. report. Right. Right. How does that work for you? and one thing that comes to mind right now is there’s a whole bunch of new skills that are appearing. the Microsoft skills for Fabric repo Mhm. Mhm. has a new authoring Power BI reports and semantic modeling doing some design work. There’s a whole bunch of skills there that are saying this is how we do work. You may want to adopt how you use these skills with your dev process. So,

10:09 these skills with your dev process. So, let me just pause right there. Is that making sense, Tommy? Do you have any thoughts on that?, it sounds eerily familiar to team workflows when we were dealing with templates in Power BI. So, if you were to create a new report Yep. Yep. And if you were going to team, this should already be something you use, right? You should have a template and a theme file that each co-, if internally that uses. If not, you have a lot of discrepancies. And this to me is

10:35 lot of discrepancies. And this to me is just a continuation of that or an evolution of that. Interesting. That’s a really neat parallel, Tommy. Even even now I think companies still struggle with like the whole templating and theming and like,, Tommy, you build a theme file, you put it on your report. A lot of people can start from the same baseline of a report. but not everyone has the ability to like How do you describe it? What If you look at your templates and then you have people use them for a

11:05 then you have people use them for a period of time, 6 months, a year, whatever that is, and you went back and surveyed all the reports that were out there, they’re going to look different. And some of them are using the theme file, some of them are not. Some of them have the normal theme file, and maybe it’s been adjusted. There’s There’s a lot of like variability in those templates of things. Try to keep things consistent. I remember one time I worked for a client who’s doing a lot of external facing reporting. Like it was part of their application. They’re building specific reports for their customers.

11:35 reports for their customers. They wanted a very consistent language across all the different kinds of reports that we’re building, different pages. So, every report, every page, it was quite challenging to make sure that you had no slicer selected, you didn’t accidentally filter something you shouldn’t have filtered. When you’re building, you you always put it on the,, you took the desktop application, always went back to the overview page cuz that is the default page that you wanted to be on., the the header bar had the same width as the color of the page the same. There’s a whole bunch of like little details that people would just

12:05 little details that people would just modify. And so, you have a collection of like 15 reports or 10 pages across different reports. And they all look a little bit different. You’re like, “Dude, I thought we fixed this. Like, why is this so hard to make it stuff look consistently good?” good?” Right. Right. And in the same way, just like it’s easy to create a report and create your own design, just like skills. Skills are awesome. I you need to use them. I don’t know why you wouldn’t because it’s like basically using a diet version of AI to

12:35 basically using a diet version of AI to me at this point without using skills. The template analogy goes even further though, too, because what did you have to do with a template if you rolled it out to an organization or a theme file? You need someone to create it. How do we share it and then distribute it? How do we manage it and make sure everyone’s using it and then how do we adjust it? Like how or modifications? And no one has really been talking about how does a team govern the skills around the organization? It’s also this is this

13:07 organization? It’s also this is this this pulls out an interesting point, Tommy, right? In the same way that I work with a development team, right? When someone works with the skill more than someone else, I off I off there are there are times when that skill gets modified or that user who’s using it more often finds a better way to use the skill. So, maybe there’s this definition of like, “Hey, look. Here’s this set of skills skills that we’ve written and are used as a team.” team.” If people copy them down, there’s

13:38 If people copy them down, there’s nothing that says you can’t modify them, tweak them, adjust them, or fix them to make them better. How do you, to your point, Tommy, how do you version that and from the entire team are able to observe which skills are getting adjusted, adjusted, what changes are happening, and are those changes actually reflecting a better use of the skill or not? And right now, I think this also brings the point point right now, Tommy, we’re in the space where optimization of agents is becoming becoming a thing, right? Your token budgets have just gone

14:08 right? Your token budgets have just gone up on GitHub Copilot. those are more expensive now to run than you have in the past. So, you’re going to burn through more money just to run those models. models. So, now if you write a skill and that skill is egregiously verbose or very heavy in token usage, is that really a skill we should be using? Should we be thinking about something else? How do we test this? What does that look like? And I’m going to I’m going to go one more here after I I’ll end this thought

14:38 more here after I I’ll end this thought here talking about I have another comment around monitoring the output of people using their skills and token counts. So, let me let me I’ll come back to that one. I’ll write it down cuz I think there’s a really interesting story here, Tommy, that no one’s talking about and I haven’t seen or heard anything about it yet. So, I want to bring it up here in a second. Go ahead. What’s your reaction? Well, honestly, my first reaction is that’s a lot of steps forward for me because I feel like you we need to take this step one. How do you even choose what skills in an

15:08 How do you even choose what skills in an organization that should be done? That’s true. Right? So, I I completely agree with the adjusting part, but if you’re an organization starting at ground zero, which many are right How do you build one and how do you give it to me? Just Just the initial sharing part is still difficult. Right, right. There’s the logistics of it, but then there’s also the governance of it is how do you decide which skills are going to be governed skilled? What skills are going to be deployed in that you have to then tell the team what are you going to use? For example, I can have skills around scoping and discovery

15:39 have skills around scoping and discovery calls for a report building. I can have the skills for fabric that are designed. I can have skills for notebooks that in our framework. I can have skills for our documentation in the knowledge center. How do you decide, okay, these are the ones that everyone needs to use compared to the ones that are just fine for me? And I think this is a good For me, it’s hard to get to the adjusting side and even the token side without even getting to that question. And how do you actually, in a sense, manage that? How do you decide on that? How does a team work on

16:09 work on where do we start when it comes to what skills need to be governed by our department or team? I I don’t even know how you answer that. I don’t know how you answer that question., I I think you have to start with I think it all it gets born out of the individuals that are using skills more heavily than others first. Okay, this is interesting. I think these skills skills Well, someone has to set the standard. You can’t just let everyone run wild and like go do things, right? I I think

16:39 like go do things, right? I I think there needs to be some a framework around how you allow people to create skills and then make derivatives of them in your team, right? So, for example, -huh, -huh, I got to go. Not everyone in your team is going to be able to like fully step into this and do like amazing work all and on all the skills. And also, there’s a there’s actually a a wide variety of skills you can go find, right? So, you can go to awesome co-pilot on GitHub. So, you look up awesome co-pilot on GitHub, and there’s a whole bunch of skills that

17:09 there’s a whole bunch of skills that live there. Microsoft themselves has built a bunch of skills., if you go Google fabric skills or skills for fabric, fabric,, there’s you’re going to find a lot of GitHub repos that have skills already attached to them, right? All over the place. So, there’s a huge library of skill sets that are out there, but like how do we vet them? let’s just say you just want skills for just general things, Tommy, like write an email, edit this video, make a web page, like the proliferation of skills that you can go find out on the internet is just

17:41 go find out on the internet is just enormous at this point. So, to your point, now we now we have to figure out it’s it’s the same problem we have with with our reporting, right? How do you sift through that pile and just find what is good and what’s not good? What Is there any test method? Again, going back to my comment earlier, earlier, and actually maybe this is a good time to bring up my second comment, which was , performance of these things, right? So, there’s there’s right now, most people are just thinking, I don’t even know what skills I need are. That’s that’s to your point. I see the output.

18:11 I see the output. And does that skill give me a better output? output? Right? So, I think that’s where a lot of people are right now. Where I think I’m getting into and where I think the challenge will very quickly move into will be we need to be able to use skills and use skills efficiently, right? And so, every time you have a session on your GitHub Copilot, every time you run a session inside inside GitHub in the cloud or GitHub agents inside github. com, all of those sessions that you’re running with those agents, there’s like

18:41 running with those agents, there’s like a certain amount of tokens that are running with it. There’s there’s a session that’s creating some code, it’s doing some things, and it’s kicking out an answer. Right. Right. I’m fully believing, Tommy, that picking the right model to do the work that you need to get done is going to become increasingly important. And the tuning and optimization of using agents to help you build will have to change here shortly. So, you’re going to say something.

19:12 So, you’re going to say something. No, I’m curious that you said the model not the the harness because the more I I am realizing is in the environment or the harness that you’re using is the most critical. You’ve said this. I’m not saying anything new. So,, Tommy , Tommy so you’re bringing up a good point. Last night, I was just up late playing with like all the different harnesses that are out there and I’m trying to So, I’m I was literally last night I was prompting Grok and said, “Which token services give me the most amount of tokens for my dollar spend, right?”

19:42 tokens for my dollar spend, right?” Because I’m looking at all the different tokens that we’re getting and I’m looking at GitHub Copilot with enterprise licensing. Not that great a deal compared to other tools that are out there right now. I’m looking at Grok. They have their own service. They have like a Grok Max plan for $300 a month. Then there’s the Anthropic plan for Claude Code, $200 a month. You can go over to Codex, that’s ChatGPT. ChatGPT. They’ve got a $100 plan. I think they maybe even now have a $200 plan. So, each of these plans are like you could use any one of these plans and pay

20:12 use any one of these plans and pay hundreds of dollars per plan to get into the tokens that you want. But,, I think of this, Tommy, like you can switch and swap. It’s It’s It’s like a really weird mix-and-match puzzle, I guess, right? I can swap the provider. I can get different models. I can have different harnesses for each of these tools that I can use in different ways. I can use the Grok API in my VS Code harness or other harnesses or Forge Code, another harness. harness. And so, I had last time, I had installed

20:43 And so, I had last time, I had installed like three harnesses all over my machine and I was trying to figure out how How do I know which one to use, when, for what, where? Which one’s more efficient? So, even in there’s a there’s another Yeah, people were like worried about this stuff taking your jobs away. Like we just have just opened up a whole new rat’s nest of new things that that

21:05 rat’s nest of new things that that designing and architecting. and with all these options, with everything like the reason one of the reasons we have a job, Tommy, is because the Microsoft licensing is quite complex. And how do you optimize for cost? I think this is going to become the same thing. How do you agent optimize? How do you architect your agents? That’s going to be a specific skill of experience people are going to have to go get and learn learn to understand, okay, when I’m writing code, I should use this model with this

21:36 code, I should use this model with this harness. When I’m writing instructions or I’m building issues in GitHub, I want to use this agent and this harness. So, it all is like a puzzle, and based on the task you care about, that’s the harness and combination of model that you should use because if I can get a chat GPT model like,, chat GPT 5. 4 mini, if I can get that to run to write the code I need, then why do I need to be running Opus to write the same code? You’re driving a Ferrari, but all you

22:07 You’re driving a Ferrari, but all you need is a Toyota or you can a bike to get the the task done. So, is the model and task sized properly? Does that make sense? sense? 100%, but I think there’s then the next question. When does that be Once you figure out for you what works, when does that become a shared approach? And I think this is what we’re trying to get to. Right, so let’s say you’re like, you to. Right, so let’s say you’re like, what, GPT-4,, Codex that I know what, GPT-4,, Codex that I straight thing. Someone’s like, I don’t want to use it. Is that fine? Is

22:38 I don’t want to use it. Is that fine? Is that okay? And when it comes to the skills, I think it’s the same thing where where how do you decide on when does a skill become required for a team to use? And I I have a few things here that I just wanted to think? How How would you answer that question, Tommy? Like, what what’s your initial thoughts on there? How do? How do? Yeah. Yeah. I think it starts with consis- whatever is going to be a consistent standard. Just like a report, right? It’s a report template. We want to make sure that you always have a banner on top and, you

23:09 always have a banner on top and, you always have a banner on top and,, you always have whatever the know, you always have whatever the grayish background. Outside of that, it’s up to you. When I am doing things in my organization that other people want to have the same consistent out or need the same consistent output, that’s probably when a skill needs to be governed and managed together. So, hey, when you’re going to be writing your discovery, we all want discoveries to have the same in a sense format structure. Therefore, we have to use it. And also, I wanted to bring up this idea of this is a great opportunity for teams to be creative

23:40 opportunity for teams to be creative together. Mike, I’m envisioning in teams where when they have weekly stand-ups to bring about their ideas for a skill, to bring it to the group. Like, “Hey, I created this skill to,, it’s going to really help us when it comes to ticket vetting, right?” And I will actually look at our tickets and I want everyone to use this or I would like to pitch that. That becomes a decision by the team, obviously the manager, but and I think it starts with when just to your point, it starts with is this output

24:10 point, it starts with is this output something that’s going to be needs to be a standard consistent or consistent standard. Because, for example, I can write the code in the back end. Some developers would argue you want that also to be a output that’s standard, but that can be a little more,, personal. But if I’m writing as,, like my statement of works, right, Mike? Like, if I’m writing those or I have a team working on them, I don’t want someone to write it with a completely different output and structure and cost structure than I do. And if I’m writing reports or

24:41 than I do. And if I’m writing reports or the way we develop models, models, it’s probably going to be something that’s shared. And with that in mind, let me Yeah, let me let me hit the ball over back to your court here. here. My big point here is once anytime there’s something that requires a consistent standard that needs to be a governed skill. Yeah. Yeah. What else would you add to that? What What are your thoughts on this? I think that actually spans very widely, Tommy. What I think that covers almost

25:11 almost many, many or all the use cases here. Again, the only thing I would maybe add to that idea is there is going to be some Look, well, to your point. Having a document output the right bit of text, code, document format every single time is going to be consistent. Like that that makes sense. One thing I want to also maybe add in here as well. I don’t know. I don’t know if this is the time to talk about this. We’re talking about skills right now.

25:41 We’re talking about skills right now. Probably not then, but continue. We’ll see. In the same way that you’re you hire people on your team to develop things, right? right? There’s also you’re building custom agents in a way here as well, too. I was just having the conversation with my lead engineer about this exact kind my lead engineer about this exact this exact problem, right? Hey, of this exact problem, right? Hey, we I had a new feature request. I said, “Hey, it’d be really nice to have some

26:11 “Hey, it’d be really nice to have some piece of software prototype around this.” The customer gave me requirements that said I had challenges with this part of Power BI. And I thought, “Man, that’d be really neat for us to build not an app specifically, but like a solution that could help the customer.” And I thought, “Well, we could use that same idea in other places as well with our code.” And so, I’m looking at this going, “Man, this is really interesting. I’d really like to be able to build this.” And so, my engineer is like, “Well, yeah, let’s just sit down and prompt it.” Like let’s just give it We’ll work with an agent. We’ll have it give us

26:41 with an agent. We’ll have it give us We’ll We’ll work with it, have it grill us, basically, give us some detailed information about the system, the the the thing the feature you want to build, and then from that system you’re going to build, you’ll have we’ll have good requirements, we can then break them down into tasks, and we can give them to other people to work on. And I said, I love this idea, but what I’m torn on right now is I want to build two things. I want to build my net new feature, feature, but everything you’re describing is a

27:11 but everything you’re describing is a process that we’re following that I want to reuse, right? I want to be able to describe the feature I need to So, I got So, I got agent. agent. I got you. The agent should be able to say, “Okay, let me help you like let me use the grill me function, and let’s let’s go into the idea of what you’re looking for. Let’s make sure we have a shared mutual understanding of what we’re trying to create, create a markdown document that describes you the product requirements, all the things that it needs to do, all the tasks, and then,, we need like a custom

27:42 , we need like a custom agent to do like a graphics design review. We have standards, we have libraries that we’ve built that we use to build stuff. So, we want to see like a a UI reviewer. So, we need a skill around that. We need an agent that does that. And then, once the UI review do does it they add their requirements to document. So, it’s actually let me touch on this real quick, because you can actually have during your meeting, if it was in person, they’re the voice features are getting better and better, not just text-to-speech, but the conversational thing. I’ve tested this a

28:12 conversational thing. I’ve tested this a few times where you basically you start the agent as a skills and like, “Hey, I’m with my team, I’m with the UI guy, I’m with the my designer, and we’re having a chat, all three of you in the room, because I think we still think of it as one-to-one. Where it really could be an assistant that way, where it’s like, “When we are brainstorming this, we’re going to bring in this agent and these skills, and actually have that conversation where the computer is “Hey, grill us Tim’s here from UI. do you have any questions for Tim?” And, it’s

28:42 And, it’s it’s surprisingly good at that. actually, it’s scarily good at that. that’s just part of like the whole process. Like, just how active like so from from the initial concept of I have an idea, we need to tease out requirements, we need to then break those requirements down into tasks and things we can build, and then there there’s some there’s someone who’s orchestrating like a project manager role. Like we still need these things to be done, but what we’re trying to do is we’re trying to break all these Right. Right. large feature items up into smaller

29:13 large feature items up into smaller elements and then use them and build the software. I think that could be related across everywhere else that we’re talking about. But 100%. 100%. back back to your point, Tommy. I want to go back to what you were saying about skills and common skills,, , and the consistent outputs. That’s kind and the consistent outputs. That’s where we went. of where we went. That’s so true. So, that consistent output, the process of how you do business and work with your business, that’s what you’re trying to embed into custom agents, those skills, and give yourself consistent outputs. And I think

29:43 yourself consistent outputs. And I think the power of this is enabling the team to think about things this way. this way., it’s the same thing I was doing with people in Excel where I would just copy data out of one Excel file and put it in another and just be none the wiser, and that’s how we did work. Well, Power Query came out, and I had to physically stop what I was doing and say, “There’s a better way to build this stuff. How can I rethink how we build data loading processes? Let’s go use Power Query. Can I automate this? Don’t touch the raw file anymore. Don’t clean it up. Just use this new process.” And I think this is the same

30:15 And I think this is the same angst I have or pushback I’m having. Like we’re building things, we’re doing work on the day-to-day, but I’m like I still want to keep challenging ourselves to like can we rethink the process? Can we leverage agents in a different way that gets us the same output, but then make the system I want to build the system that is reusable. Right? As well as the feature that we’re trying to create. create. One thing I I want to touch this this as we get near the end when we get near the end end Yeah. Yeah. is thinking about creativity for the

30:46 is thinking about creativity for the person, their own personalization. But before we get to that, I think that this is a good point to all really bring up that’s more relevant. Is this all or nothing? And what by that is if I have a team that we want to have managed skills or governed skills, do we require every person to then understand learning AI and they must use those skills, right? Let’s say some people don’t use AI, right? Like let’s say you’re Mike let’s say you’re in a

31:16 say you’re Mike let’s say you’re in a company and you’re Mike Carlo. You’re Mike Carlo, you’re doing all the things you normally do with AI and there’s six other people, four of them know AI, two never used it. They don’t use it today. If you want to start doing this, do you have to then implement in your team that everyone is using the harness and everyone is now downloading and

31:37 everyone is now downloading and integrating AI into their process, right? Because it’s not just at this point you’re using the same skills, it’s now that everyone has to use the same process. People, process, technology, man, still comes true. So true. So do we have an all or nothing approach? If you want to use skills, even if I want to use my own individual skills, right? right? Like for a report design. If I’m the only person using report designing skills custom for me, I am going to have a different output

32:07 I am going to have a different output than every single person on my team. And probably drastically different. So do we have an all or nothing issue here? Mm. I don’t know, Tommy. This is interesting. Teams are adopting agents and agentic solutions at different speeds. Oh, yeah. Oh, yeah. Not every Not every company has every single person immediately on all the agent things. we We pretty aggressive and and

32:37 we We pretty aggressive and and really heavily pushing into the agent world and making people use and build create with agents very early on here. So I think we’re probably ahead of the game and we’re also a small company. Right. Right. Right. It’s easy for us to like pivot and move into the direction here. Now, now that we’re so heavy in agents, our costs are going up because of all the agents that we’re using that are changing costs. So now we’re focusing our attention on optimizing the agents that we use and how do we efficiently use the agents and can we run local models in combination with bigger models that like there’s a whole

33:07 bigger models that like there’s a whole bunch of other like optimization techniques that we’re looking into. But that being said I I think this I think in this question you posed Tommy, let’s just propose a scenario and see if this makes sense or resonates with you. Our team is trying to build an output of a consistent looking report. Mhm. Mhm. Whether I do it by hand in desktop by clicking the things or Tommy you do it using an agent to do the same thing. I don’t think the output should matter.

33:38 don’t think the output should matter. Meaning the output should be the same standard in both situations. Right? So in that scenario Mhm. Mhm. we’re we’re we want to define the output to be consistent. The page looks like this. This is how we style the visuals. Here’s the theme file that we’re going to use. to use. I think the output regardless if you’re using the skill or not should be consistent. Like let’s let’s take an let me give you another example. example. Sure. Sure. Your statement of work example you used earlier, right? You’ve already set the statement of work

34:08 You’ve already set the statement of work should look like this every time. Has these sections, has this feel, has this language in it. You you’ve told things on how to build your statement of works around how you like to have them written. written. Now the difference is you could have someone building your statements of work separately without any AI at all. And what that does is they have to just adhere to the standard. What the output The output doesn’t need to change. The output cannot change. It has to be consistent across both teams. The difference is, Tommy, when you do a

34:38 difference is, Tommy, when you do a statement of work, the time it takes you to build it should be less be less than it did for the other person to write every single word by hand into the statement of work. And also, Tommy, I think your output would be more consistent by using agents and skills to get the same regular output over and over again. Cuz you can have the agents check,, write up a bunch of this stuff, check and confirm, and then you actually provide maybe some lightweight testing in your,, output of your your product. And so, then in the agent can go through and test the output and make sure that

35:08 test the output and make sure that actually is working to your standards. So, So, when I when I look at that, I think the output of the work needs to be defined first. And then you can figure out how you adapt the skills to get that work done. And that way, when you have a team of mixed heavy users of agents and very light to no users of agents, you can still demand the same level of output. This will create tension in the team

35:38 This will create tension in the team because because the guy who can get the statement of works done quicker, the person who can get the reports built faster, that’s the person you’re going to always give that work to. And And you’re going to outwork or going to be able to outdeliver people doing things manually. Well, let me add another wrinkle to that, too. Regardless if it’s a person or a skill that I create, they both need two critical things. They need the the same exact same context. And more or

36:09 same exact same context. And more or less the exact same instructions. Right? So, if I give a person this, well, they still need to know my previous work. They still need to know the the structure that I I do the milestones or the cost structure or the different types of projects that I’ve done. They’re going to have to reference that cuz just like an a skill would or an agent would. And that’s the thing to get the same,, of course the writing’s going to be different. One may be longer, that’s fine. But the goal at the end of the day is

36:39 But the goal at the end of the day is take a look at this. This is deliverables. This is how we do Look,, refer back to all the work we’ve done. This is an implementation type project. This is a training project. And if they don’t have that reference, they’re not going to do anything with a standard output. Yes. Yes. Now, the problem here also, Mike, with this mix of things, and this is I think a good time to bring this up because I still want to get into managing and how do you distribute them, but that’s a little more technical. But I think this is more This is more fun of a conversation here.

37:10 conversation here. At what point though are you then limiting people’s own creativity, their own personalization? So, let’s say Mike, you’re like let’s let’s just keep running with the SOW one. You have people who help you write project plans, called project plans. And you giving them all skills, say, “When you write a project plan, refer to the transcript, use the skill.” They have They’re literally then not using any of their own human effort or their own human input, their own, personalization to that. It’s

37:40 , personalization to that. It’s literally just providing the skill that’s already been provided to them or using the skill provided to them without their own touch on it, without that person’s unique, creative creative touch. And what if they want to adjust that skill building? what? I find a more optimized way. Nope, can’t do that. We governed it., then we’re going to get a different output to your point. People are not going to like that. I would I wouldn’t, I’ll tell you what, I would be storming the gates at this because I’m like, “Then what am I

38:10 this because I’m like, “Then what am I doing here?” So, this is the other problem though when it comes to governed skills because then no one has their own unique flavor on it. Them as a human, what they can provide is gone. is gone. Yes and no, I still think I still think everything needs to be reviewed by a person at the end. Sure. Sure. There’s your language and how you write things, there’s your language and how you build stuff. so

38:40 so I I also feel like there’s a part of this in each of these tasks where you take take [snorts] [snorts] where the people good at? at? People are good in at to saying what is appealing looks wise or visual wise or, there’s there’s preferences inside how you want to build these skills or things in inside this process. Right. Right. I do think though Tommy though the output still needs to be very

39:11 output still needs to be very consistent, right? So the output cannot change in in a lot of these situations, right? You you need consistent output over and over again. And when I look at that that having that requirement of the consistent output you should be able able to allow some flexibility in the process to get to the output, but the output is rigid. And And even right now Tommy

39:44 I think you’re proposing an interesting idea. I don’t think you could even govern it right now. I don’t even think there’s any possible way you could even have consistent skills always across everyone’s computer unless you like code them into like, make a hash of them, make sure no one’s changing them or no one’s adjusting them. So I I’m not even sure like you can get to that level where the the skills you can even guarantee the skills are used across different people team members and make sure those skills are all the same skill

40:14 sure those skills are all the same skill all the time because once you push a skill to some user, it’s just text. It’s just a It’s just a markdown file. There’s a very How do you How do you deploy something with such rigid consistency that it doesn’t work? Now, Now, I’m going to propose maybe a weird idea, Tommy. Tommy. I think businesses are going to start building custom harnesses for their process, to your point. Right? Right? There’s this process that we do over and over again. Tommy’s always building a

40:45 over again. Tommy’s always building a statement of work. We’re building these reports. It’s going to look like this every single time. Or we have these this standards of things. What by a custom harness is you’re going to build code and software and embed the skills that you’re talking about into the custom harness. It’s part of the prompts that are automatically picked up and used and you cannot modify them. It’s It’s like literally So, it’s a difference between I’m going to VS code and I’m loading the skills that I need and then running that object. So, yep. So, yep. always modify the skills.

41:16 always modify the skills. It’s a difference between that and say, “Okay, Tommy, you’re going to make a custom website and with Fabric Apps, I think we had the ability to do this one, but Tommy, you’re going to say, “I’m going to make a custom site. You’re going to talk to it. You’re going to enter in text or speak to it here. You’re going to tell it what you want. The agent will take that text and run all the skills it thinks it needs to be able to produce the output. Or maybe a different way of thinking about this is maybe there’s an MCP server that you’re building, right? That So, an MPC server, I’m not

41:47 That So, an MPC server, I’m not adjusting That’s a That’s a much better, I think, distribution tool Mhm. Mhm. than a skill. I’m so happy you said this because you I’m so happy you said this because how a lot of times I bring know how a lot of times I bring something up, you’re like, “Why did you just say that? Because I’m building that.” Or I built that. Okay. Okay. And I’m like, “That’s so cool.” Well, this now is my turn. Okay, but What are you building, Tommy? How does this fit How does this really to try to accomplish this is this

42:09 really to try to accomplish this is this idea of different machines or different users to keep skills in sync. And if people have adjustments, again, how do you manage that? Well, how do you manage files, repos? So, I’ve been developing something called skill vault, which uses the Windows system or Linux system, and will actually have a location for your skills that will do system link or junctions to where Claude is, to where Copilot is, to where I think it’s Grok build now, to where

42:39 I think it’s Grok build now, to where Codex is on the local machine. So, if you actually go into the Windows computer where the Claude skills are, it’s actually referencing your skill vault. So, it’s actually not doesn’t live on your in the like,, username/. claude. It actually lives in the skill vault. So, all the updates are there, and it also works with different devices. You can say, “Hey, what’s going on on,, Carlo’s desktop computer? What skills do they have that I’m missing?” Yeah. Yeah. And then I can push them out. So, there

43:11 And then I can push them out. So, there is ways to do this. And I I I see what you’re saying with the harness, but to me, me, I want to use a skill of regardless of the harness. If I have a statement of work skill, I should be able to use that skill in skill in VS Code, in Cursor, right? Yeah Yes, I agree. But what stops me from modifying that skill? skill? Like so, Like so, Sure. Yeah, I know what you’re saying. So, I think I’m, let me let me Yeah, yeah, yeah. I know what you’re saying.

43:41 saying. Let me regurgitate your skill vault back to you. And just let me so I understand it correctly, right? So, it it it’s like a a wrapper around Like look, you’ve you’ve identified the locations of where skills should go on various computers across your team. Great. You’re able to then put a central like repo with skills that are in it. That’s the source of truth of that skill. And then you’re able to like dynamically say, “Okay, I want this skill to go from this repo to computer for this system, right? So, this harness, this computer, that’s where that skill’s going to be loaded

44:11 where that skill’s going to be loaded to. And then I can say, “Well, I actually want to compare two two skills across two different harnesses.” Those could be two different computers, it could be two different users, and you could and then you could have a consensus around what does that skill look like? Right. Right. But what I what what I’m maybe pushing back here a bit on or maybe trying to understand a little Yeah, yeah, yeah, yeah, yeah. Once that skill gets into that user’s library or wherever that location is, they can still change it, right? It’s still just text. So, it’s text, but it’s also going to reference to a repo. They have their own

44:41 reference to a repo. They have their own forked repo, the skill vault. And they push it up, they go, “Oh, someone made a commit difference. Why do, why is this part different than the than the the original?” Yeah. Yeah. And we can actually in skill vault, or at least at least you can actually see the different line difference of what was changed, what was adjusted. So, you can go, “Hey, Mike, I noticed that you took our you Mike, I noticed that you took our, set a knowledge center skill and know, set a knowledge center skill and adjusted a few things on your commit. Let’s talk about it.” Now, you can actually push that skill into the main

45:11 actually push that skill into the main vault, so that can be the thing that goes to other devices. Sure. Sure. If you want to personalize it, you can. If you want to pull back whatever the current source is, you can. And I think is it perfect right now? No, I think from an enterprise I think we’re going to have things like this, but this ability ability is going to be so essential because to your point, we’re not going to This is not emailing skills back and forth. That’s not going to work. Mhm. Mhm. And And Yes, correct. I I I think that that’s going to be part of the adjustment, too. Because you want people to bring their own flavor. I want

45:42 people to bring their own flavor. I want people if I give them a skill or we have a skill for knowledge center, they’re like, “You knowledge center, they’re like, ” what I found actually? If I know what I found actually? If I adjusted the script here and I adjusted the resources, it’s so much better at producing how-tos.” Okay. Well, again, that comes part of the vetting process, part of the government process to say, now we’re going to update that. Everyone, please do a poll on the skill vault. You’ll get the latest updates. Like we had with SharePoint with the PBIT.

46:12 PBIT. I don’t think it’s a perfect process yet. I’m not saying I have solved team management of skills, but to me there is a process and some governance that needs to be part of this. Yeah, I I’m I’m not sure I’m sold on this whole multi-repo thing yet. I maybe I need to spend some more time with it. I do agree, Tommy. Skills are difficult and what I’m finding I think in my workflows are are individuals are building skills,

46:43 individuals are building skills, but they’re not easily shared across the team. So, people are building good stuff. They’re making good outputs. People are learning things like people are learning things individually, but I’m not able to like unpack that learning and bring it out and distribute it to the broader team. And this is where I think maybe MCP and custom harnesses make a little bit more sense because in those situations like a custom harness, harness, whatever code you push, Tommy, to that custom harness is there. Right. Right. see it and you have control of it. And

47:14 see it and you have control of it. And the skills that are used and skills that are being,, built into the to the agent directly, right? Through the through the custom harness, you now have tight control around what those things are doing and what the output is. So, the other note here I would maybe that came up while you were talking, Tommy, was I don’t know how we measure what quality good looks like in this situation. So, we’re talking about the team improving skills and the team enhancing things and the team making love it. Agree. Definitely want it.

47:46 Agree. Definitely want it. But how what is the standard for success? success? Right? If the Do we and I’m where I go with this is if that skill gets modified, I don’t have a test bed to verify that it’s doing what I want in different ways. Does that make sense, Tommy? Like Yeah, Yeah, this is I I don’t think we’re going to solve this. I’m not going to solve it here, but I’m just saying like I don’t know I don’t know what this answer looks

48:16 I don’t know what this answer looks like, but I do feel like Tommy, you make a So, it almost feels like I need a system, a process where Okay, Tommy, you we’re going to make a whatever a report Power BI report skill. And this is actually one of the things I’d like to talk to Microsoft about how they built skills for skills for fabric. How are they testing it? Do they have a test harness? Are they managing it? Do they have a test harness to say, “So, like if you change one part of the description of a skill?” Like Let me think of it this way, Tommy.

48:48 Okay, Microsoft is building skills for fabric. This This is the exact same problem they’re having right now. I’m going to build the skill. I’m going to put it in this repo. Okay, how does that skill perform against Claude code, VS code, other harnesses, Codex? There’s like all these different things. So, you almost need like an an entire infrastructure around I’m going to submit this skill to something that will run all the agents, multiple models, and give me readouts of like how do I measure that

49:18 readouts of like how do I measure that the skill actually does what it wants? Claude actually just put I think that a few months ago they actually said how we’re testing skills. And I think it was very unique for them, but there is a way they have a it’s a benchmark skill actually. I’m pretty sure of this. This It’s called benchmark that allows you to measure you like you can put the test together. It’s like, “Hey, I developed the skill. I want to test it.” It breaks helps you bring out the success feature. How critical is that to Mike? Like if I

49:48 How critical is that to Mike? Like if I develop a government skill, must it go through a benchmark? Like let’s say we have one for templating or or theme design. design. Again, Again, I don’t know if this lemon I don’t know if this lemon is worth the squeeze, right? So, again, I’m going to go back to like I don’t I think you need it, but do we really need it?, Yeah. Yeah. I can see I can see this going both ways, Tommy. I can see it going one way, which is like it’s not needed, right? Why would I need to test this skill against every single agent that’s out there or harness that’s out there? That doesn’t make sense.

50:18 doesn’t make sense. But, I But, I I would need if I’m going to build a skill and publish it, I’m going to need some way of evaluating what is the efficiency of this skill in like token usage and how close it get to the desired output. So, I have to define the output of the skill and say this is the output I’m looking for, run the skill, because you can just describe many different things, you can add more instructions, you can add less instructions, you can do a whole bunch of other things inside that skill. One of the things that I find Microsoft doing right now, which I find a little bit fascinating is most of their

50:48 little bit fascinating is most of their skills for Fabric are only markdown files. That’s it. There’s the There’s the skill markdown Yeah. Yeah. and the skill markdown file is actually quite large., the description is very verbose. There’s a lot of descriptions on the skills file, but that eats up my context window every time I need to use the skill inside the agent. So, as soon as I’m asking the agent to go get it, if the description is very verbose, that’s eating up tokens if I have that in my model. And it’s also not really taking

51:18 And it’s also not really taking advantage of what a skill can do. Because a skill can have Python scripts, it can have node. It does, it can have really any language that will perform a function. that’s when you break apart a task, anything that you can give a script or a function to is advisable, I think. Because as soon as you start giving scripts and functions to the skill, the skill gets really consistent on what output’s doing. So, for example, Tommy, I was doing I have a skill that I use for pod posts. So our

51:49 for pod posts. So our our podcast that we do here, right? We do update our website parlia. tips with the podcast and we actually have a full transcript that’s happening. I’m doing the same thing for agentic thinking where we do an episode, I transcribe it and it goes on our website. So this is a very consistent skill but the if you gave the VVT or the transcript file from YouTube to an agent, there’s a lot of grunt work that’s happening. And I was trying to do it only with just the agent and it was expensive, it cost me a lot

52:19 and it was expensive, it cost me a lot of money, the agent didn’t run very efficiently and it and sometimes it would break and fail halfway through. I didn’t know why, it was very frustrating. I give it a task, go download this video or get the transcript from this video and do something with it. Well, what I found was was it was not efficient and so we had to go back to like first principles here and say, what tasks are being done in this process, turn as many of those tasks into

52:44 turn as many of those tasks into code and script and Python scripts. So now my skill is much more consistent. And I can easily get out an episode and a blog post on top of an episode that we’ve done because How long did that take you? Couple Couple hours I guess to to figure it out. It it it was it was across multiple mean it it it was it was across multiple days but I wasn’t working on it consistently. It was like an hour here and that’s not working and I figured it out. So but I worked with an agent. So this is where I’m going to double down on the creator agent is where it’s at because I could see the problem. I had I

53:15 because I could see the problem. I had I had the the vision for what the problem was, hey, we do a lot of YouTube content and I want to get that out and transcribed on on the internet. That was my goal and we had the we have a website. I had it so I now had I had to rethink how I built my process and so I had the agent work with me. We did it once where it worked but then I kept asking the agent simplify, simplify, make more skills, turn the skill into more scripts and and like all these other things and that was what got me to

53:46 other things and that was what got me to a whole collection of scripts. So, now we have like I have like five Python scripts that help me transcribe videos. It gets the text out. It It formats the blog post the right way. And to your point, Tommy, formatting websites and getting website blogs out are very consistent. You need to have a that post look very consistent in format. That’s a great use case. The output is the same. It doesn’t matter how I get there. there. I want to offer something here because I know Macintosh did this back when they were Apple. Google we know has done it.

54:18 were Apple. Google we know has done it. But, But, I want to I want to for I guess closing or really main thing is the individual experimentation here. We have no I think we’re still at the cusp of what skills can do and the capabilities of them. We’re still finding that out. I would propose that if you are a team who wants to use skills and have a government that you need I would encourage my team 3 hours a week or 4 hours a week to experiment with skills. I would say you have time in a garage,

54:48 I would say you have time in a garage, we’re calling it garage skills or where they’re going to just experiment because I don’t I still think we’re at the cusp of this. Apple did this when it was called Macintosh back in the Steve Jobs day that he gave everyone time and that’s where a lot of like a lot of the features that in a computer today came from. Xerox did that when they came up with the graphical user interface. Didn’t have a set projects and Mike, we don’t know what we don’t know. know. and I think even if I have governed skills, skills, I would encourage my team to spend time

55:20 I would encourage my team to spend time developing, experimenting, and and focusing on what a really skill can do. I agree with that, Tommy. I’m also going to throw one down here in this is well, which is not every skill you’re going to use is going to be efficient, honestly. Mhm. Oh, yeah. When I’m looking at the Fabric skills, I don’t see a lot of I don’t see a lot of scripting. the skills that Alex Powers is using, he’s got more of an orchestration thing going on. So, he has the task flow assistant that he built.

55:51 assistant that he built., , Yes. Yes. less guessing, more building, right? That’s what was his his his scenario. In that system, he has a lot more scripts. There’s a lot more scripts. It’s a mix of skills. It’s a mix of custom agents, and there’s a whole bunch of scripts that are doing very consistent input and outputs. So, you consistent input and outputs. So,, to make your agent, the large know, to make your agent, the large language model, really consistent, you need to think about what is it need to do. do. What is the consistency that that that file should,, creating files, files, functions, scripts, right? That should be consistent. You don’t want the agent

56:22 be consistent. You don’t want the agent building that stuff., you want the agent doing things that it can only do, which is like reasoning and thinking about text and summarizing and act like you you need to figure out where does the agent need to fit inside this process that you’re building, because the skill is a mix of that consistency of scripts running over and over again. This is why I keep going back to the creator agent is so important for us to get our heads around, around, Yeah. Yeah. because because you want the agent to help you create the thing that can run consistently without an agent involved,

56:53 without an agent involved, right? right? Right. Right. So, So, I’ll close on this, and I’m going to ask you the same question. Yeah. Yeah. Or I’ll ask you first. I’m a user, I’m listening. I want to get my team started with skills. I want to create my own skill. skill. How do I start? How do I create my own skill, a skill that I want to propose to my team? Yeah, so I’m I’m going to I have a pattern that I like to use when I’m creating skills and when I’m building systems. when you’re working with an agent

57:23 when you’re working with an agent or an our large language model, the first prompt you give it is usually the best part of the prompt. It gets polluted after that, and it gets confused. so when I’m I’ll build the process work with the agent with a new session. So, I’ll start a new session, I’ll build the process. I’ll do it once myself with the agent. I’ll come back at the end of building that that session with the agent, I’ll say, “What did you learn about this?” It’ll describe, “Okay, we did this. We we we downloaded this file. We made this

57:54 we we downloaded this file. We made this thing.” Da da da da. I’ll [snorts] ask it to learn some things. The next thing I’ll do is I want to create a skill around what we just learned. learned. Design for me how we would make this skill work. So, then it thinks about that, does the skill stuff, da da da da. Great. So, now it reasons about the skill part. Awesome. Awesome. The next thing I do is, okay, now that it has that skill there, the final step here is, okay, review the skill. What of this skill can we break into tasks and we can turn into scripts so

58:26 tasks and we can turn into scripts so that you understand how to do these consistent things over and over again? And I think that’s something I’m starting to do. I haven’t done that a a ton yet, Tommy, but I’m finding skills that don’t have scripting in them, it gets frustrating because you think the skill is going to do something and then it does it sometimes, but then sometimes it goes off the rails. And if you have a large context window and you’ve been using the agents for a while and you invoke a skill in the middle of the session, it sometimes doesn’t handle it right. I’ve seen just weird behavior. So, I want these skills to be very consistent and reusable.

58:58 very consistent and reusable. What do you think? What what’s your take? take? No, I I I I want to do more on the script side, too. For me, Mike, it’s honestly there’s a great skill out there that Claude developed called build skill or skill creator. creator. Yep. Creator. I’ve made a skill with the great skills, which helps with this. this. Yeah, exactly. Yeah, yeah, it’s a it’s a circle. And but also when I’m working on something, I’m like, “Oh, this would be a good skill.” I’ll ask Claude or or the the harness I’m working in, “Hey, there’s a skill

59:28 I’m working in, “Hey, there’s a skill here. Can you actually write instructions for me? I’m going to put it in the chat. And I’m going to use the skill creator skill. skill.,, and I want to go through so just write instructions for me kind so just write instructions for me in what we did. So, summarize it. of in what we did. So, summarize it. I’ll put that in a new chat. I’ll use the Grill Me skill. And along with the skill builder because you’re using the skills together. And we’ll we’ll go through thing like let’s walk through this together to your point. Here’s the things I want to be able to automate. Here’s the things I want it to do. For example, with the project plan I have, it will interact before generates

60:00 have, it will interact before generates anything. It runs through a steps of questions. It’s like, “Oh, it looks like this is this type of project. How do you want to go about it?” So, it allows for my input., and then to your point, I want to integrate like I’m going to go back to it today and I’m going to say, “How can I we modify this? Look at all the I always put skills in projects, too. This is one thing I want to also point out. Like for example, when I use the SOW skill, I have a consulting age project. So, I say, “Hey, look at all the pra- past chats here that use this skill. What are

60:31 chats here that use this skill. What are the common things that I’m doing that how can we optimize this? And Claude will go back to my chat can can Wait, I’m not sure if I understand project. Is pro- Do you mean do you mean is a project in Claude? Is Okay, project in Claude. Okay, that’s why I’m diff- cuz I’m not using Claude as much as I should. So,, that’s something that I’m not as familiar with., It’s pretty awesome. So, other things with other agents. I don’t have this concept of project. So, if you do have project well, even if you’re using like, open Claude, right? They have different agents that

61:02 right? They have different agents that you can create. And I try to be very specific on what I’m going to do with a certain thing. I’m going to ask it to use an agent I created. That way it can reference the memory. Makes sense. So, yeah. But anyways, I think the I think the big takeaway, Mike, is are we there yet where we need to start doing governance skills? Maybe not completely, but it something we need to be thinking about. Absolutely. Yeah, I I think Tommy, this is a fluid topic. We’re going to keep going back to this one. I think this is going to keep getting refined over time.

61:33 to keep getting refined over time. I think the only true way to really solve this one is building custom harnesses that start locking down your process into a consistent thing. I think that’s where companies are going to go. Cuz you I think that’s what you really need, honestly, at the end of the day. to build consistency without having too much flexibility for the team. All right, that being said, thank you very much for listening to the podcast. Tommy and I thought we were again, like always, we thought this was going to be a short episode. We’re going to go quick here with this one, but apparently there’s too many ideas for to unpack together about your team versus

62:04 unpack together about your team versus individual skills for agents. could be Microsoft Fabric, could be any agent at this point. It doesn’t really matter at this point. point. but the the problem exists. This is a a solution in the area that we have to figure out where do skills fit, who uses them, how do we distribute them, how do we govern them? I still think that’s very much to be determined at this point. But some good thoughts via the podcast. podcast. Tommy, where else can you find the podcast? podcast? You can find us on Apple, Spotify, wherever you get your podcast. Make sure to subscribe and leave a rating. It

62:34 to subscribe and leave a rating. It helps us out a ton. Do you have a question, idea, or topic that you want us to talk about how you’re using skills in a team? Let us know. Head over to powerbi. tips, leave your /podcast, leave your name and a great question, and finally, join us live every Tuesday and Thursday a. m. Central on all of powerbi. tips social media channels. you next time. Thank you for joining us. Explicit measures, pump it up, be high. Tommy and Mike lighting up the sky. Dance to the data, laughs in the mix.

63:04 Dance to the data, laughs in the mix. Fabric and AI, get your fix. Explicit measures, drop the beat now. Podcast themes, feel the crowd. Explicit measures,

Thank You

Want to catch us live? Join every Tuesday and Thursday at 7:30 AM Central on YouTube and LinkedIn.

Got a question? Head to powerbi.tips/empodcast and submit your topic ideas.

Listen on Spotify, Apple Podcasts, or wherever you get your podcasts.