OneLake Data Catalog & Governance – Ep. 382

The crew digs into OneLake data cataloging and governance: what it enables, where the current gaps are, and what to watch for as Fabric’s governance story evolves.

News & Announcements

• Got an idea or topic for a future episode?
• Subscribe / listen to the Explicit Measures Podcast
• Join Tips+ (Theme Generator)

Episode Transcript

Episode Transcript

0:33 welcome back to the exclusive measures podcast with Tommy Seth and Mike good morning morning everyone it is a morning it’s a good morning morning good morning I feel like it’s good morning why is it so good for you Tommy what I just look at this and I I realize that we’re at 384 episodes maybe it’s the end of the year and you’re thinking about what you’re thankful for maybe that was supposed to be thank 382 you’re over you’re overestimating we’re out there hopefully we get there you’re such an optimist you’re you’re in the future Tommy by two weeks by by two episodes it is pretty crazy I was looking at some of our

1:05 crazy I was looking at some of our podcast stuff they’re like by the way people have listen to us for 15, 000 hours wow why what a waste what a waste you’re never getting that time back guys what are we doing interesting 15, 000 hours that’s a lot of talk talk time one thing that astounds me about the internet whether you’re doing video or prettying content or media it’s it’s the volume it’s the volume of reach that you get now this is why I think tech companies make so much money

1:36 tech companies make so much money because they can do a little thing and reach thousands of hour like we look at the the content that we produce and it’s thousands of hours of watch time it’s just incredible so anyways this is I guess attestment to the community thank you very much for paying attention listen we would probably still do this anyways and shot into the void and scream at each other without a podcast but we enjoy that you come along with us for the ride as well just for a different Insight on that that’s almost 2 years of time oh my God what a waste

2:06 2 years of time oh my God what a waste 15 15, 000 hours yeah there’s 8760 hours in a year that’s pretty good that that’s 247 that’s like non-stop listen yeah yeah and this is coming from the heart we said when we started that we wanted to do this type of format just like if we were having a drink or just talking or meeting for breakfast and the fact that people listen to us and honestly I forget sometimes that people do listen I’d enjoy these mornings just talking to you guys and having some of the conversations like today is gonna be

2:38 the conversations like today is gonna be a good one but we really appreciate everyone who listens you guys really do I think help us do this and we’ve been doing it twice a week for four years now four years three years I don’t even know when we started started question May 4th 2021 okay for hey wow dang three years that’s pretty impressive oh you’re right4 well it will be in over another

3:08 right4 well it will be in over another year we’ll be we’ll be good all right enough yammering about all the time and the statistics of things Tommy and I and Seth all love the numbers of all the things so that’s that’s why we go tickles the right parts of our brain that being said let’s go into some news artics oh before sorry I keep doing this before we go to news let’s talk about our main topic at Microsoft night there is announcement around the one Lake data catalog and I’ve been playing around with it a little bit I’ve been reading some of the documentations

3:38 been reading some of the documentations on it I want to react to what we heard around the one Lake data catalog features I also want to react to some features that I’m I’m looking at it right now and yeah it’s it’s coming it’s coming together it’s definitely in its early phases I would say but i’ be just just going to unpack what we’ve seen so far and where we think things are going with the the one Lake data catalog I’m actually really excited about it I think this is a really rich feature set to fabric it’s actually quite needed as we

4:08 fabric it’s actually quite needed as we get larger and larger organizations with more and more data showing up so very excited to talk about that one that’s our main topic all right Tommy what’s our first news item that we have on docket here today yeah we have a a onew punch of warehouse and database updates really around the the functionality around your experience not necessarily about the product itself but one is staying up updated with the intell a sense in the fabric Warehouse just making it a lot easier to do some of the commands that you’d want to do which goes along with the other update

4:38 which goes along with the other update which is the command pallet in the fabric warehouse and in SQL database editor this really goes to me when I’m thinking about our conversations around Microsoft’s cadence of product release is it’s really much like we’re going to introduce the foundational tool but we have all these other ideas on what we want to release again nothing here is completely new like squel SS SMS has had these features for a long time but it’s really nice to see

5:09 long time but it’s really nice to see with even all the products they’re integrating into fabric the fact that we also get the nice user experience too very interesting that’s that’s good new so you made a comment there that that maybe I have a either a thought around around or let me step back two thoughts really one is intellisense I am using intellisense in the Dax query viiew very helpful it’s a little bit aggressive because if you make a new line it immediately tries to pop up a

5:40 line it immediately tries to pop up a list of functions for you right away so that’s the add columns I keep accidentally adding all the time in intellisense in Dax cor review so I hope they fix that but in general though I would say intellisense is a pretty robust feature wherever I see it and I rely upon it and I don’t think you it’s one of these subtle features that I don’t think you realize how much you rely upon it until you have a tool that does not have it where you have to everything out no AB absolutely I agree and the more more

6:10 AB absolutely I agree and the more more any of the code helpers help whether it’s name put it slap a name on it right it intellisense the even looking at the definition like hey these are required this is optional yeah yeah like with when it’s hey you’re you’re using this join a ton so I’m just going to say is it this one and it’s very a lot of them are very smart so yeah absolutely it’s a it’s a huge timesaver yeah huge timesaver because like it reminds me it reminds me now I can for somebody who doesn’t code as

6:41 can for somebody who doesn’t code as much anymore because I lead teams of coders it I probably code faster now than when I was doing it full-time because I I still remember when when you’re in the guts of something and you just go oh man okay I’ve got to join to four different tables this is going to be the same repetitive thing over and over like or here’s the slight alter and adjustment right you still have to type it all out you’re still like like or you’re copy and pasting and now with intellisense it’s

7:11 pasting and now with intellisense it’s just yeah it finishes the whole thing for you A lot of the times it’s really it’s really nice man it’s a nice cheat code these helpful tools and Mike I have the same issue Microsoft really wants me to do absolute functions it’s like but that’s not what I’m looking for every time I do new line it’s like no I don’t want to do an absolute I don’t need absolute yep these help for things like I If you took away auto correct or spelling for me in my day-to-day I wouldn’t graduate college like there’s a good chance I’d get through high school but I don’t think I’d get farther so yeah there is if it

7:43 I’d get farther so yeah there is if it doesn’t have a I I often write on whiteboards for things and there’s no red squiggly lines on whiteboards that tell me when I’ve spelled something wrong so my spelling really is atrocious when I’m not using a computer for for whatever reason I just did a dashboard in a day and one of the things we’re doing top competitors and I’m like I wasn’t sure in my ability spell competitors in that moment I’m like we’re just gonna say top comp that’s what we’re going to do so everyone just write top comp don’t worry what that means so no I but it’s nice to see this too in the web right because I think a

8:14 too in the web right because I think a lot of these things were used to in like Visual Studio or vs code or obviously powerbi desktop but Mike I hate giving you credit sometimes but I have to I have to the these products are becoming very seamless on the web and they’re becoming is to be desired like building a model in the web I’m going to write like a c d plus wait until get timle viewer when you get timle viewer timle editor inside the web dude I’m telling you already I’m using

8:45 dude I’m telling you already I’m using timle editor inside vs code it’s really slick I’m very excited to see where this is and and I think so you’re on point they are doing a great great job and like they’re taking some from the power apps books because their only way to build there is that and you feel like you’re not in the web and they’re doing a great job doing that too in fabric so I love how it pains you love how it physically pains you to say hey Mike you’re right well not just right

9:17 Mike you’re right well not just right like 100% right too I’ll give him some credit but 100% well for those for those who who are listening who are newer to the the podcast Tommy’s referring to episode one so you should go back listen to episode one where I make this prediction and here we are it’s really coming together now it’s becoming quite quite the plan to

9:40 becoming quite quite the plan to bring everything to the web and I I love I love it I think it’s good one other comment I had here Tommy you made a comment around SS SMS has already had intellisense particularly for the the SQL data warehouse experiences of things one thing I’ll note here is SS SMS has been around for a long time I don’t know the pattern of like when a new tool comes out how long it takes them to add intellisense a lot of the tools that we’ve interacted with or at least I have interacted with in the data space the the intellisense

10:11 the data space the the intellisense on ssms I’ve never not had it it’s SS SMS has always had intellisense so I don’t know how long it takes Microsoft to build a product that gets intellisense so we may actually be ahead of schedule as far as getting better intellisense based on when the product arrives till when the intellisense shows up I don’t know but the fact that a product has had intellisense for a long period of time isn’t in to me isn’t isn’t a speaking Point rather than how long did the product when was the

10:41 long did the product when was the product out and when did it first get intellisense I think that’s more of a bigger Milestone because that means Microsoft is investing in making it easier for you to write code you’re making does make sense what I’m saying no I think it’s a great yeah go ahead yeah yes and no yes and no because like the deployments of SQL work they’re like every six months or something like you can’t screw screw that up otherwise it’s a big deal related to like okay we just we just shot this out and actually Intel sense is why you need to upgrade all of your systems again like here’s

11:12 all of your systems again like here’s the patch for that true true first versus I I think one having established that functionality how like how it gets implemented plus the advancements of Technology Cloud where we’re at sure and it being in fabric I think offer up potentially faster paths to integrating existing platforms into that but that’s just like I said that’s a guess you’re bringing up there’s one other wrench here and not to divert too much but a pretty big

11:42 to divert too much but a pretty big wrench here too is I’ve been playing a lot with like the Azure AI Foundry the new AI studio and yeah I haven’t had a chance done I don’t have much experience there but I think the the bigger part is with all the co-pilot things too if I were to ask you what would you if I had to take one away from you you intellisense or co-pilot if which one are which one are you keeping and I think the intellisense was our AI has been our AI for all these years but now with the introduction of all these helper coding models you

12:15 all these helper coding models you all these helper coding models like be it again whether it’s local know like be it again whether it’s local or you’re using co-pilot or using a GitHub co-pilot is that not more valuable right now and useful than your intellisense on it depends right so if you talk about co-pilot for writing like very common languages like python that are already very well trained on that and SQL I would agree co-pilot is better than those but co-pilot inside the scope

12:45 than those but co-pilot inside the scope of Dax yeah it’s not it’s doing some interesting things it definitely does some stuff it helps me explain the functions it gets me documentation when I need it it’s it’s being used a little bit more of like a lookup tool it does suggest some measures at this point but I it’s very limited in the Dax space so if from the code languages that I need to know where do I want to apply copilot where does it find it more valuable than intellisense well for sure it’s in Python and for sure it’s in SQL those are two very robust languages and has been very well trained on them

13:17 and has been very well trained on them if you throw co-pilot at report side of things I’m a little less inclined to use it and it seems a bit more new or they need they’re still trying to refine the experience to figure out what really needs to be involved when co-pilot shows up to a report or a page or how that’s going to work so I think if I picking my choices here Tommy so to answer your question directly is it is copilot better than intellisense yes it is it’s got It’s got more legs to it and I think it has more potential to grow better than straight intell sense do I pick it

13:50 than straight intell sense do I pick it over it right now it depends what tool I’m I’m in that’s how I feel about it so anyways no I yeah that’s a good point maybe move on to your other article or did you already talk about it already the the second article here is introducing the command pallet in the fabric data warehouse this is a pure like notebook play Fields like right here this one yeah and I think that’s yeah we lumped them together in terms of again two features or I guess user experiences that have

14:20 or I guess user experiences that have been available in other products in Microsoft but now basically readily available in the web for those Tommy who don’t understand what is we say these words and again when I talk to my wife about anything about data she’s like you just say words I don’t understand the context of your words so explain for us really quickly what is the command pallet and what does that mean when that shows up in the fabric data warehouse yeah probably little context you have a lot of options and actions you can do in the data warehouse and SQL database

14:51 the data warehouse and SQL database whether it’s writing code formatting but rather than having a giant menu like your word where it’s like edit and then there’s a ton of ribbon there I can bring up this pette and search for the action that I want to do for example if I want to format my code I in in the case of the warehouse I click on F1 or type F1 and I immedately type in format if I want to search for a function if I want to save if I want to add something or look for a table all those things I can do using the command pal and just simply searching for it’s this nice

15:21 simply searching for it’s this nice little ribbon that comes up now again the only problem with web and product if you’re going from like Visual Studio that’s usually your control shift p which is print on the web so we run into those things but I think the bigger Point here is any action I want to do in the interface on in all I have to do is bring up the command pallet just by clicking F1 and search for the action that I want to do so this is interesting and I was just going through a group of people and explaining them like we were doing Dax

15:51 explaining them like we were doing Dax review we were showing them how to use the the pallet feature inside or index review so it has its own pallet there as well the command pallet it’s all bunch of commands you can go do really interesting it’s very useful my only challenge though is there’s a lot of commands in there I don’t know what they do and I don’t know where you find like what’s the list of every action in that list and what should it do based in the context of dax’s query viw so I don’t really know all the things at this point but there’s a

16:22 things at this point but there’s a couple that I did understand and knew how to work but my favorite command of all the commands out of the Monaco or the the editor with code is control shift L is my favorite and what it does is it allows you to highlight some text and it selects every instance of that text and it also puts a cursor at the end of all of that so you can have multiple cursors if you want to impress your friends this is this is just a a tip for your tip for you who listen the podcast if you don’t listen to anything else we talk about today you want to

16:52 else we talk about today you want to impress your friends and while your neighbors use multiple cursors in an editor people will be floored like wow you’re impressive like that is an impressive skill to have do that’s available in Dax in powerbi not even the queer riew in the formula bar oh of course I know this this is why I use it this is so amazing that’s why I’m bringing it up I’m taking your Paton and I’m running with it I think bring up d control shift d selects one item or control control D selects A Single

17:24 control control D selects A Single Character like a group of characters so if you highlight something it’ll find the exact phrase of it and if you control D multiple times right again and again and again it keeps selecting the next ver like the next found version of that same bit of text so if you want to if you have three areas or you’re modifying multiple things at once yeah it’s very easy to control D boom boom boom boom and then you have these things highlighted and now you have multiple cursors and you can modify around the other last trick of all this thing is if you want to just drop the cursor multiple cursors down you can use

17:55 cursor multiple cursors down you can use the ALT key and just drop click alt and click wherever you want the cursor to live so if you have crazy amounts of formatting you want to do all at once and you want to type it once but you want to have multiple cursors dude I’m telling you wow it is very powerful so anyways that’s when a professional what else is really powerful that I I just want to point out here Mike yeah in when the SQL text is formatted what what do you notice in notice in there nope we don’t notice anything

18:26 there nope we don’t notice anything it looks just good formatted literally there was literally an entire episode where I think we were combating on the the pros and cons of putting your commas before or after the wait a minute wait minut here though wait a minute wait wait there the screen the screenshot is showing they’re about to hit the button for formatting it’s it’s going to fix it it’s it’s not currently fixed that’s the problem oh that’s a good one so we

18:58 the problem oh that’s a good one so we had a t-shirt on that so either either one of two things has happened right either Microsoft has sided with Seth or someone has infiltrated the team to be the person who chose how to format the sequel so so that it’s on the other end there’s this the the in in the select statement do the

19:21 the in in the select statement do the commas belong before the the colums you’re requesting or after and Mike and I were on both sides I’ve always done it after after that’s the the the Dr Cur is that that’s how it’s formatting right now is it’s it’s after the the column in mic’s big before so I’m going to give you one little insight here the whole reason that I am commas in front of the code is because I googled this one time and I found an article that said hey we ran a whole bunch of is this valid

19:52 ran a whole bunch of is this valid code across hundreds of thousands of samples across GitHub and they said of the valid code that was written or published out to GitHub of that valid code more valid code was found when the commas were in the front versus when they were in the back and so I was like great random article on the Internet thanks I’m going to use that and change how I do every bit of code writing for the rest of my life because of that one little article so now so now I like to do I prefer to put commas in the front

20:22 do I prefer to put commas in the front whenever I’m writing SQL or other codes now when you run this in Python notebooks though the commas are in the so there may be splitting some things I think there’s other places so Tommy you might have to pull up a data warehouse here and go format some SQL code and let us know where does the commas get placed when they format it automatically are they putting it in the front they’re gonna they’re going to put it in the back that you and I both know that I I’ll take a look but Millennials are now at Microsoft I don’t know man things could be changing the people who are commas in the front are very few and

20:53 are commas in the front are very few and far between and they are going to go with the majority not the best code because I agree I have converted to to a front coder or front comma person myself they lead the way they help you get there’s less errors when I write C I’ it’s it’s anecdotal at this mean i’ it’s it’s anecdotal at this point it’s anecdotal because I I feel like I write less errors when I write with commas in the front because I know oh it’s a new line drop drop a comma if I want to comment it out I can comment out the code better it’s like it just all feels a little bit more fluid when I’m writing it so yeah

21:23 I’m writing it so yeah wild wild anyways a good a little take Seth all right we’ll jump into our main topic for now so jumping into the main topic let’s go over to our one Lake data catalog Tommy give us a quick overview there is a one L data catalog feature that is out so there’s there’s an official blog about this I’ll put the blog the one Lake catalog that is now it’s a very sparse piece of documentation at this point but I’m assuming over time as features are added

21:53 assuming over time as features are added we’ll get more and more features here so Tommy give us a quick overview of the one Lake catalog so we’ll bring it back to Chicago and one of the three main features that iron and the Microsoft team was releasing was this idea this concept of the oneel catalog now you may think oh that’s just the data Hub with different cosmetic a different dress so to speak well right now they have made some major user interface changes available today you can check it out right now if you haven’t it’s now

22:24 out right now if you haven’t it’s now called the one L catalog and really it provides all the actions that you’d want to do on any type of really artifact it’s not just your semantic models is available if I want to refresh schedule refresh if I want to take a look at some of the metrics any other actions I want to do but importantly one of the features not available yet but I think going to be a lot of our conversation today is this idea of governance in the one L catalog some of the things that they mentioned during this was the fact

22:55 they mentioned during this was the fact that you can go to the onl catalog go to the governance Tab and I can see hey 40% of your models are not tagged 30% of your models that are semantic models are not certified or or promoted at all they have no certification on this so there’s this list of features that really join in a centralized area things that you would want to do from a center of excellence or from a data governance point of view for our sake for our conversation today I think there’s a few

23:26 conversation today I think there’s a few questions that we want to ask or that I are going to come up and I think they’re going to continue to come up is how is this this impactful how does this integrate with a governance team and and really should it should this be part of a daytoday or the weekly calls the features that are available this is something that I think we all see as the Holy Grail not the one like catalog but the idea of a centralized area to look at all my data to say what can we improve on where are we lacking where should we prioritize

23:57 we lacking where should we prioritize that is something we’re we all trying to seek and there’s not one product yet that can do that so one of the questions is is that it but let’s start listen with the idea of the data Hub and now that we’ve transitioned to the one L catalog have you guys ever used the data Hub in your day-to-day for trying to find a model and like what this idea this idea is to be frankly honest the data Hub is not extremely relevant for me I’ve kind not extremely relevant for me I’ve favored did the things that I need I of favored did the things that I need I I know the workspaces I want to go to I

24:29 I know the workspaces I want to go to I live again let me give you my my navigation pattern right I’m more of again I’m not a normal user anymore in PBI I’m more of a pro user I’m I’m going into build thing create content I know where I’m going so I think I think my navigation pattern starts with more of the workspace level and then go from workspace into what I need to build to create directly from there I have found that my in my building more automated of reports it’s just very nice to have a a lot of favorites so immediately log in

25:00 a lot of favorites so immediately log in go to the favorites area I have my couple things here that I want I click on the one or two reports at the top of my list I go right to them again this is the the point we’ve made I think in the past here is think of your organization as a pyramid there’s going to be a lot of things at the bottom level of your pyramid the found like the there’s just be a lot of junk that’s made or things that are just assets that are there you look at them a couple times you move on not a big deal you don’t do a lot of extra things with them but there’s a couple reports that you’re going to have that you’re going to always look at they’re always going to be a top of mind and there’s like a handful of things you need to get access

25:31 handful of things you need to get access to a lot so I don’t do a lot of just general discovery of the one Lake catalog or the one Lake Hub at this point I don’t know how about you Seth creature I have like I know where everything is so like typically I’m diving in directly into the locations without using the data Hub so it’s it’s not it’s not a regular part of my day typically but I see the value in it for the lay users or people who have are are using powerbi lot and that’s an important distinction here and Mike I

26:02 important distinction here and Mike I think you mentioned it you’re the pro user so you don’t need to get to it and I think the data Hub initially was a adoption play it was meant the reason it exist maybe originally but I don’t feel like it is now maybe not now but I think the initial idea of having the data Hub was for the discoverability of content and I I think maybe it was actually I would lean more towards the idea of the Persona that they had in mind was more of the lay person so to speak rather than the developer with the data

26:33 rather than the developer with the data I agree I agree we tried pushing that initially with the project but it never caught on and so I think we’re all I can we see this regardless of that Persona we’re still seeing that lack of really use so one thing that Nico is is talking about here what I actually want to point was my next Point here is if you’re not using domains in your company I highly recommend and you look at using domains in your organization the reason being is the one L catalog

27:05 the reason being is the one L catalog lets you filter all of the content by the workspaces that are participating in a particular domain so this makes a lot of sense to me so what they’re doing right now is so to your point Tommy it’s more about being discoverable things me as an administrator and I’m more of a leader inside the the power organizations that I deal with it’s much better to find groups of content hey this is the HR data this is the financial data this is the sales data

27:36 financial data this is the sales data having domains on top of that and rolling all that content together makes it very easy because there’s to be honest even in small organizations there’s too many things to go look at using domains to filter down to the appropriate topics some make lake houses semantic models those things extremely helpful and I really like this new experience using domains so they’re definitely winning with domains at this point I think that’s the way to go when you start looking at this stuff so I don’t know if that solves in

28:07 stuff so I don’t know if that solves in if I’m looking at the one L catalog now there’s four elements on the left hand side all items my items endorsed and favorites and then I can filter based on the domain and tags so let me ask you guys and tags yes tags which so tags is another one that I’m like okay so I have a feature around tags that I’d like to see in in catalog but yeah keep going I like the tags idea and and I guess my question to you guys right off the bat with what they’re doing with one L catalog now with your own estimation

28:38 L catalog now with your own estimation who’s the main Persona here is is it the developer is the person building or is it the person discovering I think well it’s both honestly I think it should be a little bit of both right so it should be from like the administration side there should be some features there that let the administrators of this so for example let me give you one about tags right so you can drop down a tag item and say look I want to go to this domain

29:02 and say look I want to go to this domain call it HR and then in the domain I want to have anything that’s gold certified so you can literally go to The Domain filter down the data to by topic by subject matter area and then add a tech for okay this is gold data that would mean all your lake houses all your semantic models that are pulling from gold would show themselves the challenge of this is a little bit is for me is okay as an admin of this area not every department or every user is going to certify or bring or or do all the T tagging that they need and so for me as

29:33 tagging that they need and so for me as an admin I’d like to be able to just poke around a little bit like let’s make it really for discovery on these things and one of them is I’d like to make sure either mandate tags are being used or have the ability as an admin to easily add tags oh this is a thing that literally The Lakehouse says the name gold in it it should probably have a tag of gold right or whatever that may be right so there should be more ability or easier ability for me to then add tags one of the other things that I think is really interesting when we talk data cataloges it’s very important to have a

30:03 cataloges it’s very important to have a data owner I don’t think there’s enough visibility as to where the data owners are and how do I change a data owner on a particular semantic model or other things as well so I also think from an administrator standpoint I want to be able to say oh Tommy’s the owner of this data set I don’t know who to publish this one but I need to re realign it because Tom’s going to own this one moving forward so I think there’s some ownership pieces that need to be able to be adjusted here and that’s my my main two like at the high level

30:33 my my main two like at the high level for filtering and finding I have a whole bunch of other features here that I think would be useful when we get down to like looking into a semantic model and seeing the list of tables names columns there’s a whole bunch of other stuff here that I think we’re missing we’ve had a previous convers good good no I could see multiple different use cases and personas right to to Mike’s Point admin cleanup management promotion of of certain artifacts or put things on on the path reach out to the owners to

31:04 the path reach out to the owners to see what levels of certification are needed if if at all right like so it’s it’s the de facto one location right from a development standpoint or developer quick search for especially if you you start to build a solution that has different business units owning different data sets rather than go building something hey I’m going to go search for you I’m going to go search for who do we have this type of object know who do we have this type of object out there or tags would

31:34 out there or tags would certainly help as long as there’re standardization across the org but just the quickfinder right because before I’m going to go do any work I want to see what what we have out there that I might not have access to and I I do like that that was the approach that they took it’s not it’s not filtered down to only the things you can access it’s it’s open and you don’t have access but you can least see that the artifact exists and I think that’s one of the like so super valuable on many levels the biggest thing for the

32:05 many levels the biggest thing for the dev 2 is like who the domain owner is so you can go understand what the data set or report is doing but predominantly probably the data set techish business users may have like the same type of need related to like data access so same thing they could find where where can I go plug into this data or are there reports and then obviously the General users just looking for reports so across the board I think in in general the first

32:35 in in general the first deployment of of the options that we have meet all those needs and I I think that’s mainly predicated on the the two types of filters they’ve chosen which is domain and tags plus showing everything is a a good first step for everybody to see what’s available and and the the types of artifacts that they should be engaging in I have a hot take here on I think what the primary person or Persona here

33:06 what the primary person or Persona here is and I think it’s self-service in the world of Microsoft fabric because if you actually look at the user story of any particular type of data in in the one L catalog if I look at semantic model or I look at lak housee I can do that explor feature where the data Hub might have had the vision of like a VP going in and trying to look at content to me with the ability to filter with domains with tags well if my tags may be based on the operation or the action trying to be

33:36 operation or the action trying to be taken and the fact that it’s we’re looking at warehouses we’re looking at databases we’re looking at at Lake or semantic models the reports may be very low on the list on what the vision here for the one light catalog is we’re really dealing with Builders and there’s obviously the element of the admin admin here or the we’ll call the pro user like us where I I can look at my refresh schedule I can do a lot of other Administration things but really this looks like I just need to connect

34:07 this looks like I just need to connect to our marketing Lakehouse and I’m going to do some extra analytics on it or we need to build a report on this to me I see a lot of self-service but not the self-service that we lived in powerbi where it was report building because again there’s so many other things I can build now where I don’t know if it’s just peer develop but okay so I don’t understand that because I can see all artifacts the only the only limitation I do see right now

34:37 the only limitation I do see right now is the fact I can’t see or gaps because that’s how like that’s how things are shared predominantly today but I I guess I don’t understand your your comment I wouldn’t I wouldn’t limit it to just people building because reports are part of this so if but you made a good point though about the or apps so this is what I’m trying to say yeah but at the same but at the same time part of like fa Fabric and everything is predicated on data mesh

35:08 everything is predicated on data mesh right which is essentially the the theories behind having owners of different areas but a fundamental key part of any system whether it’s centralized or you’re separating out the ownership into different areas Etc is discoverability and that’s every level that the lay user who has no no idea what they’re doing should be able to pop in here and go I’m looking for a sales report oh look that like these artifacts are owned by the sales domain this is the owner of this report or this thing I

35:39 the owner of this report or this thing I should go talk to them right it guides people into the Avenues by W which are very challenging by the way right now without this Central repository of where the heck do I go to find something that I need be it a report be it a data set be it the owner and this is that portal I think that they’re leveraging for all of that it’s not just for the for the doers Al so let me push back a bit so

36:09 doers Al so let me push back a bit so why didn’t first two questions first why didn’t this work with the data Hub which was really focused just around reports and we know it was just reports but do anyone of your organization or Mike did you have any clients that that relied from a consumer point of view that used it I think the org apps the point that you made made I I think we’re in the very beginning stage of this I think you’re pushing on something that doesn’t exist honestly I don’t think I don’t it’s not there yet I’m I’m more of the point of it needs to be there needs to be a

36:40 it needs to be there needs to be a central location of stuff like this right no one’s going to go digging through every single the AL the alternative to this is go through one workspace at a time looking for the item you need or use the search thing at the top of the screen which has mixed results in my opinion right so you need something to to have like again there’s so much content we need controls here to filter down by what we’re looking for this is the fastest thing I’ve seen to say go to the topic area of sales and

37:12 say go to the topic area of sales and marketing here’s the list of the workspaces that are in this domain so filter by domain first that should be the first filter you start using here’s the list of all the workspaces start click into the workspaces boom boom boom boom boom and you see all the artifacts just change for you right there that’s the fastest way I’m going to see you’re going to discover things this makes a lot of sense to me my challenge here though is like not not to that point Tommy it’s too new for people to really know how to use it yet it it this is one more of these things of it’s another data point no one’s using it right now no it it’s just not out no one

37:43 right now no it it’s just not out no one even knows it exists it just got announced like a month ago so I think people are still trying to figure out what it’s there for and how to best utilize it because I think the organization the organizing patterns haven’t quite evolved yet from the community people haven’t spoken about it enough to really make it oh yeah this is an effective approach of using this and that’s absolutely fair point I think the last thing I’ll say from a consumer Builder point of view is I think we’re realizing and I think Microsoft finally realized that consumers do need to be head hand fed so to speak and that’s why

38:14 head hand fed so to speak and that’s why we have the org apps that’s a really good point though that we have the organizational apps because you could have this whole catalog of discoverability of content to consume but we already had that for consumers with the data hub for reports this is just adding in everything which a consumer if they were overwhelmed with the data Hub this is gonna this is now just another layer on that the org apps and even just a workspace app is really

38:42 and even just a workspace app is really the way a consumer is going to consume it’s the it’s the way that they’re going to find content so this is to me I’m seeing all of this build for Builders of some any some any capacity oh I think it’s I think it’s for consumers and I don’t I don’t agree with your statement of it I think they’re over overwhelmed I don’t think they’re overwhelmed by this I think they don’t even know it exists honestly I think it’s just I think it’s purely a discoverability thing at this point I think it’s going to become a pattern and there more as the community picks up on the value of it my challenge is my challenge is less around getting and

39:13 challenge is less around getting and using the tool my challenge is what data is actually being presented so if you read the Microsoft documentation and you go down and you play with one L catalog dog gonet you’re pulling in all the semantic models you have the name of the tables you have the name of the columns but it literally says it’s a type of table or column or measure I need to know what data type is it I need to know what’s the description of that column I need like so there’s things in the semantic model that exist that should just be present in these tables

39:44 should just be present in these tables there should be if I if I have the ability to see one of these columns inside the data catalog there should be a preview button I should be able to preview the data that is in that column or the review of the data in that table if I have access to it so to me there’s there’s a whole bunch of things missing here that is like if you have access to it if I have access and so there should be certain fields that are available like hey you don’t have access request access here that’s what this is for right here’s all the content that’s out there these are things that you

40:14 out there these are things that you should be able to see or not able to see based on like your permissions so there should be something flags on things maybe they say allow this to be discoverable there’s already stuff like this that can is being controlled other places maybe we we do something where an entire domain can be discoverable or not do you want this domain to be hidden from the data catalog or the one L catalog yeah I I I maybe want that so I think there’s ideas here like by default everything is shared out there in the world and you get descriptions of columns you get descriptions of models

40:45 columns you get descriptions of models like all these rich things that we should be doing in the semantic model those things need to be brought forward and put here once those are available now I think we have a data catalog that people are really wanting to use that’s I think what we’re missing at this point so as an admin of the data catalog areas I’d like to be able to go see all the data sets that are there and see which columns don’t have descriptions and then be able to edit them right there in line inside the data catalog because I’m an owner of that thing if I’m the owner I should be able

41:15 thing if I’m the owner I should be able to modify that semantic model right there and add the details that I need right inside the data catalog and now everyone has access to the definition of the tables The Columns the models that’s what I think would be very power powerful but now maybe I’m stepping too much on the toes of purview I don’t know I don’t think people are going to use it without that if if the intent of this per the Microsoft article is find explore and use you’re you’re talking about the and use part yeah use

41:45 talking about the and use part yeah use it in line or have a have a a tighter knit kind knit data exists right my my my the of data exists right my my my the heartburn I’m having is these things are already in the semantic model I’m already spending time to build them why not just you’re you’re showing me that the columns and the tables exist in these semantic models just expose them please just pull them out or let me pick what columns I want to see and give me a couple other data points that me as the admin is going to allow you to see on this data

42:16 this data set I don’t know if i’ hang my hat there but I get it I I see that but I where I’m hung up Mike is the from a security and also the type of person here who’s that for is that for you Mike the admin is that for the VP to look at a data set to look at that data right to look at that metadata so to speak who’s that feature feature for I think it’s for anyone who’s interested so if you’re talking about discoverability me as the admin for sure

42:47 discoverability me as the admin for sure admins for sure need this stuff but you should also have like so think about when you’re in power query you have these like General column statistics yeah it just tells you how many blanks how many fulls how many nulls like there’s potentially even a health score of this data set right do like there’s a whole bunch of other things that go along with this that are relevant to this story and again what I’m comparing this against is I’m comparing this against purview purview does not do a good does a better job of saying look I found your data set here’s the columns users can add descriptions

43:18 the columns users can add descriptions names or other things to those columns and comment or whatever who’s a data owner like if you I can’t be so naive to think that a single lake house is owned by one individual every table in that lake house could be owned by different individuals so there needs to be the ability of saying where does this sta come from in this lake house here’s all the tables we have great who’s the owner of every table where’s where’s that Steward of that information so if I’m not mistaken though you have to have build permission to do that if I want to do the explore feature on a semantic

43:49 do the explore feature on a semantic model I’m not I can’t just be a consumer sure then then permissions are fine then then it’s already worked out so if if you’re saying build permission you can’t even see it without build permission then you’re telling me the the resolution of like who can discover other data whether or not it’s shared or not it’s already resolved the permissions are already built into the the one L catalog which I would expect to be that’s expected I would have nothing I wouldn’t want to expect to be sharing data in there

44:19 expect to be sharing data in there that’s not supposed to be shown to the entire entire organization yeah and I think get an interesting point with the security roles and we you have the the member and build permissions now but you’re there’s almost like this second layer here of the metadata of being able to see that but not be able to do with anything with it but you did mention Something Mike where the feature that’s coming out with the oneel catalog that’s not available today everything we’ve talked about today is available you can start using it right now just fire up your computer

44:49 it right now just fire up your computer if it’s not up but the other feature that they really pushed was this idea of governance and again this is a different tab it’s in the one L catalog but I basically see a dashboard and actionable things I can do if you’ve ever built a resource in Azure what I’m talking about hey welcome to your database here are things you probably want to do you want to put security around this and the same way with fabric they’re gonna have hey here 30% of your workspaces don’t have any tags so start acting on those right now so there’s

45:19 acting on those right now so there’s going to be all these actionable dashboard things I don’t know if it’s customizable like if what your thresholds are for your business business but I want to ask you guys because as I’ve thought about this and when I saw this the first time my first thought seeing that was oh wow this would be a perfect part integration of a weekly call or your your calls with your data governance or your center of excellence to get on it that would be something that shares on a screen that has action items after that everyone can

45:51 has action items after that everyone can see and that we actually work on together no different than looking at your jurob board or your where your your is so I will ask you guys from what you have seen and heard and what’s been at least discussed by Microsoft do you see this as I don’t want to use Pinnacle but very close to what governance should be and the platform to do so no I don’t think they’re even close to what governance should be looking like in this in that SP even even having

46:21 like in this in that SP even even having a couple minor action items there’s I a couple minor action items there’s when you talk about governance mean when you talk about governance people pay really big dollars for things like Informatica and other large data cataloging tools that scan things perview is out there as well it’s trying to compete with those tools there’s a lot of money being made around the organization and discoverability of your data so I think this is going to be anecdotally it’ll be helpful I think it will add some value it’ll add more to the story this is part of the story that

46:51 the story this is part of the story that I think powerbi is missing today currently the discoverability being able to manage that and govern that yes it’s there but ultimately you’re going to need more data quality metrics like this column can’t be blank this column has sensitive data in it this one like there’s other things here that I think that are going to push you more towards that purview spot side of things that’s the tool that we really need what we’re getting now I think in in one L catalog is like a lighter weight version of that right something that’s is going to help people find some content and I was in a

47:22 people find some content and I was in a training recently and the question was how can we make sure that people will understand what data is out there so that they’re not duplicating the same data engineering or the same table they want to have domain owners make data and information and distribute that across the organization that’s what they want and like I agree with them that’s that is the right approach does this tool do it 100% right now no so I don’t think it’s gonna be pivotal I think of all of the things that Encompass data

47:52 the things that Encompass data governance right I is this is this a a meeting Tommy that you you can I think manage the metadata of the discoverable items that that you’re trying to like manage within the organization because if this is a hey we’re going to review what’s in the data catalog and look for things that are not tagged appropriately or like these are our bullets for metadata management of these artifacts yeah sure but that’s like one

48:23 artifacts yeah sure but that’s like one of 15 different things that you’re doing in data governance right it’s not it’s not the de facto like replacement for everything that you’re going to be doing from a government strategy perspective and and I I’m totally fair and I may play devil’s advocate here but I’m I’m a few a few thoughts here Mike P you mentioned like the the idea to see the empty columns and it’s like you’re talking about we’re talking about micro governance of individual semantic model issues compared to macro governance around things that are available in the fabric space the other benefit here on why wouldn’t I integrate

48:55 benefit here on why wouldn’t I integrate this part of the meeting is to use the Christmas spirit here this once it’s available out of the box will have these metrics for me this is like opening up and it’s my my toy T-Rex that just works compared to in order to get to the point you’re saying you’re talking about setting up and like oh your drone’s available but guess what it’s going to take two hours to set up there’s a lot more customization and this is all being said where I completely agree with you the government side that you’re talking about when

49:25 side that you’re talking about when build to see blank data missing data valid data both on like types of columns and also like what are our attributes is absolute necessity but we don’t have it and we don’t have a a common tool or Universal tool on the other side of the coin we’re gonna have something right out of the box where we can really accomplish a lot it’s not just something going to be nice to look at this is where we can actually accomplish a lot at least organize our content we’ve talked about the gunk that it may not have everything but again I can open up

49:56 have everything but again I can open up and start playing my T-rex right away where everything else we’re talking about that is going that is a team working on that for a while and continually having to when when I hear you talking about those things I hear or I thought that popped into my mind is this is a this is a this is a technology solution for a poor process solution in in some ways right so the technology is trying to solve something that has poor process

50:26 solve something that has poor process around it all this is fine and dandy but at the end of the day what process does your organization have that you’re going to use to help you govern all the proliferation of things you’re going to create that’s where this is going and to your point Tommy around the actual dashboard yeah let me let me have a dashboard let me look at one place where I can see all the data sets that have failed refresh in a particular domain and and what’s the list what’s that list look like right which dat which data sets have stopped refreshing because

50:56 sets have stopped refreshing because they’ve been failing so much right there’s just some simple questions I think we’re trying to ask of our data sets and without a lot of extra effort and work trying to extract all the data out for ourselves we can’t really answer those questions so this is the technology showing up to say look you have a handful of data sets that have stopped refreshing because the data set source is broken it will not work and if it’s paused and if it’s been paus for a week is anyone even really using it is it even adding value and so I think this

51:26 it even adding value and so I think this is where you’re to your point Tommy that the statistics the metrics hey you’ve got workspaces that don’t have tags that that observation is only relevant if you have a policy around things need to have tags on them right and what tags are they that comes from the central bi space so I I think this is a a tool that we’ll have to incorporate it’ll have to be we have to see how the technology works and then we have to modify our policies and our

51:56 have to modify our policies and our companies to figure out how to best utilize this tool to our advantage it’s a double part here on this one we’re trying to change data culture with this tool so that’s that I think it’s going to be some more work here for us to figure out what the pattern should be does that make sense I th% because I think you made a perfect point that dashboard is going to allow us to create those processes because we can actually see that we can actually say we’re going we’re aiming for something and I can actually say numer measure that which we don’t have

52:26 numer measure that which we don’t have today and if you do you’re already doing the apis and the customization You’ got to do apis yes exactly most companies don’t have this readily available no and it’s not easy to turn on and Microsoft reporting gives you a good stream of information but it doesn’t give you like canned reports answering specific questions it’s just here’s the data so you’ve got to figure out how to make it work on your own I agree with that well I think we’re about ready to wrap we’re at almost at time here any final thoughts on the one Lake catalog maybe Tommy give us a final

52:56 Tommy give us a final thought that you have here around where you think this is going to land is this is this going to become your game changer or is this just going to be a nice to have feature and I would love to hear from you guys too if your plan of attack here and as I’ve been looking at the one L catalog playing with it and seeing the the announcements coming from it this is something where I would def I’m going to be focusing with client as they continue to adopt fabric putting it part of a process it may not be the central location it may not be the primary Landing spot but again you I

53:29 primary Landing spot but again you I think the the best point I’ve heard today is that idea that the technolog is going to be able to adapt or change the process or create processes that we didn’t have and what we can see right now with the oneel catalog is absolutely something where I will hang my hat on and I want to see other organizations hang their hat on around creating processes that will help actionable and data quality at least from a macro point of view for the organization the adoption side we can get into and we’ll

53:59 adoption side we can get into and we’ll I I’m continue going to monitor but at least for teams for teams trying to improve their quality improve the organization of all the content that’s in in fabric this is going to help create those those processes SE any final thoughts here for you on as we wrap here yeah I you on as we wrap here yeah new feature new tool right so it’s mean new feature new tool right so it’s going to take some time to for people and organizations to figure out what aspects are useful I it appears to do a pretty good job at at

54:29 appears to do a pretty good job at at allowing the discoverability management potentially in some ways and monitoring of data and and bringing that all together to a main area I think offers up new opportunities to streamline some of those work efforts you and and it certainly does what it needs to from a lay person perspective which is give a centralized place for you to to talk to people about how they discover things that are available to them in organization and I think streamlines a lot of that communication so it’s a good it’s a good

55:00 communication so it’s a good it’s a good step and it’s a great thing to have something like this that overarching looks at the whole environment and brings it together in a a place where multiple different personas can can get some value out of it I feel like when I look at this I go this is an interesting tool it feels like it’s trying to fill a gap that hey with everything else that Microsoft has been building all the pipelines the data flows flows Microsoft has taken the best of azure and brought it directly into the fabric

55:30 and brought it directly into the fabric in the powerbi ecosystem it feels to me like perview is still on the edges on the outside of fabric and it feels to me like this is a first foray or introduction to trying to bring some of the perview features directly inside fabric it feels like the integration between Fabric and purview was hasn’t been as tight as we wanted and people are not adopting it the story is not resonating so Microsoft is is taking the effort this is anytime we see new features like this show up this is a Monumental effort by Microsoft they’ve

56:00 Monumental effort by Microsoft they’ve got to find funding they got to find people they got to go build features they got to put a PM on it like this is an effort so for them to what I’m looking at this rebuild purview to some degree speaks to the fact I think they’re committed to helping people discover things and they’re going to bring those aspects directly in the tool is it my go-to tool right now yeah probably not I it needs some more information in it so I think with some refinement some adding some metadata more metadata that was already there into it I think it actually could be a really compelling Tool but I think it needs a little bit more work on the

56:31 needs a little bit more work on the helping you discover real columns that you care about about the data sets so I think that’ll really help I do love the fact that they’ve integrated domains and filtering by domains in this I think that that is a pure underutilized feature at this point all right that being said thank you all very much for jumping to the podcast we appreciate it U are hanging on to us hopefully some of you have been around for the full three years we’ve been out there as we talked earlier in the podcast but we just want to say thank you we appreciate your time we know it’s valuable you could be doing other things like doing a

57:01 doing other things like doing a crossword puzzle or doing a sodoku or going for a walk or maybe even talking to your family you could be doing other important things that are not the podcast so we do appreciate you spending a lot of the ears with us listening to us our only ask though is if this is something we just do for fun please share with somebody else let somebody else know that you’ve enjoyed the podcast and that they might get some value from this one so hopefully you’ve unpacked some things around data catalog Tommy where else can you find the podcast you can find us on Apple and Spotify wherever your podcast put us in

57:31 Spotify wherever your podcast put us in your catalog make sure to subscribe and leave a rating it helps out a ton share with the friends since we do this for free do you have a question idea or a topic that you want us to talk about for a future episode head over to powerbi. com we appreciate you and we’ll see you next time time [Music]