Medallion Architecture in Fabric – Ep. 367

In this episode, Mike and the team discuss Medallion Architecture in Fabric - Ep.367 - Power BI tips—what it is, why it matters, and how you can apply it in your Power BI work.

News & Announcements

• Native Execution Engine available at no additional cost! | Microsoft Fabric Blog | Microsoft Fabric — We’re thrilled to announce that the Native Execution Engine is now available at no additional cost, unlocking next-level performance and efficiency for your workloads. What’s New?  The Native Execution Engine…

• x.com — We’ve detected that JavaScript is disabled in this browser. Please enable JavaScript or switch to a supported browser to continue using x.com. You can see a list of supported browsers in our Help Center.

• Medallion Architecture — The databricks medallion architecture is not really an architecture, but more [[Data Modeling#Different Levels|Approach or Pattern]]Â with three data stages: bronze, silver, and gold.

• PowerBI.tips Podcast — Subscribe and listen to the Explicit Measures podcast episodes and related content.

• Power BI Theme Generator — Power BI.tips - The worlds best theme generator for Power BI reports. Increase your speed to develop stunning reports using this free theme generator. Themes are essential for any report developer’s tool belt. Visit…

Main Discussion

The Medallion architecture (Bronze/Silver/Gold) is a proven pattern for organizing lakehouse work—but Fabric changes some of the “how” while keeping the “why.” The team discusses what the layers mean, where the pattern still fits, and how to avoid turning it into ceremony that slows delivery.

Key points:

• Bronze/Silver/Gold is about intent: raw landing → cleaned/conformed → curated/consumable outputs.
• Fabric can implement the pattern with Lakehouses, notebooks, pipelines, and semantic models—but you still need standards.
• Don’t over-layer: the right number of layers depends on data quality, reuse, and governance needs.
• The “Gold” layer is often about serving consumption (semantic model/data product), not just another table.
• The architecture should make data easier to trust and reuse; if it adds friction without value, simplify.

Looking Forward

Try one small experiment this week related to Medallion Architecture in Fabric - Ep.367 - Power BI tips—then iterate based on what you learn. Small, repeatable improvements compound fast.

Episode Transcript

0:22 [Music] out out [Music] good morning everyone and welcome to the explicit measures podcast with Tommy Seth and Mike good morning good morning and happy Tuesday [Music] [Music] gentlemen it’s a Tuesday for sure it is it is is it is it a happy Tuesday though it’s been a busy mon I go if I go that high pitch Tommy it’s got

0:53 if I go that high pitch Tommy it’s got to be a happy Tuesday it’s true but yes I think as we enter the final stretches of the year I think all of us are a little tired it gets busy I’m tired of all this political news that’s what I’m tired of get the news over with in the US got talking about that we’re not even talking about it I’m just saying I’m done with it I just wanted to hear nothing less of it too much huge I’m not feeling a happy Tuesday guys no yeah Tommy’s having rough days these days

1:24 Tommy’s having rough days these days Tommy do we need do we need to open the therapy session like well let’s go through what top what we’re talking about today and then I I can main topic and then Toby’s gonna have a therapy session all right let’s go so main main topic for today is we’re going to talk about The Medallion architecture let’s go a little bit over what it is for those of you who don’t understand it or haven’t heard about it before and then let’s just talk about its viability and is it relevant inside the context of fabric now that we have fabric does The Medallion architecture still hold true should we

1:55 architecture still hold true should we still be using it are there other patterns we should be looking at so we’ll just discuss where we think things land and we’ll go from there that’s our main topic but before we get there Tommy go ahead give us some of your news announcements this morning I’m I’m not I’m not happy guys and it’s been a tough couple of few days our last episode we talked about the joy of baseball and the world series starting and for some of you may know this I happen to be a rather fanatic towards the New York anke he always have been yep unfortunately

2:26 he always have been yep unfortunately always will be you’re GNA say the Mets no no no all every single background every single computer and the the Yankee Pinstripes there and the every baseball back there is a Yankee thing you can see you’re invested it’s invested I’m invested and right now I the hope I had last Thursday be the World Series was about to start we’ve lost the last three and not just lost them but game one we were winning up until

2:57 but game one we were winning up until with two outs to go and the bottom of the the the the 9th and one of a likable guy in the Dodgers he’s actually a likable guy he had a grand slam to walk off the game guys I don’t know if you’ve ever had someone break up with you but if you someone break up with you but if that night when someone either said know that night when someone either said no yeah and you wake up go did that happen was that real oh shoot it was that was me Friday night Saturday morning let me put it into perspective our plumbing broke Friday night no water

3:27 our plumbing broke Friday night no water we couldn’t take showers we I had to take my son to Wendy’s to go to the bathroom and by far the worst thing of this was the dang Yankees losing three games and and I’m not I’m not even I’m not even acting here I’ve questioned things I’ve been I’ve had to apologize to my wife because I’m like I didn’t think this was gon to affect me I asked I asked someone who’s very close to me I’m like how much should one care about sports I’ve been questioning things and why do I care about laundry I’m all sports is is I’m rooting for my

3:57 I’m all sports is is I’m rooting for my laundry to beat your laundry there’s nothing else it doesn’t affect my kids going to college or if I get a new client or not it’s a dumb thing sports are dumb and I don’t know why I care really don’t I a great idea it’s a great idea I couldn’t tell you who’s winning you’re losing right now just doesn’t matter to me I’ve been waking up just in a foul mood just like dang like randomly in the day just like slapping my like just head going just upset and

4:27 my like just head going just upset and sad and angry so I I have the same feeling the fact that I can’t use schemas in a semantic model I don’t know as much as I can relate I don’t think it Compares in this one I don’t know I’m pretty distraught about this one I’m also I’m also having Tommy you’re say take that emotion that you have that and that feeling there okay now think about data flows without incremental refresh think about data flows Gen 2 not being able to go the care about this stuff too I care

4:58 go the care about this stuff too I care too much that’s what too much that’s exactly right we care too much we want it to be a great the example for you Mike imagine you have been building from the basement up a three story Lego Masterpiece and you’re like all I have to do is this last story put the hand on it’s GNA be good and then every night it just begins to crumble a little and crumble and there’s nothing you can do nothing I build Legos crumbles but that’s okay that that’s an amateur Builder right there but again you actually have control over that I have no control and I’m helpless and

5:29 have no control and I’m helpless and Seth I know you follow the Packers but this is dumb like why do we why is sports such a big thing my wife says entertainment it’s not I’m not entertained by any of this I can also tell that your wife is very much annoyed with your behavior over the last couple days just because of yeah she because you you have been just stomping around the house for no no apparent reason I had to apologize to the kids because I was just I my son I told him don’t throw it back to me he threw it back and I chucked it and I said go get it and I’m like what am I doing just I I

6:00 like what am I doing just I I just perspective perspective I appreciate you guys I appreci I love power bi I just like why are the things I care like why are the things I like I just care too deeply about that’s because you’re Italian that’s what it is I know the Italian Guys sports are dumb it’s just it’s a dumb thing take a breath Tommy got it you can handle this slowly draw in for S seconds hold and hold and release it’s like or the old Seinfeld

6:31 release it’s like or the old Seinfeld episode Serenity Now Serenity Now yeah that’s exactly what awesome well hang Tommy I’m sorry to hear that your team’s not doing well and it’s causing pain but these two thing these these things too shall pass well if you’re listening I have a lot more time soon for powerbi and Microsoft fabric so there we go I stay tuned lot more nothing to do at night lot more training YouTube sessions coming because Tommy’s gonna have a whole bunch of free time fill it with powerbi awesome a

7:01 time fill it with powerbi awesome a couple other news articles here I just want to point out something I just did a recent YouTube video with esta cot with the new native execution engine and I think announced yesterday on the Microsoft blog was a article talking exactly about the native execution engine is now available you can go try it now in public preview and the best part is no additional cost other tools data bricks out there there is CA is charging you extra money for

7:31 is CA is charging you extra money for this thing called Photon it costs you more to run it Microsoft is giving you a speedier engine and and you look at some of the statistics I think they have some graphs in here about how much faster it is it’s substantially faster than everything else and you can use it for no additional cost so you get stuff done faster doesn’t cost you more and it seems amazing so I really love it super good article go check it out go read the documentation it’s very easy to turn on with one line of code you can do a bunch

8:02 with one line of code you can do a bunch of really cool things so definitely go check out the the blog article about that one I’ll put that in the chat window highly recommended it is awesome so I’ll definitely drop that link in there for you so you can go check that out I have tested that and it is a significant difference with speed it’s very impressive to your point you have you to play with a little bit all right good I was hey actually it’s like we’re not we’re not joking around here no they’re not I’m just and it’s NE because Microsoft is is still using

8:32 NE because Microsoft is is still using the normal Spark engine but it’s also built on a number of other community based tools gluten being one of them but so there’s other it’s not like Microsoft’s building this a proprietary thing but there’s a veilo and I think gluten are the other two I’m sorry there’s actually a tool called gluten I believe there’s it’s an open source library that they’re using I think it’s I thought it was gluten for for somebody who is gluten hates the word gluten yes that is hilarious I think I think it’s yeah it’s Apache gluten and then it’s

9:04 yeah it’s Apache gluten and then it’s veilo so those the two projects that they’re working on so gluten and I can use and consume gluten in my life it’s just in technology it’s just it’s

9:13 it’s just in technology it’s just it’s the Apache gluten you’re not allergic to just choose all the all the allergic Foods aache peanut that’s great so anyways very very cool article definitely go check out the documentation it’s in the chat window as well so you definitely want to read about this again if you use spark if you’re using notebooks at all this is who are you definitely check it out see if your workloads run on it and if you’re not doing anything super crazy complex with your Spark engine it will just work you can just use your existing code today and it will just run faster I code today and it will just run faster how great is that this is like a mean how great is that this is like a hidden secret that we’re we’re jumping

9:44 hidden secret that we’re we’re jumping into so anyways very very very cool items here and extremely pleased to have that part of our our library now okay one other random thing I’m going to just point out one thing here just it feels like something has happened in powerbi desktop I don’t know it just recently occurred to me I don’t know if you guys have experienced this but I was in parb desktop I was making some changes to measures and trying to calculate different things on a visual and for whatever reason I could not get the visual to render things it would not update itself for whatever

10:15 would not update itself for whatever reason I was having issues with like changing to measure and then I was bringing data in and the data was just returning blank values for whatever reason only when I went to a different page and came back the visual just kind page and came back the visual just like magically refreshed itself and of like magically refreshed itself and started working I don’t know what I did but it feels like I’m getting some very weird bug related things that are just in styling or changing the properties of a visual and they’re not quite are you dark

10:46 not quite are you dark mode that’s a good question I’m not in dark mode I was in there’s okay so there’s default there’s dark mode and there’s light mode so I was in light mode okay so which I think is not standard right this is this is the okay it’s funny and you were in dark mode when this maybe occurred right I just based on our conversation last week dark mode was one of those things I just hadn’t gotten around enabling and I was like oh I’m going to go make after our features conversation I was like I’m going to go check to make sure I have them all on or whatever yeah and I’m

11:16 them all on or whatever yeah and I’m like I’ll apply dark mode ever since then i’ I’ve got I I haven’t locked it a specific set right of like wow that weirdness right there like you described I thought it was like configuration of colors but I also now that you’re saying this could have been like measures as well in the table stuff just wasn’t working when I like pasted created a new visual the other thing that 100% was not was

11:46 was filters filters if I if I selected a different filter to not basic dropdown all of the menu dropdowns for like conditions right is is blank is not blank all that was just the white box just a white box and I I was thinking like well maybe it’s just my graphics card on my on my machine you’ve been programmed when you think it’s you or your machine that’s like the thing that’s brr that’s it’s interesting you experience I wonder if I go back to default if all my

12:16 if I go back to default if all my problems go away I custom theme are you using a custom theme because I I’ve had a very similar somewhat of a similar issue where you’re probably using the theme generator from Power tips Tom that’s probably broken this is pre theme everything worked but don’t you dare copy and pasted visual like the power okay interesting interesting okay so I so I also see in the chat window Eric is also saying that he’s also experienced some of this weird something’s not quite working in dark mode and as well so that he’s not using dark mode so it’s

12:46 that he’s not using dark mode so it’s just a maybe but not using dark mode so maybe it’s not that but I’m not sure Cent version yes okay anyway sure there’s some wonkiness going on there is okay so that being said though just be aware there’s some weird things going on we’ll keep our ears to the ground see if we can hear anything if anyone in the community is having other weird issues with that and I’m not sure there’s like default mode which I think is like how the coloring is supposed to work in powerbi and then I think there’s light mode and then there’s dark mode so maybe I was on light mode when I saw the

13:17 maybe I was on light mode when I saw the issue I’m not sure if that’s any effect there or or what else I’ll keep my ears to the ground I do like being in dark mode I’ve been building a lot in dark mode recently so I’m going to continue playing around with dark mode because I think it’s great the only the only thing I have seen come out of the team since dark mode is all of a sudden like a bunch of dark themes and I’m like no no like there is there is no I called this yeah because you have all dark right around like I want to make everything dark yes and and it’s

13:48 to make everything dark yes and and it’s kind it’s it’s subtly changing how you build the report itself because the report itself is is like you have all this dark themed stuff around you the white on the dark is going to make you naturally try and push towards a darker mode so I think using dark mode is going to push people to build darker themed things at some point I I predict there’s gonna be a toggle soon maybe yeah maybe we need an automatic every report has to come out in light and dark mode yeah yeah I want to do my power

14:20 mode yeah yeah I want to do my power query in dark I want to do my modeling in dark I want to do my Dax in dark but once we get to the report I want that to be light oh now you’re Ed some really require so the the the report portion of the report or the report portion should be in light mode or well actually it’s also silly that you have to switch between light and dark modes inside the settings and the options it’s not bad it’s but it would I’m thinking to the point to your point there Tommy it should probably be a bit more visible because you’re if you’re in the report side it makes sense

14:50 the report side it makes sense to make it more light mode I’m n picking here because I have to click on options and setting twice to get to options and settings anymore no are you sure and I still get the you do if you go through the file menu have you have you found the gear icon in the bottom right hand yeah here’s here’s a good thing let’s put all icons wherever we want all around everything that is in power so like there’s icons in the left nav there’s there’s a there’s a right nav there’s the top header bar oh and by the way the bottom right hand corner has the

15:20 way the bottom right hand corner has the gear and now that’s just for all the settings let’s just let’s just hide everything wherever we can and yet I don’t have my quick access to bar and we still deleted quick access come on come on the best article I’ve read to date on powerbi was by Mike Carlo before I knew Mike Carlo about setting up your custom access dude I’m telling you that was control one control two Center and Aline all the visuals that was like that was hot gosh bring it back well bring it back you’re never going to get it so not a problem they’ve said that no

15:52 so not a problem they’ve said that no one uses that feature apparently was only it was only used by one one user and it was used by me two users me and Tommy were the only ones using that quick access toolbar in that fashion it’s not I don’t think it’s coming back ever all right that’s okay we’ll fix we’ll fix all these little Edge case things with our theme generator our wire framing we we’ll do it in wire framing we make it we’ll make it appropriate over there speaking of which I’ll also point out too I just recently released two new wireframe designs which I’ll try and go get the links for them and put them also in the chat window I just found some

16:22 chat window I just found some interesting people building good-looking reports on the internet copied their designs threw down some of our own designs and added some style to them dropped in a couple more visuals so we’ve got one called tabs and one called teacher so there’s actually two new themes out there we are quickly becoming I think the largest collection of styled template reports inside the theme generator and I think we need to Rebrand a bit we need to be calling ourselves like the template or generator we’re doing more than just theming these days

16:52 doing more than just theming these days so anyways I just want to call those out I’ll put the links in the chat window as well if you want to go check out those themes or templates and go download them they’re out and available okay we’ve been doing a lot of intro this is a long intro for today I feel like but it’s been fun it’s been good talking let’s jump into our our main topic today so main topic today let’s talk about The Medallion architecture where did this come from and should we be building this thing inside fabric what does it look like in the fabric world of things Tommy give us a little bit of introduction around what is The Medallion architecture just let’s

17:22 Medallion architecture just let’s explain what it is and then we can go into a little bit how do we integrate it with or I feel like it do an injustice because this is this is your this is you and Seth man for years so as much as I I I would love to explain it I want you to yeah just because honestly this your guys baby all right punt Seth give us give us a quick introduction as bring it bring it around first all right bring it around all right I didn’t want I I don’t want to talk the whole time I can do okay go ahead Seth kick us off so so I

17:52 okay go ahead Seth kick us off so so I think what we’re talking about today and you can clarify for me right is the well bringing Medallion architecture do we bring the same thing into fabric right and I think some of the the points you’re you’re interweaving in here Tommy relate to how had how is The Medallion architecture different than a traditional Warehouse architecture and I I don’t where did where did like The Medallion architecture essentially came from more of the data Lake realm right Biga Delta

18:25 of the data Lake realm right Biga Delta Lake and and I think there there

18:28 Lake and and I think there there going to be Nuance here I don’t I don’t know so Mike pick up because I know you’ve worked in in both specifically but in in the warehouse realm essentially it’s it’s much more from my experience it’s structured right I have staging I have a cleansing I have and and it’s structured because you are working within fully flattened tables right it’s tables of data joins structure like building out the different stages cleaning things building out your facts Dimensions slow

18:58 building out your facts Dimensions slow changing Dimensions like there’s a process and once you set up that process you follow it right whereas in the medal architecture or lakes there you could be ingesting any type or structure of data right so your raw layer is consuming from I think a wide variety of different sources and how especially even in data bricks you create your Delta tables they can be structured in such a way that I only have have 10

19:29 such a way that I only have have 10 columns but there’s nested structures underneath those columns that I then can extrapolate right so for my bronze bronze layer is just capturing incremental history right of those changed objects in my silver I can grab the most recent and I think it’s it’s designed not to do a ton of heavy lifting in terms of data transformation but it’s it’s give me flatten like flatten out what is

20:00 me flatten like flatten out what is going to be the most useful object to the business to then scale and then gold is where you’re applying most of your business logic and and in my experience maybe the Nuance here that I’m just not like speaking directly to is there’s there’s less structure of this it’s more free form to me because it’s not as Locked In by creating objects that have keys and relationships between the tables and then you’re building out on

20:32 then you’re building out on that it’s more free form this is the flow which is also a byproduct for like it being challenging not being able to like navigate or understand how all those objects die together if you’re still in Hive Unity catalog helps that out fabric Different World but ultimately like allows you to create this ecosystem that you can use reporting and analytics on directly have that nugget of business knowledge and then build out further if you needed to

21:02 then build out further if you needed to for your facts and dimensions that’s how I think about about it I’m going to pull back I think that’s exactly everything you’re saying is spoton I’m going to put just maybe a couple of my observations again I came from so I think Seth you came from a much more database Centric world of the space like you you were SQL Server you were DBA I was not that as much I was definitely much more of the Excel Builder business user analyst a things and that’s my approach to this and I came in with more of like the data science approach when I

21:34 like the data science approach when I was getting my masters in data science I had to go spin up my own Hadoop and map ruce and all these other crazy things that like no longer I need them anymore but it gave me an understanding of like how the architecture works so I feel like the the main is me as a mean I feel like the the main is me as a an outsider to the SQL DBA World it feels a lot like it feels very similar but it also felt like a lot of the SQL world and I think SQL is the SQL and data warehousing world has slightly changed in lie of what’s happening with lake houses and The Medallion architecture anyways it’s already shifting but that being said I

22:06 already shifting but that being said I felt like SQL was more of like a row level context you’re storing rows of data you access data at row level information you put indexes on them to make sure you can skip a number of rows you get down to the data you want I felt like SQL felt very row based informational and and then what I felt we had with Medallion it was we’re using this new architecture now SQL and data Waring had this assd transactions that was the main pool for why the data warehouse you can guarantee that you’re not overriding other people’s data the the assd transaction experience

22:38 the the assd transaction experience works well in that data warehousing the challenge was when you went to Big Data there is no structure like this there is no equivalent for asset transactions and then we show up with Delta Lake and the Delta lake is really this format or structure and this I think was born out of the data bricks Community it owns that definition it’s definitely an open source project it’s it’s there but datab brick seemed to be one of the leading companies that was pushing this Delta Lake experience and I think from this we kind experience and I think from this we started saying okay how do we start

23:09 of started saying okay how do we start shaping this data and I think we we have very similar aspects in data warehousing there’s a staging table you land it in the staging table and then you say okay from the staging information do I insert or update those records inside the actual SQL final tables so I feel like in SQL I had a lot of this comparison of like staging final table that felt like a lot of what we were doing and then maybe from final tables you would do some additional transformations to make additional tables inside SQL my opinion

23:39 additional tables inside SQL my opinion here is SQL was always limited by you don’t want a lot of extra data living around the data warhouse we trying to be as efficient as you could so creating conform Dimensions having proper fact tables like Star schema that was an efficient way to use SQL when we went over to the Lakehouse the storage costs were extremely cheap and so how we were thinking about creating tables of data slightly changed because in Delta tables I don’t need to go insert or update a bunch of individual rows I

24:09 update a bunch of individual rows I could just drop a bunch of tabl data or drop a bunch of partitions stop reading them and just write new files it was faster to not delete or update or insert it was actually faster just to like ignore and then write down new records that the Delta table would just update itself to so my in here is I think you hit it spot on the Delta AR Delta Lake architecture the The Medallion architecture is store raw data and bronze D duplicate filter clean those

24:40 bronze D duplicate filter clean those topical pieces of information in silver and then join them or merge them together apply business logic in the gold layer to Output the data and I think some people get hung up they’re like oh I have one table in bronze and one table in silver and one table in Gold that’s not necessarily true you can have many different tables in any of these areas of the lake just semantics it doesn’t really matter what you’re doing with it it’s just mentally how you’re enriching the data and getting it closer to you’re going from an operational system online transactional processing oltp to

25:13 transactional processing oltp to L is that what this I don’t remember I always get the terms fixed up mixed up there but you’re going from something that is transactional in nature to something that is reporting and it’s column or storage so I think that’s a lot of what we’re doing here and there hasn’t been a single project I’ve ever done where there hasn’t been some level of Transformations required to the data period and that’s where I think the Delta Lake shines and I think that’s a context for I think our conversation today and I I firmly believe a conversation we’re going to continue to have this actually sparked from a

25:44 have this actually sparked from a message from someone named Simon spotty where the first sentence is The Medallion architecture gets a lot of hype it’s like okay we know the the that when technology changes a lot of times process changes too data The Medallion architecture actually was created by data bricks y so the the the origin of this was actually a data bricks creation again when the technology made sense and I I personally want have the conversation too where I’m finding more and more where does The

26:15 finding more and more where does The Medallion architecture in its current form make complete sense with fabric or is it going to evolve where the the gold standard is because I think we could all agree we the Baseline the gold standard of how you’re going to do data engineering or data ingestion is really The Medallion architecture in its current form so well more or less I want to I want to separate two things here right okay one is in simplified terms right whether we talk about the

26:47 talk about the original processing of data into a data warehouse or the processing of data now through a medallion architecture like the the simplified version of that is like on the on a high level both of these things are are transforming data to make it usable yeah right and if you even if you follow a process there’s going to be outliers there’s going to be different ways organizations have to plug into different things at the end of the day you’re going to have the same types of

27:18 you’re going to have the same types of like the same types of data in those systems I think what’s significantly different in the two method so like I think that is important to separate because and not muddle into like yeah we’re talking about ETL elt whatever right we’re transforming we’re extracting we’re transforming we’re we’re sourcing data and shaping it and to make it useful for analytics and

27:40 to make it useful for analytics and Reporting I think the significant difference is is when you start to analyze and look at the brass taxs of how or where these methodologies and architectures were implemented they were on completely different structured systems right so it’s it’s the object themselves and the capabilities that I think push into like into this other stream of we’re talking about The Medallion architecture and with that

28:10 Medallion architecture and with that comes these other capabilities that are not present in just a warehouse build right because to Mike’s point we’re in tables we’re doing row row byro activities or actions whereas you may not be bound by those same things in a med architecture because of the systems that you’re applying those two and I think that to me is the the important distinction I I really like that too and one other quick note here you ask a lot of powerbi Pros

28:41 here you ask a lot of powerbi Pros professionals who have really only worked in powerbi and they’re gonna go ETL isn’t that just power query man and and for a lot of a lot of organizations they had a make do if they weren’t didn’t work with the data engineering team it was power query and powerbi desktop not data flow but it was powerbi desktop so and I I’m seeing and again I’m seeing a lot of this because I’m having conversations with a few clients who a year ago they asked me about maybe a year and a half ago about a lake house or can we do that and they’re maturity he was like okay that

29:12 they’re maturity he was like okay that means they’re gonna have to spin up an Azure resource it’s GNA be this process they don’t have that that skill they didn’t the talent there but or do they want to spend the time to learn all that stuff because they got to get going like it’s it’s a we got to move faster but fabric has has made it way easier for us to integrate these things so data ingestions never been easier and so I let me just start we can start there I guess from a question point of point of view your experience with fabric right now and especially your experience with

29:43 now and especially your experience with The Medallion architecture in the current current form do you see that the traditional patterns still make 100% sense H oh 100% I don’t think there’s any in in my mind there’s no question that The Medallion architecture should be used today inside fabric where I think I question things right now are am I able to use all the new features that I need inside fabric to make it easy for me to work with The Medallion architecture the answer right now to me

30:13 architecture the answer right now to me is no right so for so when we think about a medallion architecture we’re thinking about different areas or topics of information inside our Lakehouse we have schemas right so this is something that’s new that’s been released a lake house a lake housee can have a schema the schema potentially could hold the bronze the silver and the gold data inside a single lake house today you can’t get a lake house and again maybe they’re working on the feature I don’t know when it will get here I’m guessing eventually but today a semantic model pointed at a direct Lake direct lake

30:44 pointed at a direct Lake direct lake lake house right if you’re doing direct Lake to those things you can’t pick the schema you want and those tables will not show up in this in the semantic model I just the schemas don’t work with semantic models today currently and so you’re stuck with using a SQL analytics Warehouse or SQL analytics or using the lak house directly to store things in schemas you can use a notebook the notebooks can talk to the schemas can figure out okay I have a schema and with a schema is it’s another naming convention right so you have when you’re accessing data select star from and then

31:15 accessing data select star from and then instead of just having the Lakehouse name and the table name it’s now Lakehouse name schema name silver or bronze and then the table name so it’s like a three-part naming convention this is very similar to what datab bricks does today same concept it’s a three-part naming schema so it knows where to go get those tables from so in my opinion it’s just it we’re a little bit limited we’re still playing catchup in my opinion on what the the downstream tools that consume the de Medallion architecture today if you’re

31:45 Medallion architecture today if you’re going to build a medallion architecture I’m recommending build three separate lake houses one lake house for bronze one Lake housee for silver one lake house for gold and then you move the data into those lak houses in the future hopefully we get schemas and then maybe my recommendation will change to okay well we’re going to have bronze and silver in the same Lakehouse and then we’ll put the gold tables in a separate Lakehouse because I want to again share the Lakehouse item with external users another one here that’s a a weakness in my opinion is one security that was talked about eons ago I don’t

32:15 that was talked about eons ago I don’t know whatever this the enhanced security right it’s not like there isn’t security now there’s going to be additional security down to the table level and maybe even to the column level I don’t really know but if you think about that too if I can have table level security then I don’t need two different lake houses I can have one where all the tables exist and then I give permissions down to the schema down to the tables inside those schemas and it just get works it just happens right so to me it’s more of like they’re still working on the features to make this

32:45 working on the features to make this really smooth and and an experience that’s just easy to do right now we’re just building additional artifacts that we may or may not need in the future is that Mak any sense what I’m saying there like I’m describing so that makes it a perfect sense from a best practice point of view and unlike the New York Yankees I’m going to push back here and because I I think there for the first time try hit home run we’re gonna we’re actually gonna hit the ball here we’re swing at it okay take a swing and hope you don’t miss Tommy let’s go our MVP anyway that’s another thing so

33:16 anyway that’s another thing so everything you said first I want to say I I agree with from a best practice from the ideal standard absolutely but I think what we have with Microsoft Fabric and I think what we’re going to see a lot more frequent than best practice is we actually have a scale now of best practice and on the other side of that scale is quick access because the ability to create the Lakehouse without any near skill needed or the practice needed like right if you wanted a lake house before you had to have the Azure

33:46 house before you had to have the Azure subscription you had to have these you subscription you had to have these bells and whistles already created know bells and whistles already created totally agree with you so now you’re going to tell an organization or they’re going to do it themselves I don’t think it’s going to be resulted coming in but it’s going to be hey we can get this data right now in some form or some capacity and I think that barrier to entry or rather the lack of barrier to entry that whatever the the means the quickest means to an end we’re going to see a lot more of that quick access and

34:18 see a lot more of that quick access and it may not be in the best form or it’s going to be some convoluted way well I I I do think that’s a very good point because whether or not The Medallion architecture was designed to not create the swamp right because that was like when when Lakes were out there right it instantly like there was this whole like well you you can you can create a swamp pretty quickly and what that means is you’re yeah you’re creating a bunch of data but nobody knows how to use it right and and

34:51 nobody knows how to use it right and and you do bring up a good point because I I think The Medallion architecture is the frame work that everybody who’s building in those systems should be trying to align to even if even if you don’t get it completely right like it’s not hard to be like okay when you ingest here’s the historical slice here’s where you would like pick your current if you don’t have a historical here’s where you put your current and then you’re going to apply your business logic and this is that

35:22 your business logic and this is that that gold layer like to this framework is a is really a ging structure and I think to Mike’s Point it works really well when you have a schema because you can label things in in that schema you can label things in in that schema I’m in raw right in the schema you know I’m in raw right in the schema you know I’m in raw right in the schema I’m in silver with the know I’m in silver with the schema but it isn’t it’s not necessary because even before schemas and data bricks whatever you could you could still well you could still do things in a

35:54 well you could still do things in a hacky way right like with proper naming conventions as you you can navigate and understand where or I should say you should at minimum have naming standards to allow teams to understand how to interact with the data and where they are when they are right my I my naming schema should absolutely align to bronze my naming schema should absolutely align to a silver layer my mining schema should absolutely align to a gold standard right whether or not all

36:26 a gold standard right whether or not all your data that comes into this ecosystem flows through each one of those levels maybe it doesn’t it right sometimes it comes into silver you may pull directly from a source a SQL table or something that you’re pulling up into your your realm and it’s it’s going to go on the gold layer right because it’s it’s fully fully vetted certified whatever the case may be like there’s a bunch of nuance but if somebody is trying to find something in the system

36:54 trying to find something in the system that is where that framework is absolutely pivotal because if you don’t have that oh my gosh right the only people that know are those that are either like the developers themselves or the only reason you would know is because somebody’s meticulously keeping up with documentation where a different named object right this is the Rabbit Trail of what you’re looking for and you would look look across those tables and be like I would never have been able to find that on my own and that’s something I reinforce with my team all the time if

37:25 I reinforce with my team all the time if I I one as team you have to build in such a nature that you’re building for everybody right and what that means is all I should have to do is say hey I need I need us to go triage this problem it’s in this table and that’s all they would need because if they would start at the front and be able to innately know that of course the table name is this if I want to Traverse the

37:55 this if I want to Traverse the backwards to the raw layer I don’t need to ask somebody what the table name should be it’s just this is how we’ve set up the structure of everything so it’s discoverable right and I think that’s where you you get both the benefits of a medallion architecture in the form of the data but also in the structure and framework that allows you to understand how this ecosystem is working because you don’t have all of these checks and balances and keys and foreign keys that you would in a warehouse where it’s just it’s all knit

38:26 warehouse where it’s just it’s all knit together and I can look at the table and understand what the relationships of that table are just by looking at the extraneous links between them and and that background you’re coming in where the experts coming in with the creation of this and that’s not where I fore the I guess the change here or I think where I’m coming from if you’re coming in as the consultant or the expert such as yourself internally creating this yeah the ideal is you’re going to do the in fabric of Medallion architecture because

38:57 fabric of Medallion architecture because again it’s the North Star here but I foresee or I predict that for Consultants like myself and Mike for internally for experts like Seth where the change I think in in the model is going to be or the structure is a lot of organizations are not going to adhere to that and they’re going to have much more in fabric a swamp where do you come in then and say no we’re going to change all these lak houses and all these artifacts and fabrics

39:27 artifacts and fabrics and we’re going to change them all to to Medallion architecture to me this is where I think there’s going to be a diversion where it’s going to be too difficult to try to go in where business users not just data Engineers because again we’re not playing in the playground of just data Engineers it’s going to be business users powerbi users and your data scientists all playing fabric creating their own process to me that’s where I think there’s going to be the pivot or The Improv of what type of models and what type of schema is going to be created if

39:59 type of schema is going to be created if anything the skill and of I think as a consultant and the skill of the expert here is not so much in the technical skill anymore or as much as much as one of the things we’ve talked about where this conceptual design where we’re going to have to go in and look at a current structure in fabric with multiple workspaces and all these cross over and overlap and having to conceptually think about how do we actually in a sense either migrate this or create this so business users can use it and that the

40:31 business users can use it and that the data Engineers can use it at the same time so I’m I’m going to also put some I like what you’re saying there Tommy and I think you’re right I think this is also a a data culture perspective on this a little bit as well so when I think about companies that I’ve worked with some companies come in guns of Blazing let’s just go let’s just make stuff let’s make it happen we’ll figure it out as we go like they’re coming in hot they’re going to make what they need to make and they’re going to solve business problems and to your point and I think those organizations it will be a bit more disorganized things will get created we won’t be adhering to

41:02 will get created we won’t be adhering to like standards and best practices we’ll just make tables I think that’s okay to some degree and again I’m going to always lean back on like my I always feel like I keep talking about certified data or not certified data right if something is certified we should spend some time on it we should think about how it comes in what data how do we transform the data what’s loading all those things I also think there’s this other organizations come in with a little bit more consideration or considerate towards their data culture there’s probably a smaller team there’s people

41:32 probably a smaller team there’s people at the top that are asking a specific design from a certain team and say look you need to figure out what is the certified data that we’re going to bring in use the lake house architecture or use get it into fabric we’re going to have to learn how to do that and some organizations come in and say look we are a bigger organization we’re going to have to have process around this we want this to be a maintainable long-term solution and so they bring on members experts consult who have done this or have built similar architectures like this before and I think this is a

42:02 like this before and I think this is a lot of what I see right now and this is maybe I’m pulling back a weird concept here here is the the store the thing to me that changed a lot of things here is the up until this point we’ve had a SQL Server that had storage and compute all linked into one thing one item now we are in Fabric and now we have this concept of the storage is now separate from the compute which I think is this to me if I stand back and take really big picture look at this that is the story here that’s what’s happening I

42:33 the story here that’s what’s happening I have data flows that can write to the one Lake I have data flows that can write to a SQL Server I have data flows that can write to all these different places notebooks can pick up data and move it around all these different places we now have a data warehouse which is a SQL engine that can read things from The Lakehouse do transforms on it and put data back down into the Lakehouse so when I look at all these different compute engines that can now touch and talk to and interact with the Lakehouse to me opens the world up here for a lot more opportunity and now the conversation is do we need to build a

43:04 conversation is do we need to build a SQL data warehouse or the data warehouse in Fabric or are we building a Lakehouse in fabric so I think the decisions are slightly different now but if we’re building the data warehouse in fabric versus The Lakehouse what are the implications of that why would I go One Direction or the other Microsoft has provided some guidance around those things and to be clear the the data warehouse is the old version of SQL analytics Data Warehouse from synapse

43:34 analytics Data Warehouse from synapse which is like this massive parallel processing compute engine on SQL it can then talk to and read things directly to and from the lake like very incredible things here so I really like where they’re going with this one and we have and I think to the to our conversation here is I don’t think The Medallion architecture is out of place in fabric I think it is it it’s found its home I think it makes a lot of sense here it’s just a methodology it’s not a it’s not a hard standard some people in the chat here were talking about you

44:05 the chat here were talking about you the chat here were talking about hey it’s it’s just a know hey it’s it’s just a methodology it’s just multiple layers in The Lakehouse it’s helping you like logically know where to go for what data is when things break where to go it’s just a it’s a logical organization of things yeah I think I think Tommy’s got a good point though Mike right like so we can establish a yes right you Medallion architecture makes a ton of sense in Fabric or organized things in that way yes but do the teams that are building things have experience building things

44:35 things have experience building things in the data in that Medallion architecture and this is where argue not all of them are and some are just swinging it others are asking for help to build it so I think I think what what you’ll likely encounter a lot in the future is dependent on how an organization is rolling out fabric right if if you’re if you’re assigning permissions or to a business unit or a domain and they’re just going at at it do they do they have at minimum the

45:07 it do they do they have at minimum the starting material right this would be one of those things starting material here is how you want to shape your data the gold is what you’re going to be sharing between teams right making available to the organization how you get there we recommend you use this approach a as a Str a framework for how you go approach your your data in gestion if it’s the same team you don’t have an issue but if it’s the wild west I do think that part of issues that wither Consultants or organizations have

45:38 wither Consultants or organizations have to face at some point in time is identifying and organizing somebody’s landscape right because are they mashing together raw and silver or potentially like doing everything and you just have a mega table and the only thing that they’re doing is try and Traverse the historical records that are coming in right like because because there is the like they don’t know how you would separate things out or maybe there’s a silver silver there is no silver there’s

46:05 silver silver there is no silver there’s just when you go current all the business logic’s applied and they’re running into problem like bare minimum I think that conversation is going to start right off the bat when the the different groups within an organization need to interact with each other and they’re saying where do I get this data and it like where’s your gold layer and the team goes what what do you mean what is what’s the consumable layer for me and I I think that’s where potentially refactoring of and separating out some

46:35 refactoring of and separating out some of those tables is probably going to happen because there’s a reason that it makes sense that you follow this framework so that the systems work really well the queries perform really well so you don’t have to like Wade through all of history right when you’re when you’re querying for whatever the latest record is or whatever the case may be and and that’s where I see it it happening is the more you can project some of the I guess very data engineering Centric themes that will

47:08 engineering Centric themes that will guide how domains generate data in their own ecosystem I think is really important for organizations to do but it would be something that I I think they’re going to encounter real quickly when other teams need to access their data and are having problems and and I definitely yeah two thoughts come to mind set about what your your comment there was I think a lot of times organizations struggle in this area there’s two phases I think to every project phase one is turn it on get it working period phase two is typically

47:39 working period phase two is typically come back optimize clean it up make it better and I’m seeing a what I’m seeing right now is patterns of people are now using the lake housee with data flows Gen 2 it’s not the most efficient thing in the world it’s not the fastest thing but what people are doing they’re just I know how to use that tool I’m just going to use it I’m going to use that tool to build or land data in bronze do some simple Transformations maybe in a pipeline loading data to bronze notebooks I’m sorry data flows Gen 2 is picking up data from bronze doing some transformations to it cleaning up in silver maybe making some gold layers

48:11 in silver maybe making some gold layers as well in data flows gen two and now what I’m seeing is people are like okay we hear the internet saying notebooks are faster and better to work with let’s try and you so what they’re doing now is they’re saying let’s go look at my working data flow process rebuilding it inside a notebook now and saying oh look with four commands in a notebook it’s not as scary now for me to use notebooks to do the same thing oh and by the way it cost me less compute units to do the same Transformations so at the end of the day regardless of what we’re talking about lake house or non- Lakehouse or this stuff’s the same

48:42 Lakehouse or this stuff’s the same you’re bringing in data you have records you need to update you’ve got a and you you have either this slowly changing Dimensions type stuff that you want to keep track of every single change or you just say I want the most recent value of all the change records in a table this is a pattern that’s just so common it’s just happening over and over and over again and so the more you get your head around look this is not something brand new it’s doing the same stuff we’ve always been doing at the end of the day we’re just trying to land the new records update our tables to match

49:12 records update our tables to match whatever is in production or in the operational system that’s what we’re doing and it’s it’s the same pattern over and over again one tool that I’m loving right now that I’m hoping that get more support in pipelines is the copy job experience copy job is pretty dang sweet and does a lot of things for me that I don’t have to worry about and so I think Microsoft should do a better job of just saying look we’re going to build a single activity that is the slowly changing dimension thing I don’t care how you do it Microsoft do it

49:42 don’t care how you do it Microsoft do it as efficiently as you possibly can we know semantic models like rows and keys and and we want numbers as keys and columns but we get goids from our source systems like I I really just want to come in and be like look Microsoft here’s the semantic model that I want here’s the table that for customers here’s the table in the fact table I don’t really care what you do just make the keys work I don’t really care what they are I want these you figure out the most optimal way to build and store those keys in your system and oh by the

50:13 those keys in your system and oh by the way I’m going to keep bringing in these 10 hundred thousand million records every day these are new records okay here’s my existing table take these new records and this existing table and figure out what you need to update I don’t want to deal with any of this is the we have to build so much code and logic and notebooks and patterns here like this pattern is so consistent Microsoft should just solve it and I feel like they’re trying to solve it with this thing called copy job which is helpful it works on top of the lake housee but that’s where I’m like okay I’m starting to see some Vision here

50:43 I’m starting to see some Vision here they’re going to give us specific workloads or items in the that are going to give us that that need so anyways I’m very excited about this that’s where we’re going I hope it’s that direction it’s almost like you’re saying that you would just just want to pick the destination and get things off the ground and have your co-pilot take you there whoa snap like the sweater he’s wearing like your sweatshirt so co-pilot told me to say it I’m actually not even real I’m AI generated this is not even Mike car I’m

51:14 generated this is not even Mike car I’m still sleeping actually best best llm we have’t got yet so I’ll do quickly closing thoughts because I know we’re going on time I this is gonna be a conversation we’re gonna have yes I think it’s gonna involve because one thing I you made me spark Mike is I think we should revisit the second system effect or that the that book about old computer systems and talk about that in the world of fabric Mike the biggest thing that you said that I think is going to be the real change here is yes the idea the concept and and

51:44 here is yes the idea the concept and and the ideals are the same with fabric Chang except the people and because now you have people who are coming from all different walks of life with different experiences skills and needs and behavior of what their job and their job role is all of this that we’ve talked about mallan architecture was reserved for a particular type of role at a particular type of company or more in the data engineer so that’s where I’m intrigued to see where this goes but that’s really I think this is gonna be a conversation we’re going to

52:15 gonna be a conversation we’re going to continue to evolve and have a conversation on any final thoughts for you Seth no I think fabric is is one of those systems that’s just going to you never know how somebody’s going to use it and and just because we have the different workspaces domains it could be the same teams managing that right in terms of just where they’re pulling in so you have that Medallion architecture everywhere and and that is an ideal you’re probably going to run into the exact opposite of that though too right like where you that though too right like where like it it we we sit back and we’re know like it it we we sit back and we’re like yeah you it’s still important these

52:47 like yeah you it’s still important these Frameworks are built for a purpose yes we’ve opened we’ve opened up the doors for hardcore data engineering for folks yep and and hopefully organizations recognize that and do some d diligence at least communicating the best practices or ways in which they want to see data entered into that system yep because I do I do foresee that you could easily get yourself into a state where one of these domains has completely useless data right and then you start over and and without that structure you’re going to find yourself in a world to hurt I I would have

53:18 in a world to hurt I I would have totally age with that my only observation of this one all say is look in my doing the work around notebooks and Spark and all this Lakehouse stuff it’s super cool I love it I think it’s going to be a game changer for many organizations my only thought here is nothing’s changed it’s the same type of the pattern of data engineering you’ve been doing an ssis and SQL Server prior to this it’s the same stuff we’ve just changed the medium slightly so that’s only my my final thought is don’t don’t worry we’re not shifting the ground too much the tech underneath the hood is slightly different but the the methodology the the planning the arc

53:50 methodology the the planning the arc that’s still happens it’s still the same stuff so don’t get worried about it embrace it learn how to use it and I think you’re going to find you’re going to love it I love developing a notebooks I’m playing now with t-sql notebooks which are amazing I don’t like writing SQL in a single page code editor I like having blocks of SQL it’s amazing anyways that being said thank you so much for listening to our podcast this is been a super good conversation can’t believe how fast it went went super fast but thank you for your ears we appreciate it our only ask to you as the audience is please make sure you let

54:21 audience is please make sure you let other people know about the podcast that you had some fun and you learned some interesting things here share with somebody else we appreciate that Tommy where else can you find the podcast you can find us on Apple Spotify or wherever get your podcast make sure to subscribe and leave a rating helps us out a ton do you have a question an idea or a topic that you want us to talk about a future episode head over to power. tips podcast leave your name and a great question and finally join us live every Tuesday and Thursday a. m. and join the conversation or power. tips social media channels thank you all

54:52 tips social media channels thank you all so much and we’ll see you next time [Music] out

Thank You

Thanks for listening to the Explicit Measures Podcast. If you found this episode helpful, subscribe and share it with a friend.