SQL in Fabric – Ep. 254

“SQL in Fabric” isn’t just about writing T-SQL—it’s about making lakehouse data feel usable for the teams who live in Power BI every day.

In Ep. 254, Mike, Tommy, and Seth break down what Fabric’s SQL experience enables (Delta tables in OneLake, fewer copies, easier consumption) and the real-world edges you’ll hit as you move from Excel-first workflows to repeatable pipelines.

News & Announcements

• Welcome to the 3rd generation: SQL in Microsoft Fabric — Sam de Bruyn frames Fabric as a “third-generation” SQL warehouse experience: OneLake as the storage layer, Delta as the open table format, and cloud-native scaling that’s meant to be simpler than the Synapse Dedicated era.
• Tips+ Theme Generator — If your org struggles with “every report looks different,” a repeatable theme workflow is a surprisingly high-leverage fix.
• Subscribe to the podcast — Follow along for new episodes, show links, and the full archive.
• Suggest a podcast topic — Send a question or topic you want the crew to tackle in a future show.

Main Discussion

This episode is a practical sanity-check on the promise of Fabric: a lakehouse-first world where you can still use SQL to build and serve tables—without turning data architecture into a maze of copies and “special-case” systems.

Takeaways you can apply:

• Treat Delta as the contract: once your “tables” are Delta in the lake, more engines can read the same data without translation work.
• Use SQL as the bridge, not the whole solution: the win is making lake outputs easy to consume (Power BI, analysts, and downstream tools), not recreating a pure database-only world.
• Expect new operational chores: lakehouse patterns often mean lots of small files; you’ll need a plan for compaction/cleanup and table maintenance as you iterate.
• Incremental isn’t automatically incremental: some “append forever” ingestion patterns feel different than classic incremental refresh; validate how you’ll handle late-arriving changes and corrections.
• Separate storage cost from compute cost: cheap storage encourages keeping more history, but reading/processing that history is where you pay—design for the questions you actually need to answer.
• Design for the Excel reality: most orgs still run critical processes in Excel; the migration path is converting those brittle workflows into shared, reusable pipelines with governance.
• Plan for mixed skill sets: Fabric lowers barriers for BI practitioners, but the platform still rewards people who understand modeling, SQL, and data engineering fundamentals.

Looking Forward

Start small: move one high-value Excel-driven process into a Fabric lakehouse + SQL endpoint and measure whether you actually reduced manual effort and data-copy sprawl.

Episode Transcript

0:30 good morning and welcome back to the explicit measures podcast with Seth and Mike so if you if you’d listen to our last episode you you will notice Tommy had made some comments about Familia particularly his ma yeah so regrettably I’m sorry oh Tommy didn’t make it after after last after the last episode Tommy’s gone just kidding I’m just teasing everyone here you don’t mess with Italian mothers yeah

1:00 you don’t mess with Italian mothers yeah oh there oh tomm oh there he is he is he is here I I was a that was hilarious I didn’t even know you’re gonna do that just kidding Tommy is here it’s all okay for those of you who were listening I I kept tommy’s video off for the intro of this one so Tommy was freaking out and and stff like what’s going on I don’t know what happened here so it’s okay Tommy’s fine I made Double Down how many Italian

1:30 I made Double Down how many Italian mothers does it take the screw a light bulb oh I don’t know answer answer no don’t worry about me I just raised you I’ve done everything for you but you go off on your own that’s I’ve done so much for you so goodness so so none CA because you do it exact exactly right that’s amazing so welcome back to the explicit MERS podcast everyone’s here Tommy Seth and Mike hello everyone good morning welcome

2:00 hello everyone good morning welcome back let’s let’s talk about some main topics Tommy you found some articles across the the interwebs give us some information about what’s what you’re looking here I don’t have any links for this Tommy yes I’ll put them in our private chat as well but they should be in our little perfect so we we’ll start there was a big conference in London yes I believe it was the last week and Sam de Brun who’s been writing some awesome articles who’s actually where we’re getting our topic today went to it and

2:32 getting our topic today went to it and was really talking about what the big takeaways are and it’s really in line with what Microsoft’s been doing with fabric and the lak house Delta all about Del Delta Delta all the way yes which has been good so I’ll throw this article in the chat window as well just for people to follow along read along with it seems like a great article I didn’t have a lot of chance to read about it but it looks like he’s really poking on mic Microsoft has embraced the Delta format so the Delta

3:02 embraced the Delta format so the Delta lake or Delta paret format and there’s a lot of other formats Apache has their own Hy format Apache has another one for Iceberg seems very similar in nature but a lot of these other tools like snowflake are built on top of just a different format which so it seems like Delta is now becoming the the single place where you’re going to be able to build lake houses or store data into table form a really good article wor read through he’s got some great

3:32 read through he’s got some great diagrams in there really interesting architecture diagrams as well so yeah looks good s arle to spend a little bit more time digesting and extracting out exactly what he’s talking about there there’s another article tell that you found from Chris Webb and we love Chris Webb Chris Webb was a Microsoft MVP but now has turned Microsoft employee and he everything he writes is very Niche but it’s very good and it’s it’s very detailed what he’s talking about what’s this article about

4:02 talking about what’s this article about that you found so when you used to try to explain to consumers about incremental refresh he was like oh yeah we can store the data like so where does it live I’m like well don’t you dare edit the data set or turn it off it’s there but it’s also not there in the data set because it with incr it was like a pseudo historical data it didn’t really live anywhere but the data set but we had ability to look at previous records sure well Chris and I don’t know why anyone else has written this yet it’s like well you can do this in Fabric

4:33 it’s like well you can do this in Fabric and actually now actually have the data and have the tangible data not hopefully no one deletes this data set or makes a big change in incremental referesh goes away it’s there for good in a database in one lake so it’s just utilizing what fabric already has but now that data we don’t have to rely on powerbi incr incr refresh yeah and I definitely I liked his I liked his article here and when you use a loading process whether

5:04 when you use a loading process whether it’s data flows or something else they there is this menu dialogue that appears you can you can load new data or you can append or you can replace this information right so how do how do you want to handle the new whenever you run this process every day or whatever that is what does that look like and how does that help you load the data on one hand I definitely like this article because this is another way of doing incremental refresh right so yeah idea being when I run my refresh I have a specific day that I’m going to load

5:34 a specific day that I’m going to load data from and then I will refresh that information one thing that doesn’t really handle itself very well with the new pattern inside lakehouses is and this is one thing I like about incremental refresh incremental refresh will do a drop and replace on a small of data and what chis described in fabric doesn’t technically do that it’s just always append data into your into your Lakehouse so while I understand his comments in the article I think that the the way that

6:04 article I think that the the way that Microsoft is handling refreshing of data there needs to be a bit more enhancements around hey these are the keys I care about these are the partitions I care about and allow me to in incremental refresh you’re always loading the last seven days right so if you miss some data it automatically heals itself there’s no concept of that in this new fabric thing so you have to do that in a notebook or somewhere else so I I like the article I get his point but I still think this new refreshing method is missing a couple

6:35 refreshing method is missing a couple features yeah to make it more like incremental refresh and maybe that’s just me being picky about the process but and especially that you gonna say Tommy sorry I I think we had a bit of a delay the incremental refresh is still a hack it’s only in data flows gen two and it’s not super straightforward even Microsoft’s fabric documentation is like well you have to add a filter and another query it’s not just necessarily an enabled

7:06 an enabled feature yes and that’s and that’s where I’m like I wish it was a bit more automat you’re talking about business user here that are trying to do a data engineering activities you’re going to need to make it a bit easier for us to get around doing an incremental refresh in this new data flows Gen 2 so yes it works I do like the fact that you now have the ability to write all the data down so all the data lives in a storage account which is great I have found some weird weirdness around this as well when you create a lot of Delta

7:37 when you create a lot of Delta tables and I’m still trying to discover if this is actually a bug or am I doing something wrong as you make a Delta table it Stacks up files more more more more and as you build that table of information there’s a two steps in SQL that you use to clean and make sure all the data is compacted correctly you use a a function called optimize and use a function called vacuum I think the optimized functions are working I’ve done a couple trials around using the vacuum statements to get rid of data out

8:08 vacuum statements to get rid of data out of my Lake because it’s beyond the number of days that it needs to hang on to the old partition so I just need to delete those partitions get rid of them it’s not removing the files from my Lake and my Lake keeps growing in size and not shrinking so I’ve got to figure out what what is it doing and is there a bug here is something not right are we not actually removing these files when I’m trying to vacuum out so I have some more testing to do I can’t definiely say for sure I’m doing something wrong yet but maybe maybe not I’m not quite sure so stay tuned I’ll have more

8:39 sure so stay tuned I’ll have more updates as I figure out does their Lake actually work to be fair it’s only Point what 0236 cents per gigabyte so understand well so here here’s my demo that I tried I was in fabric I turned on one of their try this demo around streaming through fabric I’m like oh cool I’ll click the little buttons and it has this little Tri sample data set like neat this is neat okay wow this is pretty fast so within like I don’t know 3 minutes it’s streaming data in it’s like here’s here’s stock amounts

9:10 it’s like here’s here’s stock amounts like here’s all the different stock amounts every minute it’s streaming in every single stock in the Fortune 500 companies into a data set I’m like this is cool I can watch the data happen in real time well I thought that was neat and then I walked away and a week

9:24 then I walked away and a week and a half later I was came back and like I wonder what that thing’s doing oh oh it’s still on oh oh my word my table now is 35 gigabyt in size after like a week and a half of running this data and I was like oh my goodness and none of was nothing was optimized I had 53, 000 files very small files in the lake and it was trying to query those files if I had to read the table I was like wow okay well it I can see that it was doing things not quite optimized yet so I ran an optimized step I was trying to clean

9:55 an optimized step I was trying to clean up some Lakehouse information it’s a good data set I really like it it’s just I don’t know how to clean it up now and and aside from going back into the lake and physically saying delete this table and delete those partitions it doesn’t seem like the vacuum step is working so anyways there will be aha moments for people using Fabric in the same way that I did you click on a couple things it’s very mindful that especially everything in Azure whenever you turn on a new service monitor it for the next couple days yeah keep keep your eye on it keep

10:26 days yeah keep keep your eye on it keep your eye on it and make it a point to not just start up a new process and walk away and not look at the costs or the size of the storage or how long things are running because if you see something that looks weird and abnormal you need to stop that thing right away and figure out what’s going on I’ve had a number of times where people have done dumb things and you have to catch them and be like whoa whoa whoa slow down I I think you I don’t think you intended to make that money cost that much cost for this thing but you can catch it quickly and it the problem sometimes goes

10:57 it the problem sometimes goes away all right sorry that was my goal in a very large soap box here speaking from experience let’s jump into our topic for today so we have an article today from the same gentleman who went to this sequel conference Sam de broom I think debam how did you say it Tommy Sam de broom I that looks good Sam Sam made another article talking about welcome to the third generation sequel inside Microsoft fabric so good article

11:28 inside Microsoft fabric so good article kick us off Tommy give us some main key points here what’s he talking about in this article and why is this the third generation of SQL so sure thing Mike so if you think back in the good old days of SQL we were really with tear data we were with on-prem SQL was not anything that any business user really knew and even that if you were to even go back to the 90s from an analytics point of view you were it if you were touching any business database from trying to create it there’s no

12:00 from trying to create it there’s no Cloud option to just spin up a SQL server or SQL database you had to do it on your computer and obviously going through access there so but that was at least the time where the Kimble published like the data warehous toolkit so we’re getting to there second generation we’re introduced to the SQL data warehouse and I think it was 2010 and Azure synapse and all of a sudden now these little more pipeline a little more ways to cleanse and

12:30 little more ways to cleanse and train your data in in ssis what is the they’re called jobs right or in yeah and like just taking that idea but really pushing the next level because obviously with the jobs you would even your plugins or was a lot of custom building so now everything lives in the cloud and also the spinning up of cloud servers so still not where we’re at today but obviously that’s really the era that we’re currently active in the majority I think of organizations are active in we’re now

13:02 of organizations are active in we’re now entering this new generation and I think we’re pro I really think we’re just on the cusp of it of the snowflake the synapse sequels serverless sequel pools and then finally the big drop Microsoft fabric on that push away also from the conference as well this push away from SQL databases SQL tables SS SMS to how do we actually Control Data through a data lake or a lake house where that’s come becoming more and more a de facto way to control or hold and

13:36 a de facto way to control or hold and manage your data so Sam does a good job in the beginning of the article saying here’s some too long didn’t read I have a lot of article NOS notes here and he does a really good job I think summarizing why he thinks fabric is going to be the next best thing right what why was this the best consolidation of the Microsoft ecosystem he has a couple bullet points here that I think I want to read through that make a lot of sense to me right SQL is part of this story but fabric gives us the ability we can now have a a data Lake

14:07 ability we can now have a a data Lake all stored in one area called one Lake we don’t have to copy data anymore we can now provide links to data that already exists elsewhere in the lake house which is very nice or or the one Lake Area fabric now uses your open- Source Delta format which by the way the SQL engine or the SQL Ser you use in fabric can go read this Delta format and go understand the tables that are there which is awesome fabric decouples storage and compute this is a story that’s been happening a number of years now in Enterprise data warehousing Microsoft now I think they’ve been doing

14:37 Microsoft now I think they’ve been doing it honestly they’ve been doing it with fabric and I’m doing with sorry synapse previously but now fabric is like then moving this same design the same technology over to to powerbi as well fabric can scale automatically you don’t have to start at zero fabric has a lake house it’s it’s a true lake house and it was built for the Cloud’s experience which yep makes total sense this was an interesting point he made the cheapest pricing tier for a fabric is five times cheaper than running synapse versus a

15:08 cheaper than running synapse versus a dedicated sequel that was an interesting point I don’t I don’t know if I have enough costs in fabric have been a little bit elusive to me just because it’s always in Pre it’s been in preview I’m a little bit more hesitant to know exactly what things are going to cost me I do know the costs of the fabric skew because I can see that in the Azure portal but there’s a story account component that goes along with this that I haven’t seen yet I’m not sure how the billing of that’s going to work in in conjunction with fabric as well and the last Point here he kind well and the last Point here he makes is fabric is expected to go

15:41 of makes is fabric is expected to go live or general AV varability near the end of 2023 and then there’s you now have this great public preview that is free right now which I’ve been really enjoying the public preview being able to play with a much larger fabric environment has been very helpful for me to get my head around how does this thing work and how can I use it any thoughts Seth you come from the traditional SQL world have you had a chance to spend some time playing around inside fabric with SQL you feel like this is a a logical evolution of where of where SQL lives inside fabric yeah

16:13 of where SQL lives inside fabric yeah unfortunately I haven’t been able to stress test because they haven’t opened it up in the regions where I oh you have regions yeah yeah like all my all my regions are not supported in fabric yet so a little bit but I think I think the the biggest things that that Sam brings up in this right are a lot of the underpinings and Technologies and infrastructures used underneath the SQL engine for quite some time even get into the multi

16:44 time even get into the multi parallel processing MPP and their efforts to use SQL and its backend to start to handle larger and larger workloads and there the way I think about is like there’s two different kind two different parts of when people talk about SQL the SQL interface is what everybody is used to right the SQL syntax how you engage with tables how do you create joins all all of that and that unified experience is one that I don’t think people want to give up yeah

17:16 don’t think people want to give up yeah what I don’t think people care about or at least the purists of dbas like obviously do but there’s a lot of infrastructure under here right so if you if you think about as Tommy’s talking about history of SQL Server Like the machines that we had to stand up to support anything of scale were massively expensive right so it it was it was a tool for the time because company like it could handle anything

17:47 company like it could handle anything that companies could build in their servers and the more interesting and fun conversations are where you had these really large companies that would build these really large server firms and then stretch the rounds of like meming capacity and like that’s where you get into a lot of the the engineering aspects and storage and partitions and indexing and I think that’s what he talks about as well throughout this article is even in some of those Solutions like getting things to work or

18:18 Solutions like getting things to work or being took took a lot of work yeah right and I think like in this net new world of okay the fabric ecosystem leveraging Delta tables like everything we do in this ecosystem is stored the same way oh it’s huge it’s it’s it’s built on a framework that is big data scales and you don’t have to mess around as much you can tweak some things Etc but there there aren’t a lot of those

18:48 but there there aren’t a lot of those same type same type of conflicts related to creating and sustaining a backend framework while everybody else is interacting with it now now purist of this are going to say like oh there absolutely is there’s a bunch of things we could do on the back yes there there are I’m not saying like everything on the back end just goes away yeah yeah but from from that perspective like this is a a whole new world and he even calls out like ultimately like this data bricks this is what it’s been doing for many years yep and I

19:19 doing for many years yep and I think the separation being like data bricks is is it that hardcore engineering tool and fabric is a different and unique experience I think that’s open for discussion but ultimately I think SQL Landing in Fabric in the way it has is really unique because it provides the interface I don’t think it’s it’s got everything like an SS SMS interface has quite yet but if like how it works and

19:51 quite yet but if like how it works and how it stores data is radically different and I think that’s that’s the the big shift and the big win with you the big shift and the big win with leveraging SQL and fabric that was know leveraging SQL and fabric that was one of the reasons why I really enjoyed looking at synapse so fa I think I think a fabric as an evolution of synaps or or synap SQL was this initial build and then they moved it or they changed some of the bits and now they’re able to run it inside Fabric or make this new thing for fabric so I believe there’s actually different engines here he he talked a little bit later on taking away some of the

20:21 later on taking away some of the confusion between what fabric and synapse SQL was doing cuz synapse SQL is running on a MPP massive parallel processing and it looks like the now current name of synapse Warehouse that now part of fabric is running on a a system called Polaris and again I’m not very up on my SQL side of things but it seems like the engine is slightly different compared to what it was in the past regardless though I really like the experience of make tables of data using

20:51 experience of make tables of data using whatever system you want that are Delta formatted the synapse engine was very easy to go in and reach into a Delta table read it and support that read of information into powerbi and so that’s what I really liked about the the syap space I like being able to write run a pipeline I like being able to read my data from the lake I like to read my Delta tables that made a lot of sense it made the architecture very easy for me to build everything in data bricks all the all the synapse tables I wanted or Delta tables I wanted then go read them and then I materialize them into views

21:22 and then I materialize them into views not materialize really but provide views inside synapse and then powerbi can go read those views and the data directly so I really like that pattern and now that it’s all bu built into into fabric really interesting I really I really like where this is going yeah and I think above and beyond just the the technology Stacks completely separated now where you do have true storage and two true compute right I think that was one of his big call outs of synapse and

21:52 one of his big call outs of synapse and that the older infrastructure and now fabric is you paid for the full capacity in synaps like you paid for everything and the increments up were significant and like reading through his article it almost seems like you only got half the capacity like you’d have to buy a full full swath and how it was implemented in terms of partitioning and indexing like you were consuming some of that capacity just with that infrastructure yes right and if you think about like how if you

22:23 and if you think about like how if you and if you think about like how if we if we even take like the the the know we if we even take like the the the storage component and I don’t I haven’t like I I I wonder how close you guys have done the analysis around the storage component but like of Chris web’s article where he’s saying hey we only pay 023 cents per gigabyte in one Lake for storage coupled with compute and we know computes the big cost driver right yes driving up the the capacity Etc but it’s usage based yes right because now on this framework where you’re storing in

22:54 framework where you’re storing in Delta paret it’s cheap yeah like massively cheap so why pay for it and like the fact that you don’t have to now and incrementally like you’re you’re storing terabytes of information for 100 bucks a month or something like it’s just the massive amount of data that we can collect store and and then use I think is is what’s interesting but if I want to read a terabyte of data I’m GNA pay for it right exactly right yes and and

23:27 for it right exactly right yes and and oh to that end around this whole storage compute layer right the the speed of which I could innovate on top of these platforms is really what gets me right I can I could in a couple minutes I can turn on a data bricks environment I get a cluster I get machines I can configure what I want I can read data I can have the storage like in a couple minutes I can have all the infastructure I need that can scale up to tens of thousands of dollars to spend if I need to to hand handle you

23:58 spend if I need to to hand handle you spend if I need to to hand handle billions of rows and tables like know billions of rows and tables like that’s instant almost that compared to like having to procure a server and plan for it and I hear a lot of people who who particularly talk to me around like this is this is when I tell I can tell customers are coming from like a SQL world or they’re reallying things on Prem the question to me becomes okay let’s build a lake house how do we size it I don’t understand what do you mean how do you size it well how many transa like what tell us what size

24:29 many transa like what tell us what size we should buy what Hardware should we buy or what should we turn on that’s like we get the right size and and immediately I go to I think you have the wrong mindset of how this works because it’s not a sizing problem everything and this is and this is how I approach things in Azure everything in Azure is buy the smallest thing you can afford go to go to the cheapest offering that Microsoft has start there if you have a SQL server and you need to put a SQL Server database up in in Azure start with the smallest option because if it doesn’t work it’s literally a couple

25:00 doesn’t work it’s literally a couple seconds of well let’s just bump it up a level let’s just let’s just turn it up there’s literally a dial that says spend more money and it will get better and faster and we will help you do that real we will yes right so from that perspective there is no trying larger size machines I always build in the smallest environment that I can and then if that doesn’t work or if it’s slow I can monitor it I can see where it’s slow is it a for example is it something in my process is it my my ET is is there something in my code that’s slow or is it actually I just need more

25:31 slow or is it actually I just need more compute to do the job and so you have indicators that tell this is cool because now I can innovate really really fast and I’m not focusing so much on the infrastructure I think this is going to be a major change for a lot of data engineering teams that are more on Prem or traditionally focused well it’ll be interesting to see if the hype like I I think over the years there was there was definitely a cloud averse MH crowd agreed like I don’t I haven’t hit that in a while you guys wouldit more

26:03 that in a while you guys wouldit more often right because you’re Consulting with other firms and and things like that a lot but it’d be interesting if people hadn’t made the leap whether or not fabric is going to make that like because it’s supposed to easier because you get all these things together it was a point that I was going to make and it’s interesting you went into it as well is okay you think about the the speed to something yes and if You’ been in technology for a while for if you’ve been in technology long enough to lose your

26:38 hair like these were huge planning cycles and I think to for some companies like you’re even saying Mike like still in a mindset where yes all right well we want to start this initiative what size data brecks do I need to make this go we’re talking about it now but we’re we’re going to start it next quarter right yes and now it’s like well you want to R& D something give me give me five minutes let me spool up the stuff and let’s yeah exactly you need a resource Group no problem I I think yeah I think I think the the the challenges around those are going to be like oh man

27:08 around those are going to be like oh man how how much is this going to cost me though right yes correct the cost Factor yes so so the transparency of that I hope we find out pretty quickly here yeah like I said I couldn’t stress test and understand what those were in my environments which is disappointing but hey hopefully I get the the live thing soon But ultimately like joining all these things together it’s a huge story for you to say hey you go to customer and it’s like all right you want to do this thing all give me a day yeah like Point

27:39 thing all give me a day yeah like Point me at the stuff give me a day and then I’ll show you this entire pipeline that I can build like that fast they might be ambitious but well depends on how much depends on what you’re doing some of this could be pretty quick I some of this could be pretty quick there’s a lot of patterns you can mean there’s a lot of patterns you can develop now that are like there’s common pattern be thing like hey watch this select join these two tables together and they would never know that it’s not like a SQL storage engine table it’s okay and that’s that’s the beauty of this is is the interfaces

28:11 the beauty of this is is the interfaces that they’ve interwoven and the underlying structure doesn’t matter to those interfaces yes it’s it’s just the most optimal way to store data yeah well here’s the thing though that I’m worried about es with the cost thing just a friendly reminder if you have a ton of fabric artifacts and you’re getting very close to the October first date when cost actually turns on correct I don’t know actually

28:41 turns on correct I don’t know actually so I I keep saying so it’s almost October 1st right now it’s what is it it’s the 29th of September 28th of September right now I still haven’t like my de My Demo environment still says 29 59 days until your trial’s up so I don’t know what happens October 1st at this point so I I’m I’m not sure how that works that’s just well did the did the free trial of of fabric also like conglomerate like

29:11 of fabric also like conglomerate like your free Pro license or like all the other free that’ll be interesting too to see what that’s doing I don’t know yeah exactly exactly right on that one well what happened to PBI where’s the PBI free Pro license license exactly come back out pop back up in the fabric Tri ends in an effort to be safe I will be turning off luckily I was at least organized all fabric items in the fabric workspaces and didn’t try to we’ll say

29:42 workspaces and didn’t try to we’ll say cross contaminate so I will be deleting all my fabric workspaces at least before October 1st just because I’ve made that mistake once with Azure yes yeah and I like I would agree especially if you’re using volumes but how much data do you actually like do you have processes running like are you using compute daily yeah I’ve been trying to the one Lake I’ve been doing a SQL database using I would say like turn off the automated things if you don’t understand like what the costs are going to be

30:12 like what the costs are going to be around those but unless you have massive amounts of data it’s Pennies on the dollar dude like what Tom been doing Tommy’s been streaming things in from all over the place he’s been no just I’m Tom it’s been it’s been busy we’ve been busy let me put it that way so Jack in the chat asks a great question is fabric cheapening our Worth to our organizations right we have these these teams of people that have been traditionally focusing on SQL and data

30:43 traditionally focusing on SQL and data engineering and making tables making sure that data is consistent and ready to to go I I think this is a great question because I don’t think this the fabric ecosystem is cheapening what we do rather it’s causing us to shift in the same way we were shifting so previously I think we were focusing a lot more on architecture the infrastructure the hardware the physical machines and what data had to go on them in addition to what data engineering did you have to do to get the data onto those machines I feel like what Fabrica is doing is it’s allowing us to focus Less on the

31:14 allowing us to focus Less on the infrastructure portion Microsoft is just going to handle it they’re going to bolt it all together they’re going to handle the virtual Network it’s going to all live in fabric it’s all going to be this nice easy to communicate ecosystem where tables can come in you can build what you need to build and share data easily so I think I think the the the need for us to actually do the data enging potentially you could lift that off to the business a bit more you could give more responsibility to other people in the organization it’s not going to rest all the shoulders of it now I think there’s going to be a

31:47 it now I think there’s going to be a an opportunity for businesses and it to continue to partner together further about what they want to build now where I think our our value is more needed than ever is how do you organize the data in the same way powerbi made reports a commodity there’s now many more reports all over the ecosystem there’s going to be a lot more data things all over the ecosystem so what is what is good data what is certified how do I use these other tools to help me build better things or better you

32:19 build better things or better you build better things or better better how do you what am I know better how do you what am I trying to get here sethy it’s the it’s the idea of the management and governance is becoming more and more important now than the infrastructure well actually in all in all reality think about that like management governance has always been I I don’t think it’s become more important I think true everybody wanted it and you just it it takes a it’s it’s a longer road to get there if we can get there faster because we have these new efficiencies whether it’s

32:49 efficiencies whether it’s because we’ve Consolidated all this stuff under our Fabric and it’s all stored in the same way or is it the enhancements of AI right where we talk about we we just had a podcast on Javen Paradox right deficiencies like I think we’ thinking in terms of being obsolete but I think what’s happening is we’re like things like fabric are allowing or will allow people to be become much more efficient at some of the things that were just blockers to where we wanted to be spending our time anyway right like I have to deal with

33:20 anyway right like I have to deal with data infrastructure business doesn’t care about what’s the business Val value behind that right the business value is behind like we collected all this data we now have these insights because we ran it through all these ETL processes we cleaned it we cleansed it and we produced value to the business and they’re making decisions based off it indexing storage Co like do they care about the cost of how much it it takes to get to that point of course they do but how the how Engineers go about doing that the code that has to be

33:51 about doing that the code that has to be written they could care less right so if that’s our job and we don’t have to deal with infrastructure as much or we don’t need the people to deal with infrastructure great cost savings if all a sudden our coding gets better because we have SQL in fabric now we have you we have SQL in fabric now we have The Notebook experience in know The Notebook experience in fabric now like so you can interact in in the ways that are best for you to become more efficient the your efficiencies increase like we’re we’re hung up on and this just ha this

34:22 hung up on and this just ha this happens regularly on teams you get hung up on logic I I don’t I don’t know how to write this thing I don’t know the function I don’t know how to create the Dax calculation I don’t know like how how long does it take you to solve that technical problem that in the future could be like I know I want to do this and one minute later you have your answer and you can move on yeah your efficiencies of trying to develop solutions to get to a point of where you want to see the data yes I think all of the sudden allows us to spend a lot more

34:54 the sudden allows us to spend a lot more time solving data related business problems or producing valuable artifacts that the business can use as opposed to where we’re now we are now where we talk about the technical challenges all the time so this this I think just reduces or the barrier of Entry but also allows for a wider audience to then engage because that level isn’t so high but that doesn’t mean you as a practitioner right now don’t have vast volumes of insight because frankly it’s

35:25 volumes of insight because frankly it’s second nature to us data and how it relates and how we can bring things together because of systems we we’ve been working in that concept is so lost for Mo most business users they don’t know how to do that yeah that’s true so fundamentally just working in data in these systems the way that you have you’re not obsolete you’re the most Valu you’re just become the more valuable people in the organization I think that’s how I see it the funny thing here is Mark says in the comments but but does it connect to excel can

35:55 but but does it connect to excel can I can I still connect as but that’s one of the things Sam points out out right table on this stuff right that’s right and and but that’s funny to me because this this is what makes me laugh I think about this and go wow we have all these we have this modern technology it’s it’s fabric it’s Delta all these cool things business users yeah but but I need an Excel yeah yeah but I need can I get a table of this just makes me laugh so I I like

36:25 this just makes me laugh so I I like the so I right now I feel like the the main tools that I really love to work in are this whole data brick so the The Lakehouse or the modern lak house whatever you want modern data warehouse whatever you want to call it I really like the tools of data bricks and power like data bricks is awesome the data the engineering experience there is incredible from a data engineering perspective I just enjoy jumping into notebooks and Building Things in that environment it’s just so cool so the fact fact that

36:57 just so cool so the fact fact that Microsoft is trying to take some some lessons from them and incorporate some of that into fabric good on them it’s still needs a lot of love it’s still not quite there from a from an engineering perspective but they’re getting there the concept is is getting better powerbi is amazing the fact that I can model data it’s it has changed fundamentally how I think about information like it has changed my entire mindset and when I talk to people or newer people around powerbi and what it’s doing and how you

37:28 powerbi and what it’s doing and how you model data there’s there’s still some heads turning trying to figure out like

37:32 heads turning trying to figure out like wait we need to grab all the granular data and load all yeah we do and so it’s it’s it’s fundamentally changing how people I think still are trying to understand how do you build a data model and it’s funny when you do this for a while you start looking at data models and you can start seeing potential problems as you work with customers or team members around oh here’s my model okay I’m looking at this model just looking at the tables in the gra graphs and how things relate let’s make sure there’s no measures here in this in this table because that’s that’s not a place

38:02 table because that’s that’s not a place where you’d want to put a measure I think things might be weird and literally a conversation later on like a week later you’re like this measure won’t work well I told you don’t put the measure there because the way you built your relationships this measure will not be able to calculate based on this other dimension that you supply so like you start seeing what happens here and sometimes I feel like I’m talking cuckoo because I’m seeing so far down the road this there’s very much implications as to how you need to calculate this thing and the impact of the visuals

38:34 thing and the impact of the visuals and what your filtering really has a huge effect on how you design and build the data model and so there’s there this really interesting integration if you understand how all this works you can see patterns really far in advance that you can indic use inside powerbi which I think is really cool and then the other one I’ll I’ll note here every business I walk into everyone everyone I talk to every time I talk to someone who’s doing P or powerbi they’re like I was just given this Excel sheet and it’s unwieldy like there’s so much

39:05 and it’s unwieldy like there’s so much in it and I think this is going to get worse honestly with the with the addition of python inside Excel now now you have to be an Excel user and a python user to be able to unwind these Excel documents it’s going to get it’s going to get nuts I was just I had an epiphany the other day right someone gave me an Excel file it that Excel file is doing multiple things across multiple teams right we have multiple people playing on different okay I have another Excel file that’s aggregating all these sub Excel files

39:35 aggregating all these sub Excel files into a file into a larger aggregation cool but what happens what happens when I have to look at python inside Excel now what happens if they when they build entire Solutions data warehouse databases basically what happens when the build business builds databases in Excel using python pieces and I got to go unwind that file now I this has changed how I need to know know what I need to know in order to work with Excel a bit more now sorry I you’re going to say something Seth no it’s just interesting to me that like you going

40:05 interesting to me that like you going down that well refreshing my memory of rabbit holes of crazy VBA code and yeah all the things that are going on Excel plus then you throw on python like add python to that now it well it It throws a different spin on for some business business users is fabric too difficult because if I wanted to wind that Excel file I have python like I have connections that’s ADF I can use python in my notebooks yes I can like

40:36 python in my notebooks yes I can like it’s just a pipeline correct except it’s reusable and repeatable and I can share it with people outside of an Excel file right it brought to mind like this really interesting idea of like oh okay well like how complex that process and it is it’s a business process an Excel file yes could be ripped out and automated into a pipeline that could be Lage for the business actually Tommy you’re making a lot of faces but you have things to say so no no no no so like I’m I’m listening and

41:07 no no no so like I’m I’m listening and I’m trying to digest because yes all the things are amazing again I think it’s really hard for us and especially the three of us as much as we try to get out of the lens that like like we see the world and what we think is best and also just the the level of knowledge that we have we’ll never no matter how much we try understand a business user who’s just starting with data forget just powerbi anymore because we’re so past that we’ve talked about we think in data we you

41:38 talked about we think in data we you talked about we think in data we we know models like the back of our know we know models like the back of our hands so you’re we you’ve seen this too just trying to get people introduced to powerbi consuming it and the interaction with it and now we’re saying for the power users yeah you have all this control over your data and there’s all these ways you can connect to things and I know a lot of of it departments are still so concerned about just the data and their reports and looking at at that a different way I if this is the way we’re going that’s fine to me this is still the majority of fabric is

42:11 this is still the majority of fabric is still I think on the data engineering side I’ve been using notebook and like and I I think there’s maybe like maybe some of the data flows where they can spin up their own one Lake to push the like their business-led data into too and then they can connect to that and other sources obviously powerbi but it’s more to me rather than for the business at least for the the normal power user who’s in that middle stage

42:42 who’s in that middle stage it’s less of a complete dat data engineering change than a level up from data flows where what data flows did for the business fabric for them is going to be a an upgrade from that but nothing I don’t think it’s going to be anything more well I’ve always said this I’ve said this multiple times I think fabric is a better deal for the business user and not as good of a deal from the data engineer or the data scientist level right because when I look at if I think about hardcore companies that are already in Cloud already building

43:13 already in Cloud already building pipelines already doing modern data warehouse things they’ve been using as your data Factory already for maybe years they’ve been building their own lakes in whatever tool they want maybe it’s a snowflake Maybe it’s data bricks right they’ve already been engineering the data somewhere else with other tools those tools are refined for 10 years and here comes fabric one year out and again Microsoft has been able to to watch and learn now what does Microsoft usually do with tools right Microsoft usually not

43:43 with tools right Microsoft usually not first to Market but when they come in they usually come in make it cheaper than everyone else so you typically see a Microsoft so what you may see is you may see a very price sensitive Market that they will entice people to move over to things of fabric or things that are Microsoft more Microsoft Centric based purely on Microsoft we’re going to build things more efficiently so to do the same workload you’ll be able to do it faster with less cost and I think I think that will be for me right as an

44:14 think that will be for me right as an engineer of data engineer already that’s what Microsoft needs to lead with they need to entice people to come to their platform and do so betting on the speed and price it’s faster and cheaper to run that’s I think where you’re really going to get people to start widely adopting if you’re already in the cloud there’s another point to your another comment Tommy to your point here is if I’m an on pren company powerbi is been likely the first thing you’ve ever done in cloud or SharePoint is like so

44:46 done in cloud or SharePoint is like so when companies start moving away from SharePoint on Prem and start moving to the cloud there’s this whole hesitation around I don’t want my dat in the cloud I don’t want the I remember Seth and I we we would go into conversation about this when we were Consultants we’re like that was a that was a real thing people were like I don’t I don’t want in the cloud it’s not how do I know it’s secure how do I know it’s not how do get leaked or not leaked dude you already have your SharePoint in Cloud well it’s SharePoint but you’ve already accepted that that’s a part of the cloud infrastructure right your active directory is now already in Cloud why is it that much harder for you

45:17 Cloud why is it that much harder for you to Leap Forward and say let’s use powerbi and I think for a lot of businesses they made that jump they allowed powerbi to come into their ecosystem and they’re like well Cloud isn’t that bad I get reports I can I have I can upload what I want I have almost unlimited scale at a pro user if I’m below one gigabyte in size and files H this is not that bad so I think I think powerbi has been this gateway to a lot of other companies being comfortable or getting more comfortable with working in cloud and moving there well I

45:48 in cloud and moving there well I in cloud and moving there well the other part of that too in in mean the other part of that too in in that conversation I think is is many many companies realize that that when people were pushing around powerbi that there was this argument well data security and and I guarantee you every company we went into we would we would be able to say okay so when’s the last time you talked to your business units are you monitoring people sending Excel files out out the door because they are and it’s company data

46:18 because they are and it’s company data and did that so they’re they’re doing that already you have holes throughout all this at least I’m deploying to a safe environment that’s seced and monitored on it people if people export a report and send it what’s the difference between that and Excel and them giving data out right like there is no difference yeah I think what’s interesting to me is as these tools evolve and I just want to Circle back a little bit from the data engineer fabric being the data engineering side sure is I agree with

46:48 engineering side sure is I agree with that now but I think there’s a correlation where like we’re it’s a new environment we’re trying to digest it as quickly as possible and and I think

46:56 quickly as possible and and I think there’s going to be new patterns where you engage your heavy Excel user into the fabric patterns but I I guess and and the reason I say that is because I think long term if co-pilot goes alongside this stuff it’s going to be very easy much easier to integrate and build the data objects that we need to the process flows Etc what what is your guys’s take though because to Tommy to your point you said like data flows there’s also power query paths like m i is the

47:27 also power query paths like m i is the evolution of how we engage and automate business processes typically through Excel is it is it going to be Fab fabric pipelines so that we get it into Delta pay tables right away or are you guys going to stick with the like just the Automation and powerbi right now with power query or potentially data flows yeah this is going to be interesting to see where this is going to start with i my opinion here is I think powerbi I think the powerbi user Market Market is going to start with what they know

47:57 is going to start with what they know and honestly how I started looking at fabric was I understand data flows it was a nice easy UI I’ll start there so for me I really liked going into fabric with a business mindset and say oh I’ll just figure out how data flow Gen 2 works oh wow this is very similar oh cool I can just write things down to my Lake H interesting so for me there was a lot of friction that came that was removed by they provided me in fabric a lot of business related tools that I’m very with what I think this is going to

48:27 very with what I think this is going to do it’s going to allow people to say okay that wasn’t that bad what else should I learn H there’s these pipeline things what could I how hard is that okay that’s not bad I can build a couple blocks of data now it may not be very efficient day one but I think people were going to start so what I think it’s going to be from my perspective business users are going to start down a journey of being able to learn more of these traditional data engineering tools and we’ll be incorporating them more and more into their workflows yeah and I’m I I’m not talking an every Excel user here

48:58 I’m not talking an every Excel user here right because we we’ve talked several I think there’s going to be challenges with all of business adopting or where do you where what workspaces or pipelines do you allow them to get into Etc but I’m talking about the folks that when you go in and you solve a big automation problem right or you you buy you win back a ton of time for a company because there’s this manual slash hack together Excel file that person knows a lot they know more code than they think they do they do right then you throw in Python and

49:28 right then you throw in Python and what’s the evolution of those users and analysts within Excel and that’s the user I’m talking about this a smaller subset yeah yeah but at the same time like the engagement for Consultants around companies that want to buyback time getting rid of those like Excel automated processes into workflows for powerbi for reuse I think there’s a better story here long term where there could be correlations between these types of hey you’re doing python okay here let me show you we’re going to do that here here’s how and why because all

49:59 that here here’s how and why because all the current Solutions and one of the things that fabric to me is like really compelling from a long-term story and strategy perspective resolves that problem where where powerbi data flows data sets like all these things are still silos of data right I can’t easily and reuse them in in the ways that I I want to for everything I can connect them right but then I also have this SP web of interwoven interconnected things that are like this relates to

50:30 things that are like this relates to this relates to this and this is supporting that and blah blah blah whereas fabric is this ecosystem that I think produces a different unified data set on the back end where all of all of these tools connect to from Source systems so are there dependencies sure but you’re not creating this like interwoven connect to a b b connects to CDE e f that just EXP explodes explodes explodes yes because that’s the world we live in right now sure so like

51:02 we live in right now sure so like bringing it back full circle that’s also one of the main reasons why I think it’s really important that you have this SQL interface because that’s going to be from a introduction standpoint one of the easiest paths to bring people into fabric because it’s going to be as long as that framework gets better and is more consistent with how they utilize SQL and if they can write their queries then and they don’t care about where stored or how it’s stored just like they never have before before today anyway they’re trying to extract data in the most efficient way possible and that

51:33 the most efficient way possible and that can be done on a on a ecosystem that is much more robust in terms of not sharding and pushing data into different systems and having to consolidate it and pull it together etc etc one of the gaps I still think I see here especially from the when I think about like the Excel user and what fabric and or powerb is doing one of them and I’ll wrap on this this comment here I still think there’s a gap around I’m an Excel user I’m going to enter in

52:03 I’m an Excel user I’m going to enter in data in a portion of an Excel file and then on a Cell by cell row by row basis I’m producing calculations that I need to quickly adjust and or modify yes there’s Excel excel’s great for doing data engineering inside the context like of a table or tables that are local to an Excel file I don’t see anywhere where fabric or powerbi is making that easier for us at this point there’s still a gap there around I’m a user I want to edit multiple cells of data in a table

52:35 multiple cells of data in a table quickly and get results immediately calculated so yeah oh so what you’re saying is fabric is missing the Excel interaction exactly they’re Miss there you go maybe 90% of the work would be done right maybe maybe data activator great idea dude Maybe data activator will give us some more of those but with Excel online you have some really cool potentials and possibilities there maybe we can do some more things with this maybe this is where this is going I’m not sure so yeah

53:07 where this is going I’m not sure so yeah we’ll see I think there’s I don’t want to discredit all the amazing working what people have done in Excel I think it’s incredible and and there’s so much literally Excel runs the world I’ve seen many very large companies make incredible decisions based on things that come out of excel that’s that’s terrifying maybe or maybe not I don’t know like yeah probably name one time you walked into one of those processes where when in that Discovery there wasn’t a wasn’t a oh oh that that shouldn’t be doing that

53:40 oh oh that that shouldn’t be doing that but but the the business process the the amount of effort has been spent on working and massaging that Excel file is incredible so there’s very smart people building these systems to to have things run and large I think you’d be shocked if you looked at like the Fortune 500 companies which which executive reports are getting rolled to the very top of that company there’s probably a lot of Excel files that are getting enrolled to those Executives and and decisions are being made on those those Excel files so how do you integrate that level of like sist like

54:12 integrate that level of like sist like sophistication thought process design into all the stuff that is fabric how do you make all of this stuff just work together in a very sess way I think we’ll get there but it’s a very interesting concept to think about anyways anyways with that I think we’re just about at time let’s go ahead and wrap the episode this has been a good episode we started a little bit early today just because we had some engagements early this morning so we had to get started so if you just joining us right now go back and watch the beginning of the episode you should be able to start from the very beginning and catch up on two time speed thank you all for watching and

54:44 speed thank you all for watching and listening to the episode we really appreciate the community here it’s wonderful thank you so much our only ask is if you like this episode if you liked what we were talking about please go out and share it somewhere online or share it with someone at a coob worker let someone else know that you’ve been listening to the explicit measures podcast and it’s good it’s you’re taking that run that drive into work that’s just so long walking from your top part of your house to the bottom if you need a a two-hour walk to get to your basement like I do maybe you want to listen the podcast so just kidding I don’t have a two-hour

55:14 just kidding I don’t have a two-hour walk it’s literally like 5 Seconds to get from upstairs to downstairs to work so anyways that was a joke Tommy where else can you find the podcast you can find the podcast anywhere you get your podcast Apple Spotify make sure to subscribe it helps us out a ton have a do you have a question idea or topic that you want us to talk about in the future episode head over to powerbi tips podcast leave your name and a great question and finally join us live every Tuesday and Thursday usually a. m. Central sometimes 7 AM and join the conversation on all powerbi

55:45 join the conversation on all powerbi tips social media channels thank you all so much and we’ll see you next

56:19 [Music] out

Thank You

Thanks for listening to the Explicit Measures Podcast. If you enjoyed this episode, share it with a coworker and drop a question or topic suggestion for a future show.