Who Owns the Connection? Managing Access and Chaos in Fabric Pipelines – Ep. 438
Mike and Tommy dig into a very real Fabric pain point: connections that get created by one developer and become invisible or unusable for everyone else. They share practical governance patterns—especially security-group driven ownership—and what Microsoft could improve to reduce friction and ‘paper cuts’ in team-based pipelines.
News & Announcements
- Enhance data prep with AI-powered capabilities in Data Wrangler (Preview) — Data Wrangler in Fabric adds AI-assisted capabilities like rule-based suggestions and Copilot-driven custom operations so you can describe transformations and generate code with previews before applying. The update also highlights improved support for translating operations (including Copilot-generated steps) between pandas and PySpark, reducing “syntax friction” as you move from exploration to scalable Spark execution.
Main Discussion: Who Owns the Connection?
This episode starts from a mailbag question many teams will recognize: one person creates a connection for a Fabric pipeline, another developer opens the pipeline and gets a warning that they don’t have permissions for a connection they can’t even see. From there, Mike and Tommy unpack how connections have evolved, why they’re tricky in Fabric, and how to create a workable governance model today.
The pain: connections behave like separate “objects” (but without enough transparency)
A key frustration is that when you don’t have access to the connection used by a pipeline, Fabric can obscure the details so completely that you can’t identify the missing dependency (name, owner, etc.). Mike calls out that even if credentials should remain protected, team members with edit rights in the workspace should at least be able to see what connection the pipeline depends on and have an easy “request access” workflow.
Workspace permissions vs. global connection management
They discuss how governance historically centered on the workspace (who can edit artifacts) and gateways (who can use/edit connection settings). Fabric’s connection manager feels more global and decoupled from the workspace, which is good for reuse across workspaces—but introduces friction when multiple developers need to co-author pipelines.
A pragmatic pattern: manage connection permissions with security groups
Because there isn’t currently a clean “only these people can create connections” switch, the most scalable practice they recommend is:
- Use security groups for workspace roles (member/contributor/admin).
- When creating connections, grant access to a security group (not just individual users).
- Treat “connection sharing” as part of the team’s definition of done (a process step), because the platform doesn’t enforce it yet.
They’re blunt about the downside: if someone skips the process step, there’s no safety net—connections can still become orphaned or “owned” by one person.
What Microsoft could improve
They outline a few “this would make life easier” improvements:
- A tenant/admin-level “see all connections” view (and audits of who has access).
- Better deep links from pipeline connection warnings directly to the related connection.
- More ergonomic UX that doesn’t wipe your in-progress settings when navigating to manage connections.
- Potential future direction: workspace identities / shared identities that can own connections so teams aren’t dependent on individual users.
Looking Forward
The takeaway is that connection governance in Fabric is still maturing. Until the platform adds stronger admin tooling and workspace-aligned connection ownership, teams should lean on security groups + process to reduce the “who owns this connection?” chaos.
Episode Transcript
Full verbatim transcript — click any timestamp to jump to that moment:
0:34 Good morning and welcome back to the Explicit Measures podcast with Tommy and Mike. Good morning everyone and welcome back to the show. Feeling good, Mike. Good morning. It’s this is a recorded episode. So just letting everyone know this one has been recorded. So this is not live. Just in case you’re looking to talk to us in the chat, we won’t be there. But you’re more than welcome to communicate with each other in the chats. Typically, YouTube is the most popular one. Seems like has the most interactions going on in chat. Today’s main topic will be around who’s who owns the connection.
1:09 Managing your access and the chaos in your fabric pipelines. Just going through some of our experiences here. , it feels like some of the connection string things have been changing or at least permissions on who can edit them or create them or build things around the connection strings and just want to unpack this. What does it look like? Is there any tips or tricks that we’re finding that are going to make this easier to manage? , and maybe some opportunities here for Microsoft to switch things around here a little bit. Anyways, that being said, that’s our main topic for today. There’s a question around this
1:41 One and then let’s do a news item. Tommy, you found something around in the news. Oh well, it’s one of our favorite products. I think the fabric ecosystem is our all around data wrangler. And this is actually something I noticed is available in VS Code, but now available in the fabric UI. And really what this allows us to do is simply use co-pilot and other AI powered capabilities in data wrangler. And what they actually allow us to do is one data wrangler is going to provide some automated suggestions and also you can say hey co-pilot I want to do X Y and Z to this
2:15 Data frame or whatever you’re trying to do. Copilot will then generate the code or action that you need the function whatever you’re trying to do and allow that to be implemented into data wrangler. Mike, this is such a must needed feature that I think one puts a few of our videos already now out of commission. We got to retire them from our less learned fabric series, but more importantly, it’s about time this is available. Yes. And there’s a couple things in here. So, the AI is able to
2:47 Assist you in various areas that I think is really interesting here. , a couple of the things that I’m I’m looking at here in this article, it’s the AI pieces of this that’s fun. , you can now suggest things, , you can talk to it directly. Hey, I need you to parse this information out and it will write, , write a reg x expression that will parse this information out of this sentence or line or whatnot. I’ve actually found really good use out of using AI to write the regax for me. I don’t really know how to do it all that well. I’ve done a little
3:18 Bit. I know slightly but the syntax is quite I don’t know it’s it’s just complex if you if you write it a lot types of it too. Yeah. Yeah. If you write a lot of it you’re good at it but if you don’t write a lot of it eh I’m not as good. So I really love the I mean just the fact that you can talk to it and say hey I wanted to perform these various options or this various formatting. It will then extract text out really really well for you that way. So that’s fun. custom operations. That’s interesting. , I don’t know if I’ve been able to find any custom operations, but maybe
3:51 Custom operations in there as well. Just ask the co-pilot what you want it to do and maybe it’ll figure it out. And that’s exactly what’s available already in VS Code. And what that allows you to do is say whatever action or operation you want to do, whatever function it’s like, I want to trim, I want to take the third character, I want to take these two columns. Copilot’s going to automatically what it should be doing, what it can do really well. and now it’s simply available in the UI. So I’m I am so happy to see this. this is this just must so needed. It’s just so
4:23 Needed. So the other one here the end of the article that I thought was actually really interesting here is everything in the data wrangler experience is like a pandas data frame. Yes. Whatever data you supply to it or wherever you find a data frame in the main portion of the notebook, it takes you out of that. , if you’re using a PI Spark data frame versus a Pandanda’s data frame, it’s a you’re jumping back and forth between two different styles of data frames. And for those of you who have done some work in notebooks, sometimes
4:55 It’s a bit weird or awkward to write data back and forth from lakehouse tables or read it out because you have to change it over to this format type. And so the data the data and while it’s in memory in the notebook does matter what format that it’s in. It’s not just a standard data frame that works in every instance. There actually is like flavors of a data frame. I guess also importantly when you push it actually back to a delta table. So this is this is actually one of the things I wish I wish I didn’t start with notebooks until today because when I first started this was all a learning curve and I was like okay
5:28 Everything’s good. I have my data types right. Date time great. Push it back to delta table. Oh, it doesn’t work because the panda’s datetime is different than what actually would be good for PI Spark. Yeah, those little things that you’re not aware of, you don’t know. And I had this all this little nice notebook code and all these cells and then the very end saying no, not going to work. Denied it out. Denied. So, this is really great, too, that they’re going to translate that for you as well. I I think this is Mike. This is all stuff that you and I both know is already
6:00 Available in the AI world and it’s just nice to see now in the user interface as just already integrated. Yeah, I think I think this is one of those nice to have features or it just makes it the experience a bit easier, more smoother for users, right? I really shouldn’t I was on a podcast a different podcast recently. I think I was talking to someone about this this exact conversation and I should worry more about the concepts and not necessarily the syntax right the syntax of things is like okay how do I write this data frame specifically for pi
6:32 Spark versus pandas I shouldn’t have to care about that like the at the end of the day I’m just trying to remove some columns filter some data extract information on a rowby row basis like co-pilot should just fix those things for you and we can take away some of those technical challenges and I think we can think of it like a when you work with a piece of wood and and you’re designing something you’re cutting something out of the piece of wood and there’s like these sharp edges on it. Feel like co-pilot right now is really sanding the edges on the chair or the table that you’re making. It’s making the edges a bit
7:04 Smoother and easier to handle instead of having like this really sharp edge on everything. You bring up a conversation I think we may need to say it for an entire episode because if a lot of people are relying on Copilot, they don’t have to worry about the syntax, but they’re owning these these operations. Sure. Well, right. So, to your example with the wood, well, let’s say I just hired someone out to do it. I have no idea what I’m doing, which I don’t usually with anything has to do with wood or carpentry. Yeah. But that’s all getting done and I’m I still own it.
7:37 Well, what if something did break like or are we really at a point right now where AI can do it and I don’t need that technical expertise at an organization or in a team because it’s like you first think well someone know what they’re doing. That’s interesting Tommy. Well, do you remember high school shop class or not even that middle school shop class? Did you go to Did you ever go to shop? They wouldn’t let you in. The first couple things you build are like super shoddy. Like they work, but they don’t work great. , the
8:10 Edges are not quite lined. The the wood pieces of wood don’t go together very well. You overuse the glue when you shouldn’t have used too much glue. And there’s like, , beads of wood glue on everything like the So, I almost think of it like when you’re writing code, it’s the same thing, right? until you learn your way around it. Those first couple of projects are just going to be shoddy, not perfect. That’s okay. I think it’s part of the learning experience. But then over time, , you start learning how to craft better, right? You you upgrade the the skills that you’re
8:43 Working with. You actually start buying more expensive pieces of wood because, , now you’re not just working with pine. You’re you’re now working with like maple or I don’t know, another nice wood. I don’t do wood carpentry so I couldn’t tell you but like it’s it gets better potentially over time. So I think this is maybe the same way here right? So the AI is I I think of it more of less of like I think my sanding of the edges make it make sense here. I think the analogy still holds but I also think it might be like someone else who’s there beside you
9:17 Like the expert that’s there also right cuz you can ask questions of it and say hey is this the right way or I wrote this whole thing in again to this last statement of the blog here I wrote all this in pandas could you just translate this to pi spark so it’s more scalable right right and then it can it can guide you through that experience and say here’s here’s the equivalent of what you’ve written I mean the the amount code being written in pi spark and pandas. There’s there’s an inordinate amount of that stuff. So
9:48 The AI should definitely be able to know what this is doing. And this is this is the stuff that I’m like I really find AI valuable here is because I can ask it to build a component or a function or something and it’s pulling on thousands, hundreds of thousands of functions that are similar in nature and writing one that’s similar. I can’t even I couldn’t even have read all those hundreds of thousands of functions to get to a point where I would understand them all. So anyways, this is really interesting. I like the fact that that AI is coming to data
10:21 Wrangler. I think Tom, I think we we both would admit here data wrangler is an underrated part of the product. Mhm. And the more places we can make it easier to just expose data wrangler in easy to use spaces all across the notebook experience, I think the better. Honestly, 100%. Yeah, I think it brings I think it blends that world of like power query heavy users and then notebook and then developers in notebooks. I think I think it really starts letting that user experience something that feels familiar
10:54 In power query but now can do it in a bit more performant tool like data wrangler and then ultimately move into notebooks which is I think in general faster and costs less cus. completely under running a data flow. Yeah, awesome% great article. The article will also be in the description of this video. So, if you want to check out this article and read it for yourself, there actually is a really nice little YouTube video that goes along with it. Transform your data with AI. This whole article is written by Aaron. I believe Aaron is the PM for this product. Anyways, check
11:26 It out. It’s got a lot of views. This came out a while ago, May 22nd. So, it’s been like out for a couple months, but just now catching up on it. All right, that being said, Tommy, we have a mailbag today. So, let’s jump into our main topic today, talking more about chaos in fabric pipelines and yeah, give us a quick run through of the question for today. This is exciting, too, because this is one of our first mailbags we’ve gotten that’s just fabric focused, fabric, nothing else. So, really excited to see the our our five listeners finally getting fabric
12:00 Question. All the two of them. Exactly. Yes. Exactly. Let’s get into it. My team is co-creating some data pipelines in fabric. We have run into issues with connections where one person will create a connection and then the other will not be able to access the pipeline and receive a warning message that they do not have the permissions for that connection. What is interesting is e that even as a tenant admin I could not find the connection in the manage connection sections to add myself to it. The other user had to go at me. How
12:36 Should connections be managed? Is there a way to only allow certain individuals to create connections? I feel like this is a miss by Microsoft and allows anyone to create connections and is a struggle to manage. Any advice on governing this? This is a great question. Yeah. All right, Tommy. Like let’s let’s start with the let’s start with the observation. Have you also had challenges managing connections in the same way? Someone else can creates a connection, you can’t see it, you can’t get access to it.
13:08 What’s your experience here? Well, the funny thing first off, this is not this is not unique to pipelines. this is something that we’ve also had issues with with data flows gen one where someone created a j a data flow and no one else could edit it. You actually had to overtake the data flow in order to do any modifications to it. So the same idea or same struggle has existed throughout the PowerBI life cycle in the service now pertaining especially to pipelines. Yeah. Yeah. This is something where it’s
13:41 A weird almost feels like a bug. Yeah. I’ I’ve looked at this and I think they they’ve changed how the connection things typically work. I believe the connection it feels like to me observing this is how how it’s changed over the years. Right. Initially connections were made like in the item itself, right? So you’d make a connection inside the item and what would happen is it would immediately check the connection properties like when you logged in. Hey, Tommy’s logging into a pipeline that was written by Mike. Well, well, I guess let’s go back
14:12 To your like data flow gen one stuff as well, right? I’m trying to enter into a data a pipeline that I didn’t have access to, right? So that’s number one problem there. That way all of my connection details could be attached to that particular pipeline. Therefore, we know I think I think that’s why that was happening initially was look you have you’re connecting to SharePoint you’re connecting to a SQL server whatever whatever the database whatever the connections are in that data flow they were owned by a person. So the whole data flow item was a single ownership. Now that you’re in
14:46 Pipelines like we don’t really want to have single ownership on things anymore. we we want, , I want to be able to put the anyone in the workspace who’s like a member to be able to go in and see it and edit it and manage it and things like that as well. So, what happened here, I think, is they disassociated the ability for you to have the connections stored inside the pipeline. And actually, if you think about this from like a continuous integration, continuous deployment space, dev, test, and prod, I mean, there’s there’s likely different connections in each of those environments, which would also require
15:20 Different user permissions potentially or permissions to the server, whatever that is. So, I think I think the I think the right mindset here is to disassociate the connections away from the artifacts in the workspace. The challenge becomes what happens when multiple people need to manage that. Yeah, and my the standard way of governance really around PowerBI and Fabric 2 today, I know they’re working on that individual permissions, but it’s really the workspace permissions. That’s really the de facto default way that we want to manage any permissions around
15:52 Governance, around data ownership, around the modifications or even viewing. And this should be no different for pipelines. if I have editabilities in a workspace, if I am a member or contributor, well changed a bit in our fabric days, but that I should be able to go into any artifact and modify because I don’t need to worry about like for example in a semantic model where I need build permissions if I’m not part of the data set or part of the workspace. But again, so standard governance practices dictate that if I’m
16:25 A edit, if I have the ability to edit in a workspace, then I should be able to go into any artifact and modify it. But that’s not the case here. Yeah. And it feels like I mean, if I if I had to look at what’s going on here, again, I’m trying to like rearticulate this into like maybe a different scenario or something. It feels like the connection itself is becoming its own item. And that item is not stored inside the workspace. The connection is an item that lives in this now connection manager. Right? So the gateway and
16:59 Connection manager seems to be like the place where all connections live. And I I would agree with so a couple things I’ve observed this very much. My observation has been when someone creates a connection and I can’t use it. What happens is I go into the pipeline there’ll be a little notice at the top that says hey you don’t have access to all the connections. Therefore, you can’t run this pipeline. , be aware. Okay, fine. But when I go in there, the connection is just gone. I can’t see the name of it. I can’t see who owns it. I can’t see the name. Like, there’s it’s just a gooid that attaches
17:32 Itself to the pipeline. And I’m like, I understand there’s a permission issue here, but I’m part of the workspace and I’m like, if you I would I could understand like, hey, look, I’m a I’m a viewer. then I would understand like hiding and offiscating the the name of the connection away. But if I’m a member or contributor to the workspace, at least give me the name of the connection, right? Or at least give me a link to go see the connection and see that it’s owned by somebody else in the connection manager, right? So what’s the harm in letting me see the connection
18:06 Without actually putting my permissions or credentials into it, right? So there’s no like request permission button that tells you Tommy that I’m I’m accessing this pipeline. I need to be added to this permission. So I guess my mindset is because the connections are now their own objects. They’re now living in this other page all together. I feel like we need to bring that item the connection back into the workspace and make the like just make a new item. Just make a new artifact in the
18:39 Workspace. That’s like a connection. More items. We need more artifacts in the work. Of course we do. That’s what we’re going to do. We’re going to we’re going to load this sucker up like all the way. I we did not have enough. So, first off, we’re going on that what was that game show that we would play when we were we watch kids. You’d go to like a a store and they’d give you like a shopping cart and you’d have to get as much stuff into the shopping cart as you could. Yeah. Yeah. Yeah. what I’m talking about? Yes. It’s the crazy game where you Yeah. You had like five minutes to get 100 items, whatever. Yeah. Or something like that. or or you just you just here’s a shopping cart go through this whole store and pick out anything you want we’ll buy
19:11 Everything in the shopping cart whatever that is anyways it feels like we’re doing that like just throw everything in the workspace just put it all there and I’m asking for one more thing which is like the connection itself cuz then I feel like you could govern the connection in the workspace two different ways you could govern it by individual users who could access the connection that could be one way you govern the connection or the alternative would be you could govern the connection by right the the workspace management, right? So, obviously all admins would be able to see all connections in the workspace because they’re an admin.
19:43 Makes sense. But then you should be able to set permissions on the on the the connection that says who who do I want to be able to edit or see the details? , only admins, , members, contributors, and definitely not viewers. So, like at least that way, , maybe contributors can see the connection, but they can’t edit it. Maybe members can edit the connection but they can’t they can they can they can’t delete it or something like that and then admins can do it everything right so that would be nice because then you could leverage again the workspace user permissions
20:17 Against the connection and then Tommy when the members of the workspace show up to work on the pipeline okay by default you automatically have access to all these connections that makes to me that makes more sense to bring the connections out of the connections portal page and put it in the workspace does that make sense yeah I’m trying I’m breaking it down in my head and the one thing I feel like you don’t want as much I can see that but you also don’t want that to fly in the face or be a difference than any other artifact in the workspace. So to your point connection mean or your credent
20:49 Credentials right so if I have access to a gateway right to your point that lives in the manage connections and gateway section which is completely different permissions has nothing to do with your workspace permissions or what work lives in right and if that lives there but then all of a sudden you have the artifacts then you’re flying the face it gets it gets complicated very quickly so I guess yeah I think what you’re describing though is sharing connections across workspaces now becomes much more difficult because I’m potentially using a connection from a separate workspace
21:22 That needs to be both and now I’m duplicating them which defeats the purpose because you want one connection for both places. Yeah. Yeah. This is the hard part because and it gets really complicated on how do you actually manage it? That’s a good point because now how the question I think becomes now how do you really want to manage the connections? Are they in fact truly global or are they actually linked or related to the workspace? Like should should they be global or should they be workspace related? Right now they’re just built as global. Right? Anyone who has permissions and email like email address
21:55 Is attached to the work connection they can use them at any workspace anywhere which is nice. I think right but then you also lose like now you have to add I don’t know I just well so it’s hard going from workspace to global and then global back to workspace. Maybe that’s the issue that I’m This is a good point to dive into because let’s break that down from how did we do governance around connections prefabric which I think can stay we can apply those same practices. We didn’t really we well we did if you had a gateway so we had a we
22:28 Had a team that’s true I’ll stop back on my statement especially yeah gateway gateways were the way to manage that connection string right right so with gateways that was really the de facto way to do it because again with data flows the difference was you weren’t managing had nothing to do with connections that was the artifact that you could not manage or go into but connections if I like if I’m in any workspace and I have a connection to let’s say the the e-commerce database, whatever it is, any semantic model that I create in any workspace, if as long as I have the permissions in the
23:00 Connections area to access that, I can connect that semantic model to the gateway and I can access the gateway because of it. So, and that worked to me that was a global way because you’re more than likely going to use connections across workspaces. It’s just it’s just one of those things. you’re going to have your overlapping databases, lakeouses, whatever it may be, is going to more than likely touch more than one workspace. So that makes sense to me. Now, as we wrap
23:32 This in the more complication of fabric, the problem gets into pipelines are very different than semantic models because you’re in the process of modifying or creating something or writing back. So you do want a little extra permissions, too. So I don’t know if you’re seeing that difference there where anything I was doing with the gateway I was just accessing to refresh. I was not modifying. I was not creating. I was only using it to refresh to get the new data. However, if I do this in a
24:05 Pipeline, if I have connections, well then I have the ability if I have access to connections to modify, edit or in a sense delete something that exists. So I really don’t want that to be managed the same. Not always. I think I think if you go into the pipeline you can give people like use access on a connection but not edit access on a connection. So even so I mean to your point Tommy like you’re like one of the things that we had in the gateway which was there were people who were setting up connections on the gateway but there was like the person setting the connection. There’s
24:38 Credentials that were being used. Those credentials could be used in like a service account. So a service account could be attached to them. It’s not specifically linked to a user, but users used the connection, right? Users had the permission to like so what user is allowed to edit the connection, what users allowed to use these connections. And that’s that’s the permissions they had on the gateway. I think they’re mirroring that same experience now inside the portal page with connections, which is there people that can edit it and there people that can use it. So you can use and edit the connection the same way. And I’m just going to I double check that statement just to make sure
25:10 That I have to log in here real quick. So, I’ll I’ll log into Fabric real quick just to confirm that in that gateway connection management manage connections screen. You can have people use connections as well as the ability to edit or manage them as well. but that being said, I I really I do agree with you Tommy. The way we’ve managed these things in the past is different now. And going back to our question from the mailbags because I do want to make sure that we address and answer the specifically the questions. One one is how should
25:42 Connections be managed? And then is there a way to only let certain ini individuals create connections? And I’m going to go back to this one. What have you seen any good practices in this new connection management way? Tommy, have you seen anything that has emerged in your experience that is a good practice that you would like to leverage within connections? Nothing that can to me that can be applied universally. Nothing that I think that you could say you could stamp at any every organization. It has been to this point pretty pretty custom
26:17 Pretty based on that team and their structure because right now to me that managed connections and from an governance point of view it doesn’t have all the bells and whistles and features that you would want that I can apply to any organization with any type of structure. And that’s where I think we’re running into questions like this. So let me go in a couple a couple layers here, right? So how should connections be managed? I I think there’s a difference between users
26:50 Who are just ad hoc making reports and like connecting to SharePoint and bringing files in and maybe even connect to a SQL server. I think there’s a difference between that and I’m going to be building some central IT reporting that’s going to be managed by a larger team. Right? If you’re if you’re a small team of individuals that are building things, I think you manage connections differently than if you’re a larger team because the idea is we’re going to need to hand off permissions to other people potentially. And in larger teams, they fluctuate a little bit. Some people come in, some people go out, right? So, I’m going to look at this two different ways, right? On the smaller
27:22 Side of the teams of things, it would be nice to have coordination between your individuals, but I think you’re going to have a hard time getting everyone to align to the same thing. There’s a bit of again back to this other question. Is there any way to only allow certain individuals to create connections? Not that I’m aware of. I’m not aware of anything that will then refuse the connection creation screen for individual users, which would be nice because then you could really control this and say, look, if we don’t establish the connection and we don’t hand it out to you, then you can’t use it. Well, then that defeats most
27:55 Of the purpose of PowerBI, doing your own things, making your own connections, creating your own data, , , , MRS or whatever you want to call it, right? That defeats the purpose here. So, I don’t I don’t think that’s the right answer. However, what I would say is as your team starts growing, I do think that having security groups is another good place here. We talk about a lot of security groups at the workspace level. There’s other things you may need to have access to the security group. So this is where I think when you’re starting to talk
28:28 About more centrally managed or when people are accessing information. Yeah, it’s more important here to manage things at security group level. Now I what I will admit here is as let’s think about the process here. I’m just going to unpack the process, right? So we make a workspace. It’s going to be a team of people that are going to work in this workspace. So, we already know we’re going to need to build security groups for member, contributor, maybe even admin. Maybe the admin just gets their name on it. I don’t know. We’ll we’ll see. I don’t
29:00 Know how big your organization is. Right. So, in that situation, we’re already creating security groups for different member levels of people. In the connection string space, I can create the connection, but I me as the user, it’s very important for me to attach not just me, but also the security group onto that connection. then anyone who’s a part of the security group can then manage the connection. Right? So to me that’s where I think things should go. Okay, I’ll pause. Here’s what I’m here’s what I’m hearing and I think I’m I’m I’ve really come to this conclusion. A lot of situations
29:33 That we have like this and this is not the only one but let’s be honest to that manage connection string it’s it’s it’s messy. It’s ugly. It it works but it usually works for a single admin. And what I think we’re trying to do here is we’re trying to have a technology solution to a process problem. That to me is really where we’re at. You can do security groups. You can create an app that has some API that connects that has a managed screen connection, but all this is I don’t think this is a technology solution we’re looking for.
30:05 And how do we actually answer this question or more importantly, how do we solve this question or this problem? This is a process thing. And this is really something that the technology alone is not going to solve for you. And this is why you’re dealing with a few things where we don’t have the features available that you would want to manage this in an automated fashion that does not exist. If I’m already working in a team and I’m creating connections to me honestly there should be a process in place to the person who asked this mailbag. If
30:38 I’m working in a team already in a single workspace where I know I’m going to edit something with you. Let’s say you and I are part of the same pipeline. We’re working on the same things. Well, if I’m creating a connection or a pipeline and I’m not making sure that the first thing I do is give you access to that or the team access to that, I think that’s a penalty on me. To be honest, that’s a problem on my end, not the technologies end. We don’t have a techn technological solution or features available to solve this for us. We don’t we don’t. So honestly, this would be
31:11 Something that’s implemented on you and I as the workspace, the contract that you and I write together. It’s like, hey, I’m creating a pipeline. It’s going to go through our normal databases or whatever the strings are, and I know at some point I’m going on vacation, right? So this goes back to like the same things we’ve dealt with before. It’s like, do you have the backtrack? And if I have not given my team or people, my colleagues the ability to edit this, then that’s on me because we know that there’s not a solution there. So, I’m going to I’m going to say that
31:43 I’m going to stop there before I keep going on and I want to hear your thoughts about this because to me, the more I’m looking at this, this is a process solution, not a techn we’re we’re looking at this the wrong way. I I in so in in part of this I agree with you Tommy yes it is part process but I think the way the technology has implemented connections is not done in an easy way to incorporate the process right so I agree with you it’s it’s definitely right I mean as we’re building connections as we’re building
32:15 Things in workspaces again small again this is like the difference between like small projects and small projects I’m probably less interested in managing all these little details right there’s so much to manage It’s up to your point Tommy in smaller projects the process can take over. Did you share it with everyone on the team? Have you gone in and made a connection to something? is there is there a method by which we can like So to that end there is some need there to make it process driven. I’m I’m cool with that. Yeah. But I’m also looking at going where’s the place
32:48 Where this becomes less friction. Right. If I’m making the pipeline, I’m making the connection, I’m doing it inside the pipeline activity, where’s where’s what on creation of the connection, where do I assign who can use it? Where do I assign which users are supposed to be attached to that connection? So I I agree with you Tommy, but I think the technology should enforce good process around management of connections. And so I think right now what I’m saying is right the technology today does not enforce good process to manage the connection. And so we had to rely solely
33:21 On our process to make sure the connection actually is secured the right way. We have the right people access it and it’s in the workspace correctly, which is fine. Again, Microsoft hasn’t built it yet. I’m just I’m think I’m on the side of I’m a bit more complaining about I wish Microsoft would just give us a little bit more features in the creation experience or think about like let me say it this way another way. Okay. So, what what if what if connections were managed globally like they are right now? But what if you could make a shortcut to a connection in
33:54 The workspace? So, and then the shortcut to the connection is basically anyone who’s in this workspace automatically gets permissions to this shortcut or anyone who has access to this shortcut, you can set permissions based on that. And what that does is it just exposes the connection down into the workspace, lets you manage the permissions of it, and then another another option here too, Tommy, is , maybe you don’t want to use the default connection string. And one that was recently introduced was which is Azure Key. So key volt can now hold connections and
34:28 Key volt has a similar effect. You can you can you can also delegate groups and security and teams and people into that as well. And you could set that up initially and then go into having create connections there. But again, it’s adding a lot more complexity. It’s not so much more complex. Way more technical. I’m just saying it’s another alternative out there. So, it’s good that we have options, but I think to your point, Tommy, like it’s there is a process that needs to be in place at a company. Does the technology support it? I would argue right now it’s not a very
35:01 Good support. And the other thing that was mentioned here in the question which is any advice on how to govern this is there any way what was the question here towards the end it was towards the end here is I feel like this is a miss by Microsoft and any creating connections is a struggle to manage any advice on this one I would also look at this going when we expose these connections I feel like we we need to be able to I I feel like as a user, we need more
35:33 Flexibility. Okay, so let’s let’s dive into this because this is what I actually wanted to ask you. If you had this ideal user interface with the options and dropdowns, what would that look like then? Right? So, if we don’t have it now and we’re all complaining that we don’t have it, then what are we looking for? What what do you see on your screen that would manage this for you? Because this is not I think this is a lot harder to envision than we’re saying. So maybe I’m missing something here. Maybe I’m reading between the lines a bit too much. But when I look at this, I
36:06 Go there’s this concept like a workspace identity. Mhm. I think that solves a lot of our problems here. So use any user of the workspace can create. So maybe and maybe I’m thinking about this whole thing the wrong way. Maybe shortcuts to connections isn’t the right way to think about this one. But maybe the idea here is look when we’re in a workspace that is a team based thing maybe the right way of connecting to things is let’s first make an identity at the workspace level and that is the identity that is then being used to
36:39 Manage the connection so that anyone who has access like so the permissions of the workspace like when I’m accessing that connection I’m doing so on behalf of the workspace so it’s like a delegated permission I’m in the workspace I have the right permissions to the identity and therefore that identity allows me to see what I need to see at the connection space. So maybe maybe I’m thinking in the world of again back to the technology part right what’s maybe missing here is workspace identities where workspace identities can be attached to a connection and then anyone in the workspace who’s a member or contributor
37:12 Of the different levels you now have the ability to say you’re at the right permissions level you can then use that identity and therefore you can see all the connection pieces you could update it you could just view it you could just use it so maybe that’s the right approach here is to go more of that direction as opposed to the other way that I’m thinking is you really don’t like when you get bigger solutions when their solution gets more than just one person you want to start removing the credentials of the user away from the data access. You want to start focusing on like app registrations and service principles and like common identities but you still need to secure
37:44 Them. I don’t want to be passing around a client ID and client secret to my team members to use a connection. I want it to be I want it to live securely in the workspace. And this is to me I think streamlining this this is making more and more sense the way connections live in fabric right again because they have the ability to modify create delete and more than just refresh. I’m leaning with you on this where there should be a default setting or really connections should be managed on the workspace level. You could you can have global connections. You can either delegate or set something that way. But by default
38:17 In fabric again because I can do so much. Well, if I’m going to streamline this, workspaces already are such a great security feature. What it entails is more than just a folder or a SharePoint library. We know how secure and how well workspaces work. So, let’s utilize that. To me, I set this up where I have a my workspace. I’m part of a security group to your point where it’s like this. I’m part of the marketing dev and all those workspaces that are marketing dev or whatever that
38:50 May be are part of that access. So all those connections then should have that credentials and that would to me solve it because I’m already part of that workspace. I don’t need to give that connection every time I create a connection to your point. Jim, Karen, Susie and Mike have access. I have to do that every single time I have to create a connection. That would be idiotic. And cannot manage that. you cannot manage the creation of that the deletion or the anything archival but security groups where I’m again I’m creating everything in the marketing development whatever
39:22 You want to call it if as long as the workspace has access to that I can give connections that way we don’t have the ability now to say connections are managed on the workspace level would be great and I think that to me is the way to go I think in the future Microsoft if you’re listening but if we don’t have that then I think a really good way to manage that now is that any workspace that my team’s on is really part of a security or an some type of group. Every workspace that I’m in is part of that
39:54 Group. So whatever again marketing dev security group has access workspace member whatever you want to call it. And if I do need to create a connection all I have to do is just simply add a new security group to it. I don’t have to add 18 people. So the management’s there and I I I think the more we’re looking into this, the more I’m seeing that that security group or the group level setting is the way to go because we don’t have the technology now to manage this on a way that I think we would see ideal and we don’t we really don’t want to do this on a global
40:26 Setting either. So to me, I’m looking at this and I’m seeing if I had a UI for this, it’s really the ability to set security groups to the different connections. I hopefully should be able to see first off all the connections in my workspace which that’s another I think that’s another miss. I would agree here as well like the last question. How do any advice on this week or I guess it’ be more in the middle of the question here is as an admin I want to be able to go into the admin connection portal and say I want to see every connection for all users cuz I’m the admin like I should be
40:59 Able to see all of it. And I agree with you Tommy there. I think 100% that’s a miss in my book. I can’t see all the different connections across my whole organization. I should be able to see all of them and who has access to them. That should be like a given. Especially even if you’re even if you’re a workspace admin, I’m sorry. You should be able to have this access because what’s going to happen and if it hasn’t happened already, it’s like the old days, man. When I remember someone, he said, “I’m I’m moving to Barcelona. His 18 reports gone. We had a we had to build them by looking at them because we
41:34 Couldn’t download all all this stuff which is and the thing I don’t like as much as I I’ve been pushing the process thing is there’s no back there’s no netting right on relying on just the process alone because if someone doesn’t follow that there is no netting for someone who fails to do it and as as much as I would push that so to round this up if I were to say my solution here is you put a as tight-knit of a process you can around anytime a connection’s created
42:06 That has to be part added to part of a work security group the security group that you have already have access to that’s already been set up. Unfortunately with that method is you have to understand too there is no net if you fall if someone fails to do that and we don’t have anything now there really and that’s the thing to important to understand there really is no backdrop for that if if if that doesn’t fall but it’s the best way because what’s the other way is you’re giving too much access to me you’re giving too much security and you don’t know who has access to it
42:39 I mean what we can use APIs to try to look at all the connections but again we know that’s We’re trying to look at it things that are I can pull it out of the box. I mean, I’m even trying to go into the the connections and the API stuff. I have some projects that I’ve done where we have tables of data being pulled out from the APIs. I’m even trying to look at like, okay, what are all the connections that are in my tenant or things that were connecting to, who had the permissions to them? Even that I’m not even sure I can even list as an API call as an admin or service principle. I’m not sure I can even list all the connections out like even get them all
43:11 Out with the user permissions based on the like. So your point Tommy like maybe what what I’m thinking here is it would be nice be able to have here’s all the connections of my tenant here’s all the users of them and you could forcefully go through and say oh I need to make six new connections were added therefore I will automatically add an admin or an admin security group to those connections that way they could at least see everything right today that doesn’t exist at least that I’m aware of or I haven’t dug hard enough to go find it may exist I just don’t know but to your to your point there Tommy though like there’s a missing gap here around
43:43 How do we find all the connections everywhere and as an admin it almost feels like we need like right I’m switching gears thinking thought here is it almost feels like at the workspace level there should be like anformational page like here’s all the things in your workspace and here’s all the connections that are used in this workspace and who has permissions for them then you could go through and just to your point Tommy have a highle glance at here’s all the connections we’re using everywhere and that could be part of your process. Right? So, now we’re incorporating I need more information
44:15 Back out about the workspace that will help me change my process to then make sure, hey, did you add the security group to all these connections? Look, you made six new connections. Those items don’t they’re not attached to a person. They’re only attached to a person, not the security group. And maybe this is where Git will help you out a bit when you make a pipeline and there’s a Git connection. So, you can you can see pipelines see connections like being referenced in there. Maybe you can check that out and go figure out something. It gives you the gooey though. I bet it gives you it
44:47 Probably gives you the gooid. It probably would be great. But that that is not that’s something worth looking into as well. But let’s see if we can wrap up on these like we’re getting closer to time here. Maybe we can wrap up on these final thoughts here and like we’ve talked a lot about what’s going on here. , okay. So, I want to very clearly address how should we manage connections? I think it’s to your point, Tommy, you need to build a process. There’s got to be some internal process that you say in your company, this is how we’re going to
45:18 Manage these things. The technology today, I don’t think gives us the full flexibility of what we would want to do from an administrative standpoint to get everything done. Now, I do think it’s a good practice here to start attaching security groups to things. Mhm. I think that makes sense. You can have security groups at the workspace level. You can make connections. But to your point, Tommy, if you create a new connection, it’s important to note when you’re creating it, making sure that you’re attaching the right permissions to it so everyone can see it because you’re going
45:49 To get locked out. So, that’s that’s just how should you manage it? Build process and try to add security groups wherever you can. And and I think that’s exactly where where I’m leading on this too is if the really person is trying to ask how do I automate this as much as possible but also have the same security that I need. And I think you have a security group that you and your team have access to that’s part of the workspace and any new connection has to be added to that security group. All that being said, there is no backdrop to that as well and it has to be aware to the team that there’s no like oh well if
46:22 We don’t do this there’s another way and unfortunately we have to understand the technology right now the features we have are limited so this isn’t we’re relying on process maybe too heavily but that’s just the way we’re playing right now that’s that’s just the those are the rules of the game but I would completely agree with you if connections are such an issue right now you have to put a processor on the not just the creation but the management of that. The best way to automate that is through adding everyone who can edit is part of that security group. And and to be clear like
46:56 When you’re in the pipeline and you’re clicking on a connection that exists in the pipeline somewhere connecting to an API connecting something there is no place to add like here’s the connection here’s the users that have access to this connection. I would think if you’re going to edit the connection, I think one of the things that would just solve a lot of this, again, this is just a friction point that I have is just add the user permissions at the time of connection creation, right? Let me let me if I’m if I’m creating the connection and you’re going to push me into this global connection space, just let me add the permissions. But it doesn’t do that.
47:31 And also, there’s no like deep links either. Like it would be nice if the connections in the pipeline would deep link you over to the connection that lives inside the ad the the manage connections page. That’s another frustrating. Yeah. Right. So it just find it. So if you’re not going to So I have to go find or go search for it. I have to go find it. Go back to another page. Go search for the name then maybe get it and then say okay who’s the check. To your point, Tommy, it’s the process requires too many steps at this point to make it so you’re overcompensating of things that should that should just be
48:03 Easier to manage. Basically, the most frustrating thing I know we’re getting on time. The most frustrating thing that makes me yell out loud is if I’m putting connections on, let’s say, a semantic model and I accidentally click on the little widget icon to go to manage connections. I lose every single drop down I may have had a connection. That is an audible. Yeah, I’m frustrated. Like what’s going on here? Yeah, agree with that one 100%. So that’s another friction point that I see. And again, this is these are things that we’ve observed when we work with connections. maybe these get
48:36 Better over time in the future. you do have workspace identities that live today. So that that is something that does exist today. you can get out those work, but again, can you add them to the connections yet? I haven’t done a lot of testing around that yet. So that’s that’s an area where you we’ll need to spend more time researching that to to get to the to that answer a bit more. Okay, that was the first part of the question. The second part of the question here was is there a way to only allow certain individuals create connections? The answer that I’m aware of is no. No, everyone can create connections if you have a powerbi.com access if you’re a pro user, right? You
49:10 Can create connections and make content. Now a viewer of a workspace can’t make content. Can’t make connections. So viewers of workspaces I believe is the only way you would be able to block someone from making connections. You couldn’t publish the workspace. That’s the only way, right? Cuz because everything here hinges on creating artifacts in either fabric or into the workspace. So the only way I can think of to make sure that no individuals can make their own connections again. So the caveat here is you can create things in
49:45 The workspace, but you’re not managing connections, right? That’s the trade-off. You’re either going to get all or nothing. You could be a viewer on a workspace, which means you can see the content there, but now you can’t create anything. And therefore, by default, you wouldn’t be able to create connections, but that doesn’t exclude you from going to my workspace and creating connections there, unless you turn that off, too. So, there’s a lot of things you think you have to turn off to reduce the connections, but in doing so, you actually hinder anyone’s ability to actually build anything in PowerBI, which is also not what you want. Right. Right. Yeah. Because that’s actually you
50:18 Brought up a good point where if I can’t if I can view well I can’t use them I can view it I can’t use it correct so correct nor could you create anything in the workspace just it doesn’t work so yeah I I don’t think there’s there’s the short answer to this one is no you can’t restrict that there’s no additional permissions of like con connection creator privileges like there there’s none of that because it it basically def I mean my opinion here it defeats everything you’re doing in the workspace pretty much% let me let me take this one step further around this connection no
50:52 Creating connections things. So this is what we’ve been focusing is let me be the prouser in creating connections in workspaces. If you abstract that layer one level deeper and say if your users are only creating reports, copying reports, building patching reports or building PowerBI reports, right? If you’re just if you’re just Yeah, I know where you’re going with this. Creating existing content and copying it. The only other way I know of how doing this is now you can man you could create semantic models
51:25 For people to play with and then you can use those semantic models in embedding. That’s the only thing I cuz then cuz then you remove all the experience at powerbay.com away from them and you’re purely giving them a here’s the semantic model, here’s the report, here’s the pageionate report. you’re only giving them create report experiences. And when you do that, it strips away the ability for them to create anything that’s fabric. It strips away the ability for for create any connections. So if you So let me I’m going to I’m going to say okay, I’m going to change my answer.
51:56 Maybe, right? The answer is maybe now with lots of effort. Right? So if you if you are giving access to the directly to the workspace, the answer is no because you can’t when as soon as you turn off that ability to create connections, you turn off everything everything’s ability. Yeah. If you’re using PowerBI embedding now the answer is you can create catered experiences that allow users to create reporting objects and table exports but they cannot create pipelines and do data engineering. So the the data so I guess it’s there the break line for me now is if you’re
52:28 Creating report level content yes but if you’re creating pipeline data engineering level content no because you you just can’t do that. Yeah that’s yes with a lot of caveats. That’s the same way as like can I make a bolognese? Well, yes, but you do need a pig from Osci. You do need tomatoes. You got to go to you got to go to Naples. So, but yeah, you can do it. You can do it. Yeah. I And I I agree. I think there is an discussion here where if you just you don’t have to worry about the view permissions on a workspace or really permissions at all on a workspace if
53:02 We’re dealing with the semantic model level because then I can do that on the semantic model. Yes. which is is that the only artifact that you can we can do that with if I’m thinking off the top of my head where I can change the permissions where I can build or just view and I think this is semantic model the only one I can actually do that right now because if I have a pipeline I can’t give someone view permissions right I’m just trying to think right now yeah I think with notebooks we can do that too you can do the developer the editor or the you can run it notebooks can be viewed inside an or app as well. So
53:37 There is there is this concept of like viewing a notebook but not editing the notebook. Artifact I can give permissions. Yeah. I I believe when I look at the the artifacts that are inside the PowerB workspace there there needs to be two levels of per like permissioning on all the items, right? Is there a view level version of this or is there like a edit level of permissioning? Right. So so there seems like there needs to be two layers of what’s going on for permission for any model. Yeah. Yes, correct. Exactly right. So like if if you come in to the item and you’re the item must obey the
54:12 Workspace level of that user and then the level be below it. Yeah. Okay. So I think those are the two main questions that were asked here. Can it be can the connections be managed? and then what’s the best practices for manage connections? And then can you allow only certain individuals create connections? Not really. Not without a lot of effort. Let’s just say it that way. not without a lot of effort to do to build something that would make it you’re managing the connections for people. And the last one, any advice on this? This is the whole episode’s been roughly our advice and our pain points around this whole experience. So hopefully Microsoft
54:45 You’re listening and maybe we can, , reach out. We’re happy to talk about connections with you about what our experience is and how does this work and I’m not sure if we’ve given been given a lot of vision for like where connections are going to go and how can we more easily manage them centrally but man I it’s a big to me it’s a big miss that there is no admin see all connections page feels like it needs to exist I can hear the pain of Microsoft engineers if they’re listening to this like geez this is an issue for them too goodness because I mean Here’s the thing though.
55:18 We’ve introduced so many cool features, but there are so many of these. We added these with just PowerBI, these gotchas, these little things that you would never know. Paper cuts. Mhm. Little tiny things that just annoy you a little bit. Exactly. So, agreed. All right. That being said, thank you very much for listening to us for an hour yammer about connections. Who thought that would be such an engaging topic? , but there’s a lot to think about here. these connections and as your organization goes from a small organization to a larger one, you’re going to want to think or unpack your process on how you manage these. How do you handle this
55:50 Between different developers and even if you’re going from one developer to two, you’re going you’re going to run into this problem. Like so even just even at a small team, you’re going to to bump into this challenge. So at least something you need to think about. That being said, thank you so much for listening. Please, if you if you like this episode, if you found this useful to you, share it with somebody else. We’d love to to see it out. share it on social media. We will definitely jump in and give you a good thumbs up for that if you do also share it. Tommy, where else can you find the podcast? You can find us on Apple, Spotify, wherever you get your podcast, make sure to subscribe and leave a rating. It helps
56:22 Us out a ton. And please share with a friend. We do this for free. If you have a question, idea, or topic that you want us to talk about on future episode, well, head over to powerbi.tips/mpodcast. Leave your name and a great question like today. And finally, join us live every Tuesday and Thursday, 7:30 a.m. Central, and join the conversation on all of PowerBI. Tips social media channels. Thank you all so much, and we’ll see you next time. Heat. Heat.
Thank You
Want to catch us live? Join every Tuesday and Thursday at 7:30 AM Central on YouTube and LinkedIn.
Got a question? Head to powerbi.tips/empodcast and submit your topic ideas.
Listen on Spotify, Apple Podcasts, or wherever you get your podcasts.