Knowledge

When teams first adopt Databricks, they typically start with a "yes to everything" approach. Data scientists get full access to explore, experiment, and innovate. This works well&mdash; until it doesn't. Three of the most common scenarios we see:
<ul>
<li>Data usage becomes untraceable</li>
<li>Compliance requirements&nbsp;get overlooked</li>
<li>Security policies&nbsp;become hard to enforce</li>
</ul>
Even with the Unity Catalog, organizations struggle to answer basic questions.. Who is using what data? Why are they using it (for which project?) Are they following our organisation's compliance rules?
There is a real cost to governance failures, it's not only about a box left unchecked. It's seen in valuable time spent tracking down data usage, in the increased risk of violations that can trigger serious penalties, in&nbsp; security vulnerabilities and, maybe most frustratingly for those involved, in duplicate (or often triplicate, quadruplicate or more) work across teams.
<h2>Building Effective Governance in Databricks</h2>
The first impulse is to fully restrict access, and this would be the easiest for one side - the compliance side. But it's not feasible. Effective governance is that which effectively supports the business, not hinders it. So we need to try to make governance part of the natural workflow of companies. And there are multiple ways in which we can adapt&nbsp; Databricks governance strategies to make them work for both sides.
<h3>1. Clear Data Ownership in Databricks</h3>
Data ownership often becomes unclear as teams grow and projects multiply. So successful organizations address this by establishing clear ownership patterns early on. They define not just who owns each dataset, but also how that ownership translates into practical responsibilities.
For example, we can assign dataset owners based on business domains rather than technical teams. These owners become responsible for defining usage policies and approving access requests. More importantly, they track how their data is being used across different projects, ensuring compliance with regulatory requirements while maintaining team agility.
<h3>2. Self-Service Governance in Databricks</h3>
Traditional access management often creates bottlenecks, with IT teams becoming overwhelmed by access requests. Modern governance approaches attempt to solve this by looking at embedding access controls into the daily workflow of data teams.
A practical implementation could look like this:
<ul>
<li>Structure data access per data product output</li>
<li>Self-service interface for access requests</li>
<li>Automate workflows routing requests to appropriate approvers</li>
<li>Regular data access reviews</li>
</ul>
This approach typically reduces access request processing time from months or weeks to hour or minutes, while again - maintaining strict compliance standards.
<h3>3. Automated compliance tracking in Databricks</h3>
Manual compliance tracking doesn't scale. No manual process ever does. So organizations successful with Databricks governance look at automating compliance processes. Reducing the need for manual input drastically reduces friction between teams and can improve the adoption of any process. Automating in this case can mean:
<ul>
<li>Automatically logging all&nbsp;data access requests</li>
<li>Tracking data lineage across all projects</li>
<li>Generating compliance reports automatically</li>
<li>Setting up automated alerts for potential policy violations</li>
</ul>
We could imagine how implementing these can lead to significantly less time spent on routine checks and to increased focus on strategic governance improvements.
<h2>Having Our Cake And Eating It Too : Innovation PLUS Governance</h2>
It's worth repeating the key to successful governance here - it's&nbsp; finding the right balance between control and flexibility. This flexibility is vital to maintain innvation and ensure a healthy growth. Organizations that get this right understand that they need to give data scientists freedom but also that they need to do it within defined boundaries. This can be achieved by looking at automating routine governance tracks, effectively making compliance the path of least resistance, and enabling quick access to approved data sources, which is what data scientists often struggle the most with.
We often talk to Data Leaders that struggle with effective governance and understand that things aren't on the right track but aren't sure where to look at. So it's important to first clearly understand the specifics of our current setup. Do we know who's using our critical data? Can teams easily request and get access to data? Are we meeting compliance requirements without manual work?
If we find ourselves answering "no" to any of these, it might be time to rethink our approach to Databricks governance.
Start with an honest mapping of the data landscape to identify critical datasets, document current usage patterns and list compliance requirements. Once there's a clear and shared understanding of the actual data landscape and the requirements of all business functions, move to implementing self-service governance:
<ul>
<li>Teams request access through a simple interface</li>
<li>Approvals follow&nbsp;predefined rules</li>
<li>Usage&nbsp;and requests are tracked automatically</li>
</ul>
Data scientists get to keep their flexibility, compliance teams get their oversight and everyone saves time.
<h2>The Path Forward: Bring Self-Service Governance To Databricks</h2>
We've seen this scenario play out often at customers so we've built a tool to help solve these problems. The Data Product Portal is a fully open-source project and free to use. It lets you continue using your preferred data science and analytics tools like Databricks, while offering self-service automated governance. It's best suited for data teams within businesses that aim to build their own use cases in a self-service way, embracing data product thinking.
So imagine this: two teams, each with four data scientists. Team A is building a demand forecasting model and Team B is developing a pricing optimization system.
Team B needs Team A's forecast outputs. Their options are:
<ul>
<li>Request&nbsp;access to Team A's entire&nbsp;workspace (security risk)</li>
<li>Have Team A manually&nbsp;export and share results (time-consuming)</li>
<li>Create duplicate data pipelines (inefficient and error-prone)</li>
</ul>
They risk spending more time solving access issues than improving their models.
When using Portal, the organisation enables Team B to access only the specific forecast outputs they need, see the data lineage (where the data comes from) and get automated access to updates. One of the biggest benefits to working within a data product framework is that it makes accessing the latest version of the data product output the default modus operandi. Meanwhile, Portal empowers Team A to maintain control of their workspace while sharing only the necessary outputs, track who uses their data and automate access management.
In a nutshell, the key point we want to make is that effective Databricks governance doesn't mean choosing between innovation and control. It means making governance simple enough that teams want to follow it by automating the boring parts and reducing friction through self-service capabilities.
Check out <a href="https://github.com/conveyordata/data-product-portal">the project's repo</a> for more details, reach out to us anytime on our <a href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2r47r0icj-PAYbbklmH6gPz9A6bJ144Q">Slack channnel</a> or book <a href="../../../../book-demo" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiJhcGk6OnBhZ2UucGFnZSIsInRhcmdldENvbnRlbnRUeXBlSWQiOjUzLCJ1cmwiOiJodHRwczovL2NvbnZleW9yZGF0YS5jb20vYm9vay1kZW1vIiwidGV4dCI6ImEgZ3VpZGVkIGRlbW8iLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiJodHRwczovL2NvbnZleW9yZGF0YS5jb20iLCJ1cmxBZGRpdGlvbiI6IiJ9">a guided demo</a>,&nbsp; we'd love to show you around.

Data scientists love Databricks for its power and flexibility. But as organizations scale their Databricks implementations, a pattern emerges: the very freedom that makes Databricks powerful can become its biggest governance challenge.

The Hidden Cost of Databricks Freedom: Balancing Innovation with Governance

As a platform engineer, you will succeed when offering more than "a bag of tools" to business users. Offer technology-independent UIs that suit the particular needs of your company.

Give Your Business Users A Friendly Interface To Your Data Platform

The Data Product Portal is an open-source tool to help organisations build and manage data products at scale. It’s intuitive, flexible, and geared towards making data product management straightforward and effective.

How Portal Integrates With Data Platforms

Latest Blogs

imec is the world’s leading independent nanoelectronics R&D hub and the home of more than 5,500 people. As data is core to good research, imec had the ambition to roll-out a broad range of data & analytics capabilities company-wide.

imec

Latest Cases

Learn how Conveyor provides you with all the tools needed as a data engineer and how your development gets accelerated with templates for the most popular frameworks. It minimizes the time setting up infrastructure and maximizes the time available to create value from data.

How to use Conveyor as a data engineer

The multitude of tools designed to solve specific data challenges can really be hard to set up and maintain and is a nightmare for your end user. In this video, we will explain how Conveyor overcomes this challenge.

How Conveyor fits in your data landscape

Data is created in all corners of your organization and there is no longer a single person that knows all data available within your company. Learn how Conveyor addresses this challenge with the introduction of project-based access management for data.