Blogs

When teams first adopt Databricks, they typically start with a "yes to everything" approach. Data scientists get full access to explore, experiment, and innovate. This works well&mdash; until it doesn't. Three of the most common scenarios we see:
<ul>
<li>Data usage becomes untraceable</li>
<li>Compliance requirements&nbsp;get overlooked</li>
<li>Security policies&nbsp;become hard to enforce</li>
</ul>
Even with the Unity Catalog, organizations struggle to answer basic questions.. Who is using what data? Why are they using it (for which project?) Are they following our organisation's compliance rules?
There is a real cost to governance failures, it's not only about a box left unchecked. It's seen in valuable time spent tracking down data usage, in the increased risk of violations that can trigger serious penalties, in&nbsp; security vulnerabilities and, maybe most frustratingly for those involved, in duplicate (or often triplicate, quadruplicate or more) work across teams.
<h2>Building Effective Governance in Databricks</h2>
The first impulse is to fully restrict access, and this would be the easiest for one side - the compliance side. But it's not feasible. Effective governance is that which effectively supports the business, not hinders it. So we need to try to make governance part of the natural workflow of companies. And there are multiple ways in which we can adapt&nbsp; Databricks governance strategies to make them work for both sides.
<h3>1. Clear Data Ownership in Databricks</h3>
Data ownership often becomes unclear as teams grow and projects multiply. So successful organizations address this by establishing clear ownership patterns early on. They define not just who owns each dataset, but also how that ownership translates into practical responsibilities.
For example, we can assign dataset owners based on business domains rather than technical teams. These owners become responsible for defining usage policies and approving access requests. More importantly, they track how their data is being used across different projects, ensuring compliance with regulatory requirements while maintaining team agility.
<h3>2. Self-Service Governance in Databricks</h3>
Traditional access management often creates bottlenecks, with IT teams becoming overwhelmed by access requests. Modern governance approaches attempt to solve this by looking at embedding access controls into the daily workflow of data teams.
A practical implementation could look like this:
<ul>
<li>Structure data access per data product output</li>
<li>Self-service interface for access requests</li>
<li>Automate workflows routing requests to appropriate approvers</li>
<li>Regular data access reviews</li>
</ul>
This approach typically reduces access request processing time from months or weeks to hour or minutes, while again - maintaining strict compliance standards.
<h3>3. Automated compliance tracking in Databricks</h3>
Manual compliance tracking doesn't scale. No manual process ever does. So organizations successful with Databricks governance look at automating compliance processes. Reducing the need for manual input drastically reduces friction between teams and can improve the adoption of any process. Automating in this case can mean:
<ul>
<li>Automatically logging all&nbsp;data access requests</li>
<li>Tracking data lineage across all projects</li>
<li>Generating compliance reports automatically</li>
<li>Setting up automated alerts for potential policy violations</li>
</ul>
We could imagine how implementing these can lead to significantly less time spent on routine checks and to increased focus on strategic governance improvements.
<h2>Having Our Cake And Eating It Too : Innovation PLUS Governance</h2>
It's worth repeating the key to successful governance here - it's&nbsp; finding the right balance between control and flexibility. This flexibility is vital to maintain innvation and ensure a healthy growth. Organizations that get this right understand that they need to give data scientists freedom but also that they need to do it within defined boundaries. This can be achieved by looking at automating routine governance tracks, effectively making compliance the path of least resistance, and enabling quick access to approved data sources, which is what data scientists often struggle the most with.
We often talk to Data Leaders that struggle with effective governance and understand that things aren't on the right track but aren't sure where to look at. So it's important to first clearly understand the specifics of our current setup. Do we know who's using our critical data? Can teams easily request and get access to data? Are we meeting compliance requirements without manual work?
If we find ourselves answering "no" to any of these, it might be time to rethink our approach to Databricks governance.
Start with an honest mapping of the data landscape to identify critical datasets, document current usage patterns and list compliance requirements. Once there's a clear and shared understanding of the actual data landscape and the requirements of all business functions, move to implementing self-service governance:
<ul>
<li>Teams request access through a simple interface</li>
<li>Approvals follow&nbsp;predefined rules</li>
<li>Usage&nbsp;and requests are tracked automatically</li>
</ul>
Data scientists get to keep their flexibility, compliance teams get their oversight and everyone saves time.
<h2>The Path Forward: Bring Self-Service Governance To Databricks</h2>
We've seen this scenario play out often at customers so we've built a tool to help solve these problems. The Data Product Portal is a fully open-source project and free to use. It lets you continue using your preferred data science and analytics tools like Databricks, while offering self-service automated governance. It's best suited for data teams within businesses that aim to build their own use cases in a self-service way, embracing data product thinking.
So imagine this: two teams, each with four data scientists. Team A is building a demand forecasting model and Team B is developing a pricing optimization system.
Team B needs Team A's forecast outputs. Their options are:
<ul>
<li>Request&nbsp;access to Team A's entire&nbsp;workspace (security risk)</li>
<li>Have Team A manually&nbsp;export and share results (time-consuming)</li>
<li>Create duplicate data pipelines (inefficient and error-prone)</li>
</ul>
They risk spending more time solving access issues than improving their models.
When using Portal, the organisation enables Team B to access only the specific forecast outputs they need, see the data lineage (where the data comes from) and get automated access to updates. One of the biggest benefits to working within a data product framework is that it makes accessing the latest version of the data product output the default modus operandi. Meanwhile, Portal empowers Team A to maintain control of their workspace while sharing only the necessary outputs, track who uses their data and automate access management.
In a nutshell, the key point we want to make is that effective Databricks governance doesn't mean choosing between innovation and control. It means making governance simple enough that teams want to follow it by automating the boring parts and reducing friction through self-service capabilities.
Check out <a href="https://github.com/conveyordata/data-product-portal">the project's repo</a> for more details, reach out to us anytime on our <a href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2r47r0icj-PAYbbklmH6gPz9A6bJ144Q">Slack channnel</a> or book <a href="../../../../book-demo" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiJhcGk6OnBhZ2UucGFnZSIsInRhcmdldENvbnRlbnRUeXBlSWQiOjUzLCJ1cmwiOiJodHRwczovL2NvbnZleW9yZGF0YS5jb20vYm9vay1kZW1vIiwidGV4dCI6ImEgZ3VpZGVkIGRlbW8iLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiJodHRwczovL2NvbnZleW9yZGF0YS5jb20iLCJ1cmxBZGRpdGlvbiI6IiJ9">a guided demo</a>,&nbsp; we'd love to show you around.

Data scientists love Databricks for its power and flexibility. But as organizations scale their Databricks implementations, a pattern emerges: the very freedom that makes Databricks powerful can become its biggest governance challenge.

The Hidden Cost of Databricks Freedom: Balancing Innovation with Governance

As a platform engineer, you will succeed when offering more than "a bag of tools" to business users. Offer technology-independent UIs that suit the particular needs of your company.

Give Your Business Users A Friendly Interface To Your Data Platform

The Data Product Portal is an open-source tool to help organisations build and manage data products at scale. It’s intuitive, flexible, and geared towards making data product management straightforward and effective.

How Portal Integrates With Data Platforms

The more people you add to a team, the more lines of communication you introduce. The same holds for the number of tools in your data landscape

How To Conquer The Complexity Of The Modern Data Stack

A new category of tools entered the data scene: data product catalogs. We believe such a tool is useful not only when it provides an overview, but also empowers users to navigate their data landscape.

The Missing Piece To Data Democratization Is Not A Catalog

The Missing Piece to Democratize Data Is Not A Catalog

Traditional medallion architecture is showing its age. It wasn't designed for the agility and self-service needs of modern data teams. So, what lies beyond medallion?

Beyond Medallion: New Data Architectures For The Age Of Data Products

In the fast-evolving world of data, companies are discovering that the key to success for scaling their data initiatives is not to rely on a single data team to handle all requests from the business. Instead they want to enable self-service capabilities to allow each domain or department to build their own data products at their own pace with their own budgets.
However, putting this vision into practice can be very challenging, especially when you put all aspects related to data governance, data platforms and data catalogs&nbsp;together. It can become messy very quickly as it is hard to provide an easy to understand consistent view across all these dimensions and technologies.
Today we are very happy to announce the&nbsp;Data Product Portal&nbsp;&mdash; an open-source tool to help organisations build and manage data products at scale. It&rsquo;s intuitive, flexible, and geared towards making data product management straightforward and effective.
<h1 id="9533" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">What problem does it solve</h1>
Imagine you&rsquo;re building a data pipeline. You take some input data, process it with python, DBT or another tool and generate output for others to use. This scenario applies whether you&rsquo;re using Starburst with Ranger for access controls, or other platforms technologies like Snowflake, AWS, Databricks, BigQuery or Microsoft Fabric.
<img src="https://media.prod.dataminded-dev.nl/1_Noue_Lmz_Vd_L_9n_Vp_Vr6h_UVA_6ec23f20b3.webp" alt="1_NoueLmzVdL_9nVpVr6hUVA.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Typical data pipeline consuming remote data, data from files and databases</figcaption>
</figure>
&nbsp;
In large organisations, you typically have multiple departments or domains, and you can&rsquo;t simply share all data between all departments by default due to legal, compliance, regulatory or confidentiality reasons. This means that departments or domains have to keep control how and why their data is being used by others.
<img src="https://media.prod.dataminded-dev.nl/1_K_Mqruo_H_Tid_Stx_Ky_K_Vr_AHNQ_2dd73e0f41.webp" alt="1_KMqruoHTidStxKyKVrAHNQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Scaling data pipelines across a large organisation with multiple departments</figcaption>
</figure>
&nbsp;
The moment you start sharing data across multiple departments or domains, you are immediately faced with the following questions:
&nbsp;
<blockquote>
Who has access to what data, when can they use it and for what reason are they using it?
</blockquote>
If you are working with data, you want to know where to access and find the data and tooling to build your data pipelines.
To answer these questions, companies start data governance initiatives, where they start generating policies per user to manage access to data in tools like Ranger, AWS IAM, Snowflake or any other tool. When people have been working on multiple use cases, all of these policies accumulate together. This &ldquo;spaghetti&rdquo; of permissions will lead towards very broad access permissions for a large group of people. After a while you will end up in the situation you wanted to avoid in the first place where:
&nbsp;
<blockquote>
Everybody ends up with access to all data and you no longer know why and how your data is being used.
</blockquote>
<h1 id="a357" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm pe no np nq pf ns nt nu pg nw nx ny bj" data-selectable-paragraph="">Data products as a governance model</h1>
There are many different definitions out there about what is a data product and every organisation or person working with data has their own opinion about it, but we found the following definition to be useful and quite universal for many organisational structures.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_ji_BF_p71h_Wo_YE_5_Mki_XG_Ntg_0bd0979863.webp" alt="1_jiBF_p71hWoYE5MkiXGNtg.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Definition of a data product</figcaption>
</figure>
&nbsp;
We propose to define a&nbsp;data product&nbsp;as: an initiative with a clear goal, owned by a department or domain of the business that consists of the combination of:
<ol class="">
<li id="8b22" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Input data: Access to datasets created by a combination of other data products.</li>
<li id="47a8" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Output data: Read/write access to output data that can be combined into a dataset that can be shared with other data products. This data is stored in specific locations (e.g. databases or buckets).</li>
<li id="1ea3" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Private data: A safe location to store private/internal data for local processing with no intention of sharing</li>
<li id="1f27" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Transformation artefacts: All code and outputs describing your transformations, scheduling and tooling configuration needed to build, access and run your data pipeline separated from other data products.</li>
<li id="5481" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Team roles: Defined roles of team members that have specific permissions on how to interact with the data product (e.g. data product owner, data engineer, business analyst).</li>
</ol>
&nbsp;
<blockquote class="op">
It is important to note that people can work on multiple data products, but have to choose on what data product they are working on. They only get access to that data product scope related to tooling and data.
</blockquote>
In this definition, data products are not only data assets that are the output for sharing with others, but also the tooling, artefacts and roles of people that interact with that data product.
When multiple data products start interacting, your data product lineage will start looking like this:
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_p_Vu_Poa_NU_Ja_Ckgn9_i31mw_6224405452.webp" alt="1_pVu-PoaNUJaCkgn9-i31mw.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">How different data products are interacting with each other</figcaption>
</figure>
&nbsp;
The main benefits of this approach are:
<ul class="">
<li id="39a5" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz pu pi pj bj" data-selectable-paragraph="">Clear Data Usage: You always know who is using your data and why.</li>
<li id="10c1" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Simplified Access Management: Easier to handle access requests and data revocations</li>
<li id="34ed" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Natural Data Lineage:&nbsp;Understand how your data flows from one data product to another.</li>
</ul>
Adopting the data product governance model if very powerful in scaling data initiatives across departments, while still keeping control and self-service capabilities. However, effective tools are essential for managing the processes around this to make it truly self-service.
<h1 id="fbcc" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">Introducing the Data Product Portal</h1>
The&nbsp;Data Product Portal&nbsp;is a practical tool that helps you build data products at scale, that is both useful for both people working with data and people who want to have control over how their data is being used.
It is designed to simplify the creation and management of data products at scale. It is beneficial and easy to use for both data professionals and those overseeing the use of data and data governance.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_Ph_T_Gak_S_Vpw_A8_Oi5_P_Ki13_YQ_56d99d19ee.webp" alt="1_PhTGakSVpwA8Oi5PKi13YQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Data product model translation to policies and configuration</figcaption>
</figure>
&nbsp;
This is where the Data Product Portal comes into play:
<ol class="">
<li id="706e" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Guided Setup: Step-by-step assistance involving the right stakeholders for creating data products, requesting access, adding users and registering new data for sharing with other data products.</li>
<li id="57cf" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Tech Translation: Converts high-level concepts into specific configurations settings for platforms like AWS, Azure, Databricks, Snowflake, and others, making sure that each data product is correctly separated and not impacting each other.</li>
<li id="79df" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">User-Friendly Interface: Makes it easy for business users and people working with data to understand and navigate the data landscape.</li>
<li id="174b" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Self-service:&nbsp;Enables departments and teams to start new data initiatives easily without having to depend on a central team.</li>
<li id="5061" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Comprehensive Overview: Combines data catalogs, data platforms and data governance aspects into a single 360 overview of all ongoing data initiatives.</li>
</ol>
<h1 id="7eb9" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">How to get started</h1>
The Data Product Portal is available as an open source project on&nbsp;<a class="af pv" href="https://github.com/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">Github</a>. Getting started is as simple as running&nbsp;docker compose up&nbsp;and visiting&nbsp;localhost:8080
We invite you to check us out and give us a star if you like what you see. Your contributions are invaluable to us &mdash; whether it&rsquo;s through feedback, suggestions, or direct involvement in development.
For Kubernetes deployments, check out our helm chart&nbsp;<a class="af pv" href="https://gallery.ecr.aws/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">here</a>.
If you have questions or want to share your thoughts? Join our&nbsp;<a class="af pv" href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2lo245t3l-bJxcAz1trqrcQNnGARWMDg" target="_blank" rel="noopener ugc nofollow">community on Slack</a>&nbsp;and connect with us directly. We can&rsquo;t wait to hear from you!
We are excited to bring the Data Product Portal into your hands as an open source initiative. Don&rsquo;t hesitate to share with us what you think about it!
&nbsp;

Scale data initiatives across your entire organisation while enabling self-service and maintaining governance.

Introducing Data Product Portal

Introducing The Data Product Portal: A New Open-Source Tool For Data Governance

<blockquote>
A data platform is about enabling an organization to work with data at scale, in a self-service manner.
</blockquote>
That is how Kristof Martens, tech lead at multiple of <a href="https://www.dataminded.com/" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5kYXRhbWluZGVkLmNvbS8iLCJ0ZXh0IjoiRGF0YW1pbmRlZCdzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">Dataminded's</a> customers, describes a data platform. That is also what Wim Vancuyck, data lead at <a href="https://www.imec-int.com/en" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5pbWVjLWludC5jb20vZW4iLCJ0ZXh0IjoiaW1lYyIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">imec</a>, envisions in his quest to enable more than 5000 researchers to work with data at imec, the world&rsquo;s leading independent nano-electronics R&amp;D hub. We recently sat together and discussed why companies should actually not build their own internal data platform. These are our main conclusions.
<h1 class="pw-post-body-paragraph mz na fv nb b gt nc nd ne gw nf ng nh ni nj nk nl nm nn no np nq nr ns nt my fo bj">The impact of company culture</h1>
Let&rsquo;s start with explaining the different factions that we typically see while deciding on a data platform: business, IT and governance. Typically business teams want to go fast and have something available yesterday. That&rsquo;s why they mostly lean towards buying off the shelf tooling: click install and off you go.
IT departments on the other hand focus on standardization and development best-practices, and tend to lean towards building a solution, as it offers you more flexibility and control. And let&rsquo;s be honest: building a data platform often looks easy at the start. The third party, the governance team, has a more narrow view on a data platform: a data catalog will solve all problems. In the end, it&rsquo;s the faction with most power in the company that influences its buy vs build mentality.
Funny enough, at imec it&rsquo;s the other way around: the R&amp;D department exists of people that are used to coding, which led to building and rebuilding quite often. The IT department on the contrary is small compared to the amount of researchers, loves to work with partners and buy tools to scale fast and introduce standardization. It&rsquo;s all about avoiding re-inventing the wheel over and over again by all those researchers.
This buy mentality, combined with the desire to keep costs under control, the love for open source, and the preference to avoid vendor lock-in, has led to a modular approach to the data platform at imec. This data platform consists of multiple&nbsp;<a class="af nu" href="https://azure.microsoft.com/en-us" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2F6dXJlLm1pY3Jvc29mdC5jb20vZW4tdXMiLCJ0ZXh0IjoiQXp1cmUiLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiIiLCJ1cmxBZGRpdGlvbiI6IiJ9">Azure</a>&nbsp;components, combined with tools like&nbsp;<a class="af nu" href="https://www.mathworks.com/products/matlab.html" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5tYXRod29ya3MuY29tL3Byb2R1Y3RzL21hdGxhYi5odG1sIiwidGV4dCI6Ik1hdGxhYiIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">Matlab</a>, or Conveyor. Conveyor can be considered as the cornerstone, enabling multiple open source technologies as a managed services and properly glued together. In theory, this modular approach allows imec to migrate the platform to other cloud providers, again mitigating a vendor lock-in.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">The pitfalls of building your own data platform</h1>
&ldquo;We can build this ourselves!&rdquo; A common heard argument to build your own data platform, and probably the authors are right. The question is not whether you can build your own data platform, but whether you want or should do so.&nbsp;At imec they have the guideline
<blockquote>
Buy where you can, build where you want to make the difference.
</blockquote>
Often the platform is not where you make the difference: what you use it for is the value add.
Remember that a data platform is about enabling an organization to work with data at scale in a self-service manner. This raises a lot of questions, like:
<ul class="">
<li id="9c4d" class="mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my qa qb qc bj" data-selectable-paragraph="">How will I provide proper separation between teams and projects?</li>
<li id="fe53" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How can I encourgag best-practices?</li>
<li id="8dd7" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I share data?</li>
<li id="3365" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I keep track of governance and compliance concerns?</li>
<li id="b9e1" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I implement a data product lifecycle?</li>
</ul>
It is hard to come up with an answer to all these questions beforehand. If you want to have all of this prior to a first use-case, this can easily take some years. Glueing tested and proven components together, as imec does, speeds up this process drastically. After the consideration phase, imecs data platform was up and running in just 4 months. Even more: within the same 4 months the first 14 use-cases were live.
Conveyor was chosen to avoid reinventing the wheel. It offered a set of open standards which were on imecs wishlist, and they were already properly integrated. Imec could have done this themselves, but they acknowledged the fact that this has been done before. Due to experience, others might be better at it and could take the long-term maintenance out of their hands. This long-term maintenance might be the most neglected pitfall of building your own data platform: if you build it yourself, you need to maintain it yourself.&nbsp;Often platform teams get bogged down in operational issues after a while and can no longer add any new functionality.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Scaling your data platform</h1>
When talking about scaling a data platform, we immediately think about scaling the infrastructure but scaling a data platform also involves:
<ul>
<li>Increasing the amount of use cases running on top of it.</li>
<li>The number of people&nbsp;working with it and the knowledge required&nbsp;to work on it.</li>
<li id="da21" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">The ability to securely share data between use cases.</li>
</ul>
<blockquote>
Imec is a good example that technically scaling the platform is not the biggest challenge, scaling people is.
</blockquote>
It is hard to onboard people to work with data. Both internally where you need to sell the platform as externally where you need to attract experts. This is one of the reasons they prefer buying platforms and components, to avoid hiring a lot of people in the data platform team. Data workers might be getting infused in the business teams, a consequence of an ongoing data mesh discussion. A small data platform team should enable them to focus on value creation with standards, guidelines and paved roads. This renewed focus is the new IQ, efficience is key.
This separation of responsibilities is recognized by Kristof amongst many companies. A data platform team will take care of shared services, compute, infrastructure and data access, allowing use case teams to focus on building and maintaining the data use cases themselves. It&rsquo;s the platforms team responsibility to sell the platform to more and more people internally. You can only do this with a proper customer-centric mindset.
<h1>The cost of innovation</h1>
Sometimes standardization is considered the opposite of innovation. The goal of imec is to balance both. Yes, the IT department wants to introduce standards, but researchers value freedom and flexibility. The key is having a good offering tailored to each user profile.&nbsp;Remember the customer-centric mindset of the platform team? The users at imec are strongly demanding paved roads/standardization within the tools of their preference because it takes away complexity, it becomes an enabler, and even an accelerator for innovation.
The modular approach at Imec limits the dependency on one single partner. The open standards offered by Conveyor or the interchangeable Azure components allow to change them when desired with new, innovative tools over time.
But what does this approach cost? Make a business case when buying a tool and compare it to the cost building it. However, do not forget the maintenance cost on the long run. Even open source technologies cost money as you need to keep maintaining their installation and keep upgrading them. Imec argues that the tools they have bought have a lower Total Cost of Ownership than a build approach. In order to keep costs under control they focus on avoiding licenses without added value as well as idle resources.
As a small remark, Kristof notices that many companies find it easier to accept the people cost as opposed to the tooling cost. Hiring extra people often does not require a procurement process and rarely the business case of people versus licenses is made.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Security and compliance</h1>
Off the shelf data platforms usually have a sensible story that covers most if not all security aspects. Companies want to have control over who can access which data, who can alter production processes, how to enable software engineering best-practices, how to comply to regulatory requirements, &hellip;
When buying a tool, it is mostly a matter of translating the concepts that these data platforms offer to the existing company processes. As this translation is never a 100% match, companies often favor building a custom data platform than updating the company processes.
Kristof advocates that it makes sense to change the company processes to the concepts of a data platform. Again these are proven concepts, distilled from experiences at many different customers. The agility of your organization heavily impacts the feasibility of this change. Yet, if your level of agility does not allow for this, do you believe your company is the right one to build a tool itself?
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Looking back</h1>
Imec&rsquo;s approach has always been to buy where possible and to build where you can make the difference. However they do not believe in buying a Suisse knife that can do everything and probably comes with the cost of a Rolls Royce. Therefor imec takes a modular approach, which requires you to write glue code between the different selected components. This is where Conveyor came in: a proven technology. An added benefit is that Imec does not need to maintain the components, nor the glue between them.
Overall, imec is very happy with the choices made. The world of technology and data is moving fast, which means that over time you might no longer have the best platform possible. This does not imply that you made the wrong choices, but rather stresses the importance of building a modular platform. With this approach, you have the flexibility to change components when desired.
A data platform is there to scale the self-service usage of data throughout your organization. We notice that it&rsquo;s more difficult to scale the number of users than your infrastructure. Your data platform should be customer-centric in order to convince more and more people to use it. A great user experience is crucial. A small data platform team focusing on this enablement, frees up your data workers to focus on value adding use-caes.&nbsp;Furthermore, their development with the use of paved roads is what allows you to reach your data ambitions.

Let’s start with explaining the different factions that we typically see while deciding on a data platform: business, IT and governance. Typically business teams want to go fast and have something available yesterday. 

When Should You NOT Build A Custom Internal Data Platform?

When You Should NOT Build Your Own Internal Data Platform

Central data teams aiming to build a data platform and deliver data use-cases at the same time, have been created and are often organized under IT. Business departments on the other hand, have attempted to create value from their data silo by buying tools and hiring consultants as business leaders want results now, not when such a central team can plan them.
A number of waves to re-centralize data teams have occurred, driven by Enterprise Data Warehouses, Company-Wide Data Lakes and many more. Only recently with the rise of data mesh, strategies are adapting to the distributed nature of data. Rather than re-centralizing, large scale companies are decentralizing their data teams, outside of IT, embedded in the business. These teams are empowered with a data platform, still build within IT. What are the benefits of a data platform team?
<h1>Wait what, data is not IT?</h1>
As IT stands for Information Technology, the role of an IT department can become quite broad. As a minimal responsibility, an IT department is responsible for maintaining the hardware and software systems within an organization. Briefly summarized: the IT department makes sure the rest of the company is properly equipped to perform their tasks.
While looking to (semi-)tech companies, product development is often also ranked under IT. These products are build for the company&rsquo;s customers or for internal users, yet development is led by a single department with a clear business goal. Processes are put in place to strengthen the cooperation between the development team and the business leaders.
This is the main distinction with data: data use-cases - machine learning algorithms and predictions, reports, &hellip; - are all individual products and all might be build for different stakeholders. There is no single department or business leader who can easily prioritize amongst them, nor who can capture and translate all business requirements. These data use-cases however, might all be build using the same data platform. It&rsquo;s this data platform that has a common goal: enabling your data workers to reach value from their use-cases.
<h1>A data platform</h1>
But, what is a data platform? For most software components it is high-level very clear: a CRM (Customer Relationship Management) system contains all my customers and manages my relations and possibly my interactions with them, a billing system creates invoices and sends them to customers, &hellip; A data platform on the other hand, well, it does something with data, but what?
A data platform covers a vast amount of capabilities. To be able to valorize data, you most likely want to combine data from multiple of your software systems. None of these software systems are build to contain all your company&rsquo;s data. A data platform therefore needs to store all data and in the first place actually ingest the data. Afterwards you want to apply business transformation logic to this data and structure it for use in reports or train a machine learning algorithm on the data. Those results then need to be presented to a consumer of your insights and updated on a regular basis. All these capabilities combined are what you call your data platform. All these capabilities are often tackled by distinct tools (un)fortunately.
This data platform is the software (and hardware or cloud infrastructure) system people use to massage data and get value out of it.
<h1>In comes the data platform team</h1>
Hence maintaining the data platform is an IT responsibility. Compared to typical software systems like a CRM system, the data platform consists of multiple components and I will refer to the team maintaining all these tools as the &ldquo;Data Platform Team&rdquo;. Should this data platform team be organized within your IT department? Possibly. Not necessarily. And not the focus of the point I would like to make today.
The data platform team should provide and maintain a platform that has the capability of data ingestion, data storage, data transformation, reporting, machine learning, scheduling, &hellip;. Rapidly this platform consists of 6 or more tools. If the platform team needs to maintain all of these tools and integrate them with each other, you can imagine that the team needs to be quite large and requires some time to set everything up. As this does not deliver direct value, rather supports the value creation, you will understand that this will rapidly become a budget challenge.
Therefore, the responsibility of the data platform team is to select the required components, preferably managed services to lower the maintenance, and integrate them, while safeguarding a clear division of capabilities. This should allow a shift in the focus of data workers from building and maintaining a data platform towards delivering value from data.
<h1>A shift of focus to deliver value</h1>
Through the usage of your data platform, your data workers can focus on transforming data, training machine learning models and delivering data products which drive value. It doesn&rsquo;t really matter whether your data workers are still centrally organized or federated in a data mesh set-up: through your data platform, you have liberated your data workers from maintaining their own tools. This allows them to take a deep-dive in the business processes and business goals and search for patterns that build up towards these company goals.

For years, organizations have been building and using data platforms to get value out of data. For years, companies have been struggling to organize to optimize the value creation from data. For years , organizations have failed to maximize the return on their data investments? Why is this, and what should you to, do increase your chances?