Blogs

Every December, <a href="https://reinvent.awsevents.com/" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI3LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3JlaW52ZW50LmF3c2V2ZW50cy5jb20vIiwidGV4dCI6IkFXUydzIHJlOkludmVudCBjb25mZXJlbmNlIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">AWS's re:Invent conference</a> brings a flood of announcements that can feel overwhelming, even for technical teams. This year was no different, with over 30 new data-related features announced. As a data leader, you might be wondering what this all means for your organization, beyond the technical specs.
From a strategic perspective, three key themes emerged at AWS Re:Invent 2024
<ol>
<li dir="ltr" role="presentation">AWS heavily invested in AI capabilities, launching everything from coding assistants to supercomputers. They launched Amazon Q Developer and Nova foundation models, directly challenging OpenAI's dominance and Anthropic's Claude in the enterprise AI space. (Although they also invested 8 billion in Anthropic). The introduction of multi-agent collaboration in Bedrock also takes aim at Microsoft's Copilot suite.</li>
<li dir="ltr" role="presentation">They introduced new data governance tools, particularly around metadata management and security controls. With Amazon SageMaker Data, AI Governance and enhanced data lineage features, they are positioning themselves against Databricks' Unity Catalog and pure-play data governance platforms like Collibra.</li>
<li dir="ltr" role="presentation">And they announced significant improvements to their storage solutions, with S3 Tables bringing Apache Iceberg support and queryable metadata features, competing with Snowflake's native tables and Databricks' Delta Lake.&nbsp;</li>
</ol>
<img style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif; display: block; margin-left: auto; margin-right: auto;" src="https://media.prod.dataminded-dev.nl/aws_s3tables_iceberg_a28a1f3013.png" alt="aws-s3tables-iceberg.png" width="648" height="366">
<pre dir="ltr" style="text-align: center;">Image source: <a href="https://bigdata.2minutestreaming.com/p/meet-your-new-data-lakehouse-s3-iceberg">https://bigdata.2minutestreaming.com/p/meet-your-new-data-lakehouse-s3-iceberg</a>&nbsp;</pre>
Unsurprisingly, each announcement adds layers of complexity to an already complex landscape. Yet, we believe it should be clear by now (at least to larger organisations) that complexity in the data landscape isn't a problem to solve anymore, we&rsquo;re way past that point. It&rsquo;s simply a new reality to embrace.&nbsp;
So with every big data industry event like AWS&rsquo;s annual conference, a more useful approach for the Data Leader is to try to understand how to evaluate, prioritize and adopt what truly adds value. In a nutshell, consider how to effectively manage the added complexity that new tech developments inevitably bring, instead of how to solve it. This can be done by following a structured approach to working with new capabilities - through clear processes for evaluation, adoption, and integration into existing workflows.
The recent SageMaker features, for instance, promise to democratize AI development. But for organisations without clear processes for collaboration and governance, this democratization could lead to chaos. Similarly, AWS's new storage features like S3 Tables offer new capabilities for organizing and querying data. But the technical ability to query data more efficiently doesn't address the organizational challenges of determining who should have access to what data and why.
<h1>Evaluate New Tech Announcements Through The Data Product Lens</h1>
So while keeping up with new tech updates is important, leaders need a systematic approach to evaluation - one that focuses less on technical details and more on how new capabilities can be effectively integrated into existing organizational processes.
This is where data product thinking comes in. Instead of fighting socio-technical complexity, it provides a framework for operating within it. By viewing data as a product with clear ownership, defined interfaces, and standardized processes, organizations can navigate complexity confidently.&nbsp;&nbsp;
So whether you're evaluating <a href="https://www.aboutamazon.com/news/aws/amazon-nova-ai-canvas-reel-aws-reinvent" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI3LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5hYm91dGFtYXpvbi5jb20vbmV3cy9hd3MvYW1hem9uLW5vdmEtYWktY2FudmFzLXJlZWwtYXdzLXJlaW52ZW50IiwidGV4dCI6IkFXUydzIGxhdGVzdCBkYXRhL0FJIGFubm91bmNlbWVudHMgIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">AWS's latest data/AI announcements </a>or considering any new data technology, you can use this perspective to cut through the noise. It makes it easier to understand what may be an opportunity and what may be a hindrance. For organisations that are already managing large data systems and are well aware of the drawbacks of centralised data architectures, evaluating new capabilities through the data product lens can mean asking things like:&nbsp;How does this new capability fit into our existing data product architecture? Does it help standardize how teams deliver and consume data?&nbsp;Can it strengthen our data product interfaces and contracts?
For example, AWS's new S3 Tables feature could strengthen data product interfaces by providing standardized ways to expose and consume data. Their enhanced governance features could help automate data product quality measurements and access controls.&nbsp;
&nbsp;
<img style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif; display: block; margin-left: auto; margin-right: auto;" src="https://media.prod.dataminded-dev.nl/s3_tables_83213f85d2.webp" alt="s3-tables.webp" width="650" height="365">
&nbsp;
<h1 dir="ltr">AWS Re:Invent 2024: The Data Product Approach</h1>
So here&rsquo;s how we can look at AWS's announcements through specific data product considerations.
<h2 dir="ltr" style="padding-left: 40px;">Does any of this help data product teams deliver value faster?</h2>
The new AI capabilities in Amazon SageMaker, like HyperPod flexible training plans, promise faster model development through optimized compute resources. This speed advantage materializes if your organization has clear processes for model deployment and validation. Without them, faster development could actually lead to more technical debt and governance challenges.
<h2 dir="ltr" style="padding-left: 40px;">Will any of this make governance models more efficient?</h2>
Amazon SageMaker's Data and AI Governance features offer automated policy enforcement and improved data lineage. These require significant setup and maintenance. Organizations with well-defined data product ownership models will find these features amplify their existing governance processes. Those still struggling with basic questions like "who owns this data?" might find these tools add another layer of complexity without solving fundamental challenges.
<h2 dir="ltr" style="padding-left: 40px;">Can any of this improve collaboration between data product owners and consumers?</h2>
The enhanced storage features, like S3 Tables, could either streamline or complicate collaboration, depending on your organization's maturity. While they enable better data organization and querying, they also introduce new concepts and workflows that teams need to understand. Success depends less on the technical capabilities and more on having clear processes for how teams should work together.
These questions help shift the focus from technical capabilities to organizational readiness and process maturity. This is the approach we take at Dataminded when delivering strategic data consultancy and also with <a href="https://github.com/conveyordata/data-product-portal" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI3LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2dpdGh1Yi5jb20vY29udmV5b3JkYXRhL2RhdGEtcHJvZHVjdC1wb3J0YWwiLCJ0ZXh0IjoiPHN0cm9uZz5Qb3J0YWw8L3N0cm9uZz4iLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiIiLCJ1cmxBZGRpdGlvbiI6IiJ9">Portal</a>, a new open-source data product management tool that provides a guided process for creating data products and that enables governance by design for data initiatives.
<h1 dir="ltr">Conclusion</h1>
As with any new announcement, there's excitement around the AWS releases for opportunities to get more out of our data initiatives. The challenge lies in how we approach these advances. By viewing them through the lens of data product thinking and process, we can better prioritize and focus on standardized approaches that help teams collaborate effectively and deliver value consistently.
<h1 dir="ltr">TL;DR</h1>
For leaders that embrace the decentralized governed approach of data products, several announcements made at&nbsp;AWS re:Invent 2024 show promising alignment with data product thinking.
<ul>
<li dir="ltr">Amazon SageMaker's Data and AI Governance features could automate quality measurements for data products</li>
<li dir="ltr">S3 Tables' support for Apache Iceberg could make data product interfaces more standardized and interoperable</li>
<li dir="ltr">Enhanced data lineage features could help track relationships between data products</li>
</ul>
These developments suggest the industry is moving toward more structured approaches to data management, though implementation success will still depend on clear organizational processes.

For leaders that embrace the decentralized governed approach of data products, several announcements made at AWS re:Invent 2024 show promising alignment with data product thinking.

AWS re:Invent 2024: A Data Leader's Summary 

AWS Re:Invent 2024: A Data Leader's Summary

When teams first adopt Databricks, they typically start with a "yes to everything" approach. Data scientists get full access to explore, experiment, and innovate. This works well&mdash; until it doesn't. Three of the most common scenarios we see:
<ul>
<li>Data usage becomes untraceable</li>
<li>Compliance requirements&nbsp;get overlooked</li>
<li>Security policies&nbsp;become hard to enforce</li>
</ul>
Even with the Unity Catalog, organizations struggle to answer basic questions.. Who is using what data? Why are they using it (for which project?) Are they following our organisation's compliance rules?
<a href="https://data-minded-webinars.webinargeek.com/bringing-governance-to-databricks-enable-self-service-without-losing-control?cst=conveyorwebsite" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2RhdGEtbWluZGVkLXdlYmluYXJzLndlYmluYXJnZWVrLmNvbS9icmluZ2luZy1nb3Zlcm5hbmNlLXRvLWRhdGFicmlja3MtZW5hYmxlLXNlbGYtc2VydmljZS13aXRob3V0LWxvc2luZy1jb250cm9sP2NzdD1jb252ZXlvcndlYnNpdGUiLCJ0ZXh0IjoiTkVXISBSZWdpc3RlciBmb3Igb3VyIHVwY29taW5nIHdlYmluYXIgJmd0OyZndDsgQ1RSTC1aIFRvIERhdGEgQ2hhb3M6IEJyaW5naW5nIFNlbGYtU2VydmljZSBHb3Zlcm5hbmNlIFRvIERhdGFicmNpY2tzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">NEW!</a><a href="https://data-minded-webinars.webinargeek.com/bringing-governance-to-databricks-enable-self-service-without-losing-control?cst=conveyorwebsite" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2RhdGEtbWluZGVkLXdlYmluYXJzLndlYmluYXJnZWVrLmNvbS9icmluZ2luZy1nb3Zlcm5hbmNlLXRvLWRhdGFicmlja3MtZW5hYmxlLXNlbGYtc2VydmljZS13aXRob3V0LWxvc2luZy1jb250cm9sP2NzdD1jb252ZXlvcndlYnNpdGUiLCJ0ZXh0IjoiTkVXISBSZWdpc3RlciBmb3Igb3VyIHVwY29taW5nIHdlYmluYXIgJmd0OyZndDsgQ1RSTC1aIFRvIERhdGEgQ2hhb3M6IEJyaW5naW5nIFNlbGYtU2VydmljZSBHb3Zlcm5hbmNlIFRvIERhdGFicmNpY2tzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ=="> CTRL-Z To Data Chaos: Bringing Self-Service Governance To Databricks</a>
There is a real cost to governance failures, it's not only about a box left unchecked. It's seen in valuable time spent tracking down data usage, in the increased risk of violations that can trigger serious penalties, in&nbsp; security vulnerabilities and, maybe most frustratingly for those involved, in duplicate (or often triplicate, quadruplicate or more) work across teams.
<h2>Building Effective Governance in Databricks</h2>
The first impulse is to fully restrict access, and this would be the easiest for one side - the compliance side. But it's not feasible. Effective governance is that which effectively supports the business, not hinders it. So we need to try to make governance part of the natural workflow of companies. And there are multiple ways in which we can adapt&nbsp; Databricks governance strategies to make them work for both sides.
<h3>1. Clear Data Ownership in Databricks</h3>
Data ownership often becomes unclear as teams grow and projects multiply. So successful organizations address this by establishing clear ownership patterns early on. They define not just who owns each dataset, but also how that ownership translates into practical responsibilities.
For example, we can assign dataset owners based on business domains rather than technical teams. These owners become responsible for defining usage policies and approving access requests. More importantly, they track how their data is being used across different projects, ensuring compliance with regulatory requirements while maintaining team agility.
<h3>2. Self-Service Governance in Databricks</h3>
Traditional access management often creates bottlenecks, with IT teams becoming overwhelmed by access requests. Modern governance approaches attempt to solve this by looking at embedding access controls into the daily workflow of data teams.
A practical implementation could look like this:
<ul>
<li>Structure data access per data product output</li>
<li>Self-service interface for access requests</li>
<li>Automate workflows routing requests to appropriate approvers</li>
<li>Regular data access reviews</li>
</ul>
This approach typically reduces access request processing time from months or weeks to hour or minutes, while again - maintaining strict compliance standards.
<h3>3. Automated compliance tracking in Databricks</h3>
Manual compliance tracking doesn't scale. No manual process ever does. So organizations successful with Databricks governance look at automating compliance processes. Reducing the need for manual input drastically reduces friction between teams and can improve the adoption of any process. Automating in this case can mean:
<ul>
<li>Automatically logging all&nbsp;data access requests</li>
<li>Tracking data lineage across all projects</li>
<li>Generating compliance reports automatically</li>
<li>Setting up automated alerts for potential policy violations</li>
</ul>
We could imagine how implementing these can lead to significantly less time spent on routine checks and to increased focus on strategic governance improvements.
<h2>Having Our Cake And Eating It Too : Innovation PLUS Governance</h2>
It's worth repeating the key to successful governance here - it's&nbsp; finding the right balance between control and flexibility. This flexibility is vital to maintain innvation and ensure a healthy growth. Organizations that get this right understand that they need to give data scientists freedom but also that they need to do it within defined boundaries. This can be achieved by looking at automating routine governance tracks, effectively making compliance the path of least resistance, and enabling quick access to approved data sources, which is what data scientists often struggle the most with.
We often talk to Data Leaders that struggle with effective governance and understand that things aren't on the right track but aren't sure where to look at. So it's important to first clearly understand the specifics of our current setup. Do we know who's using our critical data? Can teams easily request and get access to data? Are we meeting compliance requirements without manual work?
If we find ourselves answering "no" to any of these, it might be time to rethink our approach to Databricks governance.
<a href="https://data-minded-webinars.webinargeek.com/bringing-governance-to-databricks-enable-self-service-without-losing-control?cst=conveyorwebsite" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2RhdGEtbWluZGVkLXdlYmluYXJzLndlYmluYXJnZWVrLmNvbS9icmluZ2luZy1nb3Zlcm5hbmNlLXRvLWRhdGFicmlja3MtZW5hYmxlLXNlbGYtc2VydmljZS13aXRob3V0LWxvc2luZy1jb250cm9sP2NzdD1jb252ZXlvcndlYnNpdGUiLCJ0ZXh0IjoiTkVXISBSZWdpc3RlciBmb3Igb3VyIHVwY29taW5nIHdlYmluYXIgJmd0OyZndDsgQ1RSTC1aIFRvIERhdGEgQ2hhb3M6IEJyaW5naW5nIFNlbGYtU2VydmljZSBHb3Zlcm5hbmNlIFRvIERhdGFicmNpY2tzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">NEW! Register for our upcoming webinar &gt;&gt; CTRL-Z To Data Chaos: Bringing Self-Service Governance To Databrcicks</a>
Start with an honest mapping of the data landscape to identify critical datasets, document current usage patterns and list compliance requirements. Once there's a clear and shared understanding of the actual data landscape and the requirements of all business functions, move to implementing self-service governance:
<ul>
<li>Teams request access through a simple interface</li>
<li>Approvals follow&nbsp;predefined rules</li>
<li>Usage&nbsp;and requests are tracked automatically</li>
</ul>
Data scientists get to keep their flexibility, compliance teams get their oversight and everyone saves time.
<h2>The Path Forward: Bring Self-Service Governance To Databricks</h2>
We've seen this scenario play out often at customers so we've built a tool to help solve these problems. The Data Product Portal is a fully open-source project and free to use. It lets you continue using your preferred data science and analytics tools like Databricks, while offering self-service automated governance. It's best suited for data teams within businesses that aim to build their own use cases in a self-service way, embracing data product thinking.
So imagine this: two teams, each with four data scientists. Team A is building a demand forecasting model and Team B is developing a pricing optimization system.
Team B needs Team A's forecast outputs. Their options are:
<ul>
<li>Request&nbsp;access to Team A's entire&nbsp;workspace (security risk)</li>
<li>Have Team A manually&nbsp;export and share results (time-consuming)</li>
<li>Create duplicate data pipelines (inefficient and error-prone)</li>
</ul>
They risk spending more time solving access issues than improving their models.
When using Portal, the organisation enables Team B to access only the specific forecast outputs they need, see the data lineage (where the data comes from) and get automated access to updates. One of the biggest benefits to working within a data product framework is that it makes accessing the latest version of the data product output the default modus operandi. Meanwhile, Portal empowers Team A to maintain control of their workspace while sharing only the necessary outputs, track who uses their data and automate access management.
In a nutshell, the key point we want to make is that effective Databricks governance doesn't mean choosing between innovation and control. It means making governance simple enough that teams want to follow it by automating the boring parts and reducing friction through self-service capabilities.
Check out <a href="https://github.com/conveyordata/data-product-portal">the project's repo</a> for more details, reach out to us anytime on our <a href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2r47r0icj-PAYbbklmH6gPz9A6bJ144Q">Slack channnel</a> or <a href="../../../../book-demo" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiJhcGk6OnBhZ2UucGFnZSIsInRhcmdldENvbnRlbnRUeXBlSWQiOjUzLCJ1cmwiOiJodHRwczovL2NvbnZleW9yZGF0YS5jb20vYm9vay1kZW1vIiwidGV4dCI6ImJvb2sgYSBndWlkZWQgZGVtbyIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6Imh0dHBzOi8vY29udmV5b3JkYXRhLmNvbSIsInVybEFkZGl0aW9uIjoiIn0=">book a guided demo</a>, we'd love to show you around.

Data scientists love Databricks for its power and flexibility. But as organizations scale their Databricks implementations, a pattern emerges: the very freedom that makes Databricks powerful can become its biggest governance challenge.

The Hidden Cost of Databricks Freedom: Balancing Innovation with Governance

As a platform engineer, you will succeed when offering more than "a bag of tools" to business users. Offer technology-independent UIs that suit the particular needs of your company.

Give Your Business Users A Friendly Interface To Your Data Platform

The Data Product Portal is an open-source tool to help organisations build and manage data products at scale. It’s intuitive, flexible, and geared towards making data product management straightforward and effective.

How Portal Integrates With Data Platforms

The more people you add to a team, the more lines of communication you introduce. The same holds for the number of tools in your data landscape

How To Conquer The Complexity Of The Modern Data Stack

A new category of tools entered the data scene: data product catalogs. We believe such a tool is useful not only when it provides an overview, but also empowers users to navigate their data landscape.

The Missing Piece To Data Democratization Is Not A Catalog

The Missing Piece to Democratize Data Is Not A Catalog

Traditional medallion architecture is showing its age. It wasn't designed for the agility and self-service needs of modern data teams. So, what lies beyond medallion?

Beyond Medallion: New Data Architectures For The Age Of Data Products

In the fast-evolving world of data, companies are discovering that the key to success for scaling their data initiatives is not to rely on a single data team to handle all requests from the business. Instead they want to enable self-service capabilities to allow each domain or department to build their own data products at their own pace with their own budgets.
However, putting this vision into practice can be very challenging, especially when you put all aspects related to data governance, data platforms and data catalogs&nbsp;together. It can become messy very quickly as it is hard to provide an easy to understand consistent view across all these dimensions and technologies.
Today we are very happy to announce the&nbsp;Data Product Portal&nbsp;&mdash; an open-source tool to help organisations build and manage data products at scale. It&rsquo;s intuitive, flexible, and geared towards making data product management straightforward and effective.
<h1 id="9533" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">What problem does it solve</h1>
Imagine you&rsquo;re building a data pipeline. You take some input data, process it with python, DBT or another tool and generate output for others to use. This scenario applies whether you&rsquo;re using Starburst with Ranger for access controls, or other platforms technologies like Snowflake, AWS, Databricks, BigQuery or Microsoft Fabric.
<img src="https://media.prod.dataminded-dev.nl/1_Noue_Lmz_Vd_L_9n_Vp_Vr6h_UVA_6ec23f20b3.webp" alt="1_NoueLmzVdL_9nVpVr6hUVA.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Typical data pipeline consuming remote data, data from files and databases</figcaption>
</figure>
&nbsp;
In large organisations, you typically have multiple departments or domains, and you can&rsquo;t simply share all data between all departments by default due to legal, compliance, regulatory or confidentiality reasons. This means that departments or domains have to keep control how and why their data is being used by others.
<img src="https://media.prod.dataminded-dev.nl/1_K_Mqruo_H_Tid_Stx_Ky_K_Vr_AHNQ_2dd73e0f41.webp" alt="1_KMqruoHTidStxKyKVrAHNQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Scaling data pipelines across a large organisation with multiple departments</figcaption>
</figure>
&nbsp;
The moment you start sharing data across multiple departments or domains, you are immediately faced with the following questions:
&nbsp;
<blockquote>
Who has access to what data, when can they use it and for what reason are they using it?
</blockquote>
If you are working with data, you want to know where to access and find the data and tooling to build your data pipelines.
To answer these questions, companies start data governance initiatives, where they start generating policies per user to manage access to data in tools like Ranger, AWS IAM, Snowflake or any other tool. When people have been working on multiple use cases, all of these policies accumulate together. This &ldquo;spaghetti&rdquo; of permissions will lead towards very broad access permissions for a large group of people. After a while you will end up in the situation you wanted to avoid in the first place where:
&nbsp;
<blockquote>
Everybody ends up with access to all data and you no longer know why and how your data is being used.
</blockquote>
<h1 id="a357" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm pe no np nq pf ns nt nu pg nw nx ny bj" data-selectable-paragraph="">Data products as a governance model</h1>
There are many different definitions out there about what is a data product and every organisation or person working with data has their own opinion about it, but we found the following definition to be useful and quite universal for many organisational structures.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_ji_BF_p71h_Wo_YE_5_Mki_XG_Ntg_0bd0979863.webp" alt="1_jiBF_p71hWoYE5MkiXGNtg.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Definition of a data product</figcaption>
</figure>
&nbsp;
We propose to define a&nbsp;data product&nbsp;as: an initiative with a clear goal, owned by a department or domain of the business that consists of the combination of:
<ol class="">
<li id="8b22" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Input data: Access to datasets created by a combination of other data products.</li>
<li id="47a8" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Output data: Read/write access to output data that can be combined into a dataset that can be shared with other data products. This data is stored in specific locations (e.g. databases or buckets).</li>
<li id="1ea3" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Private data: A safe location to store private/internal data for local processing with no intention of sharing</li>
<li id="1f27" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Transformation artefacts: All code and outputs describing your transformations, scheduling and tooling configuration needed to build, access and run your data pipeline separated from other data products.</li>
<li id="5481" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Team roles: Defined roles of team members that have specific permissions on how to interact with the data product (e.g. data product owner, data engineer, business analyst).</li>
</ol>
&nbsp;
<blockquote class="op">
It is important to note that people can work on multiple data products, but have to choose on what data product they are working on. They only get access to that data product scope related to tooling and data.
</blockquote>
In this definition, data products are not only data assets that are the output for sharing with others, but also the tooling, artefacts and roles of people that interact with that data product.
When multiple data products start interacting, your data product lineage will start looking like this:
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_p_Vu_Poa_NU_Ja_Ckgn9_i31mw_6224405452.webp" alt="1_pVu-PoaNUJaCkgn9-i31mw.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">How different data products are interacting with each other</figcaption>
</figure>
&nbsp;
The main benefits of this approach are:
<ul class="">
<li id="39a5" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz pu pi pj bj" data-selectable-paragraph="">Clear Data Usage: You always know who is using your data and why.</li>
<li id="10c1" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Simplified Access Management: Easier to handle access requests and data revocations</li>
<li id="34ed" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Natural Data Lineage:&nbsp;Understand how your data flows from one data product to another.</li>
</ul>
Adopting the data product governance model if very powerful in scaling data initiatives across departments, while still keeping control and self-service capabilities. However, effective tools are essential for managing the processes around this to make it truly self-service.
<h1 id="fbcc" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">Introducing the Data Product Portal</h1>
The&nbsp;Data Product Portal&nbsp;is a practical tool that helps you build data products at scale, that is both useful for both people working with data and people who want to have control over how their data is being used.
It is designed to simplify the creation and management of data products at scale. It is beneficial and easy to use for both data professionals and those overseeing the use of data and data governance.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_Ph_T_Gak_S_Vpw_A8_Oi5_P_Ki13_YQ_56d99d19ee.webp" alt="1_PhTGakSVpwA8Oi5PKi13YQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Data product model translation to policies and configuration</figcaption>
</figure>
&nbsp;
This is where the Data Product Portal comes into play:
<ol class="">
<li id="706e" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Guided Setup: Step-by-step assistance involving the right stakeholders for creating data products, requesting access, adding users and registering new data for sharing with other data products.</li>
<li id="57cf" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Tech Translation: Converts high-level concepts into specific configurations settings for platforms like AWS, Azure, Databricks, Snowflake, and others, making sure that each data product is correctly separated and not impacting each other.</li>
<li id="79df" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">User-Friendly Interface: Makes it easy for business users and people working with data to understand and navigate the data landscape.</li>
<li id="174b" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Self-service:&nbsp;Enables departments and teams to start new data initiatives easily without having to depend on a central team.</li>
<li id="5061" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Comprehensive Overview: Combines data catalogs, data platforms and data governance aspects into a single 360 overview of all ongoing data initiatives.</li>
</ol>
<h1 id="7eb9" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">How to get started</h1>
The Data Product Portal is available as an open source project on&nbsp;<a class="af pv" href="https://github.com/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">Github</a>. Getting started is as simple as running&nbsp;docker compose up&nbsp;and visiting&nbsp;localhost:8080
We invite you to check us out and give us a star if you like what you see. Your contributions are invaluable to us &mdash; whether it&rsquo;s through feedback, suggestions, or direct involvement in development.
For Kubernetes deployments, check out our helm chart&nbsp;<a class="af pv" href="https://gallery.ecr.aws/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">here</a>.
If you have questions or want to share your thoughts? Join our&nbsp;<a class="af pv" href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2lo245t3l-bJxcAz1trqrcQNnGARWMDg" target="_blank" rel="noopener ugc nofollow">community on Slack</a>&nbsp;and connect with us directly. We can&rsquo;t wait to hear from you!
We are excited to bring the Data Product Portal into your hands as an open source initiative. Don&rsquo;t hesitate to share with us what you think about it!
&nbsp;

Scale data initiatives across your entire organisation while enabling self-service and maintaining governance.

Introducing Data Product Portal

Introducing The Data Product Portal: A New Open-Source Tool For Data Governance

<blockquote>
A data platform is about enabling an organization to work with data at scale, in a self-service manner.
</blockquote>
That is how Kristof Martens, tech lead at multiple of <a href="https://www.dataminded.com/" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5kYXRhbWluZGVkLmNvbS8iLCJ0ZXh0IjoiRGF0YW1pbmRlZCdzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">Dataminded's</a> customers, describes a data platform. That is also what Wim Vancuyck, data lead at <a href="https://www.imec-int.com/en" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5pbWVjLWludC5jb20vZW4iLCJ0ZXh0IjoiaW1lYyIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">imec</a>, envisions in his quest to enable more than 5000 researchers to work with data at imec, the world&rsquo;s leading independent nano-electronics R&amp;D hub. We recently sat together and discussed why companies should actually not build their own internal data platform. These are our main conclusions.
<h1 class="pw-post-body-paragraph mz na fv nb b gt nc nd ne gw nf ng nh ni nj nk nl nm nn no np nq nr ns nt my fo bj">The impact of company culture</h1>
Let&rsquo;s start with explaining the different factions that we typically see while deciding on a data platform: business, IT and governance. Typically business teams want to go fast and have something available yesterday. That&rsquo;s why they mostly lean towards buying off the shelf tooling: click install and off you go.
IT departments on the other hand focus on standardization and development best-practices, and tend to lean towards building a solution, as it offers you more flexibility and control. And let&rsquo;s be honest: building a data platform often looks easy at the start. The third party, the governance team, has a more narrow view on a data platform: a data catalog will solve all problems. In the end, it&rsquo;s the faction with most power in the company that influences its buy vs build mentality.
Funny enough, at imec it&rsquo;s the other way around: the R&amp;D department exists of people that are used to coding, which led to building and rebuilding quite often. The IT department on the contrary is small compared to the amount of researchers, loves to work with partners and buy tools to scale fast and introduce standardization. It&rsquo;s all about avoiding re-inventing the wheel over and over again by all those researchers.
This buy mentality, combined with the desire to keep costs under control, the love for open source, and the preference to avoid vendor lock-in, has led to a modular approach to the data platform at imec. This data platform consists of multiple&nbsp;<a class="af nu" href="https://azure.microsoft.com/en-us" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2F6dXJlLm1pY3Jvc29mdC5jb20vZW4tdXMiLCJ0ZXh0IjoiQXp1cmUiLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiIiLCJ1cmxBZGRpdGlvbiI6IiJ9">Azure</a>&nbsp;components, combined with tools like&nbsp;<a class="af nu" href="https://www.mathworks.com/products/matlab.html" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5tYXRod29ya3MuY29tL3Byb2R1Y3RzL21hdGxhYi5odG1sIiwidGV4dCI6Ik1hdGxhYiIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">Matlab</a>, or Conveyor. Conveyor can be considered as the cornerstone, enabling multiple open source technologies as a managed services and properly glued together. In theory, this modular approach allows imec to migrate the platform to other cloud providers, again mitigating a vendor lock-in.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">The pitfalls of building your own data platform</h1>
&ldquo;We can build this ourselves!&rdquo; A common heard argument to build your own data platform, and probably the authors are right. The question is not whether you can build your own data platform, but whether you want or should do so.&nbsp;At imec they have the guideline
<blockquote>
Buy where you can, build where you want to make the difference.
</blockquote>
Often the platform is not where you make the difference: what you use it for is the value add.
Remember that a data platform is about enabling an organization to work with data at scale in a self-service manner. This raises a lot of questions, like:
<ul class="">
<li id="9c4d" class="mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my qa qb qc bj" data-selectable-paragraph="">How will I provide proper separation between teams and projects?</li>
<li id="fe53" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How can I encourgag best-practices?</li>
<li id="8dd7" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I share data?</li>
<li id="3365" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I keep track of governance and compliance concerns?</li>
<li id="b9e1" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I implement a data product lifecycle?</li>
</ul>
It is hard to come up with an answer to all these questions beforehand. If you want to have all of this prior to a first use-case, this can easily take some years. Glueing tested and proven components together, as imec does, speeds up this process drastically. After the consideration phase, imecs data platform was up and running in just 4 months. Even more: within the same 4 months the first 14 use-cases were live.
Conveyor was chosen to avoid reinventing the wheel. It offered a set of open standards which were on imecs wishlist, and they were already properly integrated. Imec could have done this themselves, but they acknowledged the fact that this has been done before. Due to experience, others might be better at it and could take the long-term maintenance out of their hands. This long-term maintenance might be the most neglected pitfall of building your own data platform: if you build it yourself, you need to maintain it yourself.&nbsp;Often platform teams get bogged down in operational issues after a while and can no longer add any new functionality.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Scaling your data platform</h1>
When talking about scaling a data platform, we immediately think about scaling the infrastructure but scaling a data platform also involves:
<ul>
<li>Increasing the amount of use cases running on top of it.</li>
<li>The number of people&nbsp;working with it and the knowledge required&nbsp;to work on it.</li>
<li id="da21" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">The ability to securely share data between use cases.</li>
</ul>
<blockquote>
Imec is a good example that technically scaling the platform is not the biggest challenge, scaling people is.
</blockquote>
It is hard to onboard people to work with data. Both internally where you need to sell the platform as externally where you need to attract experts. This is one of the reasons they prefer buying platforms and components, to avoid hiring a lot of people in the data platform team. Data workers might be getting infused in the business teams, a consequence of an ongoing data mesh discussion. A small data platform team should enable them to focus on value creation with standards, guidelines and paved roads. This renewed focus is the new IQ, efficience is key.
This separation of responsibilities is recognized by Kristof amongst many companies. A data platform team will take care of shared services, compute, infrastructure and data access, allowing use case teams to focus on building and maintaining the data use cases themselves. It&rsquo;s the platforms team responsibility to sell the platform to more and more people internally. You can only do this with a proper customer-centric mindset.
<h1>The cost of innovation</h1>
Sometimes standardization is considered the opposite of innovation. The goal of imec is to balance both. Yes, the IT department wants to introduce standards, but researchers value freedom and flexibility. The key is having a good offering tailored to each user profile.&nbsp;Remember the customer-centric mindset of the platform team? The users at imec are strongly demanding paved roads/standardization within the tools of their preference because it takes away complexity, it becomes an enabler, and even an accelerator for innovation.
The modular approach at Imec limits the dependency on one single partner. The open standards offered by Conveyor or the interchangeable Azure components allow to change them when desired with new, innovative tools over time.
But what does this approach cost? Make a business case when buying a tool and compare it to the cost building it. However, do not forget the maintenance cost on the long run. Even open source technologies cost money as you need to keep maintaining their installation and keep upgrading them. Imec argues that the tools they have bought have a lower Total Cost of Ownership than a build approach. In order to keep costs under control they focus on avoiding licenses without added value as well as idle resources.
As a small remark, Kristof notices that many companies find it easier to accept the people cost as opposed to the tooling cost. Hiring extra people often does not require a procurement process and rarely the business case of people versus licenses is made.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Security and compliance</h1>
Off the shelf data platforms usually have a sensible story that covers most if not all security aspects. Companies want to have control over who can access which data, who can alter production processes, how to enable software engineering best-practices, how to comply to regulatory requirements, &hellip;
When buying a tool, it is mostly a matter of translating the concepts that these data platforms offer to the existing company processes. As this translation is never a 100% match, companies often favor building a custom data platform than updating the company processes.
Kristof advocates that it makes sense to change the company processes to the concepts of a data platform. Again these are proven concepts, distilled from experiences at many different customers. The agility of your organization heavily impacts the feasibility of this change. Yet, if your level of agility does not allow for this, do you believe your company is the right one to build a tool itself?
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Looking back</h1>
Imec&rsquo;s approach has always been to buy where possible and to build where you can make the difference. However they do not believe in buying a Suisse knife that can do everything and probably comes with the cost of a Rolls Royce. Therefor imec takes a modular approach, which requires you to write glue code between the different selected components. This is where Conveyor came in: a proven technology. An added benefit is that Imec does not need to maintain the components, nor the glue between them.
Overall, imec is very happy with the choices made. The world of technology and data is moving fast, which means that over time you might no longer have the best platform possible. This does not imply that you made the wrong choices, but rather stresses the importance of building a modular platform. With this approach, you have the flexibility to change components when desired.
A data platform is there to scale the self-service usage of data throughout your organization. We notice that it&rsquo;s more difficult to scale the number of users than your infrastructure. Your data platform should be customer-centric in order to convince more and more people to use it. A great user experience is crucial. A small data platform team focusing on this enablement, frees up your data workers to focus on value adding use-caes.&nbsp;Furthermore, their development with the use of paved roads is what allows you to reach your data ambitions.

Let’s start with explaining the different factions that we typically see while deciding on a data platform: business, IT and governance. Typically business teams want to go fast and have something available yesterday. 