Knowledge

Every December, <a href="https://reinvent.awsevents.com/" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI3LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3JlaW52ZW50LmF3c2V2ZW50cy5jb20vIiwidGV4dCI6IkFXUydzIHJlOkludmVudCBjb25mZXJlbmNlIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">AWS's re:Invent conference</a> brings a flood of announcements that can feel overwhelming, even for technical teams. This year was no different, with over 30 new data-related features announced. As a data leader, you might be wondering what this all means for your organization, beyond the technical specs.
From a strategic perspective, three key themes emerged at AWS Re:Invent 2024
<ol>
<li dir="ltr" role="presentation">AWS heavily invested in AI capabilities, launching everything from coding assistants to supercomputers. They launched Amazon Q Developer and Nova foundation models, directly challenging OpenAI's dominance and Anthropic's Claude in the enterprise AI space. (Although they also invested 8 billion in Anthropic). The introduction of multi-agent collaboration in Bedrock also takes aim at Microsoft's Copilot suite.</li>
<li dir="ltr" role="presentation">They introduced new data governance tools, particularly around metadata management and security controls. With Amazon SageMaker Data, AI Governance and enhanced data lineage features, they are positioning themselves against Databricks' Unity Catalog and pure-play data governance platforms like Collibra.</li>
<li dir="ltr" role="presentation">And they announced significant improvements to their storage solutions, with S3 Tables bringing Apache Iceberg support and queryable metadata features, competing with Snowflake's native tables and Databricks' Delta Lake.&nbsp;</li>
</ol>
<img style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif; display: block; margin-left: auto; margin-right: auto;" src="https://media.prod.dataminded-dev.nl/aws_s3tables_iceberg_a28a1f3013.png" alt="aws-s3tables-iceberg.png" width="648" height="366">
<pre dir="ltr" style="text-align: center;">Image source: <a href="https://bigdata.2minutestreaming.com/p/meet-your-new-data-lakehouse-s3-iceberg">https://bigdata.2minutestreaming.com/p/meet-your-new-data-lakehouse-s3-iceberg</a>&nbsp;</pre>
Unsurprisingly, each announcement adds layers of complexity to an already complex landscape. Yet, we believe it should be clear by now (at least to larger organisations) that complexity in the data landscape isn't a problem to solve anymore, we&rsquo;re way past that point. It&rsquo;s simply a new reality to embrace.&nbsp;
So with every big data industry event like AWS&rsquo;s annual conference, a more useful approach for the Data Leader is to try to understand how to evaluate, prioritize and adopt what truly adds value. In a nutshell, consider how to effectively manage the added complexity that new tech developments inevitably bring, instead of how to solve it. This can be done by following a structured approach to working with new capabilities - through clear processes for evaluation, adoption, and integration into existing workflows.
The recent SageMaker features, for instance, promise to democratize AI development. But for organisations without clear processes for collaboration and governance, this democratization could lead to chaos. Similarly, AWS's new storage features like S3 Tables offer new capabilities for organizing and querying data. But the technical ability to query data more efficiently doesn't address the organizational challenges of determining who should have access to what data and why.
<h1>Evaluate New Tech Announcements Through The Data Product Lens</h1>
So while keeping up with new tech updates is important, leaders need a systematic approach to evaluation - one that focuses less on technical details and more on how new capabilities can be effectively integrated into existing organizational processes.
This is where data product thinking comes in. Instead of fighting socio-technical complexity, it provides a framework for operating within it. By viewing data as a product with clear ownership, defined interfaces, and standardized processes, organizations can navigate complexity confidently.&nbsp;&nbsp;
So whether you're evaluating <a href="https://www.aboutamazon.com/news/aws/amazon-nova-ai-canvas-reel-aws-reinvent" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI3LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5hYm91dGFtYXpvbi5jb20vbmV3cy9hd3MvYW1hem9uLW5vdmEtYWktY2FudmFzLXJlZWwtYXdzLXJlaW52ZW50IiwidGV4dCI6IkFXUydzIGxhdGVzdCBkYXRhL0FJIGFubm91bmNlbWVudHMgIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">AWS's latest data/AI announcements </a>or considering any new data technology, you can use this perspective to cut through the noise. It makes it easier to understand what may be an opportunity and what may be a hindrance. For organisations that are already managing large data systems and are well aware of the drawbacks of centralised data architectures, evaluating new capabilities through the data product lens can mean asking things like:&nbsp;How does this new capability fit into our existing data product architecture? Does it help standardize how teams deliver and consume data?&nbsp;Can it strengthen our data product interfaces and contracts?
For example, AWS's new S3 Tables feature could strengthen data product interfaces by providing standardized ways to expose and consume data. Their enhanced governance features could help automate data product quality measurements and access controls.&nbsp;
&nbsp;
<img style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif; display: block; margin-left: auto; margin-right: auto;" src="https://media.prod.dataminded-dev.nl/s3_tables_83213f85d2.webp" alt="s3-tables.webp" width="650" height="365">
&nbsp;
<h1 dir="ltr">AWS Re:Invent 2024: The Data Product Approach</h1>
So here&rsquo;s how we can look at AWS's announcements through specific data product considerations.
<h2 dir="ltr" style="padding-left: 40px;">Does any of this help data product teams deliver value faster?</h2>
The new AI capabilities in Amazon SageMaker, like HyperPod flexible training plans, promise faster model development through optimized compute resources. This speed advantage materializes if your organization has clear processes for model deployment and validation. Without them, faster development could actually lead to more technical debt and governance challenges.
<h2 dir="ltr" style="padding-left: 40px;">Will any of this make governance models more efficient?</h2>
Amazon SageMaker's Data and AI Governance features offer automated policy enforcement and improved data lineage. These require significant setup and maintenance. Organizations with well-defined data product ownership models will find these features amplify their existing governance processes. Those still struggling with basic questions like "who owns this data?" might find these tools add another layer of complexity without solving fundamental challenges.
<h2 dir="ltr" style="padding-left: 40px;">Can any of this improve collaboration between data product owners and consumers?</h2>
The enhanced storage features, like S3 Tables, could either streamline or complicate collaboration, depending on your organization's maturity. While they enable better data organization and querying, they also introduce new concepts and workflows that teams need to understand. Success depends less on the technical capabilities and more on having clear processes for how teams should work together.
These questions help shift the focus from technical capabilities to organizational readiness and process maturity. This is the approach we take at Dataminded when delivering strategic data consultancy and also with <a href="https://github.com/conveyordata/data-product-portal" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI3LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2dpdGh1Yi5jb20vY29udmV5b3JkYXRhL2RhdGEtcHJvZHVjdC1wb3J0YWwiLCJ0ZXh0IjoiPHN0cm9uZz5Qb3J0YWw8L3N0cm9uZz4iLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiIiLCJ1cmxBZGRpdGlvbiI6IiJ9">Portal</a>, a new open-source data product management tool that provides a guided process for creating data products and that enables governance by design for data initiatives.
<h1 dir="ltr">Conclusion</h1>
As with any new announcement, there's excitement around the AWS releases for opportunities to get more out of our data initiatives. The challenge lies in how we approach these advances. By viewing them through the lens of data product thinking and process, we can better prioritize and focus on standardized approaches that help teams collaborate effectively and deliver value consistently.
<h1 dir="ltr">TL;DR</h1>
For leaders that embrace the decentralized governed approach of data products, several announcements made at&nbsp;AWS re:Invent 2024 show promising alignment with data product thinking.
<ul>
<li dir="ltr">Amazon SageMaker's Data and AI Governance features could automate quality measurements for data products</li>
<li dir="ltr">S3 Tables' support for Apache Iceberg could make data product interfaces more standardized and interoperable</li>
<li dir="ltr">Enhanced data lineage features could help track relationships between data products</li>
</ul>
These developments suggest the industry is moving toward more structured approaches to data management, though implementation success will still depend on clear organizational processes.

For leaders that embrace the decentralized governed approach of data products, several announcements made at AWS re:Invent 2024 show promising alignment with data product thinking.

AWS re:Invent 2024: A Data Leader's Summary 

AWS Re:Invent 2024: A Data Leader's Summary

When teams first adopt Databricks, they typically start with a "yes to everything" approach. Data scientists get full access to explore, experiment, and innovate. This works well&mdash; until it doesn't. Three of the most common scenarios we see:
<ul>
<li>Data usage becomes untraceable</li>
<li>Compliance requirements&nbsp;get overlooked</li>
<li>Security policies&nbsp;become hard to enforce</li>
</ul>
Even with the Unity Catalog, organizations struggle to answer basic questions.. Who is using what data? Why are they using it (for which project?) Are they following our organisation's compliance rules?
<a href="https://data-minded-webinars.webinargeek.com/bringing-governance-to-databricks-enable-self-service-without-losing-control?cst=conveyorwebsite" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2RhdGEtbWluZGVkLXdlYmluYXJzLndlYmluYXJnZWVrLmNvbS9icmluZ2luZy1nb3Zlcm5hbmNlLXRvLWRhdGFicmlja3MtZW5hYmxlLXNlbGYtc2VydmljZS13aXRob3V0LWxvc2luZy1jb250cm9sP2NzdD1jb252ZXlvcndlYnNpdGUiLCJ0ZXh0IjoiTkVXISBSZWdpc3RlciBmb3Igb3VyIHVwY29taW5nIHdlYmluYXIgJmd0OyZndDsgQ1RSTC1aIFRvIERhdGEgQ2hhb3M6IEJyaW5naW5nIFNlbGYtU2VydmljZSBHb3Zlcm5hbmNlIFRvIERhdGFicmNpY2tzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">NEW!</a><a href="https://data-minded-webinars.webinargeek.com/bringing-governance-to-databricks-enable-self-service-without-losing-control?cst=conveyorwebsite" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2RhdGEtbWluZGVkLXdlYmluYXJzLndlYmluYXJnZWVrLmNvbS9icmluZ2luZy1nb3Zlcm5hbmNlLXRvLWRhdGFicmlja3MtZW5hYmxlLXNlbGYtc2VydmljZS13aXRob3V0LWxvc2luZy1jb250cm9sP2NzdD1jb252ZXlvcndlYnNpdGUiLCJ0ZXh0IjoiTkVXISBSZWdpc3RlciBmb3Igb3VyIHVwY29taW5nIHdlYmluYXIgJmd0OyZndDsgQ1RSTC1aIFRvIERhdGEgQ2hhb3M6IEJyaW5naW5nIFNlbGYtU2VydmljZSBHb3Zlcm5hbmNlIFRvIERhdGFicmNpY2tzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ=="> CTRL-Z To Data Chaos: Bringing Self-Service Governance To Databricks</a>
There is a real cost to governance failures, it's not only about a box left unchecked. It's seen in valuable time spent tracking down data usage, in the increased risk of violations that can trigger serious penalties, in&nbsp; security vulnerabilities and, maybe most frustratingly for those involved, in duplicate (or often triplicate, quadruplicate or more) work across teams.
<h2>Building Effective Governance in Databricks</h2>
The first impulse is to fully restrict access, and this would be the easiest for one side - the compliance side. But it's not feasible. Effective governance is that which effectively supports the business, not hinders it. So we need to try to make governance part of the natural workflow of companies. And there are multiple ways in which we can adapt&nbsp; Databricks governance strategies to make them work for both sides.
<h3>1. Clear Data Ownership in Databricks</h3>
Data ownership often becomes unclear as teams grow and projects multiply. So successful organizations address this by establishing clear ownership patterns early on. They define not just who owns each dataset, but also how that ownership translates into practical responsibilities.
For example, we can assign dataset owners based on business domains rather than technical teams. These owners become responsible for defining usage policies and approving access requests. More importantly, they track how their data is being used across different projects, ensuring compliance with regulatory requirements while maintaining team agility.
<h3>2. Self-Service Governance in Databricks</h3>
Traditional access management often creates bottlenecks, with IT teams becoming overwhelmed by access requests. Modern governance approaches attempt to solve this by looking at embedding access controls into the daily workflow of data teams.
A practical implementation could look like this:
<ul>
<li>Structure data access per data product output</li>
<li>Self-service interface for access requests</li>
<li>Automate workflows routing requests to appropriate approvers</li>
<li>Regular data access reviews</li>
</ul>
This approach typically reduces access request processing time from months or weeks to hour or minutes, while again - maintaining strict compliance standards.
<h3>3. Automated compliance tracking in Databricks</h3>
Manual compliance tracking doesn't scale. No manual process ever does. So organizations successful with Databricks governance look at automating compliance processes. Reducing the need for manual input drastically reduces friction between teams and can improve the adoption of any process. Automating in this case can mean:
<ul>
<li>Automatically logging all&nbsp;data access requests</li>
<li>Tracking data lineage across all projects</li>
<li>Generating compliance reports automatically</li>
<li>Setting up automated alerts for potential policy violations</li>
</ul>
We could imagine how implementing these can lead to significantly less time spent on routine checks and to increased focus on strategic governance improvements.
<h2>Having Our Cake And Eating It Too : Innovation PLUS Governance</h2>
It's worth repeating the key to successful governance here - it's&nbsp; finding the right balance between control and flexibility. This flexibility is vital to maintain innvation and ensure a healthy growth. Organizations that get this right understand that they need to give data scientists freedom but also that they need to do it within defined boundaries. This can be achieved by looking at automating routine governance tracks, effectively making compliance the path of least resistance, and enabling quick access to approved data sources, which is what data scientists often struggle the most with.
We often talk to Data Leaders that struggle with effective governance and understand that things aren't on the right track but aren't sure where to look at. So it's important to first clearly understand the specifics of our current setup. Do we know who's using our critical data? Can teams easily request and get access to data? Are we meeting compliance requirements without manual work?
If we find ourselves answering "no" to any of these, it might be time to rethink our approach to Databricks governance.
<a href="https://data-minded-webinars.webinargeek.com/bringing-governance-to-databricks-enable-self-service-without-losing-control?cst=conveyorwebsite" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2RhdGEtbWluZGVkLXdlYmluYXJzLndlYmluYXJnZWVrLmNvbS9icmluZ2luZy1nb3Zlcm5hbmNlLXRvLWRhdGFicmlja3MtZW5hYmxlLXNlbGYtc2VydmljZS13aXRob3V0LWxvc2luZy1jb250cm9sP2NzdD1jb252ZXlvcndlYnNpdGUiLCJ0ZXh0IjoiTkVXISBSZWdpc3RlciBmb3Igb3VyIHVwY29taW5nIHdlYmluYXIgJmd0OyZndDsgQ1RSTC1aIFRvIERhdGEgQ2hhb3M6IEJyaW5naW5nIFNlbGYtU2VydmljZSBHb3Zlcm5hbmNlIFRvIERhdGFicmNpY2tzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">NEW! Register for our upcoming webinar &gt;&gt; CTRL-Z To Data Chaos: Bringing Self-Service Governance To Databrcicks</a>
Start with an honest mapping of the data landscape to identify critical datasets, document current usage patterns and list compliance requirements. Once there's a clear and shared understanding of the actual data landscape and the requirements of all business functions, move to implementing self-service governance:
<ul>
<li>Teams request access through a simple interface</li>
<li>Approvals follow&nbsp;predefined rules</li>
<li>Usage&nbsp;and requests are tracked automatically</li>
</ul>
Data scientists get to keep their flexibility, compliance teams get their oversight and everyone saves time.
<h2>The Path Forward: Bring Self-Service Governance To Databricks</h2>
We've seen this scenario play out often at customers so we've built a tool to help solve these problems. The Data Product Portal is a fully open-source project and free to use. It lets you continue using your preferred data science and analytics tools like Databricks, while offering self-service automated governance. It's best suited for data teams within businesses that aim to build their own use cases in a self-service way, embracing data product thinking.
So imagine this: two teams, each with four data scientists. Team A is building a demand forecasting model and Team B is developing a pricing optimization system.
Team B needs Team A's forecast outputs. Their options are:
<ul>
<li>Request&nbsp;access to Team A's entire&nbsp;workspace (security risk)</li>
<li>Have Team A manually&nbsp;export and share results (time-consuming)</li>
<li>Create duplicate data pipelines (inefficient and error-prone)</li>
</ul>
They risk spending more time solving access issues than improving their models.
When using Portal, the organisation enables Team B to access only the specific forecast outputs they need, see the data lineage (where the data comes from) and get automated access to updates. One of the biggest benefits to working within a data product framework is that it makes accessing the latest version of the data product output the default modus operandi. Meanwhile, Portal empowers Team A to maintain control of their workspace while sharing only the necessary outputs, track who uses their data and automate access management.
In a nutshell, the key point we want to make is that effective Databricks governance doesn't mean choosing between innovation and control. It means making governance simple enough that teams want to follow it by automating the boring parts and reducing friction through self-service capabilities.
Check out <a href="https://github.com/conveyordata/data-product-portal">the project's repo</a> for more details, reach out to us anytime on our <a href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2r47r0icj-PAYbbklmH6gPz9A6bJ144Q">Slack channnel</a> or <a href="../../../../book-demo" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjI2LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiJhcGk6OnBhZ2UucGFnZSIsInRhcmdldENvbnRlbnRUeXBlSWQiOjUzLCJ1cmwiOiJodHRwczovL2NvbnZleW9yZGF0YS5jb20vYm9vay1kZW1vIiwidGV4dCI6ImJvb2sgYSBndWlkZWQgZGVtbyIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6Imh0dHBzOi8vY29udmV5b3JkYXRhLmNvbSIsInVybEFkZGl0aW9uIjoiIn0=">book a guided demo</a>, we'd love to show you around.

Data scientists love Databricks for its power and flexibility. But as organizations scale their Databricks implementations, a pattern emerges: the very freedom that makes Databricks powerful can become its biggest governance challenge.

The Hidden Cost of Databricks Freedom: Balancing Innovation with Governance

As a platform engineer, you will succeed when offering more than "a bag of tools" to business users. Offer technology-independent UIs that suit the particular needs of your company.

Give Your Business Users A Friendly Interface To Your Data Platform

Latest Blogs

imec is the world’s leading independent nanoelectronics R&D hub and the home of more than 5,500 people. As data is core to good research, imec had the ambition to roll-out a broad range of data & analytics capabilities company-wide.

imec

Latest Cases

Learn how Conveyor provides you with all the tools needed as a data engineer and how your development gets accelerated with templates for the most popular frameworks. It minimizes the time setting up infrastructure and maximizes the time available to create value from data.

How to use Conveyor as a data engineer

The multitude of tools designed to solve specific data challenges can really be hard to set up and maintain and is a nightmare for your end user. In this video, we will explain how Conveyor overcomes this challenge.

How Conveyor fits in your data landscape

Data is created in all corners of your organization and there is no longer a single person that knows all data available within your company. Learn how Conveyor addresses this challenge with the introduction of project-based access management for data.