Blogs

Organisations that want to have self-service data product teams working on many use cases in parallel are adopting data product thinking. 

Published on: 2 Oct 2024

How To Structure Data For Self-Service Data Teams

In the fast-evolving world of data, companies are discovering that the key to success for scaling their data initiatives is not to rely on a single data team to handle all requests from the business. Instead they want to enable self-service capabilities to allow each domain or department to build their own data products at their own pace with their own budgets.
However, putting this vision into practice can be very challenging, especially when you put all aspects related to data governance, data platforms and data catalogs&nbsp;together. It can become messy very quickly as it is hard to provide an easy to understand consistent view across all these dimensions and technologies.
Today we are very happy to announce the&nbsp;Data Product Portal&nbsp;&mdash; an open-source tool to help organisations build and manage data products at scale. It&rsquo;s intuitive, flexible, and geared towards making data product management straightforward and effective.
<h1 id="9533" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">What problem does it solve</h1>
Imagine you&rsquo;re building a data pipeline. You take some input data, process it with python, DBT or another tool and generate output for others to use. This scenario applies whether you&rsquo;re using Starburst with Ranger for access controls, or other platforms technologies like Snowflake, AWS, Databricks, BigQuery or Microsoft Fabric.
<img src="https://media.prod.dataminded-dev.nl/1_Noue_Lmz_Vd_L_9n_Vp_Vr6h_UVA_6ec23f20b3.webp" alt="1_NoueLmzVdL_9nVpVr6hUVA.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Typical data pipeline consuming remote data, data from files and databases</figcaption>
</figure>
&nbsp;
In large organisations, you typically have multiple departments or domains, and you can&rsquo;t simply share all data between all departments by default due to legal, compliance, regulatory or confidentiality reasons. This means that departments or domains have to keep control how and why their data is being used by others.
<img src="https://media.prod.dataminded-dev.nl/1_K_Mqruo_H_Tid_Stx_Ky_K_Vr_AHNQ_2dd73e0f41.webp" alt="1_KMqruoHTidStxKyKVrAHNQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Scaling data pipelines across a large organisation with multiple departments</figcaption>
</figure>
&nbsp;
The moment you start sharing data across multiple departments or domains, you are immediately faced with the following questions:
&nbsp;
<blockquote>
Who has access to what data, when can they use it and for what reason are they using it?
</blockquote>
If you are working with data, you want to know where to access and find the data and tooling to build your data pipelines.
To answer these questions, companies start data governance initiatives, where they start generating policies per user to manage access to data in tools like Ranger, AWS IAM, Snowflake or any other tool. When people have been working on multiple use cases, all of these policies accumulate together. This &ldquo;spaghetti&rdquo; of permissions will lead towards very broad access permissions for a large group of people. After a while you will end up in the situation you wanted to avoid in the first place where:
&nbsp;
<blockquote>
Everybody ends up with access to all data and you no longer know why and how your data is being used.
</blockquote>
<h1 id="a357" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm pe no np nq pf ns nt nu pg nw nx ny bj" data-selectable-paragraph="">Data products as a governance model</h1>
There are many different definitions out there about what is a data product and every organisation or person working with data has their own opinion about it, but we found the following definition to be useful and quite universal for many organisational structures.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_ji_BF_p71h_Wo_YE_5_Mki_XG_Ntg_0bd0979863.webp" alt="1_jiBF_p71hWoYE5MkiXGNtg.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Definition of a data product</figcaption>
</figure>
&nbsp;
We propose to define a&nbsp;data product&nbsp;as: an initiative with a clear goal, owned by a department or domain of the business that consists of the combination of:
<ol class="">
<li id="8b22" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Input data: Access to datasets created by a combination of other data products.</li>
<li id="47a8" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Output data: Read/write access to output data that can be combined into a dataset that can be shared with other data products. This data is stored in specific locations (e.g. databases or buckets).</li>
<li id="1ea3" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Private data: A safe location to store private/internal data for local processing with no intention of sharing</li>
<li id="1f27" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Transformation artefacts: All code and outputs describing your transformations, scheduling and tooling configuration needed to build, access and run your data pipeline separated from other data products.</li>
<li id="5481" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Team roles: Defined roles of team members that have specific permissions on how to interact with the data product (e.g. data product owner, data engineer, business analyst).</li>
</ol>
&nbsp;
<blockquote class="op">
It is important to note that people can work on multiple data products, but have to choose on what data product they are working on. They only get access to that data product scope related to tooling and data.
</blockquote>
In this definition, data products are not only data assets that are the output for sharing with others, but also the tooling, artefacts and roles of people that interact with that data product.
When multiple data products start interacting, your data product lineage will start looking like this:
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_p_Vu_Poa_NU_Ja_Ckgn9_i31mw_6224405452.webp" alt="1_pVu-PoaNUJaCkgn9-i31mw.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">How different data products are interacting with each other</figcaption>
</figure>
&nbsp;
The main benefits of this approach are:
<ul class="">
<li id="39a5" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz pu pi pj bj" data-selectable-paragraph="">Clear Data Usage: You always know who is using your data and why.</li>
<li id="10c1" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Simplified Access Management: Easier to handle access requests and data revocations</li>
<li id="34ed" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Natural Data Lineage:&nbsp;Understand how your data flows from one data product to another.</li>
</ul>
Adopting the data product governance model if very powerful in scaling data initiatives across departments, while still keeping control and self-service capabilities. However, effective tools are essential for managing the processes around this to make it truly self-service.
<h1 id="fbcc" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">Introducing the Data Product Portal</h1>
The&nbsp;Data Product Portal&nbsp;is a practical tool that helps you build data products at scale, that is both useful for both people working with data and people who want to have control over how their data is being used.
It is designed to simplify the creation and management of data products at scale. It is beneficial and easy to use for both data professionals and those overseeing the use of data and data governance.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_Ph_T_Gak_S_Vpw_A8_Oi5_P_Ki13_YQ_56d99d19ee.webp" alt="1_PhTGakSVpwA8Oi5PKi13YQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Data product model translation to policies and configuration</figcaption>
</figure>
&nbsp;
This is where the Data Product Portal comes into play:
<ol class="">
<li id="706e" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Guided Setup: Step-by-step assistance involving the right stakeholders for creating data products, requesting access, adding users and registering new data for sharing with other data products.</li>
<li id="57cf" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Tech Translation: Converts high-level concepts into specific configurations settings for platforms like AWS, Azure, Databricks, Snowflake, and others, making sure that each data product is correctly separated and not impacting each other.</li>
<li id="79df" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">User-Friendly Interface: Makes it easy for business users and people working with data to understand and navigate the data landscape.</li>
<li id="174b" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Self-service:&nbsp;Enables departments and teams to start new data initiatives easily without having to depend on a central team.</li>
<li id="5061" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Comprehensive Overview: Combines data catalogs, data platforms and data governance aspects into a single 360 overview of all ongoing data initiatives.</li>
</ol>
<h1 id="7eb9" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">How to get started</h1>
The Data Product Portal is available as an open source project on&nbsp;<a class="af pv" href="https://github.com/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">Github</a>. Getting started is as simple as running&nbsp;docker compose up&nbsp;and visiting&nbsp;localhost:8080
We invite you to check us out and give us a star if you like what you see. Your contributions are invaluable to us &mdash; whether it&rsquo;s through feedback, suggestions, or direct involvement in development.
For Kubernetes deployments, check out our helm chart&nbsp;<a class="af pv" href="https://gallery.ecr.aws/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">here</a>.
If you have questions or want to share your thoughts? Join our&nbsp;<a class="af pv" href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2lo245t3l-bJxcAz1trqrcQNnGARWMDg" target="_blank" rel="noopener ugc nofollow">community on Slack</a>&nbsp;and connect with us directly. We can&rsquo;t wait to hear from you!
We are excited to bring the Data Product Portal into your hands as an open source initiative. Don&rsquo;t hesitate to share with us what you think about it!
&nbsp;

Scale data initiatives across your entire organisation while enabling self-service and maintaining governance.

Introducing Data Product Portal

Introducing The Data Product Portal: A New Open-Source Tool For Data Governance

<blockquote>
A data platform is about enabling an organization to work with data at scale, in a self-service manner.
</blockquote>
That is how Kristof Martens, tech lead at multiple of <a href="https://www.dataminded.com/" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5kYXRhbWluZGVkLmNvbS8iLCJ0ZXh0IjoiRGF0YW1pbmRlZCdzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">Dataminded's</a> customers, describes a data platform. That is also what Wim Vancuyck, data lead at <a href="https://www.imec-int.com/en" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5pbWVjLWludC5jb20vZW4iLCJ0ZXh0IjoiaW1lYyIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">imec</a>, envisions in his quest to enable more than 5000 researchers to work with data at imec, the world&rsquo;s leading independent nano-electronics R&amp;D hub. We recently sat together and discussed why companies should actually not build their own internal data platform. These are our main conclusions.
<h1 class="pw-post-body-paragraph mz na fv nb b gt nc nd ne gw nf ng nh ni nj nk nl nm nn no np nq nr ns nt my fo bj">The impact of company culture</h1>
Let&rsquo;s start with explaining the different factions that we typically see while deciding on a data platform: business, IT and governance. Typically business teams want to go fast and have something available yesterday. That&rsquo;s why they mostly lean towards buying off the shelf tooling: click install and off you go.
IT departments on the other hand focus on standardization and development best-practices, and tend to lean towards building a solution, as it offers you more flexibility and control. And let&rsquo;s be honest: building a data platform often looks easy at the start. The third party, the governance team, has a more narrow view on a data platform: a data catalog will solve all problems. In the end, it&rsquo;s the faction with most power in the company that influences its buy vs build mentality.
Funny enough, at imec it&rsquo;s the other way around: the R&amp;D department exists of people that are used to coding, which led to building and rebuilding quite often. The IT department on the contrary is small compared to the amount of researchers, loves to work with partners and buy tools to scale fast and introduce standardization. It&rsquo;s all about avoiding re-inventing the wheel over and over again by all those researchers.
This buy mentality, combined with the desire to keep costs under control, the love for open source, and the preference to avoid vendor lock-in, has led to a modular approach to the data platform at imec. This data platform consists of multiple&nbsp;<a class="af nu" href="https://azure.microsoft.com/en-us" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2F6dXJlLm1pY3Jvc29mdC5jb20vZW4tdXMiLCJ0ZXh0IjoiQXp1cmUiLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiIiLCJ1cmxBZGRpdGlvbiI6IiJ9">Azure</a>&nbsp;components, combined with tools like&nbsp;<a class="af nu" href="https://www.mathworks.com/products/matlab.html" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5tYXRod29ya3MuY29tL3Byb2R1Y3RzL21hdGxhYi5odG1sIiwidGV4dCI6Ik1hdGxhYiIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">Matlab</a>, or Conveyor. Conveyor can be considered as the cornerstone, enabling multiple open source technologies as a managed services and properly glued together. In theory, this modular approach allows imec to migrate the platform to other cloud providers, again mitigating a vendor lock-in.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">The pitfalls of building your own data platform</h1>
&ldquo;We can build this ourselves!&rdquo; A common heard argument to build your own data platform, and probably the authors are right. The question is not whether you can build your own data platform, but whether you want or should do so.&nbsp;At imec they have the guideline
<blockquote>
Buy where you can, build where you want to make the difference.
</blockquote>
Often the platform is not where you make the difference: what you use it for is the value add.
Remember that a data platform is about enabling an organization to work with data at scale in a self-service manner. This raises a lot of questions, like:
<ul class="">
<li id="9c4d" class="mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my qa qb qc bj" data-selectable-paragraph="">How will I provide proper separation between teams and projects?</li>
<li id="fe53" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How can I encourgag best-practices?</li>
<li id="8dd7" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I share data?</li>
<li id="3365" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I keep track of governance and compliance concerns?</li>
<li id="b9e1" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I implement a data product lifecycle?</li>
</ul>
It is hard to come up with an answer to all these questions beforehand. If you want to have all of this prior to a first use-case, this can easily take some years. Glueing tested and proven components together, as imec does, speeds up this process drastically. After the consideration phase, imecs data platform was up and running in just 4 months. Even more: within the same 4 months the first 14 use-cases were live.
Conveyor was chosen to avoid reinventing the wheel. It offered a set of open standards which were on imecs wishlist, and they were already properly integrated. Imec could have done this themselves, but they acknowledged the fact that this has been done before. Due to experience, others might be better at it and could take the long-term maintenance out of their hands. This long-term maintenance might be the most neglected pitfall of building your own data platform: if you build it yourself, you need to maintain it yourself.&nbsp;Often platform teams get bogged down in operational issues after a while and can no longer add any new functionality.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Scaling your data platform</h1>
When talking about scaling a data platform, we immediately think about scaling the infrastructure but scaling a data platform also involves:
<ul>
<li>Increasing the amount of use cases running on top of it.</li>
<li>The number of people&nbsp;working with it and the knowledge required&nbsp;to work on it.</li>
<li id="da21" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">The ability to securely share data between use cases.</li>
</ul>
<blockquote>
Imec is a good example that technically scaling the platform is not the biggest challenge, scaling people is.
</blockquote>
It is hard to onboard people to work with data. Both internally where you need to sell the platform as externally where you need to attract experts. This is one of the reasons they prefer buying platforms and components, to avoid hiring a lot of people in the data platform team. Data workers might be getting infused in the business teams, a consequence of an ongoing data mesh discussion. A small data platform team should enable them to focus on value creation with standards, guidelines and paved roads. This renewed focus is the new IQ, efficience is key.
This separation of responsibilities is recognized by Kristof amongst many companies. A data platform team will take care of shared services, compute, infrastructure and data access, allowing use case teams to focus on building and maintaining the data use cases themselves. It&rsquo;s the platforms team responsibility to sell the platform to more and more people internally. You can only do this with a proper customer-centric mindset.
<h1>The cost of innovation</h1>
Sometimes standardization is considered the opposite of innovation. The goal of imec is to balance both. Yes, the IT department wants to introduce standards, but researchers value freedom and flexibility. The key is having a good offering tailored to each user profile.&nbsp;Remember the customer-centric mindset of the platform team? The users at imec are strongly demanding paved roads/standardization within the tools of their preference because it takes away complexity, it becomes an enabler, and even an accelerator for innovation.
The modular approach at Imec limits the dependency on one single partner. The open standards offered by Conveyor or the interchangeable Azure components allow to change them when desired with new, innovative tools over time.
But what does this approach cost? Make a business case when buying a tool and compare it to the cost building it. However, do not forget the maintenance cost on the long run. Even open source technologies cost money as you need to keep maintaining their installation and keep upgrading them. Imec argues that the tools they have bought have a lower Total Cost of Ownership than a build approach. In order to keep costs under control they focus on avoiding licenses without added value as well as idle resources.
As a small remark, Kristof notices that many companies find it easier to accept the people cost as opposed to the tooling cost. Hiring extra people often does not require a procurement process and rarely the business case of people versus licenses is made.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Security and compliance</h1>
Off the shelf data platforms usually have a sensible story that covers most if not all security aspects. Companies want to have control over who can access which data, who can alter production processes, how to enable software engineering best-practices, how to comply to regulatory requirements, &hellip;
When buying a tool, it is mostly a matter of translating the concepts that these data platforms offer to the existing company processes. As this translation is never a 100% match, companies often favor building a custom data platform than updating the company processes.
Kristof advocates that it makes sense to change the company processes to the concepts of a data platform. Again these are proven concepts, distilled from experiences at many different customers. The agility of your organization heavily impacts the feasibility of this change. Yet, if your level of agility does not allow for this, do you believe your company is the right one to build a tool itself?
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Looking back</h1>
Imec&rsquo;s approach has always been to buy where possible and to build where you can make the difference. However they do not believe in buying a Suisse knife that can do everything and probably comes with the cost of a Rolls Royce. Therefor imec takes a modular approach, which requires you to write glue code between the different selected components. This is where Conveyor came in: a proven technology. An added benefit is that Imec does not need to maintain the components, nor the glue between them.
Overall, imec is very happy with the choices made. The world of technology and data is moving fast, which means that over time you might no longer have the best platform possible. This does not imply that you made the wrong choices, but rather stresses the importance of building a modular platform. With this approach, you have the flexibility to change components when desired.
A data platform is there to scale the self-service usage of data throughout your organization. We notice that it&rsquo;s more difficult to scale the number of users than your infrastructure. Your data platform should be customer-centric in order to convince more and more people to use it. A great user experience is crucial. A small data platform team focusing on this enablement, frees up your data workers to focus on value adding use-caes.&nbsp;Furthermore, their development with the use of paved roads is what allows you to reach your data ambitions.

Summary notes from a round-table webinar on imec's approach to their data platform with Wim Vancuyck, Kristof Martens and Wannes Rosiers.

Why NOT Build Your Own Data Platform

Central data teams aiming to build a data platform and deliver data use-cases at the same time, have been created and are often organized under IT. Business departments on the other hand, have attempted to create value from their data silo by buying tools and hiring consultants as business leaders want results now, not when such a central team can plan them.
A number of waves to re-centralize data teams have occurred, driven by Enterprise Data Warehouses, Company-Wide Data Lakes and many more. Only recently with the rise of data mesh, strategies are adapting to the distributed nature of data. Rather than re-centralizing, large scale companies are decentralizing their data teams, outside of IT, embedded in the business. These teams are empowered with a data platform, still build within IT. What are the benefits of a data platform team?
<h1>Wait what, data is not IT?</h1>
As IT stands for Information Technology, the role of an IT department can become quite broad. As a minimal responsibility, an IT department is responsible for maintaining the hardware and software systems within an organization. Briefly summarized: the IT department makes sure the rest of the company is properly equipped to perform their tasks.
While looking to (semi-)tech companies, product development is often also ranked under IT. These products are build for the company&rsquo;s customers or for internal users, yet development is led by a single department with a clear business goal. Processes are put in place to strengthen the cooperation between the development team and the business leaders.
This is the main distinction with data: data use-cases - machine learning algorithms and predictions, reports, &hellip; - are all individual products and all might be build for different stakeholders. There is no single department or business leader who can easily prioritize amongst them, nor who can capture and translate all business requirements. These data use-cases however, might all be build using the same data platform. It&rsquo;s this data platform that has a common goal: enabling your data workers to reach value from their use-cases.
<h1>A data platform</h1>
But, what is a data platform? For most software components it is high-level very clear: a CRM (Customer Relationship Management) system contains all my customers and manages my relations and possibly my interactions with them, a billing system creates invoices and sends them to customers, &hellip; A data platform on the other hand, well, it does something with data, but what?
A data platform covers a vast amount of capabilities. To be able to valorize data, you most likely want to combine data from multiple of your software systems. None of these software systems are build to contain all your company&rsquo;s data. A data platform therefore needs to store all data and in the first place actually ingest the data. Afterwards you want to apply business transformation logic to this data and structure it for use in reports or train a machine learning algorithm on the data. Those results then need to be presented to a consumer of your insights and updated on a regular basis. All these capabilities combined are what you call your data platform. All these capabilities are often tackled by distinct tools (un)fortunately.
This data platform is the software (and hardware or cloud infrastructure) system people use to massage data and get value out of it.
<h1>In comes the data platform team</h1>
Hence maintaining the data platform is an IT responsibility. Compared to typical software systems like a CRM system, the data platform consists of multiple components and I will refer to the team maintaining all these tools as the &ldquo;Data Platform Team&rdquo;. Should this data platform team be organized within your IT department? Possibly. Not necessarily. And not the focus of the point I would like to make today.
The data platform team should provide and maintain a platform that has the capability of data ingestion, data storage, data transformation, reporting, machine learning, scheduling, &hellip;. Rapidly this platform consists of 6 or more tools. If the platform team needs to maintain all of these tools and integrate them with each other, you can imagine that the team needs to be quite large and requires some time to set everything up. As this does not deliver direct value, rather supports the value creation, you will understand that this will rapidly become a budget challenge.
Therefore, the responsibility of the data platform team is to select the required components, preferably managed services to lower the maintenance, and integrate them, while safeguarding a clear division of capabilities. This should allow a shift in the focus of data workers from building and maintaining a data platform towards delivering value from data.
<h1>A shift of focus to deliver value</h1>
Through the usage of your data platform, your data workers can focus on transforming data, training machine learning models and delivering data products which drive value. It doesn&rsquo;t really matter whether your data workers are still centrally organized or federated in a data mesh set-up: through your data platform, you have liberated your data workers from maintaining their own tools. This allows them to take a deep-dive in the business processes and business goals and search for patterns that build up towards these company goals.

For years, organizations have been building and using data platforms to get value out of data. For years, companies have been struggling to organize to optimize the value creation from data. For years , organizations have failed to maximize the return on their data investments? Why is this, and what should you to, do increase your chances?

The benefits of a data platform team

To state the obvious: a data team is there to bring value to the company. But is it this obvious? Haven&rsquo;t companies too often created a data team out of necessity? Too often data initiatives have started since competitors were doing this, or since doing data sounds sexy. And when the data team is indeed in pursuit of value, how do you measure this? Even more, if you succeed to measure value, how do you organize yourself to optimize the outcome? All valid questions, for which we want to articulate a response today.
<h2>Understanding the Data Team's Raison d'&ecirc;tre</h2>
Let&rsquo;s start with the reason of existence of your data team. Going just a few years back in time, often we&rsquo;ve heard companies saying: &ldquo;we need to do Big Data&rdquo;. But when asking what doing Big Data meant, no one could reply. It&rsquo;s all about V&rsquo;s they said: it&rsquo;s the Variety of data, the Velocity and the Volume of data. But actually, it was all about the hype and what we then considered Big Data is now rather small. That wasn&rsquo;t the first time data was hyped and it won&rsquo;t be the last time neither.
For a long time, the actual reason of existence of a data team, has been to create operational reports, building up to KPI-trees which allow to steer your entire business. This still remains valid and valuable use of data, but it&rsquo;s only the tip of the iceberg. There is more data than ever before, and algorithms and models are popping up everywhere, which leads to an increasing amount of possible use-cases every day.
More and more companies identify these possibilities and succeed in implementing them. Data team tasks shift from doing purely analytical reporting, to embedding solutions in operational workflows and steering automated decision making. It&rsquo;s a shift in time from using data after the facts to prior to events. This requires a whole other approach of organizing your team. Suddenly things like on-call support 24/7 become relevant.
<h2>The needs of a modern data team</h2>
Next to the challenging requirements of availability - data steering operational process should be there - the omni-presence of data also impacts the companies needs with regards to building a data organization. Simply said: as of a certain size, one can not expect that a single person knows all data available in the company, let alone understands all business goals.
In short, to obtain value from data, you need to:
<ul>
<li>Understand the data: knowing the exact meaning of every data field and the business process or input channel that has created it.</li>
<li>Recognize the business goals: know what you want to accomplish, the why of your data work next to the what.</li>
<li>Have sufficient technical data knowledge: master both data engineering or data science technology, as well as methodology like data machine learning models or data modelling.</li>
<li>Keep everything running, all the time: again master technology and methodology, with a stress on keeping infrastructure and tools up and running and preventing models to drift.</li>
</ul>
<h2>Why central data teams fail</h2>
A central, single-disciplinary team won&rsquo;t succeed. You can not expect someone to be the technical expert, both for infrastructure as engineering, as well as the business expert, leaving data model expertise still out of scope.
This is not a data specific challenge and the good thing is: you can learn from others. Within software engineering, you see the rise of cross-functional teams or at least the rise of agile frameworks to increase the connection between technical and business experts.
Elaborating on the similarity between data needs and the rise of cross-functional teams in software engineering, you rapidly end up with a bunch of different roles needed. Some of these roles of course might be taken up by the same person, depending on your context. So what are these roles:
<ul>
<li>Product owner: someone that protects the why</li>
<li>Functional analyst or business process expert: someone that can translate the why into the what</li>
<li>Data expert: someone that understands the input data</li>
<li>Technical data expert: someone that can transform the input data into valuable output. Note that depending on the use-case this can be a data engineer, a data visualization expert or both.</li>
<li>A devops expert: someone that can provide and maintain the tools and build the CI/CD pipeline. Again, this might even be distinct people</li>
</ul>
Central data teams are grouping these technical data experts, possibly in combination with the devops experts. This means you group the people building the data platform with those using it, rather than the data workers with those that can guide the value creation from data. This is exactly why central data teams too often continued to tackle tech challenges rather than business challenges.
<h2>And what to do about it</h2>
As a data team, we can steal the solution with pride from software engineering teams. It&rsquo;s all about bringing the why and what of data together. In cross-functional teams or using agile frameworks: up until a certain size or number of activities (of a company), bringing the business skills closer to a central data team will make sense, as of a certain size or number of activities, federating the data skills will fit better.
The recent introduction of product-thinking to data is both a result of this need to bring why and what together, as well as trigger to create cross-functional teams. This product-thinking has also led to the introduction of platform-thinking, which essentially means that you consider your data platform as a product to be used by your data workers, build and maintained by a separate cross-functional team.
Again up until a certain size (now of the data team), combining the responsibility to manage a data platform and building use-cases is logical - just as in start-ups, where everyone steps in to perform which ever task is needed - yet as of a certain size, focus becomes increasingly relevant and you will need to separate the team in platform team and data value units federated across your organization.

Unlock the true potential of your data team's role in your company. Discover why mere data reporting is outdated, and how to evolve towards proactive data utilization. Learn the key to balancing technical expertise with business acumen, ensuring your team effectively drives value creation. Embrace cross-functional collaboration for sustainable success.

Blogs

How To Structure Data For Self-Service Data Teams

Introducing The Data Product Portal: A New Open-Source Tool For Data Governance

Why NOT Build Your Own Data Platform

The benefits of a data platform team

How to organise a data team to get the most out of data