Knowledge

Organisations that want to have self-service data product teams working on many use cases in parallel are adopting data product thinking. 

Published on: 2 Oct 2024

How To Structure Data For Self-Service Data Teams

In the fast-evolving world of data, companies are discovering that the key to success for scaling their data initiatives is not to rely on a single data team to handle all requests from the business. Instead they want to enable self-service capabilities to allow each domain or department to build their own data products at their own pace with their own budgets.
However, putting this vision into practice can be very challenging, especially when you put all aspects related to data governance, data platforms and data catalogs&nbsp;together. It can become messy very quickly as it is hard to provide an easy to understand consistent view across all these dimensions and technologies.
Today we are very happy to announce the&nbsp;Data Product Portal&nbsp;&mdash; an open-source tool to help organisations build and manage data products at scale. It&rsquo;s intuitive, flexible, and geared towards making data product management straightforward and effective.
<h1 id="9533" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">What problem does it solve</h1>
Imagine you&rsquo;re building a data pipeline. You take some input data, process it with python, DBT or another tool and generate output for others to use. This scenario applies whether you&rsquo;re using Starburst with Ranger for access controls, or other platforms technologies like Snowflake, AWS, Databricks, BigQuery or Microsoft Fabric.
<img src="https://media.prod.dataminded-dev.nl/1_Noue_Lmz_Vd_L_9n_Vp_Vr6h_UVA_6ec23f20b3.webp" alt="1_NoueLmzVdL_9nVpVr6hUVA.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Typical data pipeline consuming remote data, data from files and databases</figcaption>
</figure>
&nbsp;
In large organisations, you typically have multiple departments or domains, and you can&rsquo;t simply share all data between all departments by default due to legal, compliance, regulatory or confidentiality reasons. This means that departments or domains have to keep control how and why their data is being used by others.
<img src="https://media.prod.dataminded-dev.nl/1_K_Mqruo_H_Tid_Stx_Ky_K_Vr_AHNQ_2dd73e0f41.webp" alt="1_KMqruoHTidStxKyKVrAHNQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Scaling data pipelines across a large organisation with multiple departments</figcaption>
</figure>
&nbsp;
The moment you start sharing data across multiple departments or domains, you are immediately faced with the following questions:
&nbsp;
<blockquote>
Who has access to what data, when can they use it and for what reason are they using it?
</blockquote>
If you are working with data, you want to know where to access and find the data and tooling to build your data pipelines.
To answer these questions, companies start data governance initiatives, where they start generating policies per user to manage access to data in tools like Ranger, AWS IAM, Snowflake or any other tool. When people have been working on multiple use cases, all of these policies accumulate together. This &ldquo;spaghetti&rdquo; of permissions will lead towards very broad access permissions for a large group of people. After a while you will end up in the situation you wanted to avoid in the first place where:
&nbsp;
<blockquote>
Everybody ends up with access to all data and you no longer know why and how your data is being used.
</blockquote>
<h1 id="a357" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm pe no np nq pf ns nt nu pg nw nx ny bj" data-selectable-paragraph="">Data products as a governance model</h1>
There are many different definitions out there about what is a data product and every organisation or person working with data has their own opinion about it, but we found the following definition to be useful and quite universal for many organisational structures.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_ji_BF_p71h_Wo_YE_5_Mki_XG_Ntg_0bd0979863.webp" alt="1_jiBF_p71hWoYE5MkiXGNtg.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Definition of a data product</figcaption>
</figure>
&nbsp;
We propose to define a&nbsp;data product&nbsp;as: an initiative with a clear goal, owned by a department or domain of the business that consists of the combination of:
<ol class="">
<li id="8b22" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Input data: Access to datasets created by a combination of other data products.</li>
<li id="47a8" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Output data: Read/write access to output data that can be combined into a dataset that can be shared with other data products. This data is stored in specific locations (e.g. databases or buckets).</li>
<li id="1ea3" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Private data: A safe location to store private/internal data for local processing with no intention of sharing</li>
<li id="1f27" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Transformation artefacts: All code and outputs describing your transformations, scheduling and tooling configuration needed to build, access and run your data pipeline separated from other data products.</li>
<li id="5481" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Team roles: Defined roles of team members that have specific permissions on how to interact with the data product (e.g. data product owner, data engineer, business analyst).</li>
</ol>
&nbsp;
<blockquote class="op">
It is important to note that people can work on multiple data products, but have to choose on what data product they are working on. They only get access to that data product scope related to tooling and data.
</blockquote>
In this definition, data products are not only data assets that are the output for sharing with others, but also the tooling, artefacts and roles of people that interact with that data product.
When multiple data products start interacting, your data product lineage will start looking like this:
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_p_Vu_Poa_NU_Ja_Ckgn9_i31mw_6224405452.webp" alt="1_pVu-PoaNUJaCkgn9-i31mw.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">How different data products are interacting with each other</figcaption>
</figure>
&nbsp;
The main benefits of this approach are:
<ul class="">
<li id="39a5" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz pu pi pj bj" data-selectable-paragraph="">Clear Data Usage: You always know who is using your data and why.</li>
<li id="10c1" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Simplified Access Management: Easier to handle access requests and data revocations</li>
<li id="34ed" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz pu pi pj bj" data-selectable-paragraph="">Natural Data Lineage:&nbsp;Understand how your data flows from one data product to another.</li>
</ul>
Adopting the data product governance model if very powerful in scaling data initiatives across departments, while still keeping control and self-service capabilities. However, effective tools are essential for managing the processes around this to make it truly self-service.
<h1 id="fbcc" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">Introducing the Data Product Portal</h1>
The&nbsp;Data Product Portal&nbsp;is a practical tool that helps you build data products at scale, that is both useful for both people working with data and people who want to have control over how their data is being used.
It is designed to simplify the creation and management of data products at scale. It is beneficial and easy to use for both data professionals and those overseeing the use of data and data governance.
&nbsp;
<img src="https://media.prod.dataminded-dev.nl/1_Ph_T_Gak_S_Vpw_A8_Oi5_P_Ki13_YQ_56d99d19ee.webp" alt="1_PhTGakSVpwA8Oi5PKi13YQ.webp">
<figure class="of og oh oi oj lw lo lp paragraph-image">
<figcaption class="ok ol om lo lp on oo be b bf z dw" data-selectable-paragraph="">Data product model translation to policies and configuration</figcaption>
</figure>
&nbsp;
This is where the Data Product Portal comes into play:
<ol class="">
<li id="706e" class="mc md ft me b mf mg mh mi mj mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz ph pi pj bj" data-selectable-paragraph="">Guided Setup: Step-by-step assistance involving the right stakeholders for creating data products, requesting access, adding users and registering new data for sharing with other data products.</li>
<li id="57cf" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Tech Translation: Converts high-level concepts into specific configurations settings for platforms like AWS, Azure, Databricks, Snowflake, and others, making sure that each data product is correctly separated and not impacting each other.</li>
<li id="79df" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">User-Friendly Interface: Makes it easy for business users and people working with data to understand and navigate the data landscape.</li>
<li id="174b" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Self-service:&nbsp;Enables departments and teams to start new data initiatives easily without having to depend on a central team.</li>
<li id="5061" class="mc md ft me b mf pk mh mi mj pl ml mm mn pm mp mq mr pn mt mu mv po mx my mz ph pi pj bj" data-selectable-paragraph="">Comprehensive Overview: Combines data catalogs, data platforms and data governance aspects into a single 360 overview of all ongoing data initiatives.</li>
</ol>
<h1 id="7eb9" class="nb nc ft be nd ne nf ng nh ni nj nk nl nm nn no np nq nr ns nt nu nv nw nx ny bj" data-selectable-paragraph="">How to get started</h1>
The Data Product Portal is available as an open source project on&nbsp;<a class="af pv" href="https://github.com/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">Github</a>. Getting started is as simple as running&nbsp;docker compose up&nbsp;and visiting&nbsp;localhost:8080
We invite you to check us out and give us a star if you like what you see. Your contributions are invaluable to us &mdash; whether it&rsquo;s through feedback, suggestions, or direct involvement in development.
For Kubernetes deployments, check out our helm chart&nbsp;<a class="af pv" href="https://gallery.ecr.aws/conveyordata/data-product-portal" target="_blank" rel="noopener ugc nofollow">here</a>.
If you have questions or want to share your thoughts? Join our&nbsp;<a class="af pv" href="https://join.slack.com/t/dataproductportal/shared_invite/zt-2lo245t3l-bJxcAz1trqrcQNnGARWMDg" target="_blank" rel="noopener ugc nofollow">community on Slack</a>&nbsp;and connect with us directly. We can&rsquo;t wait to hear from you!
We are excited to bring the Data Product Portal into your hands as an open source initiative. Don&rsquo;t hesitate to share with us what you think about it!
&nbsp;

Scale data initiatives across your entire organisation while enabling self-service and maintaining governance.

Introducing Data Product Portal

Introducing The Data Product Portal: A New Open-Source Tool For Data Governance

<blockquote>
A data platform is about enabling an organization to work with data at scale, in a self-service manner.
</blockquote>
That is how Kristof Martens, tech lead at multiple of <a href="https://www.dataminded.com/" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5kYXRhbWluZGVkLmNvbS8iLCJ0ZXh0IjoiRGF0YW1pbmRlZCdzIiwidHlwZSI6ImV4dGVybmFsIiwiZG9tYWluIjoiIiwidXJsQWRkaXRpb24iOiIifQ==">Dataminded's</a> customers, describes a data platform. That is also what Wim Vancuyck, data lead at <a href="https://www.imec-int.com/en" target="_blank" rel="noopener" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5pbWVjLWludC5jb20vZW4iLCJ0ZXh0IjoiaW1lYyIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">imec</a>, envisions in his quest to enable more than 5000 researchers to work with data at imec, the world&rsquo;s leading independent nano-electronics R&amp;D hub. We recently sat together and discussed why companies should actually not build their own internal data platform. These are our main conclusions.
<h1 class="pw-post-body-paragraph mz na fv nb b gt nc nd ne gw nf ng nh ni nj nk nl nm nn no np nq nr ns nt my fo bj">The impact of company culture</h1>
Let&rsquo;s start with explaining the different factions that we typically see while deciding on a data platform: business, IT and governance. Typically business teams want to go fast and have something available yesterday. That&rsquo;s why they mostly lean towards buying off the shelf tooling: click install and off you go.
IT departments on the other hand focus on standardization and development best-practices, and tend to lean towards building a solution, as it offers you more flexibility and control. And let&rsquo;s be honest: building a data platform often looks easy at the start. The third party, the governance team, has a more narrow view on a data platform: a data catalog will solve all problems. In the end, it&rsquo;s the faction with most power in the company that influences its buy vs build mentality.
Funny enough, at imec it&rsquo;s the other way around: the R&amp;D department exists of people that are used to coding, which led to building and rebuilding quite often. The IT department on the contrary is small compared to the amount of researchers, loves to work with partners and buy tools to scale fast and introduce standardization. It&rsquo;s all about avoiding re-inventing the wheel over and over again by all those researchers.
This buy mentality, combined with the desire to keep costs under control, the love for open source, and the preference to avoid vendor lock-in, has led to a modular approach to the data platform at imec. This data platform consists of multiple&nbsp;<a class="af nu" href="https://azure.microsoft.com/en-us" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL2F6dXJlLm1pY3Jvc29mdC5jb20vZW4tdXMiLCJ0ZXh0IjoiQXp1cmUiLCJ0eXBlIjoiZXh0ZXJuYWwiLCJkb21haW4iOiIiLCJ1cmxBZGRpdGlvbiI6IiJ9">Azure</a>&nbsp;components, combined with tools like&nbsp;<a class="af nu" href="https://www.mathworks.com/products/matlab.html" target="_blank" rel="noopener ugc nofollow" data-internal-link="eyJpZCI6bnVsbCwic291cmNlQ29udGVudFR5cGVVaWQiOiJhcGk6OnBvc3QucG9zdCIsInNvdXJjZUNvbnRlbnRUeXBlSWQiOjE4LCJzb3VyY2VGaWVsZE5hbWUiOiJsaW5rIiwidGFyZ2V0Q29udGVudFR5cGVVaWQiOiIiLCJ0YXJnZXRDb250ZW50VHlwZUlkIjpudWxsLCJ1cmwiOiJodHRwczovL3d3dy5tYXRod29ya3MuY29tL3Byb2R1Y3RzL21hdGxhYi5odG1sIiwidGV4dCI6Ik1hdGxhYiIsInR5cGUiOiJleHRlcm5hbCIsImRvbWFpbiI6IiIsInVybEFkZGl0aW9uIjoiIn0=">Matlab</a>, or Conveyor. Conveyor can be considered as the cornerstone, enabling multiple open source technologies as a managed services and properly glued together. In theory, this modular approach allows imec to migrate the platform to other cloud providers, again mitigating a vendor lock-in.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">The pitfalls of building your own data platform</h1>
&ldquo;We can build this ourselves!&rdquo; A common heard argument to build your own data platform, and probably the authors are right. The question is not whether you can build your own data platform, but whether you want or should do so.&nbsp;At imec they have the guideline
<blockquote>
Buy where you can, build where you want to make the difference.
</blockquote>
Often the platform is not where you make the difference: what you use it for is the value add.
Remember that a data platform is about enabling an organization to work with data at scale in a self-service manner. This raises a lot of questions, like:
<ul class="">
<li id="9c4d" class="mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my qa qb qc bj" data-selectable-paragraph="">How will I provide proper separation between teams and projects?</li>
<li id="fe53" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How can I encourgag best-practices?</li>
<li id="8dd7" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I share data?</li>
<li id="3365" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I keep track of governance and compliance concerns?</li>
<li id="b9e1" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">How do I implement a data product lifecycle?</li>
</ul>
It is hard to come up with an answer to all these questions beforehand. If you want to have all of this prior to a first use-case, this can easily take some years. Glueing tested and proven components together, as imec does, speeds up this process drastically. After the consideration phase, imecs data platform was up and running in just 4 months. Even more: within the same 4 months the first 14 use-cases were live.
Conveyor was chosen to avoid reinventing the wheel. It offered a set of open standards which were on imecs wishlist, and they were already properly integrated. Imec could have done this themselves, but they acknowledged the fact that this has been done before. Due to experience, others might be better at it and could take the long-term maintenance out of their hands. This long-term maintenance might be the most neglected pitfall of building your own data platform: if you build it yourself, you need to maintain it yourself.&nbsp;Often platform teams get bogged down in operational issues after a while and can no longer add any new functionality.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Scaling your data platform</h1>
When talking about scaling a data platform, we immediately think about scaling the infrastructure but scaling a data platform also involves:
<ul>
<li>Increasing the amount of use cases running on top of it.</li>
<li>The number of people&nbsp;working with it and the knowledge required&nbsp;to work on it.</li>
<li id="da21" class="mz na fv nb b gt qd nd ne gw qe ng nh ni qf nk nl nm qg no np nq qh ns nt my qa qb qc bj" data-selectable-paragraph="">The ability to securely share data between use cases.</li>
</ul>
<blockquote>
Imec is a good example that technically scaling the platform is not the biggest challenge, scaling people is.
</blockquote>
It is hard to onboard people to work with data. Both internally where you need to sell the platform as externally where you need to attract experts. This is one of the reasons they prefer buying platforms and components, to avoid hiring a lot of people in the data platform team. Data workers might be getting infused in the business teams, a consequence of an ongoing data mesh discussion. A small data platform team should enable them to focus on value creation with standards, guidelines and paved roads. This renewed focus is the new IQ, efficience is key.
This separation of responsibilities is recognized by Kristof amongst many companies. A data platform team will take care of shared services, compute, infrastructure and data access, allowing use case teams to focus on building and maintaining the data use cases themselves. It&rsquo;s the platforms team responsibility to sell the platform to more and more people internally. You can only do this with a proper customer-centric mindset.
<h1>The cost of innovation</h1>
Sometimes standardization is considered the opposite of innovation. The goal of imec is to balance both. Yes, the IT department wants to introduce standards, but researchers value freedom and flexibility. The key is having a good offering tailored to each user profile.&nbsp;Remember the customer-centric mindset of the platform team? The users at imec are strongly demanding paved roads/standardization within the tools of their preference because it takes away complexity, it becomes an enabler, and even an accelerator for innovation.
The modular approach at Imec limits the dependency on one single partner. The open standards offered by Conveyor or the interchangeable Azure components allow to change them when desired with new, innovative tools over time.
But what does this approach cost? Make a business case when buying a tool and compare it to the cost building it. However, do not forget the maintenance cost on the long run. Even open source technologies cost money as you need to keep maintaining their installation and keep upgrading them. Imec argues that the tools they have bought have a lower Total Cost of Ownership than a build approach. In order to keep costs under control they focus on avoiding licenses without added value as well as idle resources.
As a small remark, Kristof notices that many companies find it easier to accept the people cost as opposed to the tooling cost. Hiring extra people often does not require a procurement process and rarely the business case of people versus licenses is made.
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Security and compliance</h1>
Off the shelf data platforms usually have a sensible story that covers most if not all security aspects. Companies want to have control over who can access which data, who can alter production processes, how to enable software engineering best-practices, how to comply to regulatory requirements, &hellip;
When buying a tool, it is mostly a matter of translating the concepts that these data platforms offer to the existing company processes. As this translation is never a 100% match, companies often favor building a custom data platform than updating the company processes.
Kristof advocates that it makes sense to change the company processes to the concepts of a data platform. Again these are proven concepts, distilled from experiences at many different customers. The agility of your organization heavily impacts the feasibility of this change. Yet, if your level of agility does not allow for this, do you believe your company is the right one to build a tool itself?
<h1 class="pw-post-body-paragraph mz na fv nb b gt ow nd ne gw ox ng nh ni oy nk nl nm oz no np nq pa ns nt my fo bj">Looking back</h1>
Imec&rsquo;s approach has always been to buy where possible and to build where you can make the difference. However they do not believe in buying a Suisse knife that can do everything and probably comes with the cost of a Rolls Royce. Therefor imec takes a modular approach, which requires you to write glue code between the different selected components. This is where Conveyor came in: a proven technology. An added benefit is that Imec does not need to maintain the components, nor the glue between them.
Overall, imec is very happy with the choices made. The world of technology and data is moving fast, which means that over time you might no longer have the best platform possible. This does not imply that you made the wrong choices, but rather stresses the importance of building a modular platform. With this approach, you have the flexibility to change components when desired.
A data platform is there to scale the self-service usage of data throughout your organization. We notice that it&rsquo;s more difficult to scale the number of users than your infrastructure. Your data platform should be customer-centric in order to convince more and more people to use it. A great user experience is crucial. A small data platform team focusing on this enablement, frees up your data workers to focus on value adding use-caes.&nbsp;Furthermore, their development with the use of paved roads is what allows you to reach your data ambitions.

Summary notes from a round-table webinar on imec's approach to their data platform with Wim Vancuyck, Kristof Martens and Wannes Rosiers.

Why NOT Build Your Own Data Platform

Latest Blogs

imec is the world’s leading independent nanoelectronics R&D hub and the home of more than 5,500 people. As data is core to good research, imec had the ambition to roll-out a broad range of data & analytics capabilities company-wide.

imec

Latest Cases

Learn how Conveyor provides you with all the tools needed as a data engineer and how your development gets accelerated with templates for the most popular frameworks. It minimizes the time setting up infrastructure and maximizes the time available to create value from data.

How to use Conveyor as a data engineer

The multitude of tools designed to solve specific data challenges can really be hard to set up and maintain and is a nightmare for your end user. In this video, we will explain how Conveyor overcomes this challenge.

How Conveyor fits in your data landscape

Data is created in all corners of your organization and there is no longer a single person that knows all data available within your company. Learn how Conveyor addresses this challenge with the introduction of project-based access management for data.