Conveyor Product & Features

Let’s start with what we don’t cover: you should bring your own storage of data. Conveyor will fulfill most of your other data platform requirements. Conveyor is build on top of Kubernetes and uses Airflow as an orchestrator for all your data pipelines.

To kick-start your data processing jobs, you can start from our dbt or Spark templates, which can run on the Kubernetes cluster. Want to use something else? Not a problem! If you can wrap it in a Docker container, we can run it for you. This also allows you to use the data ingestion framework of your choice within Conveyor.

With our online IDEs, we offer a readily available development environment, which includes notebooks to interact with data in your experimentation phase. Moving from idea, over experimentation to production-great code is a major part of our solution. But we don’t stop there: Conveyor simplifies your deployment cycle and allows you to easily integrate our commands within your CI/CD pipeline.

Yes, but… Conveyor is not a no-code platform, but we do acknowledge that mastering all intricacies of data engineering can be overwhelming. Conveyor aims to take away a lot of the complexity of the common data engineering frameworks. Through our templates, you can benefit from a structure which simplifies your code creation. The heavy-lifting of then getting your code in production? That is our strength!

All our underlying tooling is Open Source. We believe that you want to avoid the hassle of setting everything up, integrating and maintaining all these tools. But you can do this yourself and step away from Conveyor if desired. You can move your Airflow DAGs to your own Airflow and host Spark and dbt yourself. Be aware that you will lose our great CLI, the Conveyor plugin for Airflow and our focus on developer experience.

We track cost per environment, think of a development, production, or single team environment, and project. This allows you to perfectly know who is spending what and evaluate whether this is acceptable.

Keeping track is one thing, but making it actionable is what matters. We have a relentless focus on minimizing node overhead, to get the best out of your resources. We allow you to pick t-shirt sized resources to optimally fulfill the needs of your jobs, and enable you to choose spot instances over on-demand pricing to limit costs of your environments and data pipelines.

Conveyor is running on a Kubernetes cluster in your own cloud account. This Kubernetes cluster is used to host your Airflow environments as well as to perform your data processing. The use of the cluster to host Airflow is a non-negotiable, but for processing services, you can bring your own. Willing to run transformations within your warehouse or bring your own VMs to run a custom data transformation framework? It’s all up to you.

Conveyor gets installed in a cloud account which you provide. Next to that, Conveyor has a control plane which runs in a separate cloud account, either hosted within a cloud environment of ours, or again at your side. Updates are initiated by the Conveyor team. Through this control plane, we can update everything tagged as Conveyor within your cloud account. If the control plane runs at our side, we will push all updates automatically and you will always have the latest version available. If the control plane runs at your end on the other hand, we agree on timings to update the solution.

Partially that’s up to you. But no worries, we can help you with that. With the use of our templates, you will have a solid structure that separates Airflow DAGs from actual code and kick-starts the development of both. By the way, we also include a structure to add tests to your code, don’t forget to benefit from this best-practice!

The Conveyor CLI offers a set of powerful deployment commands. Think about Conveyor build, deploy, promote, … These commands can be triggered from your CI/CD pipeline, bringing the full power of Conveyor to your CI/CD process.

This immediately means that you can also bring more automation to Conveyor: automatic builds on commits, running tests at PR’s, deploys at merges. All is available.

Yes, you can, and yes, you can. Conveyor has been built by Data Minded, a respected data engineering consultancy firm and has initially been sold in combination with consultancy. But we have moved beyond that phase. The largest Conveyor clients use Conveyor completely independent, without any consultancy. But of course, we stay linked to Data Minded and through them, Conveyor experts are available to kick-start your adoption.

Well, yesterday. No, but when you are already using Airflow, self-managed or cloud-managed, and dbt or Spark. Things can go incredibly fast. Porting your code to Conveyor is as easy as pointing towards the code repository. Some refactoring to adhere to our structure will need to happen. But at one of our first customers, we saw the first use-cases being ported in a matter of days.