Pricing example

Use the following steps to estimate the Conveyor and instance costs for a particular workload. 

Jobs and environments

Let’s say you have two environments (staging and production). Each environment runs 10 DAGs with each one an average of 20 tasks. We will calculate the monthly costs for only the production environment were the jobs run daily.

In our example, the 20 tasks are split into 2 types:

• 10 Spark jobs running one mx.small driver and five mx.medium executors for 30 min.

• 10 container jobs running one mx.medium instance for 10 min.

Compute the number of mx.medium hours

Conveyor proposes multiple job compute types. In order to simplify calculations, we convert any types's number of hours of compute back to equivalent mx.medium hours. By definition, this type has 1 cpu core and 4 GiB mem.

The mx.small has 0.5 cpu cores, so this means that 1 hour of this instance is equivalent to 0.5 hours of an mx.medium.

Let’s do it for the production environment.

Spark hours

Spark executor: 10 DAGs * 10 tasks * 5 executors * 1/2hour* 30days = 7500 mx.medium hours

Spark driver: 10 * 10 * 1/2 hours *30 = 1500 mx.small hours = 750 mx.medium hours.

Container hours

Container jobs: 10 * 10 * 1/2 hours *30 = 1500 mx.medium hours

Total hours

Monthly you will use 9750 mx.medium hours.

Conveyor license costs

The Conveyor license costs are computed in core-hours. Because the mx.medium has a single cpu core, you can reuse the total amount of mx.medium hours without any additional multiplication.

For the production environment this means that 9750 instance hours will cost you 487.5 EUR in licensing cost for one month using the Conveyor Team plan.

Cloud cost

An mx.medium job is a Conveyor concept. The underlying Kubernetes cluster itself can run on many types of instances. Kubernetes will scale the number of actual worker nodes for you based on the required number of cpu cores and memory. This means that for an estimation, you will need to roughly convert the mx.medium (1 cpu, 4 GiB mem) to the actually provisioned cloud instance.

Let’s say on AWS a mx.medium job is running on a m6i.mxlarge (4 cpu, 16 GiB mem) instance with an hourly cost of 0.214 EUR. A single mx.medium job hour is then roughly equivalent to 0.25 of a m6i.mxlarge hour, thus costing 0.0535 EUR per hour.

For our production environment, this means that the 9750 instance hours results in a cost of 521.63 EUR when running on-demand.

If you would run on spot instances, we can reduce this cost up to 90%, but let’s take a more conservative 70% reduction: 156.48 EUR.

This principle is the same on any cloud provider.

Total cost

The total conveyor cost is the sum of the license and the cloud cost.

Our monthly total cost for the production environment would be 1009.13 EUR when running on-demand, and 643.98 EUR when running on spot instances.