Built by platform engineers
for platform engineers
SparkPilot exists because teams running Spark on EKS eventually build the same control-plane layer and carry that maintenance forever. We built a faster pilot path and a cleaner rollout path.
The gap nobody was filling
Runtime optimizers like Ocean for Apache Spark make running jobs cheaper. FinOps platforms like Kubecost show you what you spent. Observability tools like Unravel diagnose failures after the fact.
None of them sit at pre-dispatch, the moment before a job starts when you can still stop it. Before IAM misconfigurations waste startup cost. Before a team blows its monthly budget at 2 AM. Before a bad EMR release label causes a silent failure.
That is where SparkPilot sits. SparkPilot checks each submission against governance and cost rules before dispatch.
How we work
We keep product claims grounded in what customers can use now. We would rather be clear and reliable than broad and vague.
Observability tools tell you what went wrong after a job ran. We prevent the bad run from starting. The value is in the gate, not the dashboard.
SparkPilot runs in your AWS account. Your Spark job artifacts, S3 buckets, and VPC stay in your perimeter. BYOC is the foundation.
Data engineers are the end users. Platform engineers are the buyers. We build for the person who has to set up IRSA bindings at midnight, not just the person who submits jobs.
How we got here
First preflight engine shipped with IAM, OIDC, and quota checks before job dispatch.
First production Spark job ran through SparkPilot on a live EKS cluster.
A second platform team completed BYOC-Lite setup and first run end-to-end.
Open-sourced core preflight and provider integrations. Airflow and Dagster providers are available from source.
Want to scope your pilot architecture?
We can walk through architecture details, scope your pilot, and confirm technical fit in one call.