The AI control tower for data infrastructure on AWS

Jobs cost pennies.
Infrastructure is the bill.

SparkPilot deploys the infrastructure your Spark jobs run on, shuts down idle capacity, and governs every run that touches it. Per-run reconciliation against your AWS bill is in early access.

Deploys a dedicated EKS + EMR cluster with Spot-first executor capacity — or connects to the cluster you already run in minutes via CloudFormation Launch Stack.
Governs before it spends: IAM simulation (EMR on EKS today; see limitations), policy and quota gates, and a budget hard-block that stops an over-budget run before dispatch.
Stops idle spend with AWS-native controls and reconciles per-run cost against your CUR (early access) to prove what you saved.

Request pilotView pilot guide

Coming soon: billed-truth reconciliation GA, verdict CLI + CI check, Airflow & Dagster providers GA, agent-safety API.

preflight.yaml

checks:
  - iam_simulation: passed
  - irsa_binding: passed
  - spot_capacity: passed
  - budget_guardrail: BLOCKED
    reason: "Team Q3 spend would exceed $12,000 limit"
    action: "Reduce executor count or request budget increase"

status: rejected_before_dispatch
estimated_cost_usd: 0.00

20+

Preflight checks before every dispatch

Spark runtimes in one control plane

5-step

Guided pilot to production

Pilot Evaluation

What you can review before rollout

Start with a live demo, then use pilot artifacts to align technical, security, and buyer stakeholders.

Live guided demo

Available now

Walk through submission, preflight, run tracking, and diagnostics with your team and workload shape.

Request pilot

Pilot screenshot pack

In beta

Redacted screenshots and run summaries are shared during active pilot evaluations.

Request pilot evidence pack

Recorded walkthrough clips

Coming soon

Short onboarding and run-operations clips are coming soon.

See upcoming features in your pilot call

The Problem

What teams replace in week one

Platform teams running Spark on shared EKS clusters hit the same operational bottlenecks. SparkPilot replaces manual run prep with a governed workflow.

Manually validate IAM trust policy and OIDC association before every job

SparkPilot checks 20+ conditions automatically and fails fast with remediation steps

Hunt through 6 different CloudWatch log groups to find your job output

Each run links directly to the correct log stream

Estimate cost in a spreadsheet after the job finishes

Estimated cost is recorded per run, and actuals can be reconciled against CUR billing data when configured

Engineers share a cluster namespace with no isolation or quota enforcement

Each team gets scoped access, resource quotas, and budget guardrails

Capabilities

Core capabilities for pilot and rollout

These are the capabilities teams use first. Each card includes an availability label so teams can plan rollout clearly.

Preflight Safety Gates

Available now

IAM, IRSA, OIDC, resource quota, and Spot capacity checks run before a single byte moves. Bad configs are blocked with clear remediation steps.

CUR-Aligned Cost Attribution

Early access

Each run records an estimated cost and reconciles against AWS Cost and Usage Report data in Athena when CUR integration is configured.

Multi-Tenant Isolation

Available now

Tenants, teams, environments, and runs are fully scoped. Each environment gets its own namespace, IRSA bindings, and resource quotas.

Governance and Audit

Available now

Role-based access is enforced across SparkPilot APIs, with team-environment scopes, budget guardrails, and audit events for key control-plane actions.

Budget Guardrails

Available now

Set monthly budget limits per team with warn thresholds — and a hard block at the budget line that stops an over-budget run before dispatch.

Structured Diagnostics

Available now

When a run fails, SparkPilot classifies the cause such as OOM kill, Spot interruption, S3 access denied, timeout, or user error.

Run Lifecycle

How SparkPilot handles run operations

Teams submit through the API or CLI (Airflow and Dagster providers in preview). SparkPilot manages dispatch, state reconciliation, and diagnostics so operators do not stitch together raw AWS calls.

queued

dispatching

accepted

running

succeeded · failed · cancelled · timed_out

Scheduler

Polls for queued runs and dispatches them to EMR on EKS today. Manages concurrency limits and environment-level queueing.

Reconciler

Continuously polls EMR for job state changes and writes structured transitions from accepted to running to succeeded or failed.

Provisioner

Manages environment lifecycle for BYOC-Lite today. Full BYOC (VPC + EKS + EMR Terraform) is a roadmap item.

Supported Engines

One control plane for EMR on EKS, Serverless, and EC2

SparkPilot routes submissions to EMR on EKS today, with beta coverage for EMR Serverless and EMR on EC2. Databricks routing is planned as a coming-soon extension.

EMR on EKS

Available now

Native EMR virtual cluster on your EKS cluster for production Spark workloads.

EMR Serverless

Coming soon

Planned routing to an EMR Serverless application. Dispatch client exists; end-to-end provisioning, reconciliation, and cancel propagation are still being validated.

EMR on EC2

Coming soon

Planned step-submission routing to existing EMR on EC2 clusters. Dispatch client exists; the reconciler and cancel path do not yet route to this engine.

Databricks on AWS

Coming soon

Planned support for Databricks Jobs API routing from the SparkPilot control plane.

How It Works

From pilot kickoff to rollout in five steps

Step 1

Define pilot scope

Align on one workload family, success criteria, and owner roles before setup starts.

Open the pilot guide

Step 2

Connect your AWS account

Create the cross-account IAM role and OIDC association. SparkPilot validates trust, permissions, and namespace prerequisites.

Step 3

Choose deployment model

BYOC-Lite connects to your existing EKS cluster quickly. Full BYOC provisioning from Terraform modules is on the roadmap.

Step 4

Submit your first governed run

Encode submission patterns as versioned templates, including Spot configurations, Graviton preferences, S3 paths, and Spark baselines.

Step 5

Review outcomes and decide rollout

Compare pilot results against your success criteria, then move to production rollout with the same control plane.

Integrations and Interfaces

Use SparkPilot from orchestrators, terminal, or API

SparkPilot supports workflow engines and engineer-first interfaces, so teams can adopt it through existing DAGs, CI pipelines, and terminal-driven operations.

Apache Airflow

Preview: SparkPilotSubmitRunOperator with full deferrable trigger support, installable from source for pilot DAGs — sync or async.

Operator | Hook | Sensor | Async Trigger

Dagster

Preview: native @asset definitions and ops for run submission, polling, and cancellation, installable from source.

Assets | Ops | Config Schema

SparkPilot CLI

Engineers can submit, inspect, cancel, and tail runs from terminal workflows without opening the dashboard.

run-submit | run-list | run-logs | usage-get

SparkPilot API

Teams integrate SparkPilot into internal portals and automation through authenticated REST endpoints, including OAuth2 client_credentials for machine clients.

REST API | M2M Auth | RBAC | Audit Trail

Airflow and Dagster providers are in preview, installable from source today. API and CLI interfaces are provided for active pilot workflows and automation.

Open integration guide

Why SparkPilot

What you don't get with DIY or EMR Serverless

DIY gives you primitives. EMR Serverless removes cluster management. Neither gives you a multi-tenant control plane with built-in governance.

Capability	DIY on AWS	EMR Serverless	SparkPilot
Preflight IAM/OIDC validation
Multi-tenant namespace isolation on EKS
Per-team cost attribution (estimates today; CUR reconciliation in early access)
Budget guardrails with a pre-dispatch hard block at the budget line
Spot diversification validation at preflight
Airflow and Dagster providers (preview)
Kubernetes-native control plane
No infra management required