01 / SPARK CONTROL PLANE
Change Spark with proof.
Connect AWS, run under policy and budget guardrails, diagnose failures, understand cost, and prove business results before cutover — all inside your own account.
02 / THE PROBLEM
A runtime upgrade that finishes is not a runtime upgrade that worked.
Row counts match while a decimal(38,6) → decimal(38,1) cast silently changes every aggregate underneath them. That is not carelessness on the left — it is the most careful thing available without a comparator. A useful control plane must be able to say no. SparkPilot's approval means something because it can withhold approval.
03 / PROOF
We seeded a rounding difference to prove the comparator can fail.
Every figure in this record is recomputable from the evidence trace. The negative control is deliberate: a comparator that never returns a failure is not a comparator. The clean run of the same workload returns EQUIVALENT, 10 of 10 matching.
CHECKS
q1_input_sample
CHECKS
q1_pricing_summary
04 / THE OPERATING LOOP
Assurance isn't a separate tool. It's the last stage of the loop.
The same control plane that dispatches the run is the one that measures it, compares it, and decides whether it may be promoted.
- 01CONNECTCross-account role, scoped.
- 02RUNGates, then dispatch.
- 03OBSERVEState, duration, cost state.
- 04DIAGNOSECause, evidence, remediation.
- 05CHANGEBaseline and candidate, paired.
- 06VERIFYDifferences are named.
- 07PROMOTE / BLOCKA verdict, with provenance.
05 / THE PRODUCT
Twelve surfaces. One control plane, one audit path.
Runtime-change assurance is the sharpest proof point, not the product. Everything below submits through the same preflight, the same budget gate, and the same audit trail.
06 / GOVERNANCE
The cheapest run is the one that never dispatched.
Policies, budgets, and scopes are evaluated at preflight — before AWS spends a cent. A hard block rejects the submission; a soft warning still lands in the audit trail.
07 / WHERE RESULTS LAND
A verdict nobody sees isn't governance.
Subscribe a channel to run and system events. Delivery is admin-configured and the webhook is stored encrypted.
08 / THE ENVIRONMENT
We can build it. We can delete it. Or you can bring your own.
Three engines, three genuinely different shapes of commitment. The control stays with your engineers in all three.
Whichever you choose, the control plane only ever assumes a role into your account. Compute, data, and evidence never leave it.
And in every path, the boundary is the same.
SparkPilot dispatches and measures. The compute, the data, and the evidence stay in the account they already live in.
Engines today: EMR on EKS, EMR Serverless, EMR on EC2.
09 / COST
Money is decided at four moments. Three of them happen before the run.
A dashboard shows you what you already spent. Most of the leverage is upstream of it — in the engine you pick, the shape of the create call, and the hours nobody is watching.
Engine choice, priced — while the choice is still free.
At environment creation, before a dollar is committed, the product prints the monthly floor of each engine with the arithmetic attached. For a low-volume team this single screen is the largest lever in the product.
+ smallest practical always-on capacity
You pay for the seconds a job runs.
Cost begins when a cluster does.
Cost-correct by construction.
Cost correctness is a property of the create call, not something the customer has to remember afterwards. This is how SparkPilot's own EMR Serverless create is written; the product today attaches to an application you bring.
Cheaper by default, and it refuses the unbounded case.
The money nobody watches.
We measured a real idle Spark environment for a month. It cost $230.53. Five cents of that was Spark.
This bill does not get smaller when you run less. It is what the cluster costs to exist, and you pay it once per cluster — dev, staging, prod, per team, per region. At 30 clusters, the per-cluster lines above come to $6,072 a month of standing charge before a single job is submitted — an illustration of how a per-cluster charge scales, not a measurement of anyone's estate. The account-level remainder is excluded, because it does not recur once per cluster.
Two thirds of that is the node lever, and it is only available if this cluster does not need high availability — one system node is a single point of failure for cluster DNS. A cluster that must stay highly available avoids $27.96 a month, not $76.73.
Pull every lever and this cluster still bills $153.80 a month while idle. SparkPilot surfaces these three with the command to run; you execute them in your own account, because performing them needs permissions we do not ask for — disabling audit logging requires a permission that would also let us expose your Kubernetes API server. Retention alone does not remove the log charge: it is ingestion, not storage.
Read your own number rather than ours: it is split_line_item_unused_cost in your Cost and Usage Report. That is the field this page is built on, and it is the field that would prove us wrong.
When your EKS version leaves standard support, the cluster control-plane fee goes up six times.
Extended support adds $0.50 per hour to the $0.10 cluster fee: $0.10 to $0.60 per cluster-hour, charged whether or not anything is running. Your Spark compute is billed exactly as before — this is the control-plane fee, not the workload. We warn you 90 days out, with the exact date.
10 / INTEGRATES INTO YOUR STACK
SparkPilot does not replace your orchestrator.
Your scheduler keeps scheduling. The gates, the cost state, and the evidence come from here. Four interfaces into the same control plane.
Run:ai gates chips. Kueue gates cores. Nobody gates dollars.
Everything below is a destination, not a feature. It is on this page because a roadmap you can read is worth more than one you have to ask for — and because we would rather you hold us to it.
The same verdict, dispatch and reconcile path applied to training and inference spend. GPU quota tools today are denominated in chips, not dollars, and none of them reconcile against the bill. That gap is the reason this is first on the list.
11 / PRICING
Four ways in. The gaps are printed.
A dash means the capability is not in that tier. We would rather lose the deal than discover it during the pilot.
| CAPABILITY | Scan Free | Assessment $12,500 | Assurance from $24,000/yr | Platform from $750/mo |
|---|---|---|---|---|
| Portability review of job definitions and source | ✓ Included | ✓ Included | ✓ Included | ✓ Included |
| Constructs with no target equivalent, and what they cost to fix | ✓ Included | ✓ Included | ✓ Included | ✓ Included |
| Execution on both source and target platform | Not included | ✓ Included | ✓ Included | Not included |
| Row-level output comparison | Not included | ✓ Included | ✓ Included | Not included |
| Measured wall clock and resource consumption, both sides | Not included | ✓ Included | ✓ Included | Not included |
| Cutover recommendation with coverage counts | Not included | ✓ Included | ✓ Included | Not included |
| Re-checked on every runtime or config change | Not included | Not included | ✓ Included | Enterprise rung |
| Governed dispatch in your own AWS account | Not included | Not included | Not included | ✓ Included |
| Preflight, policy and quota gates before dispatch | Not included | Not included | Not included | ✓ Included |
| Per-run audit trail and failure diagnostics | Not included | Not included | Not included | ✓ Included |
- Portability review of job definitions and source
- Constructs with no target equivalent, and what they cost to fix
- Execution on both source and target platform — not included
- Row-level output comparison — not included
- Measured wall clock and resource consumption, both sides — not included
- Cutover recommendation with coverage counts — not included
- Re-checked on every runtime or config change — not included
- Governed dispatch in your own AWS account — not included
- Preflight, policy and quota gates before dispatch — not included
- Per-run audit trail and failure diagnostics — not included
- Portability review of job definitions and source
- Constructs with no target equivalent, and what they cost to fix
- Execution on both source and target platform
- Row-level output comparison
- Measured wall clock and resource consumption, both sides
- Cutover recommendation with coverage counts
- Re-checked on every runtime or config change — not included
- Governed dispatch in your own AWS account — not included
- Preflight, policy and quota gates before dispatch — not included
- Per-run audit trail and failure diagnostics — not included
- Portability review of job definitions and source
- Constructs with no target equivalent, and what they cost to fix
- Execution on both source and target platform
- Row-level output comparison
- Measured wall clock and resource consumption, both sides
- Cutover recommendation with coverage counts
- Re-checked on every runtime or config change
- Governed dispatch in your own AWS account — not included
- Preflight, policy and quota gates before dispatch — not included
- Per-run audit trail and failure diagnostics — not included
- Portability review of job definitions and source
- Constructs with no target equivalent, and what they cost to fix
- Execution on both source and target platform — not included
- Row-level output comparison — not included
- Measured wall clock and resource consumption, both sides — not included
- Cutover recommendation with coverage counts — not included
- Re-checked on every runtime or config change — Enterprise rung
- Governed dispatch in your own AWS account
- Preflight, policy and quota gates before dispatch
- Per-run audit trail and failure diagnostics
12 / GETTING STARTED