SparkPilot
Request pilot

01 / AI SPARK ENGINEER

Give your Spark engineer a job.

Describe what you need. SparkPilot builds or adapts the workload, prepares the environment in your AWS account, runs it, and keeps the result operating.

Request pilotTry the engineerRead the evidence record
RECORDED SESSION · 3 NOV 2026

Recorded on SparkPilot's own workspace. A SparkPilot team member asked in Control Tower and confirmed both proposals on their cards; the run succeeded on EMR Serverless, and the last answer is the engineer's own report of it.

02 / THE JOB

Hand over the task. The engineer handles the steps.

TODAY
Find the environment. Configure the bucket. Write the job. Submit the run.
WHO DOES THE WORK / YOUR TEAM, ONE SCREEN AT A TIME
WHAT YOU SEE / THE PIECES, IN DIFFERENT PLACES
WITH SPARKPILOT
INSPECTPREPAREBUILDRUN
WHO DOES THE WORK / SPARKPILOT, ONE ASSIGNMENT
WHAT YOU SEE / THE CODE, THE CHANGE, THE RUN, THE OUTPUT

It asks you only what needs your judgment. Change your mind later and it proposes the code change, shows you the exact diff, and waits for your approval.

03 / PROOF

SparkPilot compares a change against the old results.

A separate validation example. One TPC-H workload ran on two EMR releases, and we deliberately seeded a rounding difference into the candidate run so the comparison had something to catch. It found 5 findings and withheld approval. The same workload without the seeded difference returns EQUIVALENT, 10 of 10 matching.

The evaluation as SparkPilot shows it, on its Evaluations page.

04 / THE OPERATING LOOP

Compare results before cutover.

The same control plane that dispatches the run is the one that measures it, compares it, and decides whether it may be promoted.

  1. 01
    CONNECT
    Cross-account role, scoped.
  2. 02
    RUN
    Gates, then dispatch.
  3. 03
    OBSERVE
    State, duration, cost state.
  4. 04
    DIAGNOSE
    Cause, evidence, remediation.
  5. 05
    CHANGE
    Baseline and candidate, paired.
  6. 06
    VERIFY
    Differences are named.
  7. 07
    PROMOTE / BLOCK
    A verdict, with provenance.

05 / THE PRODUCT

Run and review Spark from one workspace.

Manage workspaces, workloads, runs, costs, and evaluations in one place.

CONTROL TOWER
Ask in plain language. Answers cite the workspace evidence you can see, and nothing changes without your approval.
MISSION BRIEF
Five setup steps, checked on every visit: session, identity, AWS account, first run, and first evaluation.
WORKLOADS
Every saved job, its last outcome, and what it cost.
RUNS
Preflight → dispatch → live telemetry → retry. On every engine, one board.
COSTS
Standing infrastructure cost, idle-spend levers, cost-allocation tags, estimate → reconciled.
NOTEBOOKS
Write Spark in-product; submits through the same preflight, budget gate, and audit path as every run.
EVALUATIONS
Change verdicts: output equivalence and cutover recommendation kept separate.
MIGRATIONS
Databricks portability assessment: inventory jobs, scan source, leave with a remediation list.
POLICIES
Preflight guardrails: hard-block rejects the submission, soft-warn records to the audit trail. Scoped global → workspace → environment.
ACCESS & GOVERNANCE
User identities, teams, team↔environment scopes, budget guardrails with warn and block thresholds.
NOTIFICATIONS
Slack delivery for run and system events, admin-only, webhook stored encrypted. Email behind an operator flag.
INTEGRATIONS
CLI, REST API, and the Airflow and Dagster providers in preview.
REVIEW THE CHECKS BEFORE THE VERDICT

06 / GOVERNANCE

Check rules before dispatch.

Policies, budgets, and scopes are evaluated at preflight, before AWS spends a cent. A hard block rejects the submission; a soft warning still lands in the audit trail.

The checks SparkPilot recorded before it dispatched the run in the conversation at the top of this page.

07 / WHERE RESULTS LAND

Send results to your team.

Subscribe a channel to run and system events. Delivery is admin-configured and the webhook is stored encrypted.

run.failed · run.timed_out · run.dispatch_failed
run.preflight_failed · run.schedule_submission_failed
evaluation.evaluated · evaluation.revalidation_required
activation.refused

08 / THE ENVIRONMENT

Choose how Spark runs in your account.

Three engines, three genuinely different shapes of commitment. The control stays with your engineers in all three.

Spark execution and datasets stay in your AWS account. SparkPilot stores run records and notebooks created in the app. See data handling.

PATH 01
EMR Serverless
NO CLUSTER TO MANAGE
Run on an EMR Serverless application without managing a cluster.
Pre-initialized workers and interactive sessions can keep capacity running between jobs. Check the application's capacity and auto-stop settings.
PROVISIONING
CONNECT YOURS OR LET SPARKPILOT CREATE ONE
IDLE COST
DEPENDS ON CAPACITY
AUTO-STOP
15 MIN ON APPS WE CREATE
TEARDOWN
APPLICATION AUTO-STOP
PATH 02
EMR on EKS
WE BUILD IT, OR YOU BRING IT
One click builds the whole environment in five resumable stages, and one action destroys it.
Or point us at the EKS cluster you already run, and we use it as it is. Cheapest at steady state with spot pricing.
IF WE BUILD IT · FIVE RESUMABLE STAGES
01
NETWORK
VPC, two AZs, a NAT gateway per AZ.
02
EKS
1.35 in Auto Mode, IRSA and Pod Identity.
03
EMR
Virtual cluster with a scoped execution role.
04
BOOTSTRAP
Validate what was built before using it.
05
RUNTIME
A real job runs before it is handed over.
CAPACITY
DRIVERSON-DEMAND
EXECUTORSSPOT-FIRST
Two Karpenter node pools. No static node groups. Karpenter owns capacity, so there is nothing to keep warm.
TEARDOWN
terraform destroy
Every resource the build created, removed in reverse. No orphaned NAT gateway quietly billing for a year.
PATH 03
EMR on EC2
LONG-LIVED, AND YOURS TO SHAPE
For teams that want the cluster to persist and want to own instance selection.
SparkPilot detects a missing EMR auto-termination policy and recommends a setting for you to apply.
CLUSTER LIFETIME
YOURS TO DECIDE
IDLE COST BETWEEN CLUSTERS
CHECK RETAINED RESOURCES
AUTO-TERMINATION
DETECTED · NATIVE AWS POLICY

And in every path, the boundary is the same.

SparkPilot dispatches and measures. The compute, the data, and the evidence stay in the account they already live in.

PROVISIONING / TERRAFORM · RESUMABLE
ACCESS / ASSUMEROLE · EXTERNALID
AUTH / M2M OAUTH

Engines today: EMR on EKS, EMR Serverless, EMR on EC2.

09 / COST

Control costs before and after each run.

A dashboard shows you what you already spent. Most of the leverage is upstream of it: in the engine you pick, the shape of the create call, and the hours nobody is watching.

MOMENT 01
BEFORE YOU COMMIT

Compare engine costs before choosing.

Compare each engine's cost drivers before choosing a runtime. Monthly cost depends on region, capacity, runtime and retained resources.

EMR on EKS
Cluster bills hourly
EKS cluster fees depend on the support tier.
Worker capacity and EMR jobs add charges.
Price the cluster, workers and jobs
EMR Serverless
Bills while it works
Pre-initialized workers can bill while idle.
Check capacity, sessions and auto-stop.
Price vCPU, memory, storage and runtime
EMR on EC2
Bills until terminated
Cluster charges stop at termination.
S3 data and other retained AWS resources may continue to bill.
Per-cluster hours, priced on create
MOMENT 02
BEFORE A RUN

Check idle settings.

Connect an EMR Serverless application in your AWS account. Preflight reads its capacity and auto-stop settings when access permits.

·Pre-initialized workers can bill while the application is started.
·Auto-stop limits the idle period before the application stops.
·Check maximum capacity before submitting jobs.
·Review network and storage charges in your AWS configuration.
·Cost-allocation tags require activation and billing-data reconciliation before they support per-run actuals. For EKS, see AWS split cost allocation data.
MOMENT 03
WHILE IT RUNS

Capacity and cost controls

·EMR on EKS can place executors on Spot capacity when available.
·Arm64 estimates use the selected engine's rate card.
·Karpenter can consolidate unused nodes when configured.
·The console confirms a finite rate-card estimate before submission.
·Hard-budget checks reserve estimated costs for admitted runs.
·Quotas and configured YuniKorn queue limits are checked before dispatch.
MOMENT 04
WHILE IT SITS IDLE

Find what is sitting idle.

SparkPilot reports whether idle EKS Auto Mode node-pool capacity is being reclaimed, and what it is worth. On environments SparkPilot builds, admins can stop or restore it.

A customer-installed stack starts in ReportOnly mode; scheduled stops require an opt-in cluster tag.

Notebook sessions default to a 30-minute idle stop, configurable per environment.

90 DAYS AHEAD OF THE CLIFF

When your EKS version leaves standard support, the cluster control-plane fee goes up six times.

Extended support adds $0.50 per hour to the $0.10 cluster fee: $0.10 to $0.60 per cluster-hour, charged whether or not anything is running. Your Spark compute is billed exactly as before. This is the control-plane fee. We warn you 90 days out, with the exact date.

EKS 1.34 · STANDARD SUPPORT ENDS
2026-12-02
TODAY$0.10/hr · $73/mo
AFTER$0.60/hr · $438/mo
Control-plane fee only, per cluster, us-east-1 list price at 730 hours/month. Dates are UTC, per the AWS release calendar. Version dates and rates read from the AWS EKS and Pricing APIs on 2026-09-18.
6×
AND WHAT WE WON'T DO
NO FABRICATED $0.00
An unknown renders unavailable. A cost that reads $0.00 because we weren't allowed to look is indistinguishable from one that reads $0.00 because nothing was spent.
GROSS, NOT NET
AWS credits covered all of it, so you were charged $0.00. Credits are finite, and this is the figure you'll pay when they run out.
AWS AUTO-TERMINATION
On EMR on EC2, SparkPilot detects a missing auto-termination policy and offers to enable the AWS policy.

10 / INTEGRATES INTO YOUR STACK

Connect your existing orchestrator.

Your scheduler keeps scheduling. The gates, the cost state, and the evidence come from here. Four interfaces into the same control plane.

01
Request access to the private provider source.
Airflow preview access
02
Install from the granted checkout on your scheduler.
pip install -e providers/airflow
03
Add a connection with your API address and machine credentials, then use the operator.
SparkPilotSubmitRunOperator(job_id=…)
ACCESS
Preview source is private. A real scheduler-driven run remains unverified.
Request Airflow preview ↗
10 / WHAT IS NEXTROADMAP · NOT SHIPPED

Future workload support

GPU AND INFERENCE WORKLOADS
ROADMAP

We plan to extend run controls and cost attribution to training and inference workloads.

RAY / KUBERAY
ROADMAP
Distributed Python beside Spark, under the same policy and budget gates.
SAGEMAKER
ROADMAP
Job governance for training and processing runs.
BEDROCK
ROADMAP
Budget controls for token and inference spending.
DATABRICKS
ROADMAP
Jobs API routing, so a governed run does not care which platform executes it.
SNOWFLAKE
ROADMAP
Warehouse spend inside the same reconciliation.
VERDICT CLI / GITHUB ACTION
ROADMAP
The gate at pull-request time, before the DAG ever merges.
AGENT-SAFETY API
ROADMAP
The same pre-execution gate, exposed to agents rather than people.

11 / GETTING STARTED

Assess your next Spark change.

Request pilotTry the engineerGetting startedRead the evidence record
SPARKPILOTSPARK CHANGE WITH EVIDENCE