Limitations

What SparkPilot does not do yet

Check the evidence coverage and current product limits before planning a migration.

Migration and assurance scope

The published reference covers one workload and one runtime-upgrade pair.

One workload and one runtime upgrade

The executed migration evidence covers one reference workload on EMR Serverless, upgraded from emr-6.15.0 to emr-7.5.0 in a real AWS account. Its 10 equality assertions matched. Other workloads and migration paths remain unproven.

Jobs inventoried: 0. Source files scanned: 0.

The reference report inventoried no estate jobs and scanned no source files. Early compatibility scans require an engineer; automated customer-estate scanning is unproven.

Other migration paths are unproven

Only the reference source-and-target pair has been executed. A different engine, vendor, or runtime jump needs its own evaluation.

A matching pair did not authorize cutover

The reference pair was equivalent, but its report returned INSUFFICIENT_EVIDENCE because it had no estate inventory or static scan. A separate seeded rounding difference returned DO_NOT_CUT_OVER. Each verdict covers only the workload and checks recorded in its report.

Current scope limits

EKS control-plane charges continue while the cluster exists

An idle environment still bills while its EKS cluster exists. AWS lists $0.10 per cluster-hour in standard support and $0.60 in extended support. SparkPilot has no pause control; the only way to end that cluster charge is to delete the cluster. Worker capacity controls may reduce other charges. Other resources may still bill. SparkPilot's rate-card figure is an estimate.

AWS EKS pricing ↗
IAM simulation runs on EMR on EKS only

Pre-dispatch iam:SimulatePrincipalPolicy checks cover EMR on EKS with BYOC-Lite onboarding. On EMR Serverless and EMR on EC2, preflight enforces policy and quota gates but does not simulate per-action IAM permissions before submit.

Engine coverage is uneven beyond EMR on EKS

EMR on EKS is the end-to-end governed path today, with completed runs re-verified against AWS. EMR Serverless runs dispatch through the current control plane, with preflight, in a production workspace. EMR on EC2 can be created now; its engine-level provision, submit and in-flight cancel path passed once, on 2026-06-01, before the current dispatch architecture, and governed dispatch through the current control plane has not been proven there. Steady-state reconciliation on both engines has not been proven.

Budget blocking works; its failure explanation does not yet

Team budgets can warn or block dispatch. A block prevents job submission, but existing AWS resources may still bill. The failure screen may attribute a budget block to environment setup.

CUR reconciliation is early access

Billed-truth reconciliation against your Cost and Usage Report — down to Split Cost Allocation Data on EKS — is built and has been exercised end to end against a simulated report; no billed run has yet been reconciled through a real CUR, and the production onboarding path is not yet reliable. It ships as early access until both are proven in production.

Per-run cost attribution on EMR Serverless is not CUR-proven

Cost-allocation tag propagation for Serverless is built, but the tag-to-CUR proof so far covers EMR on EKS only. Per-run billed attribution on Serverless is coming soon. On EMR on EC2, per-run billed attribution is not possible at all: EMR bills the cluster, and a step is not a billable resource.

Airflow and Dagster providers are preview

Both private packages have been tested with stubs. A scheduler-driven run in a customer deployment remains unverified.

No dead-letter handling for failed dispatch

Failed dispatches have no dead-letter replay. Some infrastructure failures require resubmission.

Control Tower scope

Control Tower answers questions from authorized workspace evidence. Every change waits for approval, except routine work an admin lets it do without asking. Run diagnostics use rules.

Not supported today

These workloads and integrations are unavailable today.

GPU and inference workloads

GPU scheduling and inference serving are unavailable.

Ray / KubeRay

Ray clusters are unsupported.

SageMaker

SageMaker job governance is unavailable.

Bedrock

Bedrock usage governance is unavailable.

Databricks

Jobs API integration is planned.

Snowflake

Snowflake integration is unavailable.

Multi-cloud

SparkPilot supports AWS only.

Evaluate your workload

Bring a workload to check access, preflight, dispatch, cost, and evidence against these limits.