What SparkPilot checks before every run
SparkPilot runs a preflight before it dispatches any Spark run. Every run gets a verdict made of individual checks, each with a status and — when something is wrong — a specific fix. A hard failure blocks dispatch before AWS bills you. This page lists the checks and what each status means, so a verdict is never a black box.
The three statuses
Every check resolves to one of three statuses. The run's overall verdict is a hard block if any single check fails.
The check is satisfied. Nothing to do — this part of the run is cleared for dispatch.
Advisory. The run is still allowed to dispatch, but something is worth your attention — for example an EMR release that is no longer current, or spend that has crossed a budget warning threshold.
A hard block. If any check fails, the run does not dispatch: no AWS job is submitted, so you are not billed for that Spark run — one that was never going to be valid. The verdict tells you which check failed and how to fix it.
What the preflight checks
The main checks are grouped into families below, each listing the check codes you will see in a verdict so you can map a failure straight to the fix. Some families apply only to particular engines or onboarding modes, noted where that is the case.
SparkPilot confirms it can reach into your account the way it is supposed to — by assuming a role you created, with no long-lived keys — before it tries to run anything.
environment.customer_role_arniam.assume_role_chainiam.runtime_identityiam.no_static_credentialsissue3.sts_caller_identityissue3.iam_simulate_principal_policyissue3.eks_describe_clusterissue3.irsa_trust_subject
The STS caller-identity, per-action IAM simulation, EKS describe, and execution-role trust (IRSA) checks run on EMR on EKS with BYOC-Lite onboarding only. On other engines, preflight enforces policy and quota gates but does not simulate per-action IAM permissions before submit — see the limitations page for the exact scope.
The target environment has to actually be provisioned and wired before a run can land on it.
environment.statusenvironment.virtual_clusteremr.security_configuration
The EMR release, execution role, logging destination, and Spark configuration have to be ones SparkPilot can safely dispatch, on a release that is still supported.
config.emr_release_labelconfig.emr_release_currencyconfig.graviton_release_supportconfig.execution_roleconfig.log_group_prefixrun.spark_conf_policyrun.vpa_compatrun.s3_express_config
Before a run spends money, SparkPilot looks at the team's budget position and whether the cluster queue has room. Budget warning thresholds evaluate on every preflight where a team budget is configured, and crossing the block threshold stops the run before dispatch — an over-budget run has been blocked in production with zero AWS spend, the error naming the budget and the cost basis it was judged on.
team_budgetyunikorn_queue_capacity
Any governance rule you configure for a tenant, team, or environment becomes its own preflight check — for example a policy.max_vcpu rule. Rules set to hard enforcement fail the verdict and block dispatch; rules set to soft enforcement raise a warning and let the run proceed.
policy.<rule_type>
For EMR on EKS environments onboarded with BYOC-Lite, SparkPilot validates the cross-account and in-cluster wiring your onboarding created — such as the OIDC association, the execution-role trust, the EKS access-entry mode, cluster and namespace configuration, IAM pass-role, and Spot capacity.
byoc_lite.oidc_associationbyoc_lite.execution_role_trustbyoc_lite.access_entry_modebyoc_lite.eks_cluster_arnbyoc_lite.eks_namespacebyoc_lite.namespace_collisionbyoc_lite.iam_pass_rolebyoc_lite.spot_capacity
This lists representative onboarding checks, not the full set, and they run only on BYOC-Lite EMR on EKS environments.
When your environment uses AWS Lake Formation for fine-grained access control, SparkPilot checks that the EMR release supports it, the required service-linked role exists, and the Lake Formation permissions are in place before dispatch.
fgac.emr_releasefgac.service_linked_rolefgac.lf_permissions
Scope, stated plainly
Per-action IAM permission simulation covers EMR on EKS and BYOC-Lite, not every engine. Budget blocking is live and has stopped a real over-budget run before it spent anything. For the full list of current boundaries, read the limitations page — it is kept as current as this one.