SparkPilot security controls
Spark jobs run in your AWS account. SparkPilot uses scoped, revocable access to check and dispatch them.
Current controls
Spark jobs use your AWS account's compute and storage. SparkPilot dispatches them through a cross-account role.
SparkPilot assumes a cross-account IAM role that you create, gated by a per-tenant ExternalId. Preflight checks the role; AWS STS rejects a wrong ExternalId. SparkPilot holds no long-lived AWS credentials for your account.
The BYOC-Lite CloudFormation template lets SparkPilot validate your setup; create, run and cancel work on EMR on EKS, EMR Serverless and EMR on EC2; read its job logs; write notebook and deploy files to your artifacts bucket; and update the execution role's trust policy. Review the template before deploying it.
Before dispatch, preflight checks IAM, trust policy, OIDC, IRSA, quota, and capacity; failures block the run with remediation. IAM permission simulation applies to EMR on EKS / BYOC-Lite only.
Tenants, teams, environments, and runs are scoped end to end, and API access is enforced by role (RBAC). On EMR on EKS, each environment maps to its own virtual cluster and namespace with IRSA-scoped execution credentials, and SparkPilot validates that trust and namespace configuration on every preflight. The strength of in-cluster isolation depends on the namespace, IRSA, and quota boundaries you configure in your own cluster — SparkPilot checks them; it does not replace them.
People's access to SparkPilot APIs is enforced by role, scoped to teams and environments. A machine credential acts as a workspace admin. Control-plane actions (run submission, preflight verdicts, dispatch) are recorded as audit events tied to the acting identity and tenant.
Traffic to SparkPilot is served over TLS. At rest, the control-plane database, dispatch queues, secrets, snapshots, and application logs are encrypted with a dedicated AWS KMS customer-managed key. CloudTrail and load-balancer access logs use S3-managed encryption. The runtime role's grant on that key is confined by a ViaService condition to the dispatch queues; the task-execution role holds a separate decrypt grant used only to resolve Secrets Manager payloads at container start.
Interactive users sign in with Auth0. Automation uses OAuth2 client credentials. Revoked credentials stop working before token expiry.
Current limits
SparkPilot has no SOC 2 report or ISO 27001 certification. We do not claim HIPAA or PCI DSS compliance. IAM permission simulation covers EMR on EKS and BYOC-Lite. See the limitations page for more boundaries.
Review SparkPilot with your security team
Bring a workload. We can review its access role, preflight checks, and audit record with your team.