Security you can verify, not just trust
SparkPilot exists to put a trustworthy gate in front of expensive AWS compute, so we hold our own security to the same standard: your data stays in your account, our access is role-scoped and revocable, and every claim below maps to a real control. What we have not yet earned the right to claim, we say so plainly.
How SparkPilot protects your account
These are the controls in place today. None of them require your data to leave your AWS account.
SparkPilot governs and dispatches Spark jobs that run inside your own AWS account. Your VPC, your S3 buckets, your EKS cluster, and your IAM policies stay under your control. SparkPilot's control plane orchestrates execution; your data does not leave your account to reach it.
SparkPilot never holds long-lived credentials into your account. It assumes a cross-account IAM role that you create, gated by a per-tenant ExternalId that only you and SparkPilot share. The assume-role chain is validated on every preflight, and a wrong ExternalId fails closed at AWS STS — the trust boundary is enforced by AWS itself, not by us.
The BYOC-Lite bootstrap CloudFormation stack grants the minimum permissions SparkPilot needs to validate and dispatch on EMR on EKS — deliberately narrow. It does not grant broad administrative access, and it omits permissions we do not require. You can read the template before you deploy it.
Before a job is dispatched, a composed preflight of 15+ IAM, trust-policy, OIDC, IRSA, quota, and capacity checks must pass; failures block dispatch with specific remediation. On EMR on EKS, this includes simulating IAM permissions before submit. IAM permission simulation runs on EMR on EKS / BYOC-Lite only — see the honest limitations page for the exact scope.
Tenants, teams, environments, and runs are scoped end to end, and API access is enforced by role (RBAC). On EMR on EKS, each environment maps to its own virtual cluster and namespace with IRSA-scoped execution credentials, and SparkPilot validates that trust and namespace configuration on every preflight. The strength of in-cluster isolation depends on the namespace, IRSA, and quota boundaries you configure in your own cluster — SparkPilot checks them; it does not replace them.
Access to SparkPilot APIs is enforced by role, scoped to teams and environments. Control-plane actions — run submission, preflight verdicts, dispatch — are recorded as audit events tied to the acting identity and tenant.
Traffic to SparkPilot is served over TLS. At rest, the control-plane database, dispatch queues, secrets, snapshots, and logs are encrypted with a dedicated AWS KMS customer-managed key. The runtime role's grant on that key is confined by a ViaService condition to the dispatch queues, and the key is resource-scoped so no role can decrypt data outside its own grant.
Interactive users sign in through OIDC (Auth0). Machine and automation clients authenticate with OAuth2 client_credentials — proven against the deployed API. Credentials can be revoked, and a revoked credential is rejected even before its token would otherwise expire.
What we do not claim yet
Honesty is part of the security posture. SparkPilot does not currently hold formal security certifications (such as SOC 2, ISO 27001, HIPAA, or PCI) and this page does not imply one. IAM permission simulation covers EMR on EKS and BYOC-Lite, not every engine. For the full list of current boundaries, read our limitations page — we keep it as current as this one.
Evaluating SparkPilot for a security-conscious team?
Bring a real workload. The parts that ship — role-scoped, RBAC-enforced access, IAM preflight, and per-run audit — are proven with live evidence, and we will walk your security reviewer through exactly how they work. Policy and quota gates are built and tested; we will show you those too, and be clear about what has run in a live account versus what has not.