Connect SparkPilot to your
existing workflow stack
Keep Airflow and Dagster in place, then run submission, diagnostics, and cost controls through one governed control plane.
Orchestrator and interface coverage
Submit SparkPilot runs, wait for terminal states, and manage retries from your existing DAGs. Currently exercised via the Airflow compatibility stubs in tests — a first real Airflow scheduler run is on the validation list.
Installable from source: hook, operator, sensor, and deferrable trigger support
Use resources, ops, and assets to submit and monitor Spark workloads in Dagster Cloud or OSS. Currently exercised via Dagster compatibility stubs in tests — a first real Dagster scheduler run is on the validation list.
Installable from source: resource + ops + assets for submit, wait, cancel
Run SparkPilot through internal portals, CI jobs, or terminal workflows without changing team ownership.
REST API, RBAC, audit trail, run-submit and run-logs commands
How teams operate with SparkPilot
- Admin configures identity, team scopes, and budget guardrails in the app.
- Orchestrators submit jobs through Airflow, Dagster, API, or CLI.
- SparkPilot runs preflight checks, dispatches jobs, and tracks lifecycle events.
- Operators review runs, diagnostics, and cost visibility in one place.
What you can review during evaluation
Walk through Airflow or Dagster submission, preflight checks, run tracking, and diagnostics with your workflow shape.
Redacted screenshots for buyer and security reviews are shared during active pilot evaluations.
Short onboarding clips are coming soon, alongside planned workflow extensions such as Apache Iceberg governance.
Connect your orchestrator in a pilot call
We will walk through your Airflow or Dagster setup, scope one workload, and confirm integration fit before you commit to a rollout.