Connect SparkPilot to your
existing workflow stack
Connect your tools to SparkPilot's run API. Airflow and Dagster packages are available for private preview.
Orchestrator and interface coverage
Preview provider for run submission and status checks from Airflow. Tested with compatibility stubs; a real scheduler run remains unverified.
Private source access required before installation.
Preview package for submitting and monitoring Spark workloads from Dagster. Tested with compatibility stubs; a real scheduler run remains unverified.
Private source access required before installation.
Submit and inspect runs from your own tools with a machine credential. It acts as a workspace admin, so keep it in your secret store.
Versioned endpoints for runs, logs, and evidence.
Private-source command line client for pilot automation. It needs machine credentials and an API address.
Request source access before installation.
How teams operate with SparkPilot
- Admin configures identity, team scopes, and budget guardrails in the app.
- Teams can submit through the REST API. CLI, Airflow, and Dagster access is by preview request.
- SparkPilot runs preflight checks, dispatches jobs, and tracks lifecycle events.
- Operators review runs, diagnostics, and cost visibility in one place.
What you can review during evaluation
Review API submission, preflight, run tracking, and diagnostics with one workload. The CLI, Airflow, and Dagster integrations are private previews.
Redacted screenshots for buyer and security reviews are shared during active pilot evaluations.
Short onboarding clips are coming soon, alongside planned workflow extensions such as Apache Iceberg governance.
Connect your orchestrator in a pilot call
We will walk through your Airflow or Dagster setup, scope one workload, and confirm integration fit before you commit to a rollout.