Control Tower

Answers cite the workspace evidence you can see, and nothing changes without your approval.

  1. Run iceberg-net-revenue-daily-v2 now.
    SparkPilot2 sources

    I prepared a one-run dispatch for iceberg-net-revenue-daily-v2 using its saved configuration. The incremental run-cost estimate is $0.19; it excludes shared infrastructure and items such as S3, logs, NAT gateway, EBS, idle capacity, shared platform costs, and EMR Serverless storage. The run timeout is 30 minutes.

    Nothing has been queued or started yet. Confirm the proposal on its card to queue it; preflight and budget gates will still be checked before AWS dispatch.

    Server statusConfirmation recorded

    Run the proposed job

    Environment
    EMR Serverless · us-east-1
    What it does
    Queue one Spark run using the saved job configuration.
    estimated_cost_usd_micros
    189345
    cost_basis
    incremental_estimate
    includes_shared_infrastructure
    false
    excluded_costs
    [ "ebs", "nat_gateway", "cloudwatch_logs", "s3", "idle_capacity", "shared_platform", "emr_serverless_storage" ]
    timeout_seconds
    1800
    resource_ceiling
    { "driver_vcpu": 1, "driver_memory_gb": 4, "executor_vcpu": 2, "executor_memory_gb": 8, "executor_instances": 2 }
    After you confirm
    Confirmation creates one queued run. The scheduler must still pass preflight and budget gates before AWS dispatch.
    Technical details
    job_id
    9deb973b-0d5b-4868-9f11-61ed03daec2f
    spark_conf
    {}
    environment_id
    1a8b3660-5fe3-4d84-84d6-e57f0b651b9a
    Proposal ID and hash
    06806d5b-8744-47e2-83c2-fdc0b49008e8sha256:9210b9c6e2f78f400792eeada5048b81c96d555ddee78e30ebd69ce9bd4c075d

    Approved by a SparkPilot team member. An admin can let SparkPilot do this without asking next time

  2. Run it every night at 3 a.m. New York time.
    SparkPilot2 sources

    I prepared the schedule change for iceberg-net-revenue-daily-v2: every night at 3:00 a.m. America/New_York, starting Tuesday, November 3. It will run seven times per week; the currently running job is unaffected.

    Nothing has changed yet—confirm the proposal with the button on its card to save the new schedule. If another orchestrator already starts this workload, confirming this would create duplicate runs.

    Server statusConfirmation recorded

    Change the job's schedule

    What it does
    Change this job's schedule.
    Workload
    iceberg-net-revenue-daily-v2
    Schedule now
    0 2 * * * (America/New_York)
    Schedule after
    0 3 * * * (America/New_York)
    Next starts
    [ "Tue 2026-11-03 03:00 America/New_York", "Wed 2026-11-04 03:00 America/New_York", "Thu 2026-11-05 03:00 America/New_York" ]
    Runs started per week
    7
    After you confirm
    Confirmation saves the schedule. SparkPilot starts a run of the job's active version at each time, submitted the same way as a run started by hand. A run already in progress is not affected. No time before the change is made up. If an orchestrator such as Airflow also starts this job, it would then run twice. It is refused if the schedule changes first.
    required_roles
    [ "admin", "operator" ]
    Technical details
    job_id
    9deb973b-0d5b-4868-9f11-61ed03daec2f
    Proposal ID and hash
    1f9f4dd6-338d-4e8b-a51f-e69aa064de3dsha256:f6bdadf673b747c40b46e27000c0cf96de7d21d5e8b628609ec345e9f8d77105

    Approved by a SparkPilot team member.

  3. SparkPilot followed up on its own work: The run of iceberg-net-revenue-daily-v2 succeeded

    SparkPilot4 sources

    iceberg-net-revenue-daily-v2 succeeded on its first attempt. It ran from about 2:48 a.m. to 2:53 a.m. UTC and exited cleanly.

    Outcome: the job verified the daily net-revenue results already present for October 1–3:

    • October 1: gross $300.00, refunds $70.00, net $230.00, 2 orders, average order value $115.00, refund rate 23.33%.
    • October 2: gross $650.00, refunds $30.00, net $620.00, 2 orders, average order value $310.00, refund rate 4.62%.
    • October 3: gross $620.00, refunds $0.00, net $620.00, 2 orders, average order value $310.00, refund rate 0.00%.

    It found no changed rows, so it did not create a new table snapshot—the existing daily net-revenue table was already up to date. That is a successful no-op, not a failure.

    What should happen next: no repair or rerun is needed. Let the saved nightly 3:00 a.m. New York schedule run normally. The run’s incremental compute cost is currently an estimate of about $0.027, not a reconciled billed actual; it excludes shared infrastructure and costs such as S3, logs, NAT gateway, EBS, idle capacity, shared platform, and EMR Serverless storage. There was also a non-blocking preflight notice that a newer EMR release is available; review release compatibility before changing it, but it did not affect this run.