A check that has never failed proves nothing.
Anyone can ship a comparator that always says “looks fine.” So we seeded a rounding difference into our own reference migration and made ours fail. Below is the mechanism and every number behind both answers: an EMR Serverless 6.15.0 → 7.5.0 runtime upgrade, executed on real AWS in account 400920695203, us-east-1, in both directions of the verdict.
Two runs, two verdicts
Same workload, same comparator. The only difference is that a rounding difference was deliberately introduced into the second one.
One workload — tpch-q1-reference-v1 — executed on both platforms. 10 equality assertions ran and no differences were found. The recommendation still stops short of PROCEED: no estate inventory or static scan was attached, and equivalence on one pair is not a cut-over verdict.
| Measure | emr-6.15.0 | emr-7.5.0 |
|---|---|---|
| Spark | 3.4.1-amzn-2 | 3.5.2-amzn-1 |
| Wall clock | 48.5s | 19.6s |
| Cost | $0.0157 | $0.0202 |
The verdict is scoped to the workloads that were assessed. It says nothing about workloads that were not run.
A rounding difference was seeded into the workload on purpose, so that a passing verdict could be distinguished from a comparator that cannot fail. It was caught three independent ways: the output schema drifted, the row content hash changed, and three aggregates moved.
- Tolerance for this comparison was
rel=0.0, abs=0.0— exact equality, so a rounding change is a difference, not a rounding error. - The seeded run is
00g7nppt0p4fu80b, a real job run in the same account and region.
This is the most important run on the page. Without it, the equivalent verdict is an unfalsifiable claim.
How a verdict is produced
Five steps. Nothing in the chain is a questionnaire, a checklist, or a heuristic read of your code — every step below is an execution or a comparison of what execution returned.
One versioned workload is the unit under test. The reference migration used tpch-q1-reference-v1. The version is what a verdict attaches to, so a later change to the workload invalidates the verdict instead of silently inheriting it.
The same workload is submitted to the source platform and the target platform. In the reference migration that meant emr-6.15.0 (Spark 3.4.1-amzn-2) and emr-7.5.0 (Spark 3.5.2-amzn-1), as real EMR Serverless job runs in AWS account 400920695203, us-east-1 — not a dry run and not a simulation. Naming the engine matters: these runs were submitted by the assurance harness, not dispatched through SparkPilot's governed control plane. The assurance run and the dispatch path are separate things.
Output schema, a row content hash, aggregate values, wall clock, and the platform's own reported resource consumption. Evidence is taken from the platform, so the comparison does not depend on trusting either vendor's summary of itself.
The tolerance is stated up front rather than discovered afterwards. The reference comparison ran 10 equality assertions at rel=0.0, abs=0.0 — exact — and found no differences. Verdict: EQUIVALENT.
PROCEED_FOR_ASSESSED_SCOPE on the equivalent pair; DO_NOT_CUT_OVER when a difference is found. The scoping is deliberate and it is load-bearing: the verdict covers the workloads that were actually executed and compared, and says nothing about workloads that were not run.
What gets compared
Three axes were exercised by the reference migration, each with the figure it produced.
Column types, a row content hash over the result set, and aggregate values, each asserted at the declared tolerance. This is the assertion that caught the seeded difference three separate ways.
Wall clock for the run on each platform, measured on the runs themselves rather than inferred from a benchmark. Faster is not automatically better — it is evidence a reviewer weighs alongside the equality result.
Cost is the platform's own reported resource consumption for the run priced at that platform's published public rates. It is not a bill, and it is not a modelled forecast. Where a platform exposes no consumption feed the value reads not measured — never zero.
The negative control, in full
Everything the comparator reported on the seeded run, at tolerance rel=0.0, abs=0.0. These five findings are why the equivalent verdict on the other run is worth reading.
| Assertion | Reference run | Seeded run | Result |
|---|---|---|---|
| Schema — sum_charge | decimal(38,6) | decimal(38,1) | type drift |
| Row content hash | 984eb4735fc1b990… | 5e7671c91ac6d5cc… | differs |
| Aggregate — max | 1418207490.410181 | 1418207475.3 | differs |
| Aggregate — min | 1391007086.081212 | 1391007105.8 | differs |
| Aggregate — sum | 8427644657.016188 | 8427644818.5 | differs |
On a narrow screen, swipe horizontally to read the full table.
Run identifiers
The EMR application and job-run identifiers behind the figures on this page, in AWS account 400920695203, us-east-1.
- Reference migration
application 00g7nppcfcrf8i09 · job run 00g7nppqou31tg0b - Reference migration
application 00g7nppdqs2mr409 · job run 00g7npps1fnug00b - Seeded negative control
job run 00g7nppt0p4fu80b
Read the report itself
Everything on this page summarises generated artifacts. The artifacts themselves are published here, unedited — including the report that answers its own headline with INSUFFICIENT_EVIDENCE because no estate was inventoried.
- Reference migration report (Markdown) — the EMR 6.15.0 → 7.5.0 equivalence run, verbatim
- Seeded negative-control report (Markdown) — the same comparator returning DO_NOT_CUT_OVER
- Sample portability assessment (Markdown) — the assessment deliverable's shape, built from a fixture estate
- Raw run manifests (JSON) — 7.5.0 baseline and seeded control, the comparator's actual inputs
What this does not prove
The scoping is the product. A verdict that quietly covered work nobody executed would be worth less than no verdict at all, so here is exactly where the evidence stops.
The reference migration report records it plainly: jobs inventoried 0, source files scanned 0, inventory complete no. Nothing on this page should be read as evidence that discovery or static scanning works at estate scale, because that has not been demonstrated.
tpch-q1-reference-v1 across EMR Serverless 6.15.0 → 7.5.0. No second workload, no second platform pair, and no second migration path has been taken end to end.
PROCEED_FOR_ASSESSED_SCOPE is not a hedge bolted on afterwards — it is what the verdict means. It covers the workloads that were executed and compared. It says nothing about the ones that were not.
Both platforms completed the reference workload successfully, so this comparison never had to reconcile two different failures. Comparing how a workload fails across runtimes is part of the intent; it is not part of what has been proven.
Where this sits
Datafold diffs tables across databases. SparkPilot diffs job behavior across runtimes — outputs, runtime, cost, and failure modes.
Today that comparison is proven on outputs, wall clock, and platform-reported cost, for one workload across one runtime-upgrade pair. The reference workload succeeded on both platforms, so failure-mode divergence was not exercised — see the section above.
This is not a checklist and it is not a conversion consultancy. The differentiation is neutral execution evidence and result equivalence: the workload is run on both platforms, the outputs are compared, and the verdict states its own scope. The owned object is the versioned workload and the release attestation attached to it.
Bring us a workload and a destination
The migration above is a runtime upgrade we ran on ourselves — one workload, one pair. The useful next one is yours. Bring a workload and a target, and we will tell you before we start whether we can execute it on both sides. If we can, you get this same report — including the answer you did not want. If we cannot, you get that answer for free.