# Spark Portability Assessment — Acme Analytics (sample estate)

> **SYNTHETIC FIXTURE — this estate is invented, not a customer.** No workspace was read and no workload was executed. "Sample" here means fabricated, not a sampled subset of a real estate. Account ids and ARNs below are AWS documentation placeholders.

**databricks → emr_serverless**

## Recommendation

### Do not cut over — unresolved differences or blockers

`DO_NOT_CUT_OVER`

This report covers 0 workload(s) executed on both databricks and emr_serverless, out of 3 job(s) inventoried. It says nothing about workloads that were not run.

This recommendation follows from:

- 1 job definition(s) use constructs with no emr_serverless equivalent
- 3 distinct source-level blocker(s) (5 occurrences) found by the static scanner
- no workload was executed on both platforms, so no output equivalence has been demonstrated

## Summary

- **Decision:** `DO_NOT_CUT_OVER` — do not cut over — unresolved differences or blockers.
- **Jobs inventoried:** 3; **job definitions that cannot port as written:** 1.
- **Source files with blocking constructs:** 2 (`bronze_ingest.py`, `nightly_sales_rollup.py`). Scanner findings are file-scoped and carry no job id, so **the overlap with the job count above is unknown** — these files may belong to jobs already counted or to jobs that are not. Calling them additional would assert a relationship the same sentence says cannot be established. Either way, a job whose source appears here cannot port as written.
- **Distinct problems to fix:** 17.
  Remediation effort, by distinct problem: 5 SIMPLE (<2h each), 2 MEDIUM (2-6h each), 3 SIGNIFICANT (>6h each), 7 UNSIZED
- **Measured change:** none — no workload was run on both platforms, so this report contains no performance or cost comparison.
- **What this does not do:** SparkPilot converts nothing. Every problem above is manual work; the deliverable is the evidence, not the migration.

## Coverage

- Jobs inventoried: **3** (3 tasks)
- Workloads executed on both platforms: **0**
- Source files scanned: **4**
- Job definitions parsed without error: **all of them**
- Estate enumeration: **not performed** — job definitions were supplied to the importer, not discovered by it
- Remediation effort, by distinct problem: 5 SIMPLE (<2h each), 2 MEDIUM (2-6h each), 3 SIGNIFICANT (>6h each), 7 UNSIZED
  (buckets size one occurrence of the fix, not the whole migration; `UNSIZED` means the static match cannot size it, and it is counted here rather than dropped)

## Output equivalence

**No dual run was performed.** No claim of output equivalence is made anywhere in this report.

## Performance and cost

_No run metrics captured._

## Portability risks

### BLOCKER (4 distinct, 6 occurrences)

| Where | Issue | Effort | What has to change |
| --- | --- | --- | --- |
| job `bronze_ingest_pipeline` / task `ingest` | Delta Live Tables pipeline task | UNSIZED | DLT is proprietary and managed. The pipeline must be rebuilt as Spark jobs plus an orchestrator before it can be assessed for equivalence. |
| `bronze_ingest.py:2,4` | Delta Live Tables pipeline | SIGNIFICANT | DLT is a proprietary managed framework with no EMR equivalent. The pipeline must be rewritten as ordinary Spark jobs plus an orchestrator. |
| `bronze_ingest.py:7,8` | Auto Loader (cloudFiles) source | SIGNIFICANT | Auto Loader is Databricks-only. Replace with Structured Streaming over S3 plus SQS/SNS event notifications, or a scheduled batch listing. |
| `nightly_sales_rollup.py:12` | Unity Catalog Volume path | SIGNIFICANT | UC Volumes have no EMR equivalent. Land the data in S3 and grant the job execution role access, or front it with Lake Formation. |

### REWRITE (8 distinct, 10 occurrences)

| Where | Issue | Effort | What has to change |
| --- | --- | --- | --- |
| job `nightly_sales_rollup` / task `rollup` | Notebook task | UNSIZED | EMR does not run notebooks as jobs. Extract the notebook to a Python entrypoint; widget parameters become spark-submit arguments. |
| job `nightly_sales_rollup` / task `rollup` | Cluster uses an EC2 instance profile | UNSIZED | arn:aws:iam::111122223333:instance-profile/dbx-etl grants this job its data access. Reproduce the same boundary with an EMR job execution role before cutover. |
| job `nightly_sales_rollup` / task `rollup` | Library staged on DBFS (whl:dbfs:/FileStore/libs/acme_common-2.1.0-py3-none-any.whl) | UNSIZED | Re-stage the artifact in S3; DBFS is not reachable from EMR. |
| `nightly_sales_rollup.py:2` | Notebook magic command | SIMPLE | Magics are a notebook-runtime feature. Convert the notebook to a plain Python entrypoint; %pip becomes a packaged dependency, %run becomes an import. |
| `nightly_sales_rollup.py:6,7` | dbutils.widgets job parameters | SIMPLE | Replace with argparse over spark-submit arguments. EMR passes parameters as process arguments, not as a widget API. |
| `nightly_sales_rollup.py:9` | dbutils.secrets secret scope access | MEDIUM | Move secrets to AWS Secrets Manager or SSM Parameter Store and read them with the job execution role. Secret scopes do not exist outside Databricks. |
| `nightly_sales_rollup.py:11,21` | DBFS path | SIMPLE | Repoint to s3:// or s3a://. DBFS is a Databricks-managed mount and is not reachable from EMR. |
| `nightly_sales_rollup.py:20` | display() / displayHTML() notebook rendering | SIMPLE | Replace with .show(), or write the result to S3. These are notebook display hooks and raise NameError under spark-submit. |

### REVIEW (5 distinct)

| Where | Issue | Effort | What has to change |
| --- | --- | --- | --- |
| job `nightly_sales_rollup` / task `rollup` | Photon engine enabled | UNSIZED | Photon is a Databricks-only vectorised engine. The current runtime and cost baseline will not carry to EMR; measure the target by dual run rather than extrapolating. |
| job `nightly_sales_rollup` / task `rollup` | Databricks-only Spark conf spark.databricks.delta.optimizeWrite.enabled | UNSIZED | spark.databricks.delta.optimizeWrite.enabled='true' is ignored outside Databricks. Establish whether the behaviour it bought is required on the target. |
| job `batch_reconcile` / task `reconcile` | Python spark-submit task | UNSIZED | Directly portable to EMR. Confirm entrypoint and arguments resolve outside DBFS. |
| `compact_gold.sql:2` | Delta OPTIMIZE / ZORDER | MEDIUM | OPTIMIZE exists in Delta Lake OSS; ZORDER is Databricks-optimised and its OSS behaviour differs. Confirm layout and query performance by dual run, or move to Iceberg with sort order / clustering. |
| `compact_gold.sql:3` | Delta VACUUM | SIMPLE | Available in Delta OSS but retention defaults and enforcement differ. Pin the retention explicitly rather than relying on the Databricks default. |

## Evidence

Every claim above traces to one of these artifacts:

- **equivalence engine guards**: scripts/verify_equivalence_guards.py
- **sample estate + generator**: scripts/generate_sample_report.py

---

_Generated 2026-08-14T06:00:00+00:00 from commit `b10aac5`. Any platform rates cited above were read on that date and change without notice._