Data pipeline / quality
The problem was not just connecting the boxes
I wanted to see the whole path from a retail source to a dashboard. The more useful question was where bad data or PII should stop, and how to know that a second run did not duplicate the warehouse.
What I tried
- Keep source-shaped data in Bronze with watermarks, then normalize and deduplicate in Silver.
- Apply the documented delete, hash, and mask treatments to synthetic PII with PySpark.
- Put a blocking quality gate before the warehouse so an invalid fixture cannot continue.
The data flow
- 1
FastAPI source → Bronze Parquet
- 2
Bronze → PII-protected PySpark Silver
- 3
Quality gate → DuckDB star schema → local evidence report
The useful failure case
A report saying “quality passed” is only half the story. I kept a fixture with duplicate, orphan, and invalid order-item data so the gate could stop a warehouse load when it should.
What came out of it
The local demo runs twice to check idempotency and runs the bad fixture separately to prove that a blocking failure prevents warehouse loading. The interesting output is the stop condition, not only the final tables.
Evidence from the local review path
The committed project review documents three checks from one demo command:
- Good run: status passed and the quality gate passes before DuckDB loading.
- Bad fixture: the gate fails and warehouse loading is not attempted.
- Second good run: idempotency is checked and fact keys remain unique.
What's still missing
- All data is synthetic and the demo runs locally.
- Airflow stops at local DuckDB; cloud publication is a manual command outside the demo path.
- There is no evidence here for production-scale volume or performance.
What I'd change today
- Start with the quality report and failure fixtures before adding more serving views.
- Put the contract for each layer in one reader-friendly place.
- Add schema-drift and late-arriving-data cases to the review path.
Source and demo
The repository has the complete setup, tests, and implementation details.