Data engineering / mobility
I wanted to see the whole path
The project began with a production-style data-platform picture, but the review path stays small and local. It does not download a full NYC TLC month into the repository.
What I tried
- Create a Parquet fixture and profile it before checking timestamps, amounts, passengers, zones, and speed.
- Separate valid, duplicate-warning, and rejected rows, then load only the allowed rows into DuckDB.
- Build dimensions, facts, and marts with dbt before serving them through FastAPI, React, and local Dagster assets.
From file to dashboard
- 1
TLC-style Parquet → profile and validate
- 2
Validated / rejected partitions → DuckDB staging → dbt marts
- 3
FastAPI read-only API → React dashboard; Dagster orchestrates locally
A bad row is still useful output
The fixture contains one valid trip, one duplicate warning, and one row with invalid amounts. Keeping the rejected row visible lets the quality page explain why it did not continue.
What came out of it
The fixture is intentionally tiny, but the demo runs twice to check month-level idempotency and writes a quality report that keeps valid, warning, and rejected rows separate.
What's still missing
- The demo does not download the full NYC TLC dataset and the fixture does not represent its volume.
- It is local-first with no production deployment, authentication, or cloud account.
- The optional official sample needs network access and an external DATA_DIR outside the repository.
What I'd change today
- Put the data contract next to the quality report before adding more dashboard surface.
- Add late-arriving-data and schema-drift cases to the fixture.
- Reduce the first dashboard pass and finish one anomaly story end to end.
Source and demo
The repository has the complete setup, tests, and implementation details.