Data Quality

An open toolcase with every tool in its own cut-out, and one oddly shaped empty slot in the middle with a blank paper tag tied to it

Where Should the Library Stop?

My data profiler gives the same untouched file a lower score on New Year’s Day. I found that by using it as a dependency for the first time. On the glue a real pipeline needs, and the line between what a library should do and what it should leave to the caller.

A calm, uniform green dot; under a magnifying glass its inside is a tangle of gears, threads, loose papers and one cracked cog

Green Is a Policy, Not a Fact

My backup verifier passed 24 of 24 checks while the backup job sat in failed. It is not the only green thing I have built that answered a narrower question than the one I was asking. On what a green light actually proves, and the checks that earn it.

Writer A checks epoch 1 against base A, the transaction is rebased onto base B, and the premise the check established is lost before the commit lands

I tried to put data quality inside the commit. One pub(crate) stopped it.

A falsification experiment with a written kill list and a negative result. Delta has a purpose-built idempotent commit; iceberg-rust rebases a transaction onto a base the caller never checked, so an external writer cannot make a commit conditional. The green concurrency test was a false pass — 0 of 40 rounds ever raced — and only a failpoint showed the duplicate.

Guardrails for Tabular ML: A Data Engineer's Take on Data Leakage, Poisoning, and Brittle Pipelines

Most ML pipeline failures are not exotic model bugs — they are data issues that nobody encoded as checks. This article walks through building guardrails using pandas, Apache DataFusion, data contracts, and the Arrow C Data Interface.