Local profiling · Python + Rust

Know your data before it ships.

Find missing values, inferred types and unusual values in files or tables. Gate a batch on quality limits or changes from an accepted baseline. Get a pass, fail or inconclusive result.

Runs on your machine. Deterministic analysis.
Open source under MIT or Apache-2.0.

An example report

orders.csv

8 rows · 3 columns · full scan

ColumnTypeNulls
order_idinteger0%
amountinteger0%
regionstring25%

amount
Median 30 · range 10–60

Amount frequencies: 10 occurs once, 20 twice, 30 three times, 40 once, 50 zero times, 60 once

Quality score93.19 / 100

Gate: fail

Why it fails: region has 2 missing cells out of 8, above the 20% null limit. The score meets the minimum of 90.

Default analysis of this CSV with dataprof 0.14.1. The report also flags the small sample size.

Try it on a small dataset

From CSV to a quality check

Use the same data as the report above. Then replace it with your own file.

1 Install the package

Python 3.11–3.15. The package has no Python dependencies.

python -m pip install dataprof

2 Get the example data

Save orders.csv in your working directory, beside your script.

3 Profile and check

Python
import dataprof as dp

report = dp.profile("orders.csv")
print(report.quality_score)
print(report["region"].null_percentage)

result = report.check(
    min_quality_score=90,
    max_null_percentage={"*": 20},
)
print(result.verdict)
report.save("report.json")

Output

93.19
25.0
fail

The score passes. The null check fails: 25% of region is missing.

The full report is saved to report.json.

Python API guide
Use the same check in CI
python -m dataprof.check orders.csv --min-quality 90 --max-null "*=20"

This CSV exits with code 1. Exit codes: 0 pass, 1 fail, 2 inconclusive, 3 argument, policy, input, baseline or output error.

Read the data. Keep the evidence.

A report you can use in your pipeline

Understand each column

Inspect types, null counts, numeric summaries, patterns and distinct values. Save a report to compare it with a later run.

Choose your quality limits

Set a minimum score, a min_rows floor or limits on missing values. Compare with a saved baseline to catch score drops and removed columns.

See what was analyzed

Read sampling_applied, source_exhausted and quality_sampled_dimensions. Numeric is_approximate and column unique_count_is_approximate mark sampled statistics and estimated distinct counts. Uncomputed metrics stay missing.

For the same logical data, schema semantics, options and analyzed population, serialized metrics agree across input paths and engines within the numeric contract. Provenance is compared separately. Raw floats can differ, and values near f64's limits belong to an approximate domain.

Keep the last accepted run

A baseline checkpoint

Save a candidate report on each run. Promote it to the baseline only on a pass. Options marked 0.14 need that release or later.

Start with absolute checks

python -m dataprof.check orders.csv --min-rows 1 \
  --unique-column order_id --max-null-count order_id=0 \
  --report candidate.json && mv candidate.json baseline.json

Compare later runs with the baseline

python -m dataprof.check orders.csv --min-rows 1 \
  --unique-column order_id --max-null-count order_id=0 \
  --baseline baseline.json --max-quality-drop 5 \
  --max-row-drop 10 --max-row-growth 20 --forbid-removed-columns \
  --timeliness-clock rolling --report candidate.json --json > verdict.json \
  && mv candidate.json baseline.json

A changed or unrecorded metric definition can leave an affected check definition_not_comparable. Inspect the reason before deliberately refreshing the baseline. Retyped columns appear in schema diagnostics; those diagnostics alone do not fail the gate.

Read the checkpoint recipe

Freshness and missing days

In 0.14, max_date_age={"published_at": 3} limits the newest date's age. max_date_gap={"published_at": 1} rejects a gap longer than one day between consecutive distinct dates. Use reference_date= when profiling to fix the clock; temporal_columns and temporal_pairs declare date roles and start/end relations.

Rows and keys

In 0.14, max_row_count_drop and max_row_count_growth limit percentage changes from the baseline. unique_columns checks repeated non-null keys. identifier_columns selects the key for the uniqueness score and key_duplicates finding.

Whole-row duplicates

The row tracker stays exact up to 1,000,000 distinct rows by default, or fewer under memory pressure. In 0.14, raise duplicate_rows_exact_limit to retain a larger exact set even past the memory budget. An estimated duplicate_rows count leaves max_duplicate_rows unevaluated.

A complete scan may still use a quality sample. quality_score_bounds records score intervals where available; a full-source score check can decide when the interval settles the threshold. See the freshness and key recipes.

Supported inputs

Start with the data you have

Files, in-memory tables and streams use a shared report model.

Files

CSV, JSON, JSONL and Parquet.

CSV detects comma, semicolon, pipe and tab delimiters. JSON accepts an array of objects or a single object; JSONL accepts one object per line. Native Parquet profiling needs no Python table library. Rust enables parquet and arrow by default.

Python tables and bytes

pandas and polars DataFrames, Arrow tables, batches and C streams.

Also accepts dictionaries, bytes and BytesIO. Byte inputs require a format. Install the table library you use; pandas and polars conversion also needs pyarrow. The pandas extra installs pandas.

Async and database sources

Async files, bytes, HTTP URLs and database queries.

The wheel includes dataprof.asyncio. Rust uses async-streaming and parquet-async for remote Parquet. PostgreSQL, MySQL and SQLite need connector features, and a Python source build. The incremental engine uses bounded accumulators. Connector setup.

The quality model

A score with evidence behind it

Quality analysis assesses up to seven dimensions, informed by ISO 8000-8 and ISO/IEC 25012. The aggregate uses only dimensions the data supports.

A score can pass while an individual check fails, as in the CSV above. Choose limits that fit your data.

Completeness
Missing cells and fully populated rows.
Consistency
Type consistency, format and encoding issues.
Uniqueness
Duplicate rows and key uniqueness.
Accuracy
Outliers and range violations.
Timeliness
Future dates and stale timestamps.
Validity
Conformance to detected semantic patterns.
Precision
Decimal scale in floating-point columns.

In 0.14, native_type preserves a source's declared column type where available. Declared decimal columns are read as decimals, including when the statistics pack is omitted. Nested columns contribute null and duplicate counts; quality_columns_without_value_checks names those whose values received no value-level quality checks.

Benchmark evidence

Inspect the profiling costs

Explore timings, variation and workload details from CI artifacts. The benchmark pages explain the measurement boundaries and how to reproduce a run.

Browse benchmarks

Keep going

Documentation and project links