Understand each column
Inspect types, null counts, numeric summaries, patterns and distinct values. Save a report to compare it with a later run.
Local profiling · Python + Rust
Find missing values, inferred types and unusual values in files or tables. Gate a batch on quality limits or changes from an accepted baseline. Get a pass, fail or inconclusive result.
8 rows · 3 columns · full scan
| Column | Type | Nulls |
|---|---|---|
order_id | integer | 0% |
amount | integer | 0% |
region | string | 25% |
amount
Median 30 · range 10–60
Quality score93.19 / 100
Gate: failWhy it fails: region has 2 missing cells out of 8, above the 20% null limit. The score meets the minimum of 90.
Default analysis of this CSV with dataprof 0.14.1. The report also flags the small sample size.
Try it on a small dataset
Use the same data as the report above. Then replace it with your own file.
Python 3.11–3.15. The package has no Python dependencies.
python -m pip install dataprofSave orders.csv in your working directory, beside your script.
import dataprof as dp
report = dp.profile("orders.csv")
print(report.quality_score)
print(report["region"].null_percentage)
result = report.check(
min_quality_score=90,
max_null_percentage={"*": 20},
)
print(result.verdict)
report.save("report.json")
93.19
25.0
failThe score passes. The null check fails: 25% of region is missing.
The full report is saved to report.json.
python -m dataprof.check orders.csv --min-quality 90 --max-null "*=20"This CSV exits with code 1. Exit codes: 0 pass, 1 fail, 2 inconclusive, 3 argument, policy, input, baseline or output error.
In an existing Cargo project. Requires Rust 1.96 or later.
cargo add dataprofSave orders.csv in your Cargo project directory.
Replace src/main.rs with this example, then run cargo run.
use dataprof::Profiler;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let report = Profiler::new()
.analyze_file("orders.csv")?;
println!("Rows: {}", report.execution.rows_processed);
if let Some(score) = report.quality_score() {
println!("Quality: {score:.2}");
}
for col in &report.column_profiles {
println!(
"{} {:?} nulls={}",
col.name, col.data_type, col.null_count,
);
}
Ok(())
}
Rows: 8
Quality: 93.19
order_id Integer nulls=0
amount Integer nulls=0
region String nulls=2Feature flags add Arrow, Parquet, async streaming and database connectors.
Rust API referenceRead the data. Keep the evidence.
Inspect types, null counts, numeric summaries, patterns and distinct values. Save a report to compare it with a later run.
Set a minimum score, a min_rows floor or limits on missing values. Compare with a saved baseline to catch score drops and removed columns.
Read sampling_applied, source_exhausted and quality_sampled_dimensions. Numeric is_approximate and column unique_count_is_approximate mark sampled statistics and estimated distinct counts. Uncomputed metrics stay missing.
For the same logical data, schema semantics, options and analyzed population, serialized metrics agree across input paths and engines within the numeric contract. Provenance is compared separately. Raw floats can differ, and values near f64's limits belong to an approximate domain.
Keep the last accepted run
Save a candidate report on each run. Promote it to the baseline only on a pass. Options marked 0.14 need that release or later.
python -m dataprof.check orders.csv --min-rows 1 \
--unique-column order_id --max-null-count order_id=0 \
--report candidate.json && mv candidate.json baseline.json
python -m dataprof.check orders.csv --min-rows 1 \
--unique-column order_id --max-null-count order_id=0 \
--baseline baseline.json --max-quality-drop 5 \
--max-row-drop 10 --max-row-growth 20 --forbid-removed-columns \
--timeliness-clock rolling --report candidate.json --json > verdict.json \
&& mv candidate.json baseline.json
A changed or unrecorded metric definition can leave an affected check definition_not_comparable. Inspect the reason before deliberately refreshing the baseline. Retyped columns appear in schema diagnostics; those diagnostics alone do not fail the gate.
In 0.14, max_date_age={"published_at": 3} limits the newest date's age. max_date_gap={"published_at": 1} rejects a gap longer than one day between consecutive distinct dates. Use reference_date= when profiling to fix the clock; temporal_columns and temporal_pairs declare date roles and start/end relations.
In 0.14, max_row_count_drop and max_row_count_growth limit percentage changes from the baseline. unique_columns checks repeated non-null keys. identifier_columns selects the key for the uniqueness score and key_duplicates finding.
The row tracker stays exact up to 1,000,000 distinct rows by default, or fewer under memory pressure. In 0.14, raise duplicate_rows_exact_limit to retain a larger exact set even past the memory budget. An estimated duplicate_rows count leaves max_duplicate_rows unevaluated.
A complete scan may still use a quality sample. quality_score_bounds records score intervals where available; a full-source score check can decide when the interval settles the threshold. See the freshness and key recipes.
Supported inputs
Files, in-memory tables and streams use a shared report model.
CSV, JSON, JSONL and Parquet.
CSV detects comma, semicolon, pipe and tab delimiters. JSON accepts an array of objects or a single object; JSONL accepts one object per line. Native Parquet profiling needs no Python table library. Rust enables parquet and arrow by default.
pandas and polars DataFrames, Arrow tables, batches and C streams.
Also accepts dictionaries, bytes and BytesIO. Byte inputs require a format. Install the table library you use; pandas and polars conversion also needs pyarrow. The pandas extra installs pandas.
Async files, bytes, HTTP URLs and database queries.
The wheel includes dataprof.asyncio. Rust uses async-streaming and parquet-async for remote Parquet. PostgreSQL, MySQL and SQLite need connector features, and a Python source build. The incremental engine uses bounded accumulators. Connector setup.
The quality model
Quality analysis assesses up to seven dimensions, informed by ISO 8000-8 and ISO/IEC 25012. The aggregate uses only dimensions the data supports.
A score can pass while an individual check fails, as in the CSV above. Choose limits that fit your data.
In 0.14, native_type preserves a source's declared column type where available. Declared decimal columns are read as decimals, including when the statistics pack is omitted. Nested columns contribute null and duplicate counts; quality_columns_without_value_checks names those whose values received no value-level quality checks.
Benchmark evidence
Explore timings, variation and workload details from CI artifacts. The benchmark pages explain the measurement boundaries and how to reproduce a run.
Keep going
Find setup instructions and examples in the getting started guide and Python API guide.
Release notesStart with the contributing guide, or pick a good first issue. Usage questions and ideas belong in GitHub Discussions.
GitHub’s Cite this repository button generates APA and BibTeX from CITATION.cff.
Benchmark research material