0.10.0 release notes → Rust core · Python native · ISO 8000/25012

Know your data before it bites.

dataprof profiles tabular data at Rust speed: column statistics, type and pattern detection, and standards-based quality scoring — with bounded memory, so datasets far larger than RAM are fair game.

Start in 30 seconds Live benchmarks

crates.io PyPI docs.rs MIT or Apache-2.0

python

import dataprof as dp

report = dp.profile("orders.csv")

report.rows, report.columns
(1_048_576, 14)

report.quality_score
87.4  # 0–100 across assessed dimensions

age = report["customer_age"]
age.data_type, age.null_percentage
('integer', 2.31)

# token-bounded summary for an agent, PII never echoed
print(report.to_llm_context(max_tokens=500))

The first ten minutes

Built for the questions you ask unfamiliar data

Find sparse columns, unstable types, duplicate keys, stale timestamps, and suspicious values before they turn into pipeline bugs.

Which columns are thin, empty, or structurally broken?

Null counts, completeness metrics, and schema shape in one pass.

Did this feed drift or spike somewhere suspicious?

Numeric summaries, outlier signals, and range checks.

Are these IDs really unique, or just pretending to be keys?

Distinct counts, uniqueness ratios, and duplicate warnings.

Are my timestamps plausible and fresh?

Future-date detection, stale-data signals, and timeliness scoring.

Did parsing silently go wrong?

Type inference, pattern matches, format violations, and source metadata.

What changed since yesterday?

Save a baseline report and diff it against today’s data, column by column.

Quickstart

Two polished entry points

A Python package that feels natural in notebooks, and a compact Rust facade for services, ETL jobs, and batch tools.

uv pip install dataprof

import dataprof as dp

report = dp.profile("data.csv")          # files, dicts, bytes, DataFrames, Arrow
print(report.quality_summary())        # per-dimension ISO scores

report.save("report.json")             # full report, reloadable
print(report.to_markdown())            # a table for a PR comment

before = dp.ProfileReport.load("report.json")
delta = before.compare(dp.profile("data_clean.csv"))

Python 3.10+. Pre-built wheels have zero Python dependencies — pandas is optional, for DataFrame-typed exports and Parquet byte buffers. See the Python API guide.

cargo add dataprof

use dataprof::Profiler;

let report = Profiler::new().analyze_file("data.csv")?;
println!("Rows: {}", report.execution.rows_processed);
println!("Quality: {:.1}%", report.quality_score().unwrap_or(0.0));

for col in &report.column_profiles {
    println!("{} {:?} nulls={}", col.name, col.data_type, col.null_count);
}

MSRV 1.96. Feature flags cover Arrow, Parquet, async streaming, and PostgreSQL / MySQL / SQLite connectors — or go lean with default-features = false. See docs.rs.

Why it feels modern

Fast by construction, honest about memory

Fast first-pass signal

Surface null pockets, type drift, duplicate keys, and outliers quickly — before you commit to a full analysis.

True streaming

Bounded-memory profiling with online algorithms. Files bigger than RAM are a normal Tuesday.

Multi-format by default

CSV, JSON, JSONL, Parquet, live databases, DataFrames, and Arrow batches — one tool across all of them.

Two polished entry points

A compact Rust facade and a Python package that feels native in notebooks, scripts, and data apps.

Async-ready

Rust async APIs and opt-in Python builds cover stream pipelines, services, and remote Parquet sources.

Agent-safe summaries

Token-bounded LLM context with sensitive values never echoed — profiling that plugs into agent workflows.

ISO 8000-8 · ISO/IEC 25012

A quality model you can point an auditor at

When quality analysis is requested, dataprof assesses up to seven dimensions informed by international data-quality standards. Its configurable aggregate score uses only the dimensions the data could actually support.

Completeness

Missing-cell percentage, share of fully-populated rows, columns past the null threshold.

Consistency

Data type consistency, format violations, encoding issues.

Uniqueness

Duplicate rows, key uniqueness, high-cardinality warnings.

Accuracy

Outlier ratio, range violations, negatives in positive-only columns.

Timeliness

Future dates, stale-data ratio, temporal ordering violations.

Validity

Conformance to confidently detected semantic patterns, with weak evidence left unassessed.

Precision

Consistency of observed decimal scale within floating-point columns.

Supported inputs

Meet your data where it lives

Format	Engine	Notes
CSV	Incremental, Columnar	Auto-detects `,` `;` `\|` `\t` delimiters
JSON / JSONL	Incremental	Array-of-objects or one object per line
Parquet	Columnar	Schema and counts from metadata, no row scan needed
Database query	Async	PostgreSQL, MySQL, SQLite via connection string
pandas / polars DataFrame	Columnar	Python API
Arrow RecordBatch	Columnar	Zero-copy via PyCapsule, or the Rust API
dict / bytes / BytesIO	Columnar	Python API, no dependencies
Async byte stream	Incremental	Any `AsyncRead` source (HTTP, WebSocket, …)

Measured, not promised

Every change is benchmarked in CI

Criterion runs on each push to master, and the full reports — throughput, scaling behavior, end-to-end pipeline timings — are published here, generated straight from the run’s artifacts.

Browse benchmark reports

Academic work

A reproducible performance study

dataprof is the subject of a paper submitted to IEEE ScalCom 2026, benchmarking it against YData Profiling, Polars, and pandas across execution efficiency, memory scalability, energy consumption, and zero-copy interoperability in constrained Edge AI environments.

A. Bozzo, “A Compiled Paradigm for Scalable and Sustainable Edge AI: Out-of-Core Execution and SIMD Acceleration in Telemetry Profiling,” IEEE ScalCom 2026 (under review).

Repository & reproducible benchmarks →

BibTeX

@inproceedings{bozzo2026compiled,
  author={Bozzo, Andrea},
  title={A Compiled Paradigm for Scalable and Sustainable Edge AI:
         Out-of-Core Execution and SIMD Acceleration in Telemetry Profiling},
  booktitle={2026 IEEE International Conference on Scalable Computing
             and Communications (ScalCom)},
  year={2026},
  note={Under review}
}