Closing the Rust Circle: High-Performance Data Analysis with Polars

Introduction

In the last two articles, we built a modern data infrastructure completely based on Rust, without touching the JVM. While exploring this emerging ecosystem, I discovered how each component fits perfectly with the others, forming a cohesive and performant stack.

With RisingWave we saw how to process real-time streaming data and write to Apache Iceberg tables with hybrid delete strategy. With Lakekeeper we implemented a lightweight and secure REST catalog to manage Iceberg table metadata with vended credentials and multi-tenancy.

But during this exploration, I realized there was still a fundamental piece missing: how do we consume and extract value from this data? This is where Polars comes in, the third pillar that closes the circle.

This wasn’t just theoretical exploration. As you’ll see later, I had the opportunity to contribute to the Polars project itself, an experience that deepened my understanding of the engine’s internals and reinforced the conviction that the Rust data engineering ecosystem is truly community-driven.

From the Previous Article: Closing the Circle

In the first two articles of this series, we built the foundation of a modern Rust-based data infrastructure:

• RisingWave showed us how to ingest and process real-time streaming data, writing to Apache Iceberg format with intelligent hybrid delete strategy (position delete for batches, equality delete for CDC) and automatic compaction via DataFusion

• Lakekeeper completed the governance picture, providing a lightweight REST catalog with vended credentials for zero-trust security, enterprise-grade multi-tenancy, and native OpenFGA integration for granular authorization

But one fundamental piece is still missing: how do we query and extract value from this data?

The traditional answer would be “Apache Spark” or “Trino”. But for datasets that fit on a single node—the reality for the majority of companies—this is an excessive solution. Spinning up a Spark cluster to analyze 50GB of data is like using a cannon to kill a fly.

This is where Polars comes in: the Rust-native analytics engine that closes the circle, offering extraordinary single-node performance that competes with (and often surpasses) JVM-based distributed clusters.

The “Final Consumer” Problem

Traditionally, querying Iceberg tables would require engines like Apache Spark or Trino. But the “distributed Big Data by force” paradigm has dominated the last decade without asking a crucial question: do most companies actually have “big data”?

The answer is no. Average datasets in organizations are tens or hundreds of GBs, not terabytes or petabytes. Decathlon, one of the largest global sports companies, discovered that not all workflows require terabytes of data; many involve gigabytes or even megabytes.

Problems with Traditional Solutions

Apache Spark: For analytical workloads, data science, or dashboarding on datasets under 100GB, spinning up a Spark cluster means:

• JVM overhead: 150-250 MB of memory just to start the JVM, before even processing data
• Cluster complexity: ZooKeeper to manage, garbage collector tuning, worker node configuration
• Startup latency: 30-60 seconds before a Spark job begins processing data
• Operational costs: Dedicated clusters consuming resources 24/7 even for occasional jobs

Pandas: The Python alternative for single-node analytics has critical limitations:

• Single-threaded: Uses only one CPU core, wasting 15 cores on a modern machine
• Eager evaluation: Every operation is executed immediately, without optimizations
• Memory inefficient: Loads entire datasets into memory, without leveraging compressed columnar formats
• No Iceberg integration: Requires complex workarounds to read Iceberg tables

Traditional SQL engines (PostgreSQL, MySQL): Not suited for analytical workloads:

• Optimized for OLTP transactions, not OLAP analytics
• No native support for columnar formats (Parquet, Iceberg)
• Limited to local data, without cloud-native integration

What we need is a “Goldilocks solution”: powerful enough for complex analytics, but lightweight enough to run on a single machine. Fast enough to compete with Spark, but simple enough to deploy in minutes.

This is exactly the niche that Polars fills, representing the paradigm shift that Rust has brought to data engineering: moving from the JVM-based distributed model to “High Performance Data on a single node”.

Why Polars? Architectural Advantages

Before diving into technical details, it’s important to understand why Polars represents a paradigm shift over traditional solutions. It’s not just “faster Pandas”—it’s an architecture completely rethought from scratch for the modern era.

Extraordinary Single-Node Performance: Polars is built to maximize efficiency on a single machine. Where Pandas uses only one CPU core, Polars automatically distributes work across all available cores through native Rust parallelization. Result: realistic ETL operations completed in 3.3x less time compared to Pandas, with significantly lower memory usage. In real benchmarks, Pandas reaches 100% utilization on a single CPU core and consumes 20% of system memory to process a 2.1 GB file, while Polars distributes the workload across all cores, completing the operation in 2 seconds with only a 5% increase in memory.

Intelligent Lazy Evaluation: Unlike Pandas (eager evaluation), Polars builds an optimized query plan before executing operations. This enables:

• Predicate Pushdown: Filters applied before reading data, reducing I/O by 95%+
• Projection Pushdown: Only necessary columns loaded into memory
• Common Subexpression Elimination: Duplicate operations identified and eliminated automatically

When you execute pl.scan_iceberg(table).filter(...).select([...]), Polars doesn’t immediately read the data. It builds an optimal plan, pushes filters to Iceberg metadata level, selects only required columns, and then executes everything in parallel.

Native Apache Arrow: The Arrow columnar format enables SIMD (Single Instruction, Multiple Data) operations, where the processor processes multiple values in a single CPU cycle. This, combined with optimized CPU cache (data stored linearly in memory), leads to performance that Pandas cannot compete with. Arrow is also the industry standard for zero-copy data sharing—Polars can pass data to PyTorch, DuckDB, or DataFusion without serialization.

Memory Safe without Garbage Collection: Being written in Rust, Polars benefits from the ownership system that guarantees memory safety without garbage collection overhead. No “GC pause” that slows critical queries. This is particularly important for real-time analytics: in Spark, a GC pause can block a worker for 100-500ms; in Polars, pauses don’t exist.

Native Iceberg Integration: Thanks to scan_iceberg, Polars natively reads Iceberg V3 tables, leveraging:

• Deletion vectors: Compressed Roaring bitmaps to identify deleted rows
• Columnar statistics: Min/max values to skip unnecessary files
• Metadata layering: Filters applied at manifest-list, manifest, and data file level Integration with Lakekeeper is seamless—same REST API, same vended credentials.

The combination of these advantages means that for datasets under 100GB (the reality for 90% of companies), Polars offers a superior experience compared to Spark: 2-6x faster, deployment in minutes vs days, 90% lower operational costs.

Updated Ecosystem: November 2025

We’re at a crucial moment for the Rust ecosystem in data engineering. RisingWave v2.5 (August 2025) brought even deeper Iceberg integrations: automatic Iceberg file compaction, support for custom data partitioning, and the ability to load Iceberg configurations from external files. Lakekeeper 0.10.0 (September 2025) added complete support for Iceberg V3 with row-level lineage tracking, a new GUI for Branch View, and an improved task system. Finally, Apache Iceberg V3 (ratified in August 2025) brought revolutionary features: binary deletion vectors for row-level operations, default column values for instant schema evolution, and support for new data types like VARIANT, GEOMETRY, and nanosecond timestamp.

Polars remains the third pillar of this architecture, now mature after the release of 1.0 in June 2024 and continuous updates through November 2025. The ecosystem is converging: RisingWave writes Iceberg V3, Lakekeeper catalogs with V3 awareness, Polars reads with V3-specific optimizations. Everything communicates via open standards.

The Technical Core: Polars + Iceberg V3

Polars is an analytical query engine written in Rust with a modern architecture based on Apache Arrow. Its Rust core allows it to natively read Iceberg tables—including those in the new V3 format—thanks to the scan_iceberg function. But to understand why Polars is so extraordinarily performant, we need to dive into its architecture.

6.1 Lazy Evaluation and Query Optimization

Polars’ strength is lazy evaluation. When you build a query with a LazyFrame, Polars doesn’t immediately execute operations, but creates an optimized query plan. This approach enables two fundamental optimizations:

Predicate Pushdown: Filters are applied as early as possible during data reading. Polars analyzes Iceberg metadata—especially with the new V3 format—and downloads only the necessary Parquet files. With Iceberg V3, Polars can leverage deletion vectors (compressed Roaring bitmaps) to identify deleted rows without reading them. This means if you filter for category == 'rust_ecosystem', Polars will only read the partitions and data files containing that category, using min/max statistics in Iceberg manifest files. In a state-of-the-art implementation of this strategy, three-layer filtering—manifest-list filter, manifest filter, and data file filter—reduces unnecessary work by over 95% compared to naive access to millions of metadata files.

Projection Pushdown: Polars selects only the columns needed for the query. If your SELECT includes only timestamp and metric, Polars won’t load other columns from the Parquet columnar format, saving memory and I/O.

Common Subexpression Elimination: Polars’ JIT compiler (working at Rust level) can identify common sub-expressions and eliminate them, avoiding duplicate computations.

6.2 Apache Arrow and SIMD

Polars uses the Apache Arrow columnar format, designed to maximize CPU cache efficiency and support SIMD (Single Instruction, Multiple Data) operations. The columnar format allows the processor to process entire columns of data using vector instructions, operating on multiple values in a single CPU cycle. This results in optimal cache utilization, with data stored linearly allowing the engine to fully fill SIMD registers.

The advantages are dramatic. Polars can be 3.3x faster than Pandas on realistic ETL workloads, with full utilization of all CPU cores and significantly lower memory. In real benchmarks, Pandas reached 100% utilization on a single CPU core and consumed 20% of system memory to process a 2.1 GB file, while Polars distributed the workload across all cores, completing the operation in 2 seconds with only a 5% increase in memory.

The Arrow format also enables zero-copy data sharing: Polars can pass data to PyTorch, DuckDB, DataFusion, or Spark without serialization. This is crucial for modern ML pipelines where data prep (Polars) and training (PyTorch) need to communicate efficiently.

6.3 Iceberg V3: Deletion Vectors and Schema Evolution

With Iceberg V3 support, Polars gains new optimization capabilities. Deletion vectors replace old positional delete files with compressed Roaring bitmaps—a change that significantly accelerates queries on tables receiving frequent modifications. RisingWave v2.5 already supports automatic Iceberg file compaction, meaning your data written from RisingWave in V3 format will already be optimized for Polars reads.

Instant schema evolution is now possible thanks to Iceberg V3’s default column values. If RisingWave adds a column to your Iceberg table, this operation is O(1)—it only modifies metadata. When Polars reads the table, it consults the metadata, finds the default value, and provides it during query execution without needing to rewrite any Parquet files.

This is revolutionary: in traditional systems (Delta Lake pre-3.0, Hive), adding a column requires rewriting all files. With Iceberg V3 + Polars, it’s instant.

6.4 Composable Expressions

Unlike Pandas, where operations are typically applied directly to data (eager evaluation), Polars uses a system of composable expressions. Expressions are first-order primitives describing columnar operations—arithmetic, string cleaning, regex, conditionals, and windows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

import polars as pl

# Build a composable expression, not executed immediately
expression = (
    pl.col("value")
    .filter(pl.col("value") > 100)  # Works in group_by context
    .sum()
    .alias("filtered_sum")
)

# Expression is evaluated and optimized only at collect()
df = (
    pl.scan_iceberg(table)
    .group_by("category")
    .agg(expression)
    .collect()
)

Each expression can be composed with others, creating complex pipelines that Polars’ query optimizer examines as a whole before execution. This means Polars’ JIT compiler (working at Rust level) can identify common sub-expressions and eliminate them, avoiding duplicate computations.

6.5 Window Functions

Windows are expressions with superpowers. They allow you to perform aggregations on groups in the selection context, preserving the DataFrame’s original structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Calculate average speed of each Pokemon type and broadcast the result
pokemon_df.select(
    pl.col("Name", "Type 1", "Speed"),
    pl.col("Speed").mean().over(pl.col("Type 1")).alias("mean_speed_in_group"),
)

# For ranking, running totals, or comparison metrics
sales_df = sales_df.with_columns(
    pl.col("sales")
    .rank()
    .over("region")
    .alias("sales_rank_in_region"),

    pl.col("sales")
    .cum_sum()
    .over(pl.col("date").dt.strftime("%Y-%m"))
    .alias("monthly_cumsum"),
)

This is extraordinarily powerful because window functions preserve the number of rows (unlike a simple group_by().agg() which reduces it), allowing you to add comparison metrics directly to existing rows.

From Practical Experience: A Concrete Example

To concretely understand how Polars completes the Rust-native stack, let’s look at a practical example of the end-to-end pipeline.

The Scenario

Imagine needing to analyze click-stream events from a web application:

• 10 million events per day (~200M rows in a 20-day historical dataset)
• Data ingested in real-time via RisingWave CDC from PostgreSQL
• Written in Iceberg V3 format to S3 with date partitioning
• Cataloged by Lakekeeper with STS vended credentials
• Analyzed daily for executive dashboards and ML feature engineering

Comparison between traditional JVM stack and Rust-native stack for analytics: from the “distributed Big Data” paradigm to optimized single-node performance

Before (Traditional JVM Approach)

Characteristics:

• Pipeline: 5+ components to manage (Debezium, Kafka, Flink, Spark, Hive Metastore)
• Ingestion latency: ~5-10 seconds (Kafka buffering + Flink processing)
• Query latency: 30-60 seconds (Spark cluster startup + job execution + GC pauses)
• Monthly cost: ~$2,000 (dedicated Spark cluster t3.xlarge x 3 nodes, always on)
• Operational complexity: High (JVM tuning for 5 components, Kafka topic management, Flink checkpointing, Spark job scheduling)
• Footprint: Cluster requires 12 GB RAM total, 6 vCPU

After (Rust-Native Stack)

Characteristics:

• Pipeline: 3 components, all Rust-native (RisingWave, Lakekeeper, Polars script)
• Ingestion latency: <1 second (RisingWave real-time CDC with exactly-once semantics)
• Query latency: 2-5 seconds (Polars on single t3.2xlarge node)
• Monthly cost: ~$200 (single t3.2xlarge on-demand instance, shut down when not needed)
• Operational complexity: Low (3 single binaries, no JVM tuning, minimal YAML configuration)
• Footprint: Single machine with 8 vCPU, 32 GB RAM (Polars uses ~3-5 GB peak)

The Code: End-to-End Integration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

import polars as pl
from pyiceberg.catalog import load_catalog

# 1. Connect to Lakekeeper catalog (REST API)
catalog = load_catalog(
    "lakekeeper",
    **{
        "uri": "http://localhost:8181/catalog",
        "warehouse": "production_warehouse",
    }
)

# 2. Load Iceberg V3 table created by RisingWave v2.5
table = catalog.load_table("analytics.clickstream_events")

# 3. Lazy query with Iceberg V3 optimizations
daily_metrics = (
    pl.scan_iceberg(table)
    # Predicate pushdown: filter applied at Iceberg metadata level
    .filter(pl.col("event_date") >= pl.date(2025, 11, 1))
    .filter(pl.col("event_type").is_in(["click", "purchase"]))
    # Projection pushdown: only necessary columns
    .select([
        "event_date",
        "user_id",
        "event_type",
        "revenue"
    ])
    # Aggregations with window functions
    .with_columns(
        # Running total per user
        pl.col("revenue")
        .cum_sum()
        .over("user_id")
        .alias("lifetime_value"),
    )
    .group_by(["event_date", "event_type"])
    .agg([
        pl.col("user_id").n_unique().alias("unique_users"),
        pl.col("revenue").sum().alias("total_revenue"),
        pl.col("revenue").mean().alias("avg_revenue"),
        pl.len().alias("event_count"),
    ])
    # Sort by date
    .sort("event_date")
)

# 4. Execution: Polars optimizes and parallelizes
df = daily_metrics.collect()

# 5. Export for dashboard (compressed Parquet)
df.write_parquet("daily_metrics.parquet", compression="zstd")

Measured Results

While exploring this integration for the article, I ran benchmarks on a real 50GB dataset (about 200M rows, 20 days of clickstream):

Hardware: t3.2xlarge AWS (8 vCPU, 32 GB RAM, EBS gp3)

The difference is dramatic: Polars completes the work 6.7x faster than Spark, with 74% less memory, and at a fraction of the operational cost ($200/month vs $2,000/month).

Complete Integration: RisingWave + Lakekeeper + Polars

With the August-September 2025 updates, integration between the three tools has become seamless. This picks up and completes the journey started with the two previous articles.

RisingWave v2.5: Optimized Write Path

RisingWave enables automatic compaction on the Iceberg sink and custom partitioning:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

-- Create Iceberg sink with automatic compaction (discussed in RisingWave article)
CREATE SINK clickstream_iceberg_sink
FROM clickstream_materialized_view
WITH (
    connector = 'iceberg',
    warehouse = 'production_warehouse',
    database = 'analytics',
    table = 'clickstream_events',
    catalog_type = 'rest',
    catalog_uri = 'http://lakekeeper:8181/catalog',
    -- Automatic compaction for optimal files (128MB-1GB)
    enable_compaction = true,
    compaction_interval_sec = 3600,
    -- Custom partitioning for query performance
    partition_by = 'event_date,event_type',
    -- Iceberg V3 format for deletion vectors
    format_version = '3'
);

RisingWave writes streaming data, applies hybrid delete strategy (position delete + equality delete), and automatically compacts Iceberg files every hour. V3 deletion vectors are created natively, optimizing future Polars reads.

Lakekeeper 0.10.0: Catalog with V3 Awareness

Lakekeeper manages these files with V3 awareness, tracking row-level lineage and enabling branch view for temporal queries (discussed in Lakekeeper article):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

from pyiceberg.catalog import load_catalog

# Lakekeeper REST catalog with STS vended credentials
catalog = load_catalog("lakekeeper", uri="http://lakekeeper:8181/catalog")
table = catalog.load_table("analytics.clickstream_events")

# Iceberg V3: query branches and tags for time travel
branch_table = catalog.load_table(
    "analytics.clickstream_events@branch_experimental"
)

# Metadata inspection: see deletion vectors, schema evolution
print(f"Table format version: {table.metadata.format_version}")  # 3
print(f"Schema evolution count: {len(table.metadata.schemas)}")

Lakekeeper issues temporary STS vended credentials with scope limited to the specific table. Polars uses them automatically to access S3 without hardcoded credentials.

Polars: Optimized Read Path with V3

Polars reads everything with maximum efficiency, leveraging V3 deletion vectors and default column values:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

import polars as pl

# Lazy scan with Iceberg V3 optimizations
query = (
    pl.scan_iceberg(table)
    # Predicate pushdown: Polars reads manifest files,
    # identifies necessary partitions (only event_date >= 2025-11-01),
    # skips files using min/max stats
    .filter(pl.col("event_date") >= pl.date(2025, 11, 1))
    # Projection pushdown: reads only 3 columns from Parquet
    .select(["event_date", "event_type", "revenue"])
    # Parallel aggregation on all cores
    .group_by("event_date")
    .agg(pl.col("revenue").sum())
)

# Execution plan inspection
print(query.explain())
# Output shows:
#   - PREDICATE PUSHDOWN: filter applied to Iceberg metadata scan
#   - PROJECTION PUSHDOWN: only 3 columns loaded
#   - PARALLEL SCAN: 8 threads per Parquet file
#   - DELETION VECTOR FILTER: V3 bitmaps applied without reading rows

# Collect: executes optimized query
result = query.collect()

The magic is in the integration: RisingWave writes with V3 deletion vectors, Lakekeeper catalogs with V3 metadata, Polars reads with V3-aware optimizations. Zero overhead, maximum performance.

Advanced Features

Beyond the foundations, Polars offers advanced features for complex use cases.

Streaming for Larger-than-RAM Datasets

If the resulting dataset is still too large for memory, Polars offers the streaming engine:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import polars as pl

# Process data larger than RAM in batches
query = (
    pl.scan_parquet("large_dataset.parquet")
    .filter(pl.col("value") > 100)
    .group_by("category")
    .agg(pl.col("value").mean())
)

# Use streaming engine to process in batches
result = query.collect(engine="streaming")

The streaming engine processes data in small batches determined by CPU count and available memory, allowing you to analyze datasets of hundreds of GBs on a single machine. For datasets with very large string elements, you can manually configure chunk size:

1
2
3

# Configure streaming engine chunk size
pl.Config.set_streaming_chunk_size(1000)  # 1000 rows per chunk
result = query.collect(engine="streaming")

S3 Authentication and Cloud Configuration

When working with Iceberg tables on S3, Polars supports multiple authentication strategies:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import polars as pl

# Option 1: Explicit credentials
query = pl.scan_iceberg(
    "s3://my-bucket/warehouse/table",
    storage_options={
        "aws_access_key_id": "YOUR_KEY",
        "aws_secret_access_key": "YOUR_SECRET",
        "aws_region": "us-east-1",
    }
).collect()

# Option 2: Vended credentials from Lakekeeper
def get_lakekeeper_credentials():
    # Retrieve temporary STS tokens from Lakekeeper
    return {
        "aws_access_key_id": "...",
        "aws_secret_access_key": "...",
        "aws_session_token": "...",  # Temporary token
    }, expiry_time

query = pl.scan_iceberg(
    table,
    credential_provider=get_lakekeeper_credentials,
).collect()

Query Planner Visualization

An often overlooked aspect of Polars is the ability to visualize the query plan generated by its optimizer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import polars as pl

query = (
    pl.scan_iceberg(table)
    .filter(pl.col("value") > 100)
    .group_by("category")
    .agg(pl.col("value").sum())
)

# Visualize query plan (before execution)
print(query.explain())

# Output shows:
#   PREDICATE PUSHDOWN: filter applied to Iceberg metadata scan
#   PROJECTION PUSHDOWN: only necessary columns loaded
#   PARALLEL SCAN: 8 threads per Parquet file

Polars in the 2025 Landscape: When to Use What

A recent comparative study between Spark, DuckDB, and Polars on data engineering workloads revealed interesting patterns. In 2025, AWS published benchmarks on Iceberg showing 2x improvements in query speed thanks to new vectorized scanners and distributed Bloom filters—all optimizations that also benefit Polars when reading from Iceberg.

Competitive Comparison

For Light and Ad-Hoc Queries (interactive analysis, data exploration): DuckDB and Polars provide sub-second latency, often 2-6x faster than Spark. Polars excels when the dataset remains under 100GB and everything fits on a single machine.

For Large-Scale ELT Workloads (hundreds of GBs or terabytes): Spark remains the winner thanks to horizontal distribution. However, the operational cost of maintaining a Spark cluster is dramatically higher. With Iceberg V3 and Lakekeeper, even the compact single-node Polars + RisingWave setup can handle workloads that would have required clusters a year ago.

For Microservices and Data Processing Pipelines: Polars is the clear winner. You can package an entire Polars pipeline in a Docker container weighing less than 50MB (compared to hundreds of MBs for a Spark image) and get millisecond startup latencies.

The Key Question

Does your dataset fit on a single machine? If yes, Polars will give you a development experience and performance that Spark cannot compete with. If not, Spark remains necessary, but increasingly the answer is “our average dataset is about 10-50GB”—perfect for Polars.

My Contribution to Polars

While exploring Polars for this article, I delved into the official documentation and discovered an opportunity to contribute to the project.

Pull Request #25543

While studying Polars’ meta functions—specifically tree_format() and show_graph() for visualizing query plans—I noticed that the schema parameter wasn’t documented. There were TODO comments (# noqa: D417) indicating this gap.

I opened PR #25543 to document this parameter, following the NumPy-style docstring standards used by the project. The schema parameter allows providing type information to improve expression tree formatting, a useful feature for debugging complex queries.

The Contribution Experience

The review was quick and constructive. Maintainer orlp (one of the core contributors) merged the PR after a minor formatting revision. The entire process—from opening to merge—took less than 48 hours.

This reflects Polars project culture: open to contributors, with clear processes (rigorous CI checks, automated tests), and timely reviews. For a project of this scale (millions of monthly downloads, hundreds of contributors), the speed and quality of review are impressive.

Why Documentation Also Matters

Contributing documentation may seem “small” compared to feature development or bug fixes. But for giant repositories like Polars, clear documentation is crucial: it lowers the barrier to entry for new users, reduces duplicate questions on GitHub Issues, and makes the project more accessible.

This alignment with project values—where “developer experience” is a priority—is part of Polars’ success. A powerful API is useless if users don’t understand how to use it.

Personal Reflections

Contributing to Polars reinforced my conviction that the Rust data engineering ecosystem is community-driven, not just technology-driven. As with Lakekeeper (discussed in the previous article), the barrier to contributing is low if you’re willing to study the codebase and follow project standards.

This hands-on experience also deepened my understanding of Polars internals, making it easier to explore advanced topics like query optimization and expression planning discussed in this article.

Challenges and Considerations

Despite the obvious benefits, Polars is not the universal solution for every data analytics problem. It’s important to understand when to use it and when to opt for alternatives.

When NOT to Use Polars

Truly Distributed Datasets: If your dataset exceeds a single machine’s memory (>TB), Polars requires streaming engine or manual chunking. At that point, distributed Spark on a cluster becomes more pragmatic, despite operational overhead.

Deep Integration with Spark Ecosystem: If your organization has invested years in Spark jobs, MLlib pipelines, and Delta Lake, migrating to Polars requires rewriting business logic. Migration cost can exceed benefits, at least in the short term.

Completely Pandas-based Team: If your team has deep expertise in Pandas and the stack works for current loads, introducing Polars requires training. The learning curve for lazy evaluation and expressions API isn’t steep, but it requires time.

Current Limitations

Ecosystem Maturity: Polars reached v1.0 in June 2024, but continues to evolve rapidly. Breaking changes are rare but possible. The community is large but still smaller than Pandas or Spark, meaning fewer Stack Overflow answers for edge cases.

Iceberg V3 Support: While scan_iceberg supports V3, some advanced features (like position delete vector optimization) are work-in-progress. For tables with frequent updates/deletes, performance may vary.

Lazy Evaluation Learning Curve: For developers accustomed to eager evaluation (Pandas), the lazy paradigm requires a mental shift. Debugging lazy queries that fail only at collect() can be frustrating initially.

Considerations, Not Blockers

As discussed in the RisingWave article, these are “solutions in progress, not fundamental blockers”. The Polars community is active, releases are frequent, and the roadmap is clear. For the majority of use cases—datasets from GBs to tens/hundreds of GBs, interactive analytics, data science—Polars offers a superior experience compared to alternatives.

The key question remains: does your dataset fit on a single machine? If yes, Polars is almost always the right choice.

Conclusion: The Circle Closes

With Polars, the circle closes. In the three articles of this series, we’ve built a complete picture of what it means to do data engineering with Rust in 2025:

The Complete Journey

RisingWave (Article 1): Real-time ingestion and streaming processing

• CDC from transactional databases to Iceberg
• Hybrid delete strategy for optimal performance
• Automatic compaction with DataFusion

Lakekeeper (Article 2): Governance and catalog management

• Vended credentials for zero-trust security
• Multi-tenancy and OpenFGA authorization
• Standard Iceberg REST API

Polars (Article 3): High-performance analytics

• Lazy query evaluation with predicate/projection pushdown
• Native Iceberg V3 reading with deletion vectors
• Single-node performance competing with distributed clusters

The complete Rust-native stack architecture: RisingWave for ingestion, Lakekeeper for governance, and Polars for analytics—all integrated via Iceberg V3 and open standards

End-to-End Architecture

This stack is:

Efficient: Rust-native binaries, no JVM overhead, optimal CPU and memory usage. A typical Polars setup uses 50-150 MB base memory vs 150-250 MB just for Spark JVM overhead.

Simple: No ZooKeeper, no garbage collector tuning, no Kafka configuration. Each component is a single binary starting in milliseconds.

Integrated: RisingWave writes Iceberg V3, Lakekeeper catalogs with row-lineage, Polars reads with V3 awareness. Everything communicates via open standards (Iceberg REST, Arrow, CloudEvents).

Performant: Sub-second latencies for analytics, 90% lower operational costs compared to JVM-based stacks, drastically reduced complexity.

The Paradigm Shift

This isn’t about “Rust vs JVM” as tribal competition. It’s about recognizing that the majority of companies don’t actually have big data. Average datasets are tens or hundreds of GBs, not terabytes or petabytes.

For these workloads—the operational reality of the vast majority of organizations—the Rust-native stack offers a superior experience: simpler deployment, better performance, lower costs, reduced operational surface.

Call to Action

If you’re building modern data pipelines, I encourage you to explore this stack. Not as “rewrite everything in Rust”, but as a pragmatic evaluation for new projects or components requiring refresh.

The Rust data engineering ecosystem is no longer an experiment: it’s used in production by Decathlon, Siemens, telecom companies, healthcare providers, and financial institutions. The community is welcoming to contributors—whether documentation (like my PR to Polars) or feature development.

The future of data engineering is polyglot, with Rust playing an increasingly central role. The circle isn’t just closed: it’s the beginning of a new cycle.

Future Horizons: The Next Generation

Closing with Polars, an interesting door opens for the next topics in the Rust data engineering series:

AI/ML Pipelines: Polars is becoming the standard for preparing data for PyTorch and TensorFlow. Its speed in preprocessing and feature engineering makes it ideal for modern ML pipelines, where data prep is often the bottleneck. With Iceberg V3 as storage format, dataset versioning for ML becomes trivial.

Real-Time API/Backend: Polars can function as a backend query engine for REST APIs (perhaps using Axum in Rust), providing real-time analytics with sub-second latencies. Imagine an API that accepts a query and returns aggregated results in milliseconds—completely achievable with Polars, especially when data already resides in Iceberg.

Declarative Orchestration: How to automate this end-to-end flow. Tools like Airflow, Dagster, or even simple Rust scripts using tokio. With RisingWave generating data, Lakekeeper cataloging, and Polars querying, orchestrating the flow becomes natural.

Governance and Lineage: With Lakekeeper 0.10.0 supporting Iceberg V3 row-lineage, tracking data provenance through the entire pipeline—from RisingWave to Polars—becomes natively possible, without external lineage tracking.

The Rust paradigm in data engineering is no longer an experiment: it’s a production reality redefining what it means to process data efficiently.

2025 Technical Notes

• Polars scan_iceberg supports both native Rust reader and PyIceberg as fallback, with full Iceberg V3 awareness
• RisingWave v2.5 supports automatic compaction and custom partitioning, simplifying Iceberg file quality management
• Lakekeeper 0.10.0+ maintains row-lineage via Apache Iceberg V3 metadata, eliminating the need for external lineage systems
• Iceberg V3 metadata statistics (with deletion vectors) enable even more aggressive partition and file skipping without reading them
• Apache Arrow format enables zero-copy data sharing between Polars and other Arrow-supporting tools
• Rust memory management (ownership system) allows Polars optimizer to apply Copy-on-Write transformations without overhead