Benchmarking Polars vs pandas on Groupby-Heavy Analytics Workloads

Introduction

Does Polars Actually Beat pandas on Groupby-Heavy Workloads?

"Polars is faster than pandas" is true in aggregate, true in most published benchmarks, and useless for the pipeline you actually maintain. The workload that matters for most analytics code is narrower than a benchmark suite: read a frame, group by one or two keys, aggregate a few columns, repeat a few dozen times per job. This post benchmarks Polars against pandas on exactly that shape — groupby-heavy Python dataframes workloads — and tests whether the gap survives your real cardinality and dtype mix.

So I built a harness for it. My position after running it: Polars wins decisively on multi-key and filtered aggregations, wins modestly on single-key sums, and loses on small frames where per-call overhead dominates. Lazy mode is not a rounding error — it is the single biggest win on the filtered query, and it comes from pushdown, not from Rust.

Where These Numbers Come From

This post started as a reaction to a general Polars guide making broad performance claims. General guides are a poor source for numbers — but they are a fine source for the list of features worth measuring. Everything numeric below comes from a harness I ran. Every claim about why the numbers look that way is either pulled from the official Polars and pandas documentation or marked as my inference.

Why Groupby Benchmarks Are Easy to Get Wrong

The Shape of a Groupby: Hash or Sort, Aggregate, Assemble

A groupby runs in three phases, and their costs scale differently. First, the engine assigns each row to a group — by hashing the key, or by sorting the frame and taking contiguous runs. Second, it applies the aggregation kernel per group. Third, it assembles the output: allocating a new column per aggregate and writing values into per-group slots.

Which phase dominates depends on cardinality. With 12 groups, assembly is trivial and hashing is cheap; nearly all the time goes into the kernel. With 2 million groups, assembly is the expensive phase, and that is where a columnar, offset-based layout starts to matter more than raw kernel speed.

What Usually Dominates pandas Wall Time

Three things kept showing up when I profiled the pandas runs:

• Object dtypes for string keys. Hashing a Python str per row is not comparable to hashing a fixed-width integer or a dictionary-encoded string.
• Index alignment. pandas builds a MultiIndex for multi-key groupbys, and every aggregation result gets realigned against it.
• Single-threaded aggregation. pandas does not parallelize groupby().agg() across groups. Polars does.

The category dtype removes the first cost, which is why there is a separate pandas column with category keys.

Why a One-Line df.groupby(...).agg(...) Comparison Tells You Almost Nothing

Most side-by-side snippets compare two operations that are not actually the same. They use object-dtype strings in pandas and Utf8 in Polars. They measure pandas on a cold frame and Polars on one that was just constructed. They run a single iteration and report whatever the first, GC-polluted timing was. I have seen all three in published posts.

The Benchmark Harness: Dataset, Queries, and Timing Methodology

Dataset Shape, Dtype Mix, and Cardinality

One frame, 8 million rows, seven columns, generated from a seeded NumPy RNG so both engines see identical bytes:

• ts — datetime, one year of second-resolution timestamps
• region — 12 distinct values (low cardinality)
• category — 30 distinct values
• channel — 5 distinct values
• account_id — 2,000,000 distinct integers (high cardinality, included to test phase-three assembly cost)
• amount — float64, gamma-distributed
• qty — int64

The Query Set

PANDAS_QUERIES = {
"q1_sum_region": lambda d: d.groupby("region", observed=True)["amount"].sum(),
"q2_mean_region_cat": lambda d: d.groupby(["region", "category"], observed=True)["amount"]
                                 .agg(["mean", "std"]),
"q3_filtered": lambda d: d[d["ts"] >= "2024-07-01"]
                          .groupby("channel", observed=True)["qty"].sum(),
"q4_top3_region": lambda d: d.sort_values("amount", ascending=False)
                             .groupby("region", observed=True).head(3),
}

POLARS_QUERIES = {
"q1_sum_region": lambda d: d.group_by("region").agg(pl.col("amount").sum()),
"q2_mean_region_cat": lambda d: d.group_by(["region", "category"])
                                 .agg(pl.col("amount").mean().alias("mean"),
                                      pl.col("amount").std().alias("std")),
"q3_filtered": lambda d: d.filter(pl.col("ts") >= pl.datetime(2024, 7, 1))
                          .group_by("channel").agg(pl.col("qty").sum()),
"q4_top3_region": lambda d: d.sort("amount", descending=True)
                             .group_by("region").head(3),
}

Timing Methodology

Three warmup runs per query, then seven measured runs. I report the minimum and the median; if those two diverged by more than 25 percent I re-ran the query, since that usually means something else was running on the machine. POLARS_MAX_THREADS=1 for the main table, so a single-threaded Polars is compared against single-threaded pandas. Multi-threaded Polars gets its own row.

POLARS_MAX_THREADS=1 python bench_groupby.py --engine all --reps 7

Memory Methodology

Peak RSS is read per-process via resource.getrusage(RUSAGE_SELF).ru_maxrss inside a subprocess that runs exactly one query. That is a coarse number — it includes the interpreter and the input frame — but it is honest, because it is what your container's memory limit actually sees. Per-step allocator numbers would be more precise and more misleading, since intermediate Arrow buffers are freed lazily.

Benchmark Results: Timings, Memory, and Fairness

Query-by-Query Timings

All times in seconds, minimum [median], single-threaded, one 8M-row frame. "Arrow dtypes" means pdf.convert_dtypes(dtype_backend="pyarrow") with pd.options.mode.copy_on_write = True.

With POLARS_MAX_THREADS unset (8 threads on this machine), q2 eager drops from 0.74s to 0.11s and q4 from 1.06s to 0.19s. The parallel speedup is real, and pandas has no equivalent for these kernels.

Where the Gap Is Largest, and Where It Nearly Disappears

The widest ratios are q4 and q2 — the queries with more than one key or a sort step. My reading is that q4's gap is mostly pandas' sort_values plus groupby().head(), which materializes a sorted intermediate and then re-scans it. Polars' sort followed by group_by().head() does the same logical work, but with a cheaper sort over a columnar buffer.

On small frames the gap nearly disappears. On a 50,000-row version of q1:

pandas   : 3.4 ms  [median]
polars   : 6.1 ms  [median]

Polars loses. Per-call overhead, plan construction, and dtype dispatch are not free, and at 50k rows they cost more than the aggregation itself. If your workflow is a notebook with interactive one-off groupbys over frames in the tens of thousands of rows, switching engines will make your life worse.

Memory: Columnar Layout vs pandas 2.x

Peak RSS for q2, measured per subprocess:

The pandas figure is dominated by the MultiIndex for ["region", "category"] plus the intermediate aggregation frame. Copy-on-Write helps because it eliminates several defensive full-frame copies that older pandas took on the filter and sort paths. Arrow-backed dtypes narrow the string-key memory, which is why the second column beats the first — but neither closes the gap to Polars.

I did not include a disk-spilling pandas run, so the lazy column is not a streaming-out-of-core win; it is the same eager plan with pushdown. Actual out-of-core behavior is a separate experiment I have not run.

Fairness Check: PyArrow Dtypes and Copy-on-Write, Not a Straw Man

The Arrow dtypes + CoW column is the fair baseline, and it beats default pandas on every query. Anyone benchmarking default pandas against Polars and reporting a 5x gap on q1 is measuring dtype choices as much as engine design. Side by side on the same bytes, pandas 2.x closes maybe 20–25 percent of the gap. Polars keeps the rest.

Lazy Evaluation and the Query Optimizer

Reading a Plan with .explain()

Build the query against a LazyFrame and print the plan before collecting:

q = (pl.scan_parquet("data/events.parquet")
       .filter(pl.col("ts") >= pl.datetime(2024, 7, 1))
       .group_by("channel")
       .agg(pl.col("qty").sum()))

print(q.explain())

Abridged output from my run — your column ordering will differ:

AGGREGATE
	[col("qty").sum()] BY [col("channel")]
  FROM
	FILTER [(col("ts")) >= (2024-07-01 00:00:00)]
	FROM
	Parquet SCAN data/events.parquet
	PROJECT 3/7 COLUMNS

PROJECT 3/7 COLUMNS is projection pushdown: only ts, channel, and qty get read off disk. The FILTER node sitting below the scan is how predicate pushdown shows up for Parquet row-group pruning.

The Eagerness Trap

Benchmark df.group_by(...).agg(...) on an eager DataFrame and you will conclude the optimizer does nothing. In eager mode that is correct — there is no plan. The scan already happened, the columns are already in memory, and there is nothing left to push down. The optimizer is a property of LazyFrame, not of Polars.

Where Lazy Mode Actually Changes the Plan

Lazy mode moved my numbers in two places. On q3, the filtered aggregation dropped from 0.39s to 0.20s: half the win is not deserializing filtered-out rows, half is not materializing the full column set. On the small-frame case, lazy mode was slower than eager, because plan construction is pure overhead when there is nothing to prune.

For out-of-core work, collect(streaming=True) is the flag in Polars 1.x, with a newer engine="streaming" argument appearing in recent releases. I did not verify the streaming engine on a frame larger than RAM, so treat that recommendation as untested on my side.

Migrating From pandas to Polars Without Rewriting Everything

Idiom Translation Table

That last row is the most useful one in the table. over() is a window expression, not a groupby-join, so it skips the join-back that pandas' transform performs.

What Does Not Translate Cleanly

• The index. Polars has none. Code relying on .loc, .iloc, or a meaningful MultiIndex needs restructuring, not translation.
• In-place mutation. Polars frames are immutable; every operation returns a new frame. Code full of df["x"] = ... becomes a chain of with_columns.
• .apply. The equivalent, map_elements, invokes Python per element and is slow — often slower than pandas. If you lean on .apply, migration will hurt until you rewrite that logic as expressions.
• Implicit alignment. pandas aligns on index during arithmetic; Polars does not. Silent behavior differences live here.

A Staged Migration Path

Do not port the whole pipeline. Keep pd.read_csv / pd.read_sql if your ingestion code depends on pandas quirks, then hand off at the Arrow boundary:

pdf = pd.read_parquet("data/events.parquet")
result = (
    pl.from_arrow(pdf.to_arrow())   # zero-copy for Arrow-backed columns
      .lazy()
      .filter(pl.col("amount") > 0)
      .group_by(["region", "category"])
      .agg(pl.col("amount").sum())
      .collect()
)
out = result.to_pandas()            # convert back at the edge

pl.from_arrow skips the pandas-to-Polars copy for Arrow-backed dtypes. The return trip to pandas is only worth paying for if downstream code genuinely needs pandas.

When Migrating From pandas Does Not Pay Off

Small Frames and Interactive Analysis

Below roughly a few hundred thousand rows, Polars' per-call overhead eats the kernel advantage — the 50k-row number above is the evidence. Notebook sessions with one-off groupbys over modest frames are better served by pandas with Arrow dtypes.

Heavy pandas-Ecosystem Dependencies

Statsmodels, scikit-learn's set_output, and a long tail of plotting and reporting libraries speak pandas natively. If most of your runtime lives in those calls rather than the groupby, the groupby is not your bottleneck and migration will not move your numbers.

The Maintenance Cost of Two Dataframe Libraries

Running both engines means every new engineer has to know which frame they are holding and what the boundary conversion costs. That is a real tax. My rule: migrate a workload only when the groupby is measurably a large fraction of total job time and the job runs often enough that the seconds matter.

Reproduction Notes

Environment and Command

Machine   : Apple M2 Pro, 10 cores, 32 GB RAM, macOS 14.5
Python    : 3.12.4
polars    : 1.6.0
pandas    : 2.2.2
pyarrow   : 17.0.0
numpy     : 1.26.4
Threads   : POLARS_MAX_THREADS=1 unless noted
Command   : POLARS_MAX_THREADS=1 python bench_groupby.py --engine all --reps 7

Known Limitations

• One machine, one architecture, no ARM-vs-x86 comparison.
• Synthetic data with a uniform key distribution. Skewed keys change the picture — a single hot group can serialize a parallel aggregation.
• The pandas baseline is single-threaded because that is what pandas gives you; the multi-threaded Polars row is reported separately rather than mixed into the main table.
• Peak RSS is per-process, so it includes interpreter and input-frame cost.

Further Reading

Primary Sources

• Polars user guide: Lazy API — how LazyFrame and collect() fit together
• Polars user guide: Query optimizations — the pushdown rules behind the plan output above
• Polars user guide: Streaming — current engine flag and out-of-core semantics
• pandas 2.2.0 release notes — Copy-on-Write and Arrow dtype behavior changes
• pandas user guide: Copy-on-Write — why the fair-baseline column moved
• pandas user guide: PyArrow-backed dtypes — dtype_backend="pyarrow" semantics