Pandas is the de-facto standard when working with tabular datasets in Python, but it encounters significant challenges when handling large datasets. The core issue stems from Pandas’ architecture, which relies entirely on in-memory processing. This design requires loading the entire dataset into memory, resulting in a hard limit on the size of datasets that Pandas can process. When data volumes exceed this threshold, performance degrades dramatically, often resulting in system crashes or memory exhaustion errors.

In addition, Pandas operates via single-threaded execution, processing data sequentially rather than leveraging the parallel processing capabilities of multi-core systems. This creates performance bottlenecks that become increasingly pronounced as the size of data grows.

In a previous post, I demonstrated how Polars can we used as an (almost) drop-in Pandas replacement for routine data processing tasks. Here we explore the Polars LazyFrame, which has no analog in Pandas. A LazyFrame builds up a sequence of operations without immediately executing them. Instead, it constructs a deferred query plan that is only run when explicitly requested, usually by calling the .collect() method. Deferred execution allows Polars to optimize the entire pipeline and apply techniques like predicate and projection pushdown, and process data using out-of-core techniques when necessary.

In the examples that follow, the Flight Prices dataset is used. It is a 31GB CSV file containing one-way flight itineraries found on Expedia between 2022-04-16 and 2022-10-05 for airport codes ATL, DFW, DEN, ORD, LAX, CLT, MIA, JFK, EWR, SFO, DTW, BOS, PHL, LGA, IAD and OAK. The file is larger than the client’s available memory, so it isn’t possible to analyze the full Flight Prices dataset with Pandas, which requires the entire file to be loaded into memory upfront. Given that the 31GB footprint is about 2x the RAM of the client I’m working from, it should be a perfect setting to demonstrate LazyFrame functionality.

In Polars, .scan_csv creates a LazyFrame by scanning the metadata and schema of a CSV file without loading the full dataset. In contrast, .read_csv immediately reads the entire file into memory as a DataFrame, performing eager evaluation. .scan_csv supports lazy, memory-efficient workflows, while .read_csv is eager and requires the dataset to fit in RAM.

We can preview the first 5 rows of the Flight Prices dataset calling scan_csv followed by .head(5).collect() (fetch has been deprecated):

import polars as pl 

pl.Config(tbl_rows=30)
pl.Config(float_precision=4)
pl.Config(tbl_cols=None)


# Create LazyFrame based on itineraries.csv. Dataset is available here:
#   https://www.kaggle.com/datasets/dilwong/flightprices
lf = pl.scan_csv("itineraries.csv")

# Display the first 5 rows.
first5 = lf.head(5).collect()

first5

To demonstrate how LazyFrames work, we’ll execute a simple (contrived) filter/aggregate/average pipeline to compute the average ticket price by origin airport and airline, but only for non-stop flights departing in July and August. This requires filtering, grouping, and aggregation operations that benefit from lazy evaluation and query optimization:

• Filter rows where the flightDate is in July or August.
• Group by startingAirport airport and segmentsAirlineName.
• Calculate the average totalFare for each group.
• Sort the results from highest to lowest average fare.

result = (
    lf
    .filter(
        pl.col("flightDate").str.strptime(pl.Date, "%Y-%m-%d").dt.month().is_in([7, 8]) &
        pl.col("isNonStop")
    )
    .group_by(["startingAirport", "segmentsAirlineName"])
    .agg(
        pl.col("totalFare").mean().alias("avg_price")
    )
    .sort("avg_price", descending=True)
)

Notice that we didn’t include .collect() in the result pipeline. Until .collect() is called, no computation is performed: Polars just builds a query plan.

It is possible to view the query plan by calling .explain(optimized=True). This will display the optimized logical plan that Polars will execute when .collect() is called:

print(result.explain(optimized=True))

SORT BY [col("avg_price")]
  AGGREGATE
    [col("totalFare").mean().alias("avg_price")] BY [col("startingAirport"), col("segmentsAirlineName")] FROM
    simple π 5/5 ["totalFare", "startingAirport", ... 3 other columns]
      Csv SCAN [itineraries.csv]
      PROJECT 5/27 COLUMNS
      SELECTION: [(col("isNonStop")) & (col("flightDate").str.strptime([String(raise)]).dt.month().is_in([Series]))]

See this article for more information on interpreting query plans. You can optionally install Graphviz to have the query plan rendered graphically using lf.show_graph().

When calling .collect(), we include streaming=True. This ensures Polars processes the data in a streaming (out-of-core) fashion. The dataset will be processed in small chunks rather than loading the entire dataset into memory at once.

# Execute result pipeline in streaming mode.
result = result.collect(streaming=True)

print(f"type(result): {type(result)}")
print(f"result.shape: {result.shape}")

result.head(10)

type(result): <class 'polars.dataframe.frame.DataFrame'>
result.shape: (95, 3)

Once .collect(streaming=True) is called, result materializes into a Polars DataFrame. The ultimate output is a 95×3 DataFrame with average fare price by origin airport and airline. Executing the result pipeline took about 45 seconds.

Alaska Airlines seems to have higher average fares than other airlines. This is probably a consequence of the greater distance Alaska Airlines flights cover compared with other carriers, resulting in higher average fare costs. Instead we should calculate average total fare per mile. The result2 pipeline in the next cell performs this calculation. We need to convert segmentsDistance to a numeric type since it was interpreted as a text column when initially read:

# Calculate the average total fare per mile for non-stop flights in July and August.
result2 = (
    lf
    .filter(
        pl.col("flightDate").str.strptime(pl.Date, "%Y-%m-%d").dt.month().is_in([7, 8]) &
        pl.col("isNonStop")
    )
    .with_columns(
        # Cast segmentDistance to Float64. Use strict=False to convert errors to null.
        pl.col("segmentsDistance").cast(pl.Float64, strict=False).alias("miles")
    )
    .filter(
        # Ensure numeric_distance is not null and positive.
        pl.col("miles").is_not_null() & (pl.col("miles") > 0)
    )
    .group_by(["startingAirport", "segmentsAirlineName"])
    .agg(
        # Calculate the mean of totalFare / numeric_distance
        (pl.col("totalFare") / pl.col("miles")).mean().alias("avg_price_per_mile")
    )
    .sort("avg_price_per_mile", descending=True)
).collect(streaming=True)

result2.head(20)

Polars LazyFrames unlock a new level of efficiency and scalability for data analysis in Python, enabling the handling of massive datasets with ease and speed without having to maintain a dedicated compute cluster. LazyFrames overcome many of the limitations of traditional in-memory tools like pandas by leveraging deferred execution, query optimization and out-of-core computation. For additional information on LazyFrames and streaming mode, check out the links below:

• Polars Lazy API Docs
• Polars Streaming Mode
• Why Polars is Fast