In the ever-evolving field of data science, effective data manipulation tools are essential. Enter Polars, a Rust-based library garnering attention within the data community. Boasting impressive speed and versatile capabilities, Polars is redefining our data management practices. In this blog, we delve into Polars’ core functions and practical applications, shedding light on how it empowers data professionals to efficiently tackle complex tasks.

For those well-versed in Pandas, Polars offers a blend of familiarity and innovation. Although this document is not designed to substitute the official documentation, it serves as a tool to provide you with insights into the capabilities that polars offers. Our goal is to ensure the continuous updating of this document.

Our exploration of Polars is guided by insights from the Polars User Guide, Kevin Heavey’s perspectives in Modern Polars and Matt Harrison’s engaging tutorial on Polars at PyCon. We kickstart our exploration with Matt Harrison’s shared dataset, the US Department of Energy’s Automobile Fuel Economy data. Let’s begin this journey!

Setup

import pandas as pd
import polars as pl
import polars.selectors as cs

import seaborn as sns
import matplotlib.pyplot as plt

pl.Config.set_tbl_rows(2)  # limit the numbers of rows printed

Polars Version

We are using Polars 0.18.8 for this demonstration.

pl.__version__

Given the large size of the dataset and the fact that some columns aren’t really useful for us, we’ll start by picking out the information we actually need. Feel free to use the code below to work with your dataframe.

df = pl.read_csv("vehicles.csv", null_values="NA")

# process and save parquet for later use
(
    df.select(
        pl.col(
            "make",
            "model",
            "drive",
            "cylinders",
            "displ",
            "fuelType",
            "mpgData",
            "city08",
            "highway08",
            "comb08",
            "year",
        )
    )
    .with_columns(
        pl.col("make").cast(pl.Categorical, strict=False),
        pl.col("model").cast(pl.Categorical, strict=False),
        pl.col("drive").cast(pl.Categorical, strict=False),
        pl.col("fuelType").cast(pl.Categorical, strict=False),
        pl.col("cylinders").cast(pl.Int8, strict=False),
        pl.col("displ").cast(pl.Float32, strict=False),
        pl.col("mpgData").map_dict({"N": False, "Y": True}),
        pl.col("city08").cast(pl.Int8, strict=False),
        pl.col("highway08").cast(pl.Int8, strict=False),
        pl.col("comb08").cast(pl.Int8, strict=False),
        pl.col("year").cast(pl.Utf8).str.to_datetime("%Y").cast(pl.Date),
    )
    .write_parquet("vehicles.parquet")
)
df = pl.read_parquet("vehicles.parquet")

Inspect the polars namespace

Taking a closer look at the Polars namespace can be really helpful as you start to get to know the library. By exploring the functions, methods, and attributes available within the namespace, you’ll get a better sense of what the library can do. This exploration sets the groundwork for you to use the library more effectively and make the most of its features.

By issuing print(dir(pl)), we can explore the accessible functions within the namespace.

# available functions

print(dir(pl))

The same applies to the methods available for DataFrames.

# methods defined on dataframes

print(dir(df))

…and expressions for the Series

# available expressions

print(dir(pl.col("foo")))

Access documentation

If you’re familiar with Python, you might recognize that you can access documentation by either using __doc__ or by calling the help() function.

print(df["year"].dt.convert_time_zone.__doc__)

help(pl.head)

Indexing

If you’ve spent some time working with Pandas, you might find it interesting that Polars doesn’t rely on the concept of an index like Pandas does. But no need to fret—there are several other ways we can still access the rows in Polars. One of the straightforward methods—although I won’t delve too deeply into it here—is using the head, tail, and sample methods.

Use take for Series or Expressions

An interesting approach is through Polars’ take method. This method can be used with Expr, Series, or np.ndarray. However, it’s worth noting that if you intend to utilize it with DataFrames, a preliminary conversion to Series is necessary.

df.select(pl.col("make")).to_series().take(list(range(0, 5)))

Using filter

Similar to Pandas, Polars features a filter method that assists in the process of selectively filtering rows based on one or multiple conditions.

Evaluate a single condition

df.filter(pl.col("model") == "Testarossa")

Combine multiple conditions

df.filter((pl.col("model") == "Testarossa") & (pl.col("make") == "Ferrari"))

Imitate pandas’ index using with_row_count

Even in the absence of a traditional index, you can make use of the with_row_count method in Polars. This clever method comes to the rescue for tasks like indexing and filtering, providing an alternative approach.

(df.with_row_count().filter(pl.col("row_nr") == 5))

Use is_in

The is_in function can be employed to verify whether elements of a given expression are present within another Series. We can use this to filter our rows.

df.filter(pl.col("cylinders").is_in([i for i in range(6, 8)]))

Select columns

Before we dive into the realm of selecting columns, it’s important to introduce the concepts of with_columns and select, while also highlighting the distinctions between these two methods.

Difference between select and with_columns`

As evident from the examples below, both select and with_column can be utilized for both column selection and column assignment. However, there’s a crucial distinction between the two. After performing column operations, the select method drops the unselected columns and retains only the columns that underwent operations. Conversely, the with_column method appends the new columns to the dataframe, preserving the original ones.

# everything else is dropped

(
    df.select(
        (pl.col("city08") + 10).alias("added"),
        (pl.col("city08") - 10).alias("substracted"),
    )
)

# columns are kept

(
    df.with_columns(
        (pl.col("city08") + 10).alias("added"),
        (pl.col("city08") - 10).alias("substracted"),
    )
)

Now that we’ve clarified this point, let’s proceed to explore the fundamental methods for selecting columns.

Select all columns

A particularly handy tool is pl.all(), which provides the current state of the dataframe—similar to pd.assign(foo=lambda df) in Pandas. This can prove useful, particularly when dealing with operations like groupby and aggregation.

# this is analogous to df.select(pl.col("*")), where * represents the wildcard component
df.select(pl.all())

Select all columns except for…

df.select(pl.col("*").exclude("year", "comb08", "highway08"))

Select a subset of columns

df.select(pl.col("year", "comb08", "highway08"))

Select columns based on regular expression

df.select(pl.col("^.*(mo|ma).*$"))

Select columns based on dtypes

This may remind you of pandas’ select_dtypes method.

df.select(pl.col(pl.Int8, pl.Boolean))

Using selectors

Selectors offer a more intuitive approach to selecting columns from DataFrame or LazyFrame objects, taking into account factors like column names, data types, and other properties. They consolidate and enhance the functionality that’s accessible through the col() expression. More on them here.

By dtypes

df.select(cs.integer(), cs.float(), cs.temporal())

# all columns except for the ones containing float
df.select(cs.all() - cs.numeric())

# same as the one above but it uses the tilde
df.select(~cs.numeric())

By column names

df.select(cs.contains("08"))

df.select(cs.starts_with("d"))

The possibilities here are incredibly vast! I’m pretty sure you’ll find a function that suits your needs just right.

dtype conversions

Understanding and managing data types, as well as converting them, are very important—particularly when dealing with larger datasets. Luckily, with Polars, this process is wonderfully straightforward. Here’s a useful tip: in many cases, the cast method will be your trusty companion for data type conversions. It’s quite versatile and often does the job seamlessly. When it comes to converting dates, things can get a bit more intricate, but rest assured, it’s a hurdle you can easily overcome!

More info on the available dtypes can be found here.

(
    df.select(
        pl.col("cylinders").cast(pl.Int32),
        pl.col("displ").cast(pl.Int64),
        pl.col("mpgData").cast(pl.Int64),
    )
)

Working with str

We can examine the available attributes when working with strings. You might notice that it boasts some familiar attributes, like extract, replace, split, and strip, among others.

# attributes of str
print(dir(pl.col("*").str))

Basic operations

(
    df.select(
        pl.col("make").cast(pl.Utf8).str.contains("Hyundai|Kia").alias("contains"),
        pl.col("make").cast(pl.Utf8).str.starts_with("Alfa").alias("starts_with"),
        pl.col("make").cast(pl.Utf8).str.ends_with("ari").alias("ends_with"),
    )
)

We can extract the numbers from the different car models:

(
    df.select(
        pl.col("model")
        .cast(pl.Utf8)
        .str.extract(r"(\d+)", group_index=1)
        .cast(pl.Int32)
        .alias("extracted_number"),
    )
)

As per usual, we can replace values in a given column:

(
    df.select(
        pl.col("make").cast(pl.Utf8).str.replace("r", 0).alias("replaced"),
    )
)

Changing column names

Altering column names is quite reminiscent of the process in Pandas:

df.rename({"make": "car maker", "model": "car model"})

In case you would like to alter multiple column names all at once:

df.select(pl.all().map_alias(lambda name: name.upper().replace("I", "0000")))

str.split with expand='all'

I often find myself using the str.split method, and it caught me off guard that there isn’t a direct equivalent in Polars. Fingers crossed, we might come across an implementation like expand=True from Pandas, which would be a real game-changer here too!

(
    df.select(
        pl.col("drive")
        .cast(pl.Utf8)
        .str.split_exact("-", n=1)
        .struct.rename_fields(["first_part", "second_part"])
        .alias("fields"),
    ).unnest("fields")
)

Aggregation

Split-apply-combine operations are where Polars truly excels. This is largely due to its foundation in Rust, Polars’ use of lazy evaluation, parallel execution, and efficient memory management also contribute to its speed. As a result, split-apply-combine operations are significantly faster in Polars compared to conventional methods.

A simple split-apply-combine operation

(df.groupby("make").count().sort(by="count", descending=True).limit(5))

Using agg

You can use agg to calculate statistics over either a single or multiple columns all at once.

(df.groupby("make").agg(pl.count(), pl.col("model")))

Here’s an illustration of how to utilize the is_not_nan method. In Polars, unknown values are represented with floating-point precision, similar to how pandas handles them. It’s important not to mix this up with missing values, which are indicated by Null in Polars. Additionally, take note of the use of limit instead of head, a concept quite akin to SQL.

(
    df.groupby("make")
    .agg(
        pl.col("city08").mean().alias("mean_cyl"),
        pl.col("displ").mean().alias("mean_disp"),
    )
    .filter(pl.col("mean_cyl").is_not_nan())
    .sort(by=["mean_cyl", "mean_disp"], descending=[True, True])
    .limit(5)
)

Aggregated columns can be renamed immediately with the alias method.

(
    df.groupby("make", "fuelType")
    .agg(pl.col("comb08").mean().alias("filtered"))
    .filter((pl.col("fuelType") == "CNG") | (pl.col("fuelType") == "Diesel"))
)

Here’s a scenario for performing aggregation based on the year. In this case, the “year” column holds the year-related data, which can be extracted using the dt.year attribute. Keep in mind that Polars doesn’t have native plotting capabilities like pandas. To visualize the data, as a final step, you can convert the dataframe to pandas and make use of its .plot method.

(
    df.groupby(
        [
            pl.col("year").dt.year().alias("year"),
        ]
    )
    .agg([pl.col("comb08").mean(), pl.col("city08").mean()])
    .sort(
        [
            "year",
        ]
    )
    .to_pandas()
    .set_index("year")
    .plot()
)

Here’s another fantastic option for you: the groupby_dynamic feature. It’s really impressive, by the way. You can employ this when grouping based on years or any other time-related information. Additionally, you can make use of the every parameter for resampling, which closely resembles what you can do with pandas.

(
    df.sort(by="year")
    .groupby_dynamic(index_column="year", every="2y")
    .agg([pl.col("comb08").mean(), pl.col("city08").mean()])
    .sort(
        [
            "year",
        ]
    )
    .to_pandas()
    .set_index("year")
    .plot()
)

Pivot

You can also accomplish split-apply-combine tasks seamlessly using the pivot function. It’s remarkably straightforward.

(
    df.with_columns(
        pl.col("fuelType").cast(pl.Utf8)
    )  # conversion to str is needed for the next step
    .filter(pl.col("fuelType").is_in(["Regular", "Premium"]))
    .pivot(values="city08", index="make", columns="fuelType", aggregate_function="mean")
)

Handling missing data

Getting missing value count per column

The null_count function showcases the count of missing values in the dataframe. Remember, these are not the same as np.nans; it’s important to be aware of this distinction. This functionality is akin to the pandas df.isna().sum() operation if you’re familiar with pandas.

df.null_count()

Getting boolean array of missing values

If you ever need to manipulate a series based on its null values, you can also obtain a boolean mask using the is_null function. This can be really handy for targeted data manipulation.

df.select(pl.col("city08").is_null())

Filling missing values

You have a variety of options at your disposal for filling in missing values. Here’s a list of some of the most common ones, although it’s not an exhaustive rundown.

With literals

(df.with_columns(pl.col("cylinders").fill_null(pl.lit(2))))

(df.with_columns(pl.col("cylinders").fill_null(strategy="zero")))

(df.with_columns(pl.col("cylinders").fill_null(strategy="forward")))

With an expression

This can be very useful for machine learning pipelines. When addressing missing values, you can set them to the mode, median, mean, or any other value that suits your needs.

(df.with_columns(pl.col("cylinders").fill_null(pl.col("cylinders").mode())))

Using the over function

The over function in Polars is used to perform window operations, akin to what you’d achieve with window functions in SQL or the transform function in pandas. This function enables you to compute aggregations over a specified window or range of rows defined by a window specification. It’s perfect for tasks like calculating rolling averages, cumulative sums, and other operations that involve working with a specific subset of rows within the dataset.

Let’s begin by filtering the car makes to encompass only the top 5 car brands. Following that, we can utilize the over function to derive several statistics.

top5 = (
    df.select(pl.col("make"))
    .to_series()
    .value_counts()
    .sort(by="counts", descending=True)
    .limit(3)
    .select(pl.col("make"))
    .to_series()
)

(
    df.filter(pl.col("make").is_in(top5)).select(
        "make",
        "model",
        pl.col("city08").mean().over("make").alias("avg_city_mpg_by_make"),
        pl.col("city08").mean().over("model").alias("avg_city_mpg_by_model"),
        pl.col("comb08").mean().over(["make", "model"]).alias("avg_comb_mpg_by_model"),
    )
)

Other common operations

In the upcoming section, you’ll discover a mix of handy operations that I’ve ended up using quite often. These don’t exactly fit into the categories we’ve covered earlier, but they’re still quite useful!

Operating on multiple columns and renaming them

You have the flexibility to conduct column operations and then easily rename them. Additionally, you can make use of the prefix and suffix functions to streamline your workflow.

(df.with_columns((cs.numeric() * 2).prefix("changed_")))

Count unique values

Absolutely, you can also count the unique values within a series in Polars, much like you would with the unique function in pandas. This can be really useful for understanding the distribution and diversity of data within a specific column.

df.select(pl.col("make")).n_unique()

Value_counts to get number of occurrences per item

On a single column

(df.select(pl.col("make")).to_series().value_counts(sort=True))

(df.select(pl.col("make")).groupby("make").count().sort(by="count", descending=True))

On multiple columns

I noticed that Polars lacks the capability to perform value counts on multiple columns, unlike pandas’ value_counts function which operates only on series. However, I’ve discovered that combining a groupby operation with a count function can effectively achieve the same outcome for multiple columns. It’s all about finding alternative approaches that get the job done!

(
    df.select(pl.col("make", "model"))
    .groupby(["make", "model"])
    .count()
    .sort(by="count", descending=True)
)

Conditionals/ if-else statements

If you’re looking to integrate conditional if-else statements into your Polars chain, you can make use of the when, then, and otherwise functions. However, keep in mind that a series of chained when, then statements should be interpreted as if, elif, … elif, rather than if, if, … if. In other words, the first condition that evaluates to True will be selected. It’s crucial to understand this distinction when crafting your conditions.

(
    df.select(
        pl.col("make"),
        pl.when(pl.col("comb08") < 15)
        .then("bad")
        .when(pl.col("comb08") < 30)
        .then("good")
        .otherwise("very good")
        .alias("fuel_economy"),
    )
)

Alright, folks, that’s it for this introduction to Polars. I know it wasn’t exactly short, but I hope it was informative! I’ll be adding more content to this article in the future, so stay tuned. Until then, happy coding 😎💻🔎

Document version