Adam Cseresznye

🏆🥉3rd place at Kaggle's Classification with a Tabular Vector Borne Disease Dataset

Adam Cseresznye — Sun, 04 Jun 2023 00:00:00 GMT

Greetings everyone! I am thrilled to announce that I have achieved a significant milestone in my data science journey. I am delighted to share that I secured the 3rd position in Kaggle's "Classification with a Tabular Vector Borne Disease Dataset" competition, surpassing 931 talented contenders! If you're interested in learning more about my approach to the competition, I invite you to check out my detailed solution on the Kaggle discussion page.

In this exciting competition, the aim was to predict the three most probable diseases out of a total of 11 vector-borne diseases, including well-known names such as Zika, Yellow Fever, Malaria, and more. Our task involved analyzing the provided symptoms and using the MPA@3 evaluation metric for accurate predictions.

My journey in the world of Kaggle has been a relatively short one, with this competition marking my third participation. To give you some context, let me share a bit about my previous placements:

Reaching this point has been a long and rewarding endeavor. Over the past two years, I have been studying this topic in my free time while also pursuing a full-time Ph.D. research. It has been an exciting balancing act!

Participating in Kaggle competitions has been an invaluable learning experience for me. It has allowed me to apply my knowledge, explore new techniques, and connect with fellow data enthusiasts. I am deeply grateful to the organizers of Kaggle for providing a platform that fosters growth and healthy competition.

While this achievement brings a sense of accomplishment, I am already diving into new competitions and expanding my skills. Currently, I am particularly interested in reproducible data and developing robust architectures for machine learning projects to enhance code reproducibility. In fact, I am considering writing a blog on this often overlooked topic.

Stay tuned for more :)

#DataScience #KaggleCompetition #VectorBorneDiseases #MachineLearning #ReproducibleData

Top 10 things I learned from the book Effective Pandas by Matt Harrison

Adam Cseresznye — Wed, 02 Aug 2023 00:00:00 GMT

Effective Pandas by Matt Harrison is a guide to the Pandas library, a powerful Python tool for data manipulation and analysis. The book covers a wide range of topics, from basic data structures to advanced techniques for data cleaning, transformation, and visualization.

I have found Effective Pandas to be a captivating and thought-provoking read. The book offers a genuinely unique take on data wrangling, putting a great emphasis on the utility of chaining methods and utilizing the lambda function. I have found these ideas to be so useful and practical that I have revisited the book multiple times just to make sure I keep them fresh in my memory. I must have read the book from back to back at least 3-4 times.

In this article, I will share the top 10 (+1) things I learned from Effective Pandas. These are the concepts and techniques that I found to be the most useful and practical.

We will use the Real World Smartphone's Dataset by Abhijit Dahatonde from Kaggle. Let's get to it.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
pd.set_option('display.max_columns', None)

Load the dataset

df=pd.read_csv('smartphones.csv')
# some info about the dataframe, such as dimensions and dtypes of columns
df.info()

Tip #1: Use <code>pd.assign</code> more extensively

The <code>pd.assign</code> method in Pandas is a very powerful tool that can be used to create new columns, modify existing columns, or both. It is a very versatile method that can be used in a variety of ways.

One of the most important benefits of using the assign method is that it can be incorporated into method chaining. This means that you can chain multiple assign methods together to create a more concise and readable code. Another benefit of using the assign method is that it completely sidesteps the infamous <code>SettingWithCopyWarning</code>. This warning is often triggered when you try to modify an existing column in a DataFrame. However, the assign method creates a new DataFrame, so there is no need to worry about this warning.

Problem statement: Let's say we would like to capitalize the brand names located in the <code>brand_name column</code> as well as calculate the Pixels Per Inch (PPI). PPI can be calculated following the equation described by the Pixel density page on Wikipedia.

(df
 .assign(brand_name=lambda df: df.brand_name.str.capitalize(), # capitalizes the brand names 
         PPI=lambda df: (np.sqrt(np.square(df.resolution_height) + np.square(df.resolution_width))
                         .div(df.screen_size)
                         .round(1)
                        )
        )
 .loc[:, ['brand_name','model','PPI']]
 .sort_values(by='PPI',ascending=False)
 .head(5)
)

Tip #2: Simplify the management of multiple if-else conditions using <code>np.select</code>

If our goal is to incorporate if-else logic seamlessly into our code, we can effortlessly achieve this using either <code>pd.mask</code> or <code>pd.where</code>. Yet, what approach should we adopt when we need to evaluate multiple conditions instead of just two? In such situations, we have two options: we can either employ successive <code>pd.mask</code> or <code>pd.where</code> calls, or we can take advantage of the <code>np.select</code> function as an alternative solution.

Problem statement: We want to identify the top 3 and top 5 most popular processor brands in smartphones. To do this, we will first create two lists, one for the top 3 brands and one for the top 5 brands. Any processor brand that is not in either of these lists will be categorized as "Other".

# Let's create the two lists that contain the top3 and top5 brand names
top3=df.processor_brand.value_counts().head(3).index
top5=df.processor_brand.value_counts().head(5).index
print(f'Top 3 most popular processors: {top3.tolist()}')
print(f'Top 5 most popular processors: {top5.tolist()}')

'''
Here's an example that employs two successive pd.where calls:
In the first pd.where call, it checks whether the brand is in the top 3; if not, it assigns the label "Top5" to it.
Then, in the second call, it checks if the value is in the top 5; if not, it appends the category "Other".
As you can see, the logic can become intricate and difficult to grasp, especially when dealing with numerous conditions, 
making the code cumbersome and hard to manage.
'''
(df
 .assign(frequency=lambda df: df.processor_brand
         .where(df.processor_brand.isin(top3), other = 'Top5')
         .where(df.processor_brand.isin(top5), other = 'Other')
        )
 .loc[:, 'frequency']
 .value_counts()
)

'''
 Now Let's see np.select!
 It simplifies the process significantly. By providing a list of conditions we want to evaluate and their 
 corresponding values if the condition evaluates to True, we can handle multiple conditions effortlessly. 
 Additionally, we can specify a default value if none of the conditions evaluates to True, making the code 
 much more straightforward and easier to manage. 
'''
(df
 .assign(frequency=lambda df: np.select(condlist=[df.processor_brand.isin(top3), df.processor_brand.isin(top5)],
                                        choicelist=[df.processor_brand,'Top5'],
                                        default='Other'
                                       )
        )
 .loc[:, 'frequency']
 .value_counts()
)

Tip #3 Filter rows or columns with the combination of <code>pd.loc</code> and <code>lambda</code>

Some experienced Pandas users might consider the following concept trivial, but it was an eye-opener for me after reading the book. It turns out that combining <code>pd.loc</code> and <code>lambda</code> (or any custom functions) allows us to filter both rows and columns, depending on our specific needs.

Problem statement: We are interested in identifying phones with a battery capacity greater than 5000mAh.

(df
 .loc[lambda df: df.battery_capacity.gt(5000),['model', 'battery_capacity']] # here we use pd.gt() to select values greater than 5000
 .sort_values(by='battery_capacity')
)

Tip #4 Rename multiple columns effortlessly with <code>rename</code> and <code>replace</code>

OK. This is a big one for me. I used this one multiple times already. The title pretty much says it all. Let's say our column names contain spaces, which makes column selection by attribute access pretty much impossible. Now, what do we do? Well...Let's see.

Problem statement: We would like to remove all the underscores from our column names.

df.columns # original column names for reference

# column names after replacing underscores
(df
 .rename(columns = lambda x: x.replace('_', ''))
 .columns
)

Tip #5: Use <code>pd.clip</code> to easily remove outliers

# First, we'll identify the phone brands with the most number of handsets present in our dataset."
top10_brand_names = (df
                     .brand_name
                     .value_counts()
                     .head(10)
                     .index
                     .tolist()
                    )
print(top10_brand_names)

# Then we will sort them based on median price
top10_brand_names_ordered = (df
                             .loc[lambda x: x.brand_name.isin(top10_brand_names),['brand_name', 'price']]
                             .groupby('brand_name')
                             .median()
                             .sort_values(by='price')
                             .index
                             .to_list()
                            )
print(top10_brand_names_ordered)

fig, axs = plt.subplots(1,2, figsize=(10,5))

# For reference, this is what our box plot looks if we leave in the outlier values
(df
 .loc[lambda x: x.brand_name.isin(top10_brand_names),['brand_name', 'price']] # filter rows based on top10_brand_names and select columns
 .pivot(columns='brand_name',values='price') # pivot to get the brand names on the x axis later on
 .loc[:, top10_brand_names_ordered] # order the columns based on median price
 .plot
 .box(title='Top 10 most popular smartphone brands \n -outlier values included-', 
      rot=90,
      ax=axs[0]
     )
)

(df
 .loc[lambda x: x['brand_name'].isin(top10_brand_names), ['brand_name', 'price']]
 .pivot(columns='brand_name', values='price')
 .loc[:, top10_brand_names_ordered]
 .pipe(lambda df: df.assign(**{col : df[col].clip(lower=df[col].quantile(0.05), # this is called dictionary unpacking
                                                  upper=df[col].quantile(0.95))
                              for col in df.columns}))
 .plot
 .box(title='Top 10 most popular smartphone brands \n -only values between 5th and 95th percentiles included-',
      rot=90,
      ax=axs[1]
     )
)
axs[0].set(ylabel='Price in local currency')
plt.tight_layout()

Tip #6: Find corrupted entries with <code>str.extract</code>

How often have you encountered the situation where, for some reason, a column that is expected to only contain numerical values displays <code>object</code> as its <code>dtype</code>? This often indicates the presence of some string values mixed within the column. It would be beneficial to promptly identify all the erroneous values, correct them, and proceed with our analysis smoothly. Let's explore what we can do in such scenarios.

Problem statement: We would like to identify any cells in a specific column that contain non-numerical values.

df_bad_values = df.copy(deep=True) # let's prepare a copy of the original dataframe 

# let's modify some of the values in the price column randomly:
df_bad_values.loc[np.random.randint(0, high=df_bad_values.shape[0], size=10), 'price'] = '.'
df_bad_values.loc[np.random.randint(0, high=df_bad_values.shape[0], size=10), 'price'] = '-'
df_bad_values.loc[np.random.randint(0, high=df_bad_values.shape[0], size=10), 'price'] = '*'
df_bad_values.info() # the modified dataframe's price column now returns object as dtype

# let's find the corrupted values easily:
(df_bad_values
 .price
 .str.extract(r'([^a-zA-Z])') # returns NON-matching alphabetical characters
 .value_counts()
)

Tip #7: Sort values based on the <code>key</code> parameter

The <code>pd.sort_values</code> function surprised me with its versatility. Previously, I had only used its <code>by</code>, <code>axis</code>, and <code>ascending</code> parameters for sorting. However, Matt's book introduced me to its <code>key</code> parameter, which allows us to apply any function to sort the values. The only constraint is that the function must return a <code>Series</code>.

Problem statement: For whatever reason, we would like to sort the phone model names in ascending order based on their second letter.

(df
 .iloc[:, 1:3]
 .sort_values(by='model',
              key = lambda x: x.str[1],
              ascending = True
             )
 .head(10)
)

Tip #8: Reference an existing variable inside <code>pd.query</code> with @

This resembles Tip #4, as it's a technique I frequently use. Since reading Matt's book, I have started using <code>pd.query</code> extensively to filter rows based on values, instead of relying on <code>.loc</code> or <code>.iloc</code>. In case you choose to adopt <code>pd.query</code> as well, it's essential to be aware of its capability to use "@" to reference variables in the environment. This feature enhances its flexibility and makes it even more convenient to apply in various data filtering scenarios.

Problem statement: Our objective is to identify phones that meet three specific criteria: being priced below the average market price, having more processor cores than the average, and possessing a battery capacity greater than the average.

average_price=df.price.mean()
average_cores=df.num_cores.mean()
average_battery=df.battery_capacity.mean()

(df
 .query("(price <= @average_price) and (num_cores >= @average_cores) and (battery_capacity >= @average_battery)")
 .iloc[:,:3]
 .sort_values(by='price')
 .head()
)

Tip #9: Gain more insights by using <code>style.background_gradient</code>

Being a visual creature, I often struggle to comprehend data quickly just by examining the raw table alone. Fortunately, with the help of <code>style.background_gradient</code>, similar to the <code>heatmap</code> function in the <code>seaborn</code> library, we can represent cells, in terms of color gradient, based on their values. This enables us to identify trends and patterns in our data swiftly, making data analysis more intuitive and insightful.

Problem statement: Our goal is to identify the overall trends related to key descriptors found among the Top 10 smartphone brands, aiming to determine which brand offers the most value for our money.

(df
 .query("brand_name.isin(@top10_brand_names_ordered)") # filter rows based on top10 brands
 .groupby(['brand_name',])
 [['avg_rating', 'processor_speed', 'ram_capacity', 
   'screen_size', 'battery_capacity', 'price']]
 .mean()
 .sort_values(by='avg_rating')
 .transpose()
 .rename(columns=str.capitalize) # capitalize brand names
 .style
 .set_caption("Key descriptors of the Top 10 smartphone brands")
 .format(precision=1)
 .background_gradient(cmap = 'vlag', axis = 1)
 .set_table_styles([
    {'selector': 'td', 'props': 'text-align: center;'},
     {'selector': 'caption','props': 'font-size:1.5em; font-weight:bold;'}
     ,]
 )
)

Tip #10: Use <code>pd.pipe</code> to include any functions in our chain

One of the most valuable lessons I learned from Effective Pandas is the importance of arranging my code in a chain. Although it may feel somewhat restrictive at first, once you overcome the initial hurdles, you'll realize that your code becomes more readable and easier to understand. The need to invent unique names for temporary variables is completely eliminated, making coding a much happier experience.

In the chaining world, you often find yourself wanting to use various functions that are not explicitly designed for chaining. However, there's good news! You can still achieve this. The <code>pd.pipe</code> function comes to the rescue, allowing you to use any function as long as it returns a Series or DataFrame. It's a flexible solution that empowers you to seamlessly integrate different functions into your chaining workflow, making your data manipulation more efficient and enjoyable.

Problem statement: We aim to visualize the impact of RAM capacity on user satisfaction. To achieve this, we will utilize the <code>sns.lmplot</code> function, which plots the data and corresponding regression models for the Top 5 phone brands.

top5_brand_names_ordered = df.brand_name.value_counts().head().index

with sns.axes_style("darkgrid"):
    g = (df
         .query("brand_name.isin(@top5_brand_names_ordered)") # filter rows based on top10 brands
         [['brand_name', 'avg_rating', 'ram_capacity']]
         .assign(brand_name=lambda df: df.brand_name.str.capitalize())
         .rename(columns={'brand_name':'Brand name'})
         .pipe(lambda df: sns.lmplot(data=df,
                                     x='ram_capacity',
                                     y='avg_rating',
                                     hue='Brand name',
                                     # height=4,
                                     # aspect=1.2
                                    )
              )
        )

    g.set(title='Customer satisfaction correlates with RAM capacity', 
          xlabel='RAM capacity',
          ylabel='User rating'
         )
plt.tight_layout()

Tip #10 + 1: Use the "margin" parameter of <code>pd.crosstab</code> to easily calculate row/column subtotals

Despite primarily using the pandas <code>groupby</code> function for data aggregation, the <code>pd.crosstab</code> function has an enticing feature: the <code>margin</code> parameter. This option enables us to effortlessly calculate subtotals across rows and columns. Moreover, by normalizing our data, we can gain even more intuition about the questions we want to answer.

Problem statement: Our objective is to evaluate how RAM capacity impacts user satisfaction across the Top 5 brands. Additionally, we will normalize our data to compare values comprehensively across the entire dataset.

(df
 .query("brand_name.isin(@top5_brand_names_ordered)") # filter rows based on top10 brands
 .assign(brand_name=lambda df: df.brand_name.str.capitalize())
 .pipe(lambda df: pd.crosstab(index=df['ram_capacity'],
                              columns=df['brand_name'],
                              values=df['avg_rating'],
                              aggfunc='mean',
                              margins=True,
                              normalize='all'
                             )
      )
 .mul(100)
 .round(1)
)

I hope this article has convinced you to pick up Matt Harrison's Effective Pandas! There are plenty more exciting ideas in the book beyond the Top 10 I've shared here (I didn't even get into the fascinating time series part!). I hope you found these insights helpful and inspiring.

Happy coding 🐼💻🚀

How to use the Lets-Plot library by JetBrains

Adam Cseresznye — Sat, 05 Aug 2023 00:00:00 GMT

When I embarked on my data science journey, due to my academics background I quickly gravitated towards the R programming language. Like many R novices, I began with Hadley Wickham's R for Data Science book which introduced me to the wonderful ggplot2 library. As my interest in machine learning grew, I made the switch to Python. Nowadays, for most of my data plotting needs, I rely mainly on matplotlib or seaborn. Though I love these libraries, their multiple ways of accomplishing the same tasks can be a bit cumbersome and challenging to learn at first.

That's why in this article, I'm excited to introduce you to the Lets-Plot library by JetBrains. It is the closest you can get to ggplot's syntax while using Python. While some traditional Python users might find the syntax a bit unfamiliar initially, I'm here to make a case for this fantastic library and hopefully inspire you to incorporate it into your day-to-day data science activities.

To showcase (some of) the key features of Lets-Plot, we will be utilizing the penguins dataset 🐧 from Github.

Without further ado, let's dive right in and discover the power and beauty of Lets-Plot! 🐍📊

# Installation
# pip install lets-plot

# Import libraries
import numpy as np
import pandas as pd
from lets_plot import *
from lets_plot.mapping import as_discrete
LetsPlot.setup_html()

address='https://gist.githubusercontent.com/slopp/ce3b90b9168f2f921784de84fa445651/raw/4ecf3041f0ed4913e7c230758733948bc561f434/penguins.csv'
df=pd.read_csv(address)
df.head()

Syntax Similarities: <code>Lets-Plot</code> and <code>ggplot2</code>

For our first exercise, I thought it would be beneficial to replicate a basic plot inspired by Hadley's book. When comparing my code here with the one presented by him, you'll notice that there is very little difference between the two. The syntax is nearly identical, making it a smooth transition from ggplot to Lets-Plot.

Now, let's take a closer look at the code. In the <code>ggplot</code> function, we define the <code>DataFrame</code> we'll be working with, and the aesthetic mappings are set at the global level. We assign the values for the x and y axes, as well as the <code>color</code> argument, which groups the data based on the categorical variable representing the three different penguin species: Adelie, Gentoo, and Chinstrap. This color parameter is quite similar to seaborn's <code>hue</code>, making it easy for those familiar with seaborn to adapt to Lets-Plot seamlessly.

After the <code>ggplot()</code> function sets the global aesthetic mappings, the <code>geom_point()</code> function comes into play and draws the points defined by the x and y parameters, effectively creating a scatter plot. These points represent the data points from the penguins dataset, with x and y coordinates corresponding to the specified variables.

Additionally, we enhance the plot by using <code>geom_smooth(method='lm')</code>, which adds a smoothed conditional mean. The <code>lm</code> method stands for 'linear model,' indicating that the smoothing is based on a linear regression. This smoothed line helps reveal trends and patterns in the data, making it easier to observe any overall relationships between the variables.

Let's continue exploring more of what Lets-Plot has in store for us! 📊🐧🌈

(ggplot(df,
        aes(x='flipper_length_mm',
            y = 'body_mass_g',
            color='species'
           )
       )
 + geom_point() # Draw points defined by an x and y coordinate, as for a scatter plot.
 + geom_smooth(method='lm') # Add a smoothed conditional mean. ‘lm’ stands for 'linear model' as Smoothing method
)

In the previous example, we highlighted the importance of placing the <code>color</code> parameter at the global level, which grouped the data by the three penguin species and showed separate regression lines for each group. However, if we prefer to depict the regression line for the entire dataset, regardless of the group association, we can do so just as easily. All we need to do is remove the <code>color</code> parameter from the aesthetics of the <code>ggplot</code> function and place it solely in the <code>geom_point</code>.

Additionally, to enhance the plot further, we can properly label the x and y axes, add a title and subtitle. With these simple adjustments, we can achieve the same output as Hadley's original code, with little to no modification.

(ggplot(df, 
        aes(x='flipper_length_mm',
            y = 'body_mass_g',
           )
       )
 + geom_point(aes(color='species', shape='species'))
 + geom_smooth(method='lm')
 + labs(
    title = "Body mass and flipper length",
    subtitle = "Dimensions for Adelie, Chinstrap, and Gentoo Penguins",
    x = "Flipper length (mm)", y = "Body mass (g)",
  ) 
 + scale_color_viridis() # lets-plots equivalent of the scale_color_colorblind()
)

Visualizing Data Based on Categorical Variables

Lets-Plot provides numerous options to showcase our data using categorical variables. From bar plots, box plots, and violin plots to pie charts, the possibilities are diverse. You can check out their API for reference. Let's explore some examples to demonstrate the versatility of Lets-Plot in visualizing categorical data.

 
penguin_bar = (ggplot(df,aes(x='species'))
               + geom_bar()
              )

penguin_box = (ggplot(df,aes(x = 'species', y = 'body_mass_g'))
               + geom_boxplot()
              )

penguin_density = (ggplot(df,aes('body_mass_g', color='species', fill='species'))
                   + geom_density(alpha=0.5)
                  )

penguin_rel_frequency = (ggplot(df,aes(x = 'island', fill = 'species'))
                         + geom_bar(position='fill')
                        )
gggrid([penguin_bar, penguin_box, penguin_density, penguin_rel_frequency], ncol=2)

Incorporate Multiple Variables with facet_wrap

So far we've discovered how easy it is to plot data based on a single categorical variable. However, what if we want to depict relationships involving two or more categorical variables? That's where <code>facet_wrap</code> comes in handy. This versatile function bears resemblance to similar functions found in <code>seaborn</code> or <code>ggplot2</code> libraries.

To unlock the potential of <code>facet_wrap</code>, we simply need to define aesthetics, which can either be global or local to the mapping function. Then, we can use <code>facet_wrap</code> with the relevant categorical variable we want to visualize. It's as simple as that!

(ggplot(df, aes(x = 'flipper_length_mm', y = 'body_mass_g'))  
 + geom_point(aes(color = 'species', shape = 'species'), size = 4) 
 + facet_wrap('island', nrow=1)
 + labs(title = "Body mass and flipper length based on island",
        subtitle = "Dimensions for Adelie, Chinstrap, and Gentoo Penguins",
        x = "Flipper length (mm)", y = "Body mass (g)",
       )
 + theme(plot_title=element_text(size=20,face='bold'))
 + ggsize(1000,500)
)

Reordering Categorical Variables Based On Statistics

When visualizing data, a task I frequently encounter is ordering categorical variables in either ascending or descending order, based on statistics like median or mean. In my previous point on "Visualizing Data Based on Categorical Variables," you noticed that the boxplot displayed categories in an unordered manner. However, consider how we can present them in an ascending order, determined by the median. This not only enhances the aesthetics of the plot but also provides valuable insights into the relationships among the categories.

(ggplot(df,aes(as_discrete('species', order=1, order_by='..middle..'), 
               y = 'body_mass_g'))
 + geom_boxplot()
)

By incorporating the <code>as_discrete</code> function, specifying the column, the ordering direction (1 for ascending, -1 for descending), and setting the <code>order_by</code> variable to middle (representing the median), the plot has become significantly more informative. This simple addition has allowed us to organize the categorical variables in a meaningful manner, improving the visualization's clarity and aiding in the interpretation of relationships among the categories.

Chaining Pandas Code with Lets Plot Visualization

One of the best features of the <code>pandas</code> library is its remarkable customizability. With the help of the <code>pd.pipe</code> function, we can seamlessly integrate any of our own functions into method chains, as long as they return a <code>DataFrame</code> or <code>Series</code>. This opens up exciting possibilities to fully incorporate <code>Lets-Plot</code> into our code, just like <code>pandas</code>' own built-in plotting functionality.

While <code>Lets-Plot</code> may be slightly more verbose than <code>pandas</code> plotting, it offers significantly more flexibility and freedom for customization. Not to mention that some may consider it visually more appealing. With <code>Lets-Plot</code> integrated into our <code>pandas</code> code, we can effortlessly create stunning and informative plots, making data analysis an even more enjoyable experience.

(df
 .groupby('species')
 [['body_mass_g', 'flipper_length_mm']]
 .mean()
 .reset_index()
 .pipe(lambda df: (ggplot(df)
                   + geom_pie(aes(slice='body_mass_g', fill='species'), 
                              stat='identity',size=30, hole=0.2, stroke=1.0,
                              labels=layer_labels().line('@body_mass_g').format('@body_mass_g', '{.0f}').size(20)
                             )
                   + labs(title = "Body mass based on species",
                          subtitle = "Representing how Lets-Plot can be used with pd. pipe",
                          x = "", y = "",
                         )
                   + theme(axis='blank',
                          plot_title=element_text(size=20,face='bold'))
                   + ggsize(500,400)
                  )
 )
)

That's a wrap on the Lets-Plot library! There's so much more to explore and learn about this powerful tool. I hope you found this introduction helpful and consider integrating Lets-Plot into your daily data analysis routine.

Happy coding 🐍🖥️🔍🚀

Tidy Tuesday: Hot Ones Episodes

Adam Cseresznye — Wed, 09 Aug 2023 00:00:00 GMT

I've decided to get more involved in the Tidy Tuesday movement. I think it's a really enjoyable way to improve my skills in working with data, like analyzing, organizing, and visualizing it. The cool datasets they provide make it even more interesting. More information on Tidy Tuesday and their past datasets can be found here.

This week we have a dataset related to the show Hot Ones. Hot Ones is a unique web series that combines spicy food challenges with celebrity interviews. Hosted by Sean Evans, guests tackle increasingly hot chicken wings while answering questions, leading to candid and entertaining conversations. The show's blend of heat and honesty has turned it into a global sensation, offering a fresh take on interviews and captivating audiences worldwide.

Let's see what we can learn from the data 🔬🕵️‍♂️.

Import data

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt 
from scipy import stats
from lets_plot import *
from lets_plot.mapping import as_discrete
LetsPlot.setup_html()

We have three dataframes: sauces, season and episodes. For more information about the data dictionary see the GitHub repo.

sauces=pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-08/sauces.csv')
season=pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-08/seasons.csv')
episodes=pd.read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-08/episodes.csv')

sauces.sample(3)

season.sample(3)

episodes.sample(3)

:::note After taking a look at all three datasets, I think I will continue working with the sauces and episodes ones. :::

Questions

My main questions are:

What differences can be observed in the spiciness of sauces as we look across the various seasons?
Has every individual successfully completed all the episodes?
What is the average completion rate per season?
Is there a correlation between Scoville Units and completion rate?
Are there any returning guests?

Data Analysis

What differences can be observed in the spiciness of sauces as we look across the various seasons?


(sauces
 .groupby('season')
 .agg(AVG_scoville=('scoville', 'mean'),
      median_scoville=('scoville', 'median'),
     )
 .reset_index()
 .melt(id_vars='season',
       var_name='statistics',
      )
 .pipe(lambda df: (ggplot(df, aes('season', 'value',fill='statistics'))
                   + geom_bar(stat='identity', show_legend= False)
                   + facet_wrap('statistics',nrow=1,scales='free_y')
                   + labs(x='Seasons',
                          y='Scoville Units'
                         )
                   + theme(plot_title=element_text(size=20,face='bold'))
                  + ggsize(1000,600)
                  )
      )
 
)

There seems to be a shift during the season 3-5 period as can be seen in @fig-fig1. Both indicators – mean and median Scoville Units – show a consistent upward trend over this time frame and later on they stabilize.

What about the overall spread of the data? 🤔


(sauces
 .loc[:, ['season', 'scoville']]
 .pipe(lambda df: (ggplot(df, aes('season', 'scoville'))
                   + geom_boxplot()
                   + scale_y_log10()
                   + labs(x='Seasons',
                          y='log(Scoville Units)'
                         )
                   + theme(plot_title=element_text(size=20,face='bold'))
                   + ggsize(1000,600)
                  )
      )
)

Here are some observations: Season 5 exhibits the widest range, featuring sauces with Scoville Units spanning from 450 to 2,000,000. In addition, starting from season 6 onwards, the averages, medians, and ranges of Scoville Units appear to even out.

Has every individual successfully completed all the episodes?

To answer this question we will use the episodes dataframe. Keep in mind there are 300 episodes in this dataframe. The finished column can be useful here. Just by looking for entries where finished == False we will have our answer.

(episodes
 .query("finished==False")
 [['season', 'guest', 'guest_appearance_number']]
)

Taking a closer look, it seems that around 15 participants didn't make it through the entire Hot Ones interview challenge.Not bad out of 300 shows. And guess what? Eric André popped up on the show not just once, but twice! Now, the big question: did he conquer the hot seat in at least one of those interviews? Let's plot the data to make it more visual...


(episodes
 .query("finished==False")
 .groupby('season')
 .finished
 .count()
 .to_frame()
 .reset_index()
 .pipe(lambda df: (ggplot(df, aes('season', 'finished'))
                   + geom_bar(stat='identity')
                   + labs(x='Seasons',
                          y='Number of incomplete interviews'
                         )
                   + theme(plot_title=element_text(size=20,face='bold'))
                   + ggsize(600,400)
                  )
      )

)

Interestingly, a majority of these incomplete interviews belong to season 2 (@fig-fig3). This fact is quite surprising, especially when you consider that the maximum Scoville value for that season was only 550,000 – nearly a quarter of the following year's value, where only one person faced difficulty finishing the challenge.

What is the completion rate per season?

To get to the bottom of this question, let's start by figuring out how many episodes were in each season. We can grab this info from the season dataset. Just a heads-up, in season 9 they seem to threw in an extra episode. So, keep this in mind! Otherwise, you might end up with percentages that go beyond 100%.

# First we need to find out the total number of episodes per season

episodes_per_season = (season
                       [['season', 'episodes', 'note']]
                       .set_index('season')
                       # we need to extract the one extra episode in season 9
                       .assign(note=lambda df: df.note
                               .str.extract(r'([0-9.]+)')
                               .astype(float),
                        # add the two column together
                               episodes=lambda df: df.episodes
                               .add(df.note, fill_value=0)
                              )
                       .drop(columns='note')
                       .squeeze()
                      )


completion_rate = (episodes
                   .query("finished==True")
                   .groupby('season')
                   .finished
                   .sum()
                   .div(episodes_per_season)
                   .mul(100)
                   .to_frame().reset_index()
                   .rename(columns={0:'completion_rate'})
                  )
                   
(completion_rate                  
 .pipe(lambda df: (ggplot(df, aes('season', 'completion_rate'))
                   + geom_line(stat='identity')
                   + labs(x='Seasons',
                          y='% successful participants'
                         )
                   + theme(plot_title=element_text(size=20,face='bold'))
                   + ggsize(600,400)
                  )
      )
)

Taking a peek at @fig-fig4, it seems like the normalized completion rate hits its lowest point in season 1, closely followed by season 7. However, even in those seasons, the rate remains surprisingly high.

Is there a correlation between Scoville Units and completion rate?

Here's a curious thought: could there be a link between Scoville Units and the completion rate? I'm just wondering if the spiciness level affects how well participants handle the challenge. Exploring this connection might add a spicy twist to the Hot Ones experience – let's see where the data takes us!

# AVG_scoville code comes from a code snippet 
# 'What differences can be observed in the spiciness 
# of sauces as we look across the various seasons?'

AVG_scoville = (sauces
                .groupby('season')
                .agg(AVG_scoville=('scoville', 'mean'))
                .squeeze()
               )

# Let's calculate the Pearson correlation coefficient. We have to discard the last value
# as the completion rate is not defined for that

stats.pearsonr(AVG_scoville.values[:-1],completion_rate.completion_rate.values[:-1])

The Pearson correlation coefficient has its say: there's actually a moderate positive correlation(0.5, p<0.05) between Scoville Units and completion rate. Quite intriguing, isn't it? Honestly, I was expecting the opposite outcome myself! It seems like the higher the average spiciness, the more determined the guests become. Take a look at @fig-fig5.


(pd.concat([AVG_scoville,completion_rate],axis=1)
 .rename(columns={0:'success_rate'})
 .pipe(lambda df: (ggplot(df, aes('AVG_scoville', 'completion_rate'))
                   + geom_point(size=5, alpha=0.5)
                   + geom_smooth()
                   + labs(x='Average Scoville units',
                          y='% successful participants'
                         )
                   + theme(plot_title=element_text(size=20,face='bold'))
                   + ggsize(600,400)
                  )
      )
)

Are there any returning guests?

Has there been a brave soul who dared to make a return to the show for a second time? The column guest_appearance_number holds the answers you're looking for.

(episodes
 .query("guest_appearance_number > 1")
 [['guest','season', 'episode_season', 'finished']]
)

It looks like a total of 12 individuals have taken on the challenge not once, but twice. Hats off to their courage! 🎩

Final words

And there you have it, folks – that's a wrap! I hope you enjoyed exploring the Hot Ones dataset with me. Rest assured, more of these analyses are in the pipeline for the future. Stay tuned for what's to come!

🥳🎉 Two IBM certificates and some geospatial data

Adam Cseresznye — Thu, 10 Aug 2023 00:00:00 GMT

I'm happy to share that I've recently completed both IBM's Data Analyst and Data Science Professional Certificates within the past month. The course content was well-structured, and I learned a great deal from these programs. For instance, I've always been interested in learning SQL, and this was the perfect chance to start exploring it.

If you're curious about these certificates, you can find more information through the links provided below. But my learning journey doesn't stop here—I'm planning to tackle most of the courses listed in the Data Science learning path on Coursera, so there's more to come.

While I'm at it, I wanted to introduce you to a neat library called Folium, which is fantastic for working with geospatial data. I came across Folium during the capstone project of the Data Science Specialization, where we had a fun task of predicting and visualizing the success of SpaceX rocket launches.

In this post, I'll briefly share what I've learned about this library. I hope you'll find it useful too. Let's dive in!

import folium
import pandas as pd
import os
from folium import plugins

We'll be utilizing the dataset made available by https://open.toronto.ca/. This dataset includes the locations of bicycles installed on sidewalks and boulevards across the City of Toronto, wherever there's a requirement for public bicycle parking facilities. By the way, I discovered this dataset through the Awesome Public Datasets repository on GitHub. If you haven't already, I recommend checking them out.

# Let's read in the file

for file in os.listdir():
    if file.endswith(".csv"):
        toronto_df = pd.read_csv(file)

        print(f"{file} read in as pandas dataframe")

Considering the original dataset has over 17,300 entries, we'll keep things light by working with just 500 rows for now. It's all for the sake of a demonstration, after all!


toronto_df = toronto_df.sample(n=500)
toronto_df.head()

The geometry column holds the longitude and latitude information, but before we dive in, we need to extract the valuable details. No worries – we'll make use of pandas' str.extract for this task.

pattern = r"(-?\d+\.\d+),\s*(-?\d+\.\d+)"

toronto_df_processed = toronto_df.assign(
    LONGITUDE=lambda df: df.geometry.str.extract(pattern)[0],
    LATITUDE=lambda df: df.geometry.str.extract(pattern)[1],
).loc[:, ["ASSETTYPE", "STATUS", "LONGITUDE", "LATITUDE"]]
toronto_df_processed.head()

Creating the map and displaying it

Here's an example of how to create a map without any overlaid data points.


toronto_map = folium.Map(
    location=[43.651070, -79.347015], zoom_start=11, tiles="OpenStreetMap"
)
toronto_map

Superimposing bike locations on the map with `FeatureGroup`

After instantiating FeatureGroup, we can easily add the bike locations using the add_child method. It is really easy!


# let's start with a clean copy of the map of Toronto
toronto_map = folium.Map(
    location=[43.651070, -79.347015], zoom_start=11, tiles="OpenStreetMap"
)

# instantiate a feature group 
bike_stations = folium.map.FeatureGroup()

# loop through the bike stations
for lat, long in zip(toronto_df_processed.LATITUDE, toronto_df_processed.LONGITUDE):
    bike_stations.add_child(
        folium.features.CircleMarker(
            [lat, long],
            radius=5,
            color="red",
            fill=True,
            fill_color="yellow",
            fill_opacity=1,
        )
    )
# add bike stations to the map
toronto_map.add_child(bike_stations)

Adding pop-up text with relevant information

We can also enhance this by adding a pop-up box that displays custom text of our choice.


# let's start with a clean copy of the map of Toronto
toronto_map = folium.Map(
    location=[43.651070, -79.347015], zoom_start=11, tiles="OpenStreetMap"
)

# instantiate a feature group 
bike_stations = folium.map.FeatureGroup()

# loop through the bike stations
for lat, long in zip(toronto_df_processed.LATITUDE, toronto_df_processed.LONGITUDE):
    bike_stations.add_child(
        folium.features.CircleMarker(
            [lat, long],
            radius=5,
            color="grey",
            fill=True,
            fill_color="white",
            fill_opacity=1,
        )
    )

# add pop-up text to each marker on the map
latitudes = list(toronto_df_processed.LATITUDE)
longitudes = list(toronto_df_processed.LONGITUDE)
labels = list(toronto_df_processed.STATUS)

for lat, lng, label in zip(latitudes, longitudes, labels):
    folium.Marker([lat, lng], popup=label).add_to(toronto_map)

# add bike stations to map
toronto_map.add_child(bike_stations)

Clustering the rental locations with `MarkerCluster`

And the best part, which happens to be my favorite, is that we can also integrate a MarkerCluster. This comes in handy when we're dealing with numerous data points clustered closely together on the map. With a MarkerCluster, you get to see their combined values instead of each one individually. It's a fantastic feature!


# let's start with a clean copy of the map of Toronto
toronto_map = folium.Map(
    location=[43.651070, -79.347015], zoom_start=11, tiles="OpenStreetMap"
)

# instantiate a mark cluster object 
bike_stations_cluster = plugins.MarkerCluster().add_to(toronto_map)

# loop through the dataframe and add each data point to the mark cluster
for lat, lng, label, in zip(
    toronto_df_processed.LATITUDE,
    toronto_df_processed.LONGITUDE,
    toronto_df_processed.STATUS,
):
    folium.Marker(
        location=[lat, lng],
        icon=None,
        popup=label,
    ).add_to(bike_stations_cluster)

# display map
toronto_map

That's a wrap! I hope these examples have been helpful. Feel free to use these techniques in your next data science or geospatial project. Until next time, happy exploring!

Getting Started with Polars

Adam Cseresznye — Fri, 11 Aug 2023 00:00:00 GMT

In the ever-evolving field of data science, effective data manipulation tools are essential. Enter Polars, a Rust-based library garnering attention within the data community. Boasting impressive speed and versatile capabilities, Polars is redefining our data management practices. In this blog, we delve into Polars' core functions and practical applications, shedding light on how it empowers data professionals to efficiently tackle complex tasks.

For those well-versed in Pandas, Polars offers a blend of familiarity and innovation. Although this document is not designed to substitute the official documentation, it serves as a tool to provide you with insights into the capabilities that polars offers. Our goal is to ensure the continuous updating of this document.

Our exploration of Polars is guided by insights from the Polars User Guide, Kevin Heavey's perspectives in Modern Polars and Matt Harrison's engaging tutorial on Polars at PyCon. We kickstart our exploration with Matt Harrison's shared dataset, the US Department of Energy's Automobile Fuel Economy data. Let's begin this journey!

Setup

import pandas as pd
import polars as pl
import polars.selectors as cs

import seaborn as sns
import matplotlib.pyplot as plt

pl.Config.set_tbl_rows(2)  # limit the numbers of rows printed

Polars Version

We are using Polars 0.18.8 for this demonstration.

pl.__version__

Given the large size of the dataset and the fact that some columns aren't really useful for us, we'll start by picking out the information we actually need. Feel free to use the code below to work with your dataframe.

df = pl.read_csv("vehicles.csv", null_values="NA")

# process and save parquet for later use
(
    df.select(
        pl.col(
            "make",
            "model",
            "drive",
            "cylinders",
            "displ",
            "fuelType",
            "mpgData",
            "city08",
            "highway08",
            "comb08",
            "year",
        )
    )
    .with_columns(
        pl.col("make").cast(pl.Categorical, strict=False),
        pl.col("model").cast(pl.Categorical, strict=False),
        pl.col("drive").cast(pl.Categorical, strict=False),
        pl.col("fuelType").cast(pl.Categorical, strict=False),
        pl.col("cylinders").cast(pl.Int8, strict=False),
        pl.col("displ").cast(pl.Float32, strict=False),
        pl.col("mpgData").map_dict({"N": False, "Y": True}),
        pl.col("city08").cast(pl.Int8, strict=False),
        pl.col("highway08").cast(pl.Int8, strict=False),
        pl.col("comb08").cast(pl.Int8, strict=False),
        pl.col("year").cast(pl.Utf8).str.to_datetime("%Y").cast(pl.Date),
    )
    .write_parquet("vehicles.parquet")
)
df = pl.read_parquet("vehicles.parquet")

Inspect the polars namespace

Taking a closer look at the Polars namespace can be really helpful as you start to get to know the library. By exploring the functions, methods, and attributes available within the namespace, you'll get a better sense of what the library can do. This exploration sets the groundwork for you to use the library more effectively and make the most of its features.

By issuing print(dir(pl)), we can explore the accessible functions within the namespace.

# available functions

print(dir(pl))

The same applies to the methods available for DataFrames.

# methods defined on dataframes

print(dir(df))

...and expressions for the Series

# available expressions

print(dir(pl.col("foo")))

Access documentation

If you're familiar with Python, you might recognize that you can access documentation by either using __doc__ or by calling the help() function.

print(df["year"].dt.convert_time_zone.__doc__)

help(pl.head)

Indexing

If you've spent some time working with Pandas, you might find it interesting that Polars doesn't rely on the concept of an index like Pandas does. But no need to fret—there are several other ways we can still access the rows in Polars. One of the straightforward methods—although I won't delve too deeply into it here—is using the head, tail, and sample methods.

Use `take` for Series or Expressions

An interesting approach is through Polars' take method. This method can be used with Expr, Series, or np.ndarray. However, it's worth noting that if you intend to utilize it with DataFrames, a preliminary conversion to Series is necessary.


df.select(pl.col("make")).to_series().take(list(range(0, 5)))

Using `filter`

Similar to Pandas, Polars features a filter method that assists in the process of selectively filtering rows based on one or multiple conditions.

Evaluate a single condition

df.filter(pl.col("model") == "Testarossa")

Combine multiple conditions

df.filter((pl.col("model") == "Testarossa") & (pl.col("make") == "Ferrari"))

Imitate pandas' `index` using `with_row_count`

Even in the absence of a traditional index, you can make use of the with_row_count method in Polars. This clever method comes to the rescue for tasks like indexing and filtering, providing an alternative approach.

(df.with_row_count().filter(pl.col("row_nr") == 5))

Use `is_in`

The is_in function can be employed to verify whether elements of a given expression are present within another Series. We can use this to filter our rows.

df.filter(pl.col("cylinders").is_in([i for i in range(6, 8)]))

Select columns

Before we dive into the realm of selecting columns, it's important to introduce the concepts of with_columns and select, while also highlighting the distinctions between these two methods.

Difference between `select` and with_columns`

As evident from the examples below, both select and with_column can be utilized for both column selection and column assignment. However, there's a crucial distinction between the two. After performing column operations, the select method drops the unselected columns and retains only the columns that underwent operations. Conversely, the with_column method appends the new columns to the dataframe, preserving the original ones.


# everything else is dropped

(
    df.select(
        (pl.col("city08") + 10).alias("added"),
        (pl.col("city08") - 10).alias("substracted"),
    )
)

# columns are kept

(
    df.with_columns(
        (pl.col("city08") + 10).alias("added"),
        (pl.col("city08") - 10).alias("substracted"),
    )
)

Now that we've clarified this point, let's proceed to explore the fundamental methods for selecting columns.

Select all columns

A particularly handy tool is pl.all(), which provides the current state of the dataframe—similar to pd.assign(foo=lambda df) in Pandas. This can prove useful, particularly when dealing with operations like groupby and aggregation.

# this is analogous to df.select(pl.col("*")), where * represents the wildcard component
df.select(pl.all())

Select all columns except for...

df.select(pl.col("*").exclude("year", "comb08", "highway08"))

Select a subset of columns


df.select(pl.col("year", "comb08", "highway08"))

Select columns based on regular expression


df.select(pl.col("^.*(mo|ma).*$"))

Select columns based on dtypes

This may remind you of pandas' select_dtypes method.

df.select(pl.col(pl.Int8, pl.Boolean))

Using `selectors`

Selectors offer a more intuitive approach to selecting columns from DataFrame or LazyFrame objects, taking into account factors like column names, data types, and other properties. They consolidate and enhance the functionality that's accessible through the col() expression. More on them here.

By dtypes

df.select(cs.integer(), cs.float(), cs.temporal())

# all columns except for the ones containing float
df.select(cs.all() - cs.numeric())

# same as the one above but it uses the tilde
df.select(~cs.numeric())

By column names


df.select(cs.contains("08"))


df.select(cs.starts_with("d"))

The possibilities here are incredibly vast! I'm pretty sure you'll find a function that suits your needs just right.

`dtype` conversions

Understanding and managing data types, as well as converting them, are very important—particularly when dealing with larger datasets. Luckily, with Polars, this process is wonderfully straightforward. Here's a useful tip: in many cases, the cast method will be your trusty companion for data type conversions. It's quite versatile and often does the job seamlessly. When it comes to converting dates, things can get a bit more intricate, but rest assured, it's a hurdle you can easily overcome!

More info on the available dtypes can be found here.


(
    df.select(
        pl.col("cylinders").cast(pl.Int32),
        pl.col("displ").cast(pl.Int64),
        pl.col("mpgData").cast(pl.Int64),
    )
)

Working with `str`

We can examine the available attributes when working with strings. You might notice that it boasts some familiar attributes, like extract, replace, split, and strip, among others.

# attributes of str
print(dir(pl.col("*").str))

Basic operations

(
    df.select(
        pl.col("make").cast(pl.Utf8).str.contains("Hyundai|Kia").alias("contains"),
        pl.col("make").cast(pl.Utf8).str.starts_with("Alfa").alias("starts_with"),
        pl.col("make").cast(pl.Utf8).str.ends_with("ari").alias("ends_with"),
    )
)

We can extract the numbers from the different car models:


(
    df.select(
        pl.col("model")
        .cast(pl.Utf8)
        .str.extract(r"(\d+)", group_index=1)
        .cast(pl.Int32)
        .alias("extracted_number"),
    )
)

As per usual, we can replace values in a given column:


(
    df.select(
        pl.col("make").cast(pl.Utf8).str.replace("r", 0).alias("replaced"),
    )
)

Changing column names

Altering column names is quite reminiscent of the process in Pandas:

df.rename({"make": "car maker", "model": "car model"})

In case you would like to alter multiple column names all at once:

df.select(pl.all().map_alias(lambda name: name.upper().replace("I", "0000")))

`str.split` with `expand='all'`

I often find myself using the str.split method, and it caught me off guard that there isn't a direct equivalent in Polars. Fingers crossed, we might come across an implementation like expand=True from Pandas, which would be a real game-changer here too!


(
    df.select(
        pl.col("drive")
        .cast(pl.Utf8)
        .str.split_exact("-", n=1)
        .struct.rename_fields(["first_part", "second_part"])
        .alias("fields"),
    ).unnest("fields")
)

Aggregation

Split-apply-combine operations are where Polars truly excels. This is largely due to its foundation in Rust, Polars' use of lazy evaluation, parallel execution, and efficient memory management also contribute to its speed. As a result, split-apply-combine operations are significantly faster in Polars compared to conventional methods.

A simple split-apply-combine operation


(df.groupby("make").count().sort(by="count", descending=True).limit(5))

Using `agg`

You can use agg to calculate statistics over either a single or multiple columns all at once.

(df.groupby("make").agg(pl.count(), pl.col("model")))

Here's an illustration of how to utilize the is_not_nan method. In Polars, unknown values are represented with floating-point precision, similar to how pandas handles them. It's important not to mix this up with missing values, which are indicated by Null in Polars. Additionally, take note of the use of limit instead of head, a concept quite akin to SQL.

(
    df.groupby("make")
    .agg(
        pl.col("city08").mean().alias("mean_cyl"),
        pl.col("displ").mean().alias("mean_disp"),
    )
    .filter(pl.col("mean_cyl").is_not_nan())
    .sort(by=["mean_cyl", "mean_disp"], descending=[True, True])
    .limit(5)
)

Aggregated columns can be renamed immediately with the alias method.

(
    df.groupby("make", "fuelType")
    .agg(pl.col("comb08").mean().alias("filtered"))
    .filter((pl.col("fuelType") == "CNG") | (pl.col("fuelType") == "Diesel"))
)

Here's a scenario for performing aggregation based on the year. In this case, the "year" column holds the year-related data, which can be extracted using the dt.year attribute. Keep in mind that Polars doesn't have native plotting capabilities like pandas. To visualize the data, as a final step, you can convert the dataframe to pandas and make use of its .plot method.

(
    df.groupby(
        [
            pl.col("year").dt.year().alias("year"),
        ]
    )
    .agg([pl.col("comb08").mean(), pl.col("city08").mean()])
    .sort(
        [
            "year",
        ]
    )
    .to_pandas()
    .set_index("year")
    .plot()
)

Here's another fantastic option for you: the groupby_dynamic feature. It's really impressive, by the way. You can employ this when grouping based on years or any other time-related information. Additionally, you can make use of the every parameter for resampling, which closely resembles what you can do with pandas.

(
    df.sort(by="year")
    .groupby_dynamic(index_column="year", every="2y")
    .agg([pl.col("comb08").mean(), pl.col("city08").mean()])
    .sort(
        [
            "year",
        ]
    )
    .to_pandas()
    .set_index("year")
    .plot()
)

Pivot

You can also accomplish split-apply-combine tasks seamlessly using the pivot function. It's remarkably straightforward.

(
    df.with_columns(
        pl.col("fuelType").cast(pl.Utf8)
    )  # conversion to str is needed for the next step
    .filter(pl.col("fuelType").is_in(["Regular", "Premium"]))
    .pivot(values="city08", index="make", columns="fuelType", aggregate_function="mean")
)

Handling missing data

Getting missing value count per column

The null_count function showcases the count of missing values in the dataframe. Remember, these are not the same as np.nans; it's important to be aware of this distinction. This functionality is akin to the pandas df.isna().sum() operation if you're familiar with pandas.

df.null_count()

Getting boolean array of missing values

If you ever need to manipulate a series based on its null values, you can also obtain a boolean mask using the is_null function. This can be really handy for targeted data manipulation.


df.select(pl.col("city08").is_null())

Filling missing values

You have a variety of options at your disposal for filling in missing values. Here's a list of some of the most common ones, although it's not an exhaustive rundown.

With literals

(df.with_columns(pl.col("cylinders").fill_null(pl.lit(2))))

(df.with_columns(pl.col("cylinders").fill_null(strategy="zero")))

(df.with_columns(pl.col("cylinders").fill_null(strategy="forward")))

With an expression

This can be very useful for machine learning pipelines. When addressing missing values, you can set them to the mode, median, mean, or any other value that suits your needs.

(df.with_columns(pl.col("cylinders").fill_null(pl.col("cylinders").mode())))

Using the `over` function

The over function in Polars is used to perform window operations, akin to what you'd achieve with window functions in SQL or the transform function in pandas. This function enables you to compute aggregations over a specified window or range of rows defined by a window specification. It's perfect for tasks like calculating rolling averages, cumulative sums, and other operations that involve working with a specific subset of rows within the dataset.

Let's begin by filtering the car makes to encompass only the top 5 car brands. Following that, we can utilize the over function to derive several statistics.

top5 = (
    df.select(pl.col("make"))
    .to_series()
    .value_counts()
    .sort(by="counts", descending=True)
    .limit(3)
    .select(pl.col("make"))
    .to_series()
)

(
    df.filter(pl.col("make").is_in(top5)).select(
        "make",
        "model",
        pl.col("city08").mean().over("make").alias("avg_city_mpg_by_make"),
        pl.col("city08").mean().over("model").alias("avg_city_mpg_by_model"),
        pl.col("comb08").mean().over(["make", "model"]).alias("avg_comb_mpg_by_model"),
    )
)

Other common operations

In the upcoming section, you'll discover a mix of handy operations that I've ended up using quite often. These don't exactly fit into the categories we've covered earlier, but they're still quite useful!

Operating on multiple columns and renaming them

You have the flexibility to conduct column operations and then easily rename them. Additionally, you can make use of the prefix and suffix functions to streamline your workflow.

(df.with_columns((cs.numeric() * 2).prefix("changed_")))

Count unique values

Absolutely, you can also count the unique values within a series in Polars, much like you would with the unique function in pandas. This can be really useful for understanding the distribution and diversity of data within a specific column.

df.select(pl.col("make")).n_unique()

Value_counts to get number of occurrences per item

On a single column


(df.select(pl.col("make")).to_series().value_counts(sort=True))


(df.select(pl.col("make")).groupby("make").count().sort(by="count", descending=True))

On multiple columns

I noticed that Polars lacks the capability to perform value counts on multiple columns, unlike pandas' value_counts function which operates only on series. However, I've discovered that combining a groupby operation with a count function can effectively achieve the same outcome for multiple columns. It's all about finding alternative approaches that get the job done!


(
    df.select(pl.col("make", "model"))
    .groupby(["make", "model"])
    .count()
    .sort(by="count", descending=True)
)

Conditionals/ if-else statements

If you're looking to integrate conditional if-else statements into your Polars chain, you can make use of the when, then, and otherwise functions. However, keep in mind that a series of chained when, then statements should be interpreted as if, elif, ... elif, rather than if, if, ... if. In other words, the first condition that evaluates to True will be selected. It's crucial to understand this distinction when crafting your conditions.


(
    df.select(
        pl.col("make"),
        pl.when(pl.col("comb08") < 15)
        .then("bad")
        .when(pl.col("comb08") < 30)
        .then("good")
        .otherwise("very good")
        .alias("fuel_economy"),
    )
)

Alright, folks, that's it for this introduction to Polars. I know it wasn't exactly short, but I hope it was informative! I'll be adding more content to this article in the future, so stay tuned. Until then, happy coding 😎💻🔎

Document version

Version	Date	Comment
0.1	8/11/2023	Document created

The Ultimate Speed Test: Pandas vs Polars

Adam Cseresznye — Sun, 13 Aug 2023 00:00:00 GMT

Once I shared my first article about "Getting Started with Polars" I started thinking about something exciting: comparing the speed of Polars and Pandas. This curiosity was sparked by all the buzz around the brand-new Pandas 2.0 release, promising lightning-fast performance. Pandas 2.0 was announced to come packed with cool features, including the addition of Apache Arrow (pyarrow) as its backing memory format. The big perk of Apache Arrow is that it makes operations speedier and more memory-friendly. Naturally, this got me wondering: how does Pandas 2.0 measure up against Polars? Let's dive in and find out!

:::warning

Keep in mind: occasionally, I refer to some performance difference figures. Even though I repeated these experiments 100 times each, because computations can have a bit of randomness, there might be slight variations in the exact numbers.

:::

Setup

import pandas as pd
import polars as pl
import polars.selectors as cs

import numpy as np
import timeit
import random

import plotly.io as pio
import plotly.express as px

pio.templates.default = "presentation"

print(pd.__version__)
print(pl.__version__)

:::note

Throughout this article, we'll be working with the latest versions of the packages: Pandas 2.0.3 and Polars 0.18.11.

:::

Generate the synthetic data

For the purpose of benchmarking, we're going to create our own dataset. To ensure simplicity and uniformity, we'll generate a dataframe containing columns like name, year of birth, and city of residence – pretty straightforward stuff. We're aiming for a dataset of 10,000 rows. While we could certainly run this benchmark on a dataframe with more than a million rows, our intention is to provide a realistic example. The code we'll provide allows you to effortlessly explore various scenarios according to your requirements. Feel free to use it or modify it if you want to perform your own test.

def create_sample_dataframe(n_rows: int, name: str, seed: int = 42) -> pl.DataFrame:
    """
    Create a sample Polars DataFrame with the specified number of rows.

    This function generates random data for several columns and creates a Polars
    DataFrame with the specified number of rows. The generated data includes information
    about names, birth years, cities, zip codes, incomes, marital status, number of children,
    and car brands. The resulting DataFrame is written to CSV and Parquet files named
    'sample.csv' and '{name}.parquet', respectively.

    Parameters
    ----------
    n_rows : int
        The number of rows to generate in the sample DataFrame.
    name : str
        The name to be used when writing the Parquet file.
    seed : int, optional
        The seed to use when generating random data. Default is 42.

    Returns
    -------
    pl.DataFrame
        The first 5 rows of the generated sample DataFrame.
    """
    random.seed(seed)
    np.random.seed(seed)

    data = {
        "name": np.random.choice(["Alice", "Bob", "Charlie", "David", "Eva"], n_rows),
        "born": np.random.randint(1950, 1990, size=n_rows),
        "city": np.random.choice(
            ["Shanghai", "San Francisco", "London", "Munich", "Mumbai"], n_rows
        ),
        "zip_code": np.random.randint(10000, 99999, size=n_rows),
        "income": np.random.normal(50000, 10000, size=n_rows),
        "is_married": np.random.choice([True, False], n_rows),
        "children": np.random.randint(0, 5, size=n_rows),
        "car": np.random.choice(["Ford", "BMW", "Toyota", "Bentley", "Mini"], n_rows),
    }
    temp_df = pl.DataFrame(data)
    temp_df.write_csv("sample.csv")
    temp_df.write_parquet(f"{name}.parquet")
    return temp_df.head()


create_sample_dataframe(n_rows=10_000, name="sample", seed=1)

Define helper class

In order to streamline our code, we'll create a convenient helper class featuring two essential methods: get_time and get_figure. The get_time method leverages the timeit.repeat function to accurately gauge the execution time of our code. In this context, we'll set the parameters number = 1 and repeat = 100. This configuration runs the operations 100 times, allowing us to derive key statistics like the mean, median, and standard deviation.

On the other hand, the get_figure method takes these gathered results and generates a visually appealing image using Plotly.

To bring it all together and keep things simple, we've designed the run_test function. This function orchestrates the entire process, seamlessly merging the steps outlined above.

class Timeit:
    """
    A class for measuring the execution time of multiple functions.
    This class provides methods for measuring the execution time of
    multiple functions using the `timeit` module. The functions to be
    tested are passed to the constructor as a dictionary, where the
    keys are the names of the functions and the values are the functions
    themselves. The `number` and `repeat` parameters control how many times
    each function is executed and how many times the timing is repeated,
    respectively.

    The `get_time` method measures the execution time of each function and
    stores the results in a dictionary, where the keys are the names of
    the functions and the values are lists of execution times. The `get_figure`
    method generates a box plot of the execution times using the `plotly.express.box`
    function.

    Parameters
    ----------
    func_dict : dict
        A dictionary of functions to test, where the keys are the names of the
        functions and the values are the functions themselves.
    number : int, optional
        The number of times to execute each function in each timing run (default is 1).
    repeat : int, optional
        The number of times to repeat each timing run (default is 20).

    Attributes
    ----------
    results : dict or None
        A dictionary of execution times for each function, where the keys are the names
        of the functions and the values are lists of execution times. This attribute is
        `None` until `get_time` is called.

    Methods
    -------
    get_time()
        Measure the execution time of each function and store the results in the `results` attribute.
    get_figure()
        Generate a box plot of the execution times using `plotly.express.box`.
    """

    def __init__(self, func_dict, number=1, repeat=100):
        self.func_dict = func_dict
        self.number = number
        self.repeat = repeat
        self.results = None

    def get_time(self):
        """
        Measure the execution time of each function and store the results.
        This method uses the `timeit.repeat` function to measure the execution time of each function
        in `func_dict`. The results are stored in a dictionary, where the keys are the names of
        the functions and the values are lists of execution times. This dictionary is also stored
        in the `results` attribute of the `Timeit` object.

        Returns
        -------
        dict
            A dictionary of execution times for each function.
        """
        results = {}
        for k, v in self.func_dict.items():
            results[k] = timeit.repeat(v, number=self.number, repeat=self.repeat)
        self.results = results
        return results

    def get_figure(self):
        """
        Generate a box plot of the execution times.
        This method uses `plotly.express.box` to generate a box plot of the execution times
        stored in `results`.If `results` is `None`, this method raises a `ValueError`.

        Returns
        -------
        plotly.graph_objs.Figure
            A box plot figure object.

        Raises
        ------
        ValueError
            If no results are available (i.e., if `get_time` has not been called).
        """
        if self.results is None:
            raise ValueError("No results available. Please run get_time first.")
        df = pd.DataFrame(self.results)
        mean_times = df.median() * 1000
        title = f'Median Execution Times: <br><sup>{", ".join([f"{k}={v:.6f}ms" for k,v in mean_times.items()])}</sup>'
        return px.box(
            df,
            points="all",
            labels={"value": "time (sec)", "variable": ""},
            title=title,
            width=600,
            height=500,
        )


def run_test(test: dict, name: str):
    """
    Run a timing test on a dictionary of functions using the Timeit class and
    return the results as a DataFrame.

    This function takes a dictionary of functions as input, where the keys are
    the names of the functions and the values are the functions themselves. It creates
    a Timeit object with this dictionary and uses it to measure the execution time of
    each function. The results are then displayed as a box plot using the `get_figure`
    method of the Timeit object and returned as a Pandas DataFrame with columns renamed
    according to the `name` argument.

    Parameters
    ----------
    test : dict
        A dictionary of functions to test, where the keys are the names of the functions
        and the values are the functions themselves.
    name : str
        A string used to rename the columns of the returned DataFrame.

    Returns
    -------
    pd.DataFrame
        A DataFrame containing the results of the timing test, with columns renamed
        according to the `name` argument.
    """

    # Create a Timeit object
    t = Timeit(test)
    # Measure the execution time of the functions
    results = t.get_time()
    # Generate a box plot of the execution times
    fig = t.get_figure()
    fig.show()

    return pd.DataFrame(results).rename(
        columns={"Pandas": f"{name}_pandas", "Polars": f"{name}_polars"}
    )

Reading in data From a CSV File and parquet

CSV

Reading CSV files from disk is a task that data scientists often find themselves doing. Now, let's see how these two libraries compare for this particular job. To maximize the blazing-fast data handling capabilities of PyArrow, we'll equip Pandas with the engine="pyarrow" and dtype_backend="pyarrow" arguments. Let's see how these choices shape the performance!


test_dict = {
    "Pandas": lambda: pd.read_csv(
        "sample.csv", engine="pyarrow", dtype_backend="pyarrow"
    ),
    "Polars": lambda: pl.read_csv("sample.csv"),
}
read_csv = run_test(test_dict, "Read csv")

For the sake of comparison, we'll also demonstrate the timeit function invoked using Jupyter cell magic. You'll notice that the numbers generated this way are quite closely aligned with ours.

%%timeit
pd.read_csv("sample.csv", engine="pyarrow", dtype_backend="pyarrow")

%%timeit
pl.read_csv("sample.csv")

parquet

Now, let's read the data in Parquet format.


test_dict = {
    "Pandas": lambda: pd.read_parquet(
        "sample.parquet", engine="pyarrow", dtype_backend="pyarrow"
    ),
    "Polars": lambda: pl.read_parquet("sample.parquet"),
}
read_parquet = run_test(test_dict, "Read parquet")

:::note

Polars unquestionably wins this round, it can boast a speed advantage of 2 to 4 times over Pandas. :::

Selecting Columns

Alright, let's spice things up a bit and select some columns to see which library races ahead in terms of speed!

Files awaiting in-memory reading

A clever approach to conserve memory and enhance speed involves reading only the columns essential for operations. Consider a scenario where we're interested in displaying just the names from this dataset. The big question now: how do these libraries measure up in terms of speed? Let's find out!


test_dict = {
    "Pandas": lambda: pd.read_csv(
        "sample.csv", engine="pyarrow", dtype_backend="pyarrow", usecols=["name"]
    ),
    "Polars": lambda: pl.scan_csv("sample.csv").select(pl.col("name")).collect(),
}
select_col_not_in_memory = run_test(test_dict, "Select column (not in memory)")

File is in memory

As anticipated, Polars continues to showcase its swiftness. It's worth highlighting the usage of the lazy and collect methods in Polars. These nifty tools grant us access to the library's clever query optimization techniques, which play a pivotal role in significantly enhancing performance. OK, one step further: suppose our files are already loaded into memory. Would there still be a distinction in performance under this circumstance?

df_pandas = pd.read_csv("sample.csv", engine="pyarrow", dtype_backend="pyarrow")
df_polars = pl.read_csv("sample.csv")


test_dict = {
    "Pandas": lambda: df_pandas.loc[:, "name"],
    "Polars": lambda: df_polars.lazy().select(pl.col("name")).collect(),
}
select_col_in_memory = run_test(test_dict, "Select column")

:::note

While Polars showed a significant speed advantage for tasks involving pre-read files, both libraries perform similarly when the files are already in memory. :::

Filtering Rows

Now, let's explore the scenario where we filter our dataset based on one or more column values

Based on one condition

For our simple scenario, we'll be narrowing down our focus to filter data based on individuals with the name "David".


test_dict = {
    "Pandas": lambda: (df_pandas.query("name=='David'")),
    "Polars": lambda: (df_polars.lazy().filter((pl.col("name") == "David")).collect()),
}
filter_row_one_condition = run_test(test_dict, "Filter (simple)")

Based on multiple conditions

Now, for a more intricate challenge, we're going to dive into querying the data to extract individuals who meet specific criteria: those named David, born after 1980, residing in a city other than London, married, and with three children.


test_dict = {
    "Pandas": lambda: (
        df_pandas.query(
            "name=='David' and born>1980 and city != 'London' or is_married == True and children >= 3"
        )
    ),
    "Polars": lambda: (
        df_polars.lazy()
        .filter(
            (pl.col("name") == "David")
            & (pl.col("born") > 1980)
            & (pl.col("city") != "London")
            | (pl.col("is_married") == True) & (pl.col("children") >= 3)
        )
        .collect()
    ),
}
filter_row_multiple_condition = run_test(test_dict, "Filter (complex)")

:::note

Both libraries tackled this challenge quite well, yet Pandas struggled to keep pace with Polars. It's intriguing to observe that while Pandas required nearly twice the time for the more intricate task, Polars managed to complete it in almost the same amount of time. Parallelization in action.

:::

Performing operations on columns

Now, let's roll up our sleeves and dive into performing some operations on the columns.

Single operation

As a single operation, we'll simply calculate the century in which these individuals were born.


test_dict = {
    "Pandas": lambda: (df_pandas.assign(born=lambda df: df.born.div(100).round())),
    "Polars": lambda: (
        df_polars.lazy().with_columns((pl.col("born") / 100).round()).collect()
    ),
}
operate_one_column = run_test(test_dict, "Operate (one column)")

Multiple operations

Let's also explore what happens when performing multiple operations on the columns. We'll mix things up with some string operations, mapping, and math calculations to see how these libraries handle it!


test_dict = {
    "Pandas": lambda: (
        df_pandas.assign(
            born=lambda df: df.born.div(100).round(),
            name=lambda df: df.name.str.lower(),
            city=lambda df: df.city.str.upper(),
            zip_code=lambda df: df.zip_code.mul(2),
            income=lambda df: df.income.div(10).astype("str"),
            is_married=lambda df: df.is_married.map({False: 0, True: 1}),
            children=lambda df: df.children.astype("bool"),
            car=lambda df: df.car.str[0],
        )
    ),
    "Polars": lambda: (
        df_polars.lazy()
        .with_columns(
            [
                (pl.col("born") / 100).round(),
                pl.col("name").str.to_lowercase(),
                pl.col("city").str.to_uppercase(),
                pl.col("zip_code") * 2,
                (pl.col("income") / 10).cast(pl.Utf8),
                pl.col("is_married").map_dict({False: 0, True: 1}),
                pl.col("children").cast(pl.Boolean),
                pl.col("car").str.slice(0, length=1),
            ]
        )
        .collect()
    ),
}
operate_multiple_column = run_test(test_dict, "Operate (more columns)")

:::note

Once again, Polars takes the lead. While both libraries required more time for the task involving multiple operations, Polars demonstrated superior scalability in this scenario.

:::

Concatenating Data

Now, let's turn our attention to concatenating two datasets. Let the merging begin!

df_pandas2 = df_pandas.copy(deep=True)
df_polars2 = df_polars.clone()


test_dict = {
    "Pandas": lambda: pd.concat([df_pandas, df_pandas2], axis=0),
    "Polars": lambda: pl.concat([df_polars, df_polars2], how="vertical"),
}
concatenate = run_test(test_dict, "Concatenate")

:::note

Once more, Polars shines with a remarkable speed advantage.

:::

Aggregation

Simple

Time to shift our attention to aggregation. First up, a simple task: let's calculate the mean income based on names. Then, for a bit more complexity, we'll dive into computing statistics involving the income, children, and car columns. Things are about to get interesting!


test_dict = {
    "Pandas": lambda: (df_pandas.groupby("name").income.mean()),
    "Polars": lambda: (df_polars.lazy().groupby("name").mean().collect()),
}
aggregate_simple = run_test(test_dict, "Aggregate (simple)")

More complex


test_dict = {
    "Pandas": lambda: (
        df_pandas.groupby(["name", "car", "is_married"]).agg(
            born_min=("born", min),
            born_max=("born", max),
            income_mean=("income", np.mean),
            income_median=("income", np.median),
            children_mean=("children", np.mean),
            car_count=("car", "count"),
        )
    ),
    "Polars": lambda: (
        df_polars.lazy()
        .groupby(["name", "car", "is_married"])
        .agg(
            [
                pl.col("born").min().alias("born_min"),
                pl.col("born").max().alias("born_max"),
                pl.col("income").mean().alias("income_mean"),
                pl.col("income").median().alias("income_median"),
                pl.col("children").mean().alias("children_mean"),
                pl.col("car").count().alias("car_count"),
            ]
        )
        .collect()
    ),
}
aggregate_complex = run_test(test_dict, "Aggregate (complex)")

:::note

While Pandas showcased noteworthy speed for the simple aggregation, the more intricate task exposed significant disparities between the two libraries. Polars took a commanding lead in this scenario, presenting a considerably faster performance compared to Pandas. :::

Whole workflow

We're going to tackle this in two versions. First, we'll use pl.read_csv for Polars, assessing the time it takes to complete the entire workflow on a DataFrame already in memory. Then, for the second version, we'll employ pl.scan_csv for Polars. This nifty function lets us lazily read data from a CSV file or multiple files using glob patterns.

The cool part? With pl.scan_csv, the query optimizer can push down predicates and projections to the scan level. This nifty move has the potential to cut down on memory overhead. Let's dive into both versions and see how they stack up!

Using `pl.read_csv`


test_dict = {
    "Pandas": lambda: (
        df_pandas.loc[:, lambda df: ~df.columns.isin(["is_married"])]
        .query("name=='David' and born>1980 and city != 'London' and children >= 3")
        .assign(
            born=lambda df: df.born.div(100).round(),
            name=lambda df: df.name.str.lower(),
            city=lambda df: df.city.str.upper(),
            zip_code=lambda df: df.zip_code.mul(2),
            income=lambda df: df.income.div(10),
            children=lambda df: df.children.astype("bool"),
            car=lambda df: df.car.str[0],
        )
        .groupby(
            [
                "name",
                "car",
            ]
        )
        .agg(
            born_min=("born", min),
            born_max=("born", max),
            income_mean=("income", np.mean),
            income_median=("income", np.median),
            children_mean=("children", np.mean),
            car_count=("car", "count"),
        )
    ),
    "Polars": lambda: (
        df_polars.lazy()
        .select(cs.all() - cs.ends_with("married"))
        .filter(
            (pl.col("name") == "David")
            & (pl.col("born") > 1980)
            & (pl.col("city") != "London")
            & (pl.col("children") >= 3)
        )
        .with_columns(
            [
                (pl.col("born") / 100).round(),
                pl.col("name").str.to_lowercase(),
                pl.col("city").str.to_uppercase(),
                pl.col("zip_code") * 2,
                (pl.col("income") / 10),
                pl.col("children").cast(pl.Boolean),
                pl.col("car").str.slice(0, length=1),
            ]
        )
        .groupby(
            [
                "name",
                "car",
            ]
        )
        .agg(
            [
                pl.col("born").min().alias("born_min"),
                pl.col("born").max().alias("born_max"),
                pl.col("income").mean().alias("income_mean"),
                pl.col("income").median().alias("income_median"),
                pl.col("children").mean().alias("children_mean"),
                pl.col("car").count().alias("car_count"),
            ]
        )
        .collect()
    ),
}
whole_workflow_read_csv = run_test(test_dict, "Whole workflow")

Using `pl.scan_csv`


test_dict = {
    "Pandas": lambda: (
        df_pandas.loc[:, lambda df: ~df.columns.isin(["is_married"])]
        .query("name=='David' and born>1980 and city != 'London' and children >= 3")
        .assign(
            born=lambda df: df.born.div(100).round(),
            name=lambda df: df.name.str.lower(),
            city=lambda df: df.city.str.upper(),
            zip_code=lambda df: df.zip_code.mul(2),
            income=lambda df: df.income.div(10),
            children=lambda df: df.children.astype("bool"),
            car=lambda df: df.car.str[0],
        )
        .groupby(
            [
                "name",
                "car",
            ]
        )
        .agg(
            born_min=("born", min),
            born_max=("born", max),
            income_mean=("income", np.mean),
            income_median=("income", np.median),
            children_mean=("children", np.mean),
            car_count=("car", "count"),
        )
    ),
    "Polars": lambda: (
        pl.scan_csv("sample.csv")
        .select(cs.all() - cs.ends_with("married"))
        .filter(
            (pl.col("name") == "David")
            & (pl.col("born") > 1980)
            & (pl.col("city") != "London")
            & (pl.col("children") >= 3)
        )
        .with_columns(
            [
                (pl.col("born") / 100).round(),
                pl.col("name").str.to_lowercase(),
                pl.col("city").str.to_uppercase(),
                pl.col("zip_code") * 2,
                (pl.col("income") / 10),
                pl.col("children").cast(pl.Boolean),
                pl.col("car").str.slice(0, length=1),
            ]
        )
        .groupby(
            [
                "name",
                "car",
            ]
        )
        .agg(
            [
                pl.col("born").min().alias("born_min"),
                pl.col("born").max().alias("born_max"),
                pl.col("income").mean().alias("income_mean"),
                pl.col("income").median().alias("income_median"),
                pl.col("children").mean().alias("children_mean"),
                pl.col("car").count().alias("car_count"),
            ]
        )
        .collect()
    ),
}
whole_workflow_scan_csv = run_test(test_dict, "Whole workflow (scan_csv)")

As evident, the utilization of scan_csv increased the required time by about 3-4 times. However, even with this increase, Polars still manages to maintain a substantial advantage of around 5 times faster than the entire workflow executed using Pandas.

:::note

When we consider the entirety of the data processing pipeline, irrespective of the file reading approach, Polars emerges as the victor. It consistently exhibits a considerable speed advantage compared to Pandas.

:::

Putting it all together

Time to bring together all the things we've explored! Let's sum up what we've learned.



summary = pd.concat(
    [
        read_csv,
        read_parquet,
        # select_col_not_in_memory,
        select_col_in_memory,
        filter_row_one_condition,
        filter_row_multiple_condition,
        operate_one_column,
        operate_multiple_column,
        concatenate,
        aggregate_simple,
        aggregate_complex,
        whole_workflow_read_csv,
        # whole_workflow_scan_csv
    ],
    axis=1,
)

fig = (
    summary.melt(var_name="experiment", value_name="time_sec")
    .assign(
        package=lambda df: df.experiment.str[-6:],
        experiment=lambda df: df.experiment.str[:-7],
    )
    .groupby(["experiment", "package"])
    .time_sec.mean()
    .mul(1000)
    .reset_index()
    .pivot(columns="package", values="time_sec", index="experiment")
    .sort_values(
        by="polars",
    )
    .reset_index()
    .pipe(
        lambda df: px.bar(
            df,
            y="experiment",
            x=["pandas", "polars"],
            barmode="group",
            text_auto='.2s',
            orientation="h",
            labels={"value": "Average execution Time (msec)", "experiment": ""},
            title= f'<b>Speed Showdown in Data Workflow Operations</b> <br><sup><i>Pandas {pd.__version__} vs. Polars {pl.__version__}</i></sup>',
            width=1000,
            height=700,
        )
    )
)
fig.update_xaxes(tickangle=90)
fig.update_layout(yaxis=dict(automargin=True), legend_title_text="Library")
fig.update_traces(textposition='outside')

Throughout our exploration, Polars has consistently outperformed Pandas. We intentionally focused on tasks that frequently arise in data analysis, encompassing file reading, column selection, filtering, and more. However, where Polars truly shines is in aggregation, capitalizing on its multiprocessing prowess. Take a glance at @fig-fig14, and you'll notice that reading files is the most time-consuming step for Polars. To address this (and reduce memory usage at the same time), we have the nifty scan_csv function that enables Polars to operate in a lazy mode, optimizing the entire data pipeline. It's clear that Polars packs a punch in the realm of data processing!

Should you consider using Polars? Based on the findings of this experiment, it's evident that Polars can bring substantial benefits to every stage of your data processing. Its remarkable speed can make a significant impact. If you're open to investing time in understanding its API (which isn't drastically different from Pandas), I'm confident your projects will gain a significant speed boost. And don't forget, the trajectory of the project seems promising, implying a bright future ahead.

Whatever path you choose, ensure you become well-acquainted with your selected library and stay current with emerging technologies. And above all, relish the journey. Happy coding, and until next time! 🐼🐍🤓💻

Tidy Tuesday: Spam E-mail

Adam Cseresznye — Tue, 15 Aug 2023 00:00:00 GMT

Welcome to this week's Tidy Tuesday! Today, we're delving into the intriguing yet bothersome realm of email spams. You know, those unsolicited messages that flood our inboxes? They go by many names like junk email or spam mail, and they're sent in bulk. The term "spam" got its name from a Monty Python sketch where the word "Spam" was everywhere, just like these emails. Starting from the early 1990s, these spam messages have been on a steady rise, making up around 90% of all email traffic by 2014.

We've got our data from the Tidy Tuesday treasure trove over at GitHub! This dataset, from Vincent Arel-Bundock's Rdatasets package, was initially gathered at Hewlett-Packard Labs. They later kindly shared it with the UCI Machine Learning Repository.

This treasure trove of information consists of 4601 emaily sorted into spam and non-spam categorie

Setup

import polars as pl
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import polars.selectors as cs
import numpy as np
from sklearn import preprocessing
from sklearn import decomposition


from lets_plot import *
from lets_plot.mapping import as_discrete

LetsPlot.setup_html()

import plotly.io as pio
import plotly.express as px

pio.templates.default = "presentation"

df = pl.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-15/spam.csv"
).with_columns(
    [
        pl.col("crl.tot").cast(pl.Int16),
        pl.col("dollar").cast(pl.Float32),
        pl.col("bang").cast(pl.Float32),
        pl.col("money").cast(pl.Float32),
        pl.col("n000").cast(pl.Float32),
        pl.col("make").cast(pl.Float32),
        pl.col("yesno").map_dict({"y": 1, "n": 0}).cast(pl.Int8),
    ]
)

df.sample(5)

Here is the data dictionary:

Variable	Class	Description
crl.tot	double	Total length of uninterrupted sequences of capitals
dollar	double	Occurrences of the dollar sign, as percent of total number of characters
bang	double	Occurrences of ‘!’, as percent of total number of characters
money	double	Occurrences of ‘money’, as percent of total number of characters
n000	double	Occurrences of the string ‘000’, as percent of total number of words
make	double	Occurrences of ‘make’, as a percent of total number of words
yesno	character	Outcome variable, a factor with levels 'n' not spam, 'y' spam

Looking at the data dictionary, a few intriguing questions arise:

How is the distribution between True and False answers? Are non-spam emails significantly more prevalent?
What are the top words that frequently appear in spam emails?
Can we explore the relative frequency distribution disparities of words linked with spam? What's the increased likelihood of finding these specific words in a spam email versus a non-spam one?
Is there an overarching correlation within the dataset? Are certain words frequently seen together?
Could we pinpoint the ultimate worst spam email in the dataset?
Would it be feasible to employ principal component analysis to visualize this dataset effectively?

What's the distribution of True and False answers

(
    df.select(pl.col("yesno"))
    .to_series()
    .value_counts()
    .with_columns((pl.col("counts") / pl.sum("counts")).alias("percent_count"))
)

:::note As you've observed, the dataset displays a slight imbalance, with spam emails constituting approximately 40% of the dataset. :::

What are the most frequently used words in spam emails?


fig = (
    df.filter(pl.col("yesno") == 1)
    .select(cs.float())
    .melt()
    .groupby("variable")
    .agg(
        pl.col("value").mean().alias("mean"),
    )
    .sort(by="mean")
    .pipe(
        lambda df: px.bar(
            df,
            x="variable",
            y="mean",
            color="variable",
            width=600,
            height=500,
            labels={
                "mean": "Average % occurrence per total word count",
                "variable": "",
            },
        )
    )
)
fig.update_traces(hovertemplate="<br>Occurence: %{y:.2f}")  #

:::note Among the frequently used words, "bang" takes a noticeable spot, appearing on average in 0.51% of the total word count.

:::

What are the relative frequency distribution differences of the words associated with spam?


fig = (
    df.select(pl.all().exclude("crl.tot"))
    .groupby("yesno")
    .agg(
        pl.all().mean(),
    )
    .to_pandas()
    .apply(lambda x: x / x.sum(), axis=0)
    .melt(id_vars="yesno")
    .rename(columns={"yesno": "spam"})
    .assign(spam=lambda df: df.spam.map({0: "Not a spam", 1: "Spam"}))
    .assign(percentage=lambda x: ["{:.2%}".format(val) for val in x["value"]])
    .pipe(
        lambda df: px.bar(
            df,
            x="variable",
            y="value",
            color="spam",
            labels={
                "value": "Relative frequency of words compared to class ",
                "variable": "",
            },
            text="percentage",
            width=700,
            height=600,
        )
    )
)

fig.update_layout(
    showlegend=True, uniformtext_minsize=8, uniformtext_mode="hide", hovermode=False
)
fig.update_traces(
    textposition="inside",
)
fig.update_yaxes(tickformat=".0%")

:::note The word "n000" is 36x more likely to appear in a spam email. :::

What's the overall correlation


fig = (
    df.to_pandas()
    .corr()
    .round(2)
    .pipe(
        lambda df: px.imshow(
            df,
            text_auto=True,
            aspect="equal",
            zmin=-1,
            zmax=1,
            color_continuous_scale="RdBu_r",
            # origin='lower',
            width=600,
            height=600,
        )
    )
)
fig.update_xaxes(side="top")

:::note Our target variable, "yesno," exhibits the strongest moderate positive correlation with the occurrence of "n000." :::

What is the worst spam email?

To identify the most significant culprit within the dataset, we'll sum up the numerical percentage values and then sort the values accordingly.

(
    df.filter(pl.col("yesno") == 1)
    .with_columns(pl.sum_horizontal(pl.all().exclude("crl.tot", "yesno")))
    .sort(by="sum", descending=True)
    .head()
)

:::note The most concerning spam email in our dataset showcases a total occurrence of 3% for the word "bang," 9% for "money," and 4.5% for "make." Yikes. :::

Visualize data with PCA

To enhance our grasp of the data, we can employ Principal Component Analysis (PCA). This technique comes in handy when we're dealing with multiple independent variables, allowing us to reduce dimensionality and streamline the dataset.

Visualize all the principal components

X = df.select(pl.all().exclude("yesno")).to_pandas()
y = df.select(pl.col("yesno")).to_pandas()

# Before PCA we need to scale and transform our dataset

scaler = preprocessing.PowerTransformer().set_output(transform="pandas")

X = scaler.fit_transform(X)


pca = decomposition.PCA()
components = pca.fit_transform(X)
labels = {
    str(i): f"PC {i+1} ({var:.1f}%)"
    for i, var in enumerate(pca.explained_variance_ratio_ * 100)
}

fig = px.scatter_matrix(
    components,
    opacity=0.2,
    labels=labels,
    dimensions=range(6),
    color=y.squeeze().map({0: "Not a Spam", 1: "Spam"}),
    width=1000,
    height=1000,
)
fig.update_traces(diagonal_visible=False)
fig.show()

:::note As evident, the first and second principal components manage to account for 62% of the variance within the data. A distinct separation of classes is noticeable, particularly with the spam group exhibiting a broader distribution, indicating greater diversity. :::

Visualize Loadings

For a deeper comprehension of how each characteristic influences our principal components, we can delve into examining the loadings.


pca = decomposition.PCA(n_components=2)
components = pca.fit_transform(X)
loadings = pca.components_.T * np.sqrt(pca.explained_variance_)


fig = px.scatter(
    components,
    x=0,
    y=1,
    color=y.squeeze().map({0: "Not a Spam", 1: "Spam"}),
    labels={
        "0": f"PC1 ({pca.explained_variance_ratio_[0]:.0%})",
        "1": f"PC2 ({pca.explained_variance_ratio_[1]:.0%})",
    },
    template="plotly_dark",
    color_discrete_sequence=[
        "red",
        "green",
    ],
    opacity=0.4,
    width=700,
    height=700,
)
for i, feature in enumerate(X.columns):
    fig.add_annotation(
        ax=0,
        ay=0,
        axref="x",
        ayref="y",
        x=loadings[i, 0],
        y=loadings[i, 1],
        showarrow=True,
        arrowsize=1,
        arrowhead=2,
        xanchor="right",
        yanchor="top",
    )
    fig.add_annotation(
        x=loadings[i, 0],
        y=loadings[i, 1],
        ax=0,
        ay=0,
        font=dict(
            size=20,
            # color='yellow'
        ),
        xanchor="left",
        yanchor="bottom",
        text=feature,
        yshift=5,
    )
fig

:::note

The loadings plot reveals a strong correlation among the variables "crl.tot," "money," and "n000". If we were to construct a machine learning model for spam email recognition, we could opt for one of these variables to streamline our dataset. Furthermore, the disassociation of "bang" from this trio is evident, its vector positioned at a 90° angle. :::

Alright folks, our journey into the intriguing realm of spam emails has left us with some valuable insights. It's clear that words like "bang" and "n000" are key indicators to watch out for. Luckily, modern machine learning models are here to do the hard work on our behalf.

Stay vigilant out there and take care! See you in the next week's adventure! 👋📧🛡️

Tidy Tuesday: Refugees

Adam Cseresznye — Tue, 22 Aug 2023 00:00:00 GMT

Welcome to our blog dedicated to Tidy Tuesday. This week, we venture into the sobering realm of refugee statistics using the {refugees} R package. This tool grants access to the United Nations High Commissioner for Refugees (UNHCR) Refugee Data Finder, providing critical insights into forcibly displaced populations spanning over seven decades. With data from UNHCR, UNRWA, and the Internal Displacement Monitoring Centre, we'll explore a subset of this information, focusing on population statistics from 2010 to 2022.

Join us as we explore the landscape of refugee data.

Setup

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import numpy as np

from pathlib import Path
import json
from urllib.request import urlopen


from lets_plot import *
from lets_plot.mapping import as_discrete
from plotly.subplots import make_subplots
import plotly.graph_objects as go

LetsPlot.setup_html()

import plotly.io as pio
import plotly.express as px

pio.templates.default = "presentation"

try:
    df = pd.read_csv(
        "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-22/population.csv"
    )
except:
    df = pd.read_csv(
        Path.cwd().joinpath(
            "raw.githubusercontent.com_rfordatascience_tidytuesday_master_data_2023_2023-08-22_population.csv"
        )
    )

df = (
    df
    # .assign(year=lambda df: pd.to_datetime(df.year, format='%Y'))
)

df.sample(5)

Here is the data dictionary:

variable	class	description
year	double	The year.
coo_name	character	Country of origin name.
coo	character	Country of origin UNHCR code.
coo_iso	character	Country of origin ISO code.
coa_name	character	Country of asylum name.
coa	character	Country of asylum UNHCR code.
coa_iso	character	Country of asylum ISO code.
refugees	double	The number of refugees.
asylum_seekers	double	The number of asylum-seekers.
returned_refugees	double	The number of returned refugees.
idps	double	The number of internally displaced persons.
returned_idps	double	The number of returned internally displaced persons.
stateless	double	The number of stateless persons.
ooc	double	The number of others of concern to UNHCR.
oip	double	The number of other people in need of international protection.
hst	double	The number of host community members.

Looking at the data dictionary, here are the questions we will be exploring in this blog post:

How has the overall refugee count evolved over the years?
What countries experience the highest annual refugee outflows?
What countries receive the most refugees annually?
Where do most refugees from the country with the highest numbers go?
Which countries have the highest numbers of internally displaced persons?
What countries have the highest stateless populations?

In this blog, to visualize the topological data, we will be utilizing Plotly's choropleth_mapbox function as well as scatter_geo. For a quick overview about Plotly check out this website. In addition, our GeoJSON file, which represents simple geographical features along with their non-spatial attributes, is sourced from GitHub curated by Rufus Pollock.

try:
    with urlopen(
        "https://raw.githubusercontent.com/datasets/geo-boundaries-world-110m/master/countries.geojson"
    ) as response:
        countries = json.load((response))
except:
    countries = json.load(open("countries.geojson"))

How has the overall refugee count evolved over the years?

As you can see, the number of stateless persons has not changed significantly over the period studied; on the other hand, the number of internally displaced persons has dramatically increased from over 14 million in 2010 to just shy of 60 million by 2022


fig = (
    df.groupby("year")[["refugees", "asylum_seekers", "stateless", "idps"]]
    .sum()
    .reset_index()
    .melt(id_vars="year")
    .pipe(
        lambda df: px.line(
            df,
            x=df.year,
            y=df.value,
            color=df.variable,
            width=750,
            height=600,
            # markers=True,
            labels={
                "year": "Year",
                "value": "Number of individuals",
            },
        )
    )
)
fig.update_layout(legend_title_text="Categories")

What countries experience the highest annual refugee outflows?

Among all the countries under examination, the highlighted ones represent the top 5 nations with the highest registered refugee populations. It's evident that the Syrian Arab Republic consistently maintained the highest number of registered refugees from 2013 to 2022, followed closely by Afghanistan


fig = go.Figure()

top10_countries = (
    df.groupby("coo_name")
    .refugees.sum()
    .to_frame()
    .reset_index()
    .sort_values(by="refugees", ascending=False)
    .head(5)
    .coo_name.values
)

test = (
    df.groupby(["year", "coo_name"])
    .refugees.sum()
    .to_frame()
    .reset_index()
    .assign(
        color=lambda df: df.coo_name.mask(
            df.coo_name == (top10_countries[0]), "#6eb14a"
        )
        .mask(df.coo_name == (top10_countries[1]), "#2d4526")
        .mask(df.coo_name == (top10_countries[2]), "#f6d19b")
        .mask(df.coo_name == (top10_countries[3]), "#ca3780")
        .mask(df.coo_name == (top10_countries[4]), "#e9b4cd")
        .mask(~df.coo_name.isin(top10_countries), "#dadadc")
    )
)

# Loop over each unique 'coo_name'
for coo_name in test["coo_name"].unique():
    df_subset = test[test["coo_name"] == coo_name]
    if coo_name in top10_countries:
        fig.add_trace(
            go.Scatter(
                x=df_subset["year"],
                y=df_subset["refugees"],
                mode="lines+markers",
                name=coo_name,
                showlegend=True,
                line=dict(
                    color=df_subset["color"].iloc[0], dash="dash"
                ),  # Use the first color from the subset
            )
        )
    else:
        fig.add_trace(
            go.Scatter(
                x=df_subset["year"],
                y=df_subset["refugees"],
                mode="lines",
                name=coo_name,
                showlegend=False,
                line=dict(
                    color=df_subset["color"].iloc[0], dash="dot"
                ),  # Use the first color from the subset
            )
        )


fig.update_layout(
    autosize=False, width=750, height=600, yaxis_title="Sum Annual Refugee Outflows"
)
fig.show()

I have also prepared a short animation that depicts the year-by-year pattern of refugee outflows.


(
    df.assign(year=lambda df: df.year.astype(str).str.split("-", expand=True)[0])
    .groupby(["year", "coo_iso", "coo_name"])["refugees"]
    .sum()
    .to_frame()
    .reset_index()
    .pipe(
        lambda df: px.choropleth_mapbox(
            data_frame=df,
            locations="coo_iso",
            geojson=countries,
            color="refugees",
            animation_frame="year",
            animation_group="coo_iso",
            hover_name="coo_name",
            hover_data={
                "coo_iso": "",
            },
            range_color=(0, 8_000_000),
            color_continuous_scale=px.colors.sequential.Oranges,
            featureidkey="properties.iso_a3",
            zoom=1.2,
            mapbox_style="carto-positron",
            opacity=0.6,
            width=1200,
            height=700
            # **fig_dict
        )
    )
)

What countries receive the most refugees annually?

After examining which countries have the most refugees leaving, we will shift our focus to the countries with the highest number of refugees received.



fig = go.Figure()

top5_countries_accept = (
    df.groupby("coa_name")[["refugees"]]
    .sum()
    .sort_values(by="refugees", ascending=False)
    .head(5)
    .index
)

test = (
    df.groupby(["year", "coa_name"])
    .refugees.sum()
    .to_frame()
    .reset_index()
    .assign(
        color=lambda df: df.coa_name.mask(
            df.coa_name == (top5_countries_accept[0]), "#6eb14a"
        )
        .mask(df.coa_name == (top5_countries_accept[1]), "#2d4526")
        .mask(df.coa_name == (top5_countries_accept[2]), "#f6d19b")
        .mask(df.coa_name == (top5_countries_accept[3]), "#ca3780")
        .mask(df.coa_name == (top5_countries_accept[4]), "#e9b4cd")
        .mask(~df.coa_name.isin(top5_countries_accept), "#dadadc")
    )
)

# Loop over each unique 'coo_name'
for coa_name in test["coa_name"].unique():
    df_subset = test[test["coa_name"] == coa_name]
    if coa_name in top5_countries_accept:
        fig.add_trace(
            go.Scatter(
                x=df_subset["year"],
                y=df_subset["refugees"],
                mode="lines+markers",
                name=coa_name,
                showlegend=True,
                line=dict(
                    color=df_subset["color"].iloc[0], dash="dash"
                ),  # Use the first color from the subset
            )
        )
    else:
        fig.add_trace(
            go.Scatter(
                x=df_subset["year"],
                y=df_subset["refugees"],
                mode="lines",
                name=coa_name,
                showlegend=False,
                line=dict(
                    color=df_subset["color"].iloc[0], dash="dot"
                ),  # Use the first color from the subset
            )
        )


fig.update_layout(
    autosize=False, width=750, height=600, yaxis_title="Total Annual Refugee Arrivals"
)
fig.show()

Where do most refugees from the country with the highest numbers go?

Next, we wanted to find out where Syrian refugees have found welcomes for their new homes.


fig = (
    df.query("coo_name == 'Syrian Arab Rep.'")
    .groupby("coa_name")
    .refugees.sum()
    .pipe(lambda x: x / x.sum())
    .mul(100)
    .sort_values(ascending=False)
    .head(10)
    .to_frame()
    .reset_index()
    .pipe(
        lambda df: px.bar(
            df,
            y="coa_name",
            x="refugees",
            labels={
                "refugees": "Acceptance Rate of Syrian Refugees (%)",
                "coa_name": "",
            },
            height=500,
            width=1000,
        )
    )
)
fig.update_yaxes(tickangle=0, automargin=True)
fig

Which countries have the highest numbers of internally displaced persons?

Next, we'll explore the count of internally displaced persons (IDPs). An IDP is someone who has had to leave their home or usual residence because of reasons like armed conflict, violence, human rights violations, natural disasters, or humanitarian emergencies. Unlike refugees, IDPs haven't crossed international borders to find safety; they stay within their own country's borders



fig = (
    df.groupby(["coo_iso", "coo_name"])
    .idps.sum()
    .sort_values(ascending=False)
    .to_frame()
    .reset_index()
    .head(50)
    .pipe(
        lambda df: px.scatter_geo(
            df,
            locations="coo_iso",
            geojson=countries,
            hover_name="coo_name",
            # color='coo_name',
            featureidkey="properties.iso_a3",
            size="idps",
            # opacity = 0.6,
            width=1200,
            height=700,
        )
    )
)
fig

What countries have received the highest stateless populations?

Next, we focus on countries that have provided refuge to a substantial number of stateless individuals. Stateless persons are individuals who don't have the legal status of citizenship in any country. They lack the rights and protection typically granted to citizens. Stateless people often encounter significant difficulties in accessing education, healthcare, jobs, and the freedom to travel.



fig = (
    df.groupby(["coa_iso", "coa_name"])
    .stateless.sum()
    .pipe(lambda x: x / 49036122)
    .sort_values(ascending=False)
    .mul(100)
    .to_frame()
    .reset_index()
    .head(10)
    .pipe(
        lambda df: px.scatter(
            df,
            y="coa_name",
            x="stateless",
            labels={
                "stateless": "Acceptance Rate of Stateless Populations (%)",
                "coa_name": "",
            },
            height=500,
            width=800,
        )
    )
)
fig.update_yaxes(tickangle=0, automargin=True)
fig.update_layout(xaxis_range=[0, 100])

fig

And there you have it, everyone. That wraps up this week's data exploration. I'm cant't wait for what Tidy Tuesday has in store for us next week. Until then, take good care of yourselves, and I'll catch you soon!

Mastering Pandas DataFrame Styling for a Stunning Presentation

Adam Cseresznye — Sun, 27 Aug 2023 00:00:00 GMT

As you wrap up your data analysis journey, you face a fun decision: how to share your discoveries effectively. Tables and graphs both have their moments to shine. Tables work great when you want your audience to spot exact values and compare them. But, here's the twist: tables can look overwhelming at first, especially if they're crammed with data.

But don't worry! Styling and formatting are here to help. Pandas comes to the rescue. It lets you turn your data into stylish tables effortlessly.

In this article, we'll explore some tricks to make your Pandas DataFrames look awesome and tell a clear data story. For the demonstrations, we'll dive into Seaborn's built-in "tips" dataset. This dataset is a nifty data frame with 244 rows and 7 variables, offering a peek into the world of tipping. This comprehensive collection includes variables like the tip amount in dollars, the bill total in dollars, the gender of the bill payer, the presence of smokers in the party, the day of the week, the time of day, and the party size. Ready to roll? Let's jump right in!

Setup

import pandas as pd
import numpy as np


df = pd.read_csv(
    "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/tips.csv"
)
df.sample(5)

Built-in styling

In this article, we won't cover every single feature of the Styler class because that might make it too long. Instead, we'll concentrate on the practical aspects that can simplify your data analysis, improve data comprehension, and make your message clearer to your audience. So, instead of taking a deep dive, we'll start with the low-hanging fruits and explore the built-in options that Pandas offers.

Before we dive in, let's prepare an aggregated dataframe for our demonstrations. Here it is:


grouped = df.groupby(["day", "smoker", "time"]).median(numeric_only=True)
grouped

Highlighting minimum and maximum values

Check out these handy helpers: highlight_min and highlight_max. They're your go-to for spotting the lowest and highest values in each column. It's a speedy way to emphasize the most important data points in each category.

Also, please note the format(precision=1, thousands=",", decimal=".") snippet, this is not exactly a built-in style feature but has everything to do with keeping those float numbers tidy. Pandas tends to display more decimal places than we often require, which can be a bit distracting. To tone it down a notch, we can lean on the format() and format_index() methods to fine-tune the precision. Trust me, it's super useful!


(
    grouped.style.format(precision=1, thousands=",", decimal=".")
    .highlight_max(
        axis=0, props="color:white; font-weight:bold; background-color:green;"
    )
    .highlight_min(axis=0, props="color:white; font-weight:bold; background-color:red;")
)

Emphasizing Values Within a Range

Imagine you're want to find the days when tips fell between 3 and 5 dollars. In this scenario, the highlight_between method comes to the rescue. Don't forget to use the subset argument; it's your trusty sidekick when you only want to work with selected columns.


(
    grouped.reset_index()
    .style.format(precision=1, thousands=",", decimal=".")
    .highlight_between(
        left=3,
        right=5,
        subset=["tip"],
        axis=1,
        props="color:white; font-weight:bold; background-color:purple;",
    )
)

Highlight Column-Wise Outliers

Utilizing parameters such as q_left=0.05, q_right=0.95, axis=0, and defining props='opacity: 10%;', we can highlight values residing outside the 5-95 percentile range.


(
    grouped.style.format(precision=1, thousands=",", decimal=".").highlight_quantile(
        q_left=0.05, q_right=0.95, axis=0, props="opacity: 10%;"
    )
)

Spot trends Using Color Gradients

We can show data trends using both the background_gradient and text_gradient methods. These methods introduce gradient-style background colors and text shading to our visualizations. To optimize their impact, it's advisable to first arrange your data with the sort_values method before applying the background_gradient.


(
    grouped.reset_index()
    .sort_values(by="total_bill")
    .style.format(precision=1, thousands=",", decimal=".")
    .background_gradient(cmap="viridis", axis=0)
)


(
    grouped.reset_index()
    .sort_values(by="total_bill")
    .style.format(precision=1, thousands=",", decimal=".")
    .text_gradient(cmap="bwr", axis=0)
)

Display Bar Charts within Your Table

Let's explore a technique for highlighting the significance of values by embedding bar charts right within the cells. The blend of bar heights and color gradients can pack a powerful punch in your data storytelling arsenal. Don't forget to experiment with the 'align' option, a handy tool that helps you position these bars within the cells just right, giving your visuals a polished look. Feel free to play around with the settings and find what clicks best with your data tales.


(
    grouped.reset_index()
    .sort_values(by="total_bill")
    .style.format(precision=1, thousands=",", decimal=".")
    .bar(
        align="mean",
        cmap="bwr",
        height=50,
        width=60,
        props="width: 120px; border-right: 1px solid black;",
    )
)


(
    grouped.reset_index()
    .sort_values(by="total_bill")
    .style.format(precision=1, thousands=",", decimal=".")
    .bar(
        align=0,
        cmap="bwr",
        height=50,
        width=60,
        props="width: 120px; border-right: 1px solid black;",
    )
)

Formatting

Below is an illustrative example of the myriad possibilities when it comes to fine-tuning the style of a DataFrame. In this code, we've undertaken various stylistic enhancements:

Background Gradient: We've applied a nice background color gradient to the columns along the vertical axis.
Caption: We've set a descriptive caption for the DataFrame.
Visual Styling: We've specified text alignment and introduced hovering options to make highlighted values pop even more.
Column Names Formatting: We've reformatted our column names for clarity and aesthetics.
Column Hiding: We've concealed the 'smoker' column, decluttering our view.
Index Hiding: We've also hidden the index labels for a cleaner look.
Numerical Formatting: We've individually specified the numerical representation, including adding a dollar sign for currency values.

This demonstration underscores the virtually endless possibilities for customizing the appearance of your DataFrame. However, it's important to note that the physical attributes set by the set_table_styles method won't be exported to Excel, should you choose to do so. Just a handy tidbit to keep in mind.



# Start by resetting the index and renaming columns with underscores
# Replace underscores with spaces for better readability
(
    grouped.reset_index().rename(columns=lambda x: x.replace("_", " "))
    # Sort the DataFrame by the 'total bill' column
    .sort_values(by="total bill")
    # Apply Pandas Styler to format the table
    .style
    # Apply background color gradient to columns along the vertical axis (axis=0)
    .background_gradient(cmap="viridis", axis=0)
    # Set a caption for the table
    .set_caption("Exploring Dining Trends: Bill Amounts, Tips, and Party Sizes")
    # Customize the table's visual styling
    .set_table_styles(
        [
            {
                "selector": "th.col_heading",
                "props": "text-align: center; font-size: 1.5em;",
            },
            {"selector": "td", "props": "text-align: center;"},
            {
                "selector": "td:hover",
                "props": "font-style: italic; color: black; font-weight:bold; background-color : #ffffb3;",
            },
        ],
        overwrite=False,
    )
    # Apply custom formatting to the index labels (convert to uppercase)
    .format_index(str.upper, axis=1)
    # Hide the 'smoker' column from the table
    .hide(subset=["smoker"], axis=1)
    # Hide the index label (row numbers)
    .hide(axis="index")
    # Format specific columns with dollar signs and one decimal place
    .format(
        {
            "total bill": "$ {:.1f}",
            "tip": "$ {:.1f}",
            "size": "{:.0f}",
        }
    )
)

Defining Custom Styling

If the predefined styling methods don't meet your needs, you have the flexibility to create your custom styling function and use it with your DataFrame. You can apply styling element by element using the .applymap() method, or you can work on columns or rows using the .apply() method. If you need to work specifically on column header rows or indexes, both .apply_index() and .applymap_index() are at your disposal.


# Define a custom styling function that highlights 'Sun' in yellow
def highlight_sunday(value, color):
    return f"color: {color};" if value == "Sun" else None


# Sample 10 random rows from the DataFrame
(
    df.sample(10).style
    # Set formatting options for numeric values
    .format(precision=1, thousands=",", decimal=".")
    # Apply the highlight_sunday function to the 'day' column and set the color to yellow
    .applymap(highlight_sunday, color="yellow", subset=["day"])
    # Apply opacity styling to cells in the 'size' column where the value is less than or equal to 3
    .applymap(lambda value: "opacity: 50%;" if (value <= 3) else None, subset=["size"])
    # Apply red text color styling to cells in the 'tip' column where the value is less than or equal to 3
    .applymap(lambda value: "color:red;" if (value <= 3) else None, subset=["tip"])
    # Use apply_index to style index labels. If the label is 'tip', 'day', or 'size', set the color to white, otherwise black.
    .apply_index(
        lambda s: np.where(
            s.isin(["tip", "day", "size"]), "opacity: 100%;", "opacity: 10%;"
        ),
        axis=1,
    )
)

Other Useful Features

Get a Clearer Overview with `set_sticky`

What if you find yourself facing a DataFrame with more columns than can comfortably fit on your screen, yet you still wish to inspect each column individually? In the past, you might have resorted to using pd.set_option('display.max_columns', xyz) to expand the display. However, there's a much more elegant solution: set_sticky.

set_sticky introduces a clever CSS trick that permanently pins the index or column headers within a scrolling frame. In our case, although the 'tips' DataFrame doesn't have an excessive number of columns, we've concatenated 10 DataFrames together to showcase the remarkable utility of set_sticky. As you scroll horizontally, you'll notice that you can now conveniently inspect all the columns while the index remains firmly in place, thanks to the magic of set_sticky. Let's explore this feature below.


(pd.concat([df for i in range(10)], axis=1).head().style.set_sticky(axis="index"))

Concatenating DataFrame Outputs

You can combine two or more Stylers if they have the same columns. This is particularly handy when presenting summary statistics for a DataFrame


(
    df.groupby("day")[["total_bill", "tip", "size"]]
    .mean()
    .style.format(precision=1)
    .concat(
        df[["total_bill", "tip", "size"]]
        .agg(["mean", "median", "sum"])
        .style.format(precision=1)
        .relabel_index(["Average", "Median", "Sum"])
    )
)

Exporting

To save your styled DataFrame in Excel format, you can export it to an .xlsx file, preserving the styling. To do this, ensure you have the 'openpyxl' package installed.

And there you have it, folks! This concludes our brief exploration of the formatting options available right within Pandas. I hope you found this article insightful and enjoyable. Until next time, happy coding!

Tidy Tuesday: Union Membership in the United States

Adam Cseresznye — Tue, 05 Sep 2023 00:00:00 GMT

Happy Labor Day! 🛠️ As we celebrate the achievements of workers and the contributions they've made to society, what better way to delve into the world of labor and employment than with a fresh Tidy Tuesday dataset? This week's data comes from the comprehensive "Union Membership, Coverage, and Earnings from the CPS" dataset, courtesy of Barry Hirsch from Georgia State University, David Macpherson from Trinity University, and William Even from Miami University.

This dataset is a rich source of information, shedding light on the intricate dynamics of union membership, coverage, and earnings. So, grab your coffee, get comfortable, and let's embark on this Tidy Tuesday journey together, exploring the intricate tapestry of labor data!

Import data

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats
import geopandas
from lets_plot.geo_data import *

from lets_plot import *
from lets_plot.mapping import as_discrete

LetsPlot.setup_html()

There are three distinct datasets to investigate: one focusing on demographic information, another centered on wage data, and the third associated with various states. Let's read them all in.

demographic_df = pd.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-09-05/demographics.csv"
)
wages_df = pd.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-09-05/wages.csv"
)
states_df = pd.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-09-05/states.csv"
)

Data dictionaries

Here are the corresponding data dictionaries:

Demographics data: Data sources:

1973-1981: May Current Population Survey (CPS)
1982: No union questions available
1983-2022: CPS Outgoing Rotation Group (ORG) Earnings Files

The definition of union membership was expanded in 1977 to include "employee associations similar to a union".

variable	class	description
year	double	When the data was collected.
sample_size	double	The number of wage and salary workers ages 16 and over who were surveyed.
employment	double	Wage and salary employment in thousands.
members	double	Employed workers who are union members in thousands.
covered	double	Workers covered by a collective bargaining agreement in thousands.
p_members	double	Percent of employed workers who are union members.
p_covered	double	Percent of employed workers who are covered by a collective bargaining agreement.
facet	character	The sector or demographic group contained in this row of data.

Wages data: Data sources:

1973-1981: May Current Population Survey (CPS)
1982: No union questions available
1983-2022: CPS Outgoing Rotation Group (ORG) Earnings Files

The definition of union membership was expanded in 1977 to include "employee associations similar to a union".

variable	class	description
year	double	When the data was collected.
sample_size	double	The number of wage and salary workers ages 16 and over who were surveyed and provided earnings and hours worked information.
wage	double	Mean hourly earnings in nominal dollars.
at_cap	double	Percent of workers with weekly earnings at the top code of $999 through 1988, $1923 in 1989-97, and $2885 beginning in 1998, with individuals assigned mean earnings above the cap based on annual estimates of the gender-specific Pareto distribution.
union_wage	double	Mean wage among union members.
nonunion_wage	double	Mean wage among nonunion workers.
union_wage_premium_raw	double	The percentage difference between the union and nonunion wage.
union_wage_premium_adjusted	double	Estimated as exp(b)-1 where b is the regression coefficient on a union membership variable (equal to 1 if union and 0 otherwise) from a semi-logarithmic wage equation, with controls included for worker/job characteristics. Included in the all-worker wage equation are the control variables: years of schooling, potential years of experience [proxied by age minus years of schooling minus 6] and its square [both interacted with gender], and categorical variables for marital status, race and ethnicity, gender, part-time, large metropolitan area, state, public sector, broad industry, and broad occupation. Controls are omitted, as appropriate, for estimates within sectors or by demographic group [i.e., by class, gender, race, or industry sector]. Workers who do not report earnings but instead have them imputed [i.e., assigned] by the Census are removed from the estimation samples in all years, except 1994 and 1995 when imputed earners cannot be identified. Inclusion of imputed earners causes union wages to be understated, nonunion wages overstated, and union-nonunion wage differences understated. For 1994-95, the sample includes imputed earners and estimates in those years have been adjusted to remove the bias from imputation.
facet	character	The sector or demographic group contained in this row of data.

States data:

Data source: Current Population Survey (CPS) Outgoing Rotation Group (ORG) Earnings Files

variable	class	description
state_census_code	double	Census state code used in CPS
state	character	State name.
sector	character	Employment sector.
observations	double	CPS sample size.
employment	double	Wage and salary employment in thousands.
members	double	Employed workers who are union members in thousands.
covered	double	Workers covered by a collective bargaining agreement in thousands.
p_members	double	Percent of employed workers who are union members.
p_covered	double	Percent of employed workers who are covered by a collective bargaining agreement.
state_abbreviation	character	State abbreviation.
year	double	Year of the survey.

As in our previous Tidy Tuesday blog, I believe we can address the following questions using this dataset:

What are the overarching trends in the labor force, particularly regarding union memberships?
Do specific demographic groups or occupations display a higher likelihood of union membership?
Do union members experience any financial advantages or benefits compared to non-union workers?
Which states have the highest number of union members or affiliated unions?

What are the overarching trends in the labor force, particularly regarding union memberships?

The figure below illustrates a consistent upward trajectory in the workforce's growth, whereas union membership and the number of individuals covered by unions have been on a declining trend over the decades.


(
    demographic_df.query("facet.str.contains('all wage')")
    .drop(columns=["sample_size", "facet", "p_members", "p_covered"])
    .melt(id_vars="year", var_name="Description")
    .pipe(
        lambda df: ggplot(df)
        + geom_line(aes("year", "value", color="Description"), size=1.5)
        + labs(
            title="Evolution of Workforce Size and Union Membership Across the Years",
            subtitle="While employment numbers have been on the rise, the proportion of employed workers \nwho are union members and covered by unions has been steadily decreasing.",
            x="",
            y="Number of people in thousands",
            caption="© 2023 by Barry T. Hirsch, David A. Macpherson, and William E. Even.",
        )
        + theme(
            plot_subtitle=element_text(size=12, face="italic"),
            plot_title=element_text(size=15, face="bold"),
        )
    )
)

Do specific demographic groups or occupations display a higher likelihood of union membership?

Next, our investigation will focus on identifying potential demographic disparities among workforce members and uncovering the professions with the highest likelihood of union membership. We'll utilize the "p_members" column, which represents the percentage of employed workers who are union members, to delve into these aspects.

As evident in @fig-fig2, a substantial decrease in union membership is observable across nearly all demographic groups over the decades.


(
    demographic_df.query("facet.str.contains('demographics')")
    .assign(facet=lambda df: df.facet.replace({"demographics:": ""}, regex=True))
    .pipe(
        lambda df: ggplot(df, aes("year", "p_members", color="facet"))
        + geom_line(
            size=1.5,
        )
        + labs(
            title="Demographic Trends in Union Membership as a Percentage of the Total Workforce Over Time",
            subtitle="Almost all demographic groups exhibit a significant decline in union membership over the decades.",
            x="",
            y="Percent of employed workers who are union members",
            caption="© 2023 by Barry T. Hirsch, David A. Macpherson, and William E. Even.",
        )
        + theme(
            plot_subtitle=element_text(size=12, face="italic"),
            plot_title=element_text(size=15, face="bold"),
        )
        + ggsize(800, 500)
    )
)

While certain professions exhibit declining trends in union memberships, it's noteworthy that some public sector occupations, such as postal service, police, and local government jobs, maintain the highest and most consistent levels of union participation (@fig-fig3).


(
    demographic_df.query(
        "(~facet.str.contains('demographics')) and (~facet.str.contains('all'))"
    ).pipe(
        lambda df: ggplot(df, aes("year", "p_members", color="facet"))
        + geom_line(
            size=1.5,
        )
        + labs(
            title="Workforce Sector-Specific Trends in Union Membership",
            subtitle="Public sector occupations demonstrate higher union participation rates.",
            x="",
            y="Percent of employed workers who are union members",
            caption="© 2023 by Barry T. Hirsch, David A. Macpherson, and William E. Even.",
        )
        + theme(
            plot_subtitle=element_text(size=12, face="italic"),
            plot_title=element_text(size=15, face="bold"),
        )
        + ggsize(800, 500)
    )
)

Do union members experience any financial advantages or benefits compared to non-union workers?

The next figure is both straightforward and highly impactful, providing a direct comparison of average wages for union members versus non-union members.


(
    wages_df.query("facet == 'all wage and salary workers'")
    .drop(
        columns=[
            "wage",
            "sample_size",
            "at_cap",
            "union_wage_premium_raw",
            "union_wage_premium_adjusted",
            "facet",
        ]
    )
    .melt(id_vars="year", var_name="Description")
    .pipe(
        lambda df: ggplot(df, aes("year", "value", color="Description"))
        + geom_line(
            size=1.5,
        )
        + labs(
            title="Differences in Mean Hourly Earnings for Union and Non-Union Workers",
            subtitle="Union workers consistently display higher mean wages in comparison to non-union members.",
            x="",
            y="Mean hourly earnings in nominal dollars",
            caption="© 2023 by Barry T. Hirsch, David A. Macpherson, and William E. Even.",
        )
        + theme(
            plot_subtitle=element_text(size=12, face="italic"),
            plot_title=element_text(size=15, face="bold"),
        )
    )
)

As with our previous figures, we will now delve into demographic variations and job sector disparities. To unveil these potential distinctions, we will use the "union_wage_premium_raw" column, which indicates the percentage difference between union and non-union wages.

The most significant disparities in hourly wages (@fig-fig5) between unionized and non-unionized workers are observed within the construction sector. These disparities have fluctuated over the decades, ranging from below 40% to as high as 80%. It's important to note that almost all sectors exhibit higher earnings for union members, except for federal, manufacturing, and wholesale/retail workers (particularly in recent times).


(
    wages_df.query(
        "(~facet.str.contains('demographics')) and (~facet.str.contains('all'))"
    ).pipe(
        lambda df: ggplot(df, aes("year", "union_wage_premium_raw", color="facet"))
        + geom_line(
            size=1.5,
        )
        + labs(
            title="Sector-specific Differences in Mean Hourly Earnings for Union and Non-Union Workers",
            subtitle="The disparities between wages range from -20% to +80%.",
            x="",
            y="The percentage difference between the union and nonunion wage",
            caption="© 2023 by Barry T. Hirsch, David A. Macpherson, and William E. Even.",
        )
        + theme(
            plot_subtitle=element_text(size=12, face="italic"),
            plot_title=element_text(size=15, face="bold"),
        )
        + ggsize(800, 500)
        + geom_hline(yintercept=0, linetype=5, color="#6c6c6c", size=1)
    )
)

In addition to gender-specific differences, it's intriguing to observe the relationship between education levels and wage disparities between unionized and non-unionized workers. Workers with education levels below college tend to benefit more from union membership, whereas, in general, individuals with higher education levels tend to have lower wages when affiliated with a union.


(
    wages_df.query("facet.str.contains('demographics')")
    .assign(facet=lambda df: df.facet.replace({"demographics:": ""}, regex=True))
    .pipe(
        lambda df: ggplot(df, aes("year", "union_wage_premium_raw", color="facet"))
        + geom_line(
            size=1.5,
        )
        + labs(
            title="Demographics-specific Differences in Mean Hourly Earnings for Union and Non-Union Workers",
            subtitle="It appears that females tend to benefit more from union membership in comparison to males, \nparticularly in terms of the percentage difference between union and non-union wages.",
            x="",
            y="The percentage difference between the union and nonunion wage",
            caption="© 2023 by Barry T. Hirsch, David A. Macpherson, and William E. Even.",
        )
        + theme(
            plot_subtitle=element_text(size=12, face="italic"),
            plot_title=element_text(size=15, face="bold"),
        )
        + ggsize(800, 500)
        + geom_hline(yintercept=0, linetype=5, color="#6c6c6c", size=1)
    )
)

Which states have the highest number of union members or affiliated unions?

To illustrate variations in union membership across states, we will leverage Lets Plot's capabilities for visualizing spatial data. Given the extensive dataset, our focus will be on highlighting a subset of the data, specifically the most recent trends in 2022 across all sectors.

As evident in @fig-fig7, certain states like New York, Alaska, Hawaii, Washington, California, and Oregon have a significant portion of their workforce as union members. Conversely, states like Utah, South Dakota, and the Carolinas display lower percentages of union membership among their workforce.


filtered_data = states_df.query("sector == 'Total' and year == 2022")
boundaries = (
    geocode_states(filtered_data["state"])
    .scope("USA")
    .inc_res(4)
    .get_boundaries(resolution=4)
)

(
    ggplot()
    + geom_livemap()
    + geom_polygon(
        aes(color="p_members", fill="p_members"),
        data=states_df.query("sector == 'Total' and year == 2022"),
        map=boundaries,
        alpha=0.9,
        map_join=[["state"], ["state"]],
        show_legend=False,
        size=0.01,
    )
    + theme(
        axis_title="blank", axis_text="blank", axis_ticks="blank", axis_line="blank"
    )
    + scale_fill_brewer(palette="Greens")
    + ggsize(800, 500)
)

There you have it! I hope you found this week's Tidy Tuesday analysis insightful. I would encourage everyone to dive deeper into this dataset as I've only scratched the surface; there's a wealth of knowledge waiting to be uncovered here.

Happy coding, and until next time, see you in our next exploration!

Tidy Tuesday: The Global Human Day

Adam Cseresznye — Mon, 11 Sep 2023 00:00:00 GMT

This week's Tidy Tuesday takes us to The Human Chronome Project, an initiative based at McGill University in Montreal. We're exploring insights from their paper titled "The global human day in PNAS" and the accompanying dataset on Zenodo.

The publication offers a broad view of our species' activities, shedding light on how economic activities fit into the grand scheme of life. It also highlights activities with significant potential for change. For a deeper dive into our research methodology and to explore supplementary visualizations, I encourage you to peruse the supplementary materials provided with the publication.

Import data

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats
import statsmodels.formula.api as smf
import statsmodels.api as sm
from lets_plot.bistro.qq import qq_plot

import flag
from PIL import Image
from pathlib import Path

from lets_plot import *
from lets_plot.mapping import as_discrete

LetsPlot.setup_html()

While we have four distinct datasets at our disposal, our primary focus will be on two of them. The 'all_countries' dataset holds comprehensive information about the time allocated to each activity per region and country, while the 'country_regions' dataset provides the essential correspondence between country names and their ISO-2 and ISO-3 abbreviations. Let's proceed by reading them all in.

all_countries = pd.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-09-12/all_countries.csv"
)
country_regions = pd.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-09-12/country_regions.csv"
)
global_human_day = pd.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-09-12/global_human_day.csv"
)
global_economic_activity = pd.read_csv(
    "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-09-12/global_economic_activity.csv"
)

Data dictionaries

Here are the corresponding data dictionaries:

Data Dictionary

all_countries.csv

variable	class	description
Category	character	M24 categories
Subcategory	character	M24 subcategories
country_iso3	character	Country code in iso3
region_code	character	Region code
population	double	Population
hoursPerDayCombined	double	Hours per day combined for the country
uncertaintyCombined	double	Uncertainty combined. Uncertainty is in units variance.

country_regions.csv

variable	class	description
region_code	character	Region code
region_name	character	Region name
country_name	character	Country name
M49_code	double	M49 code
country_iso2	character	Country code in iso2
country_iso3	character	Country code in iso3
alt_country_name	character	Alternative country name
alt_country_name1	character	Alternative country name 1
alt_country_name2	character	Alternative country name 2
alt_country_name3	character	Alternative country name 3
alt_country_name4	character	Alternative country name 4
alt_country_name5	character	Alternative country name 5
alt_country_name6	character	Alternative country name 6
other_code1	character	Other country code 1
other_code2	character	Other country code 2

global_human_day.csv

variable	class	description
Subcategory	character	M24 subcategory
hoursPerDay	double	Hours per day for all countries
uncertainty	double	Uncertainty in units variance.

global_economic_activity.csv

variable	class	description
Subcategory	character	M24 subcategory
hoursPerDay	double	Hours per day for all countries.
uncertainty	double	Uncertainty in units variance.

As in our previous Tidy Tuesday blog, I think we can address the following questions using this dataset:

Which regions allocate the most and least time to each subcategory?
Which countries exhibit the highest and lowest time allocation for each subcategory?
Is there a discernible relationship between time spent on each category and GDP per capita?

Get GDP per capita data from Wikipedia

To prepare for our third question, let's grab GDP per capita data from Wikipedia. It will come in handy as we dive deeper into our analysis!

# Define a dictionary for country name replacements
names_to_replace = {
    "Russia": "Russian Federation",
    "South Korea": "Korea, Republic of",
    "Vietnam": "Viet Nam",
    "Hong Kong": "Hong Kong, China",
    "Iran": "Iran, Islamic Republic of",
    "Venezuela": "Venezuela, Bolivarian Republic of",
    "Tanzania": "Tanzania, United Republic of",
    "DR Congo": "Congo",
    "Palestine": "Occupied Palestinian Territory",
    "Brunei": "Brunei Darussalam",
    "North Korea": "Korea Democratic People's Republic of",
    "Moldova": "Moldova, Republic of",
    "Syria": "Syrian Arab Republic",
    "North Macedonia": "Macedonia the former Yugoslav Republic of",
    "Laos": "Lao People's Democratic Republic",
    "Eswatini": "Swaziland",
    "Curaçao": "Curacao",
    "East Timor": "Timor-Leste",
    "Saint Vincent and the Grenadines": "Saint Vincent and Grenadines",
    "Sint Maarten": "Sint Maarten (Kingdom of the Netherlands)",
    "São Tomé and Príncipe": "Sao Tome and Principe",
    "Micronesia": "Micronesia Federated States of",
    "Macau": "Macau, China",
}

# Read GDP data from Wikipedia
gdp = (
    pd.read_html("https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)")[2]
    .pipe(lambda s: s.set_axis(s.columns.map("_".join), axis=1))
    .loc[
        :,
        [
            "Country/Territory_Country/Territory",
            "UN region_UN region",
            "United Nations[15]_Estimate",
        ],
    ]
    .rename(
        columns={
            "Country/Territory_Country/Territory": "country",
            "UN region_UN region": "UN_region",
            "United Nations[15]_Estimate": "gdp",
        }
    )
    .assign(country=lambda df: df.country.replace(names_to_replace))
    .assign(
        gdp=lambda df: df.gdp.replace(r"\D+", "", regex=True),
        iso3=lambda df: df.country.map(
            country_regions.loc[:, ["alt_country_name", "country_iso3"]]
            .set_index("alt_country_name")
            .squeeze()
            .to_dict()
        ),
    )
    .query("~iso3.isna()")
)
gdp

After obtaining the GDP per capita data, we'll integrate it into our main dataset.

df = (
    all_countries.assign(
        HPD_with_u=lambda df: df.hoursPerDayCombined + df.uncertaintyCombined
    )
    .assign(
        gdp=lambda df: df.country_iso3.map(gdp.set_index("iso3").gdp.to_dict()).astype(
            float
        )
    )
    .query("~gdp.isna()")
    .assign(
        gdp_per_population=lambda df: df.gdp / df.population.div(1_000_000),
        gdp_per_population_log=lambda df: np.log10(df.gdp_per_population),
        Subcategory=lambda df: df.Subcategory.str.replace(" ", "_"),
    )
)

df

Which regions allocate the most and least time to each subcategory?

First, to provide an overview of the prevailing trends, we will begin by examining disparities in time allocation across various subcategories within different regions. Following this, we will delve into a more granular analysis (as part of our second question), considering the data at the country level. As depicted in @fig-fig1, a noticeable pattern emerges. Typically, the minimum and maximum values for each activity type appear quite similar. However, it gets intriguing when we focus on certain categories like 'Social,' 'Food Growth & Collection,' 'Schooling & Research,' 'Sleep & Bedrest,' and 'Passive.' In these categories, we observe more significant differences, with gaps ranging from 1.05 to 1.59 hours between the maximum and minimum hours spent.


maximum_values = (
    df.groupby(["region_code", "Subcategory"])
    .mean(numeric_only=True)
    .reset_index()
    .groupby(["Subcategory"])
    .apply(lambda x: x.sort_values(ascending=False, by="hoursPerDayCombined").head(1))
    .reset_index(drop=True)[["region_code", "Subcategory", "hoursPerDayCombined"]]
)

minimum_values = (
    df.groupby(["region_code", "Subcategory"])
    .mean(numeric_only=True)
    .reset_index()
    .groupby(["Subcategory"])
    .apply(lambda x: x.sort_values(ascending=True, by="hoursPerDayCombined").head(1))
    .reset_index(drop=True)[["region_code", "Subcategory", "hoursPerDayCombined"]]
)

# Concatenate the maximum and minimum records
concatenated_records = pd.concat([maximum_values, minimum_values], axis=0).reset_index(
    drop=True
)

# Assign flags, country names, and format subcategory names
(
    concatenated_records.assign(
        Subcategory=lambda df: df.Subcategory.str.replace("_", " "),
        group=lambda df: df.region_code.mask(
            df.region_code.str.contains("EU"), True
        ).mask(~df.region_code.str.contains("EU"), False),
    ).pipe(
        lambda df: ggplot(
            df,
            aes(
                "hoursPerDayCombined",
                as_discrete("Subcategory", order=1, order_by="hoursPerDayCombined"),
            ),
        )
        + geom_point(aes(color="group"), size=5, alpha=0.5, show_legend=False)
        + geom_text(
            aes(label="region_code"),
            position=position_nudge(y=0.25, x=-0),
            size=5,
            na_text="",
            angle=15,
            color="#2b8cbe",
            tooltips=layer_tooltips(
                ["Subcategory", "region_code", "hoursPerDayCombined"]
            ),
        )
        + labs(
            title="Regions with the Most and Least Time Spent on Each Subcategory",  # Title
            subtitle="""Abbreviations: EU = Europe, AF = Africa, AS = Asia, AM = Americas, ANZ = Australia and New Zealand
            E = East, W = West, N = North, S = South, C = Central, M = Middle
            Values from Europe are highlighted in green.
            """,  # Subtitle
            x="Mean Hours per day combined per Region",
            y="",  # Y-axis label
            caption="doi.org/10.1073/pnas.2219564120",  # Caption with the source DOI
        )
        + theme(
            plot_subtitle=element_text(
                size=12, face="italic"
            ),  # Customize subtitle appearance
            plot_title=element_text(size=15, face="bold"),  # Customize title appearance
        )
        + scale_x_continuous(
            trans="log10", limits=[0.001, 24]
        )  # Set limits for the x-axis
        + ggsize(800, 600)  # Set the plot size
    )
)

Which countries allocate the most and least time to each subcategory?

As depicted in @fig-fig2 below, on average, the most significant time allocation is dedicated to Sleep and Bedrest, followed by Passive, Interactive, and Social Activities, as well as Activities Centered on Eating and Drinking, including Associated Socializing. While drawing definitive conclusions from this figure may be challenging, Japan stands out as a unique case. They allocate the least amount of time to Sleep and Bedrest while devoting the most time to Passive Activities and Hygiene and Grooming—an interesting observation.


# Calculate the maximum values for each subcategory
maximum_values = (
    df.groupby(["Subcategory"])
    .apply(lambda x: x.sort_values(ascending=False, by="hoursPerDayCombined").head(1))[
        ["country_iso3", "Subcategory", "hoursPerDayCombined"]
    ]
    .reset_index(drop=True)
    .assign(
        iso2=lambda df: df.country_iso3.map(
            country_regions.set_index("country_iso3").country_iso2.squeeze().to_dict()
        )
    )
)

# Calculate the minimum values for each subcategory
minimum_values = (
    df.groupby(["Subcategory"])
    .apply(lambda x: x.sort_values(ascending=True, by="hoursPerDayCombined").head(1))[
        ["country_iso3", "Subcategory", "hoursPerDayCombined"]
    ]
    .reset_index(drop=True)
    .assign(
        iso2=lambda df: df.country_iso3.map(
            country_regions.set_index("country_iso3").country_iso2.squeeze().to_dict()
        )
    )
)

# Concatenate the maximum and minimum records
concatenated_records = pd.concat([maximum_values, minimum_values], axis=0).reset_index(
    drop=True
)

# Assign flags, country names, and format subcategory names
(
    concatenated_records.assign(
        flag=lambda df: df.iso2.apply(flag.flag),
        name=lambda df: df.country_iso3.map(
            country_regions.set_index("country_iso3").country_name.squeeze().to_dict()
        ),
        Subcategory=lambda df: df.Subcategory.str.replace("_", " "),
    ).pipe(
        lambda df: ggplot(
            df,
            aes(
                "hoursPerDayCombined",
                as_discrete("Subcategory", order=1, order_by="hoursPerDayCombined"),
            ),
        )
        + geom_point(size=5, alpha=0.25)
        + geom_text(
            aes(label="flag"),
            position=position_nudge(y=0.5, x=-0),
            size=8,
            na_text="",
            angle=15,
            color="#2b8cbe",
            tooltips=layer_tooltips(["Subcategory", "name", "hoursPerDayCombined"]),
        )
        + labs(
            title="Countries with the Most and Least Time Spent on Each Subcategory",  # Title
            subtitle="""On average, the highest time allocation is for Sleep and Bedrest, followed by Passive, Interactive, and 
            Social Activities, and Activities Centered on Eating and Drinking, including Associated Socializing""",  # Subtitle
            x="Hours per day combined for the country",
            y="",  # Y-axis label
            caption="doi.org/10.1073/pnas.2219564120",  # Caption with the source DOI
        )
        + theme(
            plot_subtitle=element_text(
                size=12, face="italic"
            ),  # Customize subtitle appearance
            plot_title=element_text(size=15, face="bold"),  # Customize title appearance
        )
        + scale_x_continuous(limits=[0, 11])  # Set limits for the x-axis
        + ggsize(800, 600)  # Set the plot size
    )
)

Is there a discernible relationship between time spent on each category and GDP per capita?

Now, let's make the most of our 'gdp_per_population_log' column. Please note that I'm not an economics expert, and our goal here is to identify intriguing trends without delving too deeply into interpretation. For a comprehensive understanding of the dataset, I recommend referring to the original publication.

Let's begin by examining the broader trends. We aim to identify activities that appear to exhibit a noteworthy correlation with GDP per population, bearing in mind that GDP per population is log-transformed for our analysis. As evident from @fig-fig3, we've come across some intriguing, and perhaps not entirely surprising, observations. For instance, the data hints at a negative relationship between the hours per day spent on schooling and research and GDP per capita. This finding might raise eyebrows, as conventional wisdom often suggests that higher education leads to increased economic output. However, it's worth considering that spending too much time in the educational system could potentially limit one's participation in economic production compared to individuals with lower levels of education. It's a thought-provoking discovery, and while it may challenge some assumptions, it certainly adds depth to our understanding of the data.


(
    df.assign(Subcategory=lambda df: df.Subcategory.str.replace("_", " ")).pipe(
        lambda df: ggplot(df, aes("hoursPerDayCombined", "gdp_per_population_log"))
        + geom_point(alpha=0.5)
        + geom_smooth()
        + facet_wrap("Subcategory", scales="free")
        + labs(
            title="Relationship between Hours per Day Combined and Log(GDP per Population)",  # Title
            subtitle="""Activities vs. GDP per Capita at the country scale. Each circle on the chart 
            represents the population-average time spent per day for an individual country.""",
            x="Hours per day combined",
            y="log(GDP per population)",  # Y-axis label
            caption="doi.org/10.1073/pnas.2219564120",  # Caption with the source DOI
        )
        + theme(
            plot_subtitle=element_text(
                size=12, face="italic"
            ),  # Customize subtitle appearance
            plot_title=element_text(size=15, face="bold"),  # Customize title appearance
        )
        + ggsize(800, 800)  # Set the plot size
    )
)

To further explore the connection between 'gdp_per_population_log' and 'hoursPerDayCombined,' we can utilize the statsmodels library to execute an Ordinary Least Squares (OLS) regression analysis. This method enables us to compute the β coefficients, in addition to establishing upper and lower confidence intervals, as well as deriving the associated p-values. To streamline this process, we'll simplify our workflow by creating a function called get_linear_regression_table. This function will initially filter the data based on the unique subcategory of interest and subsequently apply the formula: {outcome_variable} ~ {independent_variable}.

def get_linear_regression_table(
    data, outcome_variable, independent_variable, category, covariate
):
    """
    Calculate linear regression coefficients, confidence intervals, and p-values for each unique category or subcategory
    within the specified category column.

    Parameters:
    - data (pandas.DataFrame): The input DataFrame containing the data.
    - outcome_variable (str): The name of the dependent variable.
    - independent_variable (str): The name of the independent variable.
    - category (str): The column name specifying the category or subcategory for grouping.
                      Should be either 'Category' or 'Subcategory'.
    - covariate (str or None): The name of the covariate variable to include in the regression model.
                               Set to None if no covariate is needed.

    Returns:
    - results (pandas.DataFrame): A DataFrame containing the results of linear regression analysis for each unique
                                  category or subcategory. Columns include:
      - 'unique_subcategory': The unique category or subcategory.
      - 'β': The coefficient for the independent_variable in the regression model.
      - 'Lower CI': The lower confidence interval for the coefficient.
      - 'Upper CI': The upper confidence interval for the coefficient.
      - 'p_value': The p-value for the coefficient.

    Example usage:
    >>> results = get_linear_regression_table(data, 'OutcomeVar', 'IndependentVar', 'Category', 'CovariateVar')
    """
    results_list = []

    if category == "Subcategory":
        cat = "Subcategory"
        unique_categories = data[cat].unique().tolist()
    else:
        cat = "Category"
        unique_categories = data[cat].unique().tolist()

    for unique_category in unique_categories:
        temp_df = data.loc[lambda df: df[cat] == unique_category, :]

        if covariate is not None:
            # Fit the linear regression model with covariate
            reg = smf.ols(
                f"{outcome_variable} ~ {covariate} + {independent_variable}",
                data=temp_df,
            ).fit(disp=0)
        else:
            # Fit the linear regression model without covariate
            reg = smf.ols(
                f"{outcome_variable} ~ {independent_variable}", data=temp_df
            ).fit(disp=0)

        # Get the regression coefficients, confidence intervals, and p-values
        beta = reg.params[-1]
        lower_CI = reg.conf_int()[0][-1]
        upper_CI = reg.conf_int()[1][-1]
        pvalue = reg.pvalues[-1]

        # Append the results to the list as a dictionary
        results_list.append(
            {
                "unique_subcategory": unique_category,
                "β": beta,
                "Lower CI": lower_CI,
                "Upper CI": upper_CI,
                "p_value": pvalue,
            }
        )

    # Convert the list of dictionaries into a DataFrame
    results = pd.DataFrame(results_list)
    return results

According to our linear regression analysis, it appears that activities like Materials (involving the extraction of substances for various purposes), Food Processing (which includes food transformation and preservation), and Food Growth & Collection (encompassing activities related to edible organic matter) exhibit the most pronounced negative relationships with GDP per capita. Specifically, the β coefficients for these activities are -6.21, -2.86, and -1.03, respectively, with all associated p-values falling below the 0.05 threshold.
Conversely, we observe the most robust positive relationships with GDP per capita in the case of Active Recreation (involving activities with elevated metabolic activity), Infrastructure (associated with the construction and maintenance of transportation and communication structures), and Waste Management (pertaining to the handling of waste and by-products outside of inhabited areas). These activities exhibit β coefficients of 2.1, 12.17, and 25.17, respectively, with all corresponding p-values falling below the 0.05 significance level.

(
    get_linear_regression_table(
        df,
        outcome_variable="gdp_per_population_log",
        independent_variable="hoursPerDayCombined",
        category="Subcategory",
        covariate=None,
    )
    .assign(unique_subcategory=lambda df: df.unique_subcategory.str.replace("_", " "))
    .sort_values(["β", "p_value"])
    .style.format(precision=3, thousands=",", decimal=".")
    .applymap(
        lambda value: "color: red; font-weight:bold" if (value <= 0.05) else None, subset=["p_value"]
    )
    .background_gradient(cmap="viridis", axis=0, subset="β")
)

Although the existing analysis provides valuable insights, we can enhance our understanding further by building a model that considers both regions and different subcategories simultaneously. To achieve this, we'll re-run our regression model with 'gdp_per_population_log' as the outcome variable and 'hoursPerDayCombined' as the independent variable. The 'region_code' will serve as a categorical variable, with Western Europe set as the reference level. Additionally, 'Subcategory' will also be treated as a categorical variable, with 'Passive Activities' chosen as the reference level. This approach will help us determine whether the hours spent on various activities exhibit any discernible relationship with GDP per capita.

As evident from the table below, none of the coefficients associated with the various subcategories showed significant differences compared to zero. This suggests that these subcategories do not exert a substantial effect on 'gdp_per_population_log.' Additionally, the β coefficient for 'hoursPerDayCombined' was also not statistically distinct from zero.

Conversely, when considering the impact of regions, we observe a substantial influence on the correlation. Most regions exhibit negative coefficients in comparison to Western Europe. The exceptions to this trend are North America, New Zealand, and Australia, where the corresponding p-values exceeded 0.05. These findings suggest that the region indeed plays a significant role in the correlation with 'gdp_per_population_log.'

res = smf.ols(
    """gdp_per_population_log ~  hoursPerDayCombined
+ C(region_code, Treatment(reference='EU_W')) 
+ C(Subcategory, Treatment(reference='Passive')) 
""",
    data=df,
).fit(disp=0)

res.summary2().tables[1]
(
    res.summary2()
    .tables[1]
    .style.format(precision=3, thousands=",", decimal=".")
    .applymap(
        lambda value: "color: red; font-weight:bold" if (value <= 0.05) else None, subset=["P>|t|"]
    )
    .background_gradient(cmap="viridis", axis=0, subset="Coef.")
)

Regression diagnostic

Next, we'll perform some diagnostic procedures for our regression model. These diagnostics will serve the purpose of evaluating how well our model aligns with its underlying assumptions. This step is crucial for ensuring the reliability and validity of our regression analysis.
As depicted in @fig-fig4, our residuals exhibit a relatively normal distribution. While there are a few data points that may have contributed to some erroneous predictions, we have opted to maintain the current state of the analysis as the results appear reasonable.


# Create a histogram plot of residuals
hist_plot = res.resid.to_frame(name="residuals").pipe(
    lambda df: ggplot(df, aes("residuals"))
    + geom_histogram(bins=15)
    + labs(
        title="Histogram of Residuals",
    )
    + theme(plot_title=element_text(size=15, face="bold"))
)

# Create a residuals plot
residuals_plot = (
    res.resid.to_frame(name="residuals")
    .reset_index()  # Reset the index for plotting
    .pipe(
        lambda df: ggplot(df, aes("index", "residuals"))
        + geom_point(size=5, alpha=0.05)  # Plot residuals as points
        + labs(
            title="Residuals",  # Title for the residuals plot
        )
        + theme(
            plot_title=element_text(size=15, face="bold")  # Customize title appearance
        )
    )
)


# Create a Q-Q plot of residuals
qqplot = res.resid.to_frame(name="residuals").pipe(
    lambda df: qq_plot(data=df, sample="residuals", size=5, alpha=0.05)
    + labs(
        title="Q-Q Plot",
    )
    + theme(plot_title=element_text(size=15, face="bold"))
)

# Create a fit plot comparing predicted values and log(GDP per population)
fit_plot = (
    pd.concat([res.fittedvalues, df.gdp_per_population_log], axis=1)
    .rename(
        columns={
            0: "Predicted value",
            "gdp_per_population_log": "log(GDP per population)",
        }
    )
    .pipe(
        lambda df: ggplot(df, aes("Predicted value", "log(GDP per population)"))
        + geom_point(size=5, alpha=0.05)
        + labs(
            title="Fit Plot",
        )
        + theme(plot_title=element_text(size=15, face="bold"))
        + geom_abline(slope=1, size=1, linetype="dashed", color="red")
    )
)

# Combine the three plots into a grid with two columns
(gggrid([hist_plot, residuals_plot, qqplot, fit_plot], ncol=2) + ggsize(800, 600))

And there you have it, folks! We hope you've found this week's exploration insightful. Happy coding to all, and I look forward to seeing you next week for more exciting discoveries and learning opportunities. Until then, stay curious and keep exploring! 👩‍💻👨‍💻

Speed up your Python code 60x

Adam Cseresznye — Fri, 07 Jun 2024 00:00:00 GMT

Ever felt the need to turbo boost your Python code? Tired of the seemingly endless wait for your iterations or simulations to complete? Allow me to present the rustimport library. In its most basic form, rustimport empowers you to call highly optimized Rust functions from within your code. This way, you can delegate the heavy-duty tasks to Rust, instead of relying on Python, a language known for its versatility but also for its slower speed. With rustimport, you can enjoy the best of both worlds - the simplicity of Python and the performance of Rust.

Let me give you an example: imagine you’re tackling one of the problems from Project Euler, specifically problem #50. Here’s what the problem statement looks like:

The prime $41$, can be written as the sum of six consecutive primes: $$41 = 2 + 3 + 5 + 7 + 11 + 13.$$ This is the longest sum of consecutive primes that adds to a prime below one-hundred. The longest sum of consecutive primes below one-thousand that adds to a prime, contains $21$ terms, and is equal to $953$. Which prime, below one-million, can be written as the sum of the most consecutive primes?

While there are many ways to solve this problem, one approach (though not the fastest) is as follows:

import cProfile
import pstats
import io


def is_prime(n):
    if n <= 1:
        return False
    i = 2
    while i * i <= n:
        if n % i == 0:
            return False
        i += 1
    return True


def generate_primes(target):
    primes = [n for n in range(2, target + 1) if is_prime(n)]
    return primes


def longest_sum_of_consecutive_primes(primes, target):
    max_length = 0
    max_prime = 0

    for i in range(len(primes)):
        for j in range(i + max_length, len(primes)):
            sum_of_primes = sum(primes[i:j])
            if sum_of_primes > target:
                break
            if is_prime(sum_of_primes) and (j - i) > max_length:
                max_length = j - i
                max_prime = sum_of_primes

    return max_prime, max_length


def main():
    TARGET = 1_000_000
    primes = generate_primes(TARGET)
    max_prime, max_length = longest_sum_of_consecutive_primes(primes, TARGET)
    print(f"Prime: {max_prime}, Length: {max_length}")

Let's dive in the code quickly:

The is_prime(n) function checks if a number n is prime. It starts by checking if n is less than or equal to 1, in which case it returns False as these are not prime numbers. It then checks divisibility from 2 up to the square root of n. If n is divisible by any of these numbers, it returns False; otherwise, it returns True indicating that n is a prime number.
The generate_primes(target) function generates a list of prime numbers up to a given target number, which is in our case 1000. It uses a list comprehension to iterate over the range from 2 to target (inclusive), calling the is_prime(n) function for each number n in the range. The result is a list of prime numbers.
The longest_sum_of_consecutive_primes(primes, target) function finds the prime number below a given target that can be written as the sum of the most consecutive primes. It starts with two variables, max_length and max_prime, initialized to 0. It then iterates over the primes list, for each prime calculating the sum of primes from the current prime to the end of the list. If the sum exceeds the target, it breaks the inner loop. If the sum is prime and the length of the sequence is greater than max_length, it updates max_length and max_prime. The function finally returns max_prime and max_length.

Note: To evaluate the execution time and identify which functions take longer to run, we use cProfile. The statistics generated are then formatted using the pstats module. The complete code for all the examples can be found here.

    1157601 function calls in 5.517 seconds

       Ordered by: cumulative time

       ncalls  tottime  percall  cumtime  percall filename:lineno(function)
            1    0.000    0.000    5.517    5.517 PE50_v1.py:36(main)
            1    0.000    0.000    5.025    5.025 PE50_v1.py:17(generate_primes)
            1    0.134    0.134    5.025    5.025 PE50_v1.py:18(<listcomp>)
      1000570    4.894    0.000    4.894    0.000 PE50_v1.py:7(is_prime)
            1    0.288    0.288    0.491    0.491 PE50_v1.py:21(longest_sum_of_consecutive_primes)
        78526    0.197    0.000    0.197    0.000 {built-in method builtins.sum}
        78499    0.004    0.000    0.004    0.000 {built-in method builtins.len}
            1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
            1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

As you can see, this straightforward task required over 5.5 seconds to execute. The primary function, PE50_v1.py:36(main), was invoked once and took a total of 5.517 seconds. The majority of this time was spent calling the function PE50_v1.py:17(generate_primes), which itself spent most of its time calling PE50_v1.py:7(is_prime) over a million times. The function PE50_v1.py:21(longest_sum_of_consecutive_primes) also consumed a significant amount of time. Additionally, the built-in functions sum and len were called multiple times.

Replacing the `is_prime` function

From the profiling results, it's clear that is_prime is a prime candidate (pun intended) for replacement with a Rust equivalent. Let's explore how we can achieve this.

Firstly, we need to install the rustimport library. This can be done using pip: pip install rustimport. Following this, we can create a Rust file, which we'll name rs_extension.rs for this example. Now, let's write some Rust code.

// rustimport:pyo3

use pyo3::prelude::*;

#[pyfunction]
fn is_prime(n: u32) -> bool {
    if n <= 1 {
        return false;
    }
    let mut i = 2;
    while i * i <= n {
        if n % i == 0 {
            return false;
        }
        i += 1;
    }
    true
}

Upon comparing the Python is_prime function with its Rust counterpart, you'll notice some similarities. The key difference comes from Rust's statically and strongly typed system, which requires us to specify the type of the input variables and return types. Not to worry, it's simple; we anticipate an unsigned integer (positive integers only) and return a boolean value indicating whether or not the integer is prime. Let's see what kind of speed up we get.

:::warning If your Rust code isn't performing as expected, it could be because it was compiled in debug mode, which doesn't enable optimizations. However, by running the python -m rustimport build --release command, you can generate Rust code that is highly optimized and significantly faster. Give it a try!`. :::

         1157601 function calls in 0.641 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.641    0.641 PE50_v1.py:30(main)
        1    0.197    0.197    0.391    0.391 PE50_v1.py:14(longest_sum_of_consecutive_primes)
        1    0.000    0.000    0.249    0.249 PE50_v1.py:9(generate_primes)
        1    0.089    0.089    0.249    0.249 PE50_v1.py:10(<listcomp>)
    78526    0.191    0.000    0.191    0.000 {built-in method builtins.sum}
  1000570    0.160    0.000    0.160    0.000 {built-in method rs_extension.is_prime}
    78499    0.004    0.000    0.004    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

🤯🤯🤯🤯 Our execution time went from 5.517 seconds to 0.641 seconds 🚀. That's a whopping 8.6 times speed up, achieved by replacing just one function. The is_prime function now takes a total of 0.16 seconds to execute. But why stop here? We can push the boundaries even further.

Consider the longest_sum_of_consecutive_primes function. It performs a substantial amount of work, and if we could harness the power of Rust for this function, we could potentially slash the execution time even more. Let´s give it a try.

Replacing the `longest_sum_of_consecutive_primes` function

#[pyfunction]
fn longest_sum_of_consecutive_primes(primes: Vec<u32>, target: u32) -> (u32, usize) {
    let mut max_length: usize = 0;
    let mut max_prime: u32 = 0;

    for i in 0..primes.len() {
        for j in i + max_length..primes.len() {
            let sum_of_primes: u32 = primes[i..j].iter().sum();
            if sum_of_primes > target {
                break;
            }
            if is_prime(sum_of_primes) && (j - i) > max_length {
                max_length = j - i;
                max_prime = sum_of_primes;
            }
        }
    }
    (max_prime, max_length)
}

Now this Rust function looks quite a bit different from our original Python function, but it's still possible to understand what's going on even if you haven't seen Rust code before. In this case, we're supplying the function with a vector of prime numbers, each represented as u32 values, and a target number, which is 1000 in our case. The function then returns the solution to our problem: the number (max_prime) that can be expressed as the sum of the greatest number of consecutive primes, and the actual count of these consecutive primes (max-length). This is the result from our second optimization:

         1000005 function calls in 0.252 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.252    0.252 PE50_v2.py:14(main)
        1    0.000    0.000    0.248    0.248 PE50_v2.py:9(generate_primes)
        1    0.088    0.088    0.248    0.248 PE50_v2.py:10(<listcomp>)
   999999    0.160    0.000    0.160    0.000 {built-in method rs_extension.is_prime}
        1    0.004    0.004    0.004    0.004 {built-in method rs_extension.longest_sum_of_consecutive_primes}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

🤯🤯🤯🤯 Our execution time has now been reduced to just 0.252 seconds, a further improvement of 2.5 times. In total, we've achieved a staggering 21.5 times speedup compared to our original Python implementation. The longest_sum_of_consecutive_primes function, which previously took 0.391 seconds to execute, now completes in a mere 0.004 seconds. This is a significant achievement!

As you might have guessed, we're not stopping here 😈. We still have one more Python function left to convert to Rust 🦀. Let's do that and see how much more we can optimize our code!

Replacing the `generate_primes` function

#[pyfunction]
fn generate_primes(target: u32) -> Vec<u32> {
    (2..=target).filter(|&n| is_prime(n)).collect::<Vec<u32>>()
}

This Rust code, while seemingly simple, is quite powerful. It generates, or more accurately, filters out prime numbers up to a specified range. Given that Rust is heavily inspired by OCaml and Haskell, you may notice the influence of functional programming paradigms throughout the language (and in this example).

         5 function calls in 0.092 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.092    0.092 PE50_v3.py:9(main)
        1    0.088    0.088    0.088    0.088 {built-in method rs_extension.generate_primes}
        1    0.004    0.004    0.004    0.004 {built-in method rs_extension.longest_sum_of_consecutive_primes}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.print}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

😲😵‍💫🤯💥💥💥 And there you have it! 🎉 Rust did it again. The execution time has been slashed from 0.252 seconds to a mere 0.092 seconds, making it 2.7 times faster. The generate_primes function, which previously took 0.248 seconds, now completes in just 0.088 seconds. And thus our code went from taking 5.517 sec to just 0.092 sec...

Here is our final iteration:

import cProfile
import pstats
import io

# import rustimport.import_hook
import rs_extension


def main():
    TARGET = 1_000_000
    primes = rs_extension.generate_primes(TARGET)
    max_prime, max_length = rs_extension.longest_sum_of_consecutive_primes(
        primes, TARGET
    )
    print(f"Prime: {max_prime}, Length: {max_length}")

import pandas as pd
import plotly.express as px

df = pd.DataFrame(
    {
        "step": [
            "original",
            "rewrite is_prime",
            "rewrite longest_sum_of_consecutive_primes",
            "rewrite generate_primes",
        ],
        "execution_time": [5.517, 0.641, 0.252, 0.092],
    }
)
fig = px.bar(
    df,
    x="execution_time",
    y="step",
    color="step",
    title="Turbocharging Performance: A 60x Speed Boost with Rust!",
    orientation="h",
    template="plotly_white",
)
fig.update_yaxes(title="", visible=True)
fig.update_xaxes(title="Execution time (s)")
fig.update_layout(showlegend=False)


fig.show()

And that's a wrap, everyone! In this post, we've showed you how to use cProfile to profile your Python code, helping you pinpoint potential bottlenecks. We've also showcased the power of the rustimport library, which enables you to effortlessly invoke highly optimized Rust code straight from your Python script.

Have you had the chance to use these tools in your projects? Do you believe your work could benefit from Rust? I'd love to hear your thoughts. Until next time, happy coding!

Adam Cseresznye

🏆🥉3rd place at Kaggle's Classification with a Tabular Vector Borne Disease Dataset

Top 10 things I learned from the book Effective Pandas by Matt Harrison

Load the dataset

Tip #1: Use <code>pd.assign</code> more extensively

Tip #2: Simplify the management of multiple if-else conditions using <code>np.select</code>

Tip #3 Filter rows or columns with the combination of <code>pd.loc</code> and <code>lambda</code>

Tip #4 Rename multiple columns effortlessly with <code>rename</code> and <code>replace</code>

Tip #5: Use <code>pd.clip</code> to easily remove outliers

Tip #6: Find corrupted entries with <code>str.extract</code>

Tip #7: Sort values based on the <code>key</code> parameter

Tip #8: Reference an existing variable inside <code>pd.query</code> with @

Tip #9: Gain more insights by using <code>style.background_gradient</code>

Tip #10: Use <code>pd.pipe</code> to include any functions in our chain

Tip #10 + 1: Use the "margin" parameter of <code>pd.crosstab</code> to easily calculate row/column subtotals

How to use the Lets-Plot library by JetBrains

Syntax Similarities: <code>Lets-Plot</code> and <code>ggplot2</code>

Visualizing Data Based on Categorical Variables

Incorporate Multiple Variables with facet_wrap

Reordering Categorical Variables Based On Statistics

Chaining Pandas Code with Lets Plot Visualization

Tidy Tuesday: Hot Ones Episodes

Import data

Questions

Data Analysis

What differences can be observed in the spiciness of sauces as we look across the various seasons?

Has every individual successfully completed all the episodes?

What is the completion rate per season?

Is there a correlation between Scoville Units and completion rate?

Are there any returning guests?

Final words

🥳🎉 Two IBM certificates and some geospatial data

Creating the map and displaying it

Superimposing bike locations on the map with FeatureGroup

Adding pop-up text with relevant information

Clustering the rental locations with MarkerCluster

Getting Started with Polars

Setup

Polars Version

Inspect the polars namespace

Access documentation

Indexing

Use take for Series or Expressions

Using filter

Evaluate a single condition

Combine multiple conditions

Imitate pandas' index using with_row_count

Use is_in

Select columns

Difference between select and with_columns`

Select all columns

Select all columns except for...

Select a subset of columns

Select columns based on regular expression

Select columns based on dtypes

Using selectors

By dtypes

By column names

dtype conversions

Working with str

Basic operations

Changing column names

str.split with expand='all'

Aggregation

A simple split-apply-combine operation

Using agg

Pivot

Handling missing data

Getting missing value count per column

Getting boolean array of missing values

Filling missing values

With literals

With an expression

Using the over function

Other common operations

Operating on multiple columns and renaming them

Count unique values

Value_counts to get number of occurrences per item

On a single column

On multiple columns

Superimposing bike locations on the map with `FeatureGroup`

Clustering the rental locations with `MarkerCluster`

Use `take` for Series or Expressions

Using `filter`

Imitate pandas' `index` using `with_row_count`

Use `is_in`

Difference between `select` and with_columns`

Using `selectors`

`dtype` conversions

Working with `str`

`str.split` with `expand='all'`

Using `agg`

Using the `over` function

Using `pl.read_csv`

Using `pl.scan_csv`

Get a Clearer Overview with `set_sticky`