10 basic tricks to make your pandas life a bit easier

Pandas is a powerful open source data analysis and manipulation tool, built on top of the Python programming language. In this article, I will show 10 tricks regarding the pandas DataFrame to make certain programming practices a bit easier.

Of course, before we can use pandas, we have to import it by using the following command:

import pandas as pd

1. Select multiple rows and columns using .loc

countries = pd.DataFrame({
'country': ['United States', 'The Netherlands', 'Spain', 'Mexico', 'Australia'],
'capital': ['Washington D.C.', 'Amsterdam', 'Madrid', 'Mexico City', 'Canberra'],
'continent': ['North America', 'Europe', 'Europe', 'North America', 'Australia'],
'language': ['English', 'Dutch', 'Spanish', 'Spanish', 'English']})

By using the loc operator, we are able to select subsets of rows and columns on the basis of their index label and column name. Below are some examples on how to use the loc operator on the ‘countries’ DataFrame:

countries.loc[:, 'country':'continent']

countries.loc[0:2, 'country':'continent']

countries.loc[[0, 4], ['country', 'language']]

2. Filter DataFrames by category

In many cases, we may want to consider only the data points that are included in one particular category, or sometimes in a selection of categories. For a single category, we are able to do this by using the == operator. However, for multiple categories, we have to make use of the isin function:

countries[countries.continent == 'Europe']

countries[countries.language.isin(['Dutch', 'English'])]

3. Filter DataFrames by excluding categories

As opposed to filtering by category, we may want to filter our DataFrame by excluding certain categories. We do this by making use of the ~ (tilde) sign, which is the complement operator. Example usage:

countries[~countries.continent.isin(['Europe'])]

countries[~countries.language.isin(['Dutch', 'English'])]

4. Rename columns

You might want to change the name of certain columns because e.g. the name is incorrect or incomplete. For example, we might want to change the ‘capital’ column name to ‘capital_city’ and ‘language’ to ‘most_spoken_language’. We can do this in the following way:

countries.rename({'capital': 'capital_city', 'language': 'most_spoken_language'}, axis='columns')

Alternatively, we can use:

countries.columns = ['country', 'capital_city', 'continent', 'most_spoken_language']

5. Reverse row order

To reverse the row order, we make use of the loc operator. This works in the following way:

countries.loc[::-1]

However, note that now the indexes still are following the previous ordering. We have to make use of the reset_index function to reset the indexes:

countries.loc[::-1].reset_index(drop=True)

6. Reverse column order

Reversing the column order goes in a similar way as for the rows:

countries.loc[:, ::-1]

7. Split a DataFrame into two random subsets

In some cases, we want to split a DataFrame into two random subsets. For this, we make use of the sample function. For example, when creating a training and a test set out of the whole data set, we have to create two random subsets. Below, we show how to use the sample function:

countries_1 = countries.sample(frac=0.6, random_state=999)
countries_2 = countries.drop(countries_1.index)

8. Create dummy variables

students = pd.DataFrame({
'name': ['Ben', 'Tina', 'John', 'Eric'],
'gender': ['male', 'female', 'male', 'male']})

We might want to convert categorical variables into dummy/indicator variables. We can do so by making use of the get_dummies function:

pd.get_dummies(students)

To get rid of the redundant columns, we have to add drop_first=True:

pd.get_dummies(students, drop_first=True)

9. Check equality of columns

When the goal is to check equality of two different columns, one might at first think of the == operator, since this is mostly used when we are concerned with checking equality conditions. However, this operator does not handle NaN values properly, so we make use of the equals functions here. This goes as follows:

df = pd.DataFrame({'col_1': [1, 0], 'col_2': [0, 1], 'col_3': [1, 0]})

df['col_1'].equals(df['col_2'])

>>> False

df['col_1'].equals(df['col_3'])

>>> True

10. Concatenate DataFrames

We might want to combine two DataFrames into one DataFrame that contains all data points. This can be achieved by using the concat function:

df_1 = pd.DataFrame({'col_1': [6, 7, 8], 'col_2': [1, 2, 3], 'col_3': [5, 6, 7]})

pd.concat([df, df_1]).reset_index(drop=True)

Thanks for reading!

I hope this article helped you in some way, and I wish you good luck on your next project when making use of Pandas :).

A couple of days back, mister Tobias Krabel contacted me via LinkedIn to introduce me to his product, a Python library called Bamboolib, which he states to be a GUI tool for learning Pandas — Python’s data analysis and visualization library.

He states, and I quote:

Our goal is to help people quickly learn and work with pandas, and we want to onboard the next generation of python data scientists.

I have to admit, I was skeptical at first, mainly because I’m not a big fan of GUI tools and drag & drop principle in general. Still, I’ve opened the URL and watched the introduction video.

It was one of those rare times when I was legitimately intrigued.

From there I’ve quickly responded to Tobias, and he kindly offered me to test out the library and see if I liked it.

How was it? Well, you’ll have to keep reading to find the answer to that. So let’s get started.

Is it Free?

In a world where such amazing libraries like Numpy and Pandas are free to use, this question may not even pop in your head. However, it should, because not all versions of Bamboolib are free.

If you don’t mind sharing your work with others, then yeah, it’s free to use, but if that poses a problem then it will set you back at least $10 a month which might be a bummer for the average users. Down below is the full pricing list:

As the developer of the library stated, Bamboolib is designed to help you learn Pandas, so I don’t see a problem with going with the free option — most likely you won’t be working on some top-secret project if just starting out.

This review will, however, be based on the private version of the library, as that’s the one Tobias gave access to me. With that being said, this article is by no means written with the idea of persuading you to buy the license, it only provides my personal opinion.

Before jumping into the good stuff, you’ll need to install the library first.

The Installation Process

The first and most obvious thing to do is pip install:

pip install bamboolib

However, there’s a lot more to do if you want this thing fully working. It is designed to be a Jupyter Lab extension (or Jupyter Notebook if you still use those), so we’ll need to set up a couple of things there also.

In a command line type the following:

jupyter nbextension enable --py qgrid --sys-prefix
jupyter nbextension enable --py widgetsnbextension --sys-prefix
jupyter nbextension install --py bamboolib --sys-prefix
jupyter nbextension enable --py bamboolib --sys-prefix

Now you’ll need to find the major version of Jupyter Lab installed on your machine. You can obtain it with the following command:

jupyter labextension list

Mine is “1.0”, but yours can be anything, so here’s a generic version of the next command you’ll need to execute:

jupyter labextension install @jupyter-widgets/jupyterlab-manager@MAJOR_VERSION.MINOR_VERSION --no-build

Note that you need to replace “MAJOR_VERSION.MINOR_VERSION” with the version number, which is “1.0” in my case.

A couple of commands more and you’re ready to rock:

jupyter labextension install @8080labs/qgrid@1.1.1 --no-build
jupyter labextension install plotlywidget --no-build
jupyter labextension install jupyterlab-plotly --no-build
jupyter labextension install bamboolib --no-build

jupyter lab build --minimize=False

That’ it. Now you can start Juypter Lab and we can dive into the good stuff.

The First Use

Once in Jupyter, you can import Bamboolib and Pandas, and then use Pandas to load in some dataset:

Here’s how you’d use the library to view the dataset:

That’s not gonna work the first time you’re using the library. You’ll need to activate it, so make sure to have the license key somewhere near:

Once you’ve entered the email and license key, you should get the following message indicating that everything went well:

Great, now you can once again execute the previous cell. Immediately you’ll see an unfamiliar, but friendly-looking interface:

Now everything is good to go, and we can dive into some basic functionalities. It was a lot of work to get to this point, but trust me, it was worth it!

Data Filtering

One of the most common everyday tasks of any data analyst/scientist is data filtering. Basically you want to keep only a subset of data that’s relevant to you in a given moment.

To start filtering with Bamboolib, click on the Filter button.

A side menu like the one below should pop up. I’ve decided to filter by the “Age” column, and keep only the rows where the value of “Age” is less than 18:

Once you press Execute, you’ll see the actions took place immediately:

That’s great! But what more can you do?

Replacing Values

Another one of those common everyday tasks is to replace string values with the respective numerical alternative. This dataset is perfect to demonstrate value replacement because we can easily replace string values in the “Sex” column with numeric ones.

To begin, hit the Replace value button and specify the column, the value you want to replace and what you want to replace it with:

And once the Execute button is hit:

Fantastic! You can do the same for the “female” option, but it’s up to you whether you want to do it or not.

Group By

Yes, you can also perform aggregations! To get started, click on the Aggregate/Group by button and specify what should be done in the side menu.

I’ve decided to group by “Pclass”, because I want to see the total number of survivors per passenger class:

That will yield the following output:

Awesome! Let’s explore one more thing before wrapping up.

One Hot Encoding

Many times when preparing data for machine learning you’ll want to create dummy variables, ergo create a new column per unique value of a given attribute. It’s a good idea to do so because many machine learning algorithms can’t work with text data.

To implement that logic via Bamboolib, hit the OneHotEncoder button. I’ve decided to create dummy variables from the “Embarked” attribute because it has 3 distinct values and you can’t state that one is better than the other. Also, make sure to remove the first dummy to avoid collinearity issues (having variable which is a perfect predictor for some other variable):

Executing will create two new columns in the dataset, just as you would expect:

That’s nice, I’ve done my transformations, but what’s next?

Getting the Code

It was all fun and games until now, but sooner or later you’ll notice the operations don’t act in place — ergo the dataset will not get modified if you don’t explicitly specify it.

That’s not a bug, as it enables you to play around without messing the original dataset. What Bamboolib will do, however, it will generate Python code for achieving the desired transformations.

To get the code, first, click on the Export button:

Now specify how do you want it exported — I’ve selected the first option:

And it will finally give you the code which you can copy and apply to the dataset:

Is it worth it?

Until this point, I showcased briefly the main functionalities of Bamboolib — by no means was it exhaustive tutorial — just wanted to show you the idea behind it.

The question remains, is it worth the money?

That is if you decide to go with the paid route. You can still use it for free, provided that you don’t mind sharing your work with others. The library by itself is worth checking out for two main reasons:

1. It provides a great way to learn Pandas — it’s much more easy to learn by doing than by reading, and a GUI tool like this will most certainly only help you
2. It’s great for playing around with data — let’s face it, there are times when you know what you want to do, but you just don’t know how to implement it in code — Bamboolib can assist

Keep in mind — you won’t get any additional features with the paid version — the only real benefit is that your work will be private and that there’s an option for commercial use.

Even if you’re not ready to grab your credit card just yet, it can’t harm you to try out the free version and see if it’s something you can benefit from.

Thanks for reading. Take care.

Handling exceptions in python can get in some cases repetitive and ugly, we can solve that using decorators.

Functions in Python

Functions in Python are first class objects, which means they can be assigned to a variable, passed as an argument, and return from another function and store that in any data structure.

def example_function():
    print("Example Function called")some_variable = example_functionsome_variable()

As you can see we have assigned example_function to some_variable which makes some_variable callable. The output for the following will be:

Example Function called

Decorators

The first class object property of function helps us to use the concept of Decorators in Python. Decorators are functions which take as argument another function as an object, which enables us to put our logic either at the start and end of the execution of the argument function.

def decorator_example(func):
    print("Decorator called")

    def inner_function(*args, **kwargs):
            print("Calling the function")
            func(*args, **kwargs)
            print("Function's execution is over")
    return inner_function@decorator_example
def some_function():
    print("Executing the function")
    # Function logic goes here

As you can see we are using the decorator by using @decorator_example on top of the function which needs to be passed as the argument to the Decorator function. In this case, some_function will be passed as an argument to decorator_example. The output of the above snippet will be:

Decorator called
Calling the function
Executing the function
Function's execution is over

Error Handling Using Decorators

You can use Decorators for quite a lot of purposes like logging, validations, or any other common logic which needs to be put in multiple functions. One of the many areas where Decorators can be used is the exception handling.

Let’s take an example of such functions, which require handling of the same exceptions.
We will take a simple example of calculating areas. And we will be for now printing the errors if an unsupported type is passed as the argument which in this case will be a string.

Errors should never pass silently.
Unless explicitly silenced.

The normal way to do this would be to have all such functions in a try-catch, somewhat like this,

def area_square(length):
    try:
        print(length**2)
    except TypeError:
        print("area_square only takes numbers as the argument")


def area_circle(radius):
    try:
        print(3.142 * radius**2)
    except TypeError:
        print("area_circle only takes numbers as the argument")


def area_rectangle(length, breadth):
    try:
        print(length * breadth)
    except TypeError:
        print("area_rectangle only takes numbers as the argument")

Now, this looks repetitive which we should avoid to the extent we can. So we can use the magic of Decorators here and observe that the code looks a lot cleaner this way. And a clean code goes a long way.

def exception_handler(func):
    def inner_function(*args, **kwargs):
        try:
            func(*args, **kwargs)
        except TypeError:
            print(f"{func.__name__} only takes numbers as the argument")
    return inner_function


@exception_handler
def area_square(length):
    print(length * length)


@exception_handler
def area_circle(radius):
    print(3.14 * radius * radius)


@exception_handler
def area_rectangle(length, breadth):
    print(length * breadth)


area_square(2)
area_circle(2)
area_rectangle(2, 4)
area_square("some_str")
area_circle("some_other_str")
area_rectangle("some_other_rectangle")

The output of the following will be:

4
12.568
8
area_square only takes numbers as the argument
area_circle only takes numbers as the argument
area_rectangle only takes numbers as the argument

We can extend the capability of raising errors in the exception_handler by having custom exceptions and further expand its usages. This was one of the examples which enables us to handle the exceptions cleanly.