Pandas Crash Course

When working on a data science project, you spend most of your time cleaning and preprocessing data. Choosing the right library to manipulate tabular data efficiently is crucial for a smooth workflow.

Pandas is the standard open-source library for data analysis and manipulation in Python. Built on top of NumPy, it introduces the DataFrame—a highly optimized two-dimensional tabular data structure with labeled axes (rows and columns) that makes data manipulation fast and intuitive.

In this tutorial, you will build a complete data cleaning and analysis pipeline. You will learn to construct DataFrames from scratch, import real-world datasets like NBA player stats and IMDB movie rankings, handle duplicate rows, impute missing values using statistical means, and analyze variable relationships with correlation.

Prerequisites: Python 3.x, Pandas, NumPy, Seaborn, Matplotlib.

Datasets

You can download all the datasets used in this tutorial from the All CSV ML Data Files Download repository.

Getting Started

To begin working with the library, you must first import the module under its standard alias:

import pandas as pd

Prepare a Python dictionary containing lists of values to represent orange and apple counts:

data = {
    'apple': [3,1,4,5],
    'orange': [1, 5, 6, 8]
}

data

{'apple': [3, 1, 4, 5], 'orange': [1, 5, 6, 8]}

Check the data type of your newly created dictionary:

type(data)

dict

Convert your Python dictionary into a Pandas DataFrame. A DataFrame is a two-dimensional, size-mutable, and potentially heterogeneous tabular data structure. It aligns data in a tabular fashion using rows and columns:

df = pd.DataFrame(data)
df

Extract a single column to see how Pandas represents individual columns as Series. A Series is a one-dimensional array-like object that contains a sequence of values:

df['apple']

0    3
1    1
2    4
3    5
Name: apple, dtype: int64

Verify that the type of the extracted column is indeed a Pandas Series:

type(df['apple'])

pandas.core.series.Series

Reading CSV Files

To load external datasets, Pandas provides robust input/output functions. For example, read_csv() imports a comma-separated values file directly into a DataFrame:

df = pd.read_csv('nba.csv')

Display the first ten rows of the DataFrame using head() to examine the dataset structure:

df.head(10)

View the last two rows of the dataset using tail() to verify row alignments at the end of the file:

df.tail(2)

Load the CSV file while specifying the Name column to act as the row labels or index of the DataFrame:

df = pd.read_csv('nba.csv', index_col = 'Name')
df.head()

You can also use custom row labels for other datasets. For example, load the IMDB Movie dataset using the movie's rank as the index:

df = pd.read_csv('IMDB-Movie-Data.csv', index_col = 'Rank')
df.head()

Display the last five rows of the IMDB dataset to verify the tail structure:

df.tail()

To inspect the schema, data types, and non-null counts of each column, call the info() method:

df.info()

Int64Index: 1000 entries, 1 to 1000
Data columns (total 11 columns):
Title                 1000 non-null object
Genre                 1000 non-null object
Description           1000 non-null object
Director              1000 non-null object
Actors                1000 non-null object
Year                  1000 non-null int64
Runtime (Minutes)     1000 non-null int64
Rating                1000 non-null float64
Votes                 1000 non-null int64
Revenue (Millions)    872 non-null float64
Metascore             936 non-null float64
dtypes: float64(3), int64(3), object(5)
memory usage: 93.8+ KB

To check the dimensionality of the DataFrame, inspect the shape attribute:

df.shape

(1000, 11)

You can count duplicate rows by combining the duplicated() method with a sum operation:

sum(df.duplicated())

0

To demonstrate duplicate handling, append the DataFrame to itself to double the number of rows:

df1 = df.append(df)
df1.shape

(2000, 11)

Verify the number of duplicate rows in the newly concatenated DataFrame:

df1.duplicated().sum()

1000

Remove the duplicate rows using drop_duplicates() to return a cleaned copy:

df2 = df1.drop_duplicates()
df2.shape

(1000, 11)

Note that the original DataFrame remains unchanged unless you assign the result back or modify it in place:

df1.shape

(2000, 11)

To modify the DataFrame directly without creating a new copy, set the inplace parameter to True:

df1.drop_duplicates(inplace = True)
df1.shape

(1000, 11)

Column Operations

Access the column labels of the DataFrame using the columns attribute:

df.columns

Index(['Title', 'Genre', 'Description', 'Director', 'Actors', 'Year',
       'Runtime (Minutes)', 'Rating', 'Votes', 'Revenue (Millions)',
       'Metascore'],
      dtype='object')

Count the number of columns in the DataFrame:

len(df.columns)

11

Generate descriptive statistics for the numerical columns using the describe() method:

df.describe()

Extract the column names and convert them into a standard Python list:

col = df.columns
type(list(col))

list

Display the index object containing all column names:

col

Index(['Title', 'Genre', 'Description', 'Director', 'Actors', 'Year',
       'Runtime (Minutes)', 'Rating', 'Votes', 'Revenue (Millions)',
       'Metascore'],
      dtype='object')

You can rename columns by assigning a list of new names directly to the columns attribute:

col1 = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k']
df.columns = col1
df.head()

Revert back to the original column names:

df.columns = col
df.head(0)

For more precise renaming, use the rename() method with a dictionary mapping old column names to new ones:

df.rename(columns={
    'Runtime (Minutes)': 'Runtime',
    'Revenue (Millions)': 'Revenue'
}, inplace= True)

df.columns

Index(['Title', 'Genre', 'Description', 'Director', 'Actors', 'Year',
       'Runtime', 'Rating', 'Votes', 'Revenue', 'Metascore'],
      dtype='object')

Compare the updated column list with the original stored in the col variable:

col

Index(['Title', 'Genre', 'Description', 'Director', 'Actors', 'Year',
       'Runtime (Minutes)', 'Rating', 'Votes', 'Revenue (Millions)',
       'Metascore'],
      dtype='object')

Missing Values

Real-world datasets often contain missing values. You can represent these missing entries in Python using NumPy's nan constant:

import numpy as np

Access the standard representation for empty values:

np.nan

nan

Identify missing values in the DataFrame using isnull(), which returns boolean values indicating the presence of null values:

df.isnull().head(10)

Sum the boolean values to count the number of missing records in each column:

df.isnull().sum()

Title            0
Genre            0
Description      0
Director         0
Actors           0
Year             0
Runtime          0
Rating           0
Votes            0
Revenue        128
Metascore       64
dtype: int64

Alternatively, call isna() to achieve the same result:

df.isna().sum()

Title            0
Genre            0
Description      0
Director         0
Actors           0
Year             0
Runtime          0
Rating           0
Votes            0
Revenue        128
Metascore       64
dtype: int64

Remove any rows containing at least one missing value using dropna():

df1 = df.dropna()
df1.shape

(838, 11)

To drop columns that contain missing values instead of rows, set axis=1:

df2 = df.dropna(axis = 1)
df2.head(3)

Imputation

Rather than deleting rows or columns, you can fill missing cells with a specific value. Fill all missing entries with 0 using fillna():

df3 = df.fillna(0)
df3.isna().sum()

Title          0
Genre          0
Description    0
Director       0
Actors         0
Year           0
Runtime        0
Rating         0
Votes          0
Revenue        0
Metascore      0
dtype: int64

Recall the counts of null values in the original DataFrame:

df.isnull().sum()

Title            0
Genre            0
Description      0
Director         0
Actors           0
Year             0
Runtime          0
Rating           0
Votes            0
Revenue        128
Metascore       64
dtype: int64

Extract the Revenue column as a Pandas Series:

revenue = df['Revenue']
type(revenue)

pandas.core.series.Series

Display the last five elements of the revenue Series to check for missing entries:

revenue.tail()

Rank
996       NaN
997     17.54
998     58.01
999       NaN
1000    19.64
Name: Revenue, dtype: float64

Compute the average revenue to use for imputation:

revenue_mean = revenue.mean()
revenue_mean

82.95637614678897

Fill the missing entries in revenue with its calculated mean value in-place:

revenue.fillna(revenue_mean, inplace= True)
revenue.tail()

Rank
996     82.956376
997     17.540000
998     58.010000
999     82.956376
1000    19.640000
Name: Revenue, dtype: float64

Confirm that the revenue Series no longer contains null values:

revenue.isnull().sum()

0

Assign the imputed Series back to the DataFrame and check the remaining null counts:

df['Revenue'] = revenue
df.isnull().sum()

Title           0
Genre           0
Description     0
Director        0
Actors          0
Year            0
Runtime         0
Rating          0
Votes           0
Revenue         0
Metascore      64
dtype: int64

Apply the same mean-imputation process to the Metascore column:

metascore = df['Metascore']
metascore_mean = metascore.mean()
print("metascore_mean =",metascore_mean)
metascore.fillna(metascore_mean, inplace = True)
df['Metascore'] = metascore
df.isnull().sum()

metascore_mean = 58.98504273504273

Title          0
Genre          0
Description    0
Director       0
Actors         0
Year           0
Runtime        0
Rating         0
Votes          0
Revenue        0
Metascore      0
dtype: int64

Verify the statistical summary of the numerical columns now that the missing values are imputed:

df.describe()

Print the updated info schema to confirm all columns have 1000 non-null values:

df.info()

Int64Index: 1000 entries, 1 to 1000
Data columns (total 11 columns):
Title          1000 non-null object
Genre          1000 non-null object
Description    1000 non-null object
Director       1000 non-null object
Actors         1000 non-null object
Year           1000 non-null int64
Runtime        1000 non-null int64
Rating         1000 non-null float64
Votes          1000 non-null int64
Revenue        1000 non-null float64
Metascore      1000 non-null float64
dtypes: float64(3), int64(3), object(5)
memory usage: 93.8+ KB

Get a statistical summary of the non-numerical Genre column:

df['Genre'].describe()

count                        1000
unique                        207
top       Action,Adventure,Sci-Fi
freq                           50
Name: Genre, dtype: object

Count the frequencies of the top ten movie genres:

df['Genre'].value_counts().head(10)

Action,Adventure,Sci-Fi       50
Drama                         48
Comedy,Drama,Romance          35
Comedy                        32
Drama,Romance                 31
Comedy,Drama                  27
Action,Adventure,Fantasy      27
Animation,Adventure,Comedy    27
Comedy,Romance                26
Crime,Drama,Thriller          24
Name: Genre, dtype: int64

Check the count of unique movie genres present in the dataset:

len(df['Genre'].unique())

207

Correlation Analysis

Compute the pairwise Pearson correlation matrix for the numerical columns using the corr() method:

corrmat = df.corr()
corrmat

You can compute relationships between numerical attributes using correlation matrices. To visualize these correlation matrices as colors, import the Seaborn library:

import seaborn as sns

Plot the correlation matrix as a color-encoded heat map:

sns.heatmap(corrmat)

You can also use Matplotlib to generate visualizations directly from DataFrames. Import the Matplotlib pyplot interface:

import matplotlib.pyplot as plt

Generate a scatter plot of movie ratings versus revenues:

df.plot(kind = 'scatter', x = 'Rating', y = 'Revenue', title = 'Revenue vs Rating')

Create a histogram plot to view the frequency distribution of ratings:

df['Rating'].plot(kind = 'hist', title = 'Rating')

Estimate the probability density of ratings using a Kernel Density Estimation (KDE) plot:

df['Rating'].plot(kind = 'kde', title = 'Rating')

Count the frequency of each rating category:

df['Rating'].value_counts()

7.1    52
6.7    48
7.0    46
6.3    44
6.6    42
7.2    42
7.3    42
6.5    40
7.8    40
6.2    37
6.8    37
7.5    35
6.4    35
7.4    33
6.9    31
6.1    31
7.6    27
7.7    27
5.8    26
6.0    26
8.1    26
7.9    23
5.7    21
8.0    19
5.9    19
5.6    17
5.5    14
5.3    12
5.4    12
5.2    11
8.2    10
4.9     7
8.3     7
4.7     6
8.5     6
4.6     5
5.1     5
5.0     4
4.8     4
4.3     4
8.4     4
3.9     3
8.6     3
8.8     2
2.7     2
4.2     2
3.5     2
3.7     2
9.0     1
3.2     1
4.0     1
4.5     1
4.4     1
4.1     1
1.9     1
Name: Rating, dtype: int64

To visualize the distribution, spread, and outliers of ratings, you can construct a box plot. Box plots represent the five-number summary of a dataset: the minimum, first quartile ($Q_1$), median, third quartile ($Q_3$), and maximum:

Generate a box plot of movie ratings:

df['Rating'].plot(kind = 'box')

Compute numerical statistics for movie ratings to match the box plot coordinates:

df['Rating'].describe()

count    1000.000000
mean        6.723200
std         0.945429
min         1.900000
25%         6.200000
50%         6.800000
75%         7.400000
max         9.000000
Name: Rating, dtype: float64

Add a new rating category column to label movies as good or bad depending on their rating threshold:

rating_cat = []
for rate in df['Rating']:
    if rate > 6.2:
        rating_cat.append('Good')
    else:
        rating_cat.append('Bad')

rating_cat[:20]

['Good', 'Good', 'Good', 'Good', 'Bad', 'Bad', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good', 'Good']

Assign the list to the DataFrame and display the head to confirm the column addition:

df['Rating Category'] = rating_cat
df.head()

Group your data by a categorical feature and draw box plots for each group using the boxplot() method:

df.boxplot(column= 'Revenue', by = 'Rating Category')

Conclusion

In this tutorial, you explored the core functionalities of the Pandas library, including DataFrame creation, duplicate removal, column renaming, and handling missing values using mean imputation. Using the NBA and IMDB datasets, you walked through essential data cleaning steps and calculated pairwise correlation to assess linear dependencies between numeric variables.

Key takeaways:

• DataFrames and Series: A DataFrame is a two-dimensional tabular structure, while a Series represents a single column within that DataFrame.
• Wrangling and Deduplication: Tabular data often contains duplicates that can be filtered out using drop_duplicates(inplace=True) to clean the workspace.
• Handling Null Values: Missing data (NaN) should either be dropped or imputed using operations like fillna() with statistical metrics such as the column mean.
• Correlation: The corr() method helps calculate pairwise relationship coefficients to find dependencies before building predictive models.

Next steps:

• Dive deeper into visual analysis with Data Visualization with Pandas to learn how to plot line, bar, scatter, and area graphs directly from DataFrames.
• Master plotting basics with Matplotlib Crash Course to customize your plots with titles, axes, and styles.
• Learn about the different kinds of variables you will encounter in Data Variable Types Every Data Scientist Needs to choose appropriate visualization tools.