How to use Pandas to Analyze Data

By McKay Johns•November 3, 2024

Pandas is a powerful tool for analyzing data. Learn how to use it effectively for data analysis, data science, and sports analytics.

PythonRsports analyticscoding

If you are working with any sort of data in Python, you are going to be using Pandas. It doesn’t matter if you are a newbie data analyst or if you are a seasoned data scientist, Pandas is what truly unlocks the power of data.

Pandas is a Python package that allows us to work with data in a tabular format. This of this as similar to an Excel sheet but instead, it loads the data straight into our coding environment.

When I started learning to code, I heavily used Pandas to analyze sports and for different school projects. Pandas is almost a language of its own and takes some practice to learn the ins and outs.

In this article, we’ll go over the four steps of analyzing data with Pandas with some code snippets as well.

If you want to follow along and run the code, I’ll be using this CSV file here.

1. Loading Data

Pandas excels at this by providing straightforward methods to load data from a variety of sources:

CSV Files
Excel Spreadsheets
SQL Databases
JSON
HTML
and More

By loading your data into a Pandas DataFrame, you create a structured environment where each column can be manipulated and analyzed with ease.

With just this line of code, you can take a file and load it into your environment.

!https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9993ba2-b28d-4b4e-adbe-744b05bbb4a1_1530x520.png

2. Examining Data

Once your data is loaded, it's essential to understand its structure and content before diving into analysis:

Overview of DataFrame: Get a quick snapshot of your data's dimensions and types.
```
df.shape
df.columns
```
Head and Tail Methods: Preview the first or last few rows to spot any immediate anomalies.
```
df.head()
df.tail()
```
Descriptive Statistics: Generate summaries to understand distributions, means, medians, and more.
```
df.describe()
df.info()
```
Data Types and Missing Values: Identify data types for each column and detect missing or null values.
```
df.dtypes
df.isnull.sum()
df['outcome'].value_counts()
```

This step is very useful to understand your data and what it contains. Any project you do in data will start with useful data exploration so mastering these functions can help tell you a lot about the data.

3. Cleaning Data

Data in the real world is rarely perfect. Cleaning your data is a critical step to ensure the integrity of your analysis:

Handling Missing Values: Decide whether to fill in missing data or remove incomplete records.

df.isnull().sum()

# Fill missing values with a specific value (e.g., 0)
df.fillna(0, inplace=True)

# Fill missing values with the mean of the column
df['x'].fillna(df['x'].mean())

Removing Duplicates: Eliminate redundant data that could skew results.
```
df.drop_duplicates()
```
Data Type Conversion: Ensure all data is in the correct format for analysis (e.g., dates, numeric values).
```
# Convert a column to numeric type
df['x'] = pd.to_numeric(df['x'], errors='coerce')
```
Outlier Detection: Identify and address anomalies that may distort findings.

Cleaning data is one of the most important parts of working with data. You usually are going to spend a bit in this step so learning to effectively do it and set up workflows will save you a lot of time.

4. Analyzing Data

With clean data in hand, you're ready to uncover insights:

Filtering and Sorting: Isolate subsets of data based on conditions or sort data to identify trends.

# Filter the dataframe to where any value of x is greater than 30
filtered_df = df[df['x'] > 30]

# Sort a dataframe by a column
sorted_df = df.sort_values(by='minute')

Grouping and Aggregation: Summarize data by categories to reveal patterns and correlations.

# This will get the mean values of x for each different value in the outcome column

grouped_df = df.groupby('outcome')['x'].mean()

Merging and Joining: Combine multiple DataFrames to enrich your dataset.
Visualization: Create charts and graphs to present data visually (often in conjunction with libraries like Matplotlib or Seaborn).

Pandas streamlines the data analysis process, transforming raw data into an easy-to-work-with format that allows you to have clean and actionable data.

Pandas is such an important library to learn that I even have a whole section dedicated to it in the Complete Football Analytics in Python Course.

By mastering these four steps—loading, examining, cleaning, and analyzing—you equip yourself with the skills to tackle a wide array of data challenges.