Jupyter Guide

To get started, open the file on the right entitled "Step 1...". You'll complete each task for Step 1 in that Jupyter Notebook file. Remember, you must run the cells (ctrl/cmd(⌘) + Enter) for each task before moving onto the next task in the Jupyter Notebook. Continue until you have completed all tasks in this step. Then when you are ready to move onto the next step, you'll come back and click on the file for the next step until you have completed all tasks in all steps of the lab.

Introduction to Normalization

To review the concepts covered in this step, please refer to the Some Simple Normalization Techniques module of the Normalize Data to Make It Appropriate for an Analysis with Pandas course.

Data normalization is important because it makes features of a model have equal weight and makes data more robust for analysis. Here, we will practice normalizing features in a dataset using Pandas and the standard scaling technique.

This technique is a straightforward way to bring all attributes to the same scale by subtracting the mean and dividing by the standard deviation. You will use sklearn and the pandas library to normalize a randomly sampled dataset. We'll use the StandardScalartransform from sklearn to normalize the data such that its distribution will have a mean of 0 and a standard deviation of 1. Observe the effects by plotting the distribution before and after normalization.

Task 1.1: Importing Necessary Libraries

Before we can start working with data, we need to import the necessary libraries. Import pandas, numpy, matplotlib, and the StandardScaler from sklearn.preprocessing.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler

Task 1.2: Creating a Random Dataset

Create a pandas DataFrame with 100 rows and 5 columns. The columns should be filled with random numbers from a normal distribution centered at mu = 2 and with standard deviation sigma = 5. Display the first few rows of the dataframe.

array = np.random.normal(2, 5, size=(100, 5))
df = pd.DataFrame(array)
df.head()

Task 1.3: Plotting the Distribution Before Normalization

Plot the distribution of the values in the first column of the DataFrame before normalization. Use DataFrame.plot(kind='density') to create the plot.

df.plot(kind='density')

Task 1.4: Normalizing the Data

Normalize the data in the DataFrame using the StandardScaler. Display the head of the transformed data.

scaler = StandardScaler()
scaler.fit(df)
df_normalized = pd.DataFrame(scaler.transform(df))
df_normalized.head()

Task 1.5: Plotting the Distribution After Normalization

Plot the normalized data against the original data and compare the normalized vs the old distributions.

fig, ax = fig.subplots()
DataFrame.plot(ax=ax, kind="density")

fig, ax = plt.subplots()
df.plot(ax=ax, kind='density')
df_normalized.plot(ax=ax, kind='density')

Applying Min-max Scaling Technique

To review the concepts covered in this step, please refer to the Some Simple Normalization Techniques module of the Normalize Data to Make It Appropriate for an Analysis with Pandas course.

Understanding different normalization techniques is crucial because it allows you to choose the correct method based on the characteristics of your data. In this step, you will apply min-max scaling.

Let's take your data normalization skills to the next level by practicing another scaling technique: Min-Max Scaling. Min-Max scaling transforms data to fit within the range of 0 and 1. Use sklearn to try out this technique on a randomly sampled DataFrame, and observe their effects on data distribution by plotting the sample dataset before and after normalization.

Task 2.1: Import Required Libraries

Before we can start working with data, we need to import the necessary libraries. Import pandas, numpy, matplotlib, and the MinMaxScaler from sklearn.preprocessing.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler

Task 2.2: Creating a Random Dataset

Create a pandas DataFrame with 100 rows and 5 columns. The columns should be filled with random numbers from a uniform distribution in the range [8, 10]. Display the first few rows of the DataFrame.

array = np.random.uniform(8, 10, size=(100, 5))
df = pd.DataFrame(array)
df.head()

Task 2.3: Plotting the Distribution Before Normalization

Plot the distribution of the values in the first column of the DataFrame before normalization. Use DataFrame.plot(kind='density') to create the plot.

df.plot(kind='density')

Task 2.4: Apply Min-Max Scaling

Normalize the data in the DataFrame using the MinMaxScaler. Display the head of the transformed data.

scaler = MinMaxScaler()
scaler.fit(df)
df_normalized = pd.DataFrame(scaler.transform(df))
df_normalized.head()

Task 2.5: Plotting the Distribution After Normalization

Plot the normalized data against the original data and compare the normalized vs the old distributions.

fig, ax = fig.subplots()
DataFrame.plot(ax=ax, kind="density")

fig, ax = plt.subplots()
df.plot(ax=ax, kind='density')
df_normalized.plot(ax=ax, kind='density')

Experiment with Different Normalizations

To review the concepts covered in this step, please refer to the Different Types of Normalization module of the Normalize Data to Make It Appropriate for an Analysis with Pandas course.

Exploring different normalization techniques is essential to understand how each method affects your data. In this step, you will experiment with l1, l2, and max normalizations. These techniques are helpful in various data preprocessing scenarios. We'll use Sklearn to apply these normalization techniques to a randomly generated dataset and observe the differences in data distribution before and after normalization.

Task 3.1: Import Required Libraries

Similar to the previous tasks, import the necessary libraries. This time, include the Normalizer from sklearn.preprocessing.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import Normalizer

Task 3.2: Creating a Random Dataset

Create another DataFrame with 100 rows and 5 columns, filled with random numbers from a uniform distribution in the range [5, 15]. Display the first few rows of this new dataframe.

array = np.random.uniform(5, 15, size=(100, 5))
df = pd.DataFrame(array)
df.head()

Task 3.3: Plotting the Distribution Before Normalization

Plot the distribution of the values in the DataFrame before applying any normalization.

df.plot(kind='density')

Task 3.4: Apply l1 and l2 Normalization

Normalize the data using l1 and l2 normalizations. For each, display the head of the transformed data.

normalizer_l1 = Normalizer(norm='l1')
df_normalized_l1 = pd.DataFrame(normalizer_l1.fit_transform(df))
normalizer_l2 = Normalizer(norm='l2')
df_normalized_l2 = pd.DataFrame(normalizer_l2.fit_transform(df))
print(df_normalized_l1.head())
print(df_normalized_l2.head())

Task 3.5: Apply Max Normalization

Now, apply max normalization to the data and display the first few rows of the transformed data.

normalizer_max = Normalizer(norm='max')
df_normalized_max = pd.DataFrame(normalizer_max.fit_transform(df))
df_normalized_max.head()

Task 3.6: Plotting the Distributions After Normalization

Plot the distributions of the normalized data (l1, l2, and max) to compare how each normalization technique has transformed the data.

fig, ax = plt.subplots()
df.plot(ax=ax, kind='density')

fig, ax = plt.subplots()
df_normalized_l1.plot(ax=ax, kind='density')
df_normalized_l2.plot(ax=ax, kind='density')
df_normalized_max.plot(ax=ax, kind='density')