Author avatar

Deepika Singh

Predictive Analytics with PyTorch

Deepika Singh

  • Apr 1, 2020
  • 13 Min read
  • 4,343 Views
  • Apr 1, 2020
  • 13 Min read
  • 4,343 Views
Data
Pytorch
Introduction

Introduction

PyTorch is an open-source machine learning library that is widely used for developing predictive models. Predictive modeling is the phase of analytics that uses statistical algorithms to predict outcomes. The model takes data containing independent variables as inputs, and using machine learning algorithms, makes predictions for the target variable. It is often used by statisticians, data science, and machine learning professionals to make predictions.

In this guide, you will learn the basics of building predictive models using Pytorch.

Data

In this guide, you'll use a fictitious dataset of loan applicants containing 600 observations and 8 variables, as described below:

  1. Is_graduate: Whether the applicant is graduate ("1") or not ("0").

  2. Income: Annual Income of the applicant (in USD).

  3. Loan_amount: Loan amount (in USD) for which the application was submitted.

  4. Credit_score: Whether the applicants credit score is satisfactory ("1") or not ("0").

  5. approval_status: Whether the loan application was approved ("1") or not ("0").

  6. Age: The applicant's age in years.

  7. Sex: Whether the applicant is female ("1") or a male ("0").

  8. Investment: Total investment in stocks and mutual funds (in USD) as declared by the applicant.

Let's start by loading the baseline libraries. 

1import pandas as pd
2import numpy as np 
3import matplotlib.pyplot as plt
4import sklearn
5
6from sklearn.model_selection import train_test_split
7from sklearn.metrics import mean_squared_error
8from math import sqrt
python

After installing libraries, the next step is to load data and look at the basic statistics of the variables. 

1df = pd.read_csv('data.csv') 
2print(df.shape)
3df.describe()
python

Output:

1 
2(600, 8)
3
4
5|       	| Is_graduate 	| Income        	| Loan_amount   	| Credit_score 	| Age        	| Sex        	| Investment    	| approval_status 	|
6|-------	|-------------	|---------------	|---------------	|--------------	|------------	|------------	|---------------	|-----------------	|
7| count 	| 600.000000  	| 600.000000    	| 600.000000    	| 600.000000   	| 600.000000 	| 600.000000 	| 600.000000    	| 600.000000      	|
8| mean  	| 0.690000    	| 65861.466667  	| 145511.975833 	| 0.696667     	| 48.701667  	| 0.185000   	| 34417.668333  	| 0.683333        	|
9| std   	| 0.462879    	| 48628.106723  	| 86728.364583  	| 0.460082     	| 14.778362  	| 0.388622   	| 29742.580389  	| 0.465564        	|
10| min   	| 0.000000    	| 3000.000000   	| 6000.000000   	| 0.000000     	| 22.000000  	| 0.000000   	| 2100.000000   	| 0.000000        	|
11| 25%   	| 0.000000    	| 38175.000000  	| 111232.500000 	| 0.000000     	| 35.000000  	| 0.000000   	| 16678.000000  	| 0.000000        	|
12| 50%   	| 1.000000    	| 50080.000000  	| 134295.000000 	| 1.000000     	| 49.000000  	| 0.000000   	| 26439.000000  	| 1.000000        	|
13| 75%   	| 1.000000    	| 76040.000000  	| 168715.000000 	| 1.000000     	| 61.000000  	| 0.000000   	| 35000.000000  	| 1.000000        	|
14| max   	| 1.000000    	| 317370.000000 	| 466660.000000 	| 1.000000     	| 76.000000  	| 1.000000   	| 190422.000000 	| 1.000000        	|

Data Preparation

Before initiating the model, it is important to prepare the data. The lines of code below create arrays for the features and response variable. 

1target_column = ['approval_status'] 
2predictors = list(set(list(df.columns))-set(target_column))
3
4print(target_column)
5print(predictors)
python

Output:

1 ['approval_status']
2 
3 ['Sex', 'Credit_score', 'Age', 'Investment', 'Income', 'Loan_amount', 'Is_graduate']

The next step is to create the train and test datasets. This is done using the code below. The last line of code prints the shape of the training set (420 observations of 7 variables) and test set (180 observations of 7 variables). 

1X = df[predictors].values
2y = df[target_column].values
3
4X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 30)
5print(X_train.shape); print(X_test.shape)
python

Output:

1 (420, 7)
2 
3 (180, 7)

Model Building

You have created the train and test sets and are ready to train the model. You'll start by importing the required libraries to work with Pytorch library. 

1import torch
2import torch.utils.data
3import torch.nn as nn
4import torch.nn.functional as F
5from torch.autograd import Variable
python

You are all set to build the predictive model using the Artificial Neural Network (or ANN) algorithm. The basic architecture of an ANN consists of three main components.

  1. Input Layer: This is where the training observations are fed.

  2. Hidden Layers: These are the intermediate layers between the input and output layers. The model learns about the relationships involved in data in these layers.

  3. Output Layer: This is the layer where the final output is extracted from what’s happening in the previous layers.

The first step for creating the ANN model is to create a class, ANN, that inherits from the nn.Module class. The next step is to define the layers of the network using the __init__() method.

In this case, the model has four layers. Each layer will expect the first parameter to be the input size, which is seven in this case. You'll repeat the process for the remaining layers. The only change in the last stage will be that the output is one variable, representing the target column. You'll also add a dropout layer to avoid overfitting.

Once you have defined the layers, then define how they interact with each other with the def forward(self, x) function, as shown below. This means you're building a fully connected, feed-forward neural network that goes from input to output in a forward manner. The forward step begins with the activation function relu, or Rectified Linear Activation.

For the output layer, you'll use the sigmoid function to convert the probabilities to the classes one and zero. 

1class ANN(nn.Module):
2    def __init__(self, input_dim = 7, output_dim = 1):
3        super(ANN, self).__init__()
4        self.fc1 = nn.Linear(input_dim, 64)
5        self.fc2 = nn.Linear(64, 64)
6        self.fc3 = nn.Linear(64, 32)
7        self.fc4 = nn.Linear(32, 32)
8        self.output_layer = nn.Linear(32,1)
9        self.dropout = nn.Dropout(0.15)
10        
11     def forward(self, x):
12        x = F.relu(self.fc1(x))
13        x = F.relu(self.fc2(x))
14        x = self.dropout(x)
15        x = F.relu(self.fc3(x))
16        x = F.relu(self.fc4(x))
17        x = self.output_layer(x)
18        
19        return nn.Sigmoid()(x)
python

Now that you have defined the architecture of the model above, instantiate the model using the code below. 

1model = ANN(input_dim = 7, output_dim = 1)
2
3print(model)
python

Output:

1ANN(
2      (fc1): Linear(in_features=7, out_features=64, bias=True)
3      (fc2): Linear(in_features=64, out_features=64, bias=True)
4      (fc3): Linear(in_features=64, out_features=32, bias=True)
5      (fc4): Linear(in_features=32, out_features=32, bias=True)
6      (output_layer): Linear(in_features=32, out_features=1, bias=True)
7      (dropout): Dropout(p=0.2, inplace=False)
8    )

You've created the model, and now you need to make the data ready for the Pytorch library. The lines of code below carry out the conversion on the train and test arrays. 

1X_train = torch.from_numpy(X_train)
2y_train = torch.from_numpy(y_train).view(-1,1)
3
4
5X_test = torch.from_numpy(X_test)
6y_test = torch.from_numpy(y_test).view(-1,1)
python

The next step is to make this data iterable. In simple terms, this means that the model will iterate over the dataset to generate predictions. You'll use the torch.utils API provided by Pytorch to perform this task, as shown below. 

1train = torch.utils.data.TensorDataset(X_train,y_train)
2test = torch.utils.data.TensorDataset(X_test,y_test)
3
4train_loader = torch.utils.data.DataLoader(train, batch_size = 64, shuffle = True)
5test_loader = torch.utils.data.DataLoader(test, batch_size = 64, shuffle = True)
python

Model Evaluation

The fully connected ANN is ready for predictive modeling, and you've transformed the train and test arrays in the format required by Pytorch. Model evaluation is the next step. This is done by computing loss, which essentially measures the distance between the predicted and actual labels. In this case, use Binary Cross-Entropy Loss using the nn.BCELoss() function. You also need to optimize the network using the stochastic gradient descent optimizer. This is done using the lines of code below. The lr argument specifies the learning rate of the optimizer function. 

1import torch.optim as optim
2loss_fn = nn.BCELoss()
3optimizer = optim.SGD(model.parameters(), lr=0.001, weight_decay= 1e-6, momentum = 0.8)
python

After defining the loss function, the next step is to perform model evaluation on the training data using the code below. Start by defining the epoch in the first line of code, while lines two to six create lists that'll keep track of loss and accuracy during each epoch. The code from line seven onwards is used to train the model, calculate loss and accuracy for each epoch, and finally print the output. 

1# lines 1 to 6
2epochs = 2000
3epoch_list = []
4train_loss_list = []
5val_loss_list = []
6train_acc_list = []
7val_acc_list = []
8
9# lines 7 onwards
10model.train() # prepare model for training
11
12for epoch in range(epochs):
13    trainloss = 0.0
14    valloss = 0.0
15    
16    correct = 0
17    total = 0
18    for data,target in train_loader:
19        data = Variable(data).float()
20        target = Variable(target).type(torch.FloatTensor)
21        optimizer.zero_grad()
22        output = model(data)
23        predicted = (torch.round(output.data[0]))
24        total += len(target)
25        correct += (predicted == target).sum()
26
27        loss = loss_fn(output, target)
28        loss.backward()
29        optimizer.step()
30        trainloss += loss.item()*data.size(0)
31
32    trainloss = trainloss/len(train_loader.dataset)
33    accuracy = 100 * correct / float(total)
34    train_acc_list.append(accuracy)
35    trainloss_list.append(train_loss)
36    print('Epoch: {} \tTraining Loss: {:.4f}\t Acc: {:.2f}%'.format(
37        epoch+1, 
38        train_loss,
39        accuracy
40        ))
41    epoch_list.append(epoch + 1)
python

Output:

1#Truncated output for sake of brevity
2
3    Epoch: 1 	Training Loss: 10.3845	 Acc: 62.86%
4    Epoch: 2 	Training Loss: 9.0788	 Acc: 67.14%
5    Epoch: 3 	Training Loss: 9.0788	 Acc: 67.14%
6    Epoch: 4 	Training Loss: 9.0788	 Acc: 67.14%
7    Epoch: 5 	Training Loss: 9.0788	 Acc: 67.14%
8    
9    Epoch: 1996 	Training Loss: 9.0788	 Acc: 67.14%
10    Epoch: 1997 	Training Loss: 9.0788	 Acc: 67.14%
11    Epoch: 1998 	Training Loss: 9.0788	 Acc: 67.14%
12    Epoch: 1999 	Training Loss: 9.0788	 Acc: 67.14%
13    Epoch: 2000 	Training Loss: 9.0788	 Acc: 67.14%

The output shows that the training data accuracy is around 67 percent. You'll now evaluate the model performance of the test data using the lines of code below. 

1correct = 0
2total = 0
3valloss = 0
4model.eval() 
5
6with torch.no_grad():
7    for data, target in test_loader:
8        data = Variable(data).float()
9        target = Variable(target).type(torch.FloatTensor)
10
11        output = model(data)
12        loss = loss_fn(output, target)
13        valloss += loss.item()*data.size(0)
14        
15        predicted = (torch.round(output.data[0]))
16        total += len(target)
17        correct += (predicted == target).sum()
18    
19    valloss = valloss/len(test_loader.dataset)
20    accuracy = 100 * correct/ float(total)
21    print(accuracy)
python

Output:

1 0.7111111450195312

The above output shows that the test set accuracy comes out to be 71 percent. You can further fine-tune the algorithm to improve model performance.

LEARN MORE