Validating Machine Learning Models with R

Building machine learning models is an important element of predictive modeling. However, without proper model validation, the confidence that the trained model will generalize well on unseen data can never be high. Model validation helps ensure that the model performs well on new data and helps select the best model, the parameters, and the accuracy metrics.

In this guide, we will learn the basics and implementation of several model validation techniques:

1. Holdout Validation

2. K-fold Cross-Validation

3. Repeated K-fold Cross-Validation

4. Leave-One-Out Cross-Validation

In this guide, we will use a fictitious dataset of loan applicants containing 600 observations and 9 variables, as described below:

1. Marital_status: Whether the applicant is married ("Yes") or not ("No")

2. Is_graduate: Whether the applicant is a graduate ("Yes") or not ("No")

3. Income: Annual income of the applicant (in USD)

4. Loan_amount: Loan amount (in USD) for which the application was submitted

5. Credit_score: Whether the applicant's credit score is good ("Good") or not ("Bad")

6. Approval_status: Whether the loan application was approved ("Yes") or not ("No")

7. Age: The applicant's age in years

8. Sex: Whether the applicant is male ("M") or female ("F")

9. Investment: Total investment in stocks and mutual funds (in USD) as declared by the applicant

Let's start by loading the required libraries and the data.

Output:

The output shows that the dataset has four numerical (labeled as int) and five character variables (labeled as chr). We will convert these into factor variables using the line of code below.

Output:

The holdout validation approach involves creating a training set and a holdout set. The training data is used to train the model, while the holdout data is used to validate model performance. The common split ratio is 70:30, while for small datasets, the ratio can be 90:10.

The first line of code below sets the random seed for reproducibility of results. The second line loads the caTools package that will be used for data partitioning, while the third to fifth lines create the training and test sets. The training set contains 70 percent of the data (420 observations of 10 variables) and the test set contains the remaining 30 percent (180 observations of 10 variables).

Output:

To fit the logistic regression model, the first step is to instantiate the algorithm. This is done in the first line of code below, while the second line generates predictions on the test data. The third line generates the confusion matrix, while the fourth line computes and prints the accuracy.

Output:

We can see that the accuracy of the model on the test data is approximately 87.8 percent. The above technique is useful, but it has pitfalls. The split is very important and, if it goes wrong, it can lead to the model over-fitting or under-fitting the new data. This problem can be rectified using resampling methods, which repeat a calculation multiple times using randomly selected subsets of the complete data. We discuss the popular cross-validation techniques in the following sections of the guide.

In k-fold cross-validation, the data is divided into k folds. The model is trained on k-1 folds with one fold held back for testing. This process gets repeated to ensure each fold of the dataset gets the chance to be the held-back set. Once the process is completed, we can summarize the evaluation metric using the mean and/or the standard deviation.

We will use five-fold cross-validation for our problem statement as specified in the first line of code below. The second line trains the algorithm, while the third line prints the model result.

Output:

The mean accuracy for the model using k-fold cross-validation is 76.17 percent, which is less than the 88 percent accuracy achieved using the holdout validation approach.

The process of splitting the data into k folds can be repeated a number of times. This is called repeated k-fold cross-validation*, in which the final model accuracy is taken as the mean of the number of repeats.

The following lines of code use 5-fold cross validation with 3 repeats to estimate Naive Bayes on the dataset.

Output:

The mean accuracy for the model using repeated k-fold cross-validation is 75.94 percent.

Leave-one-out cross-validation, or LOOCV, is the cross-validation technique in which the size of the fold is “1” with “k” being set to the number of observations in the data. This variation is useful when the training data is of limited size and the number of parameters to be tested is not high. The lines of code below repeat the steps above.

Output:

The mean accuracy for the model using leave-one-out cross-validation is 77 percent.

In this guide, you have learned about the various model validation techniques in R. The mean accuracy result for the techniques is summarized below:

1. Holdout Validation Approach: Accuracy of 88%

2. K-fold Cross-Validation: Mean Accuracy of 76%

3. Repeated K-fold Cross-Validation: Mean Accuracy of 76%

4. Leave-One-Out Cross-Validation: Mean Accuracy of 77%

To learn more about data science using R, please refer to the following guides:

1. Interpreting Data Using Descriptive Statistics with R

2. Interpreting Data Using Statistical Models with R

3. Time Series Forecasting Using R

4. Hypothesis Testing - Interpreting Data with Statistical Models

5. Machine Learning with Text Data Using R

6. Visualization of Text Data Using Word Cloud in R

7. Exploring Data Visually with R

8. Coping with Missing, Invalid and Duplicate Data in R

9. Reshaping Data with R

10. Working with Data Types in R

11. Splitting and Combining Data with R

12. Linear, Lasso, and Ridge Regression with R