 Deepika Singh

# Linear, Lasso, and Ridge Regression with R

### Deepika Singh

• Nov 12, 2019
• 205,656 Views
• Nov 12, 2019
• 205,656 Views
Data
R

## Introduction

Machine learning is used by many organizations to identify and solve business problems. The two types of supervised machine learning algorithms are classification and regression. This guide will focus on regression models that predict a continuous outcome. You'll learn how to implement linear and regularized regression models using R.

The topics we'll cover include:

1. Linear Regression

2. Ridge Regression

3. Lasso Regression

4. Elastic Net Regression

Let’s start by looking at a real-life situation and data.

## Data

Unemployment is a critical socio-economic and political concern for any country, and hence, managing it is a chief task for any government. In this guide, we will build regression algorithms for predicting unemployment within an economy.

The data comes from US economic time series data available from http://research.stlouisfed.org/fred2. The data contains 574 rows and 5 variables, as described below:

1. `psavert`: personal savings rate

2. `pce`: personal consumption expenditures, in billions of dollars

3. `uempmed`: median duration of unemployment, in weeks

4. `pop`: total population, in million s

5. `unemploy`: number unemployed, in millions. This is the dependent variable.

### Evaluation Metrics

We will evaluate the performance of the model using two metrics: R-squared value and Root Mean Squared Error (RMSE). Ideally, lower RMSE and higher R-squared values are indicative of a good model.

``````1library(plyr)
3library(dplyr)
4library(caret)
5library(ggplot2)
6library(repr)
7
9glimpse(dat)``````
{r}

Output:

``````1Observations: 574
2Variables: 5
3\$ pce      <dbl> 507.4, 510.5, 516.3, 512.9, 518.1, 525.8, 531.5, 534.2, 544.9, 544.6, 5...
4\$ pop      <dbl> 198.7, 198.9, 199.1, 199.3, 199.5, 199.7, 199.8, 199.9, 200.1, 200.2, 2...
5\$ psavert  <dbl> 12.5, 12.5, 11.7, 12.5, 12.5, 12.1, 11.7, 12.2, 11.6, 12.2, 12.0, 11.6,...
6\$ uempmed  <dbl> 4.5, 4.7, 4.6, 4.9, 4.7, 4.8, 5.1, 4.5, 4.1, 4.6, 4.4, 4.4, 4.5, 4.2, 4...
7\$ unemploy <dbl> 2.9, 2.9, 3.0, 3.1, 3.1, 3.0, 2.9, 3.0, 2.9, 2.7, 2.7, 2.9, 2.9, 2.8, 2...``````

The output shows that all the variables in the dataset are numerical variables (labeled as 'dbl').

## Data Partitioning

We will build our model on the training set and evaluate its performance on the test set. This is called the holdout-validation approach for evaluating model performance.

The first line of code below sets the random seed for reproducibility of results. The second line creates an index for randomly sampling observations for data partitioning. The next two lines of code create the training and test set, while the last two lines print the dimensions of the training and test set. The train set contains 70 percent of the data while the test set contains the remaining 30 percent.

``````1set.seed(100)
2
3index = sample(1:nrow(dat), 0.7*nrow(dat))
4
5train = dat[index,] # Create the training data
6test = dat[-index,] # Create the test data
7
8dim(train)
9dim(test)``````
{r}

Output:

``````1 401   5
2
3 173   5
4 ``````

## Scaling the Numeric Features

The numeric features need to be scaled; otherwise, they may adversely influence the modeling process. The first line of code below creates a list that contains the names of independent numeric variables. The second line uses the `preProcess` function from the caret package to complete the scaling task. The pre-processing object is fit only to the training data, while the scaling is applied on both the train and test sets. This is done in the third and fourth lines of code below. The fifth line prints the summary of the scaled train dataset.

The output shows that now all the numeric features have a mean value of zero except the target variable, `unemploy`, which was not scaled.

``````1cols = c('pce', 'pop', 'psavert', 'uempmed')
2
3pre_proc_val <- preProcess(train[,cols], method = c("center", "scale"))
4
5train[,cols] = predict(pre_proc_val, train[,cols])
6test[,cols] = predict(pre_proc_val, test[,cols])
7
8summary(train)``````
{r}

Output:

``````1      pce               pop             psavert            uempmed           unemploy
2 Min.   :-1.2135   Min.   :-1.6049   Min.   :-1.89481   Min.   :-1.1200   Min.   : 2.700
3 1st Qu.:-0.8856   1st Qu.:-0.8606   1st Qu.:-0.76518   1st Qu.:-0.6198   1st Qu.: 6.500
4 Median :-0.2501   Median :-0.1416   Median :-0.05513   Median :-0.2447   Median : 7.500
5 Mean   : 0.0000   Mean   : 0.0000   Mean   : 0.00000   Mean   : 0.0000   Mean   : 7.791
6 3rd Qu.: 0.7797   3rd Qu.: 0.9060   3rd Qu.: 0.81629   3rd Qu.: 0.1055   3rd Qu.: 8.600
7 Max.   : 2.1244   Max.   : 1.8047   Max.   : 2.91417   Max.   : 4.1571   Max.   :15.300  ``````

## Linear Regression

The simplest form of regression is linear regression, which assumes that the predictors have a linear relationship with the target variable. The input variables are assumed to have a Gaussian distribution and are not correlated with each other (a problem called multi-collinearity).

The linear regression equation can be expressed in the following form: `y = a1x1 + a2x2 + a3x3 + ..... + anxn + b`

In the above equation:

• `y` is the target variable.
• `x1, x2, x3, ... xn` are the features.
• `a1, a2, a3, ... an` are the coefficients.
• `b` is the parameter of the model.

The parameters `a` and `b` in the model are selected through the ordinary least squares (OLS) method. This method works by minimizing the sum of squares of residuals (actual value - predicted value).

In order to fit the linear regression model, the first step is to instantiate the algorithm in the first line of code below using the `lm()` function. The second line prints the summary of the trained model.

``````1lr = lm(unemploy ~ uempmed + psavert + pop + pce, data = train)
2summary(lr)``````
{r}

Output:

``````1Call:
2lm(formula = unemploy ~ uempmed + psavert + pop + pce, data = train)
3
4Residuals:
5    Min      1Q  Median      3Q     Max
6-2.4262 -0.7253  0.0278  0.6697  3.2753
7
8Coefficients:
9            Estimate Std. Error t value Pr(>|t|)
10(Intercept)  7.79077    0.04712 165.352  < 2e-16 ***
11uempmed      2.18021    0.08588  25.386  < 2e-16 ***
12psavert      0.79126    0.13244   5.975 5.14e-09 ***
13pop          5.95419    0.37405  15.918  < 2e-16 ***
14pce         -5.31578    0.32753 -16.230  < 2e-16 ***
15---
16Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
17
18Residual standard error: 0.9435 on 396 degrees of freedom
19Multiple R-squared:  0.8542,	Adjusted R-squared:  0.8527
20F-statistic: 579.9 on 4 and 396 DF,  p-value: < 2.2e-16 ``````

The significance code `‘***’` in above output shows that all the features are important predictors. The Adjusted R-squared value of 0.8527 is also a good result. Let's evaluate the model further.

### Model Evaluation Metrics

The first step is to create a function for calculating the evaluation metrics R-squared and RMSE. The second step is to predict and evaluate the model on train data, while the third step is to predict and evaluate the model on test data.

``````1#Step 1 - create the evaluation metrics function
2
3eval_metrics = function(model, df, predictions, target){
4    resids = df[,target] - predictions
5    resids2 = resids**2
6    N = length(predictions)
7    r2 = as.character(round(summary(model)\$r.squared, 2))
10    print(as.character(round(sqrt(sum(resids2)/N), 2))) #RMSE
11}
12
13# Step 2 - predicting and evaluating the model on train data
14predictions = predict(lr, newdata = train)
15eval_metrics(lr, train, predictions, target = 'unemploy')
16
17# Step 3 - predicting and evaluating the model on test data
18predictions = predict(lr, newdata = test)
19eval_metrics(lr, test, predictions, target = 'unemploy')``````
{r}

Output:

``````1 "0.85"
2
3 "0.94"
4
5
6 "0.85"
7
8 "1.1"
9         ``````

The above output shows that RMSE, one of the two evaluation metrics, is 0.94 million for train data and 1.1 million for test data. On the other hand, R-squared value is around 85 percent for both train and test data, which indicates good performance.

## Regularization

Linear regression algorithm works by selecting coefficients for each independent variable that minimizes a loss function. However, if the coefficients are large, they can lead to over-fitting on the training dataset, and such a model will not generalize well on the unseen test data. To overcome this shortcoming, we'll do regularization, which penalizes large coefficients. The following sections of the guide will discuss various regularization algorithms.

We will be using the `glmnet()` package to build the regularized regression models. The `glmnet` function does not work with dataframes, so we need to create a numeric matrix for the training features and a vector of target values.

The lines of code below perform the task of creating model matrix using the `dummyVars` function from the caret package. The `predict` function is then applied to create numeric model matrices for training and test.

``````1cols_reg = c('pce', 'pop', 'psavert', 'uempmed', 'unemploy')
2
3dummies <- dummyVars(unemploy ~ ., data = dat[,cols_reg])
4
5train_dummies = predict(dummies, newdata = train[,cols_reg])
6
7test_dummies = predict(dummies, newdata = test[,cols_reg])
8
9print(dim(train_dummies)); print(dim(test_dummies))``````
{r}

Output:

``````1 401   4
2 173   4``````

## Ridge Regression

Ridge regression is an extension of linear regression where the loss function is modified to minimize the complexity of the model. This modification is done by adding a penalty parameter that is equivalent to the square of the magnitude of the coefficients.

`Loss function = OLS + alpha * summation (squared coefficient values)`

Ridge regression is also referred to as l2 regularization. The lines of code below construct a ridge regression model. The first line loads the library, while the next two lines create the training data matrices for the independent (x) and dependent variables (y).

The same step is repeated for the test dataset in the fourth and fifth lines of code. The sixth line creates a list of lambda values for the model to try, while the seventh line builds the ridge regression model.

The arguments used in the model are:

1. `nlambda`: determines the number of regularization parameters to be tested.

2. `alpha`: determines the weighting to be used. In case of ridge regression, the value of alpha is zero.

3. `family`: determines the distribution family to be used. Since this is a regression model, we will use the Gaussian distribution.

4. `lambda`: determines the lambda values to be tried.

The last line of code prints the model information.

``````1library(glmnet)
2
3x = as.matrix(train_dummies)
4y_train = train\$unemploy
5
6x_test = as.matrix(test_dummies)
7y_test = test\$unemploy
8
9lambdas <- 10^seq(2, -3, by = -.1)
10ridge_reg = glmnet(x, y_train, nlambda = 25, alpha = 0, family = 'gaussian', lambda = lambdas)
11
12summary(ridge_reg)``````
{r}

Output:

``````1          Length Class     Mode
2a0         51    -none-    numeric
3beta      204    dgCMatrix S4
4df         51    -none-    numeric
5dim         2    -none-    numeric
6lambda     51    -none-    numeric
7dev.ratio  51    -none-    numeric
8nulldev     1    -none-    numeric
9npasses     1    -none-    numeric
10jerr        1    -none-    numeric
11offset      1    -none-    logical
12call        7    -none-    call
13nobs        1    -none-    numeric``````

One of the major differences between linear and regularized regression models is that the latter involves tuning a hyperparameter, lambda. The code above runs the `glmnet()` model several times for different values of lambda. We can automate this task of finding the optimal lambda value using the `cv.glmnet()` function. This is performed using the lines of code below.

``````1cv_ridge <- cv.glmnet(x, y_train, alpha = 0, lambda = lambdas)
2optimal_lambda <- cv_ridge\$lambda.min
3optimal_lambda``````
{r}

Output:

``1 0.001``

The optimal lambda value comes out to be 0.001 and will be used to build the ridge regression model. We will also create a function for calculating and printing the results, which is done with the `eval_results()` function in the code below. The next step is to use the `predict` function to generate predictions on the train and test data. Finally, we use the `eval_results` function to calculate and print the evaluation metrics.

``````1# Compute R^2 from true and predicted values
2eval_results <- function(true, predicted, df) {
3  SSE <- sum((predicted - true)^2)
4  SST <- sum((true - mean(true))^2)
5  R_square <- 1 - SSE / SST
6  RMSE = sqrt(SSE/nrow(df))
7
8
9  # Model performance metrics
10data.frame(
11  RMSE = RMSE,
12  Rsquare = R_square
13)
14
15}
16
17# Prediction and evaluation on train data
18predictions_train <- predict(ridge_reg, s = optimal_lambda, newx = x)
19eval_results(y_train, predictions_train, train)
20
21# Prediction and evaluation on test data
22predictions_test <- predict(ridge_reg, s = optimal_lambda, newx = x_test)
23eval_results(y_test, predictions_test, test)``````
{r}

Output:

``````1       RMSE   Rsquare
21 0.9383428 0.8539379
3      RMSE   Rsquare
41 1.102452 0.8669624``````

The above output shows that the RMSE and R-squared values for the ridge regression model on the training data are 0.93 million and 85.4 percent, respectively. For the test data, the results for these metrics are 1.1 million and 86.7 percent, respectively. There is an improvement in the performance compared with linear regression model.

## Lasso Regression

Lasso regression, or the Least Absolute Shrinkage and Selection Operator, is also a modification of linear regression. In lasso, the loss function is modified to minimize the complexity of the model by limiting the sum of the absolute values of the model coefficients (also called the `l1-norm`).

The loss function for lasso regression can be expressed as below:

`Loss function = OLS + alpha * summation (absolute values of the magnitude of the coefficients)`

In the above function, `alpha` is the penalty parameter we need to select. Using an `l1-norm` constraint forces some weight values to zero to allow other coefficients to take non-zero values.

The first step to build a lasso model is to find the optimal lambda value using the code below. For lasso regression, the alpha value is 1. The output is the best cross-validated lambda, which comes out to be 0.001.

``````1lambdas <- 10^seq(2, -3, by = -.1)
2
3# Setting alpha = 1 implements lasso regression
4lasso_reg <- cv.glmnet(x, y_train, alpha = 1, lambda = lambdas, standardize = TRUE, nfolds = 5)
5
6# Best
7lambda_best <- lasso_reg\$lambda.min
8lambda_best``````
{r}

Output:

``1 0.001``

Once we have the optimal lambda value, we train the lasso model in the first line of code below. The second through fifth lines of code generate the predictions and print the evaluation metrics for both the training and test datasets.

``````1lasso_model <- glmnet(x, y_train, alpha = 1, lambda = lambda_best, standardize = TRUE)
2
3predictions_train <- predict(lasso_model, s = lambda_best, newx = x)
4eval_results(y_train, predictions_train, train)
5
6predictions_test <- predict(lasso_model, s = lambda_best, newx = x_test)
7eval_results(y_test, predictions_test, test)``````
{r}

Output:

``````1       RMSE  Rsquare
21 0.9378347 0.854096
3      RMSE   Rsquare
41 1.099764 0.8676104``````

The above output shows that the RMSE and R-squared values on the training data are 0.93 million and 85.4 percent, respectively. The results on the test data are 1.1 million and 86.7 percent, respectively. Lasso regression can also be used for feature selection because the coeﬃcients of less important features are reduced to zero.

## Elastic Net Regression

Elastic net regression combines the properties of ridge and lasso regression. It works by penalizing the model using both the 1l2-norm1 and the 1l1-norm1. The model can be easily built using the caret package, which automatically selects the optimal value of parameters alpha and lambda.

The first line of code creates the training control object `train_cont` which specifies how the repeated cross validation will take place. The second line builds the elastic regression model in which a range of possible alpha and lambda values are tested and their optimum value is selected. The argument `tuneLength` specifies that 10 different combinations of values for alpha and lambda are to be tested.

The last line of code prints the optimum values, which come out to be 0.4322 for alpha and 0.0065 for lambda.

``````1# Set training control
2train_cont <- trainControl(method = "repeatedcv",
3                              number = 10,
4                              repeats = 5,
5                              search = "random",
6                              verboseIter = TRUE)
7
8# Train the model
9elastic_reg <- train(unemploy ~ .,
10                           data = train,
11                           method = "glmnet",
12                           preProcess = c("center", "scale"),
13                           tuneLength = 10,
14                           trControl = train_cont)
15
16
17# Best tuning parameter
18elastic_reg\$bestTune``````
{r}

Output:

``````1      alpha      lambda
22 0.1996298 0.001183022``````

Once we have trained the model, we use it to generate the predictions and print the evaluation results for both the training and test datasets, using the lines of code below.

``````1# Make predictions on training set
2predictions_train <- predict(elastic_reg, x)
3eval_results(y_train, predictions_train, train)
4
5# Make predictions on test set
6predictions_test <- predict(elastic_reg, x_test)
7eval_results(y_test, predictions_test, test)``````
{r}

Output:

``````1       RMSE  Rsquare
21 0.9495596 0.850425
3      RMSE   Rsquare
41 1.128873 0.8605094``````

The above output shows that the RMSE and R-squared values for the elastic net regression model on the training data are 0.95 million and 85 percent, respectively. The results for these metrics on the test data are 1.12 million and 86 percent, respectively.

## Conclusion

In this guide, you have learned about linear regression models using the powerful R language. You also learned about regularization techniques to avoid the shortcomings of the linear regression models.

The performance of the models is summarized below:

1. Linear Regression Model: Test set RMSE of 1.1 million and R-square of 85 percent.

2. Ridge Regression Model: Test set RMSE of 1.1 million and R-square of 86.7 percent.

3. Lasso Regression Model: Test set RMSE of 1.09 million and R-square of 86.7 percent.

4. ElasticNet Regression Model: Test set RMSE of 1.12 million and R-square of 86 percent.

The regularized regression models are performing better than the linear regression model. Overall, all the models are performing well with decent R-squared and stable RMSE values. The most ideal result would be an RMSE value of zero and R-squared value of 1, but that's almost impossible in real economic datasets.