Validating Data Using Asserts in R

The quality of data plays a crucial role in machine learning. Without good data, errors are generated that adversely affect data analysis and model performance results. Often, these errors are difficult to detect and occur late in the analysis. Still worse, sometimes errors remain undetected and flow in to the data, producing inaccurate results. The solution to this problem lies in data validation. Enter asserts, debugging aids that test a condition and are used to programmatically check data.

In this guide, you will learn to validate data using asserts in R. Specifically, we'll be using the Assertr package, which provides variety of functions designed to verify assumptions about data early in a data analysis pipeline.

In this guide, we'll be using a fictitious dataset of loan applicants containing 600 observations and 10 variables, as described below:

1. Marital_status: Whether the applicant is married ("Yes") or not ("No").

2. Is_graduate: Whether the applicant is graduate ("Yes") or not ("No").

3. Income: Annual Income of the applicant (in USD).

4. Loan_amount: Loan amount (in USD) for which the application was submitted.

5. Credit_score: Whether the applicants credit score is satisfactory or not.

6. approval_status: Whether the loan application was approved ("Yes") or not ("No").

7. Age: The applicant's age in years.

8. Sex: Whether the applicant was a male ("M") or a female ("F").

9. Dependents: Number of dependents in the applicant's family.

10. Purpose: Purpose of applying for the loan.

Let's start by loading the required libraries and the data.

Output:

The example below demonstrates the importance of asserts, in which we summarize the average age of the applicants grouped-by their approval status. The first line of code below converts the approval_status variable into a factor, while the second line performs the required computation.

Output:

There does not seem to be anything wrong in the above output, but let's look at summary function for the Age variable.

Output:

From the output above, we can see that some of the applicants' ages are negative, which is not possible. This is incorrect data, but this error was not detected in the previous code where we performed the group-by operation. This is where the Assertr’s verify() function can be used to ensure that such mistakes don't go unidentified.

The verify function takes a data frame (dat) and a logical expression (Age >= 0). Then, it evaluates that expression for the provided data. If the condition of the expression is not met, verify raises an error alert and terminates further processing of the code pipeline. In this example, the lines of code below will perform this task.

Output:

The output shows ten instances where the age takes negative values, highlighted by the index. Finally, the error message Error: assertr stopped execution shows that the execution was stopped, which is why the desired output was not displayed.

The same task can be performed using Assertr’s assert() function. In the code below, the assert() function takes the data, dat, and applies a predicate function, within_bounds(0,Inf). We have set the range to only include positive values, but this can be altered as necessary. The next step is to apply the predicate function to the column of interest, Age. The code below raises the error alert when the condition is not met.

Output:

The first line of the output, Column 'Age' violates assertion 'within_bounds(0, Inf)' 10 times, indicates that there are ten rows with negative age values.

It can be a time consuming and inefficient to validate data points one at a time using asserts. A more efficient way is to use the family of assert functions and create a chain of such commands for data validation, as shown in the example below.

Let's assume we want to validate the following conditions in our data.

1. The data has all the ten variables described in the initial section of the guide. This is achieved with the verify(has_all_names()) command in the code below.

2. The dataset contains atleast 120 observations, which represents twenty percent of the initial data. This is achieved with the verify((nrow(.) > 120)) command below.

3. The variable Age only takes positive values. This is achieved with the verify(Age > 0) command below.

4. The variables Income and Loan_amount should have values within three standard deviations of their respective means. This is achieved with the insist(within_n_sds(3), Income) command in the code below.

5. The target variable, approval_status, contains only the binary values zero and one. This is achieved with the assert(in_set(0,1), approval_status) command in the code below.

6. Each row in the data contains at most six missing records. This is achieved with the assert_rows(num_row_NAs, within_bounds(0,6), everything()) command below.

7. Each row is unique jointly between the Income, Dependents, approval_status, Age, Sex, Purpose, Loan_amount, and Credit_score variables. This is achieved with the assert_rows(col_concat, is_uniq,...) command below.

Output:

The output shows that the first two requirements are met but the execution was halted in the third condition with the variable,Age taking negative values. Let's make this correction and create a new data frame, dat2, which only takes positive age values. This is done using the code below.

Output:

The resulting data has 590 observations because ten rows containing negative values of age were removed. We'll recheck the combination of the data conditions, specified above, using the code below.

Output:

The output shows that now there is no error alert for negative age values, since those were dropped. Instead, the insist() function found seven records where the Income variable was not within three standard deviations from the mean. The output also prints the index of these records, making it easier for us to treat them as outliers. In this way, we can go on validating the data assumptions and incorporating required corrections if needed.

In this guide, you have learned methods of validating data using asserts in R. You have applied these assertions using two functions, verify() and assert(). This knowledge will help you perform proper data validation, resulting in better data science and analytics results.

To learn more about Data Science with R, please refer to the following guides:

1. Interpreting Data Using Descriptive Statistics with R

2. Interpreting Data Using Statistical Models with R

3. Time Series Forecasting Using R

4. Hypothesis Testing - Interpreting Data with Statistical Models

5. Machine Learning with Text Data Using R

6. Visualization of Text Data Using Word Cloud in R

7. Exploring Data Visually with R

8. Coping with Missing, Invalid and Duplicate Data in R

9. Reshaping Data with R

10. Working with Data Types in R

11. Splitting and Combining Data with R

12. Linear, Lasso, and Ridge Regression with R

13. Binomial Coefficient Analysis with R

14. Implementing Marketing Analytics in R: Part 1

15. Implementing Marketing Analytics in R: Part 2