RStudio Guide

To get started, click on the 'workspace' folder in the bottom right pane of RStudio. Click on the file entitled "Step 1...". You may want to drag the console pane to be smaller so that you have more room to work. You'll complete each task for Step 1 in that R Markdown file. Remember, you must run the cells with the play button at the top right of each cell for a task before moving onto the next task in the R Markdown file. Continue until you have completed all tasks in this step. Then when you are ready to move onto the next step, you'll come back and click on the file for the next step until you have completed all tasks in all steps of the lab.

Exploring and Defining Asserts

To review the concepts covered in this step, please refer to the Introducing Asserts module of the Validating Data Using Asserts in R course.

Understanding and defining asserts is important because they are foundational to identifying and preparing problem data for analysis. This step will help learners grasp the concept of data cleaning and the role of assertions in this process.

Begin your journey into data validation with R by exploring the concept of assertions. In this step, you will:

1. Write a simple R script to define a dataset with potential data quality issues.
2. Use basic assert statements to identify rows with missing values.

Goal: Understand the basic concept of assertions and their role in data cleaning. Tools: R script, basic assert statements.

Task 1.1: Examine a Dataset with Quality Issues

The provided code creates a simple dataset named data_quality_issues that contains some rows with missing values. Print the dataset and examine its structure.

data_quality_issues <- data.frame(
  Name = c('Alice', 'Bob', NA, 'Diana', 'Evan'),
  Age = c(25, NA, 30, 22, 28),
  Score = c(85, 90, 88, NA, 95)
)
print(data_quality_issues)
str(data_quality_issues)

Task 1.2: Verifying Data Integrity with assertr

Utilize the assertr package to verify that the Name column does not contain missing values.

library(assertr)

assert(data_quality_issues, not_na, Name)

Task 1.3: Detecting a Data Problem using assertr

Now use assert to check for missing values in the Age column. Notice the difference in output when assert encounters data that does not satisfy the predicate.

assert(data_quality_issues, not_na, Age)

Validating Column Elements with assert

To review the concepts covered in this step, please refer to the Validating Elements in a Column module of the Validating Data Using Asserts in R course.

Validating elements in a column is crucial because it ensures data integrity and accuracy within individual data points. This step focuses on using the assert function from the assertr package to perform column-wise validation.

Dive deeper into column-wise data validation by:

1. Loading the assertr package in R.
2. Creating a dataset with specific column data types and values.
3. Using the assert function to validate data elements in a column against predefined criteria.

Goal: Practice column-wise data validation using the assert function. Tools: R, assertr package, assert function.

Task 2.1: Load the assertr Package and Create a Dataset

Start by loading the assertr package to use its functions for data validation. Use the provided code to create a simple dataset that contains some data quality issues. Print the dataset and examine its structure.

# Load the assertr Package
library(assertr)

# Provided code to create a simple dataset
data_quality_issues <- data.frame(
  Name = c('Alice', 'Bob', 'John', 'Diana', 'Evan'),
  Age = c(25, NA, 30, 22, 28),
  Score = c(85, 90, 88, NA, 105)
)

Task 2.2: Validate Column Types

Use the assert function from the assertr package to validate that all Score values in the data frame are numeric.

assert(data_quality_issues, is.numeric, Age)

Task 2.3: Validate Column Elements

Use the assert function from the assertr package to validate that all Score values in the data_frame are between 0 and 100.

assert(data_quality_issues, within_bounds(0, 100), Score)

Using insist for Column-Wide Validation

To review the concepts covered in this step, please refer to the Validating Elements Using the Column as a Whole module of the Validating Data Using Asserts in R course.

Validating data using information about the column as a whole is important for understanding the broader context of data points. This step introduces the insist function for such validations.

Expand your validation skills by:

1. Exploring the insist function from the assertr package.
2. Applying the insist function to ensure all values in a column meet the generated criteria.

Goal: Learn to use the insist function for column-wide validation based on aggregate data. Tools: R, assertr package, insist function.

Task 3.1: Loading the Required Packages

Before you can use the insist function for column-wide validation, you need to load the assertr package. In the following steps, we will also use the pipe operator %>%. Load the magrittr package to gain access to this operator

# Load the required packages
library(assertr)
library(magrittr)

Task 3.2: Applying the insist Function

Load the iris dataset using data(iris). Use the insist function to check whether the Sepal.Length column values in the iris dataset fall within 2 standard deviations from the mean.

iris %>%
  insist(within_n_sds(2), Sepal.Length)

Row-wise Data Validation

To review the concepts covered in this step, please refer to the Validating Rows in a Dataset module of the Validating Data Using Asserts in R course.

Row-wise validation is essential for ensuring that data across multiple columns meets certain criteria. This step covers the use of assert_rows and insist_rows functions for comprehensive row-wise checks.

In this task, you'll learn about row-wise data validation by:

1. Understanding the difference between assert_rows and insist_rows functions.
2. Applying row-wise functions to validate data across multiple columns in a dataset.
3. Exploring row reduction functions and predicate generators for advanced validation scenarios.

Goal: Gain proficiency in row-wise data validation techniques. Tools: R, assertr package, assert_rows, insist_rows functions.

Task 4.1: Loading the Required Packages

First, load the assertr package. In the following steps, we will also use the pipe operator %>%. Load the magrittr package to gain access to this operator

# Load the required packages
library(assertr)
library(magrittr)

Task 4.2: Understanding assert_rows and insist_rows

To effectively use assert_rows and insist_rows, it's important to understand their differences and how they work. Use the ? operator to view the documentation for each function.

# View documentation for assert_rows
?assert_rows

# View documentation for insist_rows
?insist_rows

Task 4.3: Checking Missing Values with Row Reduction Functions

Load the airquality dataset using the data() function. Check for any rows that have more than one NA value.

data(airquality)
airquality %>%
  assert_rows(num_row_NAs, within_bounds(0,1), everything()) 

# 2 rows have missing data

Task 4.4: Validating Unique Data with Row Reduction Functions

With the airquality dataset, use assert_rows to ensure that there are no duplicate dates on the columns Month and Day (that is, that all combinations of Month and Day are unique).

airquality %>%
  assert_rows(col_concat, is_uniq, c(Month, Day))

# all Month/Day combinations are unique

Validating Data Frames and Quality Checks

To review the concepts covered in this step, please refer to the Validating a Data Frame module of the Validating Data Using Asserts in R course.

Validating entire data frames is crucial for ensuring the overall integrity and quality of datasets. This step focuses on using assertions to check properties of data frames and perform comprehensive quality checks.

Achieve a higher level of data validation by:

1. Learning to use the verify function from the assertr package to validate properties of entire datasets.
2. Exploring the use of all.equal and identical functions for checking equality between objects.

Goal: Master the validation of entire data frames and performing quality checks. Tools: R, assertr package, verify, all.equal, identical functions.

Task 5.1: Loading the Required Packages

First, load the assertr package. In the following steps, we will also use the pipe operator %>% and the mutate function. Load the magrittr and dplyr packages to gain access to these utilities.

# Load the required packages
library(assertr)
library(magrittr)
library(dplyr)

Task 5.2: Using verify to Validate Data Frame Properties

Load the airquality dataset with the data function. Use the verify function from the assertr package to check that the average of the column Temp in the data frame airquality is greater than 70. This is a basic validation to ensure the data meets a specific condition.

data(airquality)
airquality %>%
  verify(mean(Temp) > 70)

Task 5.3: Comparing Data Frames with all.equal

The provided code loads the iris dataset, then creates a copy and modifies one column. Compare airquality with this new data frame to check if the values are considered equal using the all.equal function.

# Provided code to copy and modify a dataframe
data(iris)
iris2 <- iris %>%
  mutate(Sepal.Length = Sepal.Length + 1e-8)

# Compare the data frames
all.equal(iris, iris2)

# The tiny float difference is ignored by all.equal

Task 5.4: Identifying Exact Matches with identical

Use the identical function to check if the two data frames iris and iris2 are exactly the same. This is a stricter comparison than all.equal and useful for ensuring data integrity.

identical(iris, iris2)

# The data frames are not exactly identical