Model Evaluation: Understanding Regression Metrics

Table of Contents

1. Challenge

   Introduction

   Welcome to the Model Evaluation Code Lab. In this hands-on lab, you'll learn to compute and interpret key regression metrics, analyze prediction errors, and visualize model performance. By the end, you'll confidently evaluate regression models for real-world applications.

   Regression is a type of machine learning where models predict continuous numerical values rather than categories. Examples include predicting house prices, stock values, temperature, or sales revenue. Unlike classification (where you simply count correct vs incorrect predictions), regression requires measuring how close predictions are to actual values.

   Regression metrics quantify prediction accuracy in different ways. Some metrics like MAE (Mean Absolute Error) give you errors in the original units (dollars), while others like R-squared show how much variance your model explains. Understanding multiple metrics helps you choose the right model for your specific business needs and communicate results to stakeholders.

   IMPORTANT: Before starting any tasks, run the setup script in the terminal:

   chmod +x setup.sh
   ./setup.sh
   

   Wait for the setup to complete (it may take 1-2 minutes). You'll see "Setup complete!" when ready.

   Background

   You're a data scientist at CarvedRock predicting property prices. The business needs accurate price estimates to remain competitive. Your team has trained three regression models on housing data: Linear Regression, Ridge Regression, and Random Forest. Your job is to compute evaluation metrics, analyze prediction errors, and create visualizations to recommend the optimal model for deployment.

   This lab uses the California Housing dataset, which contains information about housing districts in California from the 1990 census. Each record represents a district with features like median income, house age, average rooms, average bedrooms, population, average occupancy, latitude, and longitude. The target variable is the median house value for each district, ranging from around $15,000 to $500,000. With over 20,000 samples, this dataset provides a realistic foundation for practicing regression model evaluation.

   Familiarizing with the Program Structure

   The lab environment includes the following key files:

   1. data_loader.py: Loads and prepares the housing price dataset
   2. model_trainer.py: Contains pre-trained regression models
   3. metrics_calculator.py: Computes regression metrics (MAE, MSE, RMSE, MSLE, R-squared)
   4. error_analyzer.py: Analyzes prediction errors and residuals
   5. visualization.py: Creates actual vs predicted plots and residual visualizations
   6. evaluate_models.py: Main script to run full model evaluation
   7. compare_models.py: Compares all models and generates recommendations

   The environment uses Python 3.x with scikit-learn for machine learning, NumPy for numerical operations, and Matplotlib for visualization. All dependencies are pre-installed in the lab environment.

   To run scripts, use the terminal with commands like python3 evaluate_models.py. The results will be saved to the output/ directory.

   Important Note: Complete tasks in order. Each task builds on the previous one. Test your code frequently by running the provided scripts to catch errors early.

   info > If you get stuck at any point during the lab, there is a solution folder containing complete solution files in your filetree.

2. Challenge

   Understanding Data and Basic Metrics

   Understanding Data Preparation

   Before evaluating any model, you need properly split data. The test set must remain unseen during training to give honest performance estimates. Using a fixed random seed ensures reproducibility - you'll get the same split every time, making debugging easier and results comparable.

   In machine learning, you typically split data into training and testing sets. The training set teaches the model patterns, while the test set measures how well those patterns generalize to new, unseen data. ### Understanding Regression Metrics

   Unlike classification, regression predicts continuous values. You can't simply count correct/incorrect predictions. Instead, you measure how far predictions are from actual values.

   Key regression metrics include:

   • MAE (Mean Absolute Error) measures average absolute difference between predictions and actual values - easy to interpret in original units
   • MSE (Mean Squared Error) penalizes larger errors more heavily by squaring differences
   • RMSE (Root Mean Squared Error) is the square root of MSE, bringing the metric back to original units
   • R-squared indicates what proportion of variance in the target your model explains (1.0 = perfect, 0 = no better than mean)
   • MSLE (Mean Squared Logarithmic Error) penalizes under-predictions more than over-predictions, useful when you care more about relative errors

   For property price prediction, RMSE tells you the typical dollar amount your predictions might be off by.

3. Challenge

   Advanced Error Metrics

   Understanding Percentage-Based Metrics

   Dollar-based errors like MAE don't tell the full story. For example, a $10,000 error on a $100,000 house (10%) is very different from a $10,000 error on a $1,000,000 house (1%).

   Percentage-based metrics provide context:

   • MAPE (Mean Absolute Percentage Error) expresses errors as percentages of actual values
   • MBE (Mean Bias Error) reveals systematic over or under-prediction

   MAPE helps business stakeholders understand model accuracy in relative terms, while MBE helps identify if your model consistently predicts too high or too low.

4. Challenge

   Analyzing Prediction Errors

   Understanding Residual Analysis

   Residuals are the differences between actual and predicted values. Analyzing residuals helps you understand where your model succeeds and fails.

   A well-behaved model should have residuals that are randomly scattered around zero with no obvious patterns. Patterns in residuals suggest the model is missing important relationships in the data. ### Understanding Outlier Detection

   Summary statistics like mean and standard deviation give you an overall picture, but they can hide important details. Outliers are predictions that deviate significantly from the norm - typically defined as errors beyond 2 standard deviations from the mean.

   Identifying outliers matters because these extreme errors often represent edge cases: luxury properties, unusual locations, or data quality issues. Understanding where your model struggles helps you decide whether to collect more data, add features, or use specialized models for these cases.

5. Challenge

   Visualizing Model Performance

   Understanding Visualization for Decision Making

   Numbers alone can be hard to interpret. Visualizations make model performance tangible:

   • Actual vs Predicted plots show how well predictions align with true values (points along the diagonal line indicate perfect predictions)
   • Residual plots reveal patterns in errors that metrics might miss

   These visual tools are essential for presenting findings to stakeholders and identifying areas for model improvement. ### Understanding Residual Distribution

   While the scatter plot shows individual predictions, a histogram reveals the overall pattern of errors. A well-behaved regression model should have residuals that follow a normal (bell-shaped) distribution centered at zero.

   If the histogram is skewed left, your model tends to over-predict. If skewed right, it under-predicts. A wide, flat distribution indicates inconsistent predictions, while a tall, narrow peak shows the model makes consistently small errors. These visual patterns help you diagnose model issues that raw numbers might miss.

6. Challenge

   Model Comparison and Recommendation

   Understanding Model Selection

   The "best" model depends on your priorities. For property pricing:

   • Low RMSE means typical predictions are close to actual values
   • High R-squared means the model explains most price variation
   • Low MAPE means percentage errors are small

   This final task brings everything together: evaluate all models with the metrics you've implemented and make a justified recommendation based on the business context.

7. Challenge

   Conclusion

   Congratulations on completing the Model Evaluation lab! You've successfully learned to compute, interpret, and visualize regression metrics for real-world property price prediction.

   What You've Accomplished

   Throughout this lab, you have:

   1. Configured Data Splitting: Set up reproducible train/test splits for honest model evaluation.
   2. Computed Basic Metrics: Calculated MAE, MSE, RMSE, MSLE and R-squared to evaluate model performance.
   3. Implemented Advanced Metrics: Added MAPE and MBE for percentage-based error analysis.
   4. Analyzed Residuals: Computed and interpreted residual statistics to understand error patterns.
   5. Identified Outliers: Located problematic predictions that exceed normal error bounds.
   6. Created Visualizations: Built actual vs predicted plots and residual distributions.
   7. Compared Multiple Models: Evaluated three regressors and recommended the best one for deployment.

   Key Takeaways

   • Multiple Metrics Matter: No single metric tells the whole story - use MAE, RMSE, and R-squared together.
   • Context Determines Priority: For property pricing, RMSE in dollars matters most to stakeholders.
   • Residual Analysis Reveals Patterns: Random residuals indicate a well-specified model.
   • Visualizations Aid Communication: Plots help non-technical stakeholders understand model quality.
   • Percentage Metrics Add Perspective: MAPE helps compare errors across different price ranges.

   Experiment Before You Go

   You still have time in the lab environment. Try these explorations:

   • Compare different regression algorithms (Lasso, ElasticNet)
   • Analyze residuals for different price ranges
   • Create custom metrics weighted by property value
   • Experiment with different train/test split ratios

   Take this opportunity to experiment and deepen your understanding!