Build and Compare Anomaly Detection Models

Table of Contents

1. Challenge

   Step 1: Introduction

   Welcome to the Build and Compare Anomaly Detection Models Code Lab!

   In this hands-on lab, you will build and compare anomaly detection models for sensor measurements collected at a chemical manufacturing plant. Your work centers around three practical concerns:

   • Data preparation: Inspect the measurements and prepare the sensor values for use by the detectors.
   • Detection methods: Implement three methods for calculating how unusual each record looks.
   • Comparison and evaluation: Examine which records the methods flag, check their decisions against the supplied labels, and choose a threshold that controls the amount of work sent for review.

   Background

   You are part of an analytics team supporting a chemical manufacturing plant. The operations team reviews historical sensor measurements to identify anomalous process behavior. The team has limited time to investigate flagged records and needs a way to prioritize its work. You've been asked to compare detection methods and recommend how to decide which measurements to investigate.

   Your comparison will help the team understand where the methods agree, which unusual measurements they miss, and how many false alarms they produce. These results will help the team judge how much investigation each method requires and what it can detect.

   The dataset

   The data comes from NoBOOM, a published dataset of sensor measurements from chemical processes. Each row contains readings from 244 sensors and an existing label marking the row as normal or anomalous. These labels let you check which anomalies the detectors find, which they miss, and how many normal records they flag.

   NoBOOM supplies these measurements as one training file and two labeled test files, already available in data/:

   • Training data contains only measurements labeled normal. These measurements establish the normal behavior that the detectors use as their reference.
   • Validation data contains both normal and anomalous measurements. You use this file to compare the detectors and choose how many records to send for review.
   • Test data contains a separate set of normal and anomalous measurements. After making your choice, you use this file to check how well that choice works on other measurements.

   Project structure

   You will complete functions in three files:

   • prepare_data.py: Describes the sensor data and prepares the measurements for the models.
   • score_models.py: Calculates an anomaly score for each record using three detection methods.
   • compare_models.py: Compares the scores, calculates evaluation metrics, and selects a model and alert threshold.

   The lab also includes provided support code and data:

   • run_lab.py: Runs your completed functions, prints their results, and saves the prepared data, model scores, and output files used in later steps.
   • support/: Helpers for reading data and producing the tables and charts used throughout this lab.
   • data/: The three sensor data files and the dataset's license notice.

   You may also notice test_anormal_experiment_007_b.csv.gz.part.* files in data/. These are setup files used to rebuild the test data file when the environment starts, so you can ignore them.

   The environment uses Python 3.10.6 with NumPy, pandas, scikit-learn, and Matplotlib, all pre-installed. All commands in this lab assume your working directory is /home/ps-user/workspace.

   How to complete the tasks

   The functions' names, parameters, and return statements are already in place. TODO comments identify the parts you will complete, and inline comments mark each placeholder and describe the kind of value it should hold:

   features = None  # Placeholder for the sensor DataFrame
   feature_types = {}  # Placeholder for the dictionary of sensor data types
   

   Replace the marked placeholders with your code, and keep the variable names intact so the existing return statements and supporting code use your calculated values.

   Each editable script includes an if __name__ == "__main__": block at the bottom with code that calls your functions and prints their results. When you reach a task, uncomment its section in the block, then run the file to check your output. You can also add your own code to the block to inspect the data or experiment with each function.

   info> Important Note: Complete tasks in order. Each task builds on the previous one. Use the Feedback/Checks panel frequently to catch errors early.

2. Challenge

   Step 2: Prepare data and generate anomaly scores

   Inspecting the measurements

   Take a closer look at the plant's measurements. The operations team is looking for unusual behavior, but how much of it is recorded in these files? The existing labels let you find out. Counting the normal and anomalous records will show you how common anomalies are in each file and give you a way to put the detectors' results in context later. You'll also check the sensor columns for their data types and missing values to see what needs attention when preparing the measurements.

   To count these records, you will use the Anomaly column. This column contains the labels 0 and 1. A value of 0 identifies a normal record, and 1 identifies an anomalous record. The other columns contain the sensor readings. Keep Anomaly separate when selecting the sensor inputs so its labels are used only to assess the detectors' decisions.

   Including Anomaly as an input would cause data leakage: the detectors would receive the answer they are meant to find, so their results would look better than they would on new measurements.

   Alongside the counts, report the fraction of records labeled anomalous. This is the anomaly prevalence, calculated by dividing the anomaly count by the total row count. The normal-to-anomaly ratio compares the two counts by dividing normal rows by anomalous rows. Use None for this ratio when the anomaly count is zero since division by zero is undefined. Run the profile checkpoint:

   cd /home/ps-user/workspace/code
   python3 run_lab.py profile
   

   In the terminal output, you should see:

   train: 21,943 rows, 244 sensors, 0 anomalies (0.00%), 0 missing cells
     Sensor types: 243 float64, 1 int64
     Normal-to-anomaly ratio: None
   validation: 9,833 rows, 244 sensors, 258 anomalies (2.62%), 0 missing cells
     Sensor types: 243 float64, 1 int64
     Normal-to-anomaly ratio: 37.11
   test: 209,451 rows, 244 sensors, 39,008 anomalies (18.62%), 0 missing cells
     Sensor types: 243 float64, 1 int64
     Normal-to-anomaly ratio: 4.37
   

   Here, run_lab.py profile called your completed profile_data function on the three data splits so you can examine the sensor types, missing values, and normal-to-anomaly ratio across all the data you'll use. It also saved the full results in /home/ps-user/workspace/output/profiles.json.

   Interpreting the profile

   Notice that train has no anomalies. The detector models will learn what normal sensor measurements look like from the train split, then use that reference to score other measurements. If you were to include anomalies in the train split, this could make them part of the normal behavior the models learn. This is why the dataset provides only normal records for this purpose.

   Now compare the normal-to-anomaly ratios in the validation and test splits. They both have more normal records than anomalies, but validation has a much stronger class imbalance. Now this can make a detector look deceptively successful. If it leaves every record unflagged, it gets all the normal records right. And, because normal records make up most of the data, it can achieve high accuracy while missing every anomaly.

   What about missing sensor values? Your results show zero missing cells in all three splits. If there were missing values, one way to handle them would be to use an imputer, which replaces each missing value with a value calculated from the available measurements. For example, if a value were missing from a sensor column, you could replace it with the median of the available values from that column.

   Because these files contain no missing values, the preparation in Task 2 doesn’t need an imputer. If you add one for other data, fit it on the train split only, as you will with the scaler, so validation and test values don’t influence it. ### Preparing the sensor values

   It was reported that the plant's sensors record values on different numerical scales. This means that if you calculate distances directly from those readings, a sensor whose readings vary over a wide numerical range can dominate the result even when those changes are ordinary for that sensor. To mitigate this, you'll use robust scaling to express each sensor's readings relative to its usual level and spread. For each sensor, RobustScaler subtracts the training median and divides by the training interquartile range. The interquartile range is the difference between the 75th and 25th percentiles.

   But why use the median and interquartile range? Extreme values have less influence on these statistics than on the mean and standard deviation, so they have less influence on the scale used to express each sensor's readings. These extreme readings are important. They may well be the anomalies your operations team needs to investigate, so you'll scale them along with the other readings and keep every row.

   You will fit the scaler on train to calculate those medians and interquartile ranges. Once you return the fitted scaler, the runner uses it to transform train, validation, and test. This keeps the scaling based only on the train split. If you recalculated the medians and interquartile ranges using the validation or test splits, their anomalies could influence the scaling too. Run the preparation checkpoint:

   cd /home/ps-user/workspace/code
   python3 run_lab.py prepare
   

   You should see progress messages followed by arrays with shapes (21943, 244), (9833, 244), and (209451, 244) for train, validation, and test.

   In each shape, the first number is the number of records and the second is the number of sensor columns. The runner saves these prepared arrays for the scoring functions. To stay within the platform's memory constraints, the runner prepares and scores the data in small batches, keeping every row.

   In your own setup, you most likely have enough memory available, so you can process the full dataset at once.

3. Challenge

   Step 3: Implement the detectors

   Great! Your sensor measurements are now ready. Now, which records should the operations team investigate? You don't yet have a score that tells the team how unusual each record looks. To produce one, you'll implement anomaly detectors. Each detector assigns an anomaly score to a record, a number that expresses how unusual its sensor readings look according to that method. In this lab, higher scores will mean more unusual records, giving the team a way to prioritize its investigations.

   You'll start by building a statistical baseline using z-scores to measure how far individual sensor readings are from their training averages. This gives the operations team a simple detector whose results you can use to assess whether other methods improve detection. You'll then implement k-nearest-neighbor distance scoring, which finds records far from examples of normal operation, and Isolation Forest, which looks for records that are easy to separate through random splits of the sensor values.

   Now, you will implement these detectors in the three functions in score_models.py. They use the data you scaled in Task 2, which the runner passes in as NumPy arrays. You'll see two parameters in each function, train and query. Here, train contains (as you might expect) the training data, and query has the rows you want to score. The runner will supply validation as query in one call and test in another. The functions you code should return one score for every row in query.

   Scoring with z-scores

   As discussed, this will be your baseline. When you evaluate the detectors, you'll be checking whether the other more elaborate methods catch anomalies that this method misses and whether they produce more false alarms.

   So, for each sensor value (every column), calculate its absolute difference from that sensor's training mean, then divide by its training standard deviation. A large result means the value is far from the training average relative to the variation seen during training.

   When you're done with each sensor you'll be left with 244 absolute z-scores for each row. Not quite celebration time yet, you actually need only one score per row. Hence, you'll use the largest absolute z-score among its sensors. This allows one unusually large deviation to raise the record's anomaly score.

   For the calculation, use the population standard deviation, which is NumPy's default with ddof=0. There's one case to account for here. If a sensor has the same value in every training record, its standard deviation is zero. Replace that zero with 1 to keep the division defined. Calculate the mean and standard deviation from the prepared training values passed to the function. Run your z-score detector:

   cd /home/ps-user/workspace/code
   python3 run_lab.py score --model ZScore
   

   You should see a progress message followed by a short table for validation, then another for test. Each table shows the first three rows' scores under ZScore.

   The runner saves the scores for all rows in validation_scores.csv and test_scores.csv in /home/ps-user/workspace/output. ### Scoring with k-Nearest-Neighbor distances

   You've now scored records by looking for individual feature values far from their training averages. Now, imagine two sensors that usually stay close in the train split. A row where one reads high and the other reads low could be unusual, even if neither value is far enough from its training average to produce a large z-score. The baseline checks each feature column separately, so it misses this relationship. Now, how could you compare the values together? With k-nearest neighbors, or kNN, you'll compare each query row with training rows across all feature columns. This method uses Euclidean distance to find the closest training rows.

   Choosing the number of neighbors

   Consider how you could use these distances to score a query row. You again want higher values to indicate a more anomalous record. As a starting point, you could take the distance to the nearest training row. But this only tells you about one match. It could be small even when every remaining training row is far away, so you'd miss the difference between having one similar row nearby and having a whole group. This is where the k comes in.

   You will find the nearest k training rows and use the distance to the last of those neighbors as the score. If the kth nearest training row is close, there are at least k normal training rows nearby, and the anomaly score is low. If reaching that neighbor requires a large distance, the score is higher. To choose k, consider what happens when its value is large. This would require more training rows nearby to keep the score low, potentially giving higher scores to rows from smaller clusters of normal data. Conversely, too few neighbors could give a low score based on only a handful of similar rows that might be anomalous altogether. Here, you will use 20. Mind you, this is not a universally correct choice; choosing k for another dataset involves comparing validation results for different values. But you won't get into this here.

   Keep in mind that the provided neighbor estimator returns distances in ascending order. Its distance array has one row for each record being scored and k columns, one for each neighbor. Run your kNN detector. This command can take a few minutes. The terminal will report when it starts scoring each split.

   cd /home/ps-user/workspace/code
   python3 run_lab.py score --model kNN
   

   The preview should look familiar from Task 3, with kNN getting its turn this time. The runner adds its scores to the same two CSV files alongside ZScore. ### Scoring with Isolation Forest

   For the team's third scoring option, you'll use Isolation Forest. This method divides the data into smaller groups and looks at how many divisions it takes to leave a row on its own. A row that differs from the others can often be separated in just a few divisions, which gives you another way to look for anomalies.

   Look at what one of those divisions does. Isolation Forest randomly chooses a feature column and a value within that column's range. It sends rows below that value to one group and the remaining rows to another, then repeats this process within the groups to build a tree. If a row has a feature value far from the others, a split can leave it on its own early in the tree. Rows with similar feature values tend to stay together for more splits. But remember, the feature columns and split values are chosen randomly. This means that one tree might separate a row after only a few splits, while another takes more splits. You don't want its score to depend on the choices made by just one random tree. Isolation Forest therefore builds many trees and uses the average path length to calculate its anomaly score, that is, the number of splits along the row's path through each tree.

   In the isolation_scores function, the IsolationForest estimator is already created as model. There's one detail to watch here. Scikit-learn returns lower scores for more unusual rows, while your other two detectors return higher scores for those rows. So remember to multiply the score_samples values by -1 so all three detectors use the same direction. Run your Isolation Forest detector:

   cd /home/ps-user/workspace/code
   python3 run_lab.py score --model IsolationForest
   

   As in the previous two tasks, you'll see a score preview for validation and test, this time under IsolationForest. The runner adds this third column to validation_scores.csv and test_scores.csv. Those files now contain the three models' scores for the comparisons in the next step.

4. Challenge

   Step 4: Compare models and select an alert threshold

   Go back for a second to the question behind building a baseline. Do kNN and Isolation Forest catch anomalies that z-scores miss, and do they produce fewer false alarms? So far, you've calculated scores, but you haven't yet checked those scores using the anomaly labels. This is where you'll investigate what each detector actually finds.

   Choosing an alert threshold

   You also need to specify which rows should be investigated. You'll do this by choosing a score threshold and flagging rows whose scores are above it. Now, how do you decide where to set that threshold? Consider how changing it affects which rows get flagged. Suppose you lower that threshold. Rows that previously fell below it may now be flagged, giving the team more rows to investigate. Some could be genuine anomalies that the higher threshold missed. Others could simply be normal rows, which would send the team down a rabbit hole investigating false alarms. This is something you want to avoid because there is limited investigation time. Therefore, you'll compare how many additional anomalies a lower threshold catches with how many additional normal rows it flags.

   Comparing score distributions

   Start by examining each detector’s score distribution on the validation split. You'll look at whether a few rows have scores well above the rest or whether many rows have similar high scores. This affects how many rows are flagged when the threshold changes. For example, rows with exactly the same score cross the threshold together, so they are either all flagged or all left unflagged.

   You'll examine this using each model's median and upper percentiles. Compare these values within each model to see how its highest scores differ from the middle of its distribution. Keep in mind that the detectors use different numerical scales. A larger kNN score than an Isolation Forest score doesn't, on its own, tell you which detector considers that row more unusual.

   Comparing rankings

   Now, are the detectors giving their highest scores to the same rows? Two models could have similar score distributions but assign their highest scores to different rows. The percentiles won't reveal that difference, so you'll also compare where each row ranks according to each model.

   For this, you'll use Spearman rank correlation. Imagine sorting the rows from lowest to highest score for each model. Each row gets a position in that ordering, called its rank. Spearman correlation compares those ranks between two models. If a row ranks highly in one model, does it tend to rank highly in the other as well? Because this comparison uses ranks, the models don't need to produce scores on the same numerical scale. Pretty neat, right?

   You'll calculate a correlation for every pair of models. A value near 1 means they tend to rank the same rows highly and the same rows low. A value near -1 means rows ranked highly by one tend to rank low in the other. A value near 0 means there is little consistent relationship between their rankings. This comparison covers all rows, so it describes overall ranking agreement; you'll examine exactly which rows the models flag in the next task. Run your score summary:

   cd /home/ps-user/workspace/code
   python3 run_lab.py summarize
   

   You should see two tables calculated from the validation scores you saved in Tasks 3–5. The first contains one row per model with columns q50, q90, q95, q99, and max. The second shows the Spearman correlations between the models. The runner saves them as score_quantiles.csv and score_correlations.csv in /home/ps-user/workspace/output.

   Interpreting the score summary

   Start with the percentile table. Within each model, compare the median with the upper percentiles to see how far the highest scores extend beyond the middle of the distribution. Then look at the correlation table. Which pair of models ranks records most similarly? Use the off-diagonal values, since each model's correlation with itself is always 1.

   Here, the maximum z-score is much higher than its 99th percentile. At least 99% of validation rows have a score at or below approximately 50.79, while the highest score is approximately 177.71. So at least one row has a score far above those of 99% of the validation rows. The table doesn't tell you how many rows have similarly extreme scores. Also, kNN and Isolation Forest have the strongest ranking agreement, with a correlation of approximately 0.86. They tend to consider similar rows more unusual, although that does not establish whether those rows are anomalies. ### Investigating different decisions

   The rankings show how similarly the models order the rows. The team, however, needs a list of rows to investigate. To turn scores into a usable list, each model uses a threshold and flags rows whose scores are greater than it. You'll now compare those flags to see where the models would send the team to investigate different rows.

   Suppose the team's review limit is 2% of the validation records. This means each model can flag no more than 2% of those records for the team to investigate.

   To keep within that limit, each model's threshold is calculated from the 98th percentile of its validation scores. Flagging only scores above that threshold limits the selection to the highest-scoring rows, with no more than 2% flagged. Remember that the models use different score scales, so each needs its own numerical threshold. These thresholds are already calculated for you and passed to find_disagreements in the thresholds dictionary. You'll use them to compare which rows each model flags under the same review limit.

   A row belongs in the disagreement results when at least one model flags it and at least one leaves it unflagged. For example, a row flagged by kNN but not by the other two models belongs in that table. The row index links each result to the same row in the validation data, so make sure to keep that index when filtering the table. Run the comparison checkpoint:

   cd /home/ps-user/workspace/code
   python3 run_lab.py compare
   

   You should see a disagreement count and a few example decisions with their labels. The runner selects examples with different combinations of model decisions and labels. The full disagreement results are saved in /home/ps-user/workspace/output/preview_disagreements.csv.

   Reviewing the disagreements

   Now look at the example decisions. For a record labeled anomalous, which models flagged it and which missed it? For a record labeled normal, which models would send the team to investigate a false alarm? Open preview_disagreements.csv to examine more records and their scores. The columns ending in _score contain the numerical scores, those ending in _alert contain the decisions, and label identifies normal records with 0 and anomalies with 1. ### Evaluating the models

   Inspect how many anomalies each model catches and how many false alarms it produces. Here, you'll compare each model's flags with the anomaly labels, including rows where all the models agreed. Each row contributes to one of four counts.

   • True positive (tp): An anomalous record is flagged.
   • False positive (fp): A normal record is flagged.
   • False negative (fn): An anomalous record is left unflagged.
   • True negative (tn): A normal record is left unflagged.

   You can use these counts to answer two practical questions for the team. Of the rows flagged for investigation, how many are labeled anomalous? This is called precision. Next, of all the labeled anomalies in the validation split, how many did the detector flag? This is recall. F1 combines precision and recall into one measure, giving a high value only when both are high. You're probably familiar with them, but it's always good to have a quick refresher on these (often confusing) concepts.

   There's an edge case to handle when calculating these metrics. If a detector flags no rows, precision would divide by zero because there are no flagged rows to count in its denominator. Use zero_division=0 when calculating precision, recall, and F1 so scikit-learn reports zero whenever one of these calculations is undefined.

   Measuring ranking across thresholds

   You'll also be evaluating the numerical scores across different thresholds to see how precision and recall change as more rows are flagged.

   You will use the precision-recall curve to show these changes, with recall on the horizontal axis and precision on the vertical axis. To express the curve as a single metric, you'll calculate its area, called PR-AUC (Area Under the Precision-Recall Curve). You will also use the trapezoidal rule, which joins adjacent curve points with straight lines. In this task, you need to also calculate average precision. This weights precision by the change in recall instead of joining the points with straight lines, so the two metrics can give different results even when no scores are tied.

   You'll also calculate ROC-AUC, the area under the receiver operating characteristic curve. This curve shows how recall changes with the false-positive rate, which is the fraction of normal rows that are flagged. For all of these curve-based metrics, pass the numerical anomaly scores to the metric functions. Run the evaluation checkpoint:

   cd /home/ps-user/workspace/code
   python3 run_lab.py evaluate
   

   You should see the validation results in three tables. Detection counts show the anomalies caught or missed and the normal records flagged or left unflagged. Alert performance shows precision, recall, F1, and how much review work each model creates. Ranking metrics show PR-AUC, average precision, and ROC-AUC across thresholds. The complete results are saved in /home/ps-user/workspace/output/preview_validation_metrics.csv.

   Interpreting the evaluation results

   Compare the anomalies caught (tp) and false alarms (fp) for each model. If two models catch the same number of anomalies, does one ask the team to investigate more normal records? Use precision and recall to relate those counts to the alerts and anomalies in the file. You can also open /home/ps-user/workspace/output/comparison.png. In the score-distribution plots, look for overlap between normal rows and anomalies. Where they overlap, a threshold cannot separate them perfectly. In the precision-recall curves, follow how precision changes as recall increases. The dashed anomaly-fraction line shows the precision you would get by flagging every row.

   Here, the statistical baseline catches as many anomalies as kNN while producing far fewer false alarms—5 compared with 40. For the operations team, that means fewer normal rows to investigate without reducing the number of anomalies caught. Both methods still miss 102 anomalies (fn). In their precision-recall curves, precision drops sharply when recall rises beyond roughly 60%, showing that catching more anomalies would also bring many more false alarms.

   Interestingly, Isolation Forest catches no anomalies at all at the current threshold. Now, remember its strong ranking agreement with kNN? Well, that agreement did not guarantee that its highest-scoring rows would be anomalies. ### Selecting a model and review percentage

   Now compare how many anomalies each detector catches and how many false alarms it produces at three requested review percentages, recorded as decimal fractions in alert_fraction: 0.5%, 1%, and 2%. At a smaller percentage, the fraction of rows actually flagged for investigation, alert_rate, should also be smaller. Your job is to check how many false alarms this removes and how many anomalies are missed as a result, using the validation split.

   You’ll then select a detector and review percentage according to the team’s priorities. The first priority is to catch the most anomalies without exceeding the 2% review limit, represented by review_cap with a value of 0.02. If two combinations catch the same number of anomalies, choose the one that produces fewer false alarms. After making that selection, you’ll check its performance on the test split. Run the selection checkpoint:

   cd /home/ps-user/workspace/code
   python3 run_lab.py select
   

   You should see a validation table with nine rows, one for each model and review percentage, followed by Selected on validation: ZScore, review fraction 2.00%. Next, you’ll see the test results in three tables covering detection counts, alert performance, and ranking metrics.

   The runner saves these results in /home/ps-user/workspace/output:

   • validation_candidates.csv: The complete candidate results
   • selection.json: The selected candidate
   • validation_metrics.csv and test_metrics.csv: Each model’s results at the selected review percentage

   Reviewing the validation selection

   Start with the z-score results at 1% and 2%. At 1%, precision reaches 100%, but recall is only 32.95%. Increasing the review percentage to 2% raises recall to 60.47%, with precision falling to 96.89%. Open validation_candidates.csv to examine the counts behind that change. The team would catch 71 additional anomalies at the cost of investigating five false alarms. This explains why the selection favors 2% under the team’s priority of catching the most anomalies.

   kNN reaches the same recall at 2%, but its lower precision means more false alarms, so z-scores win the tie.

   Checking the selection on test

   Now look at the test results. All three models are evaluated at the selected 2% review percentage. Each model’s threshold is calculated from its test scores, so the percentage stays fixed while the numerical thresholds can differ from those used on validation. Open validation_metrics.csv and test_metrics.csv to compare the selected model’s recall, false alarms, and flagged percentage across the two splits. Then compare it with the other models in the test table.

   Here, the z-score detector’s validation performance does not carry over to test. Its recall falls from 60.47% to 0.71%, and it flags only 0.13% of test rows despite the 2% allowance. This happens because 6,012 test rows share exactly the threshold score, and rows equal to the threshold aren’t flagged. Its 100% test precision means every flagged row is an anomaly, but it still misses 38,732 anomalies (fn). For the operations team, having no false alarms comes with very little detection coverage.

   kNN catches 4,189 test anomalies with no false alarms, outperforming the selected detector at this review percentage. Its 10.74% recall is the highest possible at this review percentage. Anomalies make up 18.62% of the test rows, so even a perfect detector limited to 2% of rows would leave most test anomalies undetected.

5. Challenge

   Step 5: Conclusion

   In this lab, you have:

   • Prepared the Data: Profiled feature types, missing values, and class imbalance, then fitted robust scaling on the train split.
   • Implemented Three Anomaly Detectors: Built a z-score baseline, k-nearest-neighbor distance scoring, and Isolation Forest scoring.
   • Compared Model Behavior: Examined score distributions, compared rankings, and identified rows where the models disagreed.
   • Evaluated Detection Performance: Calculated false alarms, missed anomalies, precision, recall, F1, PR-AUC, average precision, and ROC-AUC.
   • Selected a Model and Review Percentage: Evaluated each detector at several review percentages, selected a combination using validation results, and checked its performance on the test split.

   Exploring further

   For an optional coding challenge, implement the threshold calculation you've been using. Add a quantile_threshold(scores, alert_fraction) function to compare_models.py that accepts one model's scores and a requested review fraction, then returns a numerical threshold. Use NumPy's quantile calculation at 1 - alert_fraction with method="higher", and flag only scores strictly greater than the threshold. Try your function in the file's main block and compare its threshold with the value returned for that model by preview_thresholds on the same data. Test it with both distinct scores and repeated scores, checking that the fraction flagged never exceeds the requested fraction.

   You can also experiment with the review percentages. Open run_lab.py and find the REVIEW_SETTINGS dictionary near the top. Add 0.015 to its candidate_fractions list to compare each detector at 1.5% as well. Then run:

   cd /home/ps-user/workspace/code
   python3 run_lab.py select
   

   Compare the new option's recall and precision with the existing options and check whether the selection changes. As a separate experiment, lower review_cap to 0.01 and rerun the command to see which detector is selected when the team can investigate at most 1% of the rows. Compare how many anomalies it catches and how many false alarms it produces in validation_candidates.csv under /home/ps-user/workspace/output.