Featured resource
2026 Tech Forecast
2026 Tech Forecast

1,500+ tech insiders, business leaders, and Pluralsight Authors share their predictions on what’s shifting fastest and how to stay ahead.

Download the forecast
  • Lab
    • Libraries: If you want this lab, consider one of these libraries.
    • AI
Labs

Build a Time Series Anomaly Detection System

You support a network operations team monitoring equipment performance. The equipment's measurements follow a recurring daily pattern, making it difficult to distinguish normal fluctuations from unexpected behavior. The team needs a reliable way to identify potential anomalies and group repeated alerts into manageable review items. In this lab, you'll build a time-series anomaly detection system using anonymized hourly measurements from the Network Equipment KPI (NEK) dataset. You'll explore patterns in the data, prepare time-based features, and train a model to predict expected measurements. You'll then use prediction errors to identify anomalies, set an alert threshold, and group related alerts. Finally, you'll evaluate how well the system detects individual anomalies and anomalous periods, simulate a change in the normal measurement level, and prepare an operational summary with monitoring and retraining recommendations. By the end of the lab, you'll understand how to build and assess an anomaly detection system, including its limitations in real-world monitoring.

Lab platform
Lab Info
Level
Beginner
Last updated
Nov 18, 2026
Duration
1h 15m

Contact sales

By clicking submit, you agree to our Privacy Policy and Terms of Use, and consent to receive marketing emails from Pluralsight.
Table of Contents
  1. Challenge

    Step 1: Introduction

    Welcome to the Build a Time Series Anomaly Detection System Code Lab!

    In this lab, you'll build an anomaly detection system for hourly performance measurements from network equipment. You'll use past readings to predict each new measurement, score unexpected departures, and turn repeated alerts into groups an operator can review. You'll then check which labeled anomalies the system finds and how it responds when the usual level changes. Your work will center on these three practical areas:

    • Preparing time-series data. Explore recurring patterns and build features from past measurements, keeping future information out of model fitting.
    • Generating useful alerts. Train a model, set an alert threshold, and group repeated alerts to reduce separate review items.
    • Assessing the system. Check which anomalous measurements and periods it detects, and examine how it responds to a sustained change in measurement level.

    Background

    Imagine you support a network operations team that reviews measurements from its equipment. The indicator rises and falls in a recurring daily pattern, so a value that is ordinary at one point in the cycle may be surprising at another. The team needs alerts that account for this history.

    Your assignment is to build an anomaly detection pipeline and evaluate it using historical equipment measurements. The pipeline will produce timestamped scores, alert groups, and evaluation results.

    Familiarizing yourself with the program structure

    You'll complete functions in three Python files under code/:

    • prepare_data.py: Explores the measurements, separates time periods, and prepares temporal features.
    • detect_anomalies.py: Fits the model, scores departures, and creates grouped alerts.
    • evaluate_system.py: Evaluates points and episodes, compares a separate baseline-change experiment, and prepares the operational summary.

    The workspace also provides:

    • code/support/: Data loading, plotting, labeled-episode extraction, and checkpoint helpers.
    • code/run_lab.py: Runs your completed functions and saves their results.
    • data/: Measurements, timestamps, and anomaly labels from the Network Equipment KPI (NEK) dataset in NumPy (.npy) files, along with the dataset's license.

    The environment uses Python 3.10.6 with NumPy, pandas, scikit-learn, Matplotlib, and pytest. Dependencies and data are available when the lab starts.

    Commands assume your workspace is /home/ps-user/workspace. The task checkpoints run from its code/ directory. Results are written to /home/ps-user/workspace/output.

    Understanding the dataset

    You'll work with a time series from the Network Equipment KPI (NEK) dataset, specifically series 1. A KPI is a key performance indicator, a measurement used to track how well something is working. This series records one equipment performance indicator every hour, with a label marking each observation as normal or anomalous.

    The indicator is anonymized, so its physical meaning and units are unavailable. You'll work with the recorded values and use elapsed hours to describe the daily cycle because the timestamps have been shifted, concealing the original recording dates and times while preserving the intervals between measurements. A labeled anomaly does not by itself confirm an equipment failure or explain its cause.

    The series is already divided into train and test. All measurements in train come before those in test in the time series:

    • Training data (train) contains 1,073 measurements, all labeled normal. You'll use part of this period to fit the model and the remaining part to set the score threshold for alerts.
    • Test data (test) contains the next 1,073 measurements, including 464 labeled anomalous. You'll use this period to evaluate the system after fitting the model and setting that threshold.

    In the data/ folder of your workspace, you'll find the measurements in train.npy and test.npy. Matching _timestamp.npy and _label.npy files hold the timestamps and labels. The loader handles opening and combining these files for you. Your functions will work with a DataFrame, a pandas table of rows and columns, indexed by timestamp and containing three columns:

    • value contains the hourly measurement.
    • label is True for an observation labeled anomalous and False for one labeled normal.
    • published_split records whether the observation belongs to the original train or test period.

    Working through the tasks

    The partially completed functions already contain imports, parameters, and return statements. Comments beginning with TODO: Task identify the code you need to supply. Replace the marked placeholders while keeping the variable names used by the surrounding code.

    For example, scaler = None marks where the fitted scaler belongs.

    Each task has a terminal checkpoint so you can inspect the result on the real series. The editable files also contain an if __name__ == "__main__": block for trying individual functions. The first example in each file is enabled, and later examples can be uncommented as you complete their tasks.

    Important: Complete the tasks in order and use Feedback/Checks as you work.

  2. Challenge

    Step 2: Explore and prepare the time series

    The team needs to distinguish unexpected changes from the series’ usual rises and falls. To do that, you'll use the training data (train), whose measurements are all labeled normal, to examine the changes the detector should learn to expect.

    As you explore, look at how the measurements vary throughout the day and whether their typical level changes from day to day. You'll use what you find to build features that give the model information about recent behavior and recurring patterns.

    The dataset you'll be working with has already been processed to handle missing readings. You won't need to worry about this in this lab. However, that processing does not check whether every hourly timestamp is present. So, you'll check for missing hours because you don't want a feature that's intended to use the previous hour's measurement to silently use a reading from two hours earlier.

    The code in front of you loads and selects the training measurements for you. You'll work with these measurements in the function explore_series, where frame contains the data indexed by timestamp. Inside this function, the provided asfreq("1h") call creates values, a Series with one row for every hour between the first and last timestamps. If an hourly timestamp is absent, it adds a row and leaves its measurement missing. You can use len() to compare the original and hourly-aligned data to determine how many timestamps are missing.

    You'll also compare each measurement with the one 24 hours earlier to examine how closely the daily pattern repeats. This relationship is called lag-24 autocorrelation. Pandas provides the autocorr() method to calculate autocorrelation, with the lag parameter specifying how many observations apart to compare. A result near 1 indicates that paired measurements vary closely together, although individual days can still differ.

    The daily summaries use consecutive 24-hour bins starting from origin, the first observation's timestamp. Pandas provides the resample() method to group time-indexed measurements into intervals, while median() calculates the median within each interval. Using origin as the starting point keeps the daily bins aligned with the beginning of the series.

    The supplied phase numbers the hours within each bin from 0 to 23. You can use groupby() to organize measurements by phase and mean() to calculate their average values. Averaging measurements at each phase shows the daily cycle's usual shape. This repeating daily pattern is a form of seasonality.

    Daily medians provide another view. They smooth over the repeated daily rises and falls, making it easier to look for a trend, a sustained increase or decrease in measurement level over time.

    Finally, absolute differences from 24 hours earlier can identify observations worth inspecting in context, so you'll be calculating these as well. Pandas provides diff() to calculate the difference between observations separated by a specified number of periods. You can then use abs() to convert those differences into absolute values, capturing the size of each change regardless of direction. Run the exploration checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py explore
    

    You should see output similar to the following:

    Missing hours: 0
    Lag-24 correlation: 0.9402
    Wrote exploration.png, daily_medians.csv, daily_pattern.csv, and daily_change_examples.csv.
    
    • Missing hours: 0: The zero count confirms that the training data contains a timestamp for every hour between its first and last observations, which is good. Always check for missing hours in your own data.
    • Lag-24 correlation: 0.9402: The correlation is close to 1, suggesting a strong relationship between measurements taken 24 hours apart.
    • Wrote exploration.png, daily_medians.csv, daily_pattern.csv, and daily_change_examples.csv.: The runner saved the exploration results to these files in the output/ folder.

    The runner also saves your exploration results as a chart in output/exploration.png.

    In the workspace file tree, navigate to output/exploration.png and double-click the file to open it. The chart shows the daily pattern and changes in measurement level across the training data.

    The chart contains three panels:

    1. The first panel shows the hourly measurements alongside the median for each complete 24-hour bin. Compare the median line with the hourly measurements to distinguish changes in measurement level from recurring daily rises and falls.

    2. The second panel shows the first seven days of training data, making the daily rises and falls easier to see.

    3. The third panel shows the mean measurement at each daily phase, calculated across the training period.

    Compare the daily patterns in the second and third panels with the daily medians in the first. Does the typical measurement level remain steady, or does it change even as the daily cycle repeats?

    The runner also saves output/daily_change_examples.csv, which lists the five largest absolute differences between measurements taken 24 hours apart. These differences identify potential contextual anomalies worth investigating. However, all five observations are labeled normal, so the differences alone do not establish that an anomaly occurred. These differences identify candidates that the team can review further. ### Splitting the data in time

    You've explored the training data. Now you'll separate the measurements used to fit the model from those used to set its alert threshold and evaluate its results. These splits need to remain in chronological order. A random split could let the model learn from observations that would not yet have been available when making a prediction. This is a form of data leakage.

    You'll keep three consecutive periods. The first 70% of the training data (train) fits the model and scaler. The remaining 30% is used for validation, which is also called the calibration period. This data split is used to set the alert threshold, which decides which measurements trigger alerts. The complete test data (test) is reserved for checking those decisions against the supplied labels. In the code, these periods are named fit, calibration, and test.

    The provided code passes as inputs two boundary timestamps to split_periods: fit_end, where calibration begins, and test_start, where the test period begins. Your function uses these boundaries to select the rows. The row at fit_end belongs to calibration, and the row at test_start belongs to test. Run the split checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py split
    

    The fitting, calibration, and test counts should be 751, 322, and 1,073.

    Creating lag and rolling features

    Your exploration showed a strong relationship between measurements taken 24 hours apart. Yesterday’s reading at the same daily phase could therefore help the model predict the next measurement. But the measurement level can also change between days, so you’ll give the model the previous hour’s reading as well. To make these readings available as inputs, you’ll create two feature columns: one containing the measurement from one hour earlier and another containing the measurement from 24 hours earlier. These are called lag features.

    Alongside these individual readings, you’ll summarize the preceding six hours to describe recent behavior across several measurements. Their mean describes the recent level, while their standard deviation describes how much it has varied. You'll use the sample standard deviation, which is the default for the pandas std() method. These are rolling statistics: the six-hour window moves forward with each observation.

    Each window must end at the previous hour. Otherwise, the measurement you're trying to predict would contribute to its own features. You'll therefore calculate the rolling statistics from past, the series shifted by one hour, and require all six readings for each result.

    The periods you separated in the previous task are consecutive parts of the same time series. When calibration begins, measurements from the fitting period have already been observed, so they can provide the history needed for its first prediction. The same applies when the test period begins.

    At the beginning of the series, however, there are no earlier measurements to use. The first observation has no previous-hour reading, and the first 24 observations have no reading from a day earlier. Those lag features must therefore remain missing. The hourly grid is already prepared inside make_history_features; leave missing feature values in place, and the runner will exclude rows without complete inputs before fitting or scoring. Filling them with measurements from later timestamps would introduce information that was not yet available. Run the history checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py history
    

    You should see the first three rows with all four feature values available, followed by Rows without complete history: 24. The first 24 rows of the dataset have no measurement from a full day earlier, so they cannot yet have all four features and are not included in this preview. ### Representing the daily cycle

    In Task 1, you examined how the mean measurement varies across the daily cycle. To help the model learn that recurring pattern, you'll add the daily phase of each measurement as an input. This tells the model which part of the cycle it is predicting.

    There is a catch with using phase numbers directly. They run from 0 to 23, so the number drops from 23 to 0 even though only one hour has passed. To preserve that connection, you'll represent the phase with two features, sine and cosine. Together, they describe a position on a circle, where the end of one cycle meets the beginning of the next. A full turn is 2 * np.pi radians; you'll convert each phase to an angle within that turn. Both features are needed because a sine value alone can correspond to more than one phase.

    In code/prepare_data.py, you'll complete add_seasonality(features, origin). The input features DataFrame contains the lag and rolling features you created in Task 3, and origin is the first timestamp in the series.

    Inside the function, the provided code calculates elapsed_hours, the number of hours between origin and each row's timestamp. You'll use these elapsed hours because, as mentioned in Step 1, the dataset's shifted timestamps do not reveal the equipment's actual time of day. Taking elapsed_hours modulo 24 gives a repeating sequence from 0 to 23, which you'll store in phase.

    The provided code also copies features into result. You'll calculate the sine and cosine of each phase's angle and assign them to daily_sin and daily_cos in result. These two daily-cycle features serve as seasonality indicators, alongside the existing lag and rolling features in the returned DataFrame. Run the feature checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py features
    

    You should see the six feature columns: lag1, lag24, past6mean, past6std, daily_sin, and daily_cos.

    At the first row with complete feature values, 24 hours have elapsed, so the daily cycle has returned to phase 0. The sine value is approximately 0, and the cosine value is 1. These two features repeat every 24 hours, while the lag and rolling features can change from day to day.

    The runner also saves the feature table in output/features.csv if you want to have a closer look.

    Scaling the feature columns

    You now have six feature columns, but they describe different quantities: measurements, recent variation, and position in the daily cycle. Their numerical spreads differ too. The regression model you'll fit next combines these inputs using feature weights (also called coefficients), and discourages very large weights. An input with a small numerical spread may need a larger weight to have the same effect as an input with a wide spread. Without scaling, that penalty would treat the inputs differently because of their scales.

    To put the features on a comparable scale, you'll use StandardScaler. For each column, it subtracts the mean and divides by the population standard deviation. Fitting the scaler calculates those statistics; transforming the data applies them to the feature values.

    In code/prepare_data.py, you'll complete make_preprocessor(train_features). The train_features DataFrame contains the six feature columns for the fitting period. Create a StandardScaler and fit it on this DataFrame to calculate the mean and standard deviation of each column.

    Use only these fitting measurements to calculate the statistics. Including calibration or test measurements would let future data influence the scaling, undoing the separation you made in Task 2. Keep the fitted scaler in scaler for the supplied return statement; the runner will use it to transform the fitting, calibration, and test features.

    Keep unusual values after scaling because they may provide evidence of an anomaly. Run the preparation checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py prepare
    

    You should see 727 usable fitting rows, 322 calibration rows, and 1,073 test rows, each with six features.

    The fitting count is now 24 lower than in Task 2 because those initial rows lack a full day's history.

    The runner saves the fitted scaler and prepared inputs for the next step.

  3. Challenge

    Step 3: Detect departures and group alerts

    Learning expected behavior

    The features are ready. Now the model needs to learn how they relate to the measurements it should expect. You'll fit it using the features and recorded measurements from the fitting period, then use it to predict measurements in calibration and test. Comparing each prediction with what was actually recorded will show where the series departs from that learned behavior.

    You'll measure the departure with the absolute prediction error and use it as the anomaly score. Taking the absolute value matters because a reading below the prediction can be just as unexpected as one above it. This comparison also takes the reading's history and daily phase into account through the model's inputs.

    The model you'll use is called Ridge regression. It learns coefficients that combine the features to predict the measurement, with the penalty on large coefficients discussed in Task 5. The supplied alpha=1.0 controls the strength of that penalty.

    In code/detect_anomalies.py, you'll complete two functions.

    Start with fit_model(train_features, train_values). The train_features DataFrame contains the scaled features for the fitting period, and train_values contains the recorded measurements. The provided code creates the Ridge model in model and selects the measurement at each feature row's timestamp into target. You'll fit model using these features and targets. Scikit-learn provides the fit() method for this. The anomaly labels are reserved for evaluating alerts later.

    Then complete score_observations(model, query_features, values) in the same file. It receives the fitted model, the features for the measurements you want to predict, and the recorded measurements to compare with those predictions. The runner calls it separately for calibration and test.

    Inside the function, the work happens in this order:

    1. The provided code selects the matching measurements into observed.
    2. Use the model's predict() method with query_features to calculate predictions.
    3. The provided code places the predictions in expected, a pandas Series with the same timestamps.
    4. Calculate the anomaly scores in scores from observed and expected. Pandas provides abs() to convert the differences into absolute values.

    Use the model without fitting it again so the scores reflect what it learned from the fitting period. Run the scoring checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py score
    

    You should see two tables, one for calibration and one for test, with columns named value, expected, and score. The first calibration score is approximately 0.0141, and the first test score is approximately 0.0793.

    In each row, score measures how far the recorded measurement is from the model's prediction. A larger score indicates a greater departure, but you haven't yet set a threshold for alerts.

    The runner saves the complete results in output/calibration_scores.csv and output/test_scores.csv. ### Viewing scores in time

    To see whether prediction errors are isolated or continue across several hours, the scoring checkpoint also created a plot of the test measurements, predictions, and anomaly scores.

    In the workspace file tree, navigate to output/scores.png and double-click the file to open it.

    Compare the observed and expected lines in the upper panel with the anomaly scores directly below them. Where the two lines stay close, the score stays low. Larger gaps produce larger scores.

    Find the anonymized timestamp 2023-01-15 on the horizontal axis. After this timestamp, the observed measurements decline and more often fall below the predictions. The lower panel shows repeated larger anomaly scores over that period. The model still follows much of the daily cycle, but its predictions do not fully follow the changing level.

    You'll now decide which departures should become alerts. ### Setting a calibration alert budget

    The team now needs a threshold that limits how many measurements it sends for review. This is why you reserved the calibration period. Its measurements are labeled normal, and its scores show the prediction errors the model makes on normal data it was not fitted on.

    Assume the team has set a provisional alert budget of 1% of calibration observations. You'll use the upper end of the calibration scores to set the threshold. The 99th percentile, or 0.99 quantile, places the threshold near the top 1% of those scores. Only a score strictly greater than the threshold should generate an alert.

    When you calculate the 99th percentile, the result may fall between two observed scores. NumPy provides np.quantile() to calculate quantiles. To keep the alert rate within the budget, use its method="higher" option, which chooses the higher of those scores. Combined with the strict greater-than comparison, this keeps the fraction of calibration observations that trigger alerts at or below the budget. If several observations share the threshold score, none of them should trigger an alert, even if that leaves some of the budget unused.

    You'll apply the threshold calculated from the calibration scores unchanged to the test data. If more test scores exceed it, the alert rate will rise above 1%. That increase shows that more measurements differ from the model's predictions by an amount that was unusual during calibration.

    If you calculated a new threshold for every batch to keep alerts at 1%, the threshold could rise as scores increased, leaving more of those departures without alerts. The 1% budget therefore applies to the calibration data; it does not limit how many alerts the same threshold can produce on future measurements.

    In code/detect_anomalies.py, you'll complete two functions.

    Start with choose_threshold(calibration_scores, alert_fraction). It receives the calibration anomaly scores and the allowed fraction of alerts, 0.01 for this lab. The provided code checks these inputs; you'll store the calculated threshold in threshold for the return statement.

    Then complete identify_alerts(scores, threshold). It receives a timestamp-indexed Series of anomaly scores and the threshold you calculated. You'll create the Boolean Series alerts, with True for a score above the threshold and False otherwise. The runner uses this function for both calibration and test, passing the same threshold each time. Run the threshold checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py threshold
    

    You should see a fixed threshold of 0.2097721964, with 3/322 calibration alerts (0.93%) and 351/1,073 test alerts (32.71%). The runner saves output/threshold_policy.json and adds alert to the two score files.

    The calibration alert rate of 0.93% meets the chosen budget, but this alone does not show whether 1% is a suitable choice for the team. A smaller budget generally raises the threshold, reducing alerts but potentially missing more anomalies. A larger budget generally lowers the threshold, potentially catching more anomalies while creating more unnecessary reviews.

    The same threshold generates alerts for nearly a third of the test measurements. That gives the team far more alerts to review, but it does not yet tell you how many alerts match the supplied anomaly labels. You'll check that after consolidating repeated alerts.

    Consolidating repeated alerts

    Anomaly scores may stay above the threshold for several consecutive hours, generating an alert for each hour. Sending all 351 alerts as separate review items could make an operator repeatedly inspect the same period. Therefore, you'll group nearby alerts so the team can review them together.

    You'll use the time between consecutive alerts to decide when to start a new group. For each alert, compare its timestamp with the last alert in the current group. A gap of exactly two hours stays in that group; a larger gap starts another. A sequence can therefore remain in one group for longer than two hours if alerts keep arriving close together.

    Each group should retain the timestamps that actually triggered alerts. For example, alerts two hours apart can belong to one group even when the hour between them has no alert. Keeping the individual timestamps will let you compare the alerts with anomaly labels later.

    In code/detect_anomalies.py, look at group_alerts(alerts, quiet_hours=2). The alerts Series contains the Boolean decisions from Task 7, and quiet_hours specifies the allowed gap in hours. The provided code selects the timestamps with True decisions and loops through them in order. It creates the first group for you.

    For each subsequent timestamp, you'll calculate gap from the last alert in the current group. Subtracting two timestamps produces a pd.Timedelta, which you can compare with pd.Timedelta(hours=quiet_hours). You'll then set starts_new_group according to the allowed gap. The rest of the loop uses that decision to add the timestamp to the current group or begin another one. Run the grouping checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py group
    

    You should see 351 alerts grouped into 43 groups using a two-hour quiet gap. The terminal displays the first five groups, including each group's first alert, last alert, and alert count.

    The runner also saves output/alert_groups.csv, which records this information for every group.

  4. Challenge

    Step 4: Evaluate detection and baseline changes

    Evaluating individual observations

    Grouping has reduced the number of review items, but it hasn't yet told you whether the alerts are useful. To assess this, you'll compare the decisions on the test data with its anomaly labels. You'll first check individual observations, then examine periods of consecutive labeled anomalies, and finally check how the system responds when the measurement level changes.

    For each observation, there are four possible outcomes:

    • True positive: An observation labeled anomalous triggers an alert.
    • False negative: An observation labeled anomalous does not trigger an alert.
    • False positive: An observation labeled normal triggers an alert.
    • True negative: An observation labeled normal does not trigger an alert.

    A confusion matrix collects these counts so you can distinguish correct decisions from false positives and missed anomalies.

    You'll use these counts to answer two questions: how many of the alerts correspond to labeled anomalies, and how many of the labeled anomalies did the system detect?

    Precision answers the first question by measuring the proportion of alerts that match anomaly labels. Recall answers the second by measuring the proportion of labeled anomalous observations that triggered alerts. A high value for one does not guarantee a high value for the other. F1 combines precision and recall, giving a low result when either is low. These are point-level metrics: each observation is counted separately, so an alert at one hour does not count as detecting a neighboring anomalous hour.

    Evaluating episodes and alert groups

    At the point level, you measure how many labeled anomalous observations trigger alerts. However, consecutive anomalous observations can belong to the same period, which you'll call a labeled episode. Detecting many observations in one long episode can raise recall even if other episodes receive no alerts at all. To find out how many separate episodes the system detects, you'll count each episode once, treating it as detected when it receives at least one alert. The results will also show the time from the start of each detected episode to its first alert so you can assess how quickly the system responds. This is event-level evaluation, with each labeled episode treated as one event.

    To determine which episodes received alerts, you'll use the anomaly labels to establish when each episode begins and ends. Alerts at or after an episode's start timestamp and before its end timestamp belong to that episode. The end timestamp marks the first hour after the episode. An episode counts as detected if it contains at least one alert, regardless of how many alerts or groups it contains. Episode recall is the number of detected episodes divided by the total number of labeled episodes, including those with no alerts.

    For each detected episode, the first-alert delay measures the hours from its start to its first alert. A missed episode has no first alert, so its delay is None. A delay of zero would mean detection was immediate.

    You'll also check how many of the alert groups from Task 8 contain an alert that matches an anomaly label. This requires checking the individual alert timestamps you retained in each group. If at least one alert occurred at an observation labeled anomalous, the group counts as matched. An anomalous observation between two alerts does not count unless it also triggered an alert. Dividing the number of matched groups by the total number of groups gives group precision.

    Open evaluate_system.py and look at evaluate_points(labels, alerts). The labels Series contains the supplied anomaly labels, and alerts contains the Boolean decisions from Task 7. The code already selects the label for each timestamp in alerts into truth, calculates the confusion-matrix counts, and calculates F1. You'll calculate precision and recall from truth and alerts. Scikit-learn provides the precision_score() and recall_score() functions for these metrics, and both are already imported. Use zero_division=0 so an undefined result, such as precision when there are no alerts, returns zero.

    Then look at evaluate_episodes(labels, alerts, groups) in the same file. It receives the full test labels, the Boolean alert decisions, and the lists of alert timestamps you created in Task 8. The function already identifies the labeled episodes and selects each episode's alert decisions into within. It also selects the timestamps with an alert into alert_times. You'll set detected according to whether that episode contains at least one alert. You can use len() to check whether alert_times contains any timestamps. The remaining code counts detected episodes, calculates first-alert delays, and matches groups to anomaly labels to produce the summary results. Run the evaluation checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py evaluate
    

    The point-level results should show the following counts and metrics:

    TP: 311; FP: 40; FN: 153; TN: 569
    Precision: 88.60%; recall: 67.03%; F1: 76.32%
    

    Of the 351 alerts, 311 match anomaly labels and 40 fall on observations labeled normal, giving a precision of 88.60%. Recall indicates that the system detects 67.03% of the 464 labeled anomalous observations and misses the other 153. These results are saved in output/point_metrics.json.

    You should also see 10/18 detected episodes (55.56%) and 35/43 groups containing an alert matching an anomaly label (81.40%). The median first-alert delay among detected episodes is 0.0 hours. The runner saves these results in output/episode_metrics.json and output/episode_results.csv.

    High point-level precision does not mean every episode is covered. Eight episodes are missed entirely, and the zero median delay applies only to detected episodes. Several matched groups can also belong to one episode, which is why 35 matched groups correspond to only 10 detected episodes.

    In Task 1, you identified potential contextual anomalies by comparing measurements taken 24 hours apart. You can now use the model's predictions and anomaly labels to examine measurements that are anomalous because of when they occur, even though their values fall within the range commonly seen during fitting.

    The runner saves 18 such observations that triggered alerts in output/contextual_examples.csv. Each has a value within the fitting period's 1st–99th percentile range, excluding the lowest and highest 1% of fitting values. ### Testing a changing baseline

    You've assessed which labeled anomalies the system detects, but its behavior may also change when the equipment begins operating at a different measurement level. For example, a configuration change could cause measurements to remain higher than before. A sustained change in the usual measurement level is called baseline drift.

    To examine how your system responds, you'll simulate this increase and compare its anomaly scores and alert counts with those from the unchanged measurements. This will show whether the increase produces more alerts or whether the measurement level can change without drawing attention.

    You'll make this change in a separate copy of the training measurements (train), leaving the original data and previous results unchanged. The experiment increases every measurement in the last seven days of calibration by the same fixed amount. All observations in this period are labeled normal. Applying the same increase throughout the period preserves the daily rises and falls, allowing you to examine how the system responds to a higher measurement level.

    The comparison needs to use the same scaler, model, and threshold for both versions. Otherwise, you would be changing the system as well as its inputs. You'll compare:

    • Alert counts: Whether the increase produces more alerts.
    • Mean anomaly scores: Whether absolute prediction errors increase or decrease, even when scores remain below the threshold.

    The changed measurements also become inputs for subsequent predictions through the lag and rolling features you created in Task 3. These higher feature values may lead the model to predict higher measurements, even though it has not been retrained.

    There are two functions to complete in code/evaluate_system.py.

    Start with simulate_shift(values, start, offset). The values Series contains the training measurements, start marks the beginning of the final seven calibration days, and offset is the amount to add. The runner sets that amount to half the fitting measurements' standard deviation, making the change relative to their variation.

    The function already copies values into changed. Use timestamp filtering with .loc[] to add the offset from start onward, leaving the original Series and earlier measurements unchanged.

    Then complete summarize_drift(baseline_scores, shifted_scores, threshold). The runner rebuilds the lag, rolling, and daily-cycle features from the changed measurements and scores them with the existing scaler and model.

    The two score Series cover the same seven-day window: baseline_scores comes from the original measurements, and shifted_scores comes from the changed copy. Using the unchanged threshold, count scores strictly greater than the threshold in each Series. You can use sum() to count True values in a Boolean Series and mean() to calculate each Series' average score. Store these results in the placeholders used by the supplied return statement. Run the baseline-change checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py drift
    

    This separate experiment covers 168 hours with an offset of 0.145848. You should see one alert on the original measurements and zero on the shifted copy. The mean score changes from 0.064605 to 0.045787.

    The checkpoint also creates a plot of the experiment in output/drift.png. In the workspace file tree, navigate to this file and double-click it to open the chart.

    Compare the two panels:

    • Upper panel: The shifted copy remains above the original measurements throughout the seven days.
    • Lower panel: All anomaly scores for the shifted copy remain below the alert threshold set in Task 7.

    You deliberately changed the measurement level, but the system generated no alerts for the shifted copy.

    The lag and rolling features contain measurements from the shifted copy, allowing the model to predict higher measurements without retraining. The resulting prediction errors remain too small to trigger alerts.

    This shows a limitation of monitoring only alerts. The equipment's measurements could remain higher than before without prompting the team to review the change. If the team needs to detect sustained changes in measurement level, it must monitor that level separately from anomaly scores.

    The checkpoint also saves the summary results in output/drift_metrics.json and the original measurements, shifted copy, and corresponding anomaly scores in output/drift_comparison.csv. ### Preparing the operational summary

    To guide the operations team's use of the system, you'll document how alerts are generated and what to monitor. You'll also choose when the team should consider fitting the model again using updated training data, a process called retraining. You'll bring these decisions together in a short operational summary.

    In code/evaluate_system.py, open build_operational_summary. You'll prepare three parts:

    • Threshold policy: Write the anomaly-score threshold and calibration alert budget in threshold_policy. Use the values from Task 7, rounding the threshold to two decimal places and expressing the budget as a percentage. Explain how scores generate alerts.
    • Monitoring plan: Read the descriptions in monitoring_plans and choose the one supported by your results from Tasks 9 and 10.
    • Retraining triggers: Read the descriptions in retraining_policies and choose when the team should consider retraining based on the results.

    The supplied return statement combines your written threshold policy and selected descriptions into the operational summary. Run the summary checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py summary
    

    The checkpoint prints your operational summary with the threshold policy and your selected descriptions.

  5. Challenge

    Step 5: Conclusion

    In this lab, you've built an anomaly detection pipeline that uses past measurements and the daily cycle to identify unexpected changes. You have:

    • Explored the measurements: Examined the daily pattern, changes in measurement level, and potential contextual anomalies.
    • Prepared the features: Separated fitting, calibration, and test periods, created lag, rolling, and daily-cycle features, and fitted scaling using only the fitting data.
    • Generated scores and alerts: Trained the model, compared its predictions with recorded measurements, set a threshold from calibration scores, and grouped repeated alerts.
    • Assessed the results: Measured detection at both point and episode levels and examined how a separate baseline change affects the scores and alerts.
    • Prepared an operational summary: Combined the threshold policy with a monitoring plan and retraining triggers. ### Exploring further

    For an optional coding challenge, try changing the alert budget and comparing how many anomalies the system detects and how many false positives it produces.

    Open code/run_lab.py and find REVIEW_SETTINGS near the top. Change alert_fraction from 0.01 to 0.02 to allow alerts on up to 2% of calibration observations. Keep the other settings unchanged and run:

    cd /home/ps-user/workspace/code
    python3 run_lab.py threshold
    python3 run_lab.py group
    python3 run_lab.py evaluate
    

    Compare the results with Tasks 7–9. Does the larger budget detect any of the eight missed episodes? How many additional false positives and alert groups does it produce?

    You can also try 0.005 for a 0.5% budget and repeat the commands. Record the results before each run because the checkpoint files are overwritten. Consider whether additional detections justify the extra review work.

    For another experiment, change how closely alerts need to occur to belong to the same group. Restore alert_fraction to 0.01 and change quiet_hours from 2 to 4. Run:

    cd /home/ps-user/workspace/code
    python3 run_lab.py threshold
    python3 run_lab.py group
    python3 run_lab.py evaluate
    

    Compare the number of groups and group precision with Tasks 8 and 9. The individual alerts have not changed, so episode recall should stay the same. Does the longer allowed gap reduce the number of review items? Open output/alert_groups.csv to examine the groups' start times, last alert times, and alert counts.

    Finally, explore whether the drift experiment responds differently to a larger increase in measurement level. Keep alert_fraction at 0.01 and change drift_scale from 0.5 to 1.0. This increases the offset from half to one fitting-period standard deviation.

    Run the drift checkpoint:

    cd /home/ps-user/workspace/code
    python3 run_lab.py drift
    

    Compare the alert counts and mean anomaly scores with Task 10. Then open the updated output/drift.png to see whether any scores exceed the threshold.

About the author

Nicolae has been a Software Engineer since 2013, focusing on Java and web stacks. Nicolae holds a degree in Computer Science and enjoys teaching, traveling and motorsports.

Real skill practice before real-world application

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Learn by doing

Engage hands-on with the tools and technologies you’re learning. You pick the skill, we provide the credentials and environment.

Follow your guide

All labs have detailed instructions and objectives, guiding you through the learning process and ensuring you understand every step.

Turn time into mastery

On average, you retain 75% more of your learning if you take time to practice. Hands-on labs set you up for success to make those skills stick.

Get started with Pluralsight