- Lab
-
Libraries: If you want this lab, consider one of these libraries.
- AI
Build a Time Series Anomaly Detection System
You support a network operations team monitoring equipment performance. The equipment's measurements follow a recurring daily pattern, making it difficult to distinguish normal fluctuations from unexpected behavior. The team needs a reliable way to identify potential anomalies and group repeated alerts into manageable review items. In this lab, you'll build a time-series anomaly detection system using anonymized hourly measurements from the Network Equipment KPI (NEK) dataset. You'll explore patterns in the data, prepare time-based features, and train a model to predict expected measurements. You'll then use prediction errors to identify anomalies, set an alert threshold, and group related alerts. Finally, you'll evaluate how well the system detects individual anomalies and anomalous periods, simulate a change in the normal measurement level, and prepare an operational summary with monitoring and retraining recommendations. By the end of the lab, you'll understand how to build and assess an anomaly detection system, including its limitations in real-world monitoring.
Lab Info
Table of Contents
-
Challenge
Step 1: Introduction
Welcome to the Build a Time Series Anomaly Detection System Code Lab!
In this lab, you'll build an anomaly detection system for hourly performance measurements from network equipment. You'll use past readings to predict each new measurement, score unexpected departures, and turn repeated alerts into groups an operator can review. You'll then check which labeled anomalies the system finds and how it responds when the usual level changes. Your work will center on these three practical areas:
- Preparing time-series data. Explore recurring patterns and build features from past measurements, keeping future information out of model fitting.
- Generating useful alerts. Train a model, set an alert threshold, and group repeated alerts to reduce separate review items.
- Assessing the system. Check which anomalous measurements and periods it detects, and examine how it responds to a sustained change in measurement level.
Background
Imagine you support a network operations team that reviews measurements from its equipment. The indicator rises and falls in a recurring daily pattern, so a value that is ordinary at one point in the cycle may be surprising at another. The team needs alerts that account for this history.
Your assignment is to build an anomaly detection pipeline and evaluate it using historical equipment measurements. The pipeline will produce timestamped scores, alert groups, and evaluation results.
Familiarizing yourself with the program structure
You'll complete functions in three Python files under
code/:prepare_data.py: Explores the measurements, separates time periods, and prepares temporal features.detect_anomalies.py: Fits the model, scores departures, and creates grouped alerts.evaluate_system.py: Evaluates points and episodes, compares a separate baseline-change experiment, and prepares the operational summary.
The workspace also provides:
code/support/: Data loading, plotting, labeled-episode extraction, and checkpoint helpers.code/run_lab.py: Runs your completed functions and saves their results.data/: Measurements, timestamps, and anomaly labels from the Network Equipment KPI (NEK) dataset in NumPy (.npy) files, along with the dataset's license.
The environment uses Python 3.10.6 with NumPy, pandas, scikit-learn, Matplotlib, and pytest. Dependencies and data are available when the lab starts.
Commands assume your workspace is
/home/ps-user/workspace. The task checkpoints run from itscode/directory. Results are written to/home/ps-user/workspace/output.Understanding the dataset
You'll work with a time series from the Network Equipment KPI (NEK) dataset, specifically series 1. A KPI is a key performance indicator, a measurement used to track how well something is working. This series records one equipment performance indicator every hour, with a label marking each observation as normal or anomalous.
The indicator is anonymized, so its physical meaning and units are unavailable. You'll work with the recorded values and use elapsed hours to describe the daily cycle because the timestamps have been shifted, concealing the original recording dates and times while preserving the intervals between measurements. A labeled anomaly does not by itself confirm an equipment failure or explain its cause.
The series is already divided into
trainandtest. All measurements intraincome before those intestin the time series:- Training data (
train) contains 1,073 measurements, all labeled normal. You'll use part of this period to fit the model and the remaining part to set the score threshold for alerts. - Test data (
test) contains the next 1,073 measurements, including 464 labeled anomalous. You'll use this period to evaluate the system after fitting the model and setting that threshold.
In the
data/folder of your workspace, you'll find the measurements intrain.npyandtest.npy. Matching_timestamp.npyand_label.npyfiles hold the timestamps and labels. The loader handles opening and combining these files for you. Your functions will work with aDataFrame, a pandas table of rows and columns, indexed by timestamp and containing three columns:valuecontains the hourly measurement.labelisTruefor an observation labeled anomalous andFalsefor one labeled normal.published_splitrecords whether the observation belongs to the originaltrainortestperiod.
Working through the tasks
The partially completed functions already contain imports, parameters, and return statements. Comments beginning with
TODO: Taskidentify the code you need to supply. Replace the marked placeholders while keeping the variable names used by the surrounding code.For example,
scaler = Nonemarks where the fitted scaler belongs.Each task has a terminal checkpoint so you can inspect the result on the real series. The editable files also contain an
if __name__ == "__main__":block for trying individual functions. The first example in each file is enabled, and later examples can be uncommented as you complete their tasks.Important: Complete the tasks in order and use Feedback/Checks as you work.
-
Challenge
Step 2: Explore and prepare the time series
The team needs to distinguish unexpected changes from the series’ usual rises and falls. To do that, you'll use the training data (
train), whose measurements are all labeled normal, to examine the changes the detector should learn to expect.As you explore, look at how the measurements vary throughout the day and whether their typical level changes from day to day. You'll use what you find to build features that give the model information about recent behavior and recurring patterns.
The dataset you'll be working with has already been processed to handle missing readings. You won't need to worry about this in this lab. However, that processing does not check whether every hourly timestamp is present. So, you'll check for missing hours because you don't want a feature that's intended to use the previous hour's measurement to silently use a reading from two hours earlier.
The code in front of you loads and selects the training measurements for you. You'll work with these measurements in the function
explore_series, whereframecontains the data indexed by timestamp. Inside this function, the providedasfreq("1h")call createsvalues, aSerieswith one row for every hour between the first and last timestamps. If an hourly timestamp is absent, it adds a row and leaves its measurement missing. You can uselen()to compare the original and hourly-aligned data to determine how many timestamps are missing.You'll also compare each measurement with the one 24 hours earlier to examine how closely the daily pattern repeats. This relationship is called lag-24 autocorrelation. Pandas provides the
autocorr()method to calculate autocorrelation, with the lag parameter specifying how many observations apart to compare. A result near 1 indicates that paired measurements vary closely together, although individual days can still differ.The daily summaries use consecutive 24-hour bins starting from
origin, the first observation's timestamp. Pandas provides theresample()method to group time-indexed measurements into intervals, whilemedian()calculates the median within each interval. Usingoriginas the starting point keeps the daily bins aligned with the beginning of the series.The supplied
phasenumbers the hours within each bin from 0 to 23. You can usegroupby()to organize measurements by phase andmean()to calculate their average values. Averaging measurements at each phase shows the daily cycle's usual shape. This repeating daily pattern is a form of seasonality.Daily medians provide another view. They smooth over the repeated daily rises and falls, making it easier to look for a trend, a sustained increase or decrease in measurement level over time.
Finally, absolute differences from 24 hours earlier can identify observations worth inspecting in context, so you'll be calculating these as well. Pandas provides
diff()to calculate the difference between observations separated by a specified number of periods. You can then useabs()to convert those differences into absolute values, capturing the size of each change regardless of direction. Run the exploration checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py exploreYou should see output similar to the following:
Missing hours: 0 Lag-24 correlation: 0.9402 Wrote exploration.png, daily_medians.csv, daily_pattern.csv, and daily_change_examples.csv.Missing hours: 0: The zero count confirms that the training data contains a timestamp for every hour between its first and last observations, which is good. Always check for missing hours in your own data.Lag-24 correlation: 0.9402: The correlation is close to 1, suggesting a strong relationship between measurements taken 24 hours apart.Wrote exploration.png, daily_medians.csv, daily_pattern.csv, and daily_change_examples.csv.: The runner saved the exploration results to these files in theoutput/folder.
The runner also saves your exploration results as a chart in
output/exploration.png.In the workspace file tree, navigate to
output/exploration.pngand double-click the file to open it. The chart shows the daily pattern and changes in measurement level across the training data.The chart contains three panels:
-
The first panel shows the hourly measurements alongside the median for each complete 24-hour bin. Compare the median line with the hourly measurements to distinguish changes in measurement level from recurring daily rises and falls.
-
The second panel shows the first seven days of training data, making the daily rises and falls easier to see.
-
The third panel shows the mean measurement at each daily phase, calculated across the training period.
Compare the daily patterns in the second and third panels with the daily medians in the first. Does the typical measurement level remain steady, or does it change even as the daily cycle repeats?
The runner also saves
output/daily_change_examples.csv, which lists the five largest absolute differences between measurements taken 24 hours apart. These differences identify potential contextual anomalies worth investigating. However, all five observations are labeled normal, so the differences alone do not establish that an anomaly occurred. These differences identify candidates that the team can review further. ### Splitting the data in timeYou've explored the training data. Now you'll separate the measurements used to fit the model from those used to set its alert threshold and evaluate its results. These splits need to remain in chronological order. A random split could let the model learn from observations that would not yet have been available when making a prediction. This is a form of data leakage.
You'll keep three consecutive periods. The first 70% of the training data (
train) fits the model and scaler. The remaining 30% is used for validation, which is also called the calibration period. This data split is used to set the alert threshold, which decides which measurements trigger alerts. The complete test data (test) is reserved for checking those decisions against the supplied labels. In the code, these periods are namedfit,calibration, andtest.The provided code passes as inputs two boundary timestamps to
split_periods:fit_end, where calibration begins, andtest_start, where the test period begins. Your function uses these boundaries to select the rows. The row atfit_endbelongs tocalibration, and the row attest_startbelongs totest. Run the split checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py splitThe fitting, calibration, and test counts should be
751,322, and1,073.Creating lag and rolling features
Your exploration showed a strong relationship between measurements taken 24 hours apart. Yesterday’s reading at the same daily phase could therefore help the model predict the next measurement. But the measurement level can also change between days, so you’ll give the model the previous hour’s reading as well. To make these readings available as inputs, you’ll create two feature columns: one containing the measurement from one hour earlier and another containing the measurement from 24 hours earlier. These are called lag features.
Alongside these individual readings, you’ll summarize the preceding six hours to describe recent behavior across several measurements. Their mean describes the recent level, while their standard deviation describes how much it has varied. You'll use the sample standard deviation, which is the default for the pandas
std()method. These are rolling statistics: the six-hour window moves forward with each observation.Each window must end at the previous hour. Otherwise, the measurement you're trying to predict would contribute to its own features. You'll therefore calculate the rolling statistics from
past, the series shifted by one hour, and require all six readings for each result.The periods you separated in the previous task are consecutive parts of the same time series. When calibration begins, measurements from the fitting period have already been observed, so they can provide the history needed for its first prediction. The same applies when the test period begins.
At the beginning of the series, however, there are no earlier measurements to use. The first observation has no previous-hour reading, and the first 24 observations have no reading from a day earlier. Those lag features must therefore remain missing. The hourly grid is already prepared inside
make_history_features; leave missing feature values in place, and the runner will exclude rows without complete inputs before fitting or scoring. Filling them with measurements from later timestamps would introduce information that was not yet available. Run the history checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py historyYou should see the first three rows with all four feature values available, followed by
Rows without complete history: 24. The first 24 rows of the dataset have no measurement from a full day earlier, so they cannot yet have all four features and are not included in this preview. ### Representing the daily cycleIn Task 1, you examined how the mean measurement varies across the daily cycle. To help the model learn that recurring pattern, you'll add the daily phase of each measurement as an input. This tells the model which part of the cycle it is predicting.
There is a catch with using phase numbers directly. They run from 0 to 23, so the number drops from 23 to 0 even though only one hour has passed. To preserve that connection, you'll represent the phase with two features, sine and cosine. Together, they describe a position on a circle, where the end of one cycle meets the beginning of the next. A full turn is
2 * np.piradians; you'll convert each phase to an angle within that turn. Both features are needed because a sine value alone can correspond to more than one phase.In
code/prepare_data.py, you'll completeadd_seasonality(features, origin). The inputfeaturesDataFrame contains the lag and rolling features you created in Task 3, andoriginis the first timestamp in the series.Inside the function, the provided code calculates
elapsed_hours, the number of hours betweenoriginand each row's timestamp. You'll use these elapsed hours because, as mentioned in Step 1, the dataset's shifted timestamps do not reveal the equipment's actual time of day. Takingelapsed_hoursmodulo 24 gives a repeating sequence from 0 to 23, which you'll store inphase.The provided code also copies
featuresintoresult. You'll calculate the sine and cosine of each phase's angle and assign them todaily_sinanddaily_cosinresult. These two daily-cycle features serve as seasonality indicators, alongside the existing lag and rolling features in the returned DataFrame. Run the feature checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py featuresYou should see the six feature columns:
lag1,lag24,past6mean,past6std,daily_sin, anddaily_cos.At the first row with complete feature values, 24 hours have elapsed, so the daily cycle has returned to phase 0. The sine value is approximately
0, and the cosine value is1. These two features repeat every 24 hours, while the lag and rolling features can change from day to day.The runner also saves the feature table in
output/features.csvif you want to have a closer look.Scaling the feature columns
You now have six feature columns, but they describe different quantities: measurements, recent variation, and position in the daily cycle. Their numerical spreads differ too. The regression model you'll fit next combines these inputs using feature weights (also called coefficients), and discourages very large weights. An input with a small numerical spread may need a larger weight to have the same effect as an input with a wide spread. Without scaling, that penalty would treat the inputs differently because of their scales.
To put the features on a comparable scale, you'll use
StandardScaler. For each column, it subtracts the mean and divides by the population standard deviation. Fitting the scaler calculates those statistics; transforming the data applies them to the feature values.In
code/prepare_data.py, you'll completemake_preprocessor(train_features). Thetrain_featuresDataFrame contains the six feature columns for the fitting period. Create aStandardScalerand fit it on this DataFrame to calculate the mean and standard deviation of each column.Use only these fitting measurements to calculate the statistics. Including calibration or test measurements would let future data influence the scaling, undoing the separation you made in Task 2. Keep the fitted scaler in
scalerfor the supplied return statement; the runner will use it to transform the fitting, calibration, and test features.Keep unusual values after scaling because they may provide evidence of an anomaly. Run the preparation checkpoint:
cd /home/ps-user/workspace/code python3 run_lab.py prepareYou should see
727usable fitting rows,322calibration rows, and1,073test rows, each with six features.The fitting count is now 24 lower than in Task 2 because those initial rows lack a full day's history.
The runner saves the fitted scaler and prepared inputs for the next step.
-
Challenge
Step 3: Detect departures and group alerts
Learning expected behavior
The features are ready. Now the model needs to learn how they relate to the measurements it should expect. You'll fit it using the features and recorded measurements from the fitting period, then use it to predict measurements in calibration and test. Comparing each prediction with what was actually recorded will show where the series departs from that learned behavior.
You'll measure the departure with the absolute prediction error and use it as the anomaly score. Taking the absolute value matters because a reading below the prediction can be just as unexpected as one above it. This comparison also takes the reading's history and daily phase into account through the model's inputs.
The model you'll use is called Ridge regression. It learns coefficients that combine the features to predict the measurement, with the penalty on large coefficients discussed in Task 5. The supplied
alpha=1.0controls the strength of that penalty.In
code/detect_anomalies.py, you'll complete two functions.Start with
fit_model(train_features, train_values). Thetrain_featuresDataFrame contains the scaled features for the fitting period, andtrain_valuescontains the recorded measurements. The provided code creates the Ridge model inmodeland selects the measurement at each feature row's timestamp intotarget. You'll fitmodelusing these features and targets. Scikit-learn provides thefit()method for this. The anomaly labels are reserved for evaluating alerts later.Then complete
score_observations(model, query_features, values)in the same file. It receives the fitted model, the features for the measurements you want to predict, and the recorded measurements to compare with those predictions. The runner calls it separately for calibration and test.Inside the function, the work happens in this order:
- The provided code selects the matching measurements into
observed. - Use the model's
predict()method withquery_featuresto calculatepredictions. - The provided code places the predictions in
expected, a pandas Series with the same timestamps. - Calculate the anomaly scores in
scoresfromobservedandexpected. Pandas providesabs()to convert the differences into absolute values.
Use the model without fitting it again so the scores reflect what it learned from the fitting period. Run the scoring checkpoint:
cd /home/ps-user/workspace/code python3 run_lab.py scoreYou should see two tables, one for calibration and one for test, with columns named
value,expected, andscore. The first calibration score is approximately0.0141, and the first test score is approximately0.0793.In each row,
scoremeasures how far the recorded measurement is from the model's prediction. A larger score indicates a greater departure, but you haven't yet set a threshold for alerts.The runner saves the complete results in
output/calibration_scores.csvandoutput/test_scores.csv. ### Viewing scores in timeTo see whether prediction errors are isolated or continue across several hours, the scoring checkpoint also created a plot of the test measurements, predictions, and anomaly scores.
In the workspace file tree, navigate to
output/scores.pngand double-click the file to open it.Compare the observed and expected lines in the upper panel with the anomaly scores directly below them. Where the two lines stay close, the score stays low. Larger gaps produce larger scores.
Find the anonymized timestamp
2023-01-15on the horizontal axis. After this timestamp, the observed measurements decline and more often fall below the predictions. The lower panel shows repeated larger anomaly scores over that period. The model still follows much of the daily cycle, but its predictions do not fully follow the changing level.You'll now decide which departures should become alerts. ### Setting a calibration alert budget
The team now needs a threshold that limits how many measurements it sends for review. This is why you reserved the calibration period. Its measurements are labeled normal, and its scores show the prediction errors the model makes on normal data it was not fitted on.
Assume the team has set a provisional alert budget of 1% of calibration observations. You'll use the upper end of the calibration scores to set the threshold. The 99th percentile, or
0.99quantile, places the threshold near the top 1% of those scores. Only a score strictly greater than the threshold should generate an alert.When you calculate the 99th percentile, the result may fall between two observed scores. NumPy provides
np.quantile()to calculate quantiles. To keep the alert rate within the budget, use itsmethod="higher"option, which chooses the higher of those scores. Combined with the strict greater-than comparison, this keeps the fraction of calibration observations that trigger alerts at or below the budget. If several observations share the threshold score, none of them should trigger an alert, even if that leaves some of the budget unused.You'll apply the threshold calculated from the calibration scores unchanged to the test data. If more test scores exceed it, the alert rate will rise above 1%. That increase shows that more measurements differ from the model's predictions by an amount that was unusual during calibration.
If you calculated a new threshold for every batch to keep alerts at 1%, the threshold could rise as scores increased, leaving more of those departures without alerts. The 1% budget therefore applies to the calibration data; it does not limit how many alerts the same threshold can produce on future measurements.
In
code/detect_anomalies.py, you'll complete two functions.Start with
choose_threshold(calibration_scores, alert_fraction). It receives the calibration anomaly scores and the allowed fraction of alerts,0.01for this lab. The provided code checks these inputs; you'll store the calculated threshold inthresholdfor the return statement.Then complete
identify_alerts(scores, threshold). It receives a timestamp-indexed Series of anomaly scores and the threshold you calculated. You'll create the Boolean Seriesalerts, withTruefor a score above the threshold andFalseotherwise. The runner uses this function for both calibration and test, passing the same threshold each time. Run the threshold checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py thresholdYou should see a fixed threshold of
0.2097721964, with3/322calibration alerts (0.93%) and351/1,073test alerts (32.71%). The runner savesoutput/threshold_policy.jsonand addsalertto the two score files.The calibration alert rate of 0.93% meets the chosen budget, but this alone does not show whether 1% is a suitable choice for the team. A smaller budget generally raises the threshold, reducing alerts but potentially missing more anomalies. A larger budget generally lowers the threshold, potentially catching more anomalies while creating more unnecessary reviews.
The same threshold generates alerts for nearly a third of the test measurements. That gives the team far more alerts to review, but it does not yet tell you how many alerts match the supplied anomaly labels. You'll check that after consolidating repeated alerts.
Consolidating repeated alerts
Anomaly scores may stay above the threshold for several consecutive hours, generating an alert for each hour. Sending all 351 alerts as separate review items could make an operator repeatedly inspect the same period. Therefore, you'll group nearby alerts so the team can review them together.
You'll use the time between consecutive alerts to decide when to start a new group. For each alert, compare its timestamp with the last alert in the current group. A gap of exactly two hours stays in that group; a larger gap starts another. A sequence can therefore remain in one group for longer than two hours if alerts keep arriving close together.
Each group should retain the timestamps that actually triggered alerts. For example, alerts two hours apart can belong to one group even when the hour between them has no alert. Keeping the individual timestamps will let you compare the alerts with anomaly labels later.
In
code/detect_anomalies.py, look atgroup_alerts(alerts, quiet_hours=2). ThealertsSeries contains the Boolean decisions from Task 7, andquiet_hoursspecifies the allowed gap in hours. The provided code selects the timestamps withTruedecisions and loops through them in order. It creates the first group for you.For each subsequent
timestamp, you'll calculategapfrom the last alert in the current group. Subtracting two timestamps produces apd.Timedelta, which you can compare withpd.Timedelta(hours=quiet_hours). You'll then setstarts_new_groupaccording to the allowed gap. The rest of the loop uses that decision to add the timestamp to the current group or begin another one. Run the grouping checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py groupYou should see
351alerts grouped into43groups using a two-hour quiet gap. The terminal displays the first five groups, including each group's first alert, last alert, and alert count.The runner also saves
output/alert_groups.csv, which records this information for every group. - The provided code selects the matching measurements into
-
Challenge
Step 4: Evaluate detection and baseline changes
Evaluating individual observations
Grouping has reduced the number of review items, but it hasn't yet told you whether the alerts are useful. To assess this, you'll compare the decisions on the test data with its anomaly labels. You'll first check individual observations, then examine periods of consecutive labeled anomalies, and finally check how the system responds when the measurement level changes.
For each observation, there are four possible outcomes:
- True positive: An observation labeled anomalous triggers an alert.
- False negative: An observation labeled anomalous does not trigger an alert.
- False positive: An observation labeled normal triggers an alert.
- True negative: An observation labeled normal does not trigger an alert.
A confusion matrix collects these counts so you can distinguish correct decisions from false positives and missed anomalies.
You'll use these counts to answer two questions: how many of the alerts correspond to labeled anomalies, and how many of the labeled anomalies did the system detect?
Precision answers the first question by measuring the proportion of alerts that match anomaly labels. Recall answers the second by measuring the proportion of labeled anomalous observations that triggered alerts. A high value for one does not guarantee a high value for the other. F1 combines precision and recall, giving a low result when either is low. These are point-level metrics: each observation is counted separately, so an alert at one hour does not count as detecting a neighboring anomalous hour.
Evaluating episodes and alert groups
At the point level, you measure how many labeled anomalous observations trigger alerts. However, consecutive anomalous observations can belong to the same period, which you'll call a labeled episode. Detecting many observations in one long episode can raise recall even if other episodes receive no alerts at all. To find out how many separate episodes the system detects, you'll count each episode once, treating it as detected when it receives at least one alert. The results will also show the time from the start of each detected episode to its first alert so you can assess how quickly the system responds. This is event-level evaluation, with each labeled episode treated as one event.
To determine which episodes received alerts, you'll use the anomaly labels to establish when each episode begins and ends. Alerts at or after an episode's start timestamp and before its end timestamp belong to that episode. The end timestamp marks the first hour after the episode. An episode counts as detected if it contains at least one alert, regardless of how many alerts or groups it contains. Episode recall is the number of detected episodes divided by the total number of labeled episodes, including those with no alerts.
For each detected episode, the first-alert delay measures the hours from its start to its first alert. A missed episode has no first alert, so its delay is
None. A delay of zero would mean detection was immediate.You'll also check how many of the alert groups from Task 8 contain an alert that matches an anomaly label. This requires checking the individual alert timestamps you retained in each group. If at least one alert occurred at an observation labeled anomalous, the group counts as matched. An anomalous observation between two alerts does not count unless it also triggered an alert. Dividing the number of matched groups by the total number of groups gives group precision.
Open
evaluate_system.pyand look atevaluate_points(labels, alerts). ThelabelsSeries contains the supplied anomaly labels, andalertscontains the Boolean decisions from Task 7. The code already selects the label for each timestamp inalertsintotruth, calculates the confusion-matrix counts, and calculates F1. You'll calculateprecisionandrecallfromtruthandalerts. Scikit-learn provides theprecision_score()andrecall_score()functions for these metrics, and both are already imported. Usezero_division=0so an undefined result, such as precision when there are no alerts, returns zero.Then look at
evaluate_episodes(labels, alerts, groups)in the same file. It receives the full test labels, the Boolean alert decisions, and the lists of alert timestamps you created in Task 8. The function already identifies the labeled episodes and selects each episode's alert decisions intowithin. It also selects the timestamps with an alert intoalert_times. You'll setdetectedaccording to whether that episode contains at least one alert. You can uselen()to check whetheralert_timescontains any timestamps. The remaining code counts detected episodes, calculates first-alert delays, and matches groups to anomaly labels to produce the summary results. Run the evaluation checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py evaluateThe point-level results should show the following counts and metrics:
TP: 311; FP: 40; FN: 153; TN: 569 Precision: 88.60%; recall: 67.03%; F1: 76.32%Of the 351 alerts, 311 match anomaly labels and 40 fall on observations labeled normal, giving a precision of
88.60%. Recall indicates that the system detects67.03%of the 464 labeled anomalous observations and misses the other 153. These results are saved inoutput/point_metrics.json.You should also see
10/18detected episodes (55.56%) and35/43groups containing an alert matching an anomaly label (81.40%). The median first-alert delay among detected episodes is0.0hours. The runner saves these results inoutput/episode_metrics.jsonandoutput/episode_results.csv.High point-level precision does not mean every episode is covered. Eight episodes are missed entirely, and the zero median delay applies only to detected episodes. Several matched groups can also belong to one episode, which is why 35 matched groups correspond to only 10 detected episodes.
In Task 1, you identified potential contextual anomalies by comparing measurements taken 24 hours apart. You can now use the model's predictions and anomaly labels to examine measurements that are anomalous because of when they occur, even though their values fall within the range commonly seen during fitting.
The runner saves 18 such observations that triggered alerts in
output/contextual_examples.csv. Each has a value within the fitting period's 1st–99th percentile range, excluding the lowest and highest 1% of fitting values. ### Testing a changing baselineYou've assessed which labeled anomalies the system detects, but its behavior may also change when the equipment begins operating at a different measurement level. For example, a configuration change could cause measurements to remain higher than before. A sustained change in the usual measurement level is called baseline drift.
To examine how your system responds, you'll simulate this increase and compare its anomaly scores and alert counts with those from the unchanged measurements. This will show whether the increase produces more alerts or whether the measurement level can change without drawing attention.
You'll make this change in a separate copy of the training measurements (
train), leaving the original data and previous results unchanged. The experiment increases every measurement in the last seven days of calibration by the same fixed amount. All observations in this period are labeled normal. Applying the same increase throughout the period preserves the daily rises and falls, allowing you to examine how the system responds to a higher measurement level.The comparison needs to use the same scaler, model, and threshold for both versions. Otherwise, you would be changing the system as well as its inputs. You'll compare:
- Alert counts: Whether the increase produces more alerts.
- Mean anomaly scores: Whether absolute prediction errors increase or decrease, even when scores remain below the threshold.
The changed measurements also become inputs for subsequent predictions through the lag and rolling features you created in Task 3. These higher feature values may lead the model to predict higher measurements, even though it has not been retrained.
There are two functions to complete in
code/evaluate_system.py.Start with
simulate_shift(values, start, offset). ThevaluesSeries contains the training measurements,startmarks the beginning of the final seven calibration days, andoffsetis the amount to add. The runner sets that amount to half the fitting measurements' standard deviation, making the change relative to their variation.The function already copies
valuesintochanged. Use timestamp filtering with.loc[]to add the offset fromstartonward, leaving the original Series and earlier measurements unchanged.Then complete
summarize_drift(baseline_scores, shifted_scores, threshold). The runner rebuilds the lag, rolling, and daily-cycle features from the changed measurements and scores them with the existing scaler and model.The two score Series cover the same seven-day window:
baseline_scorescomes from the original measurements, andshifted_scorescomes from the changed copy. Using the unchangedthreshold, count scores strictly greater than the threshold in each Series. You can usesum()to countTruevalues in a Boolean Series andmean()to calculate each Series' average score. Store these results in the placeholders used by the supplied return statement. Run the baseline-change checkpoint:cd /home/ps-user/workspace/code python3 run_lab.py driftThis separate experiment covers
168hours with an offset of0.145848. You should see one alert on the original measurements and zero on the shifted copy. The mean score changes from0.064605to0.045787.The checkpoint also creates a plot of the experiment in
output/drift.png. In the workspace file tree, navigate to this file and double-click it to open the chart.Compare the two panels:
- Upper panel: The shifted copy remains above the original measurements throughout the seven days.
- Lower panel: All anomaly scores for the shifted copy remain below the alert threshold set in Task 7.
You deliberately changed the measurement level, but the system generated no alerts for the shifted copy.
The lag and rolling features contain measurements from the shifted copy, allowing the model to predict higher measurements without retraining. The resulting prediction errors remain too small to trigger alerts.
This shows a limitation of monitoring only alerts. The equipment's measurements could remain higher than before without prompting the team to review the change. If the team needs to detect sustained changes in measurement level, it must monitor that level separately from anomaly scores.
The checkpoint also saves the summary results in
output/drift_metrics.jsonand the original measurements, shifted copy, and corresponding anomaly scores inoutput/drift_comparison.csv. ### Preparing the operational summaryTo guide the operations team's use of the system, you'll document how alerts are generated and what to monitor. You'll also choose when the team should consider fitting the model again using updated training data, a process called retraining. You'll bring these decisions together in a short operational summary.
In
code/evaluate_system.py, openbuild_operational_summary. You'll prepare three parts:- Threshold policy: Write the anomaly-score threshold and calibration alert budget in
threshold_policy. Use the values from Task 7, rounding the threshold to two decimal places and expressing the budget as a percentage. Explain how scores generate alerts. - Monitoring plan: Read the descriptions in
monitoring_plansand choose the one supported by your results from Tasks 9 and 10. - Retraining triggers: Read the descriptions in
retraining_policiesand choose when the team should consider retraining based on the results.
The supplied return statement combines your written threshold policy and selected descriptions into the operational summary. Run the summary checkpoint:
cd /home/ps-user/workspace/code python3 run_lab.py summaryThe checkpoint prints your operational summary with the threshold policy and your selected descriptions.
-
Challenge
Step 5: Conclusion
In this lab, you've built an anomaly detection pipeline that uses past measurements and the daily cycle to identify unexpected changes. You have:
- Explored the measurements: Examined the daily pattern, changes in measurement level, and potential contextual anomalies.
- Prepared the features: Separated fitting, calibration, and test periods, created lag, rolling, and daily-cycle features, and fitted scaling using only the fitting data.
- Generated scores and alerts: Trained the model, compared its predictions with recorded measurements, set a threshold from calibration scores, and grouped repeated alerts.
- Assessed the results: Measured detection at both point and episode levels and examined how a separate baseline change affects the scores and alerts.
- Prepared an operational summary: Combined the threshold policy with a monitoring plan and retraining triggers. ### Exploring further
For an optional coding challenge, try changing the alert budget and comparing how many anomalies the system detects and how many false positives it produces.
Open
code/run_lab.pyand findREVIEW_SETTINGSnear the top. Changealert_fractionfrom0.01to0.02to allow alerts on up to 2% of calibration observations. Keep the other settings unchanged and run:cd /home/ps-user/workspace/code python3 run_lab.py threshold python3 run_lab.py group python3 run_lab.py evaluateCompare the results with Tasks 7–9. Does the larger budget detect any of the eight missed episodes? How many additional false positives and alert groups does it produce?
You can also try
0.005for a 0.5% budget and repeat the commands. Record the results before each run because the checkpoint files are overwritten. Consider whether additional detections justify the extra review work.For another experiment, change how closely alerts need to occur to belong to the same group. Restore
alert_fractionto0.01and changequiet_hoursfrom2to4. Run:cd /home/ps-user/workspace/code python3 run_lab.py threshold python3 run_lab.py group python3 run_lab.py evaluateCompare the number of groups and group precision with Tasks 8 and 9. The individual alerts have not changed, so episode recall should stay the same. Does the longer allowed gap reduce the number of review items? Open
output/alert_groups.csvto examine the groups' start times, last alert times, and alert counts.Finally, explore whether the drift experiment responds differently to a larger increase in measurement level. Keep
alert_fractionat0.01and changedrift_scalefrom0.5to1.0. This increases the offset from half to one fitting-period standard deviation.Run the drift checkpoint:
cd /home/ps-user/workspace/code python3 run_lab.py driftCompare the alert counts and mean anomaly scores with Task 10. Then open the updated
output/drift.pngto see whether any scores exceed the threshold.
About the author
Real skill practice before real-world application
Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.
Learn by doing
Engage hands-on with the tools and technologies you’re learning. You pick the skill, we provide the credentials and environment.
Follow your guide
All labs have detailed instructions and objectives, guiding you through the learning process and ensuring you understand every step.
Turn time into mastery
On average, you retain 75% more of your learning if you take time to practice. Hands-on labs set you up for success to make those skills stick.