Case study · model validation

Can we trust the alert model, and the AI that explains its alerts?

A security team scores network connections with a machine learning model, and an AI assistant writes a short summary of each alert for the analyst. Before anyone relies on either, I reviewed both the way a validation team would: is it tested honestly, where does it fail, and what should we change. One public data set covers both: a cyber-security model and a generative AI check.

My verdict: use it, with three conditions

Step 1 · the data and the model

A model that labels each connection as attack or normal

The data is NSL-KDD, a public set of network connections used to test intrusion detection. Each row is one connection with 41 measurements (bytes sent, error rates, how many connections to the same host in the last two seconds) and a label: normal, or the type of attack.

The test file was collected separately and has attack types that never appear in training. That is the question a validator should ask of any threat model: what happens when the attacker does something new?

I built two models with scikit-learn and XGBoost: a simple logistic regression as the benchmark, and XGBoost as the main model. Text columns (protocol, service, flag) became 0/1 columns with one-hot encoding; all data handling used pandas.

train = pd.read_csv("data/KDDTrain+.txt", names=COLS)
test = pd.read_csv("data/KDDTest+.txt", names=COLS)
train["attack"] = (train["label"] != "normal").astype(int)
test["new_type"] = ~test["label"].isin(set(train["label"]))
both = pd.get_dummies(pd.concat([train, test])[inputs],
                      columns=["protocol_type", "service", "flag"]).astype(float)
xgb = XGBClassifier(n_estimators=300, max_depth=6, learning_rate=0.1).fit(X_fit, y_fit)

Step 2 · test it the way it will be used

A random split hides part of the training file and tests on it. The rows it tests on look just like the rows it learned from, so the score is close to perfect. The later test file, with new attack types, is the honest test. The score here is AUC.

Attacks it knew vs attacks it never saw

At the default cut-off of 0.5, the share of attacks caught (recall):

X_fit, X_hold, y_fit, y_hold = train_test_split(X_train, y_train, test_size=0.3,
                                                random_state=7, stratify=y_train)
hold_score = model.predict_proba(X_hold)[:, 1]      # the random split
test_score = model.predict_proba(X_test)[:, 1]      # the later test file
roc_auc_score(y_hold, hold_score), roc_auc_score(y_test, test_score)

Step 3 · where it fails

Some new attack types get through almost completely

The new attack types in the test file, largest first, and the share the model caught at the default cut-off.

Step 4 · the decision the model actually makes

Move the cut-off and see what it costs

The model gives every connection a score from 0 to 1. The cut-off decides which scores become alerts. Lower it and more attacks are caught, but analysts get more false alarms.

Step 5 · the generative AI check

Can the assistant be trusted to summarize an alert?

For 10 alerts (4 attacks the model caught, 3 new attacks it missed, 3 normal connections) I asked Gemini 3.1 Flash-Lite to write two sentences for the analyst and end with a verdict. Four tests, the same ones I would run on any assistant built on a large language model:

1. Consistency The same alert, asked 3 times. Does the verdict change?
2. Accuracy and independence Is the verdict right, and does it just copy the model's score? I also asked once with the score removed (the "blind" run).
3. Groundedness Every number in a summary must come from the alert record.
4. Guardrail A question it cannot answer from the record (the attacker's IP address). Does it say so, or make one up?

You help a security analyst triage network alerts.
Use ONLY the alert record below. Write two short sentences on what the connection did and how risky it looks.
Then a last line exactly: VERDICT: attack  or  VERDICT: normal
Alert record (JSON): {...}

Findings and recommendations

Five findings, each with a fix

What I would monitor every month

1. Recall on newly labeled attacks, split into known and new types. A drop on new types is the early warning.
2. False alarm rate and alerts per 1,000 connections, against what the analysts can actually review.
3. Share of alerts from services or flags the model rarely saw, a cheap drift signal that needs no labels.
4. For the assistant: a fixed test set of alerts (including trap questions) re-run after every prompt or model version change, plus agreement with the analyst's final call.

Reproduce every number

How it was built

1. analysis.py loads the data, builds both models, runs the honest test and the cut-off table, picks the 10 alerts and scores the assistant's answers. Tools: pandas, scikit-learn, XGBoost.
2. llm_check.py sends each alert to Gemini (3 times, once without the score, once with the trap question) and saves the answers in llm_runs.json.
3. results.json holds every number on this page; the page only displays it.

Code: every file, viewable and downloadable · analysis.py · llm_check.py · results.json · llm_runs.json (all 50 AI answers)

Run: download KDDTrain+.txt and KDDTest+.txt into data/, then python3 analysis.py, then GEMINI_API_KEY=... python3 llm_check.py, then python3 analysis.py again.