Code · Can we trust an alert model, and the AI that explains its alerts?

Code: Can we trust an alert model, and the AI that explains its alerts?

Every number on the case study page comes from these files. View any file in the browser, download one, or get them all as a zip.

File	What it does	Open
analysis.py	Loads the data, builds both models, runs the honest test, the cut-off table and the failure analysis, picks 10 alerts and scores the AI's answers. Writes results.json.	View · Download (9.4 KB)
llm_check.py	Sends each alert to Gemini 3.1 Flash-Lite (3 times, once without the model score, once with a trap question) and saves the answers.	View · Download (2.4 KB)
results.json	Every number on the page.	View · Download (7.9 KB)
llm_runs.json	All 50 AI answers, word for word.	View · Download (14.2 KB)

Download all (alert-model-check.zip)

Run it yourself

Download KDDTrain+.txt and KDDTest+.txt (NSL-KDD, University of New Brunswick) into a folder named data/.
pip install pandas scikit-learn xgboost
python3 analysis.py
Optional, to ask the AI again: GEMINI_API_KEY=... python3 llm_check.py, then python3 analysis.py once more.

Data: NSL-KDD (Canadian Institute for Cybersecurity, UNB) · Google Gemini 3.1 Flash-Lite. The data files are not re-hosted here; the links go to the original source.