How Quantum Is the Advantage? A Fair, Calibration- and Noise-Aware Benchmark and Attribution Audit of Quantum Machine Learning for Network Intrusion Detection

AI summary

Overview

• Research area: Quantum machine learning applied to network intrusion detection (QML-IDS), with a focus on benchmarking methodology and causal attribution of reported quantum gains.
• Technical level: Advanced. The paper assumes familiarity with variational quantum circuits, quantum kernels, PCA-based dimensionality reduction, calibration metrics (ECE, Brier), NISQ noise channels, and multiple-hypothesis correction (Benjamini-Hochberg).
• Scope (1 sentence): A single leakage-controlled, equal-budget benchmark of hybrid variational quantum circuits and quantum-kernel SVMs against five tuned classical baselines across four NIDS datasets, extended with a "quantum-attribution audit," a simulated noise sweep, and false-discovery-rate-controlled significance testing.

What This Paper Is About

Published QML-for-intrusion-detection papers routinely report accuracies and F1 scores in the 97 to 99.9% range, but recent rigorous work suggests these gains come from classical preprocessing (PCA) and implicit regularisation rather than from quantum effects. Rather than asking whether a quantum model can post a high score, the authors ask how much of any advantage is genuinely quantum. They build one fair, reproducible protocol and an audit that decomposes each measured difference into its quantum and classical parts.

Key Contributions

1. A fair, reproducible multi-dataset QML-IDS benchmark. Hybrid VQCs and QSVMs are evaluated against five honestly-tuned classical baselines (Random Forest, XGBoost, RBF-SVM, tuned MLP, 1D-CNN) on NSL-KDD, UNSW-NB15, CICIDS2017, and NF-ToN-IoT-v2 for binary detection, under one protocol with strict leakage control and official splits honoured where they exist. Code, seeds, and splits are released.
2. An equal-budget fairness design. Two explicitly labelled feature views are reported: a full view (classical models on all features) and a same-budget view (every model on the identical low-dimensional input the quantum models receive — 8 features = 8 qubits in the main configuration), removing the silent feature-budget asymmetry that confounds prior work.
3. A quantum-attribution audit. For each quantum model the authors build a parameter-matched classical control sharing the identical classical front-end, add a random-feature kernel as the explicit classical surrogate for the quantum feature map, and run a regularisation sweep — a direct constructive test of the Bellante et al. critique in the practical NISQ regime.
4. Operationally meaningful evaluation. Beyond accuracy/F1, the paper reports detection rate and false-positive rate together, low-FPR operating points (TPR@0.1%FPR, TPR@1%FPR), calibration (Brier, ECE), significance testing with Benjamini-Hochberg FDR control, and degradation under depolarising, amplitude-damping, phase-damping, and bit-flip readout noise.

Main Findings

• Classical wins on aggregate, everywhere. Under the same-budget view, the best classical baseline beats the leading 8-qubit quantum model on F1 on every dataset: NSL-KDD 0.782 vs. 0.730; UNSW-NB15 0.891 vs. 0.836; CICIDS2017 1.000 vs. 0.968; NF-ToN-IoT-v2 0.991 vs. 0.949. Full-view classical F1 was 0.803, 0.912, 1.000, and 0.996 respectively.
• The audit attributes the "gain" to classical components. A parameter-matched classical MLP sharing the quantum front-end reached AUPRC 0.944, and tuned Random Forest reached F1 0.764 — matching or exceeding every hybrid VQC on aggregate metrics. For the twelve-qubit hybrid, classical controls won significantly (F1 −0.039, p=0.021; AUPRC −0.036, p=0.035; TPR@1%FPR −0.121, p=0.026), consistent with overfitting at width rather than a quantum benefit.
• Two quantum advantages survive FDR control. (1) The IQP-encoded quantum-kernel SVM beats its direct classical analogue, the random-feature (Nyström/RBF-sampler) kernel, on AUPRC (Δ=+0.075, p=0.001, BH q=0.011), ROC-AUC (Δ=+0.134, p=0.002, q=0.018), and TPR@1%FPR (Δ=+0.152, p=0.017, q=0.047). (2) A four-qubit hybrid VQC reaches TPR@1%FPR = 0.517 versus 0.467 for the tuned Random Forest (Δ=+0.050, p=0.005, BH q=0.030) on NSL-KDD.
• The operating-point edge is small-circuit specific. The four-qubit hybrid also leads at TPR@0.1%FPR (0.407 vs. 0.272 for the matched MLP), but that comparison does not survive FDR control (p=0.428, q=0.62). Its F1 is essentially a tie with Random Forest (0.759 vs. 0.764, p=0.776, q=0.84), and a qubit-count ablation shows both F1 and low-FPR detection decline with width — consistent with an implicit-regularisation mechanism rather than raw expressivity.
• Random cross-validation inflates results by roughly 0.20 F1. On NSL-KDD, models scoring F1 ≈ 0.98 under within-training cross-validation collapse to F1 ≈ 0.78 on the official test split — a consistent ≈ 0.20 drop across all models and both views.
• Two datasets are near-saturated. CICIDS2017 and NF-ToN-IoT-v2 have weak discriminative power, since every tuned classical model scores F1 ≥ 0.97 there; high absolute quantum scores on those datasets should be read as "this task is easy."
• Noise degrades gracefully, and mild noise acts as a regulariser. Relative to the noiseless six-qubit run (ROC-AUC 0.846, TPR@1%FPR 0.422, ECE 0.191), depolarising, amplitude-damping, and phase-damping noise at p ∈ {0.01, 0.05} left low-FPR detection essentially unchanged (0.421 to 0.430) and slightly raised ROC-AUC to between 0.878 and 0.897. Only bit-flip readout noise materially hurt (TPR@1%FPR 0.379 at p=0.05).
• Corrected evidence still favours classical overall. Across 108 direction-aware comparisons on NSL-KDD, 13 quantum-favouring and 27 classical-favouring differences survived at q < 0.05. The authors note that the low-FPR hybrid result survives Benjamini-Hochberg but not the more conservative Holm correction, under which only the quantum-kernel-vs-surrogate results remain significant — so they treat the quantum-kernel result as the study's most robust positive.
• Calibration is comparable. ECE was approximately 0.18 to 0.22 for hybrids versus 0.193 for Random Forest, with the four-qubit hybrid marginally better.
• The CPU simulator is far faster than the GPU at these qubit counts. lightning.qubit took 1.12 s vs. lightning.gpu's 8.89 s at 4 qubits (8.0×), 2.08 s vs. 28.0 s at 8 qubits (13.5×), and 3.94 s vs. 54.3 s at 12 qubits (13.8×) — reported as an 8 to 14× gap growing with width.

Methodology in Plain English

The authors built one experimental pipeline and ran everything through it. Every dataset is split so that no test information leaks into training: NSL-KDD uses its official KDDTrain+/KDDTest+ split (whose test set contains 17 attack signatures absent from training, so the task measures generalisation to novel attacks), UNSW-NB15 uses its official CSVs, and CICIDS2017 and NF-ToN-IoT-v2 use a single seeded stratified 70/30 split. Preprocessing is fit on training data only, leaky identifier columns (IP/port tuples, flow IDs, timestamps, row indices) are dropped, and model selection happens by cross-validation inside the training set only.

Because quantum models can only accept a handful of inputs, the authors report two views: classical models on all features (the practical ceiling), and every model on the exact same 8-dimensional PCA-reduced input the quantum models get. They found that min-max scaling before PCA retains far more information than standard scaling (0.841 versus roughly 27% variance on NSL-KDD with eight components), so they min-max scale first.

The quantum side covers six hybrid VQCs spanning encoding (angle, IQP/ZZ, amplitude, re-uploading) and ansatz (strongly-entangling, basic-entangler) combinations, plus two QSVMs (an IQP fidelity-kernel and an angle projected-kernel). The classical side is five hyperparameter-searched baselines refit per seed.

The distinctive step is the attribution audit. For every quantum model the authors build a classical twin that sees the same compressed input and has a comparable number of parameters — a matched MLP for each hybrid VQC, and an RBF-SVM plus a random-feature kernel approximation for each QSVM. They also sweep classical regularisation strength to test whether tuning alone closes any gap. If a matched classical twin ties the quantum model, the "advantage" cannot be credited to the quantum circuit. Everything is repeated over fixed seeds, compared with McNemar's test, paired t/Wilcoxon tests with Cohen's d, and a paired bootstrap with 2,000 resamples, then corrected for multiple comparisons with Benjamini-Hochberg.

Why This Matters

Impact on research. The paper supplies a benchmark harness and an attribution methodology rather than another accuracy record, and it explicitly states that its contribution stands whether quantum wins, ties, or loses. It generalises the Bellante et al. regularisation critique from a fault-tolerant, PCA-specific case study to the NISQ hybrid-VQC and QSVM models the community actually deploys, across four datasets. The ≈0.20 F1 cross-validation-to-test gap is a direct caution for essentially the entire QML-IDS literature.

Real-world applications:

• Enterprise and ISP network monitoring: the honest recommendation of a tuned Random Forest or XGBoost gives practitioners a cost-effective default with no quantum hardware requirement.
• Security operations centres with strict false-alarm budgets: the TPR@1%FPR analysis directly addresses the operational constraint that false positives flood analysts.
• IoT and NetFlow telemetry security: the NF-ToN-IoT-v2 results speak to flow-level detection on constrained, high-volume telemetry.
• Research infrastructure planning: the backend-crossover finding saves quantum ML groups real compute time by showing that CPU simulation outperforms GPU simulation at these circuit widths.

Industry relevance. The audit framework gives reviewers, procurement teams, and standards bodies a concrete checklist for contesting vendor or academic claims of quantum advantage in security: match the front-end, match the parameter count, sweep regularisation, and correct for multiple comparisons. The released code, seeds, and splits lower the barrier to independent replication.

Future Directions

• Real-hardware confirmation of the low-FPR signal. The authors identify this as the single most valuable next experiment, since it would distinguish a genuine quantum inductive bias from a simulation artefact. All quantum results here are simulated, with no real-QPU access, and the noise model omits correlated/non-Markovian noise, crosstalk, connectivity, and transpilation.
• Probing larger circuit widths. The 4 GB GPU caps width at roughly 16 qubits noiseless and 12 noisy, with the main configuration at 8, so the larger-width regime remains unexplored.
• Characterising trainability directly. Barren plateaus and gradient variance are not characterised in this work, leaving open whether the small-circuit advantage is fundamentally a regularisation effect that wider circuits lose.
• Strengthening statistical power and scope. The seed budget is small and uneven (five seeds for classical baselines on NSL-KDD, three elsewhere and for all quantum runs, two for the noise sweep), and the two-sided Wilcoxon floor at three paired seeds is p=0.25. Extending to adversarial evasion, unsupervised, online, and federated detection settings is explicitly out of scope.

Target Audience

Researchers and reviewers working on quantum machine learning for security, benchmarking, or evaluation methodology; NIDS practitioners choosing between classical and quantum detectors; and journal or conference reviewers who need a concrete template for auditing quantum-advantage claims. The paper is written for readers comfortable with significance testing and quantum circuit terminology, though its central argument — that fair baselines and matched controls change the conclusion — is accessible to anyone who evaluates machine learning models.

Authors’ abstract

Quantum machine learning (QML) for network intrusion detection (NIDS) is routinely reported to reach near-perfect accuracy, yet the most rigorous studies find that well-tuned classical models remain competitive, and that apparent quantum gains may be artefacts of classical dimensionality reduction and implicit regularisation rather than genuine quantum effects. We ask not whether a quantum model can post a high accuracy, but how quantum the advantage really is. We present a unified, reproducible QML-IDS benchmark evaluating hybrid variational quantum circuits and quantum-kernel SVMs against five honestly-tuned classical baselines across four standard NIDS datasets (NSL-KDD, UNSW-NB15, CICIDS2017, NF-ToN-IoT-v2) under one leakage-controlled protocol, with an equal-budget feature view, imbalance- and calibration-aware metrics with significance testing, and a simulated NISQ noise sweep. We introduce a quantum-attribution audit (parameter-matched classical controls, a random-feature kernel, and a regularisation sweep) that quantifies how much of any gain is genuinely attributable to the quantum component. Tuned classical models (Random Forest, XGBoost) match or exceed the quantum models on aggregate detection on every dataset, and the audit attributes this to classical preprocessing and regularisation rather than quantum effects. Two advantages survive false-discovery-rate correction: the quantum-kernel SVM out-ranks its direct classical surrogate (a random-feature kernel) on AUPRC and ROC-AUC, and a small four-qubit hybrid out-detects the best classical baseline at the 1% false-positive operating point on the distribution-shifted NSL-KDD task (p = 0.005, BH q = 0.030). Code, seeds, and splits are released; our contribution stands whether quantum wins, ties, or loses.