Pareto-optimal quantum kernel selection for unsupervised anomaly detection on real malware beaconing data

AI summary

Overview

• Research area: Quantum machine learning, specifically quantum kernel methods, applied to unsupervised anomaly detection on real network security data (malware beaconing traffic).
• Technical level: Intermediate — the paper is readable with a working knowledge of kernels, support vector machines, and basic quantum circuit concepts; the mathematics is presented but is not the main barrier.
• Scope: The paper proposes a fully label-free, two-objective protocol for selecting quantum kernels — maximising a detection-quality proxy (normalised pseudo discrepancy, NPD) and the geometric separation from a tuned classical kernel (geometric difference, GD) — and benchmarks fidelity and projected quantum kernels against a classical one-class SVM on real malware beaconing data, on simulators and on a 20-qubit IQM Garnet processor.

What This Paper Is About

Quantum kernel methods are often proposed as candidates for a practical quantum advantage in machine learning, but claims of advantage require two very different things to be true at once: the kernel must actually detect well on the task, and its geometry must be genuinely different from the classical kernels available for that task. These two quantities are usually reported separately, and in a real unsupervised deployment there are no labels with which to tune either of them. The paper's goal is to build a selection protocol that optimises both quantities simultaneously without ever touching a label, and to test whether the resulting models actually beat a classical detector on a difficult, data-scarce security problem: detecting periodic command-and-control traffic from infected hosts.

Key Contributions

1. A multi-objective (NPD + GD) Pareto framework for systematically searching for empirical — but not provable — quantum advantage in kernel methods, using Optuna with the NSGA-II genetic sampler and selecting the "knee point" of the resulting Pareto front.
2. NPD as a fully label-free criterion for quantum kernel hyperparameter tuning, extending a proxy previously used only for classical detectors to the selection of quantum feature maps, layer counts, OCSVM ν, and projected-kernel bandwidth γ.
3. The first unsupervised quantum kernel benchmark on real malware beaconing data, built from Stratosphere Research Laboratory traffic and the CTU-13 botnet capture dataset, with all anomalous flows held exclusively in the test set.
4. An empirical characterisation of the performance–separability trade-off, showing that fidelity kernels reach the strongest detection while staying below the g_{C→Q} ≈ √N scale, whereas projected kernels exceed that scale without a corresponding detection gain — plus a simulator-versus-hardware comparison on IQM's 20-qubit Garnet processor.

Main Findings

• The classical baseline is non-trivial but beatable. A NPD-tuned classical OCSVM with an RBF kernel (γ = 8.99, ν = 0.472) reaches AUC 0.765, F1(A) 0.599, and F1(N) 0.561 on the test set. The NPD-selected fidelity kernel beats it on the statevector simulator with AUC 0.840, but at a geometric difference of only 4.297.

• Fidelity kernels stay in the classical-equivalent regime. Across all three optimisation objectives (NPD, GD, Pareto), the Hamiltonian feature map was selected, and GD never exceeded 7.14 — well below the √N = 14.14 threshold, above which the reference classical kernel is no longer guaranteed to match or beat the quantum model. The Pareto knee sits at NPD 0.725 and GD 5.936.

• Projected kernels reach large geometric separation. The GD-maximising projected kernel reaches g_{C→Q} = 272, roughly nineteen times the √N = 14.14 threshold, using a Hamiltonian map with one layer and γ = 9.99. The Pareto-selected projected kernel reaches g_{C→Q} = 88.6 at NPD 1.421, and the NPD-selected one reaches g_{C→Q} = 32.0 at NPD 2.94.

• Large separation does not translate into better detection. The GD-maximising projected kernel (GD 272) achieves AUC 0.738, close to the classical baseline and to the NPD-selected projected kernel (AUC 0.777, GD 32), despite an almost tenfold larger GD. The Pareto-selected projected kernel reaches AUC 0.782 versus the baseline's 0.765 — a marginal gain. The best overall model remains the NPD-selected fidelity kernel at AUC 0.840 with GD just 4.297.

• Large GD can be an artefact of kernel bandwidth. The authors explain that large γ drives K_Q toward the identity matrix, for which g²_{C→Q} = ‖K_C⁻¹‖_∞, a quantity set by the smallest eigenvalue of the classical kernel matrix. A classical RBF kernel with a larger bandwidth would reproduce the same identity-like geometry; the same narrow-bandwidth limit is the memorising regime where generalisation deteriorates.

• NPD identifies a good region, not a precise ranking. Within each kernel family the NPD–AUC relationship is weak over 0 ≲ NPD ≲ 2, yet the largest-NPD configurations of both families (AUC 0.840 fidelity, 0.777 projected) lie above the classical baseline. A few individual models reached about 0.87 (fidelity) and 0.86 (projected) without having the largest NPD. The authors note this matches the original NPD study, where NPD-guided Bayesian optimisation came close to the best attainable AUC across 38 benchmark datasets (84.0 versus 85.8, against 78.7 with default hyperparameters), while the highest NPD did not always correspond to the best model.

• Noise reshapes the ranking. On the IQM noisy simulator, the NPD-selected fidelity kernel falls from AUC 0.840 to 0.790, the Pareto fidelity configuration only slightly from 0.812 to 0.798, and the GD-selected fidelity configuration actually improves from 0.666 to 0.752. The projected kernel is unchanged under NPD and GD selection (0.777 and 0.738) but drops sharply for the Pareto configuration, from 0.782 to 0.647. Under noise the best model is the Pareto-selected fidelity kernel at 0.798, and the fidelity-versus-projected margin narrows to 0.790 versus 0.777; both remain above the 0.765 baseline.

• The Pareto fronts are not converged. The authors state explicitly that the fronts reflect a budget of 100 NSGA-II trials, not converged approximations, and that the single-objective TPE runs found configurations outside the joint fronts (NPD 2.94 with GD 32.0 dominating the projected front's high-NPD end; GD 7.14 dominating the fidelity front's low-NPD end). The separation between kernel families relative to √N holds across all three runs.

• Hardware results are incompletely reported in the available text. The paper states that models were deployed on the 20-qubit IQM Garnet processor and that hardware results are evaluated on the 1,000 flows (rather than the 777 aggregated samples used for simulator results), but the specific QPU detection scores are not reported in the provided content.

Methodology in Plain English

The team treats a classical anomaly detector — a one-class SVM — as the consumer of a similarity matrix (a kernel). Instead of feeding it an ordinary classical similarity, they compute the similarity with a quantum circuit: either a fidelity kernel, which is the overlap between two encoded quantum states, or a projected quantum kernel, which compares the single-qubit reduced density matrices of those states through an RBF-like exponential with a tunable bandwidth γ. Four data encodings (Z, IQP, CNOT, Hamiltonian) and one to five layers were searched.

Because labels are unavailable, model selection uses two signals. The first is the normalised pseudo discrepancy (NPD): split the training data, train on one part, then compare the anomaly scores the model gives to the held-out real data against the scores it gives to synthetic samples drawn from an axis-aligned Gaussian fitted to the training set. A larger gap between the two score distributions suggests the model has learned what "normal" means. The second is the geometric difference (GD) to a tuned classical kernel, which measures how far the quantum kernel's geometry is from one the classical model could reproduce; values well below √N mean the classical model is guaranteed to do about as well.

They then run three searches per kernel family — maximise NPD alone, maximise GD alone, and optimise both jointly with the NSGA-II genetic algorithm — and, from the joint Pareto front, pick the knee point closest to the utopia point where both objectives would be maximal. Selected configurations are evaluated on a held-out test set using AUC and F1 reported twice (anomalies as positive and normal as positive), first on a statevector simulator, then on a noisy simulator tuned to IQM hardware characteristics (relaxation and dephasing times, gate infidelities, readout errors), and finally on the Garnet processor.

The data comes from real traffic: 200 randomly sampled normal flows for training (no malware at all), and 1,000 test flows of which 457 are anomalous and 543 normal, described by six statistical features such as forward packets per second, standard deviation of the FFT of forward traffic, and mean forward packet length. Simulator scores are aggregated per unique source–destination pair, keeping the lowest score per pair, giving 777 evaluation samples (255 anomalous, 522 normal).

Why This Matters

Impact on research. The paper makes a methodological point that is easy to state and easy to ignore: geometric separation and predictive quality are two different things and should be reported and optimised together. It demonstrates concretely that a kernel can sit far outside the √N regime purely because its bandwidth was pushed to a near-identity geometry, which is a classical artefact rather than evidence of quantum structure. It also provides the first evidence that label-free proxies developed for classical anomaly detection transfer, imperfectly, to quantum kernels — identifying a good region of hyperparameter space but not finely ranking configurations within it.

Real-world applications:

• Network intrusion and command-and-control detection, where labelled malware traffic is scarce, quickly goes stale due to concept drift, and is fragmented across organisations by privacy and security constraints.
• Unsupervised threat hunting in security operations centres, where detectors must be trained on normal traffic alone and cannot rely on a labelled validation set.
• Data-scarce anomaly detection in regulated industries such as finance and insurance, where sharing labelled incident data across institutions is legally or commercially restricted.
• Industrial and physical-process monitoring, where the same label-free selection protocol could be reused for any one-class detection task.

Industry relevance. The work is a collaboration between Multiverse Computing, IQM Quantum Computers, and the Allianz Quantum Hub, and is framed explicitly as a practical evaluation rather than a theoretical construction. Its headline conclusion — that at current noise levels neither kernel family yet reaches the classical baseline on hardware — is a useful, sober calibration point for organisations deciding when quantum kernels are worth piloting.

Future Directions

• Converging the Pareto fronts. The authors acknowledge that 100 NSGA-II trials did not explore either extreme of the front, since single-objective runs found dominating configurations outside it; larger budgets or hybrid single-/multi-objective searches are the obvious next step.
• Better label-free proxies. NPD locates a good region but does not rank configurations well within it. The paper explicitly flags the question of how far label-free proxies track detection performance as deserving dedicated study, including whether alternatives to NPD behave differently for quantum kernels.
• Closing the hardware gap. Neither kernel family reached the classical baseline at current noise levels on the 20-qubit Garnet processor. Determining which circuit structures are most noise-robust — the paper shows the impact depends more on the circuit than on the kernel family — and whether error mitigation or different encodings change the picture remains open.
• Testing on other tasks and data regimes. The framework is presented as applicable beyond malware beaconing, but whether the performance–separability trade-off observed here (fidelity kernels best in performance, projected kernels best in separation) generalises to other datasets, anomaly ratios, and contamination levels is untested. The authors also note that prior quantum-kernel anomaly work treated contamination level as known, whereas operational deployment does not.

Target Audience

Researchers working on quantum machine learning and quantum kernel methods who want a realistic, non-hyped assessment of where the field stands; machine learning practitioners in cybersecurity and fraud detection interested in unsupervised, label-free model selection; and quantum computing teams in industry — particularly those evaluating hardware readiness — who need concrete benchmarks on real data before committing to pilot programmes. The paper is also valuable for reviewers and funding bodies who need a clear template for separating "this kernel is geometrically different from a classical one" from "this kernel is better at the task."

Authors’ abstract

Quantum kernel methods are leading candidates for a practical quantum advantage in machine learning, but assessing that potential requires two quantities usually reported separately: how well a kernel performs on the task, and how far its geometry departs from the classical kernels available for the same problem. We introduce a fully unsupervised, multi-objective protocol that optimises simultaneously the normalised pseudo discrepancy (NPD), a label-free proxy for anomaly detection quality, and the geometric difference (GD) to a tuned classical reference kernel, selecting models from the resulting Pareto front. We apply it to malware beaconing detection in real network traffic, using a one-class support vector machine with fidelity and projected quantum kernels over four data encodings, on simulators and on IQM's 20-qubit Garnet processor. NPD-guided selection alone finds a fidelity kernel that beats the tuned classical baseline, but with a geometric difference too small to certify the gain as quantum. Projected kernels reach far larger geometric differences; the Pareto-selected one only marginally exceeds the baseline (AUC $0.782$ versus $0.765$, $g_{C\to Q}\approx 89>\sqrt{N}$ relative to that reference kernel), still below the NPD-selected fidelity kernel ($0.840$).