Conditional validity of quantum event classifiers under collider systematics and quantum estimation uncertainty

AI summary

Overview

Research area: Quantum machine learning applied to high-energy particle physics, combined with statistical methodology for validating scientific claims under distribution shift and finite-shot quantum measurement noise.

Technical level: Advanced. The paper assumes familiarity with quantum kernel methods (fidelity kernels, ZZ feature maps, Gram matrices), collider nuisance parameters and profile-likelihood inference, and modern anytime-valid sequential statistics (confidence sequences, optional stopping, e-values/betting constructions). The empirical content is dense and replication-heavy.

Scope (one sentence): The paper builds a fail-closed, information-conditional auditing framework that asks which claims about a deployed quantum event classifier remain justified when both collider systematics and finite-shot quantum kernel estimation act simultaneously, and instantiates it on the FAIR Universe HiggsML Uncertainty benchmark with a quantum-kernel SVM, matched classical controls, a public CMS Run2012 H→ττ sample, and a micro-scale IBM QPU run.

What This Paper Is About

Machine-learning classifiers used in particle physics are usually validated on nominal simulation, but deployed under uncertain detector calibrations and resolutions, and — for quantum models — under finite measurement shot noise that makes the realized model itself random. The authors ask a narrow, practical question: given only the information actually available at deployment time, which statements about a quantum classifier's performance can still be certified, which must be rejected, and which must be left open? Their answer is a decision framework with three verdicts (SUPPORTED, REFUTED, UNRESOLVED) applied across a hierarchy of information sets, together with the finding that a classifier can look stable while the downstream physics inference built on it is invalid.

Key Contributions

1. An information-conditional, fail-closed auditing framework. The authors formalize a hierarchy of information sets — I0 (source data plus the frozen deployment), I1 (unlabeled target features conditioned on the selected count), I2(n) (n queried binary labels plus the nominal weights revealed with them), and I3 (counts, control-region yields, exposures and nuisance estimates). A claim is certified only if the declared information resolves it at a declared error rate; heuristic signals may veto certification but never grant it.

2. An exact algebraic reduction of physics-weighted ratio claims to a bounded-mean confidence sequence (Theorem 1), plus an unidentifiability result (Proposition 1). For a fixed threshold τ chosen before labels are observed, the authors define Z_i(τ) = (u_i(c_i − τ) + τ·w_max)/w_max, show Z_i ∈ [0,1] and R ≥ τ ⇔ E[Z(τ)] ≥ τ exactly, so any time-uniform confidence sequence for a bounded mean yields per-claim Type-I control under optional stopping. Proposition 1 shows that for weight-only nuisances the feature-only I1 experiment has no distinguishing power (power at most its size α), while count-based I3 tests can have nontrivial power.

3. A separation of classifier validity from scientific-inference validity. Using a three-level inference ladder (L1: deployment-blind counting estimator; L2: score-binned Poisson profile likelihood with per-process template morphing over the official ±1σ/±2σ grid and all six nuisances profiled; L3: L2 with the actually-shifted nuisance family omitted), the authors show that signal-strength coverage can collapse at nearly unchanged AUC, and that a nominal-environment coverage gate is required per model before any shifted-environment number is interpretable.

4. A treatment of quantum estimation as changing claim semantics. They distinguish deployment-relative claims C_dep(ω): M_T(f̃_ω) ≥ M_S(f̃_ω) − δ from ideal-anchored claims C_ideal(ω): M_T(f̃_ω) ≥ M_S(f⋆) − δ, retain conditional per-claim validity for both (Proposition 2), and state a sufficient sign-stability condition (Proposition 3). Across 30 frozen finite-shot quantum-kernel deployments, every realized Gram matrix is propagated through refitting, calibration and threshold selection.

Main Findings

• No quantum advantage is claimed, and matched classical controls remove the apparent quantum effects. At a matched 2000-event budget over five training seeds, QK-SVC reaches a physics-weighted nominal AUC of 0.848 ± 0.022, above a full-feature RBF in 4/5 seeds and above a linear SVC in 5/5, but consistently below tuned trees (QK−XGB = −0.035 ± 0.013, negative in 5/5 seeds). The matched control — RBF on the identical 8 features — reaches 0.859 ± 0.016, with no consistent separation detected. The earlier "QK above RBF" contrast is attributed to a feature-set effect, not quantumness.

• Absolute benchmark metrics do not transfer across data draws. On a fresh holdout, QK−XGB = −0.039 and QK−RBF8 = −0.008 (paired structure preserved), yet every model's absolute weighted AUC sits 0.067–0.098 lower, with unweighted AUCs essentially unchanged (0.839–0.845 / 0.875–0.877 for the trees). The mechanism is heavy-tailed signal weights (max/mean ≈ 420–440; effective sample size ratio 0.005, i.e. ≈46 effective signal events per 41k-event test role), giving bootstrap CI half-widths of 0.04–0.09. Over four row-disjoint worlds, the between-world standard deviation of absolute weighted AUC is 0.030–0.050 per model (ranges up to 0.121), and the development world's re-partition variance understates the total by about a factor of two (contrast std 0.023 across worlds vs 0.013 across partitions).

• Small systematic-induced degradations are within-world replicable but draw-dependent across worlds. Within the development world, TES down-shifts degrade the QK-SVC in 5/5 seeds (+0.0024 ± 0.0010 at −2σ; the up-shift arm does not replicate) and the adverse combination degrades it in 5/5 seeds (+0.025 ± 0.024); both signs reproduce on the holdout (+0.0011; +0.0081). Two additional worlds triggered the registered cross-world falsifier: the adverse combination improves the QK-SVC in one (−0.0090) and the TES down-shift improves it in the other (−0.0050). At |ΔAUC| ≈ 0.001–0.01, these sit below the between-world variability of the paired contrasts themselves (±0.02). Weight-only nuisances have a weighted-AUC effect at the 4·10⁻⁴ level, with uniform background scaling exactly zero.

• Label-free shift sensing is real but only veto-capable, and not quantum-specific. On the development grid, frozen MMD² sensors predict replicated degradation out-of-environment (quantum ρ_S = 0.56 own-model, 0.68 transfer; matched rbf8 0.73/0.60). On 48 never-seen out-of-grid environments per world, pooled rank association is positive in both worlds and both sensors (best quantum → own 0.65; secondary world 0.22–0.62), while JES folds sit at the noise floor. Rank prediction generalizes; magnitude prediction does not (leave-one-family-out isotonic MAE 0.0005–0.012 against target means 0.0003–0.012). Blindness to normalization nuisances is exact under common random numbers: weight-only environments are byte-identical to nominal, which also degenerated the development-era veto floor and forced a re-based null (σ ≈ 7·10⁻⁵ quantum, 1.6·10⁻⁴ rbf8), under which 4–8 of 48 out-of-grid environments alarm — only the soft-MET family and the strongest prior draws.

• Fail-closed certification keeps measured error rates below α = 0.05 per claim. Unweighted, across 19,680 correlated claim-stream evaluations, false certification is 0.61%, false refutation 0.03%, and 98% of near-boundary false claims end UNRESOLVED at n = 3,000; the confirmatory holdout gives 0.69%. A fresh-world replication on archived deployment scores (certified byte-identical against a full re-derivation before any audit ran) gives 0.36% unweighted and 0.08% weighted (6/7,980). Weighted, the predeclared Monte-Carlo battery shows a worst false-certification cell of 1.5% against an 8.3% slack, and in an adversarial optional-stopping stress a naive Wald rule falsely certifies 27.8% of the time while the confidence sequence holds at 0.0%. On the benchmark, weighted false certification is 2/8,580 = 0.02% and class-conditional false certification is 0/4,700.

• Weighting costs sample efficiency, not validity. Median n*_w/n*_unw = 1.66 (IQR 1.11–3.00) on identical draws; the predeclared multiplier 2.05 exceeds the largest admissible compound scale, 2.0 × 1.01 = 2.02. A post-hoc sharp-bound sensitivity shortens jointly resolved claims by about 15% (median ratio 1.34, IQR 1.00–2.10) while reducing the resolved fraction from 36.0% to 35.0% and raising weighted false certification from 2 to 12 of 8,580 (0.14%). Audit-label draw budgets are 1.5–3.4 times a Wald-style information yardstick. 536 streams retreat from SUPPORTED to UNRESOLVED under fail-closed hardening, and 1 flips SUPPORTED → REFUTED — evidence that the metric named in a claim changes which claims are at risk.

• Weighted balanced accuracy is information-limited, not machinery-limited. A registered oracle follow-up shows weighted background rejection resolving in all 200/200 runs at margin 0.05 with zero errors, while weighted signal efficiency — and hence BA_w — would need n* ≈ 2 × 10⁷ uniform labels at margin 0.05, because the signal carries 9.7 × 10⁻⁴ of the weight mass. The authors therefore audit the components (the physics quantities) directly.

• Auxiliary-template quality bounds what I3 can restore. A pure-Poisson control-region fit failed its registered coverage falsifier (ŝ_ttbar biased +0.07 with zero coverage), because analyst templates carry Monte-Carlo statistics (2.4% in the ttbar-enriched region at this sample size) that a pure-Poisson likelihood misreads as scale shifts. The BB-lite (Barlow–Beeston-inspired Gaussian aggregate template-variance) fit is coverage-valid but honest about resolution: s_ttbar to ±10%, and s_bkg's interval saturating the official clip range in 99% of replications. Predeclared tight bands stay UNRESOLVED; only clear violations refute (12/400 refutations of the genuinely false |s_tt − 1| ≤ 0.02 claim at ttbar_scale = 1.04, zero false certifications). The diboson scale, with no viable control region in this feature space, stays UNRESOLVED by construction.

• The wrong-estimand risk at I2 is structural but unrealized at official magnitudes. The gap between true-weighted and nominal-weighted estimands is ≤ 0.003 at official normalization shifts — below certification resolution at n = 3,000 — so the I2-nominal auditor, audited against the true estimand, false-certifies 0/360, while the I3 worst-case auditor is strictly more abstinent. Feature shifts do contaminate rate evidence: under the adverse combination, selection migration biases the control-region fit (ŝ_bkg − 0.012, outside its clip coverage).

• Finite-shot quantum estimation changes the deployed object even when ranking does not move. In the primary raw pipeline, the finite-shot perturbation leaves ranking relatively stable (median Spearman correlation 0.92 between realized and ideal decision functions; median nominal-AUC change +0.001) yet moves thresholded target metrics by a median of about 0.02, mainly through downstream recalibration and operating-point selection, flipping ideal-anchored claims. The diagonal-loading sensitivity decomposes differently. Proposition 3's sufficient sign-stability condition is instantiated in 68.7% of 7,200 condition cells by a deterministic replay of 30 frozen finite-shot deployments, with verdict-flip rates of 9.2% when it holds and 60.4% when it fails — informative but sufficient-not-necessary. All evaluated far-margin deployment-relative claims were true and remained SUPPORTED in every raw and PSD-repaired realization, whereas ideal-anchored verdicts are heterogeneous across deployments. The historically executed RAW-INDEFINITE analysis is preserved, with minimum diagonal loading used only as a declared post-hoc sensitivity.

• Empirical ledgers inherit nothing automatically. A fail-closed ledger on the complete public CMS Run2012 H→ττ sample certifies only the aggregate claims its control-region information supports and refuses event-level accuracy by construction; the CMS ledger and I3 profile-likelihood procedures use their own calibrated tests and coverage gates rather than inheriting Theorem 1.

Methodology in Plain English

The authors treat a trained classifier as a frozen deployment: a fixed model, calibration and decision threshold, fitted only on nominal (θ = 0) data, with nothing retuned per environment. Each detector miscalibration (the tau and jet energy scales, and a soft missing-transverse-energy component) defines a different environment; some nuisances only rescale rates, while others move events in and out of the selection.

They then grade claims by the information available. With no target labels, a claim can only be vetoed, never certified — they enforce this with a label-free sensor comparing kernel Gram matrices (a maximum mean discrepancy, MMD²) between a source anchor and unlabeled target draws, using common random numbers across environments. With target labels, they turn each claim into a sequential test: because sampling is with replacement, per-event correctness is an exact Bernoulli stream, and for weighted accuracy they use an algebraic trick that maps the weighted ratio claim onto a bounded-mean quantity in [0,1], to which a time-uniform confidence sequence applies. Verdicts come from whether the running lower bound clears the threshold (SUPPORTED), the upper bound falls below it (REFUTED), or neither (UNRESOLVED). A claim that never leaves UNRESOLVED stays unresolved regardless of how long one waits.

For normalization nuisances, they add control-region counts, exposures and nuisance estimates, fit the rate scales to disjoint control regions with a template-variance term, and audit true-weighted claims by reweighting the labeled stream at every corner of a confidence box over the three scales and taking the worst case (exact because the estimand is linear-fractional, hence monotone, in each scale). They then push everything through a physics-level inference ladder, requiring the L2 profile-likelihood construction to pass a nominal-environment coverage calibration gate for each model before any shifted-environment number is read; the gate blocked two invalid implementations during development and one model (the scale-trained tree) remains gate-excluded and reported as such.

For the quantum side, the model is an SVM over a fidelity kernel built from a ZZ feature map (one qubit per feature, entangling repetitions, a global bandwidth scale) on standardized inputs. Exact statevector, finite-shot (compute–uncompute sampling, drawn independently per Gram entry as a device would) and hardware estimates of the same kernel are compared, and each of 30 frozen finite-shot realizations is carried all the way through refitting, recalibration and refrozen threshold selection. A micro-scale full-pipeline IBM QPU run tests fail-closed consistency only — not performance or certification at scale. Throughout, the study was designed to remain reportable under every outcome, with each registered falsifier obeyed and its correction retained in the audit trail.

Why This Matters

Impact on research. The paper argues that the standard practice of reporting a nominal benchmark score is insufficient for quantum classifiers destined for scientific pipelines, and supplies a concrete alternative that is provably calibrated per claim rather than heuristically thresholded. Its most consequential result is negative and transferable beyond quantum methods: classifier validity does not imply scientific-inference validity, and a fixed-template profile likelihood can lose coverage at nearly unchanged AUC when template-statistical uncertainty is not modeled. The explicit disclaiming of quantum advantage, and the demonstration that matched classical controls reproduce every apparent quantum-specific nominal-performance and sensing effect, sets a methodological bar for QML-for-science claims.

Real-world applications:

• Detector-systematic-aware event classification at colliders. The information hierarchy and fail-closed auditor give analysts a way to state which performance claims survive unknown calibration shifts, rather than extrapolating from a nominal validation set.
• Signal-strength (μ) inference and publication-grade claims. Because coverage of the downstream profile likelihood is checked separately from classifier accuracy, the framework targets the actual quantity physicists report — not a surrogate score.
• Benchmark design and replication practice. The measured factor-of-two underestimate of total variance from partition-level replication, and the heavy-tailed-weight diagnosis (≈46 effective signal events per 41k-event test role), are directly actionable for how HiggsML-style challenges are scored.
• Auditing stochastic or hardware-estimated models in other safety-relevant settings. The deployment-relative versus ideal-anchored distinction, and the sign-stability sufficient condition, generalize to any setting where a model's own estimate is randomized by finite sampling (noisy hardware, limited shots, stochastic inference services).

Industry relevance. Any organization deploying a model that feeds a downstream quantitative decision — risk scoring, yield estimation, instrument calibration — faces the same two-layer problem: environmental shift plus randomness in how the model itself is realized. The paper's practical lessons are that (a) ranking metrics such as AUC are a poor proxy for decision-relevant metric stability, because recalibration and threshold selection can move thresholded metrics by ~0.02 while AUC moves by +0.001; (b) per-claim sequential error control is achievable at modest label cost (budgets 1.5–3.4× a Wald-style yardstick); and (c) label-free monitoring should be granted veto power only, since its rank signal generalizes but its magnitude calibration does not.

Future Directions

• Validating under independent template estimation. The authors explicitly state that their conditional diagnostic does not establish validity when templates are estimated independently, and that inference validity is jointly limited by nuisance representability and auxiliary-template quality. Extending the coverage guarantee to that regime is the clearest open problem left standing.

• Making I3 resolution competitive rather than prior-limited. The BB-lite fit is coverage-valid but yields s_ttbar to ±10% and s_bkg intervals that saturate the official clip range in 99% of replications, leaving tight predeclared bands permanently UNRESOLVED. Better auxiliary-template statistics, or better-identified control regions (the diboson scale has none here

Authors’ abstract

Claims about a deployed quantum machine-learning classifier can fail when target data shift or when finite-shot quantum evaluation randomizes the model itself. We develop an information-conditional, fail-closed auditing framework that returns supported, refuted or unresolved verdicts with anytime-valid per-claim error control under a declared sampling protocol. On a Higgs-to-tau-tau collider benchmark, stable classifier metrics do not guarantee valid signal-strength inference: at the studied finite-template statistics, the fixed-template profile can lose coverage, even in shift-free controls, when its templates are estimated independently and template-statistical uncertainty is not modeled explicitly. Across 30 frozen finite-shot quantum-kernel deployments, every realized Gram matrix is propagated through refitting, calibration and threshold selection; in the primary raw pipeline these perturbations leave ranking nearly unchanged yet move thresholded target metrics by about 0.02, flipping ideal-anchored claims, and the diagonal-loading sensitivity decomposes differently. Matched classical controls remove apparent quantum-specific nominal-performance and sensing effects. We claim no quantum advantage.