Impact of Data Loss in Postprocessing on Training and Inference of Quantum Neural Networks

AI summary

Overview

Research area: Quantum machine learning software, specifically the classical postprocessing layer that sits between a quantum processor and a hybrid training or inference loop. The case study is the SamplerQNN class in the Qiskit Machine Learning library.

Technical level: Intermediate. The paper is readable without deep quantum mechanics, but it assumes familiarity with concepts such as qubits, transpilation, measurement shots, and variational training loops.

Scope: A single, precisely diagnosed data-loss bug in SamplerQNN postprocessing, its measured effect on inference and training across five experiments on two IBM backends, and the layout-based marginalisation fix merged as Pull Request #1041.

What This Paper Is About

Quantum machine learning code is usually written and tested on simulators, where measurement results look one way; real hardware returns measurement results in a different format, and the mismatch can silently delete most of the data before it ever reaches the model. The authors show that SamplerQNN filters measurement bit-strings using an assumption that only holds for simulators, so on real devices it discards 85 to 99.6% of valid shots and passes an unnormalised probability vector downstream without any warning. The paper traces the mechanism, quantifies its effect on both prediction and optimisation, and presents the fix that was merged into the library.

Key Contributions

1. A diagnosed root cause, not just a symptom. The authors identify the exact filter in SamplerQNN._postprocess — a check that discards any bit-string whose integer value exceeds 2^n, where n is the number of virtual qubits — and explain why it is valid for simulator output but invalid for hardware output spanning over 100 physical qubits.

2. Two distinct failure modes. They separate data loss caused by the transpiler placing virtual qubits at high physical indices (Case 1) from data loss caused by noise on ancilla qubits flipping bits that are still measured and still enter the integer conversion (Case 2), showing the second mode triggers even when virtual qubits occupy positions 0 to 3.

3. Quantified harm to both inference and training. Using five experiments on ibm_kingston (156 qubits) and ibm_torino (133 qubits), they show accuracy falling from 0.94 to 0.39 on identical raw measurements, and the loss signal reaching the optimiser compressed by 22 to 27 times during on-device fine-tuning.

4. A merged upstream fix with backward compatibility. A layout-based marginalisation patch reads the physical qubit positions from the circuit layout, extracts only those bits per shot, and rebuilds probabilities in virtual qubit space — preserving the original behaviour when no layout exists, as in local simulation.

Main Findings

• Shot survival is catastrophic and position-dependent. Across the five experiments, surviving shots ranged from 15.2% (4-qubit model on ibm_kingston, positions [0,1,2,3]) down to 0.2% (VQC with ZZFeatureMap on ibm_torino, positions [61,62,54,60]). Other rates were 4.1% (7-qubit CS, ibm_torino), 0.4% (SI+FE, positions [108,109,110,118]), and 5.2% (7-qubit VQC Z).

• Inference accuracy collapses without changing the hardware run. For the most affected experiment, both columns were computed from the same 4096 raw shots: the original filter retained 16 shots and gave accuracy 0.39, while marginalisation used all 4096 shots and gave accuracy 0.94. F1 for that model moved from 0.37 to 0.93.

• The filter reproduces the original results exactly. Method C (the filter) matched not only accuracy but all 35 individual classification metrics across the five experiments, confirming the discrepancy came from postprocessing rather than hardware execution or circuit design.

• A misleading architectural conclusion was overturned. Before correction, the Classifier Sampler with small-angle initialisation and full entanglement looked like a failure at 0.39 accuracy, which could plausibly have been blamed on noise sensitivity in a denser, more entangling circuit. After correction it reaches 0.94, identical to the standard Classifier Sampler. The real gap is between the VQC architecture and the Classifier Sampler, not between entanglement strategies.

• The VQC with ZZFeatureMap shows genuine but moderate hardware degradation. After correction it reaches 0.54 accuracy on hardware versus 0.70 on the simulator, with F1 improving from 0.31 to 0.50 — a shift from "complete failure" to "moderate hardware degradation."

• Training is harmed more severely than inference. In five COBYLA iterations run directly on hardware (ibm_torino for 4 qubits, ibm_fez for 7 qubits), the 4-qubit model's correct loss at iteration 1 was 0.33 but the optimiser received 0.012, a 27× reduction; the 7-qubit model's expected 0.41 arrived as 0.019, a 22× reduction. The optimiser saw a near-constant signal of roughly 0.01 to 0.02.

• The symptom mimics a barren plateau. Near-constant loss and vanishing gradients are the classic signature of a barren plateau, but here the cause is a compressed probability vector rather than circuit expressibility, making misdiagnosis likely.

• COBYLA never took a single optimisation step. With 12 trainable weights for the 4-qubit model, the initial simplex requires 13 evaluations; with 21 weights for the 7-qubit model, it requires 22. Only five hardware iterations were run, so 5 of 13 and 5 of 22 vertices were built. At roughly 8 minutes per iteration, completing the simplex alone would have needed 104 or 176 minutes against a total allocation of 180 minutes.

• Weight drift confirms simplex probing. Every non-zero parameter change was exactly +1.0 rad, matching the rhobeg perturbation, with no genuine optimisation. For the 7-qubit model all four perturbations were kept and the correct loss rose monotonically from 0.41 to 0.44.

• The bug existed in every released version tested. The filter is present in 0.8.4, 0.9.0, and the main branch at the time of reporting. It went undetected because the default internal QMLSampler is state-vector based and returns n-bit strings, so the threshold was never exceeded, and because earlier IBM devices had roughly 30 qubits or fewer, making low-index placement more likely.

• Inference can be recovered, training cannot. If raw counts were saved, correct inference results can be recomputed by marginalisation. On-hardware training cannot be corrected after the fact, because each uncorrected loss value already shaped the subsequent weights.

Methodology in Plain English

A user circuit has a small number of virtual qubits that do the actual computation. Before running, the transpiler maps them onto a larger chip of physical qubits — 156 on ibm_kingston, 133 on ibm_torino — and the remaining qubits, called ancillas, do nothing useful. The hardware still returns a full-length bit-string for every shot, covering every measured qubit.

SamplerQNN then converts each bit-string to an integer and keeps it only if that integer is below 2^n. If the four virtual qubits sit at positions [108, 109, 110, 118], any non-zero virtual outcome produces an integer around 1.9 × 10^33, which is thrown away. Even when virtual qubits sit at positions 0 to 3, a single noisy ancilla flipping at, say, position 20 produces an integer of at least 10^6 and the shot is discarded anyway.

To measure the damage, the authors trained five models on a noisy classical simulator using Qiskit Aer with a noise model extracted from FakeJakartaV2 and 1024 shots per circuit evaluation, then transferred the frozen weights to hardware for inference. They pulled the raw counts back from every hardware job and computed predictions three ways from the same data: over all physical qubits (a naive baseline), by marginalising to the virtual positions using the layout, and by replaying the library's filter exactly. Comparing these three isolates the software effect from the physics.

For training, they took the pre-trained models and ran five COBYLA iterations on hardware, then recomputed the loss from the stored raw measurements for all ten optimisation jobs. They also logged the trainable weights at each iteration to check whether the optimiser was actually optimising or still building its initial simplex.

Why This Matters

The paper reframes a common pattern in quantum machine learning: results that degrade on real hardware are routinely attributed to noise, decoherence, or barren plateaus. This work shows a software assumption at the boundary between library code, transpilation, and hardware output can produce the same signature, and that the two are hard to tell apart without checking whether raw bit-string counts normalise to 1. For a field where small hardware budgets and limited device access already constrain reproducibility, silently losing 85 to 99.6% of shots is a serious confound.

• Clinical classification: the study's own task is binary breast cancer classification, where the gap between 0.39 and 0.94 accuracy would lead to entirely different judgements about whether a model is deployable.

• Hybrid transfer learning: the paper examines a published pipeline where a pre-trained CNN backbone (ResNet18, EfficientNet-B0, or MobileNetV2) feeds a 4-qubit SamplerQNN on ibm_torino using SamplerV2; one backbone dropped 19 percentage points (91.03% to 71.79%) while another improved by 1.9%, a selective pattern the original authors attributed to transpilation overhead and stochastic gradients.

• Security and anomaly detection: a published quantum-enhanced graph neural network for intrusion detection switched to SamplerQNN with 128 shots and Qiskit ML 0.9.0 on ibm_fez for its hardware validation, matching the conditions for the same data loss.

• Any sampler-based variational workflow: the paper notes that quantum kernels and Bayesian inference tools do not call SamplerQNN internally and are not expected to suffer the same failure, which sharpens the scope of the risk.

Industry relevance: Qiskit is a widely used open-source quantum stack, and this bug sat in a released version across simulator-to-hardware transitions for the entire period hardware grew from roughly 30 qubits to over 100. The episode argues for validating assumptions at library boundaries as platforms scale, and for tighter feedback between users, maintainers, and hardware providers. The fix is merged (PR #1041, 7 May 2026) and slated for PyPI release in version 0.9.1.

Future Directions

• Re-auditing published results. The authors explicitly cannot confirm whether the mechanism affected the two studies they examined without access to raw measurement data, which invites a broader re-check of SamplerQNN papers produced on versions 0.8.4 to 0.9.0 where raw counts were retained.

• Completing the training study. The five-iteration fine-tuning never finished COBYLA's simplex (5 of 13 and 5 of 22 vertices), so no real optimisation occurred; a properly powered run would be needed to measure the long-run effect of compressed loss on model quality.

• Systematic validation of other framework boundaries. This is one case study in one class. The same class of silent-assumption failure could exist in other sampler-derived primitives, measurement pipelines, or transpiler-dependent postprocessing across quantum software stacks.

• Better diagnostics than normalisation checks alone. The paper suggests that distinguishing software artefacts from barren plateaus and hardware noise requires routine checks on whether probability vectors are normalised and whether raw counts are stored, raising the question of how frameworks should surface such conditions automatically rather than relying on user vigilance.

Target Audience

Quantum machine learning researchers running variational models on real hardware, and anyone who has seen unexplained accuracy drops between simulator and device. It is equally relevant to Qiskit and Qiskit ML users and maintainers, to reviewers and reproducibility advocates assessing published quantum ML claims, and to research software engineers working on hybrid quantum-classical pipelines. Readers focused on quantum algorithms rather than the software layer will find the paper most useful as a cautionary methodological reference.

Authors’ abstract

As quantum hardware scales to larger devices, the classical software layers that interface with it must evolve in step. Postprocessing routines developed and tested primarily in simulator settings can encode assumptions that no longer hold on utility-scale devices, leading to data loss that can be difficult to detect from high-level model outputs alone. We present a case study of \texttt{SamplerQNN}, the sampling-based quantum neural network class in the Qiskit Machine Learning library. Here, the postprocessing method applies a filter that assumes measurement bit-strings are in virtual qubit space. On our quantum hardware runs, where bit-strings span over 100 physical qubits, this filter led to the loss of 85 to 99.6\% of valid measurement shots, depending on the transpiler's qubit placement. The resulting probability vector is unnormalised, allowing distorted prediction and loss values to propagate through the model without an API-level warning. We demonstrate the impact across five experiments on two IBM backends: for inference, accuracy drops from 0.94 to 0.39 on the same raw measurements; for training, the loss signal is compressed by 22 to 27$\times$, substantially reducing the sensitivity of the optimiser to the objective landscape. The behaviour arises in all released versions of the library (0.8.4 to 0.9.0). We implemented a layout-based marginalisation fix, merged into the GitHub codebase as Pull Request \#1041, that makes \texttt{SamplerQNN} postprocessing forward-compatible with current and upcoming hardware.