A Sim-to-Real Study of Surface-Code Decoder Benchmarking

AI summary

Overview

Research area: Quantum error correction — specifically benchmarking surface-code decoders against real hardware rather than synthetic noise models.

Technical level: Intermediate. The paper assumes familiarity with surface codes, detection events, detector error models (DEMs), and minimum-weight perfect matching, but its central argument (that a more realistic noise model does not guarantee a more realistic decoder ranking) is stated in accessible terms.

Scope (one sentence): The authors use the public below-threshold memory-experiment data from Google's Willow 105-qubit processor as ground truth to test whether synthetic noise models of increasing fidelity rank a panel of six decoders the same way the device does, and to give the first independent hardware evaluation of NVIDIA's Ising pre-decoder.

What This Paper Is About

New decoders are almost always validated against synthetic circuit-level noise, under two implicit assumptions: that a ranking obtained this way transfers to real hardware, and that it transfers better as the noise model becomes more faithful to the device. The paper tests both assumptions directly by decoding the same circuits under a graded "ladder" of four noise models (from uniform depolarizing to a syndrome-estimated DEM fitted to the device) and comparing the resulting decoder rankings to the ranking measured on Willow hardware. The goal is to determine where synthetic benchmarks can be trusted for decoder selection, and whether NVIDIA's Ising pre-decoder keeps its reported advantage when evaluated on real device data.

Key Contributions

1. A noise-fidelity ladder. A controlled ablation that decodes identical circuits under four noise models of increasing device fidelity (uniform depolarizing, SI1000, a 25-parameter circuit-level fit, and a syndrome-estimated DEM), isolating the noise model as the only variable, and measures whether each ranks decoders as the hardware does.

2. A hardware-grounded decoder benchmark. An established decoder ranking on a below-threshold device (Willow, at d = 3, 5, and 7, two bases, 15 round counts), quantified against each synthetic model using Kendall's τ.

3. The first independent evaluation of NVIDIA's Ising pre-decoder on hardware. The authors report no accuracy–latency advantage at the code distances measured, and describe the input-mapping adapter needed to run the model on Willow data at all.

4. An open pipeline. The full pipeline, four noise-model implementations, decoder harness, and per-shot outcome of every evaluation are released at https://github.com/ShayManor/Sim2Real-Decoder-Benchmark, so decoder authors without hardware access can test synthetic claims against a device.

Main Findings

• Rank agreement tracks structure, not calibration. Uniform depolarizing noise agrees weakly with hardware (τ = 0.619, 0.714, and 0.619 at d = 3, 5, and 7), and neither the d = 3 nor the d = 7 value reaches significance (p = 0.069 for both). The three rungs carrying circuit-level structure clear τ ≥ 0.810 in all nine rung–distance cells and reach exact agreement in two.

• Calibration improves absolute error but not ranking. The two calibrated models sit closest to hardware in per-cycle error rate, yet at d = 7 the generic SI1000 template reaches τ = 1.000 while the 25-parameter fit reaches 0.810 and the syndrome-estimated DEM reaches 0.905. Neither d = 3 nor d = 5 shows this inversion.

• Rank movement is confined to near-tied decoders. Tesseract ranks first and BeliefMatching second in all fifteen model–distance cells; the two Ising variants rank sixth and seventh on the three structured models and on Willow. All other movement happens among MWPM, BP+OSD, and BP+LSD. MWPM and BP+OSD differ by 6.6×10⁻⁴ per cycle on real Willow at d = 7, across a panel that spans a factor of 16.

• The fitted ranking is robust to parameter error but sensitive to sampling error. Doubling any one of the 23 free parameters reorders at most one adjacent pair; ten of the 23 reorder nothing. Across 40 bootstrap refits, Tesseract ranks first in every draw at every distance, but the fraction of draws in which any pair swaps rises from 0.03 at d = 3 to 0.20 at d = 5 and 0.62 at d = 7.

• Thin high-distance data may explain the inversion. The corpus holds nine independent patches at d = 3, four at d = 5, and one at d = 7, while the decoding graph grows with d². The largest edge probability in the syndrome-estimated DEM reaches 0.442 at d = 7; since MWPM weights edges by −log(p/(1−p)), an edge near 0.5 carries near-zero cost and is nearly free regardless of the syndrome.

• SI1000 badly underestimates absolute error. It falls one to more than two orders of magnitude below hardware and worsens with distance, reaching 1×10⁻⁵ for Tesseract at d = 7 against 1.67×10⁻³ on hardware. Uniform noise stays near hardware for most decoders but diverges for BP+OSD and BP+LSD, reaching 1.34×10⁻¹ and 1.49×10⁻¹ at d = 7.

• Matching dominates the latency–accuracy frontier. MWPM and MWPM-RL sit together at the low-latency end at every distance, followed by BeliefMatching and then Tesseract. At d = 7 MWPM takes 0.017 ms per shot, versus 26.052 ms for BeliefMatching and 305.501 ms for Tesseract. BP+OSD reaches the frontier only at d = 3. MWPM-RL is both faster and more accurate than BP+LSD at all three distances.

• Ising shows no advantage on hardware. Another panel decoder matches or improves on the Ising pre-decoder in both per-cycle error rate and decode latency in 278 of the 280 Willow evaluations; the two exceptions are the same configuration (patch q8_5, Z basis, d = 3, r = 30), once per model size. At d = 7, MWPM reaches 3.66×10⁻³ per cycle in 0.017 ms, against 2.69×10⁻² in 5.09 ms for Ising-fast and 2.60×10⁻² in 9.61 ms for Ising-accurate.

• Ising's error rate rises with code distance. Its per-cycle rate on hardware is 1.47×10⁻² at d = 3 and 2.69×10⁻² at d = 7 (a factor of 1.84), while every other decoder improves and MWPM falls by a factor of 2.72. The rise repeats on all four synthetic models, from 2.09 on the syndrome-estimated DEM to 2.69 under uniform noise. Both released models were trained for receptive fields R = 9 and R = 13, wider than the d ≤ 7 codes measured.

• Ising is nearly insensitive to the noise model. Across the five sources its per-cycle rate spans a factor of 4.1, 5.1, and 4.5 at d = 3, 5, and 7, against 9.9, 30.1, and 154.0 for MWPM and 8.2, 136.4, and 2107 for BP+OSD. The authors attribute this to it being the one panel entry that does not read the prior; its good rank agreement therefore reflects insensitivity, not benchmark fidelity.

• The devices distinguishes X and Z bases; the fitted models do not. On Willow the X basis is worse in all 24 decoder–distance cells, by 4% to 39% (mean +13.6%). The fit rung reproduces this in only 14 of 21 cells and the syndrome-estimated DEM in 11 of 21; the fit rung even reverses the count for BeliefMatching and Tesseract (12 and 11 of 42). Because Willow prepares and measures X by conjugating Z operations with Hadamards, no syndrome statistic can separate the two rates.

• Prior sensitivity is itself a selection criterion. Under uniform noise at d = 7, BP+OSD and BP+LSD reach 31.0× and 25.2× their real-hardware error, while MWPM, Tesseract, and BeliefMatching stay within 2×. Belief propagation inherits the prior through its messages; MWPM is less exposed because a flat prior leaves matching-graph distances unchanged.

Methodology in Plain English

Ground truth. The authors take the public below-threshold memory data from Willow. Each configuration prepares a logical qubit in a known X or Z eigenstate, holds it for r rounds of rotated-surface-code stabilizer measurement, and reads out, repeated for 50,000 shots. The corpus covers d = 3, 5, 7 on 14 patches, two bases, and 15 round counts — 420 configurations.

Rebuilding the decoder inputs. The device release supplies only the measurement record and the sweep bits. The authors recompute detection events from an ideal (noiseless) circuit, comparing each stabilizer against the value fixed at preparation, one round earlier, or at the closing readout. The sweep bits must be read alongside the record, since each applies a conditional X that randomizes the initial state per shot and moves both the detectors and the observable's fixed value. The single observable bit per shot is the only ground truth hardware offers.

The ladder. Four synthetic noise models feed the same pipeline, differing only in how much device information they use: uniform depolarizing (one rate for every operation), SI1000 (fixed per-operation ratios scaled by one parameter p, Willow's reference model), a 25-parameter circuit-level fit (23 free parameters on this compilation, fitted per patch from detection-event fractions and pairwise detector correlations), and a syndrome-estimated DEM (each edge estimated independently). Crucially, logical outcomes never enter any fit, and each calibrated rung is scored on fresh samples from its own model, not on the shots that shaped it.

The panel. Six decoders, each reading a DEM as prior: PyMatching (MWPM), BeliefMatching, BP+OSD, BP+LSD, Tesseract, and the Ising pre-decoder. Two run in multiple configurations, giving eight entries: Ising at two released model sizes (Ising-fast, Ising-accurate) and MWPM re-run as MWPM-RL on a reinforcement-learning-optimized DEM. MWPM-RL has no synthetic counterpart and is excluded from rank comparisons. Tensor-network decoders are excluded as impractical at d ≤ 7, and Union-Find because MWPM already represents the matching family.

Making Ising run on Willow. Willow detection events are a flat (shots, #detectors) array in XZZX emission order, while the model expects a (4, T, D, D) tensor in CSS-style order. The authors reorder detectors by a (layer, stabilizer type, x, y) key shared with a reference synthetic circuit, apply the released dets_to_predecoder_inputs transform, and choose one of four lattice orientations (XV, XH, ZV, ZH), using XV throughout and testing the choice in an appendix.

Scoring. The logical error rate is the fraction of shots whose predicted flip differs from the recorded flip, converted to a per-cycle rate by the fidelity inversion ε_cycle = ½(1 − (1 − 2ε)^(1/r)). Rank agreement is Kendall's τ per code distance. Every decoder runs on rounds 2 ≤ r ≤ 30 of every source so cross-source comparisons are fair; costlier decoders are capped at r = 70 on Willow and r = 30 on synthetic rungs, with fewer shots. In total, 5,934 evaluations are counted in Table I.

Why This Matters

Impact on research. The paper challenges a working assumption of most decoder papers: that validation against increasingly realistic synthetic noise monotonically improves the reliability of decoder comparisons. It separates two objectives that have been conflated — matching the device in absolute error rate, and predicting which decoder a practitioner should pick — and shows a model can achieve the first while failing the second. It also supplies a reusable public benchmark and the first outside check on a vendor's learned decoder.

Real-world applications:

• Decoder selection for near-term hardware teams: choosing among MWPM, belief-matching, qLDPC, and search-based decoders for a specific device and latency budget.
• Vendor validation: independently checking accuracy and latency claims for learned or neural pre-decoders before adoption.
• Noise-model engineering: guiding whether to invest in tight device calibration or in estimators that report per-edge uncertainty.
• Benchmark design for groups without hardware access: letting academic decoder authors test synthetic claims against a public below-threshold dataset.

Industry relevance. The finding that NVIDIA's Ising pre-decoder holds no accuracy–latency advantage at d ≤ 7 on real Willow data — losing in 278 of 280 evaluations — is directly relevant to groups deciding whether to adopt learned pre-decoders in their stacks. The result that MWPM remains on the Pareto frontier at every distance (0.017 ms per shot at d = 7) sets a concrete bar. The observation that the fitted models' advantage in absolute error does not translate into rank fidelity also suggests where calibration effort pays off.

Future Directions

• Estimators that carry per-edge uncertainty. The authors argue a decoder could discount unreliable edges, which would test directly whether ill-conditioning rather than code distance drives the d = 7 inversion. Building and testing such an estimator is named as the natural next step.

• Disentangling data scarcity from distance. The corpus has one patch at d = 7, so distance and available data vary together. Repeating the comparison on a device with several patches at high distance would show whether the failure disappears when the data is there.

• Richer noise models. All four rungs are stochastic Pauli models and none represents the correlated detector flips across many rounds, or the radiation-like signatures, that Arms et al. report on this device. A model built on different principles could rank differently.

• Re-evaluating published learned decoders. The conclusion (truncated in the supplied text) begins to argue that as more hardware datasets are released, published synthetic successes should be re-checked on device — a direction the authors frame around their Ising case study.

Target Audience

Quantum error-correction researchers and decoder developers who benchmark against synthetic noise; experimentalists and hardware teams choosing decoders for a specific device; engineers evaluating learned or vendor-supplied decoders; and benchmark/methodology researchers interested in sim-to-real transfer. Readers with a working knowledge of surface codes and DEM-based decoding will get the most from it, though the core lesson — circuit-level structure matters more than calibration for ranking — is legible to newcomers.

Authors’ abstract

Quantum error-correction decoders are typically benchmarked against synthetic circuit-level noise, under the assumption that a decoder's ranking under such noise transfers to hardware and improves as the noise model becomes more realistic. The Willow processor, the first to operate below the surface-code threshold, allows us to test this assumption. We rank a panel of six decoders using a four-rung ladder of noise models with increasing fidelity, evaluated against real data across three code distances, two bases, and fifteen round counts. Rank agreement with hardware appears once the noise model gives each operation type its own error rate. Calibrating the model to the device improves absolute error rates but not rank agreement. We additionally provide the first independent evaluation of NVIDIA's Ising pre-decoder on hardware, at code distances below its training receptive field and via a mapping onto the lattice on which it was trained. Under these conditions, it holds no accuracy-latency advantage: another panel decoder matches or improves on it in both per-cycle error rate and decode latency in 278 of the 280 evaluations. We release the full pipeline and the per-shot outcome of every evaluation, so future decoders and devices can be compared.