Evaluating the performance of QEC primitives on quantum processors at large width and depth

AI summary

Overview

Research area: Quantum error correction (QEC) and quantum hardware benchmarking — specifically, characterizing mid-circuit measurement (MCM), reset, feed-forward and syndrome-extraction-style dynamic circuits on real processors.

Technical level: Intermediate. The paper assumes familiarity with stabilizer codes, QAOA and depolarizing noise models, but its core idea (use a variational algorithm whose circuit mimics a code's check structure, and watch the signal decay with depth) is explainable without detailed formalism.

Scope (one sentence): The authors introduce a low-resource, code-structured LR-QAOA benchmark that mimics the syndrome-extraction connectivity and MCM pattern of surface, color and bivariate-bicycle qLDPC codes, convert its depth-dependent signal decay into an effective two-qubit depolarizing rate, and apply it across 10 QPUs from IBM, IQM and Quantinuum.

Note on completeness: the supplied text is truncated partway through Section III.2 (mid-sentence in the discussion of Fig. 4b). Results from Sections III.4, III.5 and III.6 — the code-structure evaluation, the shot-cost ranking experiment and the quantitative memory-versus-benchmark comparison — are therefore not reported in the material available here, and no numbers for them are given below.

What This Paper Is About

Fault-tolerant quantum computing depends on repeated syndrome extraction, which uses ancilla–data entangling gates, mid-circuit measurement, reset, feed-forward and idling of spectator qubits. These primitives are usually assessed either one at a time, in isolation, or through full logical-memory experiments that demand substantial experimental expertise and sampling resources. This paper proposes a middle path: a benchmark circuit built directly from a chosen QEC code's check structure, which reproduces that code's syndrome-extraction connectivity and measurement pattern while producing a simple algorithmic signal — the approximation ratio r — whose decay with circuit depth acts as a stand-in for accumulated hardware error.

Key Contributions

1. A code-structured LR-QAOA benchmark. The cost Hamiltonian of linear-ramp QAOA (LR-QAOA) is constructed from the check structure of a target QEC code — surface-code plaquettes (four-body bulk, two-body boundary), triangular color-code faces (six-body bulk, four-body boundary), or the uniform-weight checks of a bivariate-bicycle qLDPC code — so the circuit inherits the code's syndrome-extraction connectivity and MCM pattern. Number of QAOA layers p plays the role of the number of repeated syndrome-extraction rounds.

2. A mapping from algorithmic decay to an effective hardware error. Using a normalized overlap between noisy, noiseless and random-sampling values of an observable, the authors show the signal collapses onto an exponential in accumulated error, and extract an effective depolarizing rate λ_eff that reproduces the device's degradation. They explicitly caution that λ_eff is not the physical error rate of any single operation.

3. A shot-cost theory for ranking two devices. Only the measured approximation ratio is needed to decide which of two devices is better; the paper derives the required shots S*, shows the distinguishing power peaks when the normalized overlap is 1/e ≈ 0.37 (i.e. at x* = 1/ln 2), and shows the optimal depth scales as p* ∝ 1/λ_A — so lower absolute error rates move the benchmark to larger depth rather than making ranking harder.

4. A large-scale, multi-vendor demonstration. Direct versus MCM-mediated implementations are compared across 10 QPUs from IBM, IQM and Quantinuum, with circuits containing up to 2950 MCM operations. On Quantinuum's Helios-1 and H2-1, code-structured benchmarks run for surface-code, triangular color-code and bivariate-bicycle qLDPC Hamiltonians up to 81, 91 and 48 data qubits respectively, using up to 480 MCM operations. On ibm_phoenix, the surface-code structure is compared against logical-memory experiments across 11 locations — the abstract states a correlation is observed, but the supporting numbers lie in the truncated portion of the text.

Main Findings

• MCM quality varies enormously across chips. In the minimal two-data-qubit test (H = Z_d0 Z_d1, with one ancilla mediating the interaction), the fraction of data–ancilla–data triplets scoring above the 3σ random threshold at p = 3 ranged from 97% on ibm_boston down to 48% on ibm_phoenix and 49% on iqm_garnet. Median r values were 0.778 (IQR 0.703–0.821) on ibm_boston, 0.755 (0.660–0.808) on ibm_pittsburgh, 0.686 (0.622–0.748) on ibm_kingston, 0.558 (0.471–0.653) on ibm_phoenix, 0.715 (0.639–0.754) on ibm_fez, 0.665 (0.572–0.724) on ibm_marrakesh, 0.672 (0.612–0.710) on iqm_emerald and 0.564 (0.527–0.643) on iqm_garnet. For reference, the exact noiseless value is r_ideal = 0.895 and the uniform-random band is r_rand = 0.506 ± 0.020.

• Depth degradation separates platforms. Averaged over triplets, the normalized overlap from p = 3 to p = 12 fell from 0.65 to 0.52 on ibm_boston, 0.57 to 0.46 on ibm_pittsburgh, 0.46 to 0.32 on ibm_kingston, 0.47 to 0.26 on ibm_fez, 0.37 to 0.21 on ibm_marrakesh, 0.40 to 0.06 on iqm_emerald, 0.22 to 0.00 on iqm_garnet and 0.17 to 0.23 on ibm_phoenix. The IQM devices degrade faster with depth than the IBM devices, and at p = 12 only ibm_boston and ibm_kingston retain a signal above the random threshold.

• Successive hardware generations improve measurably. The transition from Heron revision r2 (ibm_kingston) to r3 (ibm_boston) shows a substantial improvement, similar in size to the garnet-to-emerald step on IQM. For IBM devices the signal keeps improving up to p = 6, indicating that at those depths the algorithmic signal is stronger than accumulated circuit noise.

• Best-triplet performance approaches the trapped-ion baseline but does not match it. On the best ancilla of each QPU the IBM-revision separation persists. On quantinuum_helios-1E (a noisy emulator, included as a trapped-ion baseline) the normalized overlap stays between 0.97 and 1.00 for p = 3 to 12, corresponding to r = 0.890–0.982.

• Larger problems are less sensitive per unit of accumulated error. From depolarizing simulations at 3 to 15 qubits, the susceptibility of the approximation ratio scales inversely with system size, κ_r = (3.69 ± 0.04)/N_q, while the success-probability susceptibility saturates near κ_P ≈ 0.965. This means the approximation-ratio signal becomes less sensitive to accumulated error as the cost function becomes more extensive.

• Connectivity, not just noise, blocks some experiments. IQM's conditional-control architecture confines feed-forward to predefined qubit groups, with each target receiving conditional control from only one source — preventing 1D chains longer than two data qubits and preventing direct mapping of the QEC code structures studied here. On IBM heavy-hex devices, limited connectivity prevents direct mapping of the code structures without extra routing. The authors note this restriction is not a fundamental obstacle to a logical-memory experiment but does constrain adaptive fault-tolerant protocols such as magic-state cultivation and injection.

• Correlation with logical memory is claimed but its numbers are not in the available text. On ibm_phoenix, the surface-code structure at d = 3 was run in both Z and X bases alongside memory experiments in both bases at 11 placements within a single experimental window, with 4000 repetitions per memory configuration at R ∈ {2, 3, 5, 7, 10} rounds and 1000 samples per LR-QAOA point at matching depths p = R. The abstract reports that a correlation between benchmark and logical performance is observed across different regions of the QPU; the quantitative comparison is in the truncated portion.

Methodology in Plain English

The authors wanted a circuit that looks and behaves like a QEC syndrome-extraction circuit — same connectivity, same mid-circuit measurements, same repeated structure — but that produces a clean number they can track.

They got this by taking LR-QAOA, a version of QAOA where the cost and mixer angles follow linear ramps fixed by only two variational parameters, and building its cost Hamiltonian directly out of a code's checks. A surface code gives four-body bulk plaquettes and two-body boundary terms; a triangular color code gives six-body and four-body faces; a bivariate-bicycle qLDPC code gives a single family of equal-weight terms on an ℓ × m torus (for the BB[[18,4,3]] instance, 14 five-body terms on 18 data qubits). Multi-body Z terms are implemented with an ancilla that is mid-circuit measured, so the circuit exercises exactly the MCM-plus-feed-forward machinery that QEC needs. Each additional QAOA layer plays the role of one more syndrome-extraction round.

The performance metric is the approximation ratio r, computed from the measured bitstring distribution relative to the best and worst possible energies. Ideally r improves with depth; on hardware, noise drags it down. The device's decay is compared against a family of depolarizing-noise simulations and reduced to a single effective rate λ_eff. Because a direct CNOT-only implementation and an MCM-mediated implementation have the same logic and differ only by the inserted measurements and feed-forward, the gap between their λ values isolates the hardware cost of MCM.

Before going to hardware, they also work out how many shots are needed to statistically distinguish two devices given a relative error-rate gap δ; the answer scales as 1/δ² and is minimized near the ideal-to-random transition region.

For the code-structure tests they used Quantinuum's Helios-1 and H2-1 (which provided all required elements at evaluation time) and ibm_phoenix, and for the memory comparison they built distance-3 surface-code memories on ibm_phoenix, decoding with minimum-weight perfect matching on a stim-generated detector error model (uniform circuit-level noise used only as decoder edge weights, not as a device model). A logical failure is a decoded observable disagreeing with the measured logical value.

Why This Matters

Impact on research. The benchmark occupies an intermediate tier between device calibration and full logical performance. It is cheap enough that institutions without deep experimental expertise or large sampling budgets can use it to check whether a vendor's roadmap claims are materializing, and structured enough that the result is tied to a specific code's syndrome-extraction requirements rather than to generic gate fidelities. Because the circuit is built from a code's checks, the same protocol can be reused as codes and target architectures change, and it produces an interpretable hardware quantity (λ_eff) that can be tracked across device generations.

Real-world applications (potential):

• Vendor-neutral acceptance testing of dynamic-circuit capability (MCM, reset, feed-forward) before purchasing time on a platform.
• Hardware–code co-design: testing whether a proposed code's check connectivity actually maps onto a given chip's coupling graph and conditional-control rules.
• Pre-screening placements on a single chip to find regions where a full decoded-memory experiment is worth running.
• Tracking maturity of measurement-conditioned operations needed for non-Clifford operations, such as magic-state cultivation and injection.

Industry relevance. LR-QAOA has already been adopted in benchmarking frameworks including Metriq-Gym and IonQ's application-oriented benchmarking suite, so the workflow here connects to existing community infrastructure. The paper also gives a concrete cross-vendor comparison at a moment when several roadmaps explicitly target QEC demonstrations, and identifies MCM/feed-forward architecture constraints (e.g. IQM's single-source conditional control) that matter for any vendor claiming QEC readiness.

Future Directions

• Repeating the code-structured suite on more processors. Only quantinuum_helios-1, quantinuum_h2-1 and ibm_phoenix provided all the elements needed for the code-structure tests at evaluation time; extending this to heavy-hex devices would require routing or code-adapted structures that respect the coupling graph.
• Filling in the quantitative memory correlation. The abstract claims a correlation with logical-memory performance on ibm_phoenix across 11 locations, but the detailed comparison, including basis-resolved overlaps r_ovl^Z / r_ovl^X versus p_L^Z / p_L^X, needs to be validated at larger distances and on more devices than a single d = 3 chip.
• Learning whether lower λ_eff translates into lower logical failure rates. The open question the protocol is ultimately aimed at is whether physical improvements across hardware generations actually become improved logical performance; the shot-cost analysis says error-rate reductions push the optimal benchmark depth higher rather than making ranking easier, which complicates long-term longitudinal tracking.
• Benchmarking conditional-control architectures directly. Since feed-forward grouping constraints blocked multi-qubit chains on IQM devices, a natural next step is to define variants of the benchmark that target modular or limited-connectivity designs, as the authors themselves sketch in their proposed protocol.

Target Audience

Quantum computing experimentalists and QEC researchers who need a practical, low-overhead way to assess dynamic-circuit and syndrome-extraction performance on real hardware; benchmarking teams at national labs and supercomputing centers evaluating vendor platforms; hardware architects deciding how much of their connectivity and feed-forward budget to devote to QEC-relevant circuits; and technically literate readers who want a cross-vendor picture of MCM maturity without reading a full decoded-memory study.

Authors’ abstract

Quantum error correction (QEC) relies on repeated parity extraction, mid-circuit measurement (MCM), reset, feed-forward, and scheduling, yet these primitives are usually assessed either in isolation or through resource-demanding experiments. We introduce a few-sample benchmark of QEC-relevant primitives based on an MCM implementation of the quantum approximate optimization algorithm (QAOA) with a fixed set of linear parameters (LR-QAOA). For a chosen code, the QAOA Hamiltonian is constructed from its check structure, such that the resulting circuit mimics the syndrome-extraction connectivity and MCM pattern while producing a direct algorithmic signal, the approximation-ratio r. The LR-QAOA depth, defined by the number of QAOA layers, plays a role analogous to the number of repeated syndrome-extraction rounds in a QEC experiment. From QPU executions, the decay of r with depth defines an effective hardware error, which we map to an equivalent two-qubit depolarizing rate λeff. We compare direct and MCM-mediated implementations across 10 QPUs from IBM, IQM, and Quantinuum, with circuits containing up to 2950 MCM operations. On Quantinuum's Helios-1 and H2-1, we run code-structured LR-QAOA benchmarks for surface-code, triangular color-code, and bivariate-bicycle qLDPC Hamiltonians up to 81, 91, and 48 data qubits, respectively, using up to 480 MCM operations. On IBM ibm_phoenix, we implement the surface-code structure and compare the LR-QAOA response with logical-memory experiments, observing a correlation between the benchmark and the logical performance across different regions of the QPU. Its construction and low resource requirements provide a practical benchmark for comparing hardware generations and QEC implementations before full logical-memory experiments are performed.