Designing Quantum Error Correcting Codes to fit decoders via Reinforcement Learning

AI summary

Overview

Research area: Quantum error correction (QEC) and machine learning for quantum hardware — specifically the co-design of quantum low-density parity-check (qLDPC) codes and their decoders.

Technical level: Advanced. The paper assumes familiarity with stabilizer/CSS codes, Pauli algebra, belief-propagation and ordered-statistics decoding, and with deep reinforcement learning (policy gradients, PPO, actor–critic architectures).

Scope (one sentence): The paper formulates the design of Bivariate Bicycle (BB) stabilizer sets as a Markov Decision Process and trains a PPO agent with a decoder-in-the-loop reward so that the generated codes are optimised for a fixed decoder rather than for an abstract code-theoretic figure of merit.

What This Paper Is About

Code design and decoder design are usually treated as separate problems, but the performance of a heuristic or approximate decoder depends strongly on the structure of the code it is handed. This paper asks the inverse question: given a fixed decoder architecture, which code maximises that decoder's performance? The authors answer it by turning BB code construction into a sequential decision problem — the agent edits polynomials one coefficient at a time — and rewarding it with the measured area under the physical-to-logical error-rate curve produced by Monte Carlo decoding under a depolarising channel.

Key Contributions

1. An MDP formulation of BB code design. The state is the current pair of parity-check matrices (H^X=[A\mid B]), (H^Z=[B^\top\mid A^\top]); an action flips at most one coefficient of each of the four polynomials (a(x), b(x), a(y), b(y)), plus a per-polynomial "no-op" bit. Transitions are deterministic. This gives a generative model that can augment an existing code minimally rather than only sampling codes from scratch.

2. A decoder-performance reward with a logical-qubit floor. The reward is the area between the empirical physical-to-logical error-rate curve and the constant function 1, normalised by the width of the error range, with a switch to a penalty term ((k-k_{\min})/k_{\min}) whenever the code has fewer than (k_{\min}) logical qubits (in which case no decoding is performed at all).

3. A size-transferable, BB-aware policy/value architecture. A three-branch trunk fuses (i) a flattened (A|B) vector of (2(lm)^2) entries through a 3-layer tanh MLP to 256 dimensions, (ii) a transformer encoder over (2l+2m) coefficient tokens with a cyclic/Fourier positional encoding (width (9+2H), (H\in{3,4,6,7,10}) harmonics), and (iii) three logical-qubit features. The fusion layer is (323 \to 256) ((323 = 64_{\text{enc}} + 256_{\text{mlp}} + 3_k)), independent of code size; the policy head is four categorical blocks of size 7 (six coefficients plus no-op), the value head is (256 \to 1).

4. A public dataset of evaluated BB codes plus a warm-start procedure. A dataset of BB codes at various parameters, Monte Carlo evaluated with 50 samples per error point, released alongside the figure-generation code. It is used for supervised pretraining of the structure encoder ("warm start"), which can be frozen for the first (N) collector batches while the randomly initialised branches settle.

Main Findings

• Compact parameterisation shrinks the search space dramatically. Because a BB code is fully determined by the four polynomials, the agent searches over binary coefficient vectors rather than over binary matrices of size ((lm)\times(2lm)).

• Decoder-aware reward needs a logical-qubit guard. The authors state explicitly that a code with no logical qubits is "highly performant" under a pure logical-error-rate reward, which would drive the agent toward codes with fewer logical qubits. The (k_{\min}) threshold and penalty term are introduced for this reason.

• Integration over the error range must be rescaled. The raw integrated area is a very small number; the authors compensate by dividing by the width of the error range. The choice of integration (rather than sampling a single error point) is described as a design choice.

• A specific catastrophic failure mode is identified. If policy entropy collapses, an actor applying only one action is either stuck on a single code or toggles between two codes, since (b(x)\oplus A^{b(x)}\oplus A^{b(x)} = b(x)). This becomes catastrophic for certain combinations of BB code parameters and agent–environment hyperparameters.

• Error-range placement matters. At physical error rates that are too high, almost no code succeeds; at rates that are too low, a fixed number of samples gives a limited view because the sampled error may be trivial, which itself changes with code length. This can trap the agent on mediocre codes that score a positive but lower-than-possible reward.

• Episode truncation is the proposed mitigation. Episodes are capped at (\mathcal{O}(l+m)) steps and terminated with a "truncated" rather than "terminated" signal, so the agent experiences a sufficiently diverse view over the collection span.

• The MLP branch does not transfer across code sizes. Its input width is (2(lm)^2), so weights trained at ((6,6)) cannot be reused at ((9,6)). The transformer encoder branch is introduced precisely because its token width is independent of (l) and (m), making it shareable and pretrainable.

• The dataset contains duplicates. The polynomial representation of a BB code is not unique, so the same code appears more than once; since two evaluations are independent Bernoulli trials of the same probability, the authors note that pooling them could improve the estimate.

• Quantitative evaluation results are not reported in the material available. Section 6 evaluates training and the resulting policy, and Table 5 lists the dataset parameters, but the truncated content provided here does not contain the measured logical error rates, reward curves, comparisons against baselines, or the number of training steps. No specific performance figures should therefore be attributed to this paper on the basis of the available text.

Methodology in Plain English

The researchers start from a family of quantum codes (Bivariate Bicycle codes) that can be written down very compactly as four short binary polynomials. Instead of asking a network to invent a full parity-check matrix, they let it make small edits: at each step it may flip one coefficient in each polynomial, or leave that polynomial alone. Every edit produces a new code.

For each candidate code the environment runs a Monte Carlo experiment: sample a depolarising error, compute the syndrome, hand it to the fixed decoder, and check whether the residual error is a stabilizer or a logical operator (decoder "failures" and silent decoder "errors" are counted equally). Doing this at five physical error rates — 0.001, 0.00316228, 0.01, 0.03162278 and 0.1, i.e. numpy.geomspace(0.001, 0.1, 5) — gives a physical-to-logical error-rate curve, which is collapsed into one number by measuring the area between the curve and 1 and dividing by the width of the range. If the code turns out to have too few logical qubits, the decoding is skipped entirely and a penalty based on the shortfall is returned instead.

That scalar is the reward for Proximal Policy Optimisation. The actor and critic are separate networks with the same trunk design: one branch looks at the raw matrices, one branch tokenises the polynomials and processes them with a small transformer using sinusoidal position features that respect the cyclic structure (x^l = y^m = I), and a third branch carries three logical-qubit summary numbers. Before RL training, the transformer branch is pretrained in a supervised way to predict decoder performance from code parameters, giving the policy a "warm start" that improves sampling efficiency; it can be frozen for the first (N) collector batches. The PPO implementation is described as an almost drop-in reuse of prior code from the cited reference.

Why This Matters

Decoder latency, not just qubit connectivity, is expected to constrain practical fault-tolerant quantum computing: a decoder that misses its time budget can kill the application regardless of how good the code is. If codes can be tuned to the decoder that will actually run on the hardware, rather than to an idealised maximum-likelihood benchmark, then achievable logical error rates may improve without changing the decoder at all. The paper's longer-term ambition is a stack-integrated code–decoder pair engineered around detailed knowledge of a specific quantum device under test, its error modalities and failure modes.

Real-world applications:

• Fault-tolerant quantum processors: selecting stabilizer sets that match the connectivity and decoding latency of a specific superconducting or trapped-ion architecture.
• Decoder hardware/firmware co-design: producing codes whose Tanner graphs avoid short cycles that degrade belief propagation, reducing the iterations (and therefore time) a real-time decoder needs.
• Quantum memory and storage: improving logical error suppression for encoded memory where the decoder is fixed by the control stack.
• Classical LDPC engineering by analogy: the same "design the code around the decoder" recipe applies to classical communication and storage systems that use iterative decoders with latency constraints.

Industry relevance: the work is a collaboration between Imperial College London and Microsoft Research, and it targets exactly the engineering trade-off that hardware vendors face when they must commit to a decoder implementation before finalising a code family. The released dataset and evaluation tooling also lower the barrier for groups that want to benchmark decoder-aware code search without building the quantum simulation stack themselves.

Future Directions

• From a fixed decoder to a decoder family. The current formulation fixes one decoder architecture; the paper's motivating question — how to use deep knowledge of a specific device's error modalities to engineer a better code–decoder pair — is explicitly stated as still far from answered.
• Extending beyond the narrow BB parameterisation. The authors note that narrower and wider definitions of BB codes exist in the literature, and that the number of logical qubits can sometimes be designed directly by the choice of polynomials rather than learned; they chose to learn it because other constructions may not permit design.
• Improving cross-size transfer. Only the transformer encoder branch is size-agnostic; the MLP branch's (2(lm)^2) input prevents reuse across ((l,m)), so a fully size-transferable trunk is an open engineering question.
• Better statistical evaluation within the training loop. The dataset uses a fixed number of samples (50 per point) rather than running until a fixed number of logical errors is observed, and the paper flags that low-error-rate regimes give a limited view; more sample-efficient or adaptive evaluation during training is a natural next step.

Target Audience

Quantum error correction researchers interested in decoder-aware code construction; machine learning researchers working on scientific design problems where the objective is a noisy, expensive simulation; quantum hardware and control engineers who must commit to a decoder before a code family is fixed; and graduate students in either field who want a worked example of casting a code-design problem as a Markov Decision Process with a physically grounded reward. Readers mainly interested in headline logical-error-rate numbers should note that the quantitative evaluation section is not represented in the material summarised here.

Authors’ abstract

We present a reinforcement learning (RL) approach to the co-design of stabilizer sets of Quantum Error Correcting Codes (QECCs) and decoders. We show how to produce a generative model that produces Bivariate Bicycle (BB) codes based on the choice of decoder. Specifically, we fix a decoder architecture and use Proximal Policy Optimisation (PPO) to train an agent over BB codes to maximise decoder performance under a depolarising channel noise model.