Research
Multi-head Quantum Neural Networks: Replacing Transformer Blocks with NISQ Modules inside a 30-Billion-Parameter LLM
Large language models carry hidden states of thousands of dimensions, while quantum machine-learning models have been trained almost entirely on low-dimensional inputs.

- arXiv
- 2611.15250
- Published
- 2026-11-11
- Authors
- Maforikan Amoussou, Borja Aizpurua, Sukhbinder Singh, Román Orús, Hans-Arno Jacobsen
AI summary
Overview
Research area: Quantum machine learning, specifically hybrid quantum–classical large language models and NISQ-era circuit design.
Technical level: Advanced. The paper assumes familiarity with transformer architectures, mixture-of-experts models, quantum circuit ansätze, Pauli observables, and perplexity-based evaluation.
Scope: The paper demonstrates that a small multi-head quantum neural network can replace a contiguous span of transformer blocks inside a 30-billion-parameter LLM with near-parity on held-out perplexity, and that the trained circuits execute on a 156-qubit IBM Heron processor.
What This Paper Is About
Large language models carry hidden states of thousands of dimensions, while quantum machine-learning models have been trained almost entirely on low-dimensional inputs. The two standard ways of loading a large activation vector into a quantum circuit fail at production scale: amplitude encoding needs circuits of depth proportional to the dimension, and angle encoding needs one qubit per feature. The authors propose partitioning the hidden state into many small, independently processed "heads" and show that such a module can stand in for entire transformer blocks in a production-scale model.
Key Contributions
-
The multi-head QNN architecture. A quantum module whose width scales with the number of heads
krather than with the size of any single qubit register. Each head angle-encodes ad_h = 2·n_q·Lslice of the activation into its ownn_q-qubit register using dense angle encoding with layerwise uploads, and the heads are coupled by constrained classical linear "mixers" built from a fixed Sylvester–Hadamard matrix and trainable diagonals. -
Replacement of transformer blocks in Qwen3-30B-A3B. A single module of 145,281 trainable parameters replaces one block or contiguous spans of up to sixteen blocks, benchmarked against an identity-skip baseline, component ablations, and a parameter-matched classical MLP. At a span of eight blocks the replacement displaces 16.33% of the model's parameters.
-
Execution on quantum hardware. The quantum heads of five fusion checkpoints are run on a 156-qubit IBM Heron processor, for the first prompt token and, at
N = 8, at every decoding step of two 15-token generations, with per-observable deviations from statevector simulation measured. -
An honest accounting of what the evidence does and does not show. Ablations and controls reveal that the benefit does not require trained or entangled circuits, and that a parameter-matched classical MLP remains a few percent ahead in perplexity.
Main Findings
-
Fusion works up to eight blocks. A single module of 145,281 parameters, fixed across span size, replaces the contiguous span from block 10 to block
9+NforN ∈ {1, 2, 4, 8, 16}. It beats identity-skip on all three evaluation sets at all five spans: WikiText-2 perplexity falls from 8.791 to 8.144 atN = 1, and from 13.115 to 9.918 atN = 16. -
Degradation is gradual, not abrupt. WikiText-2 perplexity rises from 8.144 to 8.748 between
N = 1andN = 8, while top-1 agreement with the base model falls from 0.883 to 0.838 over the same interval. -
Crossover at eight blocks. The hybrid model is better than the original at
N = 1, 2 and 4 on both test corpora, reaches the crossover atN = 8(worse on WikiText-2 by 0.9%, unresolved on C4 and webmix-recon), and is resolved worse atN = 16(14.4% worse on WikiText-2). The token-level unmodified baseline is 8.672. -
Compression is set by span size, not module size. Each displaced transformer block holds 6.23 × 10^8 parameters against the module's 1.45 × 10^5. At
N = 8this removes 16.33% of the total 30.5-billion-parameter model (reduction factor 1.195) and 13.57% of the 3.35 billion active parameters; atN = 16, 32.65% and 27.14% respectively. -
The circuit carries the function for a single block. For a single replaced block, the quantum heads (circuits plus learned readout) recover the displaced function, and the classical mixers alone do not. For fused spans, however, the mixers alone come within 1–3% of the full module in perplexity.
-
Circuits matter most at small spans. The circuit-only variant (14,209 parameters, readout deleted) reaches 8.441 on WikiText-2 and 14.024 on C4 at
N = 4, within 2% of the full block, at a local compression ratio of 1.75 × 10^5. It leads the no-circuit variant atN = 1, 2 and 4 on all three sets, but the two are statistically indistinguishable beyondN = 4. -
The learned readout dominates the budget. It accounts for roughly 90% of the module's parameters (131,072 of 145,281). The two constrained mixers contribute 8,192 parameters, against 8.4 × 10^6 for two unconstrained 2048 × 2048 matrices.
-
Trained entanglement is not shown to be necessary. Controls that switch the entangler off (two seeds), or freeze the circuit at its initial or uniformly random angles (one seed), beat the full module on held-out perplexity.
-
The classical MLP remains ahead. A parameter-matched classical MLP of 142,537 parameters leads the full block in perplexity in all but one comparison (C4 at
N = 16, where they differ by 0.002), though downstream accuracy and top-1 agreement do not consistently show this lead. -
Results are reproducible across seeds. Repeating
N = 4andN = 8at a second training seed reproduces test log-likelihood to within 0.0016. -
Target-layer selection. Zeroing each of the 48 decoder layers in turn shows boundary layers dominate (+84.97 ΔPPL at layer 0, +11.55 at layer 47) and are excluded. Among the rest, layer 1 is most necessary (+5.06), followed by tail layers 45, 44, 43, 46 and 42 (+1.61 down to +0.84); layers 10 to 17 are most redundant (+0.03 to +0.14). Repeating the ranking on a disjoint 256-sequence validation tail gives a Spearman correlation of 0.95 over the 46 interior layers.
-
Hardware execution succeeded. Trained heads compiled and ran as expected on the 156-qubit IBM Heron processor, with the rest of the model on GPUs.
Methodology in Plain English
The authors start from an observation about redundancy: trained transformers tolerate the outright removal of blocks with little loss, and small trained modules can stand in for removed layers. They ask whether that stand-in could be a quantum circuit.
The obstacle is encoding. A 2048-dimensional activation vector is too wide for either standard encoding scheme. Their fix is to split the vector into k = 32 disjoint chunks of 64 features each, give each chunk its own 8-qubit, 4-layer circuit, and run all 32 circuits in parallel. Each circuit encodes two features per qubit per layer through rotation gates, applies a layer of variational R_x gates, and entangles with an open chain of 7 nearest-neighbour R_zz gates — chosen so that no SWAP routing is needed on a linear-connectivity device. Each head can learn its own frequency scales through a trainable affine map applied to features before encoding.
Since independent heads cannot communicate, two constrained linear "mixers" bracket the circuit layer. Each mixer is a fixed Hadamard matrix sandwiched between trainable diagonal matrices — cheap to evaluate via the fast Walsh–Hadamard transform and deliberately prevented from being a full dense matrix, so the classical path cannot trivially solve the task alone.
Measurement uses a pool of 64 Pauli observables per head (32 single-qubit terms plus 32 of 72 nearest-neighbour two-qubit candidates) rather than a single expectation value, followed by a trainable linear readout. Training happens in two stages with the base model frozen: first regress the displaced block's output using a combined MSE and cosine-similarity loss, then fine-tune end to end on next-token prediction for 20,000 steps. Everything is trained on a classical statevector simulator in PyTorch. Only afterwards are the trained circuits deployed to real hardware.
Why This Matters
The paper is explicit that it claims no quantum advantage and no new capability for the LLM itself. Its value is as an engineering existence proof: NISQ-scale circuits can take on part of a production-scale model's function and run on hardware that exists today.
-
Research impact: It reframes the question from "can quantum models compete on benchmark tasks" to "can quantum modules be installed inside models people actually deploy," and it supplies a negative-control discipline (entangler-off, frozen-angle, parameter-matched MLP) that future hybrid-LLM claims will have to meet.
-
Model compression: Replacing spans of blocks with a 145,281-parameter module displaces 16.33% of a 30.5-billion-parameter model at near-parity, suggesting a route to structurally compressed models where circuit size is decoupled from model width.
-
Hardware co-design: The linear-connectivity ansatz and native
R_zzfractional gates map logical two-qubit rotations one-to-one onto device gates with zero routing overhead, giving a concrete template for compiling model components onto near-term processors. -
Hybrid deployment: The demonstrated split — circuits on a quantum processor, the rest of the model on GPUs — is a working pattern for hybrid inference infrastructure.
Industry relevance: The work comes from Multiverse Computing with University of Toronto and Ikerbasque affiliations, and speaks directly to organizations asking whether quantum hardware has any near-term role in large-model inference. The answer here is architectural feasibility, not advantage.
Future Directions
-
Establish whether entanglement helps. The entangler-off and frozen-angle controls beat the full module, so the source of any quantum-specific benefit is unresolved. A systematic study across ansätze, seeds and observables is the obvious next step.
-
Close the gap to classical baselines. A parameter-matched MLP stays a few percent ahead in perplexity. Determining whether any circuit configuration can match or exceed it — and at what scale — remains open.
-
Move beyond classical simulability. The circuits used here can be simulated exactly, so the parameter-efficiency argument is currently about parameter counts alone. Whether a useful regime exists at larger circuit sizes is unclear, given that trainability degrades as circuits grow and the structural restrictions that mitigate this may also preserve classical simulability.
-
Add the missing matched controls. The authors note that a fine-tuned base model or a healed truncated model would isolate the training asymmetry (20,000 steps on the replacement versus no further training on the original), and neither is part of this study.
Target Audience
Quantum machine-learning researchers interested in whether NISQ circuits have any role at production model scale; machine-learning engineers working on model compression, pruning and layer replacement; quantum hardware teams designing device-native ansätze and compilation strategies; and reviewers or practitioners who need a careful template for what honest negative controls look like in hybrid quantum–classical LLM claims. Readers without background in transformer internals or variational circuits will find the paper demanding.
Authors’ abstract
We introduce the multi-head quantum neural network (QNN), a quantum module compatible with noisy intermediate-scale quantum (NISQ) devices that can replace transformer blocks inside pre-trained large language models (LLMs) with little loss in accuracy and with far fewer parameters than the blocks it replaces. Encoding a model's $D$ hidden neurons into a qubit register is the obstacle: amplitude encoding scales poorly in circuit depth and angle encoding in qubit count, so…
The full abstract on arXiv