Skip to content
Quantum.world

Research

QDiff: Differential Testing of Quantum Software Stacks

Differential testing across quantum software stacks exposes compiler and simulator disagreements that probabilistic outputs and undocumented hardware assumptions otherwise conceal.

QDiff: Differential Testing of Quantum Software Stacks
arXiv
2611.08220
Published
2026-11-05
Authors
Jiyuan Wang, Qian Zhang, Guoqing Harry Xu, Miryung Kim

AI summary

Overview

Research area: Quantum software engineering — specifically software testing and debugging of quantum software stacks (QSSes): the languages, optimizing compilers, simulators, and hardware controllers used to program quantum computers.

Technical level: Advanced. The paper assumes familiarity with quantum gates, unitary matrices, compiler testing, and statistical distribution comparison. The prose is accessible, but the techniques are specialized.

Scope (one sentence): The paper presents QDiff, a differential-testing framework that generates logically equivalent quantum circuit variants, filters out circuits whose results would be dominated by hardware noise, and statistically compares measurement distributions across Qiskit, Cirq, and Pyquil to expose software bugs that would otherwise be mistaken for quantum indeterminacy.

What This Paper Is About

Quantum programs return probabilistic measurement results, so when two logically equivalent programs behave differently, it is genuinely hard to tell whether the cause is a compiler bug, a simulator bug, a hardware defect, or just the inherent randomness of quantum measurement. QDiff attacks this ambiguity directly: it takes a seed quantum program, rewrites it into many semantically equivalent but syntactically different circuits, and then executes them on simulators and on IBM hardware, comparing the resulting measurement distributions. Divergence that survives noise-aware filtering is reported as a candidate instability, together with a pair of "witness programs" that should have behaved identically but did not.

Key Contributions

  1. The first differential testing framework for quantum software stacks. QDiff is integrated with three widely used stacks from three vendors — IBM's Qiskit, Google's Cirq, and Rigetti's Pyquil — and runs on both simulators and real hardware.

  2. Automatic generation of logically equivalent program variants using seven semantics-preserving gate transformation rules (G1–G7), built on the observation that a gate sequence is equivalent to another when their unitary matrix products match. This is paired with four semantics-modifying mutation operators (M1–M4) that diversify the seed pool between rounds.

  3. A noise-aware filtering mechanism that avoids worthless executions. Using static circuit characteristics — circuit depth (moments), the number of 2-qubit gates, per-gate error rates, and T1 relaxation time — QDiff decides whether a circuit is "worth invoking" on hardware or a noisy simulator, cutting backend invocations by 66%.

  4. An extensible equivalence-checking mechanism based on statistical distribution comparison. QDiff estimates how many measurements are needed for a reliable comparison given threshold t and confidence p, and supports pluggable comparators, with Kolmogorov–Smirnov (K-S) test and cross entropy implemented.

Main Findings

  • Bug distribution across stack layers: Of 76 recent GitHub issues collected from Pyquil, Cirq, and Qiskit, 11 were excluded as installation/tooling issues, leaving 65. These break down as 53.9% compiler (optimizations and settings), 19.7% backend (simulators and hardware), and 11.8% API and quantum gate implementation.

  • Variant generation volume: Starting from six seed programs, QDiff produced 730 variant algorithms through semantics-modifying mutations and a total of 14,799 program variants through semantics-preserving source transformations. This generation process took about 14 hours.

  • Transformation effectiveness: QDiff's source transformation produced semantically equivalent yet not-identical circuits in 34% of trials.

  • Filtering speedup: The filtering mechanism sped up differential testing by 66%, obviating the need to invoke quantum backends in those cases.

  • Bugs found: QDiff identified 6 sources of instabilities — 4 software crash bugs in Pyquil and Cirq simulation, and 2 potential root causes that may explain 25 out of 29 cases of divergence beyond expected noise on IBM hardware.

  • Measurement budgets differ by comparator: For the example circuit in Figure 3, the K-S test requires 2,828 measurement samples with t = 0.1, p = 2/3, and platform constant A = 12 for Qiskit, whereas cross entropy requires 420 measurements with A = 7. One seed program (P2, Deutsch-Jozsa) is listed with 5,293 measurements and approximately 7,998 measurement trials under t = 0.1.

  • The cost problem is real: A noisy simulator takes more than 4 minutes to perform 3,000 measurements on a simple four-qubit circuit, and D-Wave's quantum computer was reported at $15 million in 2017; IBM's public access is limited to 5 qubits, with higher qubit counts restricted.

  • Concrete confusion in practice: A Cirq VQE tutorial user believed a wrong-order mix/cost layer output was due to quantum indeterminacy until a Cirq developer labeled it a compiler bug; a StackOverflow search for "quantum error" returns over 500 posts.

  • Not reported in the available text: The provided content is truncated mid-way through RQ1's discussion of the Deutsch-Jozsa seed, so fine-grained RQ2 and RQ3 breakdowns (per-bug reproduction details, false-positive rates, or end-to-end detection precision) are not available here.

Methodology in Plain English

Step 1 — Generate equivalent variants. Every quantum gate is a unitary matrix, and a whole circuit is just the product of its gates' matrices. So if two different gate sequences multiply out to the same matrix, they must produce identical results. QDiff uses seven rewrite rules to expand complex gates into sequences of simpler ones — for example, a SWAP becomes three CNOTs, a Z becomes two S gates, a CZ becomes H-CNOT-H, and a Toffoli (CCNOT) becomes 6 CNOTs plus 9 one-qubit gates. It deliberately does the opposite of an optimizer: instead of shrinking circuits, it diversifies them while preserving semantics.

Step 2 — Diversify the seeds. Pure equivalence-preserving rewriting tends to stay within the same programming constructs, so QDiff also mutates programs without preserving semantics: insert or delete a gate, change a gate type, swap two gates, or retarget a gate to a different qubit. After each round it computes the average distribution across the equivalent variants as a reference, picks the variant furthest from that reference (largest K-S distance), and mutates it — betting that the most deviating program is most likely to expose unseen behavior.

Step 3 — Explore compiler settings and backends. QDiff sweeps compiler configurations (2 configuration types for Qiskit, 3 for Cirq, 2 for Pyquil — e.g., Qiskit swap strategies and optimization levels 0–3, Cirq's DropEmptyMoment() and MergeInteractions(), Pyquil's PRAGMA rewiring and block options) and runs each program on statevector simulators, noiseless simulators, noisy simulators, and free IBM hardware (ibmq_santiago, ibmq_yorktown, ibmq_16_melbourne, ibmq_belem, ibmq_quito).

Step 4 — Filter out circuits whose results would be noise anyway. QDiff first discards physically identical circuits by moment-by-moment comparison. Then it applies a "worthiness" test: a circuit is worth running only if its depth n_m is below a threshold t_m and its change in 2-qubit gate count d_2qubit is below a threshold δ_2qubit. The depth threshold comes from dividing T1 relaxation time by average gate execution time — for ibm_16_melbourne, the paper's worked example computes 53570/928 ≈ 57 moments. The 2-qubit threshold comes from the tolerated added error rate (default 0.1) and the device's CNOT error rate; with a 1.07% CNOT error rate, δ_2qubit is 11. This works because 2-qubit gate errors (order 10⁻²) dominate 1-qubit gate errors (order 10⁻³).

Step 5 — Decide how many shots are enough, then compare distributions. Because measurement is probabilistic, comparing two runs requires a statistical test and an adequate sample count. QDiff adapts closeness testing in L1 norm, using n = A · (1/√(1−p)) · m^(1/2) · t^(−2) samples for the K-S test and n = A · (1/√(1−p)) · m^(2/3) · t^(−4/3) for cross entropy, with confidence p set empirically to 2/3 (a common default in quantum volume measurement). The platform constant A is measured empirically per stack by repeatedly running identical programs. Each variant's distribution is compared to the average (reference) distribution; distance above threshold t counts as divergence.

Step 6 — Attribute the divergence. If a program diverges across different backends with frontend options fixed, QDiff blames a backend source. If it diverges on one backend while varying frontend options, it blames a frontend source. Otherwise it reports "other," such as a bug in a high-level gate API.

Evaluation setup: Six seed programs — X gate, Deutsch-Jozsa (4 qubits, 39 moments, 33 two-qubit gates), Bernstein-Vazirani (4, 41, 32), Grover (5, 84, 53), VQE (4, 36, 28), and QAOA (5, 29, 19) — run against Pyquil 2.19.0 with Quilc 1.19.0, Qiskit 0.21.0, and Cirq 0.9.0, using the K-S test with t = 0.1. Shor's algorithm was excluded because IBM's public access supports only up to 16 qubits and large algorithms produce unreliable hardware results.

Why This Matters

Impact on research: QDiff reframes a question that quantum developers face constantly — "is this a bug or is this just quantum randomness?" — as a testable statistical claim. It shows that compiler-testing ideas (equivalence-preserving transformation, mutation-based fuzzing, differential execution) transfer to the quantum setting, but only if you add noise-aware filtering and distribution-level equivalence checking. It also provides evidence that bugs are spread across the stack, not confined to compiler optimization rules, which matters for work on verified quantum compilers: correctness of the optimizer says little about the simulator or controller.

Real-world applications:

  • Quantum compiler and SDK quality assurance: vendoring variant generation and differential execution into CI pipelines at IBM, Google, and Rigetti to catch regressions in optimization passes, swap insertion strategies, and pragma handling.
  • Cloud quantum cost control: because hardware time is scarce and expensive (5-qubit public limits, multi-million-dollar machines, minutes-long noisy simulations), the 66% backend-invocation reduction directly translates into lower test cost and more tests per dollar.
  • Noise-aware experiment selection: the T1- and gate-error-based worthiness test is reusable by any team deciding which circuits are even meaningful to submit to a device with published calibration data.
  • Debugging educational and tutorial material: the Cirq VQE tutorial episode shows how easily users misattribute compiler bugs to indeterminacy; tooling that automatically flags divergence reduces this class of confusion.

Industry relevance: All three stacks evaluated are production vendor frameworks, and the paper reports that bugs were found on current versions and on real IBM hardware — including 4 crash bugs and 2 root causes covering 25 of 29 hardware divergence cases. That is a direct signal for the vendors whose SDKs were tested.

Future Directions

  • Extending the transformation rule set. The current approach relies on seven hand-written rules (G1–G7); a larger or automatically derived set of equivalence rules would broaden the space of reachable circuit variants, and the paper's 34% success rate at producing non-identical equivalents suggests room to improve generation yield.
  • Adding more statistical comparators with matched sample-size estimators. QDiff is explicitly designed to be extensible, but each new comparator (beyond K-S and cross entropy) requires its own measurement-estimation formula, which is an open engineering and statistical question.
  • Broadening beyond free IBM hardware. Evaluation is limited to IBM's free-access tier and up to 5–16 qubits, with rules-based thresholds tied to published IBM error rates; extending to other vendors' hardware, other qubit technologies, and larger circuits requires re-deriving the thresholds and constants.
  • Distinguishing software defects from hardware defects more sharply. QDiff explicitly targets software defects and filters out hardware-noise-dominated runs, but the two root causes it identified on IBM hardware are described as "potential" explanations; a sharper attribution method — and validation against vendor-confirmed fixes — would strengthen the diagnostic claims.

Target Audience

Quantum software engineers and compiler researchers building or maintaining QSSes; testing and fuzzing researchers interested in probabilistic and nondeterministic systems; graduate students entering quantum software engineering who need a concrete picture of where bugs actually live in the stack; and vendor QA teams at organizations shipping quantum SDKs, for whom the specific findings about Qiskit, Cirq, and Pyquil are directly actionable. Readers without background in unitary matrices, circuit moments, and distribution tests will find the methodology sections demanding.

Authors’ abstract

Over the past few years, several quantum software stacks (QSS) have been developed in response to rapid hardware advances in quantum computing. A QSS includes a quantum programming language, an optimizing compiler that translates a quantum algorithm written in a high-level language into quantum gate instructions, a quantum simulator that emulates these instructions on a classical device, and a software controller that sends analog signals to a very expensive quantum hardware…

The full abstract on arXiv
Read the original paper