Retromorphic Testing of Quantum Compiler Passes

AI summary

Overview

Research area: Quantum software engineering — specifically testing and validation of quantum compiler passes (the optimization/transformation stages of quantum compilers), with a secondary empirical study of how the four major quantum SDKs currently unit-test those passes.

Technical level: Intermediate. Readers should be comfortable with basic quantum concepts (gates, adjoints, unitaries, measurement, expectation values) and with standard software-testing vocabulary (unit tests, test oracles, black-box testing).

Scope: A systematic audit of compiler-pass unit tests in PennyLane, Qiskit, Cirq, and pytket, plus a new black-box testing framework, RetroQ, that validates passes without ever computing the expected output circuit by hand.

What This Paper Is About

Quantum compilers rewrite circuits to reduce gate count and depth, and they are supposed to preserve the circuit's semantics while doing so. Verifying that a rewrite is correct is hard because simulating large entangled circuits to obtain an expected answer is computationally intractable, and manual oracle construction does not scale. The paper first measures how the field currently tests these passes (mostly small, shallow circuits checked with content and metric assertions), then proposes and builds a testing method that sidesteps the oracle problem entirely by using the pass, the inverse of the circuit, and a Hadamard-test-style control construction to check that semantics are preserved and that the intended structural change actually happened.

Key Contributions

1. An empirical audit of compiler-pass unit tests across four frameworks. The authors built a static-analysis pipeline on Python's ast module to extract and categorize every assertion in the compiler-pass test suites of PennyLane (113 test files), Qiskit (31), Cirq (55), and pytket (1), grouping them into nine categories such as Program Metrics, Program Contents, Matrix Equivalence, Output Equivalence, Exception Handling, and Gradient Equivalence. They also instrumented the test suites to record the depth and qubit count of every circuit fed into a compiler pass: 1,518 circuits for PennyLane, 7,559 for Qiskit, 1,682 for Cirq, and 119 for pytket.

2. A retromorphic testing methodology for compiler passes. RetroQ models pass correctness as two predicates — a semantic preservation predicate (the transformed circuit is equivalent to the original up to numerical tolerance and global phase) and a structural predicate over a circuit metric such as gate count — and checks them via a two-stage process: an adjoint identity check (Stage 1) and a controlled adjoint identity check derived from the Hadamard test (Stage 2).

3. An implemented, reusable framework. RetroQ is released on GitHub with support for PennyLane and Qiskit. It takes a target quantum function, the pass under test, and a user-specified Expectation (currently gate count increase/decrease/unchanged), and produces failure reports that attribute each violation to a specific circuit variant.

4. A defect study over five real compiler passes. The evaluation covers two Qiskit-2.4.1 passes (CommutativeInverseCancellation, RemoveIdentityEquivalent) and three PennyLane-0.45.0 passes (cancel_inverses, single_qubit_fusion, pattern_matching_optimization), identifying or replicating six bugs, including previously undetected ones. From these, the paper derives five categories of compiler-transformation errors that existing approaches miss.

Main Findings

• Validation is dominated by two assertion types. Program Contents assertions account for 42.85% of PennyLane, 49.56% of Qiskit, 62.24% of Cirq, and 39.93% of pytket assertions; Program Metrics account for 16.08%, 14.39%, 3.70%, and 50.88% respectively. pytket stands out for its metric-heavy style, Cirq for a comparatively high 21.35% of Matrix Equivalence assertions, and PennyLane is the only framework testing gradient equivalence (6.85%).

• Test circuits are small and shallow. Across all four frameworks the majority of circuits fall between depth 1 and 10: Cirq, PennyLane, and Qiskit have roughly 84–94% of circuits at depth 1–5 and roughly 95% within 1–10, while pytket has 66.10% at depth 1–5 and 81.36% within 1–10. Qubit counts are mostly below 50, with Qiskit showing the widest spread and extreme outliers near 100, 200, and nearly 900 qubits from StarPrerouting and LightCone stress tests, fake backends mirroring the 127-qubit ibm_sherbrooke architecture, and circuits generated from 20×20 hexagonal lattice graphs.

• Incorrect commutation logic in Qiskit (reproduced). CommutativeInverseCancellation pushes a Swap gate through a Controlled-RZ gate and then cancels the two Swaps, even though the operations do not commute, so the gate count changes when it should not. This reproduces Qiskit Issue #16164.

• Symbolic zero-angle gates are not removed in Qiskit (reproduced). RemoveIdentityEquivalent fails to eliminate a ry(d * 0) gate where d is a symbolic Parameter; the pass is expected to reduce gate count but leaves it unchanged, and the same limitation applies to RZ and RX. This is motivated by Qiskit Issue #6468.

• Runtime crash on general unitaries in PennyLane (newly found). cancel_inverses crashes when a QubitUnitary is followed by its adjoint, raising ValueError: The truth value of an array with more than one element is ambiguous, because the pass evaluates equality between gate parameters. This was confirmed by the PennyLane team and reported on GitHub.

• Self-adjoint gates are not recognized as cancelling (newly found). In the paper's worked example, two consecutive Hadamards are not cancelled because the pass does not recognize that the adjoint of a self-adjoint gate is itself; the same limitation extends to Pauli X, Y, Z, the identity, CNOT, and Toffoli in PennyLane.

• Global phase becomes physically significant under control (newly found). With single_qubit_fusion and exclude_gates=['Hadamard'], applying the pass to the adjoint of a Hadamard-plus-RZ circuit unexpectedly fuses operations into a qp.GlobalPhase. In Stage 1 this is harmless, but in Stage 2 the phase, combined with the control, drives the auxiliary expectation value to −1 instead of +1. This was forwarded to the PennyLane team as a feature request for an optional phase-preserving configuration.

• Pattern matching ignores adjoints of self-adjoint sequences (newly found). pattern_matching_optimization fails to match H†Z†H† against a stored HZH pattern even though those gates are self-adjoint, so no gate-count reduction occurs. The paper notes this can be worked around with the qp.simplify transform.

• Stage 1 and Stage 2 catch different bug classes. Stage 1 caught failures to reduce gate count (symbolic zero-angle removal, incomplete self-adjoint recognition) and runtime crashes; Stage 2 was necessary to expose the global-phase-under-control problem, which is invisible to structural checks alone.

• Overhead is modest. The complexity of RetroQ scales with the target function and the pass applied to it, with overhead no worse than that expected from applying the pass itself.

Methodology in Plain English

The work proceeds in two parts.

First, a measurement study. The authors parse each compiler-pass test file into a Python abstract syntax tree, find every assert statement, and manually label what property each assertion is checking. Separately, they instrument the test suites so that every time a circuit is handed to a compiler pass, its depth and qubit count are logged; running the suites produces the distribution data.

Second, a testing method. The core idea is borrowed from retromorphic testing, which compares a function against its inverse rather than against a hand-computed answer. In Stage 1, RetroQ builds a baseline circuit that initializes the qubits, applies the target circuit F, then applies F's adjoint, then undoes the initialization — so every qubit should come back to |0⟩. It then builds two variants in which the compiler pass is inserted in different places relative to the adjoint: applying the pass to F†, and taking the adjoint of the pass applied to F. If the pass is correct, both variants should still return every qubit to |0⟩ and should satisfy the user's stated structural expectation (for example, gate count decreased).

Stage 2 repeats the exercise inside a Hadamard-test skeleton. An auxiliary qubit is put into superposition, the circuit-under-test is applied under control, and a second Hadamard is applied before measurement; the auxiliary qubit should read |0⟩ with expectation value +1. Three variants place the control, the adjoint, and the pass in different orders. This stage catches errors that only appear when a circuit is nested inside a controlled operation — most importantly, spurious global phases that become local, observable phases under control.

The user supplies only the target function, the pass, and a plain-language expectation such as gate_count='decrease'. RetroQ derives the expected values itself and emits a report naming which variant failed and by how much. The default initialization is the identity, which suits passes sensitive only to gate order and structure; the authors state that other initialization routines are possible in principle.

Why This Matters

Compiler passes are the layer that decides what actually runs on quantum hardware, and a silently wrong optimization corrupts every program that passes through it. The paper shows that the current safety net — unit tests built on small, shallow circuits and structural assertions — is both narrow in coverage and structurally unable to express certain correctness properties. The empirical percentages give SDK maintainers a concrete baseline to measure their own test suites against, and the discovered defects (a crash, a wrong commutation, missed cancellations, a phase bug that only surfaces under control) are the kind that ordinary unit tests miss by construction.

Real-world applications:

• Quantum SDK and compiler QA: regression testing of individual optimization passes in PennyLane, Qiskit, and other frameworks before release, without writing a manual oracle per pass.
• Uncomputation and reversible subroutines: adjoints are pervasive in uncomputation and reversible arithmetic, exactly the setting where the paper found the self-adjointness and crash bugs.
• Quantum arithmetic modules: the authors note the method generalizes to verifying arithmetic circuits, since quantum operations are unitary and reversible.
• Quantum error-correction encoding circuits: named explicitly as another class of circuit amenable to the same inverse-based checking.
• Differentiable quantum programming: PennyLane's gradient-equivalence assertions show that transformed programs must also preserve derivatives; pass bugs there propagate into training.

Industry relevance: the failures found are in shipping releases (Qiskit-2.4.1, PennyLane-0.45.0), and some were confirmed by the PennyLane team and filed upstream. As vendors compete on transpiler quality, an automated, oracle-free correctness check is directly usable in CI pipelines.

Future Directions

• Broaden the structural predicates. RetroQ currently supports only gate-count expectations (increase, decrease, unchanged). The paper's formalism already allows any metric m with comparisons <, ≤, =, ≥, >, so depth, qubit count, and pass-specific metrics are natural extensions.
• Vary the initialization. The current implementation uses the identity as the default initialization operator. Alternative state preparations could expose passes that are sensitive to input state rather than just operation order.
• Extend beyond PennyLane and Qiskit. The methodology is framework-agnostic and the paper argues it applies to any framework's transformations, but Cirq and pytket are analyzed only in the unit-test study, not exercised with RetroQ.
• Close the gap with formal verification. The paper contrasts RetroQ with Coq-based VOQC (whose manual proofs can exceed the code size by more than an order of magnitude) and with the SMT-based Giallar (limited by fixed loop templates and incomplete rewrite rules). Combining lightweight automated testing with targeted formal proofs is an open question the comparison implicitly raises.

Target Audience

Quantum software engineers and compiler developers who maintain or consume transpiler passes; researchers in quantum software engineering and testing who work on the test-oracle problem; framework maintainers at PennyLane, Qiskit, Cirq, and pytket who want a concrete benchmark for their test suites; and graduate students or advanced undergraduates with some quantum computing and software-testing background who want a readable entry point into compiler correctness.

Authors’ abstract

Quantum compilers play a critical role in transforming high-level quantum programs into optimized, hardware-compatible circuits. However, verifying the correctness of compiler passes remains challenging, as determining the expected output of large, deeply entangled quantum circuits is computationally intractable. This challenge is further amplified when compiler passes modify already complex circuit structures, making manual validation of transformed circuits impractical. In this work, we perform a systematic analysis of unit tests for quantum compiler passes in four quantum programming frameworks (PennyLane, Qiskit, Cirq, and pytket). Our findings indicate validation is dominated by program-content and program-metric assertions, and test circuits are generally small and shallow. Motivated by these observations, we introduce a testing methodology for automated validation of quantum compiler passes based on retromorphic testing and principles from the Hadamard test. This methodology analyzes a compiler pass, test circuit, and expected pass behavior to verify semantic preservation and intended structural modifications. We implement our methods in a framework, RetroQ, and apply it to compiler passes in PennyLane and Qiskit. Experimental evaluation reproduced several existing bugs as well as uncovered previously undetected defects, such as flawed symbolic parameter handling, incorrect commutation logic, failure to recognize self-adjointness of gates, and runtime crashes. These findings highlight the need for compiler-pass-specific testing methodologies to improve the reliability of the evolving quantum software stack.