ABSTRACTS: Amsterdam Benchmark Suite for the Time and Resource Analysis of Clifford+T Simulators

AI summary

Overview

• Research area: Classical simulation of quantum circuits (quantum physics / quantum computing software), specifically benchmarking simulators of non-Clifford, Clifford+T circuits.
• Technical level: Intermediate. The paper is a "tool paper" describing infrastructure, and it includes a background section defining strong vs. weak simulation, but the core contribution is process and architecture rather than new mathematics.
• Scope (one sentence): The paper introduces ABSTRACTS (Amsterdam Benchmark Suite for the Time and Resource Analysis of Clifford+T Simulators), a standardised, versioned benchmark suite, cloud execution pipeline, and public dashboard for fairly comparing classical simulators of non-Clifford quantum circuits.

What This Paper Is About

Recent years have produced many new methods for simulating non-Clifford quantum circuits on classical hardware, but each paper typically defines its own circuits, methodology, hardware and baselines, which makes fair comparisons difficult. ABSTRACTS addresses this by providing a canonical dataset of circuits organised by class, a fixed virtual hardware and execution protocol, tracked raw results, and an interactive dashboard for analysis. It is explicitly presented as a "tool paper" rather than a "benchmark paper": the included results mainly demonstrate the framework, with a comprehensive benchmarking study left for future work.

Key Contributions

1. A standardised benchmark suite and execution pipeline. ABSTRACTS bundles circuits, simulators, a cloud compute instance, results files, and a dashboard, with an automated pipeline triggered when a new simulator or circuit class is accepted into the repository.
2. A versioned consistency and reproducibility protocol. The paper formalises four design principles (Consistency, Reproducibility, Extensibility, Transparency) and states that an ABSTRACTS version uniquely identifies a consistent benchmarking configuration, with any change to circuits, hardware, execution environment or protocol triggering a new version and a rerun of all benchmarks.
3. A defined v0.1 circuit suite covering five circuit classes. The initial release includes Random Clifford+T, Clifford+Toffoli/CCZ, Exponentiated Paulis, Modified Hidden Shift, and IQP circuits, chosen because each has been used in prior benchmarking work and together they avoid bias toward any one technique.
4. A public dashboard and results record. Aggregated results are plotted interactively (selectable x/y metrics, simulators and circuit classes, linear or logarithmic axes, tag-based filtering), with raw per-circuit data recorded as JSON for each simulator.

Main Findings

• No global winner among simulators: The paper argues it is not meaningful to describe one simulator as "better" in a simplistic global sense, since simulators optimise against different metrics (T-count, qubit count, treewidth, memory, sampling overhead).
• Opposite exponential scalings observed: Two example plots compare the PyZX "Magic Cat" stabiliser decomposition strategy [20] against the Qiskit Aer Statevector simulator [19] on random Clifford+T circuits, at a fixed qubit count of 20 (left plot) and a fixed reduced T-count of 30 (right plot); the two methods scale exponentially against different metrics, so neither dominates.
• Dashboard example shows method-dependent behaviour: A dashboard screenshot plots CPU time against PyZX-reduced T-count for three simulators — PyZX BSS [22], PyZX Magic Cat [20] and Quimb [17] — on random Clifford+T circuits. The two PyZX methods scale exponentially with T-count, while Quimb (tensor contraction) is essentially agnostic of T-count and instead scales with "contraction width"; between the two PyZX methods, Magic Cat achieves markedly better scaling.
• Stabiliser decomposition α value cited: For a Clifford+T circuit with k T gates, the number of terms is written as 2^(αk), and the best known α for arbitrary circuits is α ≈ 0.396 [20, 33].
• Benchmarking coverage is deliberately partial: Of the 19 notable strong-simulation methods and software listed in the roadmap table, 4 are marked as benchmarked as of v0.1 (Gray 2018; Kissinger & van de Wetering 2021; Kissinger et al. 2022; Javadi-Abhari et al. 2024). No numerical timing or memory results are reported in this paper.
• Concrete v0.1 configuration: Benchmarks run on an AWS t3.small EC2 instance with 2 vCPUs and 2GiB of RAM, running Ubuntu 26.04 LTS (Resolute Raccoon), with a 5 minute timeout per circuit and a 3 hour global timeout per simulator.
• Tracked raw results: Successfully simulated circuits record computed amplitude, wall time, CPU time and peak memory usage; failures are recorded as exceeded circuit timeout, exceeded global benchmark timeout, or exceeded memory limit, and approximate simulators can be filtered by a specified margin of error rather than being marked failed.

Methodology in Plain English

The authors built a public repository with a fixed structure: one place for circuit classes (each with metadata), one for simulator wrappers (each defining its own environment and dependency versions), and one for results. When a simulator is submitted and passes basic review, an automated pipeline runs it against every circuit in the suite on the same cloud instance, with the same timeouts, and records per-circuit outcomes. A script then aggregates the raw JSON results into a simplified file used to generate the interactive dashboard plots. Because the setup is versioned, any change to circuits, hardware or protocol creates a new version and forces a full rerun, so results quoted as "benchmarked on ABSTRACTS v0.1" correspond to a documented, fixed configuration. The simulators themselves are not modified; wrapper code adapts each one to the common interface, and simulators with inherent randomness are encouraged to use a fixed seed.

Why This Matters

Impact on research: The lack of standardised benchmarking has made fair comparison between simulation techniques difficult and results hard to reproduce. A shared, versioned suite with fixed hardware removes the need for each paper to contextualise its own bespoke setup, and lets a citation of a single version number stand in for a long paragraph of hardware and methodology description. The project is released under the MIT licence and is free to use, submit to, and benchmark on.

Real-world applications:

• Verifying quantum algorithms and quantum computers by comparing their output against trusted classical simulation.
• Quantifying and tracking the boundary of quantum advantage, since better classical simulators raise the bar quantum hardware must meet.
• Supporting fault-tolerant quantum computing research, where near-Clifford circuits (many Clifford gates, few non-Clifford gates) are common and stabiliser-decomposition simulators are practical.
• Serving as a common testbed for developers of simulation software to profile where their method scales well or poorly.

Industry relevance: Simulation vendors and compiler/toolchain teams gain a neutral, third-party record of how their software performs against competitors across circuit classes, with simulator metadata (language, GPU or multithreading use) available for filtering so that, for example, only Python-based or non-GPU simulators are compared.

Future Directions

• Expand simulator coverage: The roadmap lists 19 notable strong-simulation methods and software as targets, of which 15 are not yet benchmarked in v0.1; some require only a wrapper, others require obtaining source code or reimplementation from theory.
• Broader circuit classes: Planned additions include circuits beyond Clifford+T (such as arbitrary phases), plus Grover/Oracle circuits, Quantum Fourier Transform and Phase Estimation circuits, and variational/QAOA circuits, potentially importing suites from T-par, RevLib, QASMBench and MQT Bench.
• New simulation tasks and conditions: Benchmarking weak simulators, computing marginal amplitudes, simulating circuits under different noise models, and improved hardware-resource profiling including GPU and multicore CPU utilisation.
• Dashboard and integrity improvements: More visualisation options such as heatmaps and 3D plots, and automated anti-cheating measures such as hidden random circuits.

Target Audience

Researchers and developers who build or use classical simulators for non-Clifford quantum circuits, and authors who want citable, reproducible performance comparisons in their own papers. It is also useful for quantum software engineers choosing a simulator for a particular circuit class, and for reviewers or readers who need to interpret simulation benchmark claims. Readers seeking new simulation algorithms will not find them here; readers seeking a benchmarking standard and infrastructure will.

Note: the paper's acknowledgements state that large language models were used to help create the YAML for GitHub Actions workflows, shell scripts for the benchmark runner and simulator wrappers, the Python results-aggregation script, and the HTML dashboard code, with all such code reviewed and verified; the manuscript itself is stated to be entirely the authors' work. The authors also acknowledge the upcoming Quantum Gauntlet project by Guillermo Alberto Perez and Matthias Lanzinger, developed independently on a comparable timeline.

Authors’ abstract

Recent years have seen a rapid growth in literature presenting new methods for simulating non-Clifford quantum circuits with classical hardware. These methods span a range of approaches, including stabiliser decomposition and tensor contraction techniques, varying in efficiency depending on circuit class, depth, non-Clifford gate count, and other metrics. A notable limitation of this literature is the lack of a standardised approach to benchmarking, with each new paper outlining its own specification, simulating its own set of circuits on the authors' own hardware. This paper seeks to address this issue by presenting a standardised and canonical benchmark suite and infrastructure for quantifying the efficiency of non-Clifford classical simulators, with a consistent dataset of circuits and providing consistent (virtual) hardware, thereby enabling a fair comparison of results.