Evaluating Verified Autonomy in Quantum Engineering

AI summary

Overview

Research area: Quantum engineering metrology and AI-agent evaluation — specifically, benchmarking large-language-model agents on realistic quantum-hardware workflows.

Technical level: Advanced. The paper assumes familiarity with surface codes, lattice surgery, decoder calibration, surrogate modeling, and agentic LLM harnesses, though its central argument is stated in accessible terms.

Scope (one sentence): The paper introduces Quantum-Harbor, a sandboxed virtual quantum laboratory, and QIQCBench, a 49-task expert-authored benchmark, to measure whether frontier AI agents can complete quantum engineering tasks and justify their answers with evidence generated during execution.

What This Paper Is About

Quantum platforms are becoming too large and too complex for humans to characterize and tune by hand, so researchers have begun pointing scientific AI agents at quantum experiments. What has been missing is a way to tell whether an agent genuinely did the work or merely produced a plausible-looking number. The paper builds a controlled laboratory where an agent's every experiment is logged and every claim is re-checked — recomputed from raw data, or retested on fresh hidden data — and uses it to score 17 frontier agentic systems on 49 quantum engineering tasks.

Key Contributions

1. Quantum-Harbor, a virtual quantum laboratory. A sandboxed environment built on the Harbor framework that gives an agent a public tool surface (Model Context Protocol) for reading hardware specs, writing lab notebooks, submitting asynchronous experiment jobs, and retrieving raw records such as per-shot bitstrings. Behind it sits an isolated "sidecar" holding hidden device parameters, noise models, and an evidence log the agent cannot touch.

2. Interchangeable hardware backends via qtypes. Abstractions for distinct platform physics — multilevel transmons under pulse control, trapped-ion chains, neutral-atom arrays — that can run as simulators with hidden parameters, as a replay backend on recorded data, or connected live to hardware from IBM, Rigetti, IonQ, and Quantinuum, without changing task definitions or the public surface.

3. Evidence-bound grading. The grader recomputes numerical results from the raw data in the evidence log, and for calibration or design outputs it replays the submitted configuration against fresh draws from the hidden parameters. This blocks the failure mode where a model reports a number it never measured.

4. QIQCBench, a 49-task benchmark on 36 qtypes. Expert-authored tasks across six scientific categories (measurement and characterization, quantum control, error correction and mitigation, compilation and resource optimization, simulation and many-body physics, communication and sensing) and five layers of the quantum computing stack, with 39 pass–fail tasks, 7 racing tasks ranked by Bradley–Terry Elo, and 3 infrastructure test cases excluded from science scores.

Main Findings

• Wide spread in verified performance. Across 17 agentic systems from 8 vendors, pass rates on the pass–fail tasks run from 78% down to 3%, decreasing steadily with no clear separation into groups.

• The benchmark separates systems that look identical elsewhere. Eight of the 17 systems score between 54.0% and 62.2% on AutomationBench-AA, and six sit within 31 Elo points of one another on GDPval-AA v2 — yet they spread across almost the whole QIQCBench range.

• Failure to submit dominates. Of 1,510 failures, 937 were no-submission trials. Seven systems exhaust their budget before submitting in more than 60% of trials, and when timed out they are usually still running experiments rather than crashed or idle. Of the remaining failures, 54% were science failures and the rest contract (protocol) failures.

• Intermittent success. Only 34% of the 249 system–task pairs that pass at least once pass all three attempts.

• Most tasks are solvable, a few are not. 37 of 39 pass–fail tasks were solved by at least one system. The two exceptions — surrogate modeling on 60 qubits and a net-zero tunable-coupler CZ gate design — had their intended solution paths confirmed by expert reviewers. Six tasks were solved only by the three highest-ranked systems.

• Simulation and many-body physics is the hardest category, with a 15% pass rate for the pool; the top three systems pass 64% of its trials while the other fourteen pass only 4%, the largest such gap of any category.

• Difficulty is not explained by stack depth. Pass rate is uncorrelated with the number of stack layers a task spans (Spearman ρ = 0.14, p = 0.40), and both unsolved tasks sit in the bottom two layers.

• Racing tasks corroborate the ranking. Of 357 race trials, 125 cleared their feasibility gates; the Bradley–Terry fit spreads the field from 1694 down to 582 with the anchor at 1000, giving the same top three systems but wider margins.

• Representative task outcomes. In the lattice-surgery decoder task on a simulated 114-qubit transmon grid (three distance-3 patches, a routing region, a distance-5 memory patch), the best run reached a memory error of 6.65×10⁻⁴ per cycle and a mean logical-CNOT error of 0.131 on fresh syndrome records, against criteria of below 10⁻³ per cycle and below 0.175 mean (0.185 per circuit) across eight logical-CNOT test circuits. Many runs on the surrogate task (60-qubit circuit, 18 non-Clifford rotation gates, six independent parameters, 40 held-out inputs, 1,770 pairwise correlations per input) progressed far but failed to complete end-to-end.

Methodology in Plain English

Every task is a miniature laboratory project rather than a quiz question. The agent receives only a high-level objective and a finite action budget (a 3-hour per-task time limit, with internet access allowed), then must decide which experiments to run, interpret raw instrument-style output, and submit a final artifact — a physical quantity, a calibration, or a design.

Each run launches two isolated containers. The agent container holds the language model under a harness (Claude Code for the two Anthropic systems, Codex CLI for the three OpenAI systems, OpenCode for the other twelve) plus workfiles; the sidecar container holds the hidden physics. Anything the agent does passes through the public tool surface, so the sidecar can log every request and outcome.

Grading is tied to that log. Numerical answers are recomputed from the raw evidence; calibrations and designs are replayed on fresh hidden data, so a lucky or fabricated answer does not survive. Tasks are built through a fixed pipeline — a domain expert writes a design brief with intended solution and pitfalls, a maintainer implements the public materials, hidden configuration, and grader as separate artifacts, a second expert reviews, and coding agents audit the action records. Contributors deliberately plant "honest" traps that occur in real labs, such as a stale notebook recommending a decoder calibration that fails once code patches are merged, with the notebook plainly labeled and budgets stated explicitly.

For racing tasks, submissions are compared pairwise against every other system's trials; a trial clearing the feasibility gate always outranks one that does not, and gate-passing trials are ordered by the recomputed metric. A Bradley–Terry model with a ridge penalty (λ = 0.5) converts wins and losses into Elo ratings scaled so each task averages 1000. Pass-rate error bars are Wilson intervals with z = 1 (roughly 68% confidence).

Why This Matters

Impact on research. The paper reframes autonomy as something verifiable rather than merely demonstrated: an agent must not only reach the right answer but support it with data it actually produced. That distinction exposes failure modes — stale priors, budget mismanagement, discarding in-hand results at submission — that final-answer benchmarks miss entirely.

Real-world applications:

• Automated calibration and decoder tuning for error-corrected processors, where correlated noise at patch boundaries is invisible in prior measurements.
• Compilation and resource optimization that maps target computations onto trapped-ion, neutral-atom, and superconducting architectures.
• Device characterization such as estimating the relaxation time of a drifted transmon or reconstructing correlated gate errors.
• Quantum communication and sensing workflows, including entanglement purification across repeater arrays and GHZ-assisted frequency estimation under non-Markovian noise.

Industry relevance. Quantum hardware vendors and cloud providers face exactly the scaling problem described: growing numbers of components, parameters, and workflows that make manual diagnosis impractical. A benchmark that predicts which agent systems can be trusted to operate instruments unsupervised — and that is deliberately separated from systems' scores on general agentic benchmarks — is directly useful for deciding what to delegate, and for judging whether an agent's reported result should be believed.

Future Directions

• Transfer beyond simulation. All reported tasks ran against the simulation backend, and the authors state that future work should determine how these findings transfer across experimental conditions and physical devices; the replay and live-hardware backends exist but were not the basis of the reported results.

• Disentangling model, harness, and scientific judgment. Task conditions, operating instructions, and resource constraints could be varied systematically to separate the effects of model capability from harness design and decision-making.

• Improving experimental judgment. Choosing informative measurements, revising stale assumptions, and knowing when to stop are the recurring failure points, and the authors argue improvements to agent harnesses — not just scientific knowledge — could substantially increase reliability.

• Reducing measurement cost. An open question is whether increasingly capable agents can lower the measurement cost and human effort that quantum engineering currently demands.

Target Audience

Researchers in quantum computing and quantum control interested in automation and AI-assisted experiment; AI-agent developers and benchmark designers who need evaluation settings where answers can be independently verified; quantum hardware and cloud-platform teams considering delegating calibration or characterization to agents; and program managers assessing the readiness of autonomous systems for laboratory work. Readers without a quantum background can follow the framing and results but will need the cited error-correction concepts to appreciate the task design.

Authors’ abstract

Reliable quantum engineering is essential for turning quantum phenomena into practical technologies. As quantum platforms grow in scale and complexity, their characterization and operation require increasing human effort and coordination. Scientific artificial intelligence agents, which can plan experiments, operate instruments, and analyze observations, offer a promising route towards autonomous quantum engineering. Yet whether current agents can perform reliably in this setting has not been systematically established. To fill this gap, we developed Quantum-Harbor, a virtual laboratory that provides a controlled execution environment for agents to interact with quantum systems. This design enables direct verification of both the actions taken and the conclusions drawn. Building on this framework, we introduce QIQCBench, a benchmark of $49$ expert-authored tasks spanning multiple layers including calibration and control, error correction and compilation, sensing and networking. Across $17$ frontier agentic systems, QIQCBench reveals wide variation in verified performance. These results expose a substantial gap between demonstrating capability and achieving reliable operation, and establish Quantum-Harbor as a foundation for measuring progress towards verified autonomy in quantum engineering.