QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

AI summary

Overview

Research area: Evaluation of large language models (LLMs) on quantum computing (QC) tasks — sitting at the intersection of LLM benchmarking and quantum information science.

Technical level: Intermediate. The abstract assumes familiarity with ranking correlations and Item Response Theory, but the core argument (aggregate scores hide task-level differences) is accessible.

Scope (one sentence): The paper introduces QC-Stark, an 11-task, execution-verified benchmark for quantum computing, and uses 2,750 model evaluations to show that overall leaderboard rankings do not reliably predict per-task performance.

What This Paper Is About

Benchmarks usually compress a model's ability into a single overall score and a single rank order. This paper asks whether that compression is trustworthy for a specialized technical domain: quantum computing. The authors build a multi-task benchmark spanning the QC workflow — circuit construction, debugging, compilation, error correction, and simulation — and test whether a model's overall standing actually reflects how it performs on each individual task.

Key Contributions

1. A new multi-task QC benchmark (QC-Stark). Eleven quantum computing tasks covering circuit construction, debugging, compilation, error correction, and simulation, each organized into five difficulty levels.
2. A large-scale evaluation sweep. 2,750 total evaluations, structured as 10 models × 11 tasks × 5 difficulty levels × 5 seeds.
3. Evidence of capability dissociations. A direct comparison of overall versus per-task rankings, showing that the two disagree enough that aggregate rankings are misleading for a subset of tasks.
4. Measurement-quality and robustness checks, plus public release. A 2-parameter Item Response Theory (IRT) model is used to validate the measurement, prompt sensitivity analysis tests ranking stability, all tasks are graded automatically by execution rather than human judgment, and code and data are released on HuggingFace.

Main Findings

• Overall rankings mask per-task variation. The headline result is that a single aggregate score hides substantial differences in how models perform across the eleven tasks.
• Rank correlation fails for 4 of 11 tasks. The Spearman correlation between overall rankings and per-task rankings is statistically insignificant for four of the eleven tasks — meaning that for those tasks, knowing a model's overall rank tells you essentially nothing reliable about its rank on that task.
• Measurement quality is validated by IRT. Fitting a 2-parameter Item Response Theory model supports the claim that the benchmark is measuring something coherent rather than noise.
• Rankings survive prompt changes. Prompt sensitivity analysis confirms that the resulting rankings are robust across prompt conditions, addressing the common objection that LLM benchmark results are artifacts of prompt phrasing.
• Evaluation requires no human grading. Every task is auto-verifiable through execution, so correctness is determined by running the artifact rather than by manual review.

The abstract does not report which specific tasks show the dissociation, which models were evaluated, the magnitude of the correlations, or per-task score distributions; those details are not available in the abstract.

Methodology in Plain English

The authors designed a set of eleven practical quantum computing exercises rather than factual quiz questions — writing circuits, finding bugs, compiling, working with error correction, and running simulations. Each task exists at five increasing difficulty levels, and each combination is run five times (seeds), producing repeated measurements rather than a single lucky or unlucky sample. Ten different LLMs attempt the full grid, generating 2,750 individual evaluations.

Grading is done by executing what the model produces, so a circuit either behaves correctly or it does not — no human judge and no LLM-as-judge step. With scores in hand, the authors compute a global ranking and separate per-task rankings, then compare them using Spearman correlation to see whether the global ordering carries over to each task. They additionally fit a 2-parameter IRT model (a standard psychometric technique that estimates item difficulty and item discrimination) to check that the benchmark behaves like a well-constructed test, and they rerun evaluations under different prompt formulations to confirm the rankings are not prompt-dependent.

Why This Matters

Impact on research. The paper argues against a common benchmarking shortcut: reporting one number and one leaderboard position per model. If four of eleven tasks show no meaningful correlation with the overall ranking, then aggregate scores are not a valid proxy for domain competence, and evaluation work in specialized technical fields should report task-level breakdowns. The execution-based, fully automatic grading design also offers a template for benchmarks that avoid the cost and subjectivity of human annotation.

Real-world applications:

• Model selection for quantum software development. Teams building assistants for circuit writing, transpilation, or debugging can pick a model based on the task they actually need rather than a general leaderboard position.
• Quantum error correction tooling. Error-correction tasks are evaluated separately, which matters for groups building LLM-assisted workflows around QEC code design.
• Quantum education and training tools. Educators can identify which subtopics a given model handles reliably before deploying it as a tutor or code helper.
• Regression testing in LLM pipelines. Because everything is execution-verified, the benchmark can be run automatically as part of a model upgrade or CI process.

Industry relevance. Quantum hardware and software companies, cloud providers offering quantum services, and LLM developers targeting scientific and engineering users all depend on knowing where a model is genuinely competent. A benchmark that exposes per-task strengths and weaknesses supports more honest capability claims than a single aggregate score.

Future Directions

• Explain the dissociations. The abstract establishes that four tasks decouple from the overall ranking but does not say why; identifying the underlying capability or data-coverage differences is the natural next question.
• Extend beyond the current grid. The benchmark covers ten models and eleven tasks — expanding model coverage, adding tasks, or increasing difficulty levels would test whether the dissociations persist or grow.
• Use the IRT model for adaptive testing. Since item difficulty and discrimination are already estimated, the benchmark could be shortened or made adaptive, giving precise per-task estimates with fewer evaluations.
• Connect benchmark scores to real QC workflows. An open question is whether per-task scores predict success on actual quantum software engineering work, which the abstract does not address.

Target Audience

Most useful to LLM evaluation and benchmarking researchers, quantum computing researchers interested in LLM capabilities, and benchmark designers who want an execution-verified, fully automatic grading model. Also valuable to practitioners choosing models for quantum software tasks and to anyone who relies on leaderboard rankings to make model-selection decisions in a specialized technical domain.

Authors’ abstract

We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models $\times$ 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface.