Quantum Fidelity-per-Cost: A Metric for Evaluation of Quantum Computing Systems

AI summary

Overview

Research area: Quantum computing systems evaluation — specifically benchmarking cloud-accessible quantum processors with monetary cost included as a first-class variable (quantum benchmarking, cloud pricing models, resource allocation).

Technical level: Intermediate. The metric itself is a simple closed-form expression, but the motivation and comparisons assume familiarity with NISQ hardware metrics such as Quantum Volume, CLOPS, TVD and KL divergence.

Scope (one sentence): A cross-provider measurement study of 247 Bell-state executions on 14 cloud QPU access-path entries (12 distinct physical QPUs) that proposes and stress-tests Quantum Fidelity-per-Cost (QFC), a score combining KL-divergence fidelity, shot count and billed monetary cost.

What This Paper Is About

Cloud quantum computers are bought, not just benchmarked, and providers bill in incompatible ways — some charge a fixed task fee plus a per-shot fee, some bill by runtime, some per shot only. Existing metrics such as Quantum Volume and CLOPS measure hardware capability or throughput but do not answer the budget-constrained question of which backend gives the most useful output per dollar. The authors build a reproducible dataset of real executions across AWS Braket, IBM Quantum Runtime, IQM Resonance and OQC Cloud, then define a single cost-aware score and test whether including price changes which device looks best.

Key Contributions

1. Cross-provider measurement dataset. 247 Bell-state executions across 14 QPU access-path entries (12 distinct physical QPUs) at shot counts from 20 to 1000, with per-run counts, derived TVD and KL divergence values, and provider-reported cost inputs, released publicly at https://github.com/caslab-code/qc-fidelity-and-cost.

2. A four-class cloud pricing taxonomy and consistent cost modeling. Per-task + per-shot, runtime-based, per-shot only, and per-gate execution, with the first three implemented for the run set (AWS Braket, IBM Quantum Runtime, IQM Resonance, OQC); the per-gate class is catalogued but not evaluated.

3. Definition of QFC, a score of the form exp(−α·D_KL(P_emp‖P_ideal))·N^γ / C(N)^β, combining a KL-based information-quality factor, a shot-count reward and billed cost, weighted by α, β, γ and evaluated at the default α = β = γ = 1 with a smoothing constant ε = 10⁻¹².

4. A ranking-disagreement finding plus a billing-regime finding. The lowest-TVD device at 1000 shots is not the top-ranked device by QFC, and this persists under reweighting; separately, a device's billing model — not its hardware — fixes how its QFC scales with shot count.

Main Findings

• Fidelity and cost rankings disagree substantially. At 1000 shots, IBM Marrakesh has the lowest mean TVD among the superconducting entries (0.0254) but ranks fifth by QFC (217.93), while IQM Sirius reaches the highest QFC (608.21) with a mean TVD of 0.0502. QFC values span roughly 1.5–2 orders of magnitude, with IQM Sirius approximately 47× the lowest-ranked entry, IBM Torino (retired, QFC 12.75).

• The gaps between devices are large but compressed by cost. At 1000 shots, mean TVD spread across access paths is a factor of five, from 0.025 (IBM Marrakesh) to 0.130 (IBM Torino, retired).

• Billing model dictates shot scaling. From 100 to 1000 shots, the per-shot-only entry moves by 1.04×, the three task-plus-shot entries by 2.13–2.52×, and the ten runtime-billed entries by 5.96–34.31×. Provider-reported runtime is largely fixed overhead at this circuit size: IQM Sirius averages 1.34 s at 50 shots and 1.73 s at 1000 (a 1.29× increase for 20× the work), and IBM Brisbane reports exactly 1 s at 100, 500 and 1000 shots.

• Rankings are shot-budget specific. OQC Toshiko Tokyo-1 ranks first at 50 and 100 shots, fifth of 14 at 500, and eleventh of 14 at 1000, without its fidelity changing appreciably.

• Cloud access path matters for the same physical QPU. IQM Emerald scores 238.01 via AWS and 73.82 via IQM Resonance, a 3.22× gap that decomposes into a 2.04× fidelity factor (mean KL 0.817 vs. 1.528) and a 1.58× cost factor ($1.90 vs. $3.00 per run). For IQM Garnet the direction reverses (AWS 224.13 vs. IQM Resonance 275.89), and the decomposition does not close (1.10× predicted against 0.81× observed) because the Resonance-path cost distribution is heavy-tailed.

• The trend is not confined to one hardware modality. IonQ Aria-1 (trapped ion, via AWS Braket, only n = 2 runs per shot level) records the lowest mean TVD anywhere in the study (0.0245 at 1000 shots) yet a QFC of 27.84, which would place it fourteenth of fifteen — roughly 22× worse per dollar than the best-value device, because a 1000-shot run costs $30.30 at $0.03 per shot plus a $0.30 task fee against $0.60 for the top-ranked entry.

• The ranking is robust to reweighting. Kendall's τ against the default ordering is +1.000 for γ = 0.5 or γ = 0, +0.824 for β = 0.5, +0.802 for α = 2, +0.714 for α = 0.5 or β = 2, and +0.560 for α = 5 — the only variant that displaces IQM Sirius, promoting IBM Marrakesh to first. IQM Sirius is top-ranked in nine of ten parameterizations. Billing IQM on exact rather than rounded seconds leaves the ordering identical (τ = +1.000) and raises IQM Sirius to 708.50.

• Even the seven IBM devices span a 17× range under one billing model at one per-second price ($1.60/s), so the spread is not merely a vendor-pricing artifact.

• The smoothing constant is a real sensitivity, not a nuisance. Varying ε from 10⁻⁶ to 10⁻¹⁵ shifts the 1000-shot ranking by Kendall τ of 0.67–0.89, though IQM Sirius stays top throughout.

Methodology in Plain English

The authors ran one deliberately small, simple circuit — a two-qubit Bell state, whose ideal output should be 00 or 11 with 50% probability each — many times on many cloud-accessible machines, and recorded the measured outcome counts. Because the ideal answer is known exactly, they could quantify how far each machine's empirical distribution strayed from it, using total variation distance as a general error measure and KL divergence as the fidelity term inside their score.

They treated a physical machine reached through two different commercial interfaces as two separate entries when pricing or runtime semantics differ, which is why 12 physical QPUs produce 14 entries. Qubit selection and routing were left to each provider's default toolchain with no manual initial layout, so the measurement reflects what an ordinary user experiences rather than best-case qubit pairs.

Cost was computed from each provider's published billing rule: a fixed task fee plus a per-shot fee, a per-shot fee alone, or per-second runtime multiplied by the provider-reported duration field. For IBM they used the Qiskit Runtime quantum_seconds field; for IQM Resonance the runtime_seconds field. They acknowledge these fields are provider-defined and may include or exclude classical overhead, queueing or compilation in different ways, and they treat that definitional difference as part of the access path being evaluated. Runtime-billed IQM entries were rounded up to the nearest whole second as a conservative reading, and a no-rounding variant confirmed the ranking was unaffected.

The score divides a bounded fidelity factor by cost and multiplies by a shot-count term, so better fidelity and more shots raise it while higher price lowers it. Because every per-run cost is recoverable in closed form from the saved counts, all robustness analysis in the paper can be regenerated without provider credentials and without re-executing anything on hardware.

Why This Matters

The paper reframes quantum hardware selection as a purchasing decision rather than a pure physics comparison, and supplies a reproducible pipeline for anyone who wants to recompute a cost-aware ranking as prices and calibrations change. Its central negative result — that fidelity-only selection can direct users to a device that costs roughly 22× more per unit of useful output — has direct budget consequences.

Real-world applications:

• Budget-constrained backend selection: research groups and early commercial users can run a short calibration circuit per candidate access path and pick by value rather than headline fidelity.
• Scheduler objective functions: current quantum cloud schedulers (Qonductor, QSRA, QuSplit) optimize predicted fidelity, runtime or utilization, but not price; QFC is directly usable as an objective they could maximize, provided fidelity and runtime are predicted rather than measured.
• Procurement monitoring: an access path can be re-scored from saved counts whenever a provider revises prices, without re-running jobs.
• Billing-model design: recognizing that runtime billing exposes users to infrastructure jitter unrelated to output quality — three IQM Garnet runs at the same shot count were billed $1.00, $3.00 and $3.50 while their TVDs were 0.038, 0.038 and 0.043 — highlights that pricing structure, not just price level, shapes value.

Industry relevance: the finding that a device's billing model, not its hardware, governs how its score scales with shot count is directly actionable for providers setting tariffs and for buyers choosing shot budgets. The related-work observation that per-shot billing is itself exploitable — packing several circuits into one shot via mid-circuit reset can cut billed cost by up to an order of magnitude — reinforces that a pricing model is a design surface.

Future Directions

• Broader workloads. The paper explicitly cannot support conclusions about deeper circuits or applications; GHZ states and small QAOA/VQE ansätze are named as natural follow-ups.
• Alternative scoring forms. Different divergences (Jensen–Shannon, Hellinger), time-to-solution cost formulations, and a reliability-weighted variant that charges providers for failed submissions — four IQM Garnet submissions failed with backend errors and one still reported 1.83 s of billable runtime for no usable result.
• Queue and wait-time cost. Named as future work; current QFC charges only execution price, not time spent waiting.
• Prediction rather than measurement. Using QFC as a scheduler objective requires forecasting the fidelity and runtime terms before execution, which the current paper does not attempt. Re-computing rankings annually as prices change is also recommended.

Target Audience

Researchers and practitioners choosing among cloud quantum backends on a budget; quantum cloud providers interested in how their tariff structure affects perceived value; benchmarking and metrics researchers extending cost-aware evaluation; and systems/scheduling researchers who want a post-hoc objective to complement pre-execution placement heuristics. Readers without quantum computing background can follow the cost and ranking argument, but the fidelity discussion assumes comfort with output distributions and divergence measures.

Authors’ abstract

Cloud-accessible quantum computing has made hardware comparison not only a physics benchmark but also a practical purchasing decision. Cost-aware comparison of quantum computers remains underexplored and is difficult to do under the heterogeneous billing models offered by various cloud-based quantum computing providers. This paper makes two main contributions to enable price-aware comparison of quantum computers. First, this work presents a cross-provider measurement study of quantum circuit execution fidelity spanning 14 cloud QPU access-path entries (12 distinct physical QPUs) across four cloud access paths: Amazon Web Services (AWS) cloud, IBM Quantum Runtime (IBM) cloud, IQM Resonance (IQM) cloud, and Oxford Quantum Circuits (OQC) cloud. Second, this work proposes and analyzes a cost-aware score, Quantum Fidelity-per-Cost (QFC), which combines Kullback--Leibler (KL) divergence from an ideal output distribution, shot count, and monetary cost into one possible metric under a documented billing model. The main empirical observation from this work is that cost-aware ranking can differ from purely fidelity-based evaluation of quantum computers, and that users may select different quantum computing backends when they consider price in their selection, as opposed to selection based on fidelity alone. This work shows that the ranking is stable under reweighting of the metric, and that a device's billing model, not its hardware, governs how its score scales with shot count. Reported QFC values change as new machines come online or as providers revise their prices.