Qlippy: A Retrieval-Augmented GenAI Assistant for Reproducible Quantum Workflows and Experiment Tracking

AI summary

Overview

Research area: Quantum software engineering, specifically tooling support for experiment tracking, provenance, and reproducibility in quantum program development, using retrieval-augmented generation (RAG).

Technical level: Intermediate. Readers benefit from familiarity with quantum programming frameworks (Qiskit), experiment tracking tools (MLflow), and basic retrieval-augmented generation concepts, but the paper explains the motivation and architecture in accessible terms.

Scope (one sentence): The paper presents and preliminarily evaluates Qlippy, a Visual Studio Code–embedded GenAI assistant that grounds its answers in a curated quantum-software-engineering corpus and injects MLflow-based experiment tracking, aligned to the QProv schema, into existing Qiskit programs.

What This Paper Is About

Quantum software development is iterative and error-prone: noisy hardware, calibration drift, and backend variability mean developers re-run experiments constantly, so tracking runs and recording provenance matters. Yet these practices are rarely adopted systematically, because the tooling (MLflow, provenance schemas like QProv) is complex and requires software-engineering knowledge that many quantum practitioners, especially novices, do not have. The paper's goal is an assistant embedded in the developer's own editor that both explains reproducibility concepts in context and automatically augments Qiskit code with experiment-tracking instrumentation, grounding every response in curated sources rather than relying on a language model's parametric memory.

Key Contributions

1. A two-environment architecture for a grounded quantum-software assistant. A Visual Studio Code extension supplies a chat interface on the developer's local workstation, connected via a REST API to a research environment hosting an MLflow tracking server and a RAG service built in Python with FastAPI and LangChain.

2. A curated, domain-specific knowledge base as non-parametric memory. A ChromaDB vector database stores a corpus of five research papers, two synthetically authored reference documents, and a curated subset of the MLflow documentation, ingested through file parsing, chunking, and embedding stages; an LLM-agnostic design allows commercial APIs or locally served open-weight models without changing the pipeline.

3. An augmentation capability that writes real tracking code, not a simulation. Rather than a toy mechanism, the assistant inserts MLflow-based tracking calls into user Qiskit programs to record runs, parameters, metrics, and artifacts, aligned with the QProv provenance schema.

4. A preliminary RAGAS-based evaluation comparing commercial and locally hosted generator models. Twenty question–answer pairs spanning five subdomains and four question types were scored by an LLM judge across four generator models, indicating that a locally deployable model can approach a commercial one on answer relevancy and factual coverage.

Main Findings

• Local model matched commercial on answer relevancy: The locally served, quantized Qwen3.5:9b achieved the highest answer relevancy (0.917) among the four generator models evaluated.

• Commercial model led on factual correctness: GPT-5.4 achieved the highest factual correctness (0.908), only 12% higher than Qwen3.5:9b's 0.814.

• Grounding reduces reliance on model scale: Because RAG shifts the knowledge burden from parametric memory to curated external sources, the advantage of large parametric models is described as less pronounced in this constrained, domain-specific setting.

• Evaluation setup: RAGAS (an LLM-as-a-Judge approach) was used with GPT-5.4-mini as judge over 20 question–answer pairs across five subdomains of experiment tracking and four question types (factual, summary, reasoning, unanswerable), scored on faithfulness, context recall, answer relevancy, and factual correctness (recall value based), with unanswerable questions excluded from the means.

• Results are relative, not definitive: The authors state that given the small question set and LLM-as-judge design, the numbers are best read as relative comparisons between configurations, and that a more rigorous evaluation is needed. Values for faithfulness and context recall are not reported in the paper content.

• Two validated demonstration scenarios: An advisory scenario (in-context explanation of tracking concepts such as what to track in the NISQ context) and an augmentation scenario (modifying a functional Qiskit program to add MLflow logging consistent with QProv guidelines), the latter judged successful when the modified program executes correctly and logs the expected data.

Methodology in Plain English

The authors built a working prototype rather than running a controlled user study. They assembled a small, hand-picked library of trustworthy material: five research papers, two reference documents they wrote themselves, and a selected portion of the MLflow documentation. This library is broken into chunks, converted into numerical embeddings, and stored in a vector database. When a developer asks a question in the chat panel inside Visual Studio Code, the system retrieves the most relevant passages from that library and hands them to a language model along with the question, so the answer is synthesized from verifiable source material instead of recalled from training data. The same retrieval step supports code modification: given an existing Qiskit program and a natural-language request, the assistant returns a modified version with MLflow tracking calls inserted to capture parameters, metrics, and provenance attributes matching QProv. The chat interface runs locally, while the retrieval service and MLflow server run in a separate research environment reached over a REST API. For evaluation, they posed the 20 curated question–answer pairs to four different generator models — two commercial (GPT-4o-mini and GPT-5.4 via the OpenAI API) and two served locally and quantized via Ollama (Gemma4:e4b and Qwen3.5:9b) — and had a separate LLM act as judge, scoring each response on four criteria.

Why This Matters

Research impact: The work argues that reproducibility and provenance infrastructure should be embedded in the development workflow rather than treated as an advanced, optional practice. It also offers evidence that, for narrow domain tasks, careful knowledge curation can substitute for raw model scale — a claim with implications for how domain-specific AI assistants are designed and evaluated.

Real-world applications:

• Quantum software education: Instructors can introduce experiment tracking and provenance without students first mastering MLflow, the QProv schema, and surrounding DevOps tooling.

• Research groups running NISQ experiments: Teams can capture circuit properties, execution parameters, backend settings, and noise conditions consistently across repeated runs, making results comparable and reproducible.

• Industry quantum teams onboarding developers: Practitioners arriving from physics or mathematics backgrounds can adopt structured tracking just in time, at the point in the workflow where it becomes relevant.

• Privacy-sensitive or cost-constrained settings: Because a locally hosted model approached a commercial one on answer relevancy and factual coverage within the RAG pipeline, deployment without recurring API fees and without sending proprietary code or unpublished artifacts to an external service becomes realistic.

Industry relevance: The assistant targets MLflow, a widely used classical machine-learning tracking system, and the QProv provenance schema, so generated workflows are intended to transfer directly to professional practice rather than to a simplified stand-in. The authors also frame the commercial-versus-local model choice as a socio-technical decision involving cost structures, governance requirements, and operational capability, not purely a performance comparison.

Future Directions

1. Strengthen the evaluation with expert input: Extend the question set beyond the current 20 pairs and add multi-expert validation, since the present study relies on a single LLM judge.

2. Formally assess provenance completeness for the augmentation scenario: The current demonstration judges success by whether the modified program executes and logs data consistent with QProv; a rigorous completeness assessment is proposed.

3. Compare against baselines: Test the grounded assistant against an ungrounded model of equal size and against general-purpose coding assistants, to isolate the effect of retrieval grounding.

4. Validate with real users and teams: The artifact has not been extensively validated with real developers, so adoption and usability in practice remain open questions, along with managing the infrastructure burden of local deployment (provisioning compute, model updates, reliability).

Target Audience

Quantum software engineering researchers and tooling developers; practitioners building Qiskit workflows who need reproducible experiment tracking; instructors teaching quantum computing who want to introduce software-engineering practices; and applied AI researchers interested in retrieval-augmented, domain-grounded assistants and the trade-offs between commercial and locally hosted models.

Authors’ abstract

Quantum software development is iterative and error-prone. Noisy hardware and repeated re-execution make experiment tracking, provenance, and reproducibility essential, yet these practices are hard to adopt because of tooling complexity and the specialized knowledge they demand. General-purpose language models can help but tend to hallucinate and lack grounding in domain-specific tooling. We present Qlippy, a retrieval-augmented GenAI assistant embedded in the development environment that grounds its responses in a curated corpus of quantum-software-engineering knowledge. Qlippy explains reproducibility and provenance concepts in context and augments existing Qiskit programs with MLflow-based experiment tracking aligned to the QProv schema. By separating knowledge from model parameters, grounding gives explicit control over the scope and provenance of the assistant's responses and reduces reliance on model scale, which points toward low-cost, privacy-preserving local deployment.