Quantum Optical Reinforcement Learning via Spectrum-Resolved Hong-Ou-Mandel Interference

AI summary

Overview

Research area: Quantum photonic machine learning — specifically quantum optical neural networks built on Hong-Ou-Mandel (HOM) interference — applied to continuous-control reinforcement learning (RL) and to the calibration of superconducting two-qubit gates.

Technical level: Advanced. The paper combines two-photon interference physics, spectral-mode encoding, statistical learning theory (Hoeffding bounds), PPO-style actor–critic RL, and transmon gate calibration.

Scope: The paper proposes a spectrum-resolved HOM (SR-HOM) architecture that turns the frequency-resolved coincidence pattern of interfering single photons into a trainable feature map, then uses it as the actor and critic of a compact optical RL agent benchmarked on five continuous-control tasks and on simulated tunable-coupler CZ/iSWAP gate recalibration.

What This Paper Is About

Standard HOM-based optical neurons integrate over all optical frequencies and output a single scalar coincidence probability, which is enough for simple overlap-based discrimination but compresses away the spectral information the interfering photons carry. That compression makes such neurons poorly suited to tasks that need structured continuous outputs and nonlinear state-action features, such as actor–critic reinforcement learning in continuous action spaces. The paper's goal is to keep the frequency-resolved coincidence pattern as a tensor-valued optical feature map, and to show that a compact actor–critic agent built from this primitive can learn continuous-control policies and can also correct drifting two-qubit gates in simulation.

Key Contributions

1. A spectrum-resolved HOM (SR-HOM) module. Instead of a single scalar, the module performs frequency-resolved coincidence detection over k disjoint frequency bins, producing a structured coincidence tensor C_ij(x, λ) that retains the two-photon spectral correlations discarded by integrated HOM readout. Coarse-graining this tensor back to one number recovers the standard HOM neuron.

2. A quantum optical actor–critic architecture. Grouped diagonal entries of the coincidence tensor generate m-dimensional continuous actions through a fixed monotone map, while a second SR-HOM module encodes the state–action pair and feeds a linear readout that produces the scalar value estimate Q. Both modules use the same physical primitive with complementary readouts.

3. A sampling-complexity bound for the diagonal readout. Using Hoeffding's inequality plus a union bound over k diagonal entries, the authors show that M ≥ (1/(2ε²)) log(2k/δ) coincidence samples suffice to estimate all action-relevant diagonal marginals to accuracy ε with confidence 1 − δ, giving O(ε⁻² log k) scaling in spectral resolution k that does not scale explicitly with the state-encoding dimension d. The corresponding photon-pair cost is O((1/P_c) ε⁻² log(k/δ)).

4. An online RL calibration protocol for tunable-coupler two-qubit gates. The same architecture is applied to correcting drifted CZ and iSWAP gates, using only measurement-derived gate diagnostics (never the hidden drift parameters or the true gate fidelity) to output bounded incremental corrections to five coupler-pulse parameters and two virtual-Z phases.

Main Findings

• Faster, more stable learning on five continuous-control benchmarks. Across LunarLanderContinuous-v3, InvertedDoublePendulum-v5, Reacher-v5, BipedalWalker-v3, and Ant-v5, the SR-HOM agent is compared against a parameter-matched two-hidden-layer MLP trained with the same PPO-style on-policy actor–critic protocol.

• LunarLanderContinuous-v3 headline numbers. SR-HOM reaches the return threshold of 200 after 666 episodes versus 2935 for the MLP — a 77.3% reduction in episodes-to-threshold and a 4.4 times improvement in sample efficiency. Post-threshold collapse rate drops by 82.3%, curve volatility by 13.2%, and the best 100-episode moving-average return improves by 74.0% (276.81 versus 159.06).

• Three benchmarks were not reached by the baseline. The MLP did not reach threshold within the aligned training horizon on InvertedDoublePendulum-v5 (SR-HOM: 35203 episodes to the 9100 threshold), BipedalWalker-v3 (SR-HOM: 8796 episodes to the 300 threshold), and Ant-v5 (SR-HOM: 17594 episodes to the 5000 reference level). On Reacher-v5, SR-HOM reached the −3.75 threshold at episode 20362 versus 48195 for the MLP.

• Peak smoothed returns favored SR-HOM everywhere. Best 100-episode moving-average returns were 9352.23 versus 1000.00 (InvertedDoublePendulum-v5), 313.09 versus 11.55 (BipedalWalker-v3), 5240.10 versus 358.03 (Ant-v5), and −3.44 versus −4.47 (Reacher-v5).

• One metric favored the baseline. On InvertedDoublePendulum-v5, curve volatility was 1377.53 for SR-HOM versus 92.63 for the MLP, and on Reacher-v5 the post-threshold collapse rate was 0.5597 for SR-HOM versus 0.9128 for the MLP (both lower for SR-HOM there); the volatility comparison on the pendulum task is the clearest exception to the otherwise consistent SR-HOM advantage.

• Gate calibration restored fidelities close to drift-free references. Under independently generated drift conditions not seen in training, uncorrected mean fidelities fell to 0.9547 (CZ) and 0.9640 (iSWAP); RL corrections raised them to 0.9917 and 0.9952, both exceeding 99.8% of their drift-free pulse references. Mean leakage fell by 27.4% for CZ and 35.4% for iSWAP.

• Simulations were run in the exact-probability limit. The reported learning curves evaluate C_ij directly from normalized spectral-mode coefficients, so finite-shot photon-counting fluctuations are not included; the sampling bound characterizes the hardware cost that would replace this.

• No experimental demonstration is reported. The three required capabilities (spectral state encoding, a reconfigurable trainable probe, and frequency-resolved coincidence detection) are each cited as individually demonstrated elsewhere, with a proposed integrated route on thin-film lithium niobate chips; the paper itself presents numerics only.

Methodology in Plain English

Each environment observation is written into the spectral wavefunction of a single photon — the color profile of the photon carries the data. A second, trainable "probe" photon carries the learnable parameters in its own spectral amplitude and phase. The two photons meet at a balanced 50:50 beam splitter, where they interfere, and the detectors record not just whether the photons emerged in separate outputs, but which pair of frequency bins they landed in. That bin-pair histogram is the coincidence tensor.

For the actor, the diagonal entries of this tensor (both photons in the same bin) are summed in disjoint groups, one group per action dimension, and passed through a fixed monotone map to produce the bounded continuous action. For the critic, the state and the action are each encoded into spectral vectors, combined with trainable probe spectra, and the resulting bin-resolved features are grouped and linearly mapped to a single value estimate.

Both networks are trained with the same PPO-style on-policy actor–critic procedure used for the baseline; only the function approximators change, which isolates the architectural effect. The MLP baseline is deliberately given an advantage: hyperparameters are searched over widths, learning rates, and exploration scales, multiple seeds are run, and the best run by 100-episode moving-average return is reported, with all curves truncated to a common episode length.

For the gate-calibration task, the agent observes finite-shot diagnostics of coherent control error and leakage for a simulated tunable-coupler transmon system with stochastic drift and control-line distortion, and outputs small correction steps; the reward uses only those diagnostics plus action regularization, and true fidelity is held back purely for validation.

Why This Matters

The paper reframes photonic interference from a similarity score into a reusable, trainable physical feature map. That matters because it gives optical hardware something to do beyond binary discrimination: producing structured, multi-dimensional outputs and nonlinear features without reconstructing the input field or performing resolution-dependent digital post-processing.

Real-world applications:

• Superconducting quantum processor maintenance: an agent that nudges coupler pulses and virtual-Z phases during idle periods to extend the interval between full recalibrations, addressing a known operational bottleneck as system size grows.
• Photonic accelerators for control and decision-making: low-latency optical front-ends for continuous-control policies in robotics or autonomous systems where the feature generation is done by interference and only a small digital readout remains.
• High-dimensional sensing and signal processing: spectral encoding of broadband inputs into single photons, with the coincidence tensor acting as a natural feature extractor for classification or prediction tasks.
• Compact optical inference hardware: the architecture is argued to be implementable on integrated platforms such as thin-film lithium niobate, where frequency-encoded gate sets have been reported.

Industry relevance: the gate-calibration result speaks directly to quantum-computing operators who lose processor uptime to recalibration; the architectural result speaks to photonic-computing efforts looking for workloads where optical interference provides a genuine representational advantage rather than replacing digital layers one-for-one.

Future Directions

1. Experimental realization. Every ingredient is cited as separately demonstrated, but no end-to-end SR-HOM module has been built; closing that gap — particularly frequency-resolved coincidence detection with dispersive grating demultiplexing onto detector arrays — is the obvious next step.

2. Finite-shot and loss-aware training. The reported learning curves assume exact probabilities. Retraining with empirical estimates Ĉ_ij = n_ij / M, detector dark counts, and imperfect mode overlap would test whether the claimed O(ε⁻² log k) sampling advantage survives realistic photon budgets.

3. Scaling and readout design. The paper argues that off-diagonal and higher-order spectral correlations are an unused resource; how to assign computational roles to off-diagonal tensor components (the authors sketch classification and prediction as extensions) and how performance scales with bin number k and action grouping remain open.

4. Hardware-in-the-loop calibration. The gate-calibration study is a simulation with a simplified drift and distortion model whose full description sits in a supplemental section that is truncated in the provided text; validating the protocol on a real transmon processor, and quantifying how much calibration overhead it actually saves, is untested here.

Target Audience

Researchers working at the intersection of quantum photonics and machine learning, particularly those building HOM-based optical neural networks or frequency-encoded photonic processors. It is also relevant to quantum-computing experimentalists interested in automated gate calibration, and to RL practitioners curious about physical substrates for function approximation. A reader needs comfort with two-photon interference, density-matrix-style spectral representations, and actor–critic methods; the paper's own exposition of the sampling bound is accessible to graduate students with a statistics background, but the photonic hardware discussion assumes familiarity with frequency-bin encoding and integrated photonics.

Authors’ abstract

Hong-Ou-Mandel (HOM) interference-based optical neural networks can offer complexity advantages on benchmark learning tasks, but conventional readout compresses the coincidence spectrum into a single scalar, limiting its use in complex settings such as continuous-action reinforcement learning. Here we introduce a spectrum-resolved HOM (SR-HOM) architecture that promotes the photons' spectral degrees of freedom to a trainable computational resource and use it to construct a compact optical actor-critic agent. Diagonal spectral responses generate continuous actions, while higher-order spectral correlations provide nonlinear state-action features for value estimation. Across five continuous-control benchmarks, SR-HOM outperforms parameter-matched multilayer-perceptron baselines, including a \(4.4\times\) improvement in sample efficiency and a \(74.0\%\) increase in best 100-episode moving-average return for LunarLanderContinuous-v3. Applied to online calibration of drifted tunable-coupler CZ and iSWAP gates for transmon qubits, simulations show it restores fidelities to \(0.9917\) and \(0.9952\) respectively, exceeding \(99.8\%\) of their drift-free calibrated values.