Time-ordered free energy in correlated quantum systems: An agentic approach

AI summary

Overview

Research area: Quantum thermodynamics, specifically the thermodynamics of computation and information — combining resource-theoretic work extraction, hidden Markov models (HMMs) from computational mechanics, and dynamic programming.

Technical level: Advanced. The paper assumes familiarity with quantum relative entropy, thermal operations, von Neumann and Shannon entropies, and Markov-chain / belief-state methods. The core ideas, however, are stated in an accessible operational narrative.

Scope (one sentence): The paper defines and computes the maximum work an online, causally constrained agent can extract from a temporally correlated sequence of quantum systems generated by a hidden Markov source, introducing "time-ordered free energy" (TOFE) and a linear-in-sequence-length dynamic programming algorithm for finding the optimal strategy.

What This Paper Is About

When a stream of quantum systems arrives one at a time and their correlations come from hidden internal states the agent cannot see, how much work can the agent actually extract if it must act online — choosing each extraction method using only what it has already observed? The paper answers this by formalizing the problem as an agent interacting with a hidden Markov model (HMM) source, then showing the optimal sequential strategy can be found efficiently rather than by brute-force search over exponentially many action histories. The maximum achievable work under these causal constraints is named the time-ordered free energy (TOFE), and the paper quantifies the unavoidable energetic penalty that causality imposes.

Key Contributions

1. Definition of time-ordered free energy (TOFE). The authors define ℱ_TO^(1:L)(ρ^(1:L)) as the maximum expected work extracted over L time steps, maximized over all causal strategies Γ whose action at time t depends only on the history h_{t−1} = (a_1, w_1, …, a_{t−1}, w_{t−1}).

2. A linear-time dynamic programming algorithm (Theorem 1). Using belief states as sufficient statistics, they show an optimal classical-causal policy Λ* can be found in time O(L) for a fixed action set — versus the exponential scaling of optimizing over all histories. They also prove (Theorem 3, Appendix G) that it suffices to search over possible eigenbases rather than the full space of density matrices.

3. A thermodynamic hierarchy and the notion of causal dissipation. They place TOFE between two benchmarks — the unconstrained non-equilibrium free energy ℱ_noneq^(1:L) (upper bound) and the greedy local-optimizing (LO) work 𝒲_LO^(1:L) (lower bound) — establishing ℱ_noneq^(1:L) ≥ ℱ_TO^(1:L) ≥ 𝒲_LO^(1:L) and the corresponding rate hierarchy f_noneq ≥ f_TO ≥ w_LO. The gap δ(Q_{1:L→}) ≔ ℱ_noneq^(1:L) − ℱ_TO^(1:L) is termed causal dissipation.

4. An information-theoretic expression for causal dissipation (Theorem 2). The dissipation is written as a minimum over policies of an expected sum of Shannon entropies of the extracted work given the history, plus a von Neumann entropy term for the final reduced post-measurement state, minus the entropy of the joint multi-time state — giving the thermodynamic cost of causality an operational measurement-based interpretation.

Main Findings

• The optimal strategy is not greedy. The paper explicitly answers a previously open question in sequential energy harvesting from quantum systems: the best strategy is not necessarily to maximize the energy gain at each time step. A "mismatched" protocol that dissipates immediate work can buy predictive information that pays off later; the dissipative term D(ξ_η‖ρ_a) is interpreted as "the price of information."

• TOFE is computable in linear time. For a fixed action set, the DP algorithm scales as O(L), and if the agent's initial belief matches the HMM's initial distribution, the maximized cumulative work equals the TOFE of Eq. (2).

• A stationary optimal policy yields a closed rate formula. In the asymptotic limit the optimal policy becomes stationary, inducing a finite-state Markov chain on belief states; the TOFE rate f_TO is obtained from the resulting limiting distribution π_lim (Eq. 8).

• Causal dissipation connects to known quantum-information quantities. For a bipartite system (L = 2), causal dissipation reduces to a measure of quantum discord. It vanishes for states with no quantum correlations (e.g., product states or classical-quantum states), is asymmetric under time ordering (δ(Q_{1:L→}) ≠ δ(Q_{1:L←})), and its asymptotic rate can vanish even when the finite-time total dissipation is non-zero. It can be viewed as a quantum extension of the cost of modularity.

• Perturbed-coin benchmarks show where the effects live. For the perturbed coin source with overlap r = |⟨φ_0|φ_1⟩|², the causal-dissipation rate (f_noneq − f_TO) vanishes whenever the dynamics are deterministic (p ∈ {0, 1}), purely classical (r = 0), or uncorrelated (p = 1/2 or r = 1) — regimes with no explicitly quantum correlations, where the causally constrained agent is perfectly efficient.

• The advantage over greedy extraction peaks at moderate stochasticity. The DP agent's advantage (f_TO − w_LO) peaks in moderately stochastic regimes and vanishes as the process approaches maximum randomness or indistinguishability, where both strategies converge.

• Sacrifice scales with uncertainty in predictable regimes. Analyzing the expected immediate dissipation against the binary entropy h₂(η_ε) of the agent's belief, the paper finds dissipation peaks at maximum uncertainty (h₂(η_ε) = 1) and decreases toward certainty (h₂(η_ε) → 0) for predictable sources, while in unpredictable regimes sacrificing energy has little value because it yields little information. Scans cover p ∈ {0.1, 0.2, 0.3, 0.4, 0.5} at fixed r = 0.1, and r ∈ {0.1, 0.3, 0.5, 0.7, 0.9} at fixed p = 0.2.

• Finite-horizon numerics confirm the theory. Simulations for L = 3, 4 show the work deficit exactly matches the predicted causal dissipation (Appendix I.3, Fig. 5). A closed-form expression for f_noneq remains intractable for this model, so the authors rigorously establish an analytical lower bound using the data processing inequality (Appendix H).

• The memoryless-memory assumption is not a limitation. The paper shows the agent need not store the entire history h_t; a belief state η_t over latent states suffices, updated by Bayesian inference after each observed work value, and Appendix J argues these updates can always be done near-reversibly with negligible heat dissipation.

Methodology in Plain English

The authors model the environment as a hidden Markov model: a machine with hidden internal states that emits a labelled quantum state σ^(x) at each step, with transition probabilities T^(x){s,s′} = Pr(S_t = s′, X_t = x | S{t−1} = s). Summing over all label sequences weighted by their probabilities gives the multi-time quantum state ρ^(1:L) of Eq. (1).

The agent has no persistent quantum memory, is restricted to thermal operations mediated by a memoryless thermal reservoir at inverse temperature β = (k_B T)^{-1}, and stores extracted work in an ideal battery modeled as a semiclassical weight. Its action set is 𝒜 = {𝒲_ρ}, protocols parametrized by target states ρ, each designed to extract all non-equilibrium free energy from ρ, using the protocol of Ref. [16], which produces work values whose statistics mirror a projective measurement in the target eigenbasis.

Rather than track the exponentially growing action-observation history, the agent compresses it into a belief state — its subjective probability distribution over which hidden state the source is in. After each round it performs a Bayesian update from the observed work value. Because belief states are sufficient statistics, the search over history-dependent policies collapses to a search over belief-dependent policies, which is solved by dynamic programming over a discretized belief space; the authors note that value functions from progressively finer discretizations converge to those of the continuous-state problem. A numerical study of the perturbed coin — a two-state HMM whose qubits can be in |φ_0⟩ or |φ_1⟩ with flip probability p — then compares the TOFE rate against the unconstrained free-energy rate and the greedy local-optimizing strategy.

Why This Matters

Impact on research. The work supplies a rigorous accounting of what temporal causality costs in quantum thermodynamics. Where most treatments of free energy assume access to the whole state at once, this paper quantifies the price of acting online — a cost the authors show is generally nonzero even in the absence of entanglement and is tied to quantum discord for two-time systems. The linear-time algorithm converts a seemingly intractable optimization (exponentially many action sequences) into a practical one, and the belief-state formulation imports computational-mechanics tools into quantum thermodynamics.

Real-world applications (as motivated in the paper):

• Information ratchets and autonomous energy harvesting, modeling organisms foraging for food or machines harvesting energy from a fluctuating environment.
• Edge devices seeking a power source, where an online, memory-limited device must decide how to tap ambient fluctuations without buffering the whole stream.
• Quantum batteries and work storage, since extracted work is formally accounted as the energy change of an ideal semiclassical weight.
• Quantum sensing and measurement, because sequential work extraction is shown to be mathematically equivalent to sequential quantum measurement within this framework — work outcomes act as indirect measurements of the system.

Industry relevance. Relevant to quantum-technology groups building autonomous or resource-constrained quantum devices, and to anyone designing online control policies where memory and causal ordering are binding constraints — the "price of information" trade-off between immediate yield and predictive knowledge is a general design principle.

Future Directions

• Non-degenerate Hamiltonians. The main text assumes an energy-degenerate Hamiltonian. Appendix A shows the relation between heat dissipation and agent-expectation mismatch persists, but the physical control cost of the rotation stage (the energy ΔU required to diagonalize a target state) remains unresolved and is explicitly left for future work.
• Learning the source, not just the hidden state. The current agent begins with a full mathematical description of the hidden Markov source and only infers which hidden state it occupies. Extending to agents that must learn the source dynamics themselves, in the spirit of quantum-state-agnostic work extraction, is proposed.
• Causal irreversibility and temporal asymmetry. Causal dissipation is a new source of irreversibility, mirroring findings in machine learning and stochastic modeling that some classical sequences are harder to predict in one temporal direction than the other. Investigating such effects in energy harvesting is suggested.
• Accounting for coherence resources. Realizing the non-degenerate rotation stage would require external sources of coherence (e.g., lasers) whose energy must be counted; the authors defer careful accounting of these resources.

Target Audience

Quantum thermodynamics and quantum-information researchers; graduate students working on work extraction, thermal operations, or the thermodynamics of learning and prediction; and applied researchers in control theory or reinforcement learning who are interested in belief-state dynamic programming under strict causal constraints and in the trade-off between immediate reward and information gain. Readers from computational mechanics will find the HMM and cost-of-modularity connections familiar, while readers from quantum foundations will find the discord link and the time-ordering asymmetry of most interest.

Note: the provided paper text is truncated partway through Appendix B (Lemma 1), so appendices C–J are referenced but not visible in the supplied content; details of those proofs are not reported here.

Authors’ abstract

How much work can an agent extract from a temporal sequence of quantum states when it can only operate online under causal constraints---deciding which energy extraction method to use with knowledge of what it has observed before? Here, we study this problem in the context of quantum state sequences that are potentially non-Markovian---generated by some underlying hidden Markov machine that the agent cannot observe. Using techniques from dynamic programming and computational mechanics, we present a method to identify the provably optimal agent strategy, with time complexity that scales linearly with sequence length. This motivates us to introduce the maximum work such agents can extract---time-ordered free energy(TOFE)---as a fundamental measure of free energy available in a temporally correlated quantum system subject to causal considerations.