- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This is a theoretical paper on fitted Q-iterations (FQI), an offline RL method to learn the optimal policy. Finite sample guarantees for the expected reward of the learned policy exist in literature for actions which are finite dimensional, but this paper treats the infinite dimensional case where the action is a function.
It identifies a key challenge to obtaining such a guarantee : classical literature requires a coverage condition linking the behaviour distribution (from which the rollout data is sampled) to policy-selected distribution. The paper shows that this condition is too restrictive for functional actions through a constructed example (Example 1, section 3.2) and also empirical evidence (Figure 1 b, Section 5 on Offline Pendulum Control). To remedy this, it introduces an alternative condition for functional actions (Assumption 2). Under this assumption, the paper transfers control of critic error from the logged state-action distribution to actions selected during policy improvement (Proposition 1).
A second challenge is that a large functional functional action space can make the argmax step of FQI computationally infeasible. To remedy this, the paper adopts the objective of FQI to continuous actions and regularizes it using an RKHS norm on the policy (Algorithm 1).
Finally, the paper bounds the finite sample error in Theorem 1. For the examples in section 4, the paper computes the asymptotic dependence of the error on various quantities, notably, (number of independent subject x trajectory length). It is instructive to note that the asymptotics depend on a quantity which they call coverage exponent, which basically measures the rate of error transfer from logged state-action distribution to actions selected during policy improvement. The experiment in section 5 demonstrates the advantage of functional action (Figure 1a) and the restrictiveness of the usual coverage condition in literature (Figure 1b) in an offline pendulum control task.
Section 5 has several experiments comparing FT-GAE and Truncated GAE over practical RL tasks and ablation analyses.
Critical strengths
- The paper gives the first finite sample theory for the reward with functional actions under a critic-relative coverage condition. Although some previous works in literature can technically handle functional actions (see for instance the paper “A Measure-Theoretic Finite-Sample Theory for Adaptive-Data Fitted Q-Iteration”, 2026 by Haussmann et al.), this is the first paper with a detailed explanation of the coverage condition, and a reasonable modification for this setting.
Critical weaknesses
- The paper is a bit lacking in experimental demonstration. The only practical experiment performed is the offline pendulum control experiment, and even there, the main assumption which drives the theory (Assumption 2) is not clearly demonstrated.
Questions for authors
- Clarification: Could the authors clarify how the plot in Figure 1 c relates to the assumptions made in the paper? I believe it is supposed to be a motivation for the assumptions in Section 3.2, but I do not see how the precise assumptions are explained by the plot.