acceptodds
Under review as a conference paper at ICLR 2027

Finite-Sample Theory for Fitted Q-Iteration When Actions Are Functions

Abstract

Offline reinforcement learning seeks optimal decision rules from previously collected data. In some applications, a decision can be an entire function, such as a fluence map in radiation therapy or a smooth movement trajectory in robotics. In this paper, we study the finite-sample theory for fitted Q-iteration (FQI) with functional actions in a discounted infinite-horizon setting. Three major difficulties arise in this setting: first, the absence of a Lebesgue probability density for functional actions complicates coverage descriptions; second, conventional coverage requirements can be restrictive; and third, the large functional action space makes greedy optimization in FQI challenging. To address these difficulties, we study smoothness-regularized policy search under a critic-relative coverage condition. This condition measures how well logged data distinguish relevant action-value differences without requiring an action density. Our main theorem gives finite-sample guarantees for learned-policy regret relative to the best value within a fixed smooth class of functional-action policies. The results allow trajectory lengths to be either bounded or growing and Q-functions to be fitted by either functional-input kernel ridge regression or adaptive functional neural networks. For a few examples, we can obtain polynomially decaying regret bounds in the number of logged transitions, up to logarithmic factors, with logarithmically many FQI iterations. Numerical experiments show gains of learned functional-action policies over constant-action policies and support our adoption of a critic-relative coverage condition.

Primary area:
learning theory
Keywords:
Offline reinforcement learning, Fitted Q-iteration, Functional actions, Functional data analysis, Finite-sample guarantees, Smoothness regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wJ62OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This is a theoretical paper on fitted Q-iterations (FQI), an offline RL method to learn the optimal policy. Finite sample guarantees for the expected reward J(π^)J(\hat{\pi}) of the learned policy π^\hat{\pi} exist in literature for actions which are finite dimensional, but this paper treats the infinite dimensional case where the action is a function.

It identifies a key challenge to obtaining such a guarantee : classical literature requires a coverage condition linking the behaviour distribution (from which the rollout data is sampled) to policy-selected distribution. The paper shows that this condition is too restrictive for functional actions through a constructed example (Example 1, section 3.2) and also empirical evidence (Figure 1 b, Section 5 on Offline Pendulum Control). To remedy this, it introduces an alternative condition for functional actions (Assumption 2). Under this assumption, the paper transfers control of critic error from the logged state-action distribution to actions selected during policy improvement (Proposition 1).

A second challenge is that a large functional functional action space can make the argmax step of FQI computationally infeasible. To remedy this, the paper adopts the objective of FQI to continuous actions and regularizes it using an RKHS norm on the policy (Algorithm 1).

Finally, the paper bounds the finite sample error J∗−J(π^M)J^* - J(\hat{\pi}_M) in Theorem 1. For the examples in section 4, the paper computes the asymptotic dependence of the error on various quantities, notably, n=NTn = NT (number of independent subject x trajectory length). It is instructive to note that the asymptotics depend on a quantity which they call coverage exponent, which basically measures the rate of error transfer from logged state-action distribution to actions selected during policy improvement. The experiment in section 5 demonstrates the advantage of functional action (Figure 1a) and the restrictiveness of the usual coverage condition in literature (Figure 1b) in an offline pendulum control task.

Section 5 has several experiments comparing FT-GAE and Truncated GAE over practical RL tasks and ablation analyses.

Critical strengths
  1. The paper gives the first finite sample theory for the reward J(π^)J(\hat{\pi}) with functional actions under a critic-relative coverage condition. Although some previous works in literature can technically handle functional actions (see for instance the paper “A Measure-Theoretic Finite-Sample Theory for Adaptive-Data Fitted Q-Iteration”, 2026 by Haussmann et al.), this is the first paper with a detailed explanation of the coverage condition, and a reasonable modification for this setting.
Critical weaknesses
  1. The paper is a bit lacking in experimental demonstration. The only practical experiment performed is the offline pendulum control experiment, and even there, the main assumption which drives the theory (Assumption 2) is not clearly demonstrated.
Questions for authors
  1. Clarification: Could the authors clarify how the plot in Figure 1 c relates to the assumptions made in the paper? I believe it is supposed to be a motivation for the assumptions in Section 3.2, but I do not see how the precise assumptions are explained by the plot.
Reviewer_ZB1LOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work studies fitted Q-iterations (offline) for the case when the chosen actions come from a function space.

The sampling distribution of the behavior policy need not be the same as that associated with the policy which is learnt. Now given the data which has been sampled by a behavior policy, there could be multiple Q functions that "agree" with it (but these functions give different policies).

In that case which Q-function should be chosen? Hence this paper states a condition (Assumption 2-ii--informational direction) to resolve this. Suppose that we change the Q-function in such a way that the corresponding induced policy has changed. Then in this case, the data distribution (which has been induced by the behavior policy) should be able to declare that this change has occured. Key result is Theorem 1, which quantifies the error associated with value function. It can give sample complexity results, and several examples are worked out after Theorem 1.

Critical strengths

It seems not many papers were published on this topic, so it seems to address a problem which was ignored by mainstream RL community. The bound/ error decompsotion intgheorem 1 is neat, and gives sample complexity results in many scenarios such as functional-input kernel ridge regression, adaptive functional neural networks.

Critical weaknesses

I find this paper to be incremental. Proof techniques are either mostly from previous works, or require minor adaptations. Here are few examples: Proposition F.1 (Egger & Hoffman), Lemma B.1 ( Munos & Szepesvari), Matrix concentration from Tropp, Proposition H.3, Proposition I.1 (Nguyen-Teng et.al), Proposition I.2 (Bolte & Pauwels) Lemma L.1 (Yu 94) Lemma I.1 (Yao, Mueller & Wang), also Lemma I.1 and Appendix K we have Schmidt-Heiber.

Paper is not well-written, and it is not easy to get what problem is being studied, and the contributions. Key challenges are also not described properly, and manuscript is riddled with cryptic sentences.

Also, the assumptions are not very transparent, and restrictive. Especially, Assumption 2-ii--informational direction looks very restrictive. Few examples of MDPs (other than th elinear case) must be provided where these assumptions are true. Assumptions 3 and 4 seem more problematic. It seems that this is the material that must be proved, rather than assumed. Experiments are very few in number, and i do not see any comparison with competitior algorithms.

Questions for authors
  1. In any RL problem,you always thave to choose a function. For each time t (=1,2,..) you have to design a history (F_t) dependent map which generates a function belonging to \Delta(A)^S, where S is the state space, and \Delta(A) is the set of probability distributions on the action space A. Here F_t denotes the entire interaction history collected until time t. This consists of past observations, transitions, and decisions. How is the current problem different?

  2. Section 3: "However, standard action-density and neighborhood conditions are difficult to apply to functional-action FQI because there exists no Lebesgue probability density for functional actions and actions can easily lie outside the behavior support. In" -> can you explain this in more detail there itself? currently it is hard for a non-expert.

  3. In Introduction section currently we have: "A common challenge in offline FQI theory is to transfer control of critic error from the logged state–action distribution to actions selected during policy improvement. Establishing such a transfer typically requires a coverage condition linking the behavior distribution to policy-selected actions." This is not at all clear, especilly given that it is just the introduction section and you cannot expect a reader to understand too much technical details.

  4. Assumption 2-ii--informational direction looks very restrictive. It seems similar to a "minimal curvature condition" in optimization literature.

  5. Please explain in simple words what are Assumptions 3 and 4? currently it seems that these must be proved (should ideally come out of your analysis) rather than assumed.

  6. Theorem 1 is confusing. It is stating the error decomposition bound. As a corollary, can you please state the sample complexity?

Reviewer_wjfNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies offline reinforcement learning where each action is a function, such as a control curve. It proves finite-sample value guarantees for fitted Q-iteration with smooth function-valued policies. These rest on a coverage condition under which rough logged actions still inform smooth policies, and give rates for kernel and neural critics.

Critical strengths

This appears to be the first finite-sample value-gap analysis of offline fitted Q-iteration with function-valued actions

Critical weaknesses

The paper is largely difficult to follow. appendix is repetitive, does not use textbook proofs and unnecessary inaccessible impeding future research

e.g. The same concentration argument is repeated for each sampling regime (Lemma D.2, Prop. L.1; Prop. H.3, Props. L.2 and L.4), and the rate assembly is written out four times. Roughly 13–14 pages could be citations.

  • Key mechanism hidden-authors have missed an opportunity to explain their novelty

IMO, The central step, Prop. 1, is classical Hilbert-scale interpolation, a three-line argument, yet it appears only in App. F.

Questions for authors
  1. Can you give one nonlinear functional-action environment where all premises of Example 2 or 3 hold together (coverage, zero compatibility gap, localization)?

  2. How does your Hölder transfer condition differ from existing transfer exponents (Hanneke & Kpotufe, 2019; Nguyen-Tang & Arora, 2024)? What is the main new mathematical step beyond classical interpolation and FQI error propagation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.