acceptodds
Withdrawn from ICLR 2027

Anchored Anytime Value Learning for Long-Horizon Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon agent training calls for anytime value learning that decouples from trajectory-completion constraints. We introduce a barrier-synchronized streaming framework that optimizes in-flight trajectory chunks, eliminating straggler-induced idle time while preserving chunk-level policy freshness. Value learning over nonterminal chunks, however, corrupts along two axes: it inflates the magnitude variance of the bootstrap propagated within a trajectory, and it scrambles the ordering of values across concurrent trajectories. Mechanistic analysis shows: by unfolding the anisotropic observation manifold into a higher-resolution belief space, thinking widens task-level (between-task) separation far faster than it changes surface-level (within-task) variation, boosting the Fisher signal-to-noise ratio of the value representation. We then propose Anchored Anytime Value Learning (A2VL): (1)longitudinal magnitude anchoring shrinks noisy boundary bootstraps toward the low relative-variance belief-state value to limit temporal error propagation; and (2)transverse ordering anchoring imposes pairwise ordering constraints across concurrent trajectories using the well-separated belief-state ordering as a detached reference. Across WebShop, ALFWorld, and Crafter, A2VL stabilizes anytime training across chunk granularities, matching episode-synchronous task performance while substantially improving wall-clock throughput.

Primary area:
reinforcement learning
Keywords:
Agentic learning, Long-horizon, Reinforcement Learning, Sparse Reward

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TEVcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Anchored Anytime Value Learning (A2VL) to improve the training efficiency of long-horizon agents. The authors introduce a streaming framework that enables training on incomplete trajectories, reducing idle time caused by varying trajectory lengths. To address unstable value estimation in incomplete trajectories, A2VL introduces two anchoring mechanisms based on belief-state values. Experiments on WebShop, ALFWorld, and Crafter show that the proposed method improves training throughput while maintaining comparable task performance to episode-synchronous training.

Critical strengths
  1. The experimental results are strong, and the ablation studies of the method's components are helpful in validating the paper's claims.
Critical weaknesses
  1. The paper is difficult to understand due to unclear writing and presentation.
  • The caption of Figure 1 is too long, making it difficult to read. I suggest making the caption more concise and removing redundant explanations.
  • The figures are too small to read clearly. I suggest increasing the font size in the figures to match that of the main text, if possible, and minimizing the unnecessary whitespace between subfigures.
  • Some terms are introduced without sufficient explanation. For example, "belief state" is formally defined in Section 2.1, but it is already used in the Introduction. In addition, several technical terms in the abstract are difficult to understand without additional context. I suggest using more accessible terminology or providing brief explanations when these terms are first introduced.
  1. The motivation behind the main claim is unclear.

In standard RL implementations, PPO typically collects fixed-length rollouts by resetting environments when episodes terminate early [1]. Therefore, the issue described in the paper does not seem to arise in standard PPO implementations. Instead, it appears to stem from differences in the underlying framework. As a result, the paper's main claim regarding the necessity of "anytime value learning" is questionable and requires further justification. The authors should clarify why this issue cannot be addressed using standard fixed-length PPO rollouts with bootstrapping.

  1. The proposed trade-off interpolation lacks novelty.

The proposed "belief-state value" appears to be closely related to the Q-function, as the agent's actions are largely determined by the thinking tokens. From this perspective, the interpolation in Equation (5) does not appear to be fundamentally novel. Similar approaches have been studied in the context of managing the bias-variance trade-off in value estimation [2]. Therefore, the authors should discuss this missing related work and clarify how their approach differs from existing methods.

  1. Error bars are missing from Table 1.

The results in Table 1 are reported over three random seeds, but no error bars or measures of variability are provided. This makes it difficult to assess the statistical significance of the reported improvements. I suggest reporting standard deviations or confidence intervals. Moreover, it is unclear whether the improvements over ChainedGAE are statistically significant, as the gains are relatively small or inconsistent in some settings.

[1] Huang, Shengyi, et al. "The 37 Implementation Details of Proximal Policy Optimization." The ICLR Blog Track 2023 (2022).

[2] De Asis, Kristopher, et al. "Multi-step Reinforcement Learning: A Unifying Algorithm." Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, No. 1, 2018.

Questions for authors
  1. Is "anytime value learning" truly necessary? Could the issue identified in the paper be addressed using standard PPO implementations?

  2. What are the key differences between the proposed method and existing approaches, particularly those addressing the bias-variance trade-off in value estimation?

  3. Are the reported experimental improvements statistically significant across different random seeds?

Reviewer_pnc7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies value learning for long-horizon language agents under streaming, chunk-based reinforcement learning. A2VL builds upon ChainedGAE with periodic Monte Carlo (MC) outcome calibration and introduces two mechanisms that use values read after the reasoning block: longitudinal magnitude anchoring, which blends observation-position and post-thinking values at open chunk boundaries, and transverse ordering anchoring, which uses detached post-thinking value differences to supervise observation-position rankings. The method is evaluated on ALFWorld, WebShop, and Crafter across three model scales.

Critical strengths
  1. The paper addresses a consequential training problem. Learning from incomplete trajectory chunks can reduce waiting for long episodes to finish. The streaming framework and its empirical evaluation make this a practically relevant contribution.

  2. The proposed anchors provide a concrete use of post-thinking value estimates. Magnitude anchoring and ordering anchoring target different aspects of critic instability. The component and directionality ablations provide useful initial evidence about their roles.

Critical weaknesses
  1. The representation analysis does not establish the claimed advantages of the value estimates. Section 3 shows greater separation between hidden representations after thinking, but separation between task states does not directly establish more accurate scalar values, better value ordering, or lower bootstrap variance. Indeed, the reported within-task representation dispersion increases after thinking. A higher representation-level Fisher SNR therefore does not, by itself, justify the variance-reduction claim in Section 4.1. The variance of the hybrid estimate also depends on the scalar variances and covariance of its two components. Direct evaluation against repeated MC continuations at fixed policy checkpoints would provide stronger evidence about value error, ranking accuracy, and target variance.

  2. The experiments do not sufficiently isolate the necessity and reliability of the proposed hybrid construction. Although Equation (5) defines a post-thinking-only bootstrap through α = 0, this endpoint is not evaluated. It is therefore unclear whether the proposed mixture is necessary, or whether directly using the post-thinking value would suffice. Moreover, Table 1 does not show consistent improvements: on WebShop with Qwen3-4B, A2VL obtains a score/success rate of 82.0/64.8, compared with 83.4/67.2 for ChainedGAE; at 7B, its task score also decreases from 85.0 to 82.1. These results require explanation and qualification of the corresponding claims. Missing uncertainty estimates in Table 1 and Figure 3 further limit assessment of the incremental gains beyond the MC-calibrated baseline.

Questions for authors
  1. Direct value validation: Can you compare observation-position values, post-thinking values, and their mixture against MC return estimates obtained from repeated continuations at fixed policy checkpoints? Please distinguish prediction error and bootstrap-estimation variance from the stochastic variance of returns.

  2. Necessity of the hybrid: How does a post-thinking-only bootstrap (α = 0) perform against the observation-only baseline (α = 1) and intermediate mixtures? Holding the ordering loss fixed would help isolate the contribution of the mixture.

  3. MC calibration: Is periodic MC outcome calibration a new contribution or an adaptation of prior work? Please specify its frequency, mixing coefficient β, and eligibility rules, and report sensitivity to the calibration frequency and strength. For Figure 3’s existing ChainedGAE (w MC) baseline, are these settings identical to those used by A2VL?

  4. Nonuniform results and uncertainty: What explains the WebShop regressions in Table 1? Please report standard deviations or confidence intervals for the main results and uncertainty bands or individual runs for Figure 3. These would help determine whether the reported differences are reproducible.

  5. Seeds and aggregation: Although the paper reports three or at least three seeds, please specify the exact run count for each experiment, the seed identifiers, and the aggregation procedure. Please also define whether the ± values in Tables 2–3 denote standard deviation, standard error, or confidence intervals.

  6. Compute and token costs: Table 4 reports the throughput benefit of streaming execution, but does not isolate A2VL’s incremental cost over ChainedGAE. Can you report total generated tokens, reasoning-token usage, training GPU-hours or wall-clock time, and time to a matched task score for these two methods?

Reviewer_ENB6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies value learning for long-horizon language agents when updates operate on incomplete trajectory chunks. It combines barrier-synchronized streaming with chained GAE and periodic Monte Carlo calibration. A2VL then reuses critic predictions after the reasoning block to blend open-boundary bootstrap targets and supervise the pairwise ordering of observation-position values. Experiments cover ALFWorld, WebShop, and Crafter with three model sizes, together with component, chunk-size, directionality, and systems studies.

Critical strengths

S1. The paper addresses a practical bottleneck in long-horizon agent RL. Reusing reasoning-position critic predictions provides a concrete and inexpensive approach to stabilizing value learning without additional rollouts. The distinction between bootstrap magnitude and cross-trajectory ordering gives the method a clear design rationale.

S2. The experiments show useful empirical gains beyond the streaming baseline, particularly on Crafter, where the 7B score increases from 30.5 to 43.1 over ChainedGAE. The chunk-size and directionality studies also provide evidence about the conditions under which the method works.

Critical weaknesses

W1. The claimed quality-throughput advantage is not established under sufficiently transparent matched conditions. The episodic and streaming arms differ in credit assignment and permitted episode lengths (Section 5.1), while updates can consume different numbers of chunks (Appendix B.4). Table 4 reports only the final ten steps of streaming PPO, without actual interaction/token totals, a precise throughput denominator, or time to a fixed score. These differences make it difficult to separate scheduling gains from changes in training data, horizon, and optimization. This affects a central contribution of the paper.

W2. The justification for treating the reasoning-position value as a reliable anchor remains indirect. Figures 2 and 6 measure hidden-representation separation, which does not establish lower scalar value error or more accurate return ordering. Within-task dispersion actually increases in absolute terms, so the variance-reduction argument also requires information about estimation errors and their covariance. Figure 3c shows outcome-conditioned means for A2VL, but does not directly compare the calibration and ordering reliability of the two value positions. The relevant diagnostics defined in Appendix D would help establish the proposed mechanism.

Questions for authors

Q1. Could you provide actual environment steps, generated/trained tokens, fetch sizes, episode limits, and full wall-clock/time-to-score results for episodic PPO, streaming ChainedGAE, and A2VL? A matched-horizon comparison with comparable credit assignment would materially strengthen the efficiency claim. Please also define the denominator of tokens/sec in Table 4.

Q2. On held-out rollouts from fixed policy checkpoints, how do the two value positions

compare in calibration error, error variance/covariance, and pairwise inversion rate? Do the individual anchors change these diagnostics in the predicted directions? Such results would substantially improve my assessment of the mechanism.

Q3. What MC-calibration coefficient, frequency, eligibility rules, and reward scaling were used? Please clarify how completed trajectories spanning policy versions enter calibration and how the calibrated targets relate to actor advantages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.