- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Anchored Anytime Value Learning (A2VL) to improve the training efficiency of long-horizon agents. The authors introduce a streaming framework that enables training on incomplete trajectories, reducing idle time caused by varying trajectory lengths. To address unstable value estimation in incomplete trajectories, A2VL introduces two anchoring mechanisms based on belief-state values. Experiments on WebShop, ALFWorld, and Crafter show that the proposed method improves training throughput while maintaining comparable task performance to episode-synchronous training.
Critical strengths
- The experimental results are strong, and the ablation studies of the method's components are helpful in validating the paper's claims.
Critical weaknesses
- The paper is difficult to understand due to unclear writing and presentation.
- The caption of Figure 1 is too long, making it difficult to read. I suggest making the caption more concise and removing redundant explanations.
- The figures are too small to read clearly. I suggest increasing the font size in the figures to match that of the main text, if possible, and minimizing the unnecessary whitespace between subfigures.
- Some terms are introduced without sufficient explanation. For example, "belief state" is formally defined in Section 2.1, but it is already used in the Introduction. In addition, several technical terms in the abstract are difficult to understand without additional context. I suggest using more accessible terminology or providing brief explanations when these terms are first introduced.
- The motivation behind the main claim is unclear.
In standard RL implementations, PPO typically collects fixed-length rollouts by resetting environments when episodes terminate early [1]. Therefore, the issue described in the paper does not seem to arise in standard PPO implementations. Instead, it appears to stem from differences in the underlying framework. As a result, the paper's main claim regarding the necessity of "anytime value learning" is questionable and requires further justification. The authors should clarify why this issue cannot be addressed using standard fixed-length PPO rollouts with bootstrapping.
- The proposed trade-off interpolation lacks novelty.
The proposed "belief-state value" appears to be closely related to the Q-function, as the agent's actions are largely determined by the thinking tokens. From this perspective, the interpolation in Equation (5) does not appear to be fundamentally novel. Similar approaches have been studied in the context of managing the bias-variance trade-off in value estimation [2]. Therefore, the authors should discuss this missing related work and clarify how their approach differs from existing methods.
- Error bars are missing from Table 1.
The results in Table 1 are reported over three random seeds, but no error bars or measures of variability are provided. This makes it difficult to assess the statistical significance of the reported improvements. I suggest reporting standard deviations or confidence intervals. Moreover, it is unclear whether the improvements over ChainedGAE are statistically significant, as the gains are relatively small or inconsistent in some settings.
[1] Huang, Shengyi, et al. "The 37 Implementation Details of Proximal Policy Optimization." The ICLR Blog Track 2023 (2022).
[2] De Asis, Kristopher, et al. "Multi-step Reinforcement Learning: A Unifying Algorithm." Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, No. 1, 2018.
Questions for authors
-
Is "anytime value learning" truly necessary? Could the issue identified in the paper be addressed using standard PPO implementations?
-
What are the key differences between the proposed method and existing approaches, particularly those addressing the bias-variance trade-off in value estimation?
-
Are the reported experimental improvements statistically significant across different random seeds?