- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose a new approach that combines token-level GAE on executed trajectories with a micro branch. For this method, at high-entropy turns the policy samples extra candidate responses, a shared critic scores them without environment execution, and group-normalized scores give candidate advantages used in the actor update. The authors also evaluate this method using multiple benchmarks.
Critical strengths
The underlying research problem is practically useful. In many real settings, it cannot replay an episode, and this work may be able to provide some insights into this.
Critical weaknesses
- The main contribution of the micro mechanism is not supported. At a certain step, zero-advantage candidate rows already lift WebShop success, but the critic adds only a very little. It also shows a near-zero Spearman correlation, and the majority of groups map to one action.
- The motivating setting is never tested. The pitch is non-replayable or costly environments, yet every experiment uses resettable simulators. The method needs a trained critic plus extra generation on many candidate rows per step, but there is no compute-matched or wall-clock comparison with GiGPO-style baselines.
Questions for authors
It might be helpful for discussing why the progress loss adds little.
Relationship to Android Coach
CAGPO integrates cross-turn GAE on executed responses with group-normalized candidate advantages in one actor update. Android Coach shares the use of a critic trained on executed trajectories to supervise unexecuted alternatives at visited contexts; we will cite it and explain this relationship. The same critic supplies token values for temporal estimation and response-end values for candidate comparison. Candidates are allocated to selected high-entropy turns; critic targets come exclusively from executed trajectories.
The normalization difference can be stated precisely. For the same candidate scores, Android Coach's ACLOO advantage is
AiACLOO=G−1G(qi−qˉ),
whereas CAGPO uses
Aimicro=σq+ϵqi−qˉ.
Both preserve the same within-group ordering. Standardization changes the relative contribution of groups with different score spreads. Android Coach also uses PRM supervision and critic pretraining in its full method; CAGPO uses real-return regression with an optional directional regularizer. We will revise the related-work discussion and contribution statement around these specific differences. A matched empirical comparison with Android Coach is absent from the current evaluation.
Role of the real-trajectory actor loss
The macro actor loss directly uses the terminal reward already obtained from executed trajectories. Removing it would discard this observed-reward policy signal while retaining supervision through critic-predicted candidate scores. Our design supplements terminal-reward learning with candidate comparisons; the relevant ablation therefore removes the candidate branch, as in Table 3.
Directional versus difference-matching supervision
Sign-hinge imposes a weaker constraint than squared difference matching. It penalizes an opposite-signed predicted change while leaving its magnitude to the base value-regression loss. Difference matching additionally fits the magnitude of the adjacent-target difference. When the target difference is zero, sign-hinge contributes zero, whereas difference matching still encourages equal adjacent predictions.