- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper adds recoverable tool observations to Agent0 and studies whether enabling recall during RL improves performance when recall is disabled at evaluation. It reports math gains on Qwen models and increased answer coverage on LongBench-v2.
Critical strengths
Evaluating trained checkpoints with recall disabled is a useful design choice. The Qwen results provide evidence that the intervention affects the learned policy beyond immediate access to retrieved text.
Critical weaknesses
-
The experiment does not establish why recoverable context is needed. Roughly 14% of training examples are deliberately constructed to be impossible for the truncation baseline (§5.2, Appendix B). The comparison therefore gives one method access to useful training examples that the other cannot solve by construction. There is no larger-observation-budget control or experiment removing this synthetic subset; §9 explicitly acknowledges the former omission. The results cannot distinguish a benefit specific to recoverable context from the simpler benefit of making more training examples solvable.
-
The proposed learning-signal mechanism is not measured. The explanation rests on converting zero-advantage groups into useful training signal, but Table 1 reports task success, KL divergence, and response length rather than the frequency of groups with nonzero reward variance. Higher success does not establish this mechanism: uniformly successful groups also have zero advantages. Moreover, §6.3 treats final memory-stress performance as the amount of recovered signal, although the policy reflects updates accumulated throughout training. These proxies do not substantiate the paper’s central explanation for the downstream gains.
Questions for authors
-
Does the math improvement with recall disabled persist when the synthetic memory-stress subset is removed, or when the baseline can directly observe the same evidence?
-
Do training logs show that recall increases the proportion of groups with nonzero reward variance, separately for synthetic and ordinary math tasks?
Thank you for these questions.
Q: The experiment does not establish why recoverable context is needed.
A: In our log files, 200 of 1,400 training examples (14.29%) are memory-stress examples. These examples are structurally unsolvable under the 512-token destructive-truncation condition but potentially solvable with recall. Our evaluation with recall disabled establishes that the effect is learned during training rather than being solely an inference-time retrieval benefit. It does not, however, establish that recoverability itself is responsible for the improvement, because the synthetic subset remains present during training.
Q: The proposed learning-signal mechanism is not measured.
A: we agree. Table 1 establishes differences in training outcomes, but it does not directly measure the proposed learning-signal mechanism. In particular, higher task success does not imply nonzero within-group advantages because both uniformly unsuccessful and uniformly successful groups have zero reward variance.
Q: Does the math improvement with recall disabled persist when the synthetic memory-stress subset is removed, or when the baseline can directly observe the same evidence?
A: Our current experiments do not include the controls or logging needed to answer them conclusively. First, the reported math improvement with recall disabled is measured after training on the complete mixture: 600 GSM8K examples, 600 MATH examples, and 200 synthetic memory-stress examples. Thus, the synthetic subset constitutes 14.3% of training. We did not train a matched pair with this subset removed, nor did we train a truncation baseline with a sufficiently large observation budget to expose the same evidence. Consequently, the current results do not establish whether the residual math improvement is specific to recoverable context or simply results from making more training examples solvable.
Q: Do training logs show that recall increases the proportion of groups with nonzero reward variance, separately for synthetic and ordinary math tasks?
A: The current logs do not directly record the proportion of rollout groups with nonzero reward variance. They contain aggregate reward, validation performance, self-consistency, and recall-use statistics, but not the complete per-group reward vector joined with task type.