- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how RL-trained language agents generalize across different inference-time scaffolds. Using Hanabi and K-language program synthesis, the authors find that RL gains depend heavily on scaffold-provided task knowledge rather than knowledge internalized by the model.
The paper introduces FIREWORKS, a 10K-example Hanabi belief-state dataset, and shows that hard-task RL encourages longer reasoning and improves downstream learning efficiency. Experiments across Qwen3 and GPT-OSS models demonstrate limited cross-scaffold transfer, improved performance on unseen reasoning tasks, and faster convergence in subsequent RL training.
Critical strengths
-
The paper addresses a relevant question about whether RL improvements genuinely transfer across inference-time scaffolds. Evaluating this phenomenon across two environments and multiple model scales provides useful empirical evidence.
-
The FIREWORKS experiments offer an interesting observation: hard-task RL improves downstream learning efficiency despite providing little immediate improvement in task competence. This distinction is potentially valuable for understanding RL post-training.
Critical weaknesses
-
The central claim that RL fails to internalize task knowledge is not convincingly established. Performance degradation under knowledge-omitting scaffolds could also reflect prompt distribution shifts or failures in knowledge retrieval.
-
All RL experiments use a single training seed, making the reported confidence intervals insufficient to establish training stability.
-
The paper lacks SFT and context-distillation baselines. Without these comparisons, it is unclear whether the observed limitations are specific to RL or simply artifacts of the training setup.
-
Hanabi move rewards rely on LLM judges without an agreement analysis, raising concerns about reward reliability and potential evaluation bias.
-
The claimed downstream learning efficiency ignores the additional 160-step FIREWORKS training cost. Moreover, the KPE experiments terminate before convergence due to policy entropy divergence, substantially weakening the conclusions about faster learning.
Questions for authors
-
In Table 3, how do you establish that the reduced gains under WATSON and No-card reflect failed knowledge internalization rather than prompt sensitivity, especially given the template effects reported in Appendix M?
-
All RL experiments use a single training seed (Appendix B). Are the key performance differences in Table 3 reproducible across independent training runs?
-
Without matched SFT or context-distillation baselines, what evidence supports attributing the failure to internalize knowledge specifically to RL rather than the training data or objective?
-
How were the LLM-generated Hanabi move rewards (R_MOVE) validated? What is the agreement between LLM judges and human evaluations?
-
In Section 4.2, does the 50-vs-130-step advantage persist when including the 160 FIREWORKS training steps and longer rollouts? How is faster convergence justified when the KPE runs terminate before convergence?
2 comments
Dear Reviewer qQPQ, thank you again for the review and valuable feedback. We addressed each of your five primary concerns with new analyses, all of which helped strengthen our paper: SFT and context-distillation baselines (under the same WATSON prompt, supplying the rules adds nothing after SFT but +3.42 to +6.38 after RL. Original-prompt numbers are in our reply to Reviewer Kv35); a direct rule-recall probe (RL: no detectable improvement, +0.9 and +1.2; rule-question SFT: +17.5); a quantification of the template effect you asked about (KPE, Table 19: a reworded, reordered prompt with the same knowledge keeps about 2/3 of the RL gain, while removing the knowledge leaves +1.1 to +2.2); and a token-based efficiency comparison (14.5% fewer generated tokens downstream). That comparison excludes the FIREWORKS stage itself (160 steps with rollouts about twice as long as the base model, see App. B) because our claim is a dissociation between zero-shot score and trainability, not an overall compute saving. We also explained our single-seed results and why we did not run human validation of the move ratings (App. B). We hope our rebuttal addressed your primary concerns and would really appreciate an appropriate increase in score if so.
Reply to Reviewer qQPQ
We thank the reviewer for the careful review and for finding the scaffold-transfer question and the FIREWORKS result useful.
[W1 / Q1] Prompt Shift or Retrieval Failure. KPE experiments with and without operator card knowledge show RL gains far more with cards (Table 3), prompt shift can't explain this and the Hanabi knowledge probe shows no gain in rule recall after RL (Table 17, App. L). Which shows it's not a retrieval problem but the failure to inject knowledge. Our reply to Reviewer 5JTd (W2a / Q2) gives more details.
[New Rebuttal Experiment] We tested both alternatives directly:
[W2 / Q2] Single Training Seed. Each RL result is one training run, which we state as a limitation (App. B, L905-906). The key differences repeat across three separately trained 4B RL runs and, in KPE, at three model sizes. Please see our reply to Reviewer Kv35 (W2).
[W3 / Q3] SFT and Context Distillation Baselines [New]. We added both these baselines, as Reviewer Kv35 also suggested; the full description and scores are in our reply to Reviewer Kv35 (W1a / Q3, RTable 5). As a summary, the rules limitation is specific to RL. After SFT the rules text adds nothing significant compared to after each RL run where it adds +3.42 to +6.38 score. Context distillation uses the same 3,586 MYCROFT states as RL.
[W4 / Q4] Validation of the Move Ratings. The move ratings follow the protocol of Ramesh et al. [35] (App. G.1). We did not run a human evaluation of the ratings (App. B, L918-920), because frontier models already play Hanabi at a high level and calibrating the ratings would need Hanabi experts, which is expensive. [35] already shows that training with an LLM judge brings models to the level of frontier models. We see the same effect: with the rules supplied, our trained 4B model reaches the level of the GPT-OSS-120B base model (13.38 under MYCROFT against 12 to 14, Table 2 and App. H). We therefore believe a separate human validation is not needed.
[W5 / Q5] Downstream Learning Efficiency. "Ignores the additional 160-step FIREWORKS training cost." We do not add the FIREWORKS steps because they train a different task. The comparison starts both models at the downstream task from the same zero-shot accuracy (0.117 against 0.127, L451-454) and asks which learns faster from there. QWEN-ST does, in both optimizer steps and generated tokens (RTable 3 in our reply to Reviewer 5JTd, W2b / Q1). The paper states the cost of the FIREWORKS stage in App. B (L922-925). "The KPE runs terminate before convergence." We compare both runs at equal steps. In KPE the two models also start at the same level; during training QWEN-ST improves faster and the gap grows (L1617-1619).
We hope the new experiments address your main concerns, and we are happy to engage further. If the concerns are resolved, we kindly ask you to consider updating the score.