acceptodds
Under review as a conference paper at ICLR 2027

Playing with Fire: What Transfers When RL Trains a Language Agent?

Abstract

Modern language agents learn specialized skills like safety, mathematical reasoning, and coding with reinforcement learning (RL) and, at inference time, are deployed in diverse, complex harnesses often unseen during training. We study whether RL transfers these learned skills across diverse inference-time settings using two environments: Hanabi, a cooperative hidden-information card game, and K, a concise array-processing programming language, with scaffolds that vary the task knowledge and reasoning support supplied to the agent. We observe that RL gains are spiky: large under scaffolds that supply the task knowledge and small where it is removed. This recurs across model sizes, families, and environments. Supplying the task knowledge after RL helps at least as much as before (+1.75 before and +2.55 after in Hanabi), suggesting that RL teaches the agent to make better use of the knowledge supplied at inference, and not the missing knowledge itself. To investigate what transfers from RL on a hard, verifiable task, we construct FIREWORKS, a dataset of 10K Hanabi belief-state transitions with deterministic verification. Training QWEN3-4B-INSTRUCT-2507 on FIREWORKS elicits longer reasoning traces and improves performance on tasks outside the training objective: supported Hanabi gameplay (+4.3 points) and non-Hanabi reasoning (Pass@8 0.15 → 0.37). Although the resulting checkpoint shows no zero-shot improvement on downstream state tracking (0.117 against 0.127 for the base model), it is a better initialization for downstream RL: on Hanabi it reaches the same state-tracking reward in about 50 optimizer steps against 130 from the base model. RL thus improves how agents use the knowledge available to them and how quickly they learn next, extending its benefits beyond the training task.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qQPQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how RL-trained language agents generalize across different inference-time scaffolds. Using Hanabi and K-language program synthesis, the authors find that RL gains depend heavily on scaffold-provided task knowledge rather than knowledge internalized by the model.

The paper introduces FIREWORKS, a 10K-example Hanabi belief-state dataset, and shows that hard-task RL encourages longer reasoning and improves downstream learning efficiency. Experiments across Qwen3 and GPT-OSS models demonstrate limited cross-scaffold transfer, improved performance on unseen reasoning tasks, and faster convergence in subsequent RL training.

Critical strengths
  1. The paper addresses a relevant question about whether RL improvements genuinely transfer across inference-time scaffolds. Evaluating this phenomenon across two environments and multiple model scales provides useful empirical evidence.

  2. The FIREWORKS experiments offer an interesting observation: hard-task RL improves downstream learning efficiency despite providing little immediate improvement in task competence. This distinction is potentially valuable for understanding RL post-training.

Critical weaknesses
  1. The central claim that RL fails to internalize task knowledge is not convincingly established. Performance degradation under knowledge-omitting scaffolds could also reflect prompt distribution shifts or failures in knowledge retrieval.

  2. All RL experiments use a single training seed, making the reported confidence intervals insufficient to establish training stability.

  3. The paper lacks SFT and context-distillation baselines. Without these comparisons, it is unclear whether the observed limitations are specific to RL or simply artifacts of the training setup.

  4. Hanabi move rewards rely on LLM judges without an agreement analysis, raising concerns about reward reliability and potential evaluation bias.

  5. The claimed downstream learning efficiency ignores the additional 160-step FIREWORKS training cost. Moreover, the KPE experiments terminate before convergence due to policy entropy divergence, substantially weakening the conclusions about faster learning.

Questions for authors
  1. In Table 3, how do you establish that the reduced gains under WATSON and No-card reflect failed knowledge internalization rather than prompt sensitivity, especially given the template effects reported in Appendix M?

  2. All RL experiments use a single training seed (Appendix B). Are the key performance differences in Table 3 reproducible across independent training runs?

  3. Without matched SFT or context-distillation baselines, what evidence supports attributing the failure to internalize knowledge specifically to RL rather than the training data or objective?

  4. How were the LLM-generated Hanabi move rewards (R_MOVE) validated? What is the agreement between LLM judges and human evaluations?

  5. In Section 4.2, does the 50-vs-130-step advantage persist when including the 160 FIREWORKS training steps and longer rollouts? How is faster convergence justified when the KPE runs terminate before convergence?

2 comments

Reply to Reviewer qQPQ

We thank the reviewer for the careful review and for finding the scaffold-transfer question and the FIREWORKS result useful.

[W1 / Q1] Prompt Shift or Retrieval Failure. KPE experiments with and without operator card knowledge show RL gains far more with cards (Table 3), prompt shift can't explain this and the Hanabi knowledge probe shows no gain in rule recall after RL (Table 17, App. L). Which shows it's not a retrieval problem but the failure to inject knowledge. Our reply to Reviewer 5JTd (W2a / Q2) gives more details.

[New Rebuttal Experiment] We tested both alternatives directly:

  1. Two SFT models that store the rules play the same WATSON prompt well without the rules (5.58 and 5.96), while the RL models score lower on it and still gain +3.42 to +6.38 when the rules are added (RTable 5 in our reply to Reviewer Kv35, W1a / Q3). Again showing it's not prompt shift.
  2. When asked the rules directly, RL with the rules in 0% and 50% of training prompts changes Recall by +0.9 [-4.7, +6.6] and +1.2 [-5.6, +8.4], while rule-question SFT raises it by +17.5 [+3.1, +31.9].
  3. The template effects of Appendix M are real but a separate effect. Our knowledge comparison keeps the template fixed: within the same WATSON prompt, adding only the rules gains +3.42 to +6.38 after RL (RTable 5, reply to Reviewer Kv35), so this gain cannot come from the template. The template effect appears on top of it: the run trained without the rules scores 10.83 in its training scaffold and 4.33 in WATSON (RTable 5).

[W2 / Q2] Single Training Seed. Each RL result is one training run, which we state as a limitation (App. B, L905-906). The key differences repeat across three separately trained 4B RL runs and, in KPE, at three model sizes. Please see our reply to Reviewer Kv35 (W2).

[W3 / Q3] SFT and Context Distillation Baselines [New]. We added both these baselines, as Reviewer Kv35 also suggested; the full description and scores are in our reply to Reviewer Kv35 (W1a / Q3, RTable 5). As a summary, the rules limitation is specific to RL. After SFT the rules text adds nothing significant compared to after each RL run where it adds +3.42 to +6.38 score. Context distillation uses the same 3,586 MYCROFT states as RL.

[W4 / Q4] Validation of the Move Ratings. The move ratings follow the protocol of Ramesh et al. [35] (App. G.1). We did not run a human evaluation of the ratings (App. B, L918-920), because frontier models already play Hanabi at a high level and calibrating the ratings would need Hanabi experts, which is expensive. [35] already shows that training with an LLM judge brings models to the level of frontier models. We see the same effect: with the rules supplied, our trained 4B model reaches the level of the GPT-OSS-120B base model (13.38 under MYCROFT against 12 to 14, Table 2 and App. H). We therefore believe a separate human validation is not needed.

[W5 / Q5] Downstream Learning Efficiency. "Ignores the additional 160-step FIREWORKS training cost." We do not add the FIREWORKS steps because they train a different task. The comparison starts both models at the downstream task from the same zero-shot accuracy (0.117 against 0.127, L451-454) and asks which learns faster from there. QWEN-ST does, in both optimizer steps and generated tokens (RTable 3 in our reply to Reviewer 5JTd, W2b / Q1). The paper states the cost of the FIREWORKS stage in App. B (L922-925). "The KPE runs terminate before convergence." We compare both runs at equal steps. In KPE the two models also start at the same level; during training QWEN-ST improves faster and the gap grows (L1617-1619).

We hope the new experiments address your main concerns, and we are happy to engage further. If the concerns are resolved, we kindly ask you to consider updating the score.

Dear Reviewer qQPQ, thank you again for the review and valuable feedback. We addressed each of your five primary concerns with new analyses, all of which helped strengthen our paper: SFT and context-distillation baselines (under the same WATSON prompt, supplying the rules adds nothing after SFT but +3.42 to +6.38 after RL. Original-prompt numbers are in our reply to Reviewer Kv35); a direct rule-recall probe (RL: no detectable improvement, +0.9 and +1.2; rule-question SFT: +17.5); a quantification of the template effect you asked about (KPE, Table 19: a reworded, reordered prompt with the same knowledge keeps about 2/3 of the RL gain, while removing the knowledge leaves +1.1 to +2.2); and a token-based efficiency comparison (14.5% fewer generated tokens downstream). That comparison excludes the FIREWORKS stage itself (160 steps with rollouts about twice as long as the base model, see App. B) because our claim is a dissociation between zero-shot score and trainability, not an overall compute saving. We also explained our single-seed results and why we did not run human validation of the move ratings (App. B). We hope our rebuttal addressed your primary concerns and would really appreciate an appropriate increase in score if so.

Reviewer_Kv35OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what transfers when RL trains a language agent under one scaffold and is tested under others. It uses two environments: Hanabi and a K-language program synthesis task. RL gains are large when the prompt supplies task knowledge (rules, operator card) and small when it does not. The authors conclude that RL improves the use of supplied knowledge and does not internalize it. They also train on FIREWORKS, a 10K-example Hanabi belief-tracking dataset. This yields longer reasoning, some transfer to other tasks, and faster downstream RL.

Critical strengths

The question is useful. The controls are careful in places. The counterfactual operator-card test and the component ladder are good designs. The rule-recall probe and the think/non-think comparison add useful evidence. Results cover several model sizes, two model families, and two environments. The limitations section is honest. FIREWORKS may be a useful dataset.

Critical weaknesses
  1. The main claim is not well supported. The rules are in every training prompt. RL has no reason to store them. Short LoRA training with a reward unrelated to memorization will not inject knowledge. The result is therefore expected. No run hides the knowledge during training. There is no SFT or distillation baseline. "Scaffold shift" is also mostly knowledge removal, not a change of scaffold. A model that does not know the rules will play poorly without them. When the same knowledge appears in a different format, gains mostly transfer (Card, SHERLOCK, the permuted reference). This weakens the "spiky" story. The Δknow argument adds more confounds. RL improves general ability and format following, so a larger gain with knowledge is expected. Scores without knowledge sit near the floor. The paper does not normalize by ceiling. The claim "RL does not internalize knowledge" is too broad for this setting.
  2. The statistics are weak and the claims go beyond them. Every RL run uses a single seed. Many effects are small and inside the confidence intervals. Examples are the −2.29 SHERLOCK drop, the 4B affinity, and the GPT-OSS phase gains. Δknow differences have no confidence intervals. The GPT-OSS phases were chosen after seeing results on the same evaluation seeds, so those "Verdict" statements are only hypotheses. The K transfer is overstated. In Table 12, Full pass@8 has a CI containing zero and pass@1 is not significant. Only pass@4 is marginal. Saying QWEN-ST "improves on K" is too strong.
  3. The hard-task RL and trainability claims lack controls. The base model is a non-reasoning model. Any RL that induces long chain-of-thought may help PI-Logic and K. The only control is PI-Logic RL, which differs in many ways. The paper has no thinking-model initialization, no length-matched control, and no other RL checkpoint as an initialization. The link between length and gains is correlational. The Hanabi downstream task is very close to FIREWORKS. The speedup may come from format and skill overlap. The 50 vs 130 steps comparison ignores the 160 extra steps and longer rollouts. The reward schedule switches at the baseline's threshold crossing. The K runs stopped at step 80 when entropy diverged, and each used one seed.
Questions for authors
  1. Is there an SFT or context-distillation baseline? Without one, we cannot tell whether the failure to internalize is specific to RL.
  2. I find the question interesting. But the claim about "how RL plays" is too large for the current evidence. Would the authors narrow the claim to the specific setting where it holds (offline, single-step, LoRA, rules always in the prompt)? To support a claim at this level, some theory or internal-parameter analysis seems needed. For example, how does the policy gradient on the model's parameters differ when the prompt has the rules and when it does not? Do the updates differ in size, direction, or location?
  3. Can the authors randomly drop a fraction of the rule knowledge during RL training? This would show whether the model learns to rely less on the prompt, and whether the spike shrinks.
3 comments

Reply to Reviewer Kv35 (1/2)

We thank the reviewer for their careful review and valuable suggestions towards improving the paper. We ran both experiments the reviewer proposed: rules dropout during RL (Q3) and SFT and context-distillation baselines (Q1).

[W1a / Q3] Storing Knowledge & Rule Dropout Training. [New Rebuttal Experiment] We ran the suggested dropout training experiment by replicating our 4B MYCROFT experimental setup (Sec. 2.5; training recipe in App. G.2, Table 9) with the rules provided in 50% or 0% of training prompts. We also trained two SFT variants with the same LoRA recipe:

  1. Context distillation, where the base model with the rules block is the teacher and the same model without it is the student, on the same 3,586 MYCROFT training states used in the matched Qwen-4B/8B/32B RL runs.
  2. Rule-question SFT. We wrote 40 questions about the Hanabi rules (e.g. what happens on a misplay). First, the base model answers each question with the rules text in its prompt, several times per question, which gives 5,008 answers. Then we train the same base model to give those answers when it sees only the question, with no rules information. This training never uses a game state, and the 40 questions are different from the questions in our knowledge probe (App. L).

The experiments (RTable 5) show the result does not depend on the rules being in every prompt: RL with both 0% and 50% rules trained models shows similar behavior as the paper setup (100%) and base model. The models gain 3.42-6.38 with the rules compared to SFT models for which adding rules doesn't have any effect showcasing RL does not store rules.

For these runs, we added five lines to the WATSON prompt that define the move labels (Play k / Discard k refer to hand positions) and evaluated all six models with it to make sure the models are not losing due to misinterpreting hands. With the original WATSON prompt of the paper the rules gains are +1.75 (Base), +2.55 / +4.88 / +2.38 (RL 100% / 50% / 0%) and +0.21 / +0.12 (SFT), so the conclusion is the same.

RTable 5: Hanabi scores without and with the rules in the held-out WATSON scaffold, and without the rules in the training scaffold (MYCROFT).

ModelWATSONWATSON+RulesRules gainMYCROFT without rules
Base1.58 [1.29, 2.04]5.83 [5.21, 6.50]+4.25 [+3.46, +4.96]0.71 [0.33, 1.25]
RL, 100% rules (default)4.83 [4.21, 5.54]8.25 [7.71, 8.92]+3.42 [+2.50, +4.33]-
RL, 50% rules4.79 [4.12, 5.29]11.17 [10.62, 11.75]+6.38 [+5.71, +7.08]8.54 [7.38, 9.54]
RL, 0% rules4.33 [3.88, 5.04]9.08 [8.42, 9.75]+4.75 [+3.75, +5.62]10.83 [9.88, 11.62]
Context distillation (SFT)5.58 [4.92, 6.21]5.54 [5.00, 6.04]-0.04 [-0.96, +0.88]6.79 [5.75, 7.79]
Rule-question SFT5.96 [5.46, 6.46]5.42 [4.88, 6.12]-0.54 [-1.29, +0.42]5.21 [4.12, 6.38]

[New Rebuttal Experiment] Spiky training remains. The new RL models gain a lot in the trained MYCROFT scaffold, but their scores are less than half in the held-out WATSON scaffold (training-prompt affinity, Sec. 3.1). Whereas knowledge injection through SFT helps in both scaffolds.

[W1b] Short LoRA Training. We would like to highlight that the QWEN-HANABI+ training pipeline involves training on over 25K game states (Fig. 8; Table 9), after 10K FIREWORKS samples, and is not a short training run. [New Rebuttal Experiment] "Reward unrelated to memorization." We believe that training the model without the rules encourages it to learn the rules in order to score better. But from [W1a / Q3] we see that the model did not learn the rules in either the RL 0% or the 50% training run, whereas the SFT variants with the same LoRA training did.

[W1c / Q1] SFT and Context Distillation Baseline [New]. We thank the reviewer for this suggestion, which was also suggested by Reviewer qQPQ. The experiments (RTable 5) show that the need to provide rules remains only after RL. Both SFT models don't need rules. Without rules, SFT models with WATSON are as good as the base model with rules (5.58 and 5.96 against 5.83), and adding rules text gives no tangible gain: -0.04 and -0.54. In contrast, the RL trained models with added rules improve by +3.42 to +6.38.

[W1d] Scaffold Shift vs Knowledge. We agree and currently show that spike follows task knowledge (Abstract, L015-017; Sec. 3.2, L355-357). But we also show a second effect of the scaffold affinity:

  1. (Already present, Table 2) From QWEN-HANABI to QWEN-HANABI+, RL under MYCROFT lowers the SHERLOCK score by 2.29 [0.83, 4.00], although SHERLOCK supplies the same rules.
  2. (Already present, App. M, Table 18) The trained scaffold keeps an advantage of +3.33 [+1.62, +5.21] at 8B and +2.33 [+0.04, +4.33] at 32B.
  3. [New Rebuttal Experiment] We also show spiky training examples with the new 0% Rule RL model between Mycroft and Watson (RTable 5).

Reply to Reviewer Kv35 (2/2)

[W1e] Δknow\Delta_{\mathrm{know}}, General Ability & Format Following. If better general ability made any added text more useful, the rules gain should improve model performance whenever added. (Already present, Table 15) Models that already know the rules gain nothing significant from it, before or after RL: Qwen3-32B -0.21 and +0.17, GPT-OSS-120B +0.05 and +0.29. [New Rebuttal Experiment] Both SFT models play a similar level with or without rules added (RTable 5). Therefore the Δknow\Delta_{\mathrm{know}} checks whether the model still needs the rules, not general ability or format improvement.

[W1f] Score Ceiling Normalization [New]. As the reviewer suggested, we show gain as a fraction of the available task headroom. Headroom = maximum score (25 in Hanabi, 100 in KPE) minus the base model score, from Table 3. The spiky phenomena is not an artifact of scaling. RL still closes a larger share with the knowledge supplied: at 4B/8B/32B the share is 0.20/0.24/0.16 in Hanabi MYCROFT (rules) against 0.10/0.08/0.12 in WATSON (no rules), and 0.22/0.30/0.13 in KPE Full (card) against 0.01/0.01/0.03 in No-card. The low starting score also does not hide gains: without the rules, all three 4B RL runs still raise WATSON by +2.75 to +3.25 over the base model, with every interval excluding zero.

[W1g / Q2] Scope of RL claim. We will clarify the scope of the claim (offline RL with LoRA) in the revision, which we currently detail in our experimental setup (Sec. 2.5; App. G.2, Table 9). We note that rules are not always required in the prompt, which we show above in response to W1a / Q3.

[W2] Confidence Intervals. We now report an interval for every quantity the reviewer names: Two exclude zero, and the third is already reported in the paper as not detectable:

  1. The -2.29 SHERLOCK drop (Table 2). The interval is [-4.00, -0.83] and excludes zero.
  2. The 4B affinity (Table 18). +1.50 [-0.29, +3.25], no detectable difference at 4B, as Table 18 reports. At 8B and 32B the intervals exclude zero: +3.33 [+1.62, +5.21] and +2.33 [+0.04, +4.33].
  3. The GPT-OSS phase gains. In phases II and IV the MYCROFT gains exclude zero, [+0.58, +1.83] and [+0.88, +2.08], while the WATSON changes do not. The gain is in the training scaffold, which is the claim.

The effects our main claims rest on are large. After RL the rules still add +3.42 to +6.38 in Hanabi (RTable 5), and in KPE the knowledge gain grows after RL by +15.9 [+12.3, +19.6], +19.8 [+15.9, +23.8] and +6.3 [+3.4, +9.2] at 4B, 8B and 32B. These are the Δknow\Delta_{\mathrm{know}} intervals the reviewer asked for. We will add all of these intervals to the revision.

Remaining points.

  • "The GPT-OSS phases were chosen after seeing results." The paper states this (App. H, L1485-1486).
  • "QWEN-ST 'improves on K' is too strong." The paper's K result is trainability: by step 80 QWEN-ST leads by +0.084 [+0.058, +0.111] (L1617-1619).
  • "Every RL run uses a single seed." Each RL result is one training run, which we state as a limitation (App. B, L905-906). The three separately trained 4B RL runs in our reply to W1a / Q3 agree with each other.

[W3a] Thinking Model Experiment [New]. Following the reviewer's suggestion, we evaluated thinking Qwen3-4B on PI-Logic before and after our 4B MYCROFT RL run, which never trained on PI-Logic/FIREWORKS. PI-Logic Avg@8 rises from 0.129 to 0.158 (+0.029, 95% CI [+0.003, +0.056]). RL training on top of the thinking model also produces gains on PI-Logic. We will add this result to the revision.

[W3b] Length-matched control. Please see RTable 4 in our reply to Reviewer 5JTd (Q3).

[W3c] Format and skill overlap. Surprisingly, despite the overlap, QWEN-ST has no head start on Hanabi state tracking (0.117 against 0.127 for the base model, L451-454). Table 7 (App. E.3) provides more analyses. We see the speedup in KPE as well (L1620-1621).

[W3d] Fairness of Convergence Comparison. We now also compare generated tokens: QWEN-ST reaches the same reward with 230.3M tokens against 269.4M for the base model (RTable 3 in our reply to Reviewer 5JTd, W2b / Q1). The run from the base model with the joint reward from the start also needs about 130 steps (L457-458), so the difference comes from the initialization and not the reward schedule. The cost of the FIREWORKS stage and the step-80 stop in KPE are stated as limitations (App. B, L922-925). Nonetheless, in KPE the difference increases as the training progresses, and in Hanabi QWEN-ST starts with no head start.

We hope the results of the two new experiments address your main concerns. If it does, please consider updating the score.

Dear Reviewer Kv35, thank you again for the detailed review. Your two suggested experiments helped strengthen our paper and the clarity of our contributions: with rules absent from 50% or 100% of RL training prompts, adding them at test time still gains +6.38 and +4.75 Hanabi score in held-out WATSON (+4.88 and +2.38 with the original WATSON prompt), whereas context-distillation and rule-question SFT models gain nothing (-0.04 and -0.54; +0.21 and +0.12 with the original prompt). On your point that the same knowledge in a different format mostly transfers: in KPE (Tables 3 and 19), a reworded & reordered prompt with the same knowledge keeps about 2/3 of the RL gain (+10.5/+14.1/+5.9 vs +17.0/+21.0/+8.5 at 4B/8B/32B), while removing the knowledge leaves only +1.1 to +2.2. So the template costs about 1/3 of the gain and knowledge removal most of it. We also reported intervals for the examples you listed and will clarify the claim's scope (offline RL with LoRA) in the revision. We hope these changes addressed your main concerns and would really appreciate and appropriate increase in score if so.

Reviewer_5JTdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what capabilities transfer when language agents are trained with reinforcement learning (RL) and subsequently evaluated under different inference-time scaffolds. Using two environments—Hanabi, a cooperative hidden-information game, and K programming—the authors systematically vary the task knowledge and reasoning support provided by the scaffold. They find that RL improvements are often substantially larger when the inference scaffold supplies relevant task knowledge, suggesting that RL primarily improves the model's ability to use available knowledge rather than necessarily internalizing it.

The paper also introduces FIREWORKS, a dataset of 10,000 verifiable Hanabi belief-state transitions for RL training. Training on FIREWORKS increases reasoning length and yields improvements beyond the training objective, including Hanabi gameplay, PI-Logic reasoning (Pass@8 improving from 0.15 to 0.37), and modest zero-shot gains in K programming. The authors further demonstrate that FIREWORKS-trained models provide a better initialization for downstream RL: on Hanabi state tracking, the trained checkpoint reaches a target reward in approximately 50 optimizer steps versus 130 for the base model, while downstream K programming also shows faster improvement.

The contributions are primarily empirical, providing a controlled study of scaffold-dependent RL transfer and evidence that task-specific RL training can improve subsequent learning on other tasks, even when immediate performance gains are limited.

Critical strengths
  1. Controlled investigation of scaffold-dependent RL generalization: The paper addresses an important question in agentic RL: whether improvements obtained during post-training transfer across different inference-time scaffolds. By evaluating the same underlying tasks while varying the availability of task knowledge and reasoning support, the authors provide useful evidence that RL gains can depend strongly on the inference scaffold. The experiments across Hanabi and K programming, multiple model sizes, and additional prompt controls strengthen this finding.
  2. Evidence of cross-domain transfer and improved downstream RL initialization. Although FIREWORKS is specific to Hanabi belief-state tracking, the authors demonstrate improvements on unrelated tasks, including PI-Logic (Pass@8 increasing from 0.15 to 0.37) and K programming. More interestingly, FIREWORKS training improves subsequent RL convergence on K programming despite only modest zero-shot gains. This suggests that RL training on a specialized verifiable task may produce transferable benefits for subsequent learning, making it a potentially useful empirical finding beyond the original training domain.
Critical weaknesses
  1. Limited methodological novelty and generality of the RL contribution: The paper primarily presents an empirical study of scaffold-dependent transfer rather than a new RL method. FIREWORKS is constructed specifically for Hanabi belief-state tracking, making improvements in Hanabi gameplay relatively expected. While the PI-Logic and K programming results provide evidence of cross-domain transfer, the evaluation remains limited to a small set of structured, verifiable reasoning tasks. It is therefore unclear how broadly the findings generalize to other RL-trained agents and environments. Given the paper's primary focus on RL, this limits the significance of its contribution.

  2. Insufficient isolation of the mechanisms underlying transfer and downstream learning improvements: The authors argue that RL primarily improves the use of available task knowledge rather than internalizing that knowledge. However, the observed scaffold dependence does not fully distinguish a failure to acquire knowledge from difficulties retrieving or applying it under different prompts. Although the paper includes relevant controls, this mechanistic conclusion remains stronger than the evidence shown. Furthermore, FIREWORKS training substantially increases reasoning length, while improvements in downstream RL convergence are primarily reported in optimizer steps. Without comparisons normalized for generated tokens or computational cost, it is difficult to determine whether the faster convergence reflects more efficient learning or increased computation per training step. These limitations weaken the paper's central conclusions about what RL transfers.

Questions for authors
  1. Compute-normalized downstream RL efficiency (clarification): FIREWORKS training substantially increases reasoning length, while the downstream RL results primarily compare convergence in optimizer steps. Do the improvements in downstream learning efficiency persist when measured in total generated tokens, training FLOPs, or wall-clock time? This would help establish whether FIREWORKS provides a genuinely more efficient initialization rather than simply enabling greater computation per optimization step.

  2. Distinguishing knowledge acquisition from knowledge utilization (clarification): The paper concludes that RL primarily improves the model's ability to use scaffold-provided knowledge rather than internalizing that knowledge. Given the observed sensitivity to prompt format, how do the authors distinguish a failure to acquire task knowledge from a failure to retrieve or apply knowledge under different scaffolds? Could the authors clarify which existing experiments most directly support this distinction and whether alternative explanations remain consistent with the results?

  3. Robustness of cross-domain transfer (clarification and additional analysis): The PI-Logic results show substantial improvements after FIREWORKS training, whereas zero-shot improvements on K programming are comparatively modest. To what extent do these cross-domain gains persist when inference-time reasoning budgets are matched between the base and FIREWORKS-trained models? This would materially affect whether the observed transfer supports improved reasoning capabilities beyond increased inference-time computation.

4 comments

Reply to Reviewer 5JTd (1/2)

We thank the reviewer for their time and feedback. Below we address questions and comments raised in the review. We refer to weaknesses by W, questions by Q and strengths by S and tables by RTable.

In brief:

  • Tokens (W2b / Q1): downstream RL from QWEN-ST converges with fewer generated tokens (-14.5%) as well as fewer optimizer steps (-55.6%).
  • Matched budget (Q3): the PI-Logic Pass@8 gain holds at every token cap (+0.06 to +0.22).
  • Internalizing knowledge (W2a / Q2): under the same prompt, SFT models store the rules and RL models do not.

[W1a] Novelty. In the current era of continuous releases of RL post-trained LLM agents, we strongly believe that careful and detailed empirical studies of how claimed post-training gains transfer to held-out scaffolds are a valuable scientific contribution to the community. This work's novelty lies primarily in our findings, which, to our knowledge, have not previously been reported:

Spiky Training. RL gains are consistent only if the inference-time prompt matches the training prompt (Sec 3.1). Even though both Mycroft and Sherlock supply the same rules in different templates, training on Mycroft transfers far less to Sherlock. We show this with two separate post-trained Qwen3-4B models, i.e. non-thinking (Table 2) and thinking variants (Tab 11).

RTable 1: Change in Hanabi score after RL under MYCROFT, in the training scaffold (MYCROFT) and in SHERLOCK.

Model trained with htrain=h_{\mathrm{train}}= MycroftΔRL(htrain)\Delta_{\mathrm{RL}}(h_{\mathrm{train}}) (Mycroft)ΔRL(htest)\Delta_{\mathrm{RL}}(h_{\mathrm{test}}) (Sherlock)
4B Non-Thinking (QWEN-HANABI to QWEN-HANABI+, Tab 2)+0.55, 95% CI [-0.75, +1.75]-2.29, 95% CI [-4.00, -0.83]
4B Thinking (Qwen3-4B, Tab 11)+3.67, 95% CI [+2.50, +4.83]-1.12, 95% CI [-2.50, +0.71]

[New Rebuttal Experiment] To further empirically validate this finding: we train Qwen3-4B (thinking) on htrainh_{\mathrm{train}} MYCROFT without supplying rules in the training prompt, and evaluate under htrainh_{\mathrm{train}} MYCROFT and htesth_{\mathrm{test}} WATSON both with and without supplying rules in the prompt during inference. We find that under htrainh_{\mathrm{train}} MYCROFT, supplying rules makes no difference (-0.04), while supplying rules is still crucial in held-out htesth_{\mathrm{test}} WATSON (+4.75).

RTable 2: Hanabi score of the model trained without rules, with and without rules at inference.

Scaffold hhWithout rulesWith rulesDifference
MYCROFT (htrainh_{\mathrm{train}})10.8310.79-0.04 [-0.96, +0.92]
WATSON (htesth_{\mathrm{test}})4.339.08+4.75 [+3.75, +5.62]

"Trainability" or Downstream RL Convergence. In the original submission, we showed that QWEN-ST showed no improvement over the base model on belief state tracking (L451-454, Sec 4.2). From this starting point, RL training on QWEN-ST converges to RST=0.6R_{ST}=0.6 in 59 optimizer steps compared to 133 for the base model.

[New Rebuttal Analysis] As the reviewer suggested, we also show that RL training converges to RST=0.6R_{ST}=0.6 with nearly 40M fewer generated tokens than the base model (see response to W2b / Q1 below).

[W1b] Generality. We highlight the generality of our findings across four model sizes, two model families and two environments requiring different and complementary skills (e.g., state tracking & cooperative reasoning for Hanabi and coding & analytical reasoning for KPE). In this work, primarily due to compute resource constraints, we choose to study two environments in depth with controlled experiments.

[W1c] Cross-Domain Transfer. We highlight that training QWEN-ST on FIREWORKS, which rewards only state tracking and NOT move quality, improved gameplay when rules were provided (Mycroft and Sherlock, Tab 2) but did not significantly improve in-domain state-tracking on Hanabi (L451-454, Sec. 4.2), which we show in Table 7 (App. E3) is because QWEN-ST more often carries a card's previous belief forward when it should have been updated. As the reviewer points out in S2, training on FIREWORKS provides strong gains on unrelated downstream tasks on both performance (PI-Logic) and training convergence (Hanabi & KPE, Sec 4.2. Also see our response to W2b / Q1 below).

Reply to Reviewer 5JTd (2/2)

[W2a, Q2] Internalizing Knowledge. We thank the reviewer for raising this point, we agree that stressing this distinction is important. If RL had internalized task knowledge (Hanabi rules or KPE operator semantics) but failed to access or retrieve them during inference, the model should pass a simple task knowledge probe. We show that the RL-post trained model fails this knowledge probe across model sizes (4/8/32B) in both Hanabi (no change in recall, Table 17 in App. L) and KPE (89-110% card permutation invariance, Table 23, App O).

[New Rebuttal Experiment] SFT models with the same prompt and LoRA recipe play WATSON as well without rules as the base model with rules (5.58, 5.96 vs 5.83), so the WATSON prompt does not block retrieval (table in our reply to Reviewer Kv35, W1a / Q3).

[W2b, Q1] Convergence vs. Generated Tokens. We thank the reviewer for this suggestion; in addition to optimizer steps (Fig 3, Sec 4.2) we now also compare the number of generated tokens for downstream convergence. We find that downstream RL from QWEN-ST needs 39M (14.5%) fewer generated tokens than from the base model, while converging to the same reward in significantly fewer optimizer steps (-55.6%). The FIREWORKS stage (App. B, L922-925) is paid once and reused by both downstream tasks.

RTable 3: Downstream RL from the base model and from QWEN-ST: steps and generated tokens to reach RST=0.6R_{ST}=0.6 in Hanabi, and KPE solve rate.

EnvironmentMetricQwen3-4B Base InitQWEN-ST (FIREWORKS) Init
Hanabi (first to reach RST=0.6R_{ST}=0.6)Optimizer Steps ↓\downarrow13359 (-55.6%)
Hanabi (first to reach RST=0.6R_{ST}=0.6)Generated Tokens ↓\downarrow [NEW]269.4M230.3M (-14.5%)
KPE at equal generated tokens (211.7M)Solve Rate ↑\uparrow0.2220.278 (+25.2%)

[Q3] Cross-Domain Gains at Matched Inference Budget. In the original submission, we mentioned QWEN-ST improves PI-Logic Pass@8 from 0.15 to 0.37 over the base model (Table 4), and Sec. 4.1 notes that it also reasons longer.

[New Rebuttal Analysis] As the reviewer suggested, we match the inference budget. We re-score the Table 4 completions with a cap on tokens per answer. Any answer longer than the cap counts as wrong.

RTable 4: PI-Logic Pass@8 under a token cap; answers longer than the cap count as wrong.

Token cap1k2k4k8k16k
Qwen3-4B-Instruct-25070.020.050.100.150.15
QWEN-ST0.080.180.260.320.37
Difference [95% CI]+0.06 [+0.01, +0.12]+0.13 [+0.07, +0.20]+0.16 [+0.08, +0.25]+0.17 [+0.08, +0.26]+0.22 [+0.14, +0.31]

QWEN-ST solves more PI-Logic problems at every budget, and its correct answers are not longer than the Base model (3,678 against 3,802 tokens on average). Therefore the gain is not explained by a larger inference budget.

We will add all new analyses to the revision. If your concerns are addressed, we would kindly ask you to consider updating your score.

Reviewer_5JTdOpenReview ↗

Rebuttal Response

Thanks to the authors for the detailed rebuttal and additional experiments. The additional results address several of my original concerns:

  1. Downstream RL efficiency: The generated-token comparison provides evidence that the FIREWORKS initialization improves downstream learning efficiency beyond the reduction in optimizer steps alone. The 14.5% reduction in generated tokens on Hanabi and the improved KPE performance at matched generated-token budgets strengthen this result. However, the cost of the initial FIREWORKS training stage is not included in these comparisons.
  2. Cross-domain transfer: The PI-Logic results under matched inference-token caps provide stronger evidence that the improvements cannot be explained solely by longer generations.
  3. Knowledge Internalization vs. Performance after RL: The additional static knowledge experiments in Table 17 provide useful evidence regarding knowledge internalization. In particular, rule recall does not improve across any of the five RL-trained checkpoints. Meanwhile, Qwen3-4B improves application accuracy from 58.0% to 69.0% (+11.0 points) despite a decline in recall. These results support the interpretation that RL can improve the application of task knowledge without a corresponding increase in explicit rule recall. However, given the estimated standard error of approximately 7 percentage points per category, the results should be interpreted as showing no detectable recall improvement rather than definitively ruling out knowledge internalization.

I do retain two broader concerns about the paper's contribution.

  1. Novelty relative to prior work: The authors emphasize spiky training as one of the central findings. However, the observation that RL gains under one prompt template do not reliably transfer to another, even when both provide the same task rules, appears closely related to the training-prompt affinity phenomenon documented in Murray et al. [30], which the authors also acknowledge. The additional knowledge probes and counterfactual reference experiments are more distinctive and strengthen the empirical contribution.

  2. Generality and scope of the conclusions: My original review raised concerns about the generality of the findings beyond the structured, verifiable tasks considered in the paper. I appreciate the authors' clarification that the experiments cover multiple model sizes and families. However, this does not fully address the concern about diversity of environments and scaffold types. The introduction motivates the problem through language agents operating with complex inference-time scaffolds involving tool interfaces, context selection, and agent orchestration. In contrast, the empirical evaluation focuses on Hanabi and K programming, where scaffold variation is primarily implemented through prompt templates, supplied task knowledge, and reasoning support. While these controlled settings are valuable for isolating specific effects, they do not directly establish whether the findings extend to more complex tool-using agents. This concern also applies to the abstract's conclusion that "RL thus improves how agents use the knowledge available to them and how quickly they learn next, extending its benefits beyond the training task." The experiments provide evidence for these effects in the studied settings, but the statement reads as a broader conclusion about RL-trained language agents. In particular, the downstream trainability result is demonstrated using a specific FIREWORKS-trained initialization rather than across diverse RL training tasks. I would encourage the authors to qualify their conclusions to better reflect the experimental scope.

Overall, the rebuttal has strengthened my assessment of the paper's empirical soundness, particularly regarding knowledge utilization, cross-domain transfer, and downstream RL trainability. However, my original concerns about the incremental novelty of the central spiky-training observation and the generality of the RL contribution remain.

Reply to Reviewer 5JTd (follow-up)

We thank the reviewer for the careful reading, and we are glad that the new results and rebuttal thus far strengthened your assessment of the paper's empirical soundness. We reply briefly to each remaining point.

Cost of the FIREWORKS stage. RTable 3 compares the downstream stage, as the reviewer notes. The paper states the cost of the first stage (Already present, App. B, L923-925): "the FIREWORKS checkpoint cost 160 steps with rollouts about twice as long, so the result is a dissociation between zero-shot score and trainability, not a saving in compute."

Recall. We will use the reviewer's wording, "no detectable recall improvement", where Table 17 is discussed. [New Rebuttal Experiment] The same probe detects the rules when they are stored: rule-question SFT raises Recall by +17.5 [+3.1, +31.9] (reply to Reviewer qQPQ).

Novelty relative to [30]. RTable 1 in our first reply showed the template component of the effect, which the paper itself links to training-prompt affinity and [30] (L259-261). Spiky training is defined by task knowledge (Abstract, L015-017), and KPE separates the template effect from the knowledge effect (Already present, Tables 3 and 19): a reworded, reordered prompt with the same knowledge keeps about two thirds of the RL gain (+10.5 / +14.1 / +5.9 against +17.0 / +21.0 / +8.5 for 4B / 8B / 32B), while removing the knowledge leaves +1.1 to +2.2 points. So changing only the wording of the prompt costs about one third of the RL gain, which is the training-prompt affinity that [30] reports. Removing the knowledge costs most of it, and this is the effect that our paper adds. [30] also does not test whether RL stores the knowledge. Our paper also studies when RL helps and RL trainability.

Generality with Tool Use Agents [New Rebuttal Experiment]. We agree that evaluating a broader set of RL tasks strengthens our claim, so we now evaluate QWEN-ST on agentic tool-use benchmarks. We report QWEN-ST and the base model it was trained on (with the FIREWORKS dataset) on four public tool-use benchmarks: τ²-bench (customer service with tools; GPT-4.1 as the simulated user), SOPBench (function calling under standard operating procedures), AgentDojo (multi-step tool use; the 97 user tasks, without attacks) and BFCL (multi-turn function calling).

RTable 6: Task success (%) as scored by each benchmark, mean of 4 runs sampled at temperature 0.6, top-p 0.95, top-k 20. Intervals are 95% paired bootstraps over tasks.

BenchmarkTasksQwen3-4B-Instruct-2507QWEN-STDifference [95% CI]
τ²-bench27829.934.5+4.7 [+1.8, +7.6]
SOPBench83026.932.7+5.8 [+4.1, +7.4]
AgentDojo9757.060.6+3.6 [-1.0, +8.5]
BFCL multi-turn80020.720.9+0.2 [-1.1, +1.4]
BFCL multi-turn: long context (1 of 4 categories)20017.119.1+2.0 [+0.5, +3.6]

QWEN-ST is significantly ahead of the base model on τ²-bench and SOPBench, as well as on AgentDojo. On BFCL the two models are level overall, and QWEN-ST is ahead on long-context, one of its four multi-turn categories. QWEN-ST was trained only on FIREWORKS, which has no external tool use and does not cover any of these domains (customer service, standard operating procedures, workspace, banking, travel and messaging tools, and multi-turn function calling).

[New Rebuttal Experiment] On τ²-bench we also remove the domain policy and the two phrases that refer to it from the agent's prompt; tools, simulated user and scoring are unchanged.

RTable 7: τ²-bench task success (%), 278 tasks, mean of 4 runs; intervals as in RTable 6.

Agent promptQwen3-4B-Instruct-2507QWEN-STDifference [95% CI]
Policy in the prompt29.934.5+4.7 [+1.8, +7.6]
Policy removed31.132.9+1.8 [-1.1, +4.8]

With the policy in the prompt, QWEN-ST is significantly ahead of the base model; with the policy removed, the difference is not significant. We will add RTables 6 and 7 to the revision.

Scope of Conclusions. RTable 6 tests the abstract's claim that RL extends its benefits beyond the training task. The spiky-training result is a claim about the scaffolds we study, which supply knowledge through the prompt (Already present, App. B, L915-917). Following the reviewer's suggestion, we will qualify the last sentence of the abstract: "In the settings we study, RL thus improves how agents use the knowledge available to them and, from the FIREWORKS checkpoint, how quickly they learn next, extending its benefits beyond the training task." This also states that the trainability result is from one initialization. Separately, the zero-shot cross-domain gain is not limited to FIREWORKS [New Rebuttal Experiment]: a thinking Qwen3-4B trained with RL on Hanabi play improves PI-Logic Avg@8 by +0.029 [+0.003, +0.056] (reply to Reviewer Kv35, W3a).

We hope this addresses the remaining concerns. If so, we would kindly ask the reviewer to consider updating the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the ACs and all reviewers for their time, constructive feedback, questions, and suggestions, all of which helped to clarify and strengthen the contributions of our paper.

The reviewers highlighted several key strengths of our work:

  1. A Useful and Relevant Question. Reviewers found the question of whether RL gains transfer across inference-time scaffolds useful and relevant to RL post-training (Kv35, qQPQ).
  2. Careful Controls across Models and Environments. Reviewers appreciated the counterfactual operator card, the component ladder and the rule-recall probe (Kv35), and results across several model sizes, two model families and two environments (Kv35, qQPQ).
  3. FIREWORKS and Trainability. Reviewers found that RL on FIREWORKS speeds up later RL despite little zero-shot gain an interesting and potentially valuable observation (5JTd, qQPQ), and FIREWORKS a useful dataset (Kv35).

We summarize the changes in the revision. All new results are in the respective reviewer response, and all new training runs use Qwen3-4B.

  1. Rules-dropout RL (Kv35, qQPQ, 5JTd). Following Kv35's suggestion, we keep the rules block in only 50% or 0% of training prompts. After RL, the rules text still adds +6.38 and +4.75 points on WATSON (base +4.25), and Recall does not rise. This shows the rules being present in all the training samples was not the cause for the lack of internalization.
  2. SFT and context-distillation baselines (Kv35, qQPQ). With the same LoRA, both remove the need for the rules (rules gain -0.04 and -0.54, n.s.), and rule-question SFT raises Recall by +17.5 points. Our finding is specific to RL.
  3. Matched token budgets (5JTd, Kv35). QWEN-ST leads Instruct on PI-Logic at every cap from 1K to 16K tokens, and a thinking Qwen3-4B after Hanabi RL also improves (Avg@8 +0.029 [+0.003, +0.056]). This shows that our finding is not just limited to the FIREWORKS-trained checkpoints.
  4. Downstream RL in tokens (5JTd, qQPQ, Kv35). From QWEN-ST, Hanabi reaches reward 0.6 in 59 vs 133 steps and 230M vs 269M tokens. We present this as faster learning, not as a saving in total compute.
  5. Intervals and separate runs (Kv35, qQPQ). We add intervals for main-text quantities and show that three separately trained 4B runs agree.
  6. Generality: Tool-use agents (5JTd). We evaluate QWEN-ST, trained only on FIREWORKS, compared to the base model on popular agentic tool-use benchmarks. We see significant improvements on τ²-bench +4.7 [+1.8, +7.6] and SOPBench +5.8 [+4.1, +7.4]; AgentDojo (+3.6 [-1.0, +8.5]) and BFCL overall (+0.2 [-1.1, +1.4]) are not significant. On τ²-bench with the policy removed from the prompt, the gain is +1.8 [-1.1, +4.8] (not significant) vs. +4.7 with it.
  7. Template vs. knowledge (5JTd, Kv35, qQPQ). In KPE (Tables 3, 19), a reworded & reordered prompt with the same knowledge keeps about 2/3 of the RL gain (+10.5/+14.1/+5.9 vs +17.0/+21.0/+8.5 at 4B/8B/32B), while removing the knowledge leaves only +1.1 to +2.2.

We highlight that after the rebuttal, Reviewer 5JTd noted that our additional results strengthened their assessment of the paper's empirical soundness (knowledge utilization, cross-domain transfer, downstream trainability).

We address each reviewer's questions in detail in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.