- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents "Learning-2-coach" (L2C), a framework which proposes to train using RL a separate LLM (the coach) to offer advice to help with answering reasoning problems, or what they call experiential knowledge, to a responding LLM (the actor). To motivate the novelty of their approach, they contrast it to (1) RL finetuning of the actor itself, and (2) using a teacher model to give advice which is then used in the training updates of actor model. Both (1) and (2) require updating the weights of the actor model, while L2C requires updating the coach model while keeping actor fixed. They use as baselines (a) the untrained actor model without coach, (b) the untrained actor model with coach but no training. They have no baselines for (1) and (2). They evaluate the impact conditioning actors responses on the coaches advice on response accuracy, in two conditions: when the advice is for the current eval question, and when the advice is for a different eval question but same task. Their results show a significant increase in accuracy with trained coach for text game tasks and a not so significant (I question this) for math reasoning problems.
Critical strengths
The paper is very clearly written well scoped. An effort was made to consider out of distribution tasks for evaluation which I appreciate.
Critical weaknesses
I have two main critical concerns and a handful of smaller ones; because of these critical concerns and the publication expectation level of ICLR, I recommend reject. First, there is no clear motivation for the work presented in the paper, we are given a solution with no clear problem. The closest we have is the mention of comparison to methods (1) and (2) i list in my summary above, but then neither of these approaches are actually used or compared in practice as baselines in the paper. The authors suggest that this method means you can keep your actor LLM weights as is and thus avoid possible overfitting or knowledge loss that can happen with finetuning a model. But the solution they suggest is in effect to train a copy of the model instead on giving a more complete retrospection of the task... but when exactly should this be a better solution than say simply finetuning a copy of your model for a given task? This isn't clear at all to me. My instinct would say, well if your coach model could offer advice aiding with performance on other - OOD - tasks down the line too. The authors present results in line with this idea in the very short 3.6, but the results are incredibly misleading as the y-axes in Figure 6 are all different an in fact performance increase seems to be for 1% only (it is also not clear if this is a significant increase as we have no cross-validation across different training runs). Which brings me to my second major concern, many of the claims are misleading or too narrowly supported. First, The choice of "same-instance accuracy" as a metric is something I struggle to see the value of and find somewhat misleading given the baselines that were chosen, both of which do not involve training, since the coach effectively has access to information about the correct answer through the deterministic reward function, so we are comparing models without this info about the right answer to a model that in theory has access to information about the answer and could if it overfit during training in the same instance regime even give the answer in its knowledge experience... (RETRACTED BASED ON REBUTTAL) Next, Table 1 which compares the same instance accuracies across models fails to mention that the L2C results are the fraction of correct results after 10 iterations of learning on each question response pairs, while the baselines to my undestanding where applicable (self refinement) are not. Section 3.3 reports positive cross-instance training effects for 2 of the 3 datasets and claims that they " [position] L2C as a mechanism for distilling reusable, environment-general knowledge that generalizes to unseen instances"; they then point reader to appendix C for more results on math reasoning dataset, but these results in the appendix C show no transfer between training regimes, this seems important and the discussion about the nature of tasks should be in the main body of the text. Next, the comparison with doubling the decoding budget of the actor only baseline leads to equal accuracy to L2C with 2 iterations which in theory in terms of tokens also has a double decoding budget (actor bugdet + coach budget) as well as training, yet the way they write about it, it seems to suggest that L2C is somehow more economic? "Compared with a doubled 32k decoding budget on math (Table 2), a single guided response at K=2 already matches the extended-budget result for Qwen3-1.7B and surpasses it for Qwen3-4B". In 3.6, the paper states "L2C improves over the untrained LLM-as-a-Coach on all four benchmarks" but as mentioned above the figure for these results is very misleading and there is no sense for whether a 1% increase is in fact a true improvement across data presentations. Other issues: (RETRACTED BASED ON REBUTTAL)-line 185 DAPO-Math-17K (Yu et al. 2025a) -- There is no information about this dataset in the cited reference, can you please give more information on this dataset; is it proprietary?
-line 214, variable z is not clear at this point, all the previous description suggests that z in eq 9 should be x, only after do we realise that because of your two training regimes, z can be x or a different input question. Maybe clarify why we switch to z for the reader
-Table 1, make clear that L2C results are your best runs with 10 iterations.
-References Fix all the capitalizations and make sure years and venues are correct for all arxiv citations if since published.
UPDATE AFTER REBUTTALS: My comment on Table 1 (that L2C results are from 10 iterations) was incorrect: Table 1 reports K=2 and I confused values between Tables 1 and 2. My comment on DAPO-Math-17K was also incorrect; the dataset is described in the cited report and is public. I have raised my score from 1 to 2 accordingly. My remaining concerns detailed in my comments to the authors are: (1) the new closed-actor motivation does not match the open-weight experiments or the chosen baselines; (2) there is no analysis of whether the coach’s guidance contains answer information (also raised by e1Gd); (3) there is no statistical comparison of methods, since seeds were reported for L2C only and there are no CIs or tests for baselines (also raised by e1Gd); and (4) a full-instruction control is needed for the text games.
Questions for authors
-Did you do cross-validation across multiple data seed runs for training? if so, can you report standard error or deviation to give a sense of result significance?
-In the text game settings, do the baseline models have access to the task instructions? Their performance seems really low, yet L2C cross seems to learn to simply give task instructions in your appendix examples, so I am wondering if learning-2-coach is really "learning-the- instructions" given the correct answers.
12 comments
Response to Reviewer 5Zof (Part 2): Methodological Questions
- "but when exactly should this be a better solution than say simply finetuning a copy of your model for a given task?"
Our motivation is to learn from an actor's trajectories when its weights are inaccessible or cannot be updated, as with a closed-source model accessed through an API. In this setting, fine-tuning the actor or a copy of it is not an available alternative. Nevertheless, we still want to extract useful experience from its trajectories to guide subsequent attempts. L2C addresses this problem by training a separate, accessible coach to produce natural-language guidance without accessing or modifying the actor's weights. Limited context windows provide an additional motivation: as interaction histories grow, retaining every full trajectory becomes impractical, whereas concise experiential knowledge can preserve useful information for later attempts.
This separation also motivates Appendix E, "Effect of Model Size." With the actor fixed at Qwen3-4B, we vary the coach across Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. The trained coaches achieve comparable guided accuracy under both K=2 and K=10, showing that a smaller model can serve effectively as the coach. Thus, L2C does not require training a same-sized copy of the actor: a lightweight coach can learn to extract useful experience while the actor remains unchanged.
We have clarified this motivation in the Introduction: actor fine-tuning may be unavailable when weights cannot be accessed or updated, while finite context windows motivate extracting concise experience.
- "since the coach effectively has access to information about the correct answer through the deterministic reward function"
The coach's training reward comes from the verifiable correctness of the frozen actor's response under its guidance. At no stage is the reference answer provided to the coach. The verifier scores the actor's guided response, and this reward is used to optimize the coach; receiving a correctness-based training signal is not the same as accessing the reference answer.
The same-instance setting evaluates whether experience extracted from an earlier attempt helps the actor perform better on that problem. Higher-quality experiential knowledge is rewarded through better downstream guided performance. Our goal is to use this verifiable reward to let the coach learn which experiential knowledge to retain, and how to customize the context for the actor. Rather than prescribing the content or wording of the guidance, L2C learns these choices from their effect on the actor's subsequent response.
- "but these results in the appendix C show no transfer between training regimes, this seems important and the discussion about the nature of tasks should be in the main body of the text."
We agree that cross-instance gains on DAPO math are much smaller than those on text-games. The original main text already acknowledged this explicitly. Section 3.3 stated: "On DAPO math with Qwen3-8B, cross-instance training yields only a small increase in transfer accuracy; Appendix C reports the results." The Introduction also qualified the transfer claim: "The gains are particularly large in text-games, where interaction trajectories expose latent environment rules. Training with the cross-instance reward further produces transferable knowledge when such rules are shared across instances."
Cross-instance rewards cannot create transferable information where the source experience provides none. They encourage the coach to extract and communicate information that is useful beyond the source instance. In text-games, different maps share latent rules, so discoveries from one trajectory can help on another map. In math, many intermediate results and solution steps are problem-specific; a source trajectory may therefore provide less relevant guidance for a different problem. This is a task-dependent explanation consistent with the results, not a claim that mathematical experience can never transfer.
Following your suggestion, we have expanded Section 3.3 in the revised main text to explain this distinction alongside the math result. The revised discussion explicitly states that cross-instance rewards promote the use of transferable information already present in a trajectory, and that the extent of transfer depends on the reusable structure shared with the target instances.
Response to Reviewer 5Zof (Part 3): Computational Cost, Reliability, and Notation
- "Next, the comparison with doubling the decoding budget of the actor only baseline leads to equal accuracy to L2C with 2 iterations which in theory in terms of tokens also has a double decoding budget (actor bugdet + coach budget) as well as training, yet the way they write about it, it seems to suggest that L2C is somehow more economic?"
We would like to distinguish actor decoding budget, additional coaching cost, and offline training cost. In the K=2 comparison in Table 2, L2C allows up to 16k tokens for the initial actor response and 16k for the guided response, totaling the same 32k actor-generation allowance as the extended-decoding baseline. Under this allowance, L2C achieves comparable accuracy at 1.7B and higher accuracy at 4B. Coach generation is additional, with an 8k-token cap; matching the actor's generation allowance does not imply matching total computation. The quoted result is an accuracy comparison under the stated decoding limits, not a measurement of lower end-to-end cost.
Training the coach is precisely the point of L2C: we invest in learning how to extract and communicate useful experience so that the frozen actor can perform better without updating its weights or simply increasing its decoding allowance. This training is performed offline, not repeated for each test problem, and the trained coach can be reused across subsequent interactions. Its training cost can therefore be amortized over deployment, although it should still be included in any total-cost comparison.
This separation is particularly relevant when the actor is a large model whose decoding is expensive. Rather than requiring an equally large coach, a smaller trainable model can extract useful guidance for the actor. Appendix E supports this design: with a fixed Qwen3-4B actor, a Qwen3-1.7B coach achieves guided accuracy comparable to larger coaches. This motivates using a lightweight coach to improve the use of an expensive actor's decoding budget.
- "In 3.6, the paper states "L2C improves over the untrained LLM-as-a-Coach on all four benchmarks" but as mentioned above the figure for these results is very misleading and there is no sense for whether a 1% increase is in fact a true improvement across data presentations."
We first report the additional training-seed results. For Qwen3-1.7B at step 100 and K=2, guided accuracy (%) is:
| Task | Seed 0 (Table 1) | Seed 42 | Seed 43 | Std. (pp) |
|---|---|---|---|---|
| FrozenLake | 65.4 | 65.9 | 65.7 | 0.25 |
| Math | 67.8 | 67.7 | 68.0 | 0.15 |
The maximum differences across the three seeds are only 0.5 percentage points on FrozenLake and 0.3 on math, supporting stable and robust L2C gains in these settings.
Section 3.6 is based on extensive repeated evaluation, not a few isolated rollouts. As detailed in Appendix B, DAPO uses 1,000 problems with 16 rollouts each; AIME 2026 uses 30 problems with 128 rollouts each; text-games use 500 held-out environment seeds with eight playthroughs each; and IFEval uses 250 prompts with 16 coaching samples each. These correspond to 16,000, 3,840, 4,000, and 4,000 guided responses, respectively. Averaging repeated rollouts reduces stochastic generation noise, although these are repeated observations on fixed benchmark instances rather than additional independent test problems.
The OOD gains are also reported numerically in Table 4: +1.5 and +1.8 percentage points on AIME 2026, and +2.9 on FrozenLake and +1.8 on Sokoban, relative to the untrained coach. Thus, the conclusion is not based on the visual size of gaps between curves. Different y-axis ranges accommodate different baseline accuracies; the improvement should be assessed from the reported percentage-point differences, not relative panel heights.
- "line 214, variable z is not clear at this point, all the previous description suggests that z in eq 9 should be x, only after do we realise that because of your two training regimes, z can be x or a different input question. Maybe clarify why we switch to z for the reader"
We refer the reviewer to Section 2 (Method), where is explicitly introduced as the target instance before defining . Here, is the source instance from which the experience is extracted, whereas is the instance on which that experience is evaluated. This distinction naturally gives the two reward settings: for the same-instance reward, and , with probes disjoint from the source, for the cross-instance reward. Equation (9) evaluates cross-instance accuracy and therefore uses the target variable consistently with this definition. Replacing it with would instead evaluate the experience on its source instance and change the meaning of the metric. The notation is thus deliberate and already established in the Method section, rather than an unexplained switch at Equation (9).
Response to Reviewer 5Zof (Part 1): Factual Clarifications
We thank the reviewer for their detailed comments and questions. We have carefully considered all feedback and provide our responses to each point below.
Before addressing the broader methodological concerns, we need to correct two factual inaccuracies in the review. These concern the experimental protocol and the dataset documentation, both of which can be checked directly in the manuscript and its cited reference. They are not changes introduced during the rebuttal.
This is incorrect: Table 1 reports K=2, not ten iterations or best-of-ten selection. Section 3.2 (p. 5) explicitly states that Table 1 reports the same-instance accuracy Acc2same with the coach trained at K=2. This consists of one initial actor response, one knowledge-extraction step, and one guided actor response. The reported metric scores that guided response; it does not select the best response from ten attempts. The K=10 experiments are presented separately in Section 3.4 and Appendix D.
The review's criticism that Table 1 gives L2C ten iterations while Self-Refinement receives only one revision therefore does not describe our experiment. In Table 1, both L2C and Self-Refinement use an initial actor response followed by one subsequent actor response; L2C additionally uses the coach to extract guidance. This distinction is already stated in the paper. It is not appropriate to characterize the table as misleading on the basis of a ten-iteration protocol that it does not use.
The cited reference does explicitly describe DAPO-Math-17K, and the dataset is publicly available. Section 3.5, "Dataset Transformation," of Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale explains the dataset's sources, selection, and transformation into problems with integer answers, and explicitly names DAPO-Math-17K. Moreover, the official DAPO project page prominently links to the dataset, which is publicly released on BytedTsinghua-SIA/DAPO-Math-17k. Our Section 3.1 also identifies the dataset and describes the approximately 14K English-language problems used in our study.
Thus, the statement that the cited reference contains no information about the dataset is factually incorrect, and the dataset is not a proprietary resource available only to us.