acceptodds
Under review as a conference paper at ICLR 2027

Learning to Coach for Experiential Learning

Abstract

Language models can learn from experience, but raw solution trajectories are often too long and noisy to provide effective guidance. In this work, we propose , a framework that trains a dedicated LLM-as-a-Coach to extract actionable experiential knowledge from an actor model's previous trajectory. The actor remains frozen, while the LLM-as-a-Coach is trained to maximize a reward given by the correctness of the actor's guided response. We study two such rewards: a same-instance reward, which improves subsequent responses on the original problem, and a cross-instance reward, which elicits knowledge that transfers to other instances. Across mathematical reasoning and interactive text-games, L2C consistently outperforms self-refinement and an untrained LLM-as-a-Coach. Running experiential learning for more iterations further improves accuracy and uses additional inference compute more effectively than enlarging the actor's decoding budget. The trained LLM-as-a-Coach also transfers to out-of-distribution tasks and adapts its guidance to the actor it coaches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Experiential Learning, LLM-as-a-Coach

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5ZofOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents "Learning-2-coach" (L2C), a framework which proposes to train using RL a separate LLM (the coach) to offer advice to help with answering reasoning problems, or what they call experiential knowledge, to a responding LLM (the actor). To motivate the novelty of their approach, they contrast it to (1) RL finetuning of the actor itself, and (2) using a teacher model to give advice which is then used in the training updates of actor model. Both (1) and (2) require updating the weights of the actor model, while L2C requires updating the coach model while keeping actor fixed. They use as baselines (a) the untrained actor model without coach, (b) the untrained actor model with coach but no training. They have no baselines for (1) and (2). They evaluate the impact conditioning actors responses on the coaches advice on response accuracy, in two conditions: when the advice is for the current eval question, and when the advice is for a different eval question but same task. Their results show a significant increase in accuracy with trained coach for text game tasks and a not so significant (I question this) for math reasoning problems.

Critical strengths

The paper is very clearly written well scoped. An effort was made to consider out of distribution tasks for evaluation which I appreciate.

Critical weaknesses

I have two main critical concerns and a handful of smaller ones; because of these critical concerns and the publication expectation level of ICLR, I recommend reject. First, there is no clear motivation for the work presented in the paper, we are given a solution with no clear problem. The closest we have is the mention of comparison to methods (1) and (2) i list in my summary above, but then neither of these approaches are actually used or compared in practice as baselines in the paper. The authors suggest that this method means you can keep your actor LLM weights as is and thus avoid possible overfitting or knowledge loss that can happen with finetuning a model. But the solution they suggest is in effect to train a copy of the model instead on giving a more complete retrospection of the task... but when exactly should this be a better solution than say simply finetuning a copy of your model for a given task? This isn't clear at all to me. My instinct would say, well if your coach model could offer advice aiding with performance on other - OOD - tasks down the line too. The authors present results in line with this idea in the very short 3.6, but the results are incredibly misleading as the y-axes in Figure 6 are all different an in fact performance increase seems to be for 1% only (it is also not clear if this is a significant increase as we have no cross-validation across different training runs). Which brings me to my second major concern, many of the claims are misleading or too narrowly supported. First, The choice of "same-instance accuracy" as a metric is something I struggle to see the value of and find somewhat misleading given the baselines that were chosen, both of which do not involve training, since the coach effectively has access to information about the correct answer through the deterministic reward function, so we are comparing models without this info about the right answer to a model that in theory has access to information about the answer and could if it overfit during training in the same instance regime even give the answer in its knowledge experience... (RETRACTED BASED ON REBUTTAL) Next, Table 1 which compares the same instance accuracies across models fails to mention that the L2C results are the fraction of correct results after 10 iterations of learning on each question response pairs, while the baselines to my undestanding where applicable (self refinement) are not. Section 3.3 reports positive cross-instance training effects for 2 of the 3 datasets and claims that they " [position] L2C as a mechanism for distilling reusable, environment-general knowledge that generalizes to unseen instances"; they then point reader to appendix C for more results on math reasoning dataset, but these results in the appendix C show no transfer between training regimes, this seems important and the discussion about the nature of tasks should be in the main body of the text. Next, the comparison with doubling the decoding budget of the actor only baseline leads to equal accuracy to L2C with 2 iterations which in theory in terms of tokens also has a double decoding budget (actor bugdet + coach budget) as well as training, yet the way they write about it, it seems to suggest that L2C is somehow more economic? "Compared with a doubled 32k decoding budget on math (Table 2), a single guided response at K=2 already matches the extended-budget result for Qwen3-1.7B and surpasses it for Qwen3-4B". In 3.6, the paper states "L2C improves over the untrained LLM-as-a-Coach on all four benchmarks" but as mentioned above the figure for these results is very misleading and there is no sense for whether a 1% increase is in fact a true improvement across data presentations. Other issues: (RETRACTED BASED ON REBUTTAL)-line 185 DAPO-Math-17K (Yu et al. 2025a) -- There is no information about this dataset in the cited reference, can you please give more information on this dataset; is it proprietary?

-line 214, variable z is not clear at this point, all the previous description suggests that z in eq 9 should be x, only after do we realise that because of your two training regimes, z can be x or a different input question. Maybe clarify why we switch to z for the reader

-Table 1, make clear that L2C results are your best runs with 10 iterations.

-References Fix all the capitalizations and make sure years and venues are correct for all arxiv citations if since published.

UPDATE AFTER REBUTTALS: My comment on Table 1 (that L2C results are from 10 iterations) was incorrect: Table 1 reports K=2 and I confused values between Tables 1 and 2. My comment on DAPO-Math-17K was also incorrect; the dataset is described in the cited report and is public. I have raised my score from 1 to 2 accordingly. My remaining concerns detailed in my comments to the authors are: (1) the new closed-actor motivation does not match the open-weight experiments or the chosen baselines; (2) there is no analysis of whether the coach’s guidance contains answer information (also raised by e1Gd); (3) there is no statistical comparison of methods, since seeds were reported for L2C only and there are no CIs or tests for baselines (also raised by e1Gd); and (4) a full-instruction control is needed for the text games.

Questions for authors

-Did you do cross-validation across multiple data seed runs for training? if so, can you report standard error or deviation to give a sense of result significance?

-In the text game settings, do the baseline models have access to the task instructions? Their performance seems really low, yet L2C cross seems to learn to simply give task instructions in your appendix examples, so I am wondering if learning-2-coach is really "learning-the- instructions" given the correct answers.

12 comments
AuthorsOpenReview ↗

Response to Reviewer 5Zof (Part 1): Factual Clarifications

We thank the reviewer for their detailed comments and questions. We have carefully considered all feedback and provide our responses to each point below.

Before addressing the broader methodological concerns, we need to correct two factual inaccuracies in the review. These concern the experimental protocol and the dataset documentation, both of which can be checked directly in the manuscript and its cited reference. They are not changes introduced during the rebuttal.

  1. "Table 1, make clear that L2C results are your best runs with 10 iterations."

This is incorrect: Table 1 reports K=2, not ten iterations or best-of-ten selection. Section 3.2 (p. 5) explicitly states that Table 1 reports the same-instance accuracy Acc2same\mathrm{Acc}^{\mathrm{same}}_2 with the coach trained at K=2K=2. This consists of one initial actor response, one knowledge-extraction step, and one guided actor response. The reported metric scores that guided response; it does not select the best response from ten attempts. The K=10 experiments are presented separately in Section 3.4 and Appendix D.

The review's criticism that Table 1 gives L2C ten iterations while Self-Refinement receives only one revision therefore does not describe our experiment. In Table 1, both L2C and Self-Refinement use an initial actor response followed by one subsequent actor response; L2C additionally uses the coach to extract guidance. This distinction is already stated in the paper. It is not appropriate to characterize the table as misleading on the basis of a ten-iteration protocol that it does not use.

  1. "There is no information about this dataset in the cited reference, can you please give more information on this dataset; is it proprietary?"

The cited reference does explicitly describe DAPO-Math-17K, and the dataset is publicly available. Section 3.5, "Dataset Transformation," of Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale explains the dataset's sources, selection, and transformation into problems with integer answers, and explicitly names DAPO-Math-17K. Moreover, the official DAPO project page prominently links to the dataset, which is publicly released on BytedTsinghua-SIA/DAPO-Math-17k. Our Section 3.1 also identifies the dataset and describes the approximately 14K English-language problems used in our study.

Thus, the statement that the cited reference contains no information about the dataset is factually incorrect, and the dataset is not a proprietary resource available only to us.

AuthorsOpenReview ↗

Response to Reviewer 5Zof (Part 2): Methodological Questions

  1. "but when exactly should this be a better solution than say simply finetuning a copy of your model for a given task?"

Our motivation is to learn from an actor's trajectories when its weights are inaccessible or cannot be updated, as with a closed-source model accessed through an API. In this setting, fine-tuning the actor or a copy of it is not an available alternative. Nevertheless, we still want to extract useful experience from its trajectories to guide subsequent attempts. L2C addresses this problem by training a separate, accessible coach to produce natural-language guidance without accessing or modifying the actor's weights. Limited context windows provide an additional motivation: as interaction histories grow, retaining every full trajectory becomes impractical, whereas concise experiential knowledge can preserve useful information for later attempts.

This separation also motivates Appendix E, "Effect of Model Size." With the actor fixed at Qwen3-4B, we vary the coach across Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. The trained coaches achieve comparable guided accuracy under both K=2 and K=10, showing that a smaller model can serve effectively as the coach. Thus, L2C does not require training a same-sized copy of the actor: a lightweight coach can learn to extract useful experience while the actor remains unchanged.

We have clarified this motivation in the Introduction: actor fine-tuning may be unavailable when weights cannot be accessed or updated, while finite context windows motivate extracting concise experience.

  1. "since the coach effectively has access to information about the correct answer through the deterministic reward function"

The coach's training reward comes from the verifiable correctness of the frozen actor's response under its guidance. At no stage is the reference answer provided to the coach. The verifier scores the actor's guided response, and this reward is used to optimize the coach; receiving a correctness-based training signal is not the same as accessing the reference answer.

The same-instance setting evaluates whether experience extracted from an earlier attempt helps the actor perform better on that problem. Higher-quality experiential knowledge is rewarded through better downstream guided performance. Our goal is to use this verifiable reward to let the coach learn which experiential knowledge to retain, and how to customize the context for the actor. Rather than prescribing the content or wording of the guidance, L2C learns these choices from their effect on the actor's subsequent response.

  1. "but these results in the appendix C show no transfer between training regimes, this seems important and the discussion about the nature of tasks should be in the main body of the text."

We agree that cross-instance gains on DAPO math are much smaller than those on text-games. The original main text already acknowledged this explicitly. Section 3.3 stated: "On DAPO math with Qwen3-8B, cross-instance training yields only a small increase in transfer accuracy; Appendix C reports the results." The Introduction also qualified the transfer claim: "The gains are particularly large in text-games, where interaction trajectories expose latent environment rules. Training with the cross-instance reward further produces transferable knowledge when such rules are shared across instances."

Cross-instance rewards cannot create transferable information where the source experience provides none. They encourage the coach to extract and communicate information that is useful beyond the source instance. In text-games, different maps share latent rules, so discoveries from one trajectory can help on another map. In math, many intermediate results and solution steps are problem-specific; a source trajectory may therefore provide less relevant guidance for a different problem. This is a task-dependent explanation consistent with the results, not a claim that mathematical experience can never transfer.

Following your suggestion, we have expanded Section 3.3 in the revised main text to explain this distinction alongside the math result. The revised discussion explicitly states that cross-instance rewards promote the use of transferable information already present in a trajectory, and that the extent of transfer depends on the reusable structure shared with the target instances.

AuthorsOpenReview ↗

Response to Reviewer 5Zof (Part 3): Computational Cost, Reliability, and Notation

  1. "Next, the comparison with doubling the decoding budget of the actor only baseline leads to equal accuracy to L2C with 2 iterations which in theory in terms of tokens also has a double decoding budget (actor bugdet + coach budget) as well as training, yet the way they write about it, it seems to suggest that L2C is somehow more economic?"

We would like to distinguish actor decoding budget, additional coaching cost, and offline training cost. In the K=2 comparison in Table 2, L2C allows up to 16k tokens for the initial actor response and 16k for the guided response, totaling the same 32k actor-generation allowance as the extended-decoding baseline. Under this allowance, L2C achieves comparable accuracy at 1.7B and higher accuracy at 4B. Coach generation is additional, with an 8k-token cap; matching the actor's generation allowance does not imply matching total computation. The quoted result is an accuracy comparison under the stated decoding limits, not a measurement of lower end-to-end cost.

Training the coach is precisely the point of L2C: we invest in learning how to extract and communicate useful experience so that the frozen actor can perform better without updating its weights or simply increasing its decoding allowance. This training is performed offline, not repeated for each test problem, and the trained coach can be reused across subsequent interactions. Its training cost can therefore be amortized over deployment, although it should still be included in any total-cost comparison.

This separation is particularly relevant when the actor is a large model whose decoding is expensive. Rather than requiring an equally large coach, a smaller trainable model can extract useful guidance for the actor. Appendix E supports this design: with a fixed Qwen3-4B actor, a Qwen3-1.7B coach achieves guided accuracy comparable to larger coaches. This motivates using a lightweight coach to improve the use of an expensive actor's decoding budget.

  1. "In 3.6, the paper states "L2C improves over the untrained LLM-as-a-Coach on all four benchmarks" but as mentioned above the figure for these results is very misleading and there is no sense for whether a 1% increase is in fact a true improvement across data presentations."

We first report the additional training-seed results. For Qwen3-1.7B at step 100 and K=2, guided accuracy (%) is:

TaskSeed 0 (Table 1)Seed 42Seed 43Std. (pp)
FrozenLake65.465.965.70.25
Math67.867.768.00.15

The maximum differences across the three seeds are only 0.5 percentage points on FrozenLake and 0.3 on math, supporting stable and robust L2C gains in these settings.

Section 3.6 is based on extensive repeated evaluation, not a few isolated rollouts. As detailed in Appendix B, DAPO uses 1,000 problems with 16 rollouts each; AIME 2026 uses 30 problems with 128 rollouts each; text-games use 500 held-out environment seeds with eight playthroughs each; and IFEval uses 250 prompts with 16 coaching samples each. These correspond to 16,000, 3,840, 4,000, and 4,000 guided responses, respectively. Averaging repeated rollouts reduces stochastic generation noise, although these are repeated observations on fixed benchmark instances rather than additional independent test problems.

The OOD gains are also reported numerically in Table 4: +1.5 and +1.8 percentage points on AIME 2026, and +2.9 on FrozenLake and +1.8 on Sokoban, relative to the untrained coach. Thus, the conclusion is not based on the visual size of gaps between curves. Different y-axis ranges accommodate different baseline accuracies; the improvement should be assessed from the reported percentage-point differences, not relative panel heights.

  1. "line 214, variable z is not clear at this point, all the previous description suggests that z in eq 9 should be x, only after do we realise that because of your two training regimes, z can be x or a different input question. Maybe clarify why we switch to z for the reader"

We refer the reviewer to Section 2 (Method), where zz is explicitly introduced as the target instance before defining r(z,e)r(z,e). Here, xx is the source instance from which the experience is extracted, whereas zz is the instance on which that experience is evaluated. This distinction naturally gives the two reward settings: z=xz=x for the same-instance reward, and z∈Pz\in\mathcal{P}, with probes disjoint from the source, for the cross-instance reward. Equation (9) evaluates cross-instance accuracy and therefore uses the target variable zz consistently with this definition. Replacing it with xx would instead evaluate the experience on its source instance and change the meaning of the metric. The notation is thus deliberate and already established in the Method section, rather than an unexplained switch at Equation (9).

Reviewer_SdxYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learning to Coach (L2C), a framework that trains an LLM-based coach to extract actionable experiential knowledge from trajectories generated by a frozen actor. The coach is optimized using the downstream correctness of the actor's guided responses. The paper investigates same-instance and cross-instance rewards and evaluates the approach on mathematical reasoning and interactive text-games.

Critical strengths
  1. The paper is clearly written, and the overall framework is intuitive and easy to follow.
  2. Training a dedicated coach while keeping the actor frozen provides a modular approach to improving agent performance through experiential learning.
  3. The experimental results show substantial improvements over an untrained coach on interactive text-games. The investigation of transferable experiential knowledge through cross-instance rewards is also interesting.
Critical weaknesses
  1. Limited evaluation of long-horizon interactions. L2C operates at the trajectory level, extracting experiential knowledge after the actor completes an episode rather than intervening during individual environment interactions. Although the paper includes interactive text-games, the evaluated trajectories are relatively short, with at most five interaction turns. This leaves its applicability to long-horizon tool-use agents insufficiently explored, particularly when errors require correction before an episode is completed.

  2. Insufficient analysis of training efficiency under different reward formulations. The comparison between same-instance and cross-instance rewards does not adequately account for their different computational costs. According to Appendix A.3, cross-instance rewards require evaluating each candidate experience on multiple probe instances, resulting in substantially more actor rollouts per training step. However, Figure 3 compares the two approaches primarily in terms of training steps, making it difficult to assess their relative sample efficiency under comparable computational budgets.

  3. Limited comparisons with alternative approaches to improving coaching quality. The experimental comparisons mainly focus on self-refinement and an untrained LLM-as-a-Coach. Stronger alternatives, such as prompt-optimized coaches or existing learned feedback methods, are not evaluated. Consequently, the experiments do not fully establish whether the improvements specifically arise from the proposed reinforcement-learning-based coaching framework rather than more general improvements in feedback quality.

Questions for authors

How would L2C extend to long-horizon tool-use agents, where failures may need to be corrected before the entire trajectory is completed? Moreover, how does its effectiveness change as the number of environment interaction steps increases? An evaluation with longer trajectories or intermediate coaching would help clarify this limitation.

  1. Could the authors compare same-instance and cross-instance rewards under matched actor rollout or generated-token budgets? Since cross-instance training evaluates each candidate experience on multiple probe instances, such comparisons would clarify its additional computational cost and relative training efficiency. An ablation on the number of probe instances would also be informative.

  2. How does L2C compare with stronger coaching baselines, such as prompt-optimized coaches or existing learned feedback methods? Such comparisons would help establish the distinct benefits of explicitly training the coach with downstream actor rewards.

3 comments
AuthorsOpenReview ↗

Response to Reviewer SdxY (Part 1): Long-Horizon Extension and Rollout Budgets

We sincerely thank the reviewer for the detailed comments and constructive suggestions. Your questions have been particularly helpful in strengthening the rigor of our experiments. We respond to each point below.

  1. "How would L2C extend to long-horizon tool-use agents, where failures may need to be corrected before the entire trajectory is completed? Moreover, how does its effectiveness change as the number of environment interaction steps increases? An evaluation with longer trajectories or intermediate coaching would help clarify this limitation."

Thank you for highlighting this limitation. Our focus is learning to extract useful experience from prior trajectories for subsequent attempts or other instances, rather than learning when to intervene during an ongoing episode. Our experiments, capped at five interaction turns per episode, do not establish effectiveness on long-horizon tool-use tasks or within-episode correction. Extending this approach to support correction within an ongoing episode is an important next step.

A potential extension is to invoke the coach at each interaction step, conditioning its guidance on the trajectory prefix, and optimize the sequence of coaching outputs using the actor’s terminal task reward while keeping the actor frozen. This would enable within-episode intervention, although sparse-reward credit assignment and additional inference cost would need to be evaluated. We will try to implement and evaluate this extension in future work.

  1. "Could the authors compare same-instance and cross-instance rewards under matched actor rollout or generated-token budgets? Since cross-instance training evaluates each candidate experience on multiple probe instances, such comparisons would clarify its additional computational cost and relative training efficiency. An ablation on the number of probe instances would also be informative."

Thank you for this suggestion. We agree that equal training steps alone do not establish equal computational cost. In Appendix G (Figure 17), we add a higher-budget same-instance baseline: each fixed knowledge snippet is evaluated through eight independent guided playthroughs on its source instance, and their mean correctness is its training reward. With 64 sources and eight coach candidates per source, this uses 4,096 guided playthroughs per step, eight times the original allocation. The actor remains frozen and K=2; these are reward evaluations, not sequential coaching rounds or best-of-eight inference. Training uses 100 steps on FrozenLake and 200 on Sokoban.

Across the completed evaluations, same-instance accuracy improves, but the transfer gap relative to cross-instance reward training persists. This supports evaluating knowledge on other instances rather than relying on more source-instance samples alone. This is a reward-sampling-budget control, not a token- or wall-clock-matched efficiency claim; the cross-instance references retain their original evaluation pools.

We also added one- and four-probe training runs in Appendix F (Table 6), alongside the eight-probe main-text setting. Both metrics improve across the reported 1-, 4-, and 8-probe settings:

TaskCross acc., 1 probeCross acc., 4 probesCross acc., 8 probesSame acc., 1 probeSame acc., 4 probesSame acc., 8 probes
FrozenLake47.553.358.642.256.364.1
Sokoban17.344.047.125.050.053.1

Values are percentages at step 100 for FrozenLake and step 200 for Sokoban. This trend is consistent with more probes providing a more reliable estimate of the knowledge's transfer utility, at a higher rollout cost per step.

AuthorsOpenReview ↗

Response to Reviewer SdxY (Part 2): Stronger Coaching Baselines

  1. "How does L2C compare with stronger coaching baselines, such as prompt-optimized coaches or existing learned feedback methods? Such comparisons would help establish the distinct benefits of explicitly training the coach with downstream actor rewards."

Thank you for recommending a stronger coaching baseline. We added LLM-as-a-Coach + GEPA to Table 1, with its configuration in Appendix A.3. GEPA optimizes the coach's instruction using guided actor correctness on training-derived data, keeping actor prompts and all model weights fixed. The instruction is selected before testing, without test feedback. Both methods use the frozen-actor, initial-response / coaching / guided-response pipeline. Same-instance accuracy (%) is:

Model / taskGEPAL2C
Qwen3-1.7B / FrozenLake42.765.4
Qwen3-4B-Instruct-2507 / Sokoban19.726.6
Qwen3-1.7B / Math66.667.8
Qwen3-4B / Math74.778.6
Qwen3-8B / Math81.482.9

L2C with the original instruction achieves higher reported accuracy than the GEPA-optimized untrained coach in all five settings, supporting the value of learning coaching behavior through downstream actor rewards. Optimization budgets are not matched; these results compare achieved accuracy, not optimization efficiency.

Prompt optimization and L2C are compatible in principle: one could first optimize the instruction with GEPA and then train the coach with L2C.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer SdxY

Dear Reviewer SdxY,

We have posted our responses to your comments. As the discussion period is nearing its end, we would appreciate it if you could take a look when you have a chance. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for your time and consideration!

The authors

Reviewer_e1GdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Given that the original problem-solving trajectories of llms are often noisy, making it difficult to provide effective guidance, the paper proposes L2C, a framework that trains a dedicated llm as a Coach to extract actionable experiential knowledge from the historical trajectories of an Executor model. In the proposed setting, the Executor remains frozen, while the Coach is trained to maximize the reward defined by the correctness of the Executor's answers after receiving guidance. The paper investigates two types of rewards: same-instance reward and cross-instance reward. Experimental results demonstrate that L2C consistently outperforms both self-refinement and an untrained Coach model on two types of tasks: mathematical reasoning and interactive text games.

Critical strengths
  1. The authors proposes a novel guidance framework that trains only the coach while keeping the actor frozen, preserving the actor’s general capabilities while making experience explicit in natural language for inspection and reuse.

  2. The paper considers two types of rewards: same-instance and cross-instance rewards, with clear comparisons of their training curves.

  3. The paper provides extensive experimental results on test-time scaling and detailed ablation studies, including the effects of increasing k and varying the Coach model size on performance.

Critical weaknesses
  1. The reward only considers whether the actor's final answer is correct. Does the coach include the correct answer or key intermediate results in experience, potentially in a few-shot-like format that the Actor can simply follow? The paper should report the proportion of experience snippets that contain information related to the final answer.

  2. The baselines are insufficient, making the gains from increasing k unclear. self-refinement only performs one full-trajectory rewrite, with no BON, Self-Consistency, or control experiment involving the same k independent attempts without guidance.

Questions for authors
  1. can authors analyze whether the experience snippets contain information related to the final answer?

  2. can the authors report results across multiple random seeds with confidence intervals and include BoN and Self-Consistency results, addressing the concerns raised in the weaknesses?

2 comments
AuthorsOpenReview ↗

Response to Reviewer e1Gd

We sincerely thank the reviewer for the detailed comments and constructive suggestions. We have carefully considered all feedback and provide our responses to each point below.

  1. "can authors analyze whether the experience snippets contain information related to the final answer?"

Thank you for raising this question. We would like to clarify that our pipeline does not directly provide the reference answer to the coach. Instead, the coach is rewarded based on the correctness of the actor's guided response. Through reinforcement learning, the coach learns what experiential knowledge to extract from the actor's prior trajectory and how to provide guidance that improves the actor's subsequent performance.

  1. "can the authors report results across multiple random seeds with confidence intervals and include BoN and Self-Consistency results, addressing the concerns raised in the weaknesses?"

Thank you for this suggestion. We report the additional training-seed and test-time comparisons below.

Training seeds. For Qwen3-1.7B on FrozenLake and math, we report the original seed 0 results from Table 1 alongside the additional seeds 42 and 43. The same-instance guided accuracies (%) at step 100 and K=2 are:

TaskSeed 0 (Table 1)Seed 42Seed 43
FrozenLake65.465.965.7
Math67.867.768.0

The results vary only slightly across the three seeds, supporting stable and robust L2C gains in these two settings.

Multi-round Self-Refinement. Appendix D (Table 5) now includes one initial response followed by nine full-trajectory revisions. The reported accuracy of the tenth response, without selecting an earlier successful attempt, is:

Model / taskSelf-Refinement, K=10L2C, K=10
Qwen3-1.7B / FrozenLake22.073.0
Qwen3-4B-Instruct-2507 / Sokoban11.529.5
Qwen3-1.7B / Math67.571.0
Qwen3-4B / Math80.082.6
Qwen3-8B / Math81.885.6

L2C has higher reported final-response accuracy in all five settings, proving its advantage over Self-Refinement is not restricted to a one-revision baseline.

Self-Consistency (SC). Standard answer-level majority voting does not directly apply to multi-turn text games, where different valid action trajectories do not yield a common final answer to vote on. We therefore evaluate SC only on math, comparing ten independent answers with SC and applying SC to the ten responses from Self-Refinement or L2C. These are voting accuracies (%), distinct from the final-response metric above:

ModelIndependent SCSelf-Refinement + SCL2C + SC
Qwen3-1.7B75.071.873.3
Qwen3-4B82.482.885.2
Qwen3-8B85.985.887.1

We do not claim uniform superiority over SC: independent SC is stronger at 1.7B, while L2C + SC improves over it by 2.8 and 1.2 points at 4B and 8B. L2C and SC are complementary: coaching improves subsequent attempts, whereas SC aggregates candidate answers.

Best-of-N. Selection with a reference-answer verifier is an oracle upper bound, not a like-for-like deployment comparison with L2C's final response, which does not use such a selector. A learned or otherwise deployable selector would be a valid baseline with its costs included; we have not evaluated that comparison. We therefore do not dismiss BoN in general, but distinguish oracle selection from the SC and final-response results reported here.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer e1Gd

Dear Reviewer e1Gd,

We have posted our responses to your comments. As the discussion period is nearing its end, we would appreciate it if you could take a look when you have a chance. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for your time and consideration!

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.