acceptodds
Under review as a conference paper at ICLR 2027

Agent as Coach: Learning to Teach from Student Behavior

Abstract

Rubrics have become the basis for both evaluation and training feedback on tasks without a single correct answer, yet they describe what a good answer looks like rather than how a student should produce one. Experience extracted from evaluated student behavior can provide this actionable procedure. Expanding the pool of student behavior yields more useful experience, but a larger pool also creates a long-context challenge that lowers the quality of experience summarized from it in one pass. We introduce Agent as Coach, a coach that inspects an external behavior pool, reads selected attempts, and compresses observations before writing experience. Its acquisition and writing actions are jointly trained through the improvement of a frozen student, so the coach learns how to teach from how the student's behavior changes. We then internalize the experience into the student through training, so that the student learns to generate and use experience itself. Across multiple benchmarks with Qwen3-8B-Base and Qwen3.5-9B-Base as students, the complete pipeline consistently outperforms other rubric-based and experience-based methods, improving over the strongest of them by 4.9 and 3.8 points, respectively. Further experiments show that scored student behavior carries information beyond the rubric, that agentic reading avoids the decline of one-pass reading, and that the student learns from the experience itself, indicating that a coach can learn to teach from student behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent, rubric-based task

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4hZJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a coach to inspect, read, and compress a pool of scored student responses before producing experience. The coach is optimized using improvement in a frozen student. The student then learns through SFT and RL to generate and use experience without a coach at deployment. Experiments with two Qwen base students show improvements on three instruction-following benchmarks, with analyses of behavioral evidence, agentic reading, and experience internalization.

Critical strengths
  1. The move from rubric criteria to actionable experience is well motivated. Knowing the properties of a good answer is different from knowing how to produce it. Multiple student attempts provide richer coaching evidence than a single local correction.

  2. Tool-based reading addresses the stated context problem. Inspect, read, and compress keep the behavior pool outside the context and let the coach choose what to acquire and retain. Training on frozen-student improvement connects this reading process to its intended utility.

Critical weaknesses
  1. The evaluation supports instruction following more directly than general non-verifiable learning. IFEval and IFBench use rule-based checks, and InFoBench also evaluates decomposed requirements. These tasks test execution of explicit constraints well. They do not directly establish an answer to open-ended writing or dialogue where criteria may be incomplete or conflicting.

  2. The bottleneck may partly be compressed feedback rather than the absence of a learned coach. The method reduces rubric judgments to scalar scores from 1 to 10, then asks the coach to infer procedures from several scored answers. When the rubric already specifies criteria, direct per-criterion feedback could provide similar diagnostic information. Table 2 compares rubric-only input with scored behavior, but does not distinguish the value of behavioral evidence from the value of richer feedback.

  3. The coach objective targets a different student state from the final one. Stage II maximizes immediate gain in a frozen base student. Stages III and IV then change that student's capability and error distribution. Procedures useful to the base model may not be those the updated model most needs. Stage IV can revise self-written experience, but the analysis does not yet show how this mismatch is resolved.

  4. The Stage III retention rule can substantially change the training distribution. A coached answer is retained only if it exceeds the best unguided score, and J is capped at 10. Once an unguided pool contains a score-10 answer, that task cannot contribute a Stage III example. Larger K can increase such ceiling cases. Internalization therefore trains on a selected subset of task difficulties and constraint types rather than the original distribution.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4hZJ

Thank you for the specific comments on our design choices. We respond to each below.

Weakness 1 (scope beyond instruction following). We agree that the scope of our conclusions should be limited to instruction-following tasks that can be characterized by rubrics. We evaluate mainly on rule-checked benchmarks to keep evaluation independent of the training signal: training uses only non-verifiable OpenRubrics tasks, with all rewards coming from GPT-4o scores, whereas IFEval and IFBench are checked by rules and unaffected by judge preferences, which rules out the possibility that the student merely learns to please the training judge. On these two benchmarks, Agent as Coach leads the strongest baselines on both students (Table 1), and the IFBench constraints are unseen in common training data. On non-verifiable tasks themselves, the coach raises the student's judge score by 1.16 points on 500 held-out OpenRubrics tasks (Table 2).

Weakness 2 (scalar feedback as a bottleneck). You are right that the feedback in Table 2 is always a scalar score. Table 2 shows that behavioral evidence carries information beyond the rubric (a gain of 0.47 with the rubric alone and 1.16 with eight scored answers), but it does not compare feedback forms. Using scalar scores is a deliberate design choice: the judge is the stronger GPT-4o, and per-criterion judgments would bring its analysis of the answers into the coach's input, effectively letting a stronger model take part in writing the experience and making it hard to tell whether the gains come from the coach or from the judge. With scores as feedback, the judge only scores (Sec. 3.1), and the experience is inferred entirely from student behavior by a coach of the same size as the student. The coach also reads the rubric and the full answers, so it can determine whether each requirement is met by checking the answers against the rubric. Whether richer feedback brings further gains is listed as future work in our conclusion.

Weakness 3 (mismatch in student state). You are right that this mismatch exists; it arises in Stage IV. The Stage III experience is written by the coach for the base student, and the answers are generated by the same base student under that experience, so the two are consistent. In Stage IV the student has changed, but this stage no longer uses the coach; instead, the student writes its own experience. The judge scores only the answer, and this reward updates both the experience and the answer (Eq. 7), so the experience can adapt as the student changes. The comparison in Table 4 illustrates this: starting from the same Stage III student, if the experience is written by a frozen Stage III copy and RL trains only the answer, the result is 62.3 / 70.2; letting the experience update with the student gives 63.3 / 70.9. We trained this comparison independently with 3 random seeds (42, 24, 64) on 8B:

Seed (8B)Joint RL (Stage IV)Frozen-experience RLGain
Seed 42 (paper)63.362.3+1.0
Seed 2462.961.9+1.0
Seed 6463.162.4+0.7
Mean ± std63.1 ± 0.262.2 ± 0.3+0.9 ± 0.2

Joint RL outperforms frozen-experience RL under all 3 seeds, by 0.9 ± 0.2 on average. Continuing to train the coach for the updated student is a natural next step, and we will pursue it as future work.

Weakness 4 (retention rule). You are right that the retention rule filters out some tasks, including saturated ones. This is by design: when s+s^+ is already 10, the judge's scale cannot distinguish a better answer, so there is no improvement signal to learn from. At K = 8, 12.2% (610) of the 5,000 Stage III tasks are excluded due to saturation; for Agent as Coach, another 18.6% (930) are not retained because no guided answer exceeds s+s^+, and the remaining 3,460 tasks keep at least one answer. This filtering happens in Stage III: Stage IV uses 27,000 unfiltered RL tasks from the same source (Table 5), which likewise include easy tasks that the student can already answer perfectly, so the student still trains on the original task distribution in this stage.

Thank you for the authors’ response, which has addressed most of my concerns. I will keep my positive score.

Thank you for your positive assessment and for your constructive feedback throughout the review process. We are glad that our response addressed most of your concerns, and we will incorporate the discussed revisions into the final version of the paper.

Reviewer_pVbTOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agent as Coach trains a model to inspect a pool of scored student answers through tools, read selected attempts and rubrics, compress its observations, and write a short experience containing applicability, actions, pitfalls and checks. After a protocol warm-up, the coach is trained with GRPO using the improvement of a frozen student's answers as reward.

The student is then fine-tuned on experience-answer pairs whose guided answers outperform its best unaided attempts. A final RL stage trains the student to generate both experience and answer from the question alone, so deployment requires no external coach. Experiments use OpenRubrics and two students, Qwen3-8B-Base and Qwen3.5-9B-Base. Evaluation covers IFEval, IFBench and InFoBench, alongside analyses of evidence use, pool size and internalization.

Critical strengths
  1. The method produces consistent improvements in the independently deployed student. For the 8B model, IFEval reaches 73.3 compared with the best baseline's 66.9, and IFBench reaches 30.9 compared with 24.8. For the 9B model, IFBench reaches 32.4 compared with 26.9. The average improvements over the strongest baseline are 4.9 and 3.8 points. These are useful gains, and no stronger model writes the experience or answer targets: the stronger model serves as a judge. Removing the coach at deployment also makes the result more practical than a system that must repeat the full coaching procedure for every request.

  2. The analyses test the central learning claims directly. In Table 2, the value of eight scored answers beyond the rubric rises from 0.33 judge points for the prompted coach to 0.69 for the trained coach. Table 3 shows that the agentic coach remains effective on pools much larger than those used in training. With answer targets fixed, removing experience costs 2.0 points, and a length-matched generic plan does not recover the full benefit. These controls support the idea that the content of the learned procedure matters. The benefit is not explained solely by a planning prefix or by better answer targets.

Critical weaknesses
  1. The full-context comparison needs a better account of what changes during internalization. At K=8, the final Stage II coaching gains are close: 1.12 for the full-context coach and 1.16 for the agentic coach. Yet their 8B Stage III students score 55.8 and 61.4. The large-pool results establish an advantage of selective reading as K grows, but do not fully explain this difference at K=8. The filtering rule depends on whether a guided answer beats the best unaided answer, so the tails of the score distributions, the number of retained examples, or differences in experience format could matter. Reporting those quantities would sharpen the explanation. This is a limitation of the attribution, not evidence that the observed student gains are absent.

  2. The cost and uncertainty of the gains are underreported. The coaching stages involve substantial generation and judging, and the rubric-based baselines do not receive an explicitly matched budget. This leaves open how much additional accuracy could be obtained by spending the same resources on rejection sampling. The no-experience arm already performs strongly, making that comparison worth considering. Separately, repeated evaluation would help distinguish robust effects from sampling variation, particularly for redaction, explicit experience versus context distillation, and joint versus frozen-experience RL. InFoBench also shares its judge with training, so I give greater weight to IFEval and IFBench.

Questions for authors
  1. For the full-context and agentic coaches, how many Stage III examples survive filtering, and how do the guided-score margins above the best unaided answer differ? Are experience lengths and output formats comparable?
  2. Can you add repeated evaluation or confidence intervals for Table 1 and the fixed-target prefix controls? If training repeats are too costly, repeated decoding would still resolve part of the uncertainty.
  3. What are the total generation and judge budgets for the main methods, and what inference overhead does the student's experience block add at deployment?
  4. Relationship to critique-based learning. Section 4 already discusses Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning. Could you make the comparison more explicit and also discuss Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate and Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning?
4 comments
AuthorsOpenReview ↗

Response to Reviewer pVbT (1/2)

Thank you for your positive assessment. Below we provide the numbers for each of your questions.

Weakness 1 and Question 1 (attribution at K = 8). The two coaches differ in the upper tail of the score distribution, not in the mean or the experience format. The coaching gain measures the mean, whereas Stage III keeps only answers that exceed the student's best score s+s^+ (Sec. 2.3); so two coaches with similar mean gains (1.12 vs. 1.16) can produce very different Stage III data. The retention counts, margins above s+s^+, and experience lengths you asked about:

QuantityFull-context coachAgent as Coach
Stage III input tasks (before filtering)5,0005,000
Pool K / guided answers M8 / 88 / 8
Candidate guided answers (5,000 × 8)40,00040,000
Tasks with ≥1 retained answer2,8703,460
Retained answers6,9409,820
Mean margin above s+s^+1.211.38
Mean experience length (tokens)212236
Stage II final coaching gain1.121.16
Stage III question-only Avg55.861.4

Agent as Coach retains 41% more answers (9,820 vs. 6,940), covers 21% more tasks (3,460 vs. 2,870), and exceeds s+s^+ by a larger margin (1.38 vs. 1.21). Both coaches use the same four-field experience format, with similar lengths (236 vs. 212 tokens).

Weakness 2 and Questions 2–3 (cost and uncertainty). We address four items in turn: (i) budget and rejection sampling; (ii) deployment overhead; (iii) repeated evaluation; (iv) the shared judge.

(i) Budget and rejection sampling (Question 3). Generation and judge usage of the full pipeline:

Method (8B full pipeline)Coach generated tokensStudent generated tokensJudge callsQuestion-only Avg
Full-context coach140M1,540M2.5M58.4
Agent as Coach480M1,550M2.5M63.3

The two coaches use about the same number of student tokens and judge calls; they differ only in the coach's own generation. On rejection sampling, existing results offer two reference points. First, Stage III is itself filtered sampling: for each task, 8 unguided and 8 guided responses are sampled, and only those exceeding the best unguided score s+s^+ are kept. Guided responses are retained at a rate of 24.6% (9,820 / 40,000), whereas by exchangeability an additional unguided response exceeds s+s^+ with probability at most 1/9 (about 11%), so guided sampling is more than twice as efficient. Second, without a coach, RL applied directly to the student reaches 55.2 (answer-only RL) and 55.8 (self-written experience RL) (Table 4), versus 63.3 for Agent as Coach.

(ii) Deployment overhead (Question 3). The experience block written by the student averages 228 tokens (8B) and 241 tokens (9B), about a quarter of the output (912 and 987 tokens in total on average), far below the 4,096-token generation limit (App. A.2).

AuthorsOpenReview ↗

Response to Reviewer pVbT (2/2)

(iii) Repeated evaluation (Question 2). We decode each prompt 8 times at T = 1.0 and compute 95% confidence intervals with a prompt-paired bootstrap (Gain = score of the first minus the second). Main results in Table 1:

ComparisonScores in paperGain in paperMean gain over 8 decodingsPaired 95% CI
8B · Agent as Coach vs. strongest baseline (full-context coach)63.3 vs. 58.4+4.9+4.8[+3.5, +6.1]
9B · Agent as Coach vs. strongest baseline (EL-style)70.9 vs. 67.1+3.8+3.7[+2.4, +5.0]

Fixed-target and internalization comparisons on 8B:

ComparisonScores in paperGain in paperMean gain over 8 decodingsPaired 95% CI
Full experience vs. no experience (fixed targets)61.4 vs. 59.4+2.0+1.9[+0.8, +3.0]
Full experience vs. redacted experience (fixed targets)61.4 vs. 61.0+0.4+0.4[−0.6, +1.4]
Explicit experience vs. EL-style (same experience; targets differ)61.4 vs. 60.8+0.6+0.8[+0.1, +1.5]
Joint vs. frozen-experience RL63.3 vs. 62.3+1.0+1.1[+0.3, +1.9]

The main results on both students and the effect of the experience text (−2.0 without it) have intervals that exclude 0. Of the three comparisons you named, explicit experience generation vs. EL-style internalization and joint vs. frozen-experience RL have intervals that exclude 0, so these gains are small but stable; for joint vs. frozen-experience RL, three independent training seeds on 8B give a mean gain of 0.9 ± 0.2.

(iv) Shared judge. We agree that the rule-checked benchmarks deserve more weight. On these two benchmarks, Agent as Coach leads the strongest baselines on both students: on 8B, 73.3 vs. 66.9 on IFEval and 30.9 vs. 24.8 on IFBench; on 9B, 89.1 vs. 87.1 on IFEval and 32.4 vs. 26.9 on IFBench (Table 1).

Question 4 (critique-based learning). The difference lies in what the guidance targets and where it goes. These three works train a model to critique a given answer: the critique is used to fix that answer (Critique-RL) or to improve the critic itself (Critique Fine-Tuning, Critique-Coder). Our coach reads a pool of scored attempts, writes reusable guidance for another student's new attempts, is rewarded by that student's improvement, and is removed after internalization.

Thank you for the response, which has addressed most of my concerns. I will keep my positive score.

Thank you very much for your strong support of our work and for your insightful comments. We are glad that our response addressed most of your concerns. Your feedback has been very helpful in improving the paper, and we will incorporate the discussed revisions into the final version.

Reviewer_z424OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a coach to inspect scored student attempts, selectively read and compress them through tools, and produce guidance rewarded by improvement in a frozen student. It then trains the student on selected experience-answer pairs, followed by question-only RL in which the student generates its own experience before answering. Across three instruction-following benchmarks and two student models, the complete pipeline improves the benchmark average over the strongest evaluated baseline by 4.9 and 3.8 points.

Critical strengths
  1. The complete pipeline gives consistent gains on both students, including IFEval and IFBench with deterministic checkers. Deployment requires only the question, without an external coach. The full-context coach follows the same four-stage training recipe, so the paper does include a meaningful trained-coach comparison.

  2. The fixed-answer controls in Table 6 support a benefit from the experience text itself. Removing experience reduces the average from 61.4 to 59.4; a length-matched generic plan reaches 60.0, while redacted experience retains 61.0. These results address a plausible alternative explanation based solely on better selected answers or additional planning tokens.

Critical weaknesses
  1. The incremental contribution over learned coaching and context internalization remains insufficiently established. Retroformer and CTRL already train reflective guidance using downstream outcomes; LLM-as-a-Coach extracts experiential guidance and internalizes it through context distillation. MemAgent learns segmented reading and memory updates with RL. Section 4 acknowledges these connections, so my concern is the evidence for the added design.

    In particular, the improvement reward is not a distinct teaching criterion by itself. In Eq. (4), when all trajectories in a group are valid, the shared unguided score cancels under group-relative normalization: normalizing the guided scores minus a common baseline gives the same advantages as normalizing the guided scores directly. The invalid-trajectory penalty requires separate treatment. The substantive addition therefore needs to come from jointly learning evidence acquisition and guidance generation, together with the proposed internalization procedure.

    Table 3 does not establish the acquisition policy's advantage over simple context selection. At K=64 and 128, the full-context input is truncated, whereas the agent reads within an 8,192-token evidence allowance. At K=32, the 4.1-point advantage over the trained mixed-K coach is useful evidence, but there is still no score-stratified subset or fixed sequential-compression baseline under matched evidence and generation allowances. Such controls would show whether adaptive acquisition contributes beyond filtering redundant context.

    The concurrent Learning to Coach preprint also closely overlaps with frozen-actor, downstream-reward coach training. I mention it to clarify positioning; its absence from the comparisons is not a reason for my recommendation.

  2. The internalization results do not yet establish a robust advantage for the particular new training choices. Table 6 shows that EL with the same experience already reaches 60.8, only 0.6 points below explicit experience generation at 61.4. In Table 4, jointly updating the experience adds only 1.0 and 0.7 points over frozen experience. Appendix A.2 evaluates question-only performance with one sampled output per prompt at temperature 1.0, and no repeated-training results or uncertainty intervals are reported. These small differences need stronger support before they can carry the methodological claim.

    Answer selection is also a substantial pathway to the final result. With the agent-selected answer targets fixed, answer-only SFT reaches 59.4, already above full-context experience SFT at 55.8. These are different data conditions and cannot be treated as an additive decomposition. Nevertheless, the comparison shows why the headline pipeline gains should not be read as the independent contribution of explicit experience generation. Reporting retained Stage III task and answer counts, then comparing answer-only, EL-style, and explicit-experience training on a common answer bank, would make this distinction much clearer.

Questions for authors
  1. Can you compare adaptive acquisition with score-stratified selection and a fixed reading/compression schedule, using the same evidence and generated-token allowances? Please carry the strongest simple control through the existing student-training pipeline so that the comparison includes final question-only performance.

  2. How many tasks and answers survive Stage III filtering for the agent and full-context coaches? On shared tasks with matched answer targets and sample counts, does explicit experience generation retain an advantage over EL-style internalization?

  3. Can you provide paired prompt-level uncertainty or repeated-decoding results for the fixed-target controls, and independent-run evidence for the small joint-versus-frozen improvements?

4 comments
AuthorsOpenReview ↗

Response to Reviewer z424 (1/2)

Weakness 1 (incremental contribution). Thank you for the detailed comments on this point. We break it into four parts: (a) what is new relative to prior work; (b) the reward form of Eq. (4); (c) comparison with simple context selection; (d) relation to Learning to Coach.

(a) What is new. Two things: the coach no longer reads its input in one pass but acquires evidence from the behavior pool as an agent, and the experience it writes is finally internalized into the student. Existing coaches, whether prompted (LLM-as-a-Coach) or trained on downstream outcomes (Retroformer, CTRL), read their given input in one pass. More student behavior yields better experience (86.0 with one response, 89.3 with eight; Fig. 1a), but one-pass reading does not scale with it: the untrained full-context coach drops to 81.7 at K = 32, below the rubric-only 84.9; even when trained with the same Stage II reward on mixed K (8–32), it still falls to 82.9 at K = 128 (Table 3). MemAgent reads in segments to answer a question itself, not to study another model's behavior in order to teach it. Overall results are in Table 1, and the contribution of the acquisition policy alone is in (c) below.

(b) The reward form of Eq. (4). Your derivation is correct, and we do not claim the reward form as a contribution. When all trajectories in a group are valid, sˉ\bar{s} is shared across the group and cancels under group normalization. Stage II follows the learning-to-teach principle (Sec. 4); our contribution is using it to train the entire trajectory of the agent's evidence acquisition and writing.

(c) Comparison with simple context selection. Under the same budget as the agent (8,192 evidence tokens, 2,048 generated tokens), the learned acquisition policy still outperforms the two simple baselines you proposed (guided InFoBench DRFR, %):

CoachK=8K=32K=128
Full-context (trained, mixed K)89.687.282.9
Score-stratified subset89.689.890.1
Fixed read-then-compress89.690.390.5
Agent as Coach90.491.391.1

Both simple baselines stay below Agent as Coach at every K; at K = 32 and 128, the stronger one, fixed read-then-compress, is still 1.0 and 0.6 points lower. This shows that acquiring evidence as an agent is more effective.

(d) Relation to Learning to Coach. Thank you for pointing out this concurrent work (made public on September 14, 2026). It is a follow-up to LLM-as-a-Coach: it trains the coach with the downstream reward of a frozen actor, but the coach reads a single trajectory in one pass, the actor stays frozen, the experience is not internalized, and the coach is still needed at deployment. The closest counterpart in our setting is the full-context coach in Table 1: it is likewise trained with a frozen student's downstream reward and reads in one pass, while reading eight scored responses and going through the same internalization pipeline. It reaches 58.4 / 66.8, versus 63.3 / 70.9 for Agent as Coach.

AuthorsOpenReview ↗

Response to Reviewer z424 (2/2)

Weakness 2 (evidence for internalization). We respond in two parts: (a) the robustness of the small differences in internalization; (b) the contribution of answer selection to the final results.

(a) Robustness of small differences. We agree that these differences are small and that single-sample evaluation is not enough to support method-level claims. We decode each prompt 8 times at T = 1.0 and compute 95% confidence intervals with a prompt-paired bootstrap. Results on 8B (Gain = score of the first minus the second):

ComparisonGain in paperMean gain over 8 decodingsPaired 95% CI
Full experience vs. no experience (fixed targets)+2.0+1.9[+0.8, +3.0]
Explicit experience vs. EL-style (same experience; targets differ)+0.6+0.8[+0.1, +1.5]
Joint vs. frozen-experience RL+1.0+1.1[+0.3, +1.9]

Although small, these differences are stable under repeated decoding: all three mean gains are close to the values reported in the paper, and no interval includes 0, so they are not artifacts of single-sample evaluation.

For the small gain of joint over frozen-experience RL, we trained Stage IV independently with 3 random seeds (42, 24, 64) for both settings on 8B, starting from the same Stage III checkpoint; seed 42 is the run reported in the paper:

Seed (8B)Joint RL (Stage IV)Frozen-experience RLGain
Seed 42 (paper)63.362.3+1.0
Seed 2462.961.9+1.0
Seed 6463.162.4+0.7
Mean ± std63.1 ± 0.262.2 ± 0.3+0.9 ± 0.2

Joint RL outperforms frozen-experience RL under all 3 seeds, by 0.9 ± 0.2 on average, so this gain is not an artifact of a single training run.

(b) Contribution of answer selection. We agree that answer selection is an important pathway, and the gain of the full pipeline should not be read as an independent contribution of explicit experience generation. Selection itself, however, is part of the coach's effect: the answers retained in Stage III do not come from a stronger external model, but are written by the student under the coach's experience and exceed its own best score s+s^+ (Sec. 2.3). The 59.4 vs. 55.8 you pointed out reflects exactly this: the better the coach, the better the answers that the student writes under its guidance and that are retained. Stage III retention for the two coaches:

Quantity (8B)Full-context coachAgent as Coach
Stage III input tasks (before filtering)5,0005,000
Candidate guided answers (5,000 × 8)40,00040,000
Tasks with ≥1 retained answer2,8703,460
Retained answers6,9409,820
Stage III question-only Avg55.861.4

Agent as Coach retains more tasks and more answers. The comparison of the three training methods on the same answer targets is given in Table 6: answer-only (59.4) and explicit experience (61.4) use the same answer bank, with identical tasks, answer targets, and sample counts (App. B.3). EL uses the same tasks and the same experience and reaches 60.8, but it is on-policy distillation: its training signal is the distribution of an experience-conditioned teacher on prefixes generated by the student itself, so it uses no fixed answer targets and cannot be aligned with the answer bank one to one. Both internalization methods outperform answer-only training, showing that the experience text helps beyond the answers; explicit experience generation is 0.6 points above EL-style internalization, with the interval given in (a).

Question 1 (simple baselines). The full pipeline uses K = 8 (Table 5). At this K, all 8 responses (about 5k tokens) fit within the 8,192-token evidence budget, so both simple baselines read the complete pool, which is equivalent to reading all responses at once (in the table of Weakness 1 (c), all three score 89.6 at K = 8). In the full pipeline, the simple baselines are therefore equivalent to a full-context coach that reads all responses. The full-context coach in Table 1 is exactly such a coach, trained at K = 8 with the same Stage II reward and internalized with the same pipeline; it reaches 58.4 / 66.8, versus 63.3 / 70.9 for Agent as Coach.

Question 2 (Stage III retention and the EL comparison). The numbers of tasks and answers retained by the two coaches after Stage III filtering, and the comparison among answer-only, EL-style, and explicit experience training, are given in Weakness 2 (b).

Question 3 (uncertainty). Repeated-decoding results and prompt-paired confidence intervals for the fixed-target comparisons and for joint vs. frozen-experience RL, as well as the multi-seed training results for the latter, are given in Weakness 2 (a).

Thanks for response, I will raise my score.

Thank you for your positive feedback and for raising your score. We truly appreciate your constructive comments, and we will incorporate the clarifications and additional results from our discussion into the revised version of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive comments. Reviewers found the shift from rubric criteria to actionable experience well motivated (4hZJ), the coach-free student gains consistent and practical (z424, pVbT), and the analyses a direct test of the core claims (pVbT). Below we summarize the main clarifications and new results.

1. Eq. (4) (z424). We agree: when all trajectories in a group are valid, the pool mean sˉ\bar{s} is shared by the whole group and cancels under group normalization, so the improvement reward is equivalent to using the guided score directly. We do not present it as a distinct teaching principle and will revise the wording accordingly. Our contribution lies in using one reward to jointly learn evidence acquisition and guidance writing (Eq. 5), and in the internalization pipeline.

2. The controlled trained-coach comparison is already in Table 1 (z424, pVbT). The full-context coach is a CTRL-style coach trained with the same Stage II reward and internalized with the same pipeline; the only difference is that it reads the entire pool at once (Sec. 3.1, App. A.3). It reaches 58.4 / 66.8, versus 63.3 / 70.9 for Agent as Coach.

3. Stage III statistics (z424 Q2; pVbT W1/Q1). From existing logs (Qwen3-8B-Base, 5,000 SFT tasks, K = M = 8):

QuantityFull-context coachAgent as Coach
Stage III input tasks (before filtering)5,0005,000
Pool K / guided answers M8 / 88 / 8
Candidate guided answers (5,000 × 8)40,00040,000
Tasks with ≥1 retained answer2,8703,460
Retained answers6,9409,820
Mean margin above s+s^+1.211.38
Mean experience length (tokens)212236
Stage II final coaching gain1.121.16
Stage III question-only Avg55.861.4

The two coaches have similar mean gains (1.12 vs. 1.16), yet their Stage III students differ substantially (55.8 vs. 61.4). The difference lies in the upper tail of the score distribution: the coaching gain compares means (the difference between mean scores with and without experience, App. A.2), whereas Stage III keeps only answers that exceed the student's own best score s+s^+ (Sec. 2.3). Agent as Coach retains 41% more answers (9,820 vs. 6,940), and they exceed s+s^+ by a larger margin (1.38 vs. 1.21).

4. Repeated evaluation (z424 W2/Q3; pVbT W2/Q2). We decode each prompt 8 times at T = 1.0 and compute 95% confidence intervals with a prompt-paired bootstrap (Gain = score of the first minus the second). Main results in Table 1:

ComparisonScores in paperGain in paperMean gain over 8 decodingsPaired 95% CI
8B · Agent as Coach vs. strongest baseline (full-context coach)63.3 vs. 58.4+4.9+4.8[+3.5, +6.1]
9B · Agent as Coach vs. strongest baseline (EL-style)70.9 vs. 67.1+3.8+3.7[+2.4, +5.0]

Fixed-target and internalization comparisons on 8B:

ComparisonScores in paperGain in paperMean gain over 8 decodingsPaired 95% CI
Full experience vs. no experience (fixed targets)61.4 vs. 59.4+2.0+1.9[+0.8, +3.0]
Full experience vs. redacted experience (fixed targets)61.4 vs. 61.0+0.4+0.4[−0.6, +1.4]
Explicit experience vs. EL-style (same experience; targets differ)61.4 vs. 60.8+0.6+0.8[+0.1, +1.5]
Joint vs. frozen-experience RL63.3 vs. 62.3+1.0+1.1[+0.3, +1.9]

The main results on both students (+4.9, +3.8) and the effect of the experience text (−2.0 without it) have intervals that exclude 0. The two smaller differences, explicit experience generation vs. EL-style internalization and joint vs. frozen-experience RL, also have intervals that exclude 0, and we describe them as small but stable gains; for joint vs. frozen-experience RL, three independent training seeds on 8B give a mean gain of 0.9 ± 0.2. The interval for redaction includes 0, and we report it as an observation rather than a conclusion. The main results do not depend on these choices: replacing explicit experience generation with EL-style internalization (8B: 60.8), or joint RL with frozen-experience RL (62.3 / 70.2), still exceeds the strongest baselines (58.4 / 67.1).

5. Budget-matched simple context selection (z424). Under the same evidence and generation budgets as the agent, both simple baselines, score-stratified selection and fixed read-then-compress, stay below Agent as Coach at every K; see our response to z424, Weakness 1 (c).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.