- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper trains a coach to inspect, read, and compress a pool of scored student responses before producing experience. The coach is optimized using improvement in a frozen student. The student then learns through SFT and RL to generate and use experience without a coach at deployment. Experiments with two Qwen base students show improvements on three instruction-following benchmarks, with analyses of behavioral evidence, agentic reading, and experience internalization.
Critical strengths
-
The move from rubric criteria to actionable experience is well motivated. Knowing the properties of a good answer is different from knowing how to produce it. Multiple student attempts provide richer coaching evidence than a single local correction.
-
Tool-based reading addresses the stated context problem. Inspect, read, and compress keep the behavior pool outside the context and let the coach choose what to acquire and retain. Training on frozen-student improvement connects this reading process to its intended utility.
Critical weaknesses
-
The evaluation supports instruction following more directly than general non-verifiable learning. IFEval and IFBench use rule-based checks, and InFoBench also evaluates decomposed requirements. These tasks test execution of explicit constraints well. They do not directly establish an answer to open-ended writing or dialogue where criteria may be incomplete or conflicting.
-
The bottleneck may partly be compressed feedback rather than the absence of a learned coach. The method reduces rubric judgments to scalar scores from 1 to 10, then asks the coach to infer procedures from several scored answers. When the rubric already specifies criteria, direct per-criterion feedback could provide similar diagnostic information. Table 2 compares rubric-only input with scored behavior, but does not distinguish the value of behavioral evidence from the value of richer feedback.
-
The coach objective targets a different student state from the final one. Stage II maximizes immediate gain in a frozen base student. Stages III and IV then change that student's capability and error distribution. Procedures useful to the base model may not be those the updated model most needs. Stage IV can revise self-written experience, but the analysis does not yet show how this mismatch is resolved.
-
The Stage III retention rule can substantially change the training distribution. A coached answer is retained only if it exceeds the best unguided score, and J is capped at 10. Once an unguided pool contains a score-10 answer, that task cannot contribute a Stage III example. Larger K can increase such ceiling cases. Internalization therefore trains on a selected subset of task difficulties and constraint types rather than the original distribution.
Questions for authors
None.
3 comments
Thank you for the authors’ response, which has addressed most of my concerns. I will keep my positive score.
Thank you for your positive assessment and for your constructive feedback throughout the review process. We are glad that our response addressed most of your concerns, and we will incorporate the discussed revisions into the final version of the paper.
Response to Reviewer 4hZJ
Thank you for the specific comments on our design choices. We respond to each below.
Weakness 1 (scope beyond instruction following). We agree that the scope of our conclusions should be limited to instruction-following tasks that can be characterized by rubrics. We evaluate mainly on rule-checked benchmarks to keep evaluation independent of the training signal: training uses only non-verifiable OpenRubrics tasks, with all rewards coming from GPT-4o scores, whereas IFEval and IFBench are checked by rules and unaffected by judge preferences, which rules out the possibility that the student merely learns to please the training judge. On these two benchmarks, Agent as Coach leads the strongest baselines on both students (Table 1), and the IFBench constraints are unseen in common training data. On non-verifiable tasks themselves, the coach raises the student's judge score by 1.16 points on 500 held-out OpenRubrics tasks (Table 2).
Weakness 2 (scalar feedback as a bottleneck). You are right that the feedback in Table 2 is always a scalar score. Table 2 shows that behavioral evidence carries information beyond the rubric (a gain of 0.47 with the rubric alone and 1.16 with eight scored answers), but it does not compare feedback forms. Using scalar scores is a deliberate design choice: the judge is the stronger GPT-4o, and per-criterion judgments would bring its analysis of the answers into the coach's input, effectively letting a stronger model take part in writing the experience and making it hard to tell whether the gains come from the coach or from the judge. With scores as feedback, the judge only scores (Sec. 3.1), and the experience is inferred entirely from student behavior by a coach of the same size as the student. The coach also reads the rubric and the full answers, so it can determine whether each requirement is met by checking the answers against the rubric. Whether richer feedback brings further gains is listed as future work in our conclusion.
Weakness 3 (mismatch in student state). You are right that this mismatch exists; it arises in Stage IV. The Stage III experience is written by the coach for the base student, and the answers are generated by the same base student under that experience, so the two are consistent. In Stage IV the student has changed, but this stage no longer uses the coach; instead, the student writes its own experience. The judge scores only the answer, and this reward updates both the experience and the answer (Eq. 7), so the experience can adapt as the student changes. The comparison in Table 4 illustrates this: starting from the same Stage III student, if the experience is written by a frozen Stage III copy and RL trains only the answer, the result is 62.3 / 70.2; letting the experience update with the student gives 63.3 / 70.9. We trained this comparison independently with 3 random seeds (42, 24, 64) on 8B:
Joint RL outperforms frozen-experience RL under all 3 seeds, by 0.9 ± 0.2 on average. Continuing to train the coach for the updated student is a natural next step, and we will pursue it as future work.
Weakness 4 (retention rule). You are right that the retention rule filters out some tasks, including saturated ones. This is by design: when s+ is already 10, the judge's scale cannot distinguish a better answer, so there is no improvement signal to learn from. At K = 8, 12.2% (610) of the 5,000 Stage III tasks are excluded due to saturation; for Agent as Coach, another 18.6% (930) are not retained because no guided answer exceeds s+, and the remaining 3,460 tasks keep at least one answer. This filtering happens in Stage III: Stage IV uses 27,000 unfiltered RL tasks from the same source (Table 5), which likewise include easy tasks that the student can already answer perfectly, so the student still trains on the original task distribution in this stage.