acceptodds
Under review as a conference paper at ICLR 2027

ECOD: Experience-Calibrated On-Policy Distillation for Multi-Turn Agents

Abstract

Sparse and delayed rewards make reinforcement learning challenging for long-horizon language agents. On-policy distillation (OPD) complements environmental rewards with dense teacher feedback on student-generated contexts. Standard OPD can misalign supervision with relative teacher competence, student trajectory outcomes, and the temporal structure of task execution. Recent variants adjust guidance through divergence-based weighting and temporal schedules, while joint calibration from task-completion experience remains underexplored. We introduce Experience-Calibrated On-Policy Distillation (ECOD), which allocates teacher supervision at the task, trajectory, and turn levels. ECOD compares teacher and student success-rate posteriors under matched task conditions, restricts direct supervision to failed student trajectories, and derives a temporal prior from successful teacher lengths. The resulting weight applies to teacher losses or teacher-derived advantages. We evaluate ECOD on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. Under matched teacher settings, ECOD improves standard OPD by up to 4.06% in relative ALFWorld unseen success. Across both model scales, ECOD-assisted SOD, TCOD-F2B, and ATOD configurations yield average paired relative gains of 6.20% on ALFWorld unseen and 7.05% on WebShop, with WebShop gains ranging from 2.29% to 21.60%. These results support experience-calibrated supervision allocation as a reusable mechanism across distinct distillation objectives.

Primary area:
reinforcement learning
Keywords:
On-policy distillation, Multi-turn language agents, Reinforcement learning, Teacher supervision allocation, Experience calibration.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ez6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ECOD allocates teacher supervision using three factors: task-conditioned teacher–student success estimates, a gate selecting failed student trajectories, and turn weights derived from successful teacher trajectory lengths. The resulting weight applies to either a teacher loss or a teacher-derived advantage, while all student trajectories remain available for reward-based learning. Experiments on ALFWorld and synthetic WebShop cover two model sizes, several training objectives, and allocation ablations.

Critical strengths
  1. The allocation rule is simple and modular. Relative task competence, student outcomes and teacher completion lengths each have a clear role, and the mechanism can be applied to both teacher-loss and teacher-advantage formulations without removing successful trajectories from reward-based learning.
  2. The budget-matched component ablations and uniform-allocation controls provide evidence that supervision placement matters beyond its average strength (Figure 4; Appendices A.9–A.10). These are more informative than comparing only against an unchanged OPD coefficient.
Critical weaknesses
  1. The 7B WebShop comparisons change teacher strength alongside allocation. Appendix A.7 assigns larger teacher coefficients to ECOD while paired baselines retain their original strengths. These comparisons therefore do not isolate the effect of allocation from coefficient choice. The matched controls are restricted to 1.5B GRPO and do not resolve this confound in the broader comparisons.
  2. The evidence for task-specific temporal calibration is limited. Equation (7) measures how long successful teacher executions last, rather than the usefulness of teacher feedback on a failed student's current history. It assigns zero supervision beyond the longest cached successful teacher execution, although student and teacher progress need not align by turn index. In Figure 4(a), a pooled temporal prior reaches 68.07% versus 68.46% for the task-specific prior; removing temporal weighting gives 65.33%. These results support temporal attenuation more clearly than the need for task-specific execution-length calibration.
  3. The incremental contribution over existing outcome-aware selective supervision remains unclear. Appendix A.2 distinguishes ECOD's task-local completion statistics from related selection and reweighting rules, but the experiments do not compare against those alternatives. For example, the cited SERL also studies selective, reward-aware supervision on ALFWorld and WebShop. Component-removal ablations support ECOD within the tested system, but do not establish the added value of its particular combination over prior selective-supervision designs. A comparison controlling for available teacher feedback would better establish that contribution.

The allocation mechanism and controlled ablations are promising, but the coefficient mismatch prevents attribution of the 7B WebShop results to allocation alone. Evidence for the added value of task-specific temporal calibration is also limited.

Questions for authors
  1. Do the 7B WebShop gains remain when paired methods use identical teacher coefficients or the same coefficient-tuning grid?
  2. Given the pooled-prior result, what supports task-specific temporal calibration over a shared decay, particularly the zero supervision weight on late failed-student states?
  3. Under comparable teacher feedback and training budgets, how does ECOD compare with prior outcome-aware selective-supervision rules or a controlled adaptation of them?
4 comments
AuthorsOpenReview ↗

Matched Teacher Coefficients, Temporal Calibration, and Complementary Supervision

We thank the reviewer for the constructive comments and recognition of the allocation controls. We added matched-coefficient and temporal-tail experiments, and respond point by point.

1. Do the 7B gains remain at identical teacher coefficients? We trained a new 7B WebShop SOD baseline with base teacher-advantage coefficient c=4.0c=4.0, matching existing SOD+ECOD. The matched pair shares the frozen teacher, initialization, task schedule, optimizer, and final update 150. Coefficient pairs below are (SOD, SOD+ECOD); SR denotes success rate and TS task score.

\text{Metric}&\text{Coefficients}&\text{SOD}&\text{SOD+ECOD}&\text{Gain}\\\\ \hline \text{SR (\\%)}&(1.0,4.0)&79.88\pm2.04&82.03\pm1.82&{\color{green}\uparrow2.15}\\\\ \text{SR (\\%)}&(4.0,4.0)&74.71\pm1.96&82.03\pm1.82&{\color{green}\uparrow7.32}\\\\ \text{TS}&(1.0,4.0)&87.50\pm1.14&90.28\pm1.26&{\color{green}\uparrow2.78}\\\\ \text{TS}&(4.0,4.0)&82.96\pm1.51&90.28\pm1.26&{\color{green}\uparrow7.32}\\\\ \end{array}$$ Tables report mean ± sample SD; pp denotes percentage points. The original 1.0/4.0 pair provides context (A.7); the 4.0/4.0 control adds a SOD baseline and reuses the ECOD endpoint (A.15, Table A8). Here 4.0 is the coefficient value. Matched SR and TS gains are 7.32 pp and 7.32 points. Base strength is matched; ECOD sets effective weighted supervision. The control uses final update 150; original main-table selection used repeated test scores (A.8). **2. What supports task-specific timing and zero late supervision?** Task conditioning retains each task's successful-execution profile. In the group with 8 teacher and 7 student successes, mean successful-teacher lengths span 4.5-10.5 turns (A.1). Pooling averages these profiles. Full ECOD reaches 68.46% SR, pooled 68.07%, and no temporal weighting 65.33%: a 0.39 pp task-specific gain, with both priors above the no-temporal control (A.9). We tested late guidance on WebShop 1.5B GRPO, changing only zero temporal weights to 0.1. Positive weights, competence/outcome masks, teacher assets, teacher-KL coefficient 0.02, reference-KL coefficient 0.01 and training protocol stay fixed. $$\begin{array}{l|cc} \text{Final update 150}&\text{SR (\\%)}&\text{TS}\\\\ \hline \text{Original ECOD}&68.46\pm1.38&80.75\pm1.71\\\\ \text{Zero tail set to }0.1&65.82\pm2.12&78.95\pm1.79\\\\ \text{ECOD gain}&{\color{green}\uparrow2.64}\text{ pp}&{\color{green}\uparrow1.80}\\\\ \end{array}$$ Guidance resumes in 143/150 updates, adding 3.00% weighted supervision relative to the original rule on the control's own batches. Base coefficients are matched; effective budgets differ (A.17, Table A10). The 2.64 pp SR and 1.80 TS gains support the zero-tail rule in this setting. The prior measures execution-length coverage, not state-specific correction value. Teacher feedback uses the student's actual history; reward learning continues beyond cached support (4.1, A.3). **3. What is the added value over outcome-aware selection?** Paired comparisons preserve existing adaptive mechanisms (A.3, A.6): SOD uses divergence-based step weights; ATOD combines disagreement/uncertainty weights with OPD-RL annealing; TCOD-F2B uses a depth curriculum. ECOD jointly adds task-local relative competence, student outcomes and teacher completion lengths. We added two training seeds per WebShop 1.5B configuration. Each baseline/ECOD pair shares teacher, initialization, task schedule, base coefficients/native schedules and the 150-update budget. Three-seed SR at update 150: $$\begin{array}{l|ccc} \text{SR (\\%)}&\text{Baseline}&\text{+ECOD}&\text{Gain (pp)}\\\\ \hline \text{SOD}&59.41\pm4.77&71.13\pm6.24&{\color{green}\uparrow11.72}\\\\ \text{TCOD-F2B}&66.67\pm1.57&70.02\pm1.25&{\color{green}\uparrow3.35}\\\\ \text{ATOD}&71.65\pm1.24&74.15\pm3.42&{\color{green}\uparrow2.51}\\\\ \end{array}$$ Eight evaluations are averaged per seed; mean ± sample SD is then computed across training-seed means. Gains use unrounded values. SOD and TCOD-F2B improve in every seed. Mean TS gains are SOD +12.27, TCOD-F2B +5.16 and ATOD +1.64 (A.13, Tables A5-A6), supporting added value on all three tested adaptive objectives. On ALFWorld unseen, uniform allocation replaying ECOD's per-update weighted-token budget reaches 72.29% SR versus ECOD's 74.16% (+1.87 pp; A.10, Table A4), supporting supervision placement at matched budgets. SERL is related outcome-aware work (A.2); our paired evidence evaluates complementary calibration on SOD, TCOD-F2B and ATOD. New WebShop endpoints use update 150 and eight evaluations on 128 tasks, temperature 0.4 and horizon 15. Coefficient/tail comparisons use one training seed and report decoding SDs; the three-seed table reports retraining SDs.
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experimental Results

Dear Reviewer,

Thank you for your valuable comments and suggestions.

We have carefully addressed your concerns in our rebuttal and conducted additional experiments to further support our findings and clarify the points you raised.

We would greatly appreciate it if you could take a look at our responses and the new experimental results. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications or additional evidence.

Thank you again for your time and consideration.

Thanks for the reply and added experiments. The matched 7B control and zero-tail ablation is helpful for my concern, but the improvement of task-specific prior over pooled one is still quite small, and there is still no direct comparison with other outcome-aware methods like SERL. So I will keep my score.

AuthorsOpenReview ↗

Clarification on Incremental Contribution and Temporal Calibration

Thank you for your thoughtful follow-up and for recognizing our additional experimental controls.

We would like to clarify the scope of our contribution. ECOD is designed as a reusable supervision-allocation mechanism that complements existing OPD objectives. Accordingly, our primary evaluation examines whether ECOD provides additional benefits to established methods that already employ adaptive teacher guidance.

We directly evaluate this with SOD, TCOD-F2B, and ATOD, which use step-wise divergence weighting, a trajectory-depth curriculum, and disagreement/uncertainty-based weighting, respectively. Across three training seeds, adding ECOD yields positive mean success-rate gains of 11.72, 3.35, and 2.51 %, respectively. Furthermore, on ALFWorld, ECOD outperforms uniform allocation under the same per-update weighted-token budget by 1.87 %. These controlled comparisons provide evidence for ECOD's incremental value across the tested adaptive OPD methods.

Regarding SERL, its primary mechanism reweights reward-driven advantages using hindsight-conditioned teacher likelihoods, alongside an action-only distillation term. ECOD instead calibrates a separate teacher contribution using task-relative competence, student outcomes, and successful-teacher execution lengths. We acknowledge that a direct comparison with SERL would help further characterize their relative strengths, while our existing experiments directly support the claimed incremental benefits on SOD, TCOD-F2B, and ATOD.

Finally, we agree that the task-specific prior provides only a modest improvement over pooling (+0.39 pp). However, both temporal priors outperform removing temporal weighting (68.46% and 68.07% vs. 65.33%), providing stronger evidence for the broader temporal-allocation mechanism.

Thank you again for your careful evaluation and constructive feedback.

Reviewer_PRpAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation (OPD) for reinforcement learning of multi-turn LLM agents, arguing that standard OPD applies uniform teacher supervision to all tasks, trajectories, and turns, which misaligns with actual needs at the task, trajectory, and turn levels. The authors propose ECOD, which computes a bounded supervision weight for each turn based on a posterior comparison of teacher and student success rates on the same task, the outcome of each student trajectory, and the length distribution of successful teacher trajectories. The weight can be applied either to the teacher distillation loss or to a teacher-derived advantage, so it can be attached to a range of existing OPD methods. Experiments on ALFWorld and WebShop with two model scales show improvements in most paired comparisons, and component ablations and teacher-strength controls are used to analyze the role of supervision allocation.

Critical strengths

The method is simple and general. ECOD only reweights the teacher term without changing the construction of the RL advantage or the rollout procedure, and can be attached to OPD methods based on either a teacher loss or a teacher-derived advantage.

The paper analyzes the misalignment of teacher supervision at three levels (task, trajectory, and turn), and the three components of the method correspond to them one-to-one, giving a clear and complete overall logic.

The experiments cover two benchmarks, two model scales, and several OPD variants, compare different allocation schemes under matched supervision budgets, and describe implementation details fairly transparently.

Critical weaknesses
  1. The teacher setting is weak, and the core motivation of the paper depends heavily on it. The teacher has the same size as the student and is simply trained with the same RL method for a limited number of steps, so the student quickly catches up with or even surpasses it during training. The task-level "competence mismatch" is therefore largely an artifact of this weak-teacher setting rather than a general problem of OPD. In the more common setting where a stronger, larger teacher guides a smaller student, the teacher is clearly better on most tasks, so the competence calibration would matter much less, and the value of the method becomes unclear.

  2. The mechanism of ECOD is essentially close to "retiring the teacher early", and its distinction from existing methods is limited. Because the competence term quickly goes to zero as the student improves, ECOD in practice behaves more like "warming up with the teacher early on, then doing mostly RL", while also greatly reducing the total amount of supervision. This is close to existing ideas such as teacher retirement and gating by teacher–student success rates, and the paper does not sufficiently show what essential advantage the fine-grained three-factor allocation has over these simpler strategies.

  3. The design of the individual components is not sufficiently justified. Supervising only failed trajectories is motivated by the observation that the student assigns higher likelihood than the teacher to its own successful actions, but this can be directly explained by sampling bias and does not show that teacher supervision on successful trajectories is harmful; moreover, successful trajectories may also contain inefficient steps. The temporal prior zeroes out supervision on later turns beyond the lengths of successful teacher trajectories, yet failed trajectories often get stuck in loops or go off track precisely in those later turns, which may be where correction is most needed. The competence term multiplies two highly correlated quantities without justification, which over-suppresses supervision on tasks where the teacher is only slightly better.

  4. The practicality of the method is limited. It requires fixing the training task schedule in advance and pre-collecting teacher rollouts on every specific task instance to estimate the statistics. This is hard to apply directly when tasks are sampled online or environment configurations keep changing, and the additional cost of pre-collection and online teacher scoring is not quantified.

  5. On the experimental side, the improvement over standard OPD in direct comparisons is limited, and in some settings performance even drops; the larger gains mainly come from comparisons with pure RL or with weaker re-implemented baselines, and the way results are presented in the abstract and the overview figure tends to overstate the effect. In some settings, the teacher strength is adjusted only for ECOD, the criterion for selecting the checkpoints in the main tables is not stated, and comparisons with simple baselines such as teacher retirement or early stopping of distillation are missing.

Questions for authors

Please see the weaknesses.

3 comments
AuthorsOpenReview ↗

ECOD Applicability, Allocation, and Controlled Evaluation

We thank the reviewer for the constructive comments. We added stronger-teacher, multi-seed and matched-coefficient experiments to address each concern.

1. Teacher applicability. Equal size controls capacity. Teachers have 150 RL updates; students start pretrained. We added a frozen 7B teacher for 1.5B students at matched coefficients (WebShop SOD, update 150).

SR: success (%); TS: task score; gains use unrounded means.

&\text{SOD}&\text{+ECOD}&\text{Gain}\\\\ \hline \text{SR}&75.59\pm1.86&79.69\pm0.84&{\color{green}\uparrow4.10}\text{ pp}\\\\ \text{TS}&86.23\pm1.45&89.84\pm1.02&{\color{green}\uparrow3.61}\\\\ \end{array}$$ Validation-selected SR improves $70.90\\%\to75.59\\%$ ($+4.69$ pp), extending SOD gains to larger teachers (A.14, Table A7). **2. Allocation versus amount.** Retirement stops guidance globally; ECOD allocates by task, trajectory and turn within each update. Uniform control replays the ECOD weighted-token budget at all 150 updates (ALFWorld unseen): $$\begin{array}{l|ccc} &\text{Uniform}&\text{ECOD}&\text{Gain (pp)}\\\\ \hline \text{SR}&72.29\pm3.15&74.16\pm0.97&{\color{green}\uparrow1.87}\\\\ \end{array}$$ Matched-budget gains support supervision placement (A.10, Table A4). **3. Beyond success-rate gating.** Success-rate gating alone leaves within-task allocation unspecified. ECOD adds outcome/teacher-length weighting atop SOD/TCOD/ATOD (4.1-4.2, A.3, A.6). With matched SOD implementation, teacher and base coefficient, WebShop 1.5B three-seed means improve SR $59.41\\%\to71.13\\%$ ($+11.72$ pp), TS $70.42\to82.69$ ($+12.27$; A.13, Tables A5-A6). **4. Components and late guidance.** Successes still inform RL and competence estimates; we omit direct imitation to preserve student exploration. The gate targets completion rather than step efficiency; likelihood gaps are descriptive evidence. Competence multiplies gap magnitude by posterior support, damping guidance when either is small without assuming independence; RL continues. WebShop 1.5B GRPO, update 150, Full ECOD SR $68.46\\%$: $$\begin{array}{l|cc} \text{Control}&\text{SR}&\text{ECOD gain (pp)}\\\\ \hline \text{Without competence}&65.33&{\color{green}\uparrow3.13}\\\\ \text{Without failure selection}&66.99&{\color{green}\uparrow1.46}\\\\ \text{Without temporal weighting}&65.33&{\color{green}\uparrow3.13}\\\\ \text{Zero tail set to }0.1&65.82&{\color{green}\uparrow2.64}\\\\ \end{array}$$ Removals match ECOD's weighted-token mass on their batches (A.9). Only zero temporal weights become $0.1$; other weights/masks stay fixed. Tail TS: $78.95$ vs ECOD $80.75$ ($+1.80$). Guidance resumes in $143/150$ updates, adding $3.00\\%$ supervision on its batches; budgets differ. Zero tail marks absent cache coverage, not correction value (4.1, A.17, Table A10). **5. Retraining variability.** Two added seeds per WebShop 1.5B configuration test retraining gains at update 150: $$\begin{array}{l|ccc} \text{SR}&\text{Baseline}&\text{+ECOD}&\text{Gain (pp)}\\\\ \hline \text{SOD}&59.41\pm4.77&71.13\pm6.24&{\color{green}\uparrow11.72}\\\\ \text{TCOD-F2B}&66.67\pm1.57&70.02\pm1.25&{\color{green}\uparrow3.35}\\\\ \text{ATOD}&71.65\pm1.24&74.15\pm3.42&{\color{green}\uparrow2.51}\\\\ \end{array}$$ Pairs share teacher, cache, initialization, task schedule and base coefficients/native schedules. We average eight evaluations per seed, then compute the across-seed mean and sample SD. SOD and TCOD-F2B gain in every seed; all mean gains are positive (A.13, Tables A5-A6). **6. Fairness and reporting.** Direct OPD SR: $71.27\\%\to74.16\\%$ (ALFWorld), $68.26\\%\to68.46\\%$ (WebShop), 1.5B GRPO. The abstract specifies averages and adjusted 7B coefficients; Figure 2 labels relative gains and decoding SD. New 7B WebShop SOD matches the SOD+ECOD base teacher-advantage coefficient $c=4.0$, teacher, initialization, task schedule and update budget: $$\begin{array}{l|ccc} c=4.0&\text{SOD}&\text{+ECOD}&\text{Gain}\\\\ \hline \text{SR}&74.71\pm1.96&82.03\pm1.82&{\color{green}\uparrow7.32}\text{ pp}\\\\ \text{TS}&82.96\pm1.51&90.28\pm1.26&{\color{green}\uparrow7.32}\\\\ \end{array}$$ Matched strength retains the gain (A.15, Table A8). Main-table checkpoints maximize repeated fixed-test success. New retraining/coefficient controls use final update 150; larger-teacher validation-selected results are separate (5.1-5.2, A.8). **7. Schedules and cost.** Fixed schedules cache $150\times16\times8=19{,}200$ teacher trajectories per teacher/schedule for reuse across seeds. Online scoring is shared with OPD; ECOD adds statistics/weights, no learned evaluator. New/changed tasks require fresh task-local statistics (A.7). WebShop endpoints: eight evaluations, 128 tasks, temperature $0.4$, horizon $15$. Outside the three-seed study, SDs reflect decoding of one training seed.

Thank you for the detailed reply and additional experiments, which address some of my concerns. However, I still feel that the effect of ECOD is close to simpler baselines, such as early teacher retirement. And having to collect task-specific statistics in advance makes it fairly hard to use in practice. So I prefer to mantain the score.

AuthorsOpenReview ↗

Follow-up on Remaining Concerns

Dear Reviewer,

Thank you for your thoughtful follow-up and for acknowledging our additional experiments.

Regarding teacher retirement, we agree that a direct comparison would further strengthen the evaluation. However, we would like to emphasize that ECOD does not retire the teacher globally. Instead, it selectively allocates supervision across tasks, trajectories, and turns throughout training. Our matched-budget experiment (74.16% vs. 72.29%) provides supporting evidence that supervision allocation itself contributes to the improvements.

Regarding practicality, we acknowledge the limitation in dynamically changing environments. For fixed training schedules, however, the pre-collected teacher statistics can be reused across training runs, while online teacher scoring is shared with existing OPD methods.

We sincerely appreciate your constructive feedback and will further clarify these distinctions and limitations in the paper.

Reviewer_a8vQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds a weighting scheme to on-policy distillation for multi-turn agents. Teacher supervision is reweighted by three factors: how much better the teacher appears on a task, whether the student's trajectory failed, and which turns the teacher's successful trajectories usually reach. The factors multiply into a single per-token weight that can be plugged into several existing distillation variants, and the paper evaluates it on ALFWorld and WebShop with two student sizes, on top of GRPO, GiGPO, OPD, SOD, TCOD-F2B and ATOD.

Critical strengths
  1. The three-way decomposition is easy to describe and does not require changing the base algorithm, so it is genuinely reusable.

  2. The paper is upfront that the gains vary by setting, including cases where a component is worse than its baseline.

Critical weaknesses
  1. The margins are small and not consistent: on the 7B model one comparison is exactly unchanged, another is negative, and one 1.5B comparison loses over six points. Since each configuration is trained once, the reported spread comes from repeated decoding of a single checkpoint, which does not tell us how much the result would move under retraining.

  2. In the 7B WebShop setting the ECOD runs apparently use adjusted teacher weights while the paired baselines keep their original ones. If that is right, the paired comparison for that block is not a clean comparison, and it is the block where the aggregate gains are as large as they are.

Questions for authors
  1. How large is the run-to-run variation under retraining, as opposed to re-decoding a fixed checkpoint, for the settings where the gain is a few points?

  2. Why do the 7B WebShop ECOD runs use different teacher weights from their paired baselines, and does the comparison hold when they match?

  3. Which of the three factors does the most work, and would the same ordering appear if the factors were applied one at a time rather than as a product?

4 comments
AuthorsOpenReview ↗

Reliability of ECOD Gains, Matched-Coefficient Comparisons, and Factor Contributions

We thank the reviewer for the constructive comments. We added retraining experiments, matched-coefficient controls, and standalone-factor evaluations, and address the questions below point by point.

1. Gain consistency and retraining variability. To address retraining variability, we added two seeds to each of three baseline/ECOD pairs: 12 new training runs.

Across three training seeds, mean success gains are 11.72/3.35/2.51 pp for SOD/TCOD-F2B/ATOD. SOD and TCOD-F2B improve in every seed.

WebShop 1.5B, final update 150, eight evaluations per checkpoint on 128 tasks. Seed columns report SR (%) mean ± decoding SD; the last reports mean ± sample SD across training-seed means. Gains use unrounded values.

\text{Method} & \text{Seed 0} & \text{Seed 1} & \text{Seed 2} & \text{Mean}\pm\text{SD}_{\rm train} \\\\ \hline \text{SOD} & 62.40\pm2.02 & 53.91\pm2.80 & 61.91\pm2.28 & 59.41\pm4.77 \\\\ \text{SOD+ECOD} & 75.88\pm1.06 & 64.06\pm2.00 & 73.44\pm1.02 & 71.13\pm6.24 \\\\ \text{Gain (pp)} & {\color{green}\uparrow13.48} & {\color{green}\uparrow10.16} & {\color{green}\uparrow11.52} & {\color{green}\uparrow11.72} \\\\ \text{TCOD-F2B} & 68.16\pm3.06 & 66.80\pm0.93 & 65.04\pm1.55 & 66.67\pm1.57 \\\\ \text{TCOD-F2B+ECOD} & 69.73\pm2.35 & 68.95\pm1.95 & 71.39\pm1.96 & 70.02\pm1.25 \\\\ \text{Gain (pp)} & {\color{green}\uparrow1.56} & {\color{green}\uparrow2.15} & {\color{green}\uparrow6.35} & {\color{green}\uparrow3.35} \\\\ \text{ATOD} & 71.88\pm1.32 & 72.75\pm1.59 & 70.31\pm1.45 & 71.65\pm1.24 \\\\ \text{ATOD+ECOD} & 70.31\pm1.82 & 75.29\pm0.93 & 76.86\pm0.83 & 74.15\pm3.42 \\\\ \text{Gain (pp)} & {\color{red}\downarrow1.56} & {\color{green}\uparrow2.54} & {\color{green}\uparrow6.54} & {\color{green}\uparrow2.51} \\\\ \end{array}$$ Pairs share teacher, cache, initialization, task schedule and base coefficients/native schedules. All three integrations also improve mean task score (mean ± sample SD across training-seed means): $$\begin{array}{l|ccc} \text{Task score}&\text{Baseline}&\text{+ECOD}&\text{Gain}\\\\ \hline \text{SOD}&70.42\pm5.82&82.69\pm7.32&{\color{green}\uparrow12.27}\\\\ \text{TCOD-F2B}&77.29\pm2.24&82.45\pm0.50&{\color{green}\uparrow5.16}\\\\ \text{ATOD}&83.25\pm1.62&84.89\pm4.10&{\color{green}\uparrow1.64}\\\\ \end{array}$$ Seed-0 ATOD+ECOD uses its original endpoint, not the main-table continuation (Appendix A.13, Tables A5–A6). **2. Fairness of the 7B WebShop comparisons.** To test whether 7B WebShop gains persist at identical coefficients, we added a SOD baseline with base teacher-advantage coefficient $c=4.0$, matching SOD+ECOD. Success and task score both improve by 7.32 points. $$\begin{array}{l|ccc} \text{Metric} & \text{SOD},\\ c=4.0 & \text{SOD+ECOD},\\ c=4.0 & \text{Gain} \\\\ \hline \text{Success (\\%)} & 74.71\pm1.96 & 82.03\pm1.82 & {\color{green}\uparrow7.32}\text{ pp} \\\\ \text{Task score} & 82.96\pm1.51 & 90.28\pm1.26 & {\color{green}\uparrow7.32} \\\\ \end{array}$$ Original ECOD coefficients were adjusted for model scale; baselines kept their defaults. The new pair fixes teacher, initialization, task schedule, optimizer and final update 150 (Appendix A.15, Table A8). The abstract and Section 5.2 specify coefficient settings behind aggregate gains. The matched result concerns SOD at coefficient 4.0. SDs reflect eight decoding evaluations of one training seed. **3. Which factors contribute most, alone and in the full product?** To compare factors, we added competence-only, outcome-only and temporal-only runs (WebShop 1.5B, GRPO). At update 100, competence and outcome selection lead in success (66 versus 65 of 128 tasks); outcome selection leads in task score. $$\begin{array}{l|cc} \text{Standalone factor} & \text{Success (\\%)} & \text{Task score} \\\\ \hline \text{Competence} & 51.56 & 68.95 \\\\ \text{Outcome selection} & 50.78 & 75.49 \\\\ \text{Temporal weighting} & 41.41 & 54.13 \\\\ \end{array}$$ Full ECOD reaches 68.46% success; removing competence, outcome selection or temporal weighting gives 65.33%, 66.99% and 65.33% (losses 3.125, 1.465 and 3.125 pp). Competence and temporal weighting lead in conditional effects (Appendix A.9). Standalone and conditional contributions differ. Standalone results use post-hoc update-100 validation, one seed/evaluation per variant, omitted factors set to one, and no budget rescaling. Removal controls use update 150, matched weighted-token mass, and repeated evaluations. The observed leaders differ across these protocols: competence/outcome alone, competence/temporal within the product (Section 5.3; Appendix A.16, Table A9).
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experimental Results

Dear Reviewer,

Thank you for your valuable comments and suggestions.

We have carefully addressed your concerns in our rebuttal and conducted additional experiments to further support our findings and clarify the points you raised.

We would greatly appreciate it if you could take a look at our responses and the new experimental results. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications or additional evidence.

Thank you again for your time and consideration.

Thank you for your response and the additional experiments. They address most of my concerns, so I will raise my score from 2 to 3.

AuthorsOpenReview ↗

Thank You for Your Feedback

Dear Reviewer,

Thank you very much for your thoughtful follow-up and for reconsidering your assessment. We sincerely appreciate your time, constructive feedback, and recognition of our clarifications and additional experimental results.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.