- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ECOD allocates teacher supervision using three factors: task-conditioned teacher–student success estimates, a gate selecting failed student trajectories, and turn weights derived from successful teacher trajectory lengths. The resulting weight applies to either a teacher loss or a teacher-derived advantage, while all student trajectories remain available for reward-based learning. Experiments on ALFWorld and synthetic WebShop cover two model sizes, several training objectives, and allocation ablations.
Critical strengths
- The allocation rule is simple and modular. Relative task competence, student outcomes and teacher completion lengths each have a clear role, and the mechanism can be applied to both teacher-loss and teacher-advantage formulations without removing successful trajectories from reward-based learning.
- The budget-matched component ablations and uniform-allocation controls provide evidence that supervision placement matters beyond its average strength (Figure 4; Appendices A.9–A.10). These are more informative than comparing only against an unchanged OPD coefficient.
Critical weaknesses
- The 7B WebShop comparisons change teacher strength alongside allocation. Appendix A.7 assigns larger teacher coefficients to ECOD while paired baselines retain their original strengths. These comparisons therefore do not isolate the effect of allocation from coefficient choice. The matched controls are restricted to 1.5B GRPO and do not resolve this confound in the broader comparisons.
- The evidence for task-specific temporal calibration is limited. Equation (7) measures how long successful teacher executions last, rather than the usefulness of teacher feedback on a failed student's current history. It assigns zero supervision beyond the longest cached successful teacher execution, although student and teacher progress need not align by turn index. In Figure 4(a), a pooled temporal prior reaches 68.07% versus 68.46% for the task-specific prior; removing temporal weighting gives 65.33%. These results support temporal attenuation more clearly than the need for task-specific execution-length calibration.
- The incremental contribution over existing outcome-aware selective supervision remains unclear. Appendix A.2 distinguishes ECOD's task-local completion statistics from related selection and reweighting rules, but the experiments do not compare against those alternatives. For example, the cited SERL also studies selective, reward-aware supervision on ALFWorld and WebShop. Component-removal ablations support ECOD within the tested system, but do not establish the added value of its particular combination over prior selective-supervision designs. A comparison controlling for available teacher feedback would better establish that contribution.
The allocation mechanism and controlled ablations are promising, but the coefficient mismatch prevents attribution of the 7B WebShop results to allocation alone. Evidence for the added value of task-specific temporal calibration is also limited.
Questions for authors
- Do the 7B WebShop gains remain when paired methods use identical teacher coefficients or the same coefficient-tuning grid?
- Given the pooled-prior result, what supports task-specific temporal calibration over a shared decay, particularly the zero supervision weight on late failed-student states?
- Under comparable teacher feedback and training budgets, how does ECOD compare with prior outcome-aware selective-supervision rules or a controlled adaptation of them?
4 comments
Follow-up on Our Rebuttal and Additional Experimental Results
Dear Reviewer,
Thank you for your valuable comments and suggestions.
We have carefully addressed your concerns in our rebuttal and conducted additional experiments to further support our findings and clarify the points you raised.
We would greatly appreciate it if you could take a look at our responses and the new experimental results. If you have any remaining questions or concerns, we would be more than happy to provide further clarifications or additional evidence.
Thank you again for your time and consideration.
Thanks for the reply and added experiments. The matched 7B control and zero-tail ablation is helpful for my concern, but the improvement of task-specific prior over pooled one is still quite small, and there is still no direct comparison with other outcome-aware methods like SERL. So I will keep my score.
Clarification on Incremental Contribution and Temporal Calibration
Thank you for your thoughtful follow-up and for recognizing our additional experimental controls.
We would like to clarify the scope of our contribution. ECOD is designed as a reusable supervision-allocation mechanism that complements existing OPD objectives. Accordingly, our primary evaluation examines whether ECOD provides additional benefits to established methods that already employ adaptive teacher guidance.
We directly evaluate this with SOD, TCOD-F2B, and ATOD, which use step-wise divergence weighting, a trajectory-depth curriculum, and disagreement/uncertainty-based weighting, respectively. Across three training seeds, adding ECOD yields positive mean success-rate gains of 11.72, 3.35, and 2.51 %, respectively. Furthermore, on ALFWorld, ECOD outperforms uniform allocation under the same per-update weighted-token budget by 1.87 %. These controlled comparisons provide evidence for ECOD's incremental value across the tested adaptive OPD methods.
Regarding SERL, its primary mechanism reweights reward-driven advantages using hindsight-conditioned teacher likelihoods, alongside an action-only distillation term. ECOD instead calibrates a separate teacher contribution using task-relative competence, student outcomes, and successful-teacher execution lengths. We acknowledge that a direct comparison with SERL would help further characterize their relative strengths, while our existing experiments directly support the claimed incremental benefits on SOD, TCOD-F2B, and ATOD.
Finally, we agree that the task-specific prior provides only a modest improvement over pooling (+0.39 pp). However, both temporal priors outperform removing temporal weighting (68.46% and 68.07% vs. 65.33%), providing stronger evidence for the broader temporal-allocation mechanism.
Thank you again for your careful evaluation and constructive feedback.
Matched Teacher Coefficients, Temporal Calibration, and Complementary Supervision
We thank the reviewer for the constructive comments and recognition of the allocation controls. We added matched-coefficient and temporal-tail experiments, and respond point by point.
1. Do the 7B gains remain at identical teacher coefficients? We trained a new 7B WebShop SOD baseline with base teacher-advantage coefficient c=4.0, matching existing SOD+ECOD. The matched pair shares the frozen teacher, initialization, task schedule, optimizer, and final update 150. Coefficient pairs below are (SOD, SOD+ECOD); SR denotes success rate and TS task score.
\text{Metric}&\text{Coefficients}&\text{SOD}&\text{SOD+ECOD}&\text{Gain}\\\\ \hline \text{SR (\\%)}&(1.0,4.0)&79.88\pm2.04&82.03\pm1.82&{\color{green}\uparrow2.15}\\\\ \text{SR (\\%)}&(4.0,4.0)&74.71\pm1.96&82.03\pm1.82&{\color{green}\uparrow7.32}\\\\ \text{TS}&(1.0,4.0)&87.50\pm1.14&90.28\pm1.26&{\color{green}\uparrow2.78}\\\\ \text{TS}&(4.0,4.0)&82.96\pm1.51&90.28\pm1.26&{\color{green}\uparrow7.32}\\\\ \end{array}$$ Tables report mean ± sample SD; pp denotes percentage points. The original 1.0/4.0 pair provides context (A.7); the 4.0/4.0 control adds a SOD baseline and reuses the ECOD endpoint (A.15, Table A8). Here 4.0 is the coefficient value. Matched SR and TS gains are 7.32 pp and 7.32 points. Base strength is matched; ECOD sets effective weighted supervision. The control uses final update 150; original main-table selection used repeated test scores (A.8). **2. What supports task-specific timing and zero late supervision?** Task conditioning retains each task's successful-execution profile. In the group with 8 teacher and 7 student successes, mean successful-teacher lengths span 4.5-10.5 turns (A.1). Pooling averages these profiles. Full ECOD reaches 68.46% SR, pooled 68.07%, and no temporal weighting 65.33%: a 0.39 pp task-specific gain, with both priors above the no-temporal control (A.9). We tested late guidance on WebShop 1.5B GRPO, changing only zero temporal weights to 0.1. Positive weights, competence/outcome masks, teacher assets, teacher-KL coefficient 0.02, reference-KL coefficient 0.01 and training protocol stay fixed. $$\begin{array}{l|cc} \text{Final update 150}&\text{SR (\\%)}&\text{TS}\\\\ \hline \text{Original ECOD}&68.46\pm1.38&80.75\pm1.71\\\\ \text{Zero tail set to }0.1&65.82\pm2.12&78.95\pm1.79\\\\ \text{ECOD gain}&{\color{green}\uparrow2.64}\text{ pp}&{\color{green}\uparrow1.80}\\\\ \end{array}$$ Guidance resumes in 143/150 updates, adding 3.00% weighted supervision relative to the original rule on the control's own batches. Base coefficients are matched; effective budgets differ (A.17, Table A10). The 2.64 pp SR and 1.80 TS gains support the zero-tail rule in this setting. The prior measures execution-length coverage, not state-specific correction value. Teacher feedback uses the student's actual history; reward learning continues beyond cached support (4.1, A.3). **3. What is the added value over outcome-aware selection?** Paired comparisons preserve existing adaptive mechanisms (A.3, A.6): SOD uses divergence-based step weights; ATOD combines disagreement/uncertainty weights with OPD-RL annealing; TCOD-F2B uses a depth curriculum. ECOD jointly adds task-local relative competence, student outcomes and teacher completion lengths. We added two training seeds per WebShop 1.5B configuration. Each baseline/ECOD pair shares teacher, initialization, task schedule, base coefficients/native schedules and the 150-update budget. Three-seed SR at update 150: $$\begin{array}{l|ccc} \text{SR (\\%)}&\text{Baseline}&\text{+ECOD}&\text{Gain (pp)}\\\\ \hline \text{SOD}&59.41\pm4.77&71.13\pm6.24&{\color{green}\uparrow11.72}\\\\ \text{TCOD-F2B}&66.67\pm1.57&70.02\pm1.25&{\color{green}\uparrow3.35}\\\\ \text{ATOD}&71.65\pm1.24&74.15\pm3.42&{\color{green}\uparrow2.51}\\\\ \end{array}$$ Eight evaluations are averaged per seed; mean ± sample SD is then computed across training-seed means. Gains use unrounded values. SOD and TCOD-F2B improve in every seed. Mean TS gains are SOD +12.27, TCOD-F2B +5.16 and ATOD +1.64 (A.13, Tables A5-A6), supporting added value on all three tested adaptive objectives. On ALFWorld unseen, uniform allocation replaying ECOD's per-update weighted-token budget reaches 72.29% SR versus ECOD's 74.16% (+1.87 pp; A.10, Table A4), supporting supervision placement at matched budgets. SERL is related outcome-aware work (A.2); our paired evidence evaluates complementary calibration on SOD, TCOD-F2B and ATOD. New WebShop endpoints use update 150 and eight evaluations on 128 tasks, temperature 0.4 and horizon 15. Coefficient/tail comparisons use one training seed and report decoding SDs; the three-seed table reports retraining SDs.