acceptodds
Under review as a conference paper at ICLR 2027

DecisionOPD: Allocating Teacher Views and State Coverage for Decision Distillation

Abstract

Training compact language agents requires effective use of costly teacher supervision. Additional queries can refine targets for familiar decisions or extend supervision to new states, and the value of either choice depends on the training available to the student. We introduce DecisionOPD, a framework for comparing these allocations through executable action distributions while accounting for teacher cost and student exposure. The resulting comparisons connect supervision quality and breadth to downstream policy performance. Refined targets retain an empirical advantage over broader coverage as training increases, although the size of that advantage remains uncertain. On ALFWorld, aligned mean targets improve task success from 38.9% to 50.0% under matched state coverage and student updates.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, LLM Agents, Dynamic Action Sets, Non-Generative Policies, Interactive Imitation Learning, Sequential Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i1JoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces DecisionOPD, a framework for analysing the tradeoff between refining targets for the same states or extending supervision to new states when distilling a teacher decision-making model into a student one. The underlying problem is that teacher models, when presented with a menu of options, will sometimes predict a different option depending on the order in which they are presented.

Critical strengths
  1. The comparison between refinement and coverage is well-controlled and effectively answers the question asked by the authors.

  2. The results are presented quite transparently, even in cases that go against the main thesis of the paper.

Critical weaknesses
  1. The idea of spending compute to re-evaluate seen states (with reversed choices) is not new (as noted by the authors), so unless I am mistaken, it seems like this work’s contribution is just analyzing this tradeoff under a fixed teacher evaluation budget?

  2. The scope is quite limited; the paper only examines one environment, one main teacher/student pair, and one initialization in the main study.

  3. The paper mentions exact McNemar tests (with Holm correction), but I could not find these results anywhere.

  4. The main results also seem very noisy. For example, Figure 2 shows that, in the high-budget setting, 2 teacher views per decision seems to do worse than 1 under a fixed budget on a different set of 72 tasks than those shown in the main paper. The paper even notices in section A.3 that repeated experiments changed nine task outcomes in each method.

  5. Figure 1 shows two line plots with only 2 x-axis ticks each. This should have simply been a table. The “Matched re-execution" shown in the plots is also not sufficiently explained.

  6. Most of the appendix also reads like an agent’s research log, including a ton of information that is not relevant to the paper.

  7. Perhaps most alarmingly, the “Limitations” section does not list a single limitation. It instead simply lists avenues for future work.

Questions for authors

Despite the explanation regarding the data stream prefix, it seems odd to not make the number of visits in the single-view extended setting double the number of visits of the single-view and two-view mean settings, especially if you are trying to show that two-view mean is superior under the same teacher budget.

Could you include results using exactly double the number of visits, even if this means you have to lower the base number of visits?

5 comments

We thank the Reviewer for noting that our experimental comparisons are well-controlled and transparent, and for the specific feedback on the visit-count ratio (1,544 vs. 1,600 visits), the missing McNemar test table, figure formatting, appendix structure, and the limitations discussion. We address every point below.

Q1 & Q2: Why Arm C uses 1,544 visits under token matching, and comparison with the earlier 1,600-visit single-view checkpoint (Section 5.5 and Appendix C.1, Tables 8 and 9):

  • Why strict teacher-token matching stops at 1,544 visits (Section 3.4 & Appendix C.1): In ALFWorld, prompt token length ℓT(Pg(xi,C(xi)))\ell_T(P_g(x_i, C(x_i))) increases as visible interaction history hih_i accumulates over an episode. On our shared stream, querying two views on the first 800 visits (Arm B) costs exactly 1,511,7261{,}511{,}726 teacher-input tokens (2×755,8632 \times 755{,}863). Because visits between indices 801 and 1,600 have slightly longer average histories than visits 1 to 800, taking 1,600 single-view visits costs 1,566,8741{,}566{,}874 teacher-input tokens, exceeding Arm B's token budget by 55,14855{,}148 tokens (+3.648%+3.648\%). To keep the equal-budget comparison strictly fair in teacher-input tokens, Arm C takes the longest contiguous single-view prefix within Arm B's token budget: 1,544 visits at 1,510,5541{,}510{,}554 tokens (within 1,1721{,}172 tokens or 0.078%0.078\% of Arm B, as Visit 1,545 requires 1,3591{,}359 tokens).
  • Results with an exact 2×2\times visit count (1,600 single-view visits vs. 800 / 824 two-view visits, and 3,200 vs. 1,609 visits — Section 5.5 & Appendix C.1, Tables 8 & 9): We also trained and evaluated a 1,600-visit single-view policy (1,566,8741{,}566{,}874 teacher tokens, exactly 2×2\times the 800 visits of Arms A and B) and report its results across both disjoint 72-task evaluation sets in Section 5.5 and Appendix C.1 (Tables 8 & 9): | Configuration | Visits | Views / Visit | Teacher Queries | Teacher Input Tokens | Main 72-Task Set (/72) | Historical 72-Task Set (/72) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | Core Arm A (Single-view base) | 800 | 1 | 800 | 755,863 | 28/72 (38.9%) | --- | | Core Arm B (Two-view aligned mean) | 800 | 2 | 1,600 | 1,511,726 | 36/72 (50.0%) | 39/72 (54.2%) | | Core Arm C (Token-matched single-view) | 1,544 | 1 | 1,544 | 1,510,554 | 26/72 (36.1%) | 36/72 (50.0%) | | Exact 2×2\times-Visit Single-View (1,600 visits) | 1,600 | 1 | 1,600 | 1,566,874 | 28/72 (38.9%) | 38/72 (52.8%) | | Earlier Low-Budget Two-View (824 visits) | 824 | 2 | 1,648 | 1,565,128 | 39/72 (54.2%) | 37/72 (51.4%) | | Earlier High-Budget Single-View (2×2\times 1,600) | 3,200 | 1 | 3,200 | 3,149,070 | --- | 41/72 (56.9%) | | Earlier High-Budget Two-View (≈1,600\approx 1{,}600) | 1,609 | 2 | 3,218 | 3,148,040 | --- | 33/72 (45.8%) |
    • On the Main 72-Task Set, the 1,600-visit single-view policy (despite using +3.65%+3.65\% more teacher tokens than Arm B) solves 28/72 (38.9%), trailing 800-visit two-view Arm B (36/72, 50.0%) by 8 tasks and trailing 824-visit two-view (39/72, 54.2%) by 11 tasks.
    • On the Historical 72-Task Set, the 1,600-visit single-view policy solves 38/72 (52.8%) compared with 39/72 (54.2%) for 800-visit two-view Arm B and 37/72 (51.4%) for 824-visit two-view. The exact 1,600-to-800 comparison uses checkpoints from different training protocols. Within the common protocol, 800 two-view visits matched the teacher-input cost of 1,544 single-view visits.

W3: Exact McNemar test results (with Holm correction)

Thank you for pointing out this omission. We have included the complete paired statistical table in Appendix B (Table 7) (referenced in Section 5.1), reporting discordant exclusive win pairs (L/R only), family-stratified 95% paired bootstrap confidence intervals, two-sided exact binomial McNemar pp-values, and Holm step-down adjusted pp-values:

  • Arm B vs. Arm A (36/72 vs. 28/72): 11 / 3 exclusive wins, net +8+8 tasks (+11.11+11.11 pp), 95% bootstrap CI [+1.39,+20.83][+1.39, +20.83] pp, exact McNemar p=0.0574p = 0.0574, Holm-adjusted p=0.1147p = 0.1147.
  • Arm B vs. Arm C (36/72 vs. 26/72): 13 / 3 exclusive wins, net +10+10 tasks (+13.89+13.89 pp), 95% bootstrap CI [+4.17,+23.61][+4.17, +23.61] pp, exact McNemar p=0.0213p = 0.0213, Holm-adjusted p=0.0638p = 0.0638.
  • Arm N vs. Arm A (33/72 vs. 28/72): 7 / 2 exclusive wins, net +5+5 tasks (+6.94+6.94 pp), 95% bootstrap CI [+0.00,+15.28][+0.00, +15.28] pp, exact McNemar p=0.1797p = 0.1797, Holm-adjusted p=0.1797p = 0.1797.
  • Supporting contrasts: F vs. A (32 vs. 28, 6/2 exclusive wins, CI [−1.39,+12.50][-1.39, +12.50] pp), B vs. F (36 vs. 32, 7/3 exclusive wins, CI [−2.78,+13.89][-2.78, +13.89] pp), B vs. N (36 vs. 33, 6/3 exclusive wins, CI [−2.78,+12.50][-2.78, +12.50] pp), and C vs. A (26 vs. 28, 4/6 exclusive wins).

W1, W2, W6, & W7: Clarifying contribution, rewriting the Appendix, and stating explicit limitations

  • Contribution beyond re-evaluating seen states: In Section 1, Section 3.3 (Equation 4), and Section 5.2, we show that our work establishes (i) why multi-view averaging improves sequential policies even when zero top-1 teacher actions change (Arm N, raising completion from 28/72 to 33/72 with identical top-1 labels, whereas one-step greedy action substitutions leave episode return unchanged on 82.35%–91.18%82.35\%\text{--}91.18\% of heldout switch points), and (ii) how per-record student exposure governs the trade-off between view refinement and state coverage as student updates scale from 200 to 400.
  • **Broader empirical evidence in the main text (Tables 3, 4, 5 & Figure 4c):**Results from our two historical initializations (Init. I and Init. II), online learner-refreshed collection runs (Table 4), 1,152-state menu stability diagnostics (Table 3), Mistral-Small-3.2-24B cross-teacher results (Section 5.6), and PIQA / HellaSwag / ARC-Challenge evaluations (Table 5) into the main text.
  • Rewritten Appendix (Appendices A–F) and explicit limitations (Section 6): We removed all research-log text and internal run identifiers from the appendix, restructuring Appendices A–F into six self-contained technical sections (Tables 6–13). In Section 6 (Discussion), we explicitly state our empirical boundaries: the equal-cost coverage comparison (Arm B vs. Arm C) uses one shared frozen stream and one initialization (seed 42 with data-order permutations 43 and 44); interactive evaluations use 72 tasks (with Holm-adjusted McNemar pp-values 0.0638–0.17970.0638\text{--}0.1797 in Table 7); cross-evaluating frozen checkpoints on a disjoint 72-task set narrows the B−CB-C gap from 10 tasks to 3 and reverses an earlier low-budget pair; and higher teacher budgets (≃3.15M\simeq 3.15\text{M} tokens) or extended student training (400 updates) shift the balance toward broader single-view coverage.
Reviewer_Qu5EOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the allocation of teacher supervision in decision-level knowledge distillation. The proposed approach obtains teacher distributions over executable actions under different option orderings, aligns and averages these distributions, and trains a student using reverse KL. The main experiments compare allocating additional teacher queries to existing states against obtaining supervision for more states, under matched teacher-input budgets.

The authors report improvements in ALFWorld task success from 28/72 to 36/72 when using two-view averaged targets on the same 800 states, and from 26/72 to 36/72 compared with broader single-view supervision at approximately equal teacher cost.

Critical strengths

The paper addresses a relevant practical question in knowledge distillation: how to allocate a limited teacher-supervision budget between improving existing targets and increasing state coverage. The experimental setup includes controlled comparisons under approximately matched teacher-query costs and student training budgets.

Critical weaknesses

1. Limited methodological novelty.

The core approach consists of querying a teacher under different option permutations, averaging the aligned distributions, and distilling the resulting targets using an existing reverse-KL objective. Permutation-based teacher debiasing and subsequent student distillation have already been investigated by Liusie et al. (2024).

The primary contribution therefore appears to be an empirical evaluation of an existing technique in an agent setting, rather than a substantial methodological advancement. The paper should more clearly articulate what new insight or capability is established beyond prior work.

2. Limited connection to standard on-policy distillation.

Despite the DecisionOPD framing, the main experiments rely on a frozen collection of states generated by an earlier behavior policy, rather than continuously collecting trajectories from the evolving student.

Although learner-refreshed experiments are included in the appendix, the paper does not establish the benefits of the proposed allocation strategy against a sufficiently controlled, strong online OPD baseline. It remains unclear whether the conclusions generalize to conventional on-policy training, where the student continuously encounters new states.

3. Limited and inconsistent empirical evidence.

The main results are based on only 72 ALFWorld tasks, and none of the three primary comparisons reaches statistical significance after Holm correction. Moreover, Appendix A.1 reports settings where single-view supervision outperforms two-view averaging (41/72 vs. 33/72), while Appendix A.4 shows that the ranking can reverse depending on the evaluation task set. These findings raise concerns about the robustness and generalizability of the conclusions.

4. Narrow evaluation and incomplete characterization of the tradeoff.

The strongest results are restricted to ALFWorld, while the improvements on PIQA, HellaSwag, and ARC-Challenge are considerably smaller.

Additionally, the analysis primarily matches teacher-input tokens and student update counts rather than complete training costs. The smaller state pool also receives greater repeated exposure during optimization.

References

[1] Liusie, A., Fathullah, Y., & Gales, M. J. F. (2024). Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024, pp. 1376–1387. https://doi.org/10.18653/v1/2024.findings-acl.81

Questions for authors
  1. Given the overlap with Liusie et al. (2024), what is the main methodological contribution beyond applying permutation-based teacher debiasing to executable actions?

  2. Why does the main evaluation focus on a fixed state dataset rather than conventional on-policy distillation? How would the proposed approach compare against a strong online OPD baseline under matched computational budgets?

  3. How do the authors reconcile the main findings with the inconsistent results in Appendix A.1 and the sensitivity to evaluation task selection in Appendix A.4?

  4. How well do the conclusions generalize to other interactive agent benchmarks, larger training budgets, and independent training seeds?

5 comments

We thank the Reviewer for the rigorous evaluation of our paper's technical contribution, experimental protocol, and empirical consistency. Below we address each question and concern in detail.

Q1 & W1: What new insight and capability does this work establish beyond applying permutation-based teacher debiasing (Liusie et al., 2024) to action menus?

  1. Multi-view refinement improves sequential control even when zero top-1 teacher actions change, whereas single-step greedy action corrections rarely alter episode return (Section 3.3, Equation 4, Section 5.2, Figure 3c, Table 1):

    • A natural hypothesis from static debiasing is that averaging two menu views helps the student because it flips erroneous top-1 teacher labels on order-sensitive states. Arm N tests whether task completion can improve while all top-choice training labels remain fixed.
    • First, in Section 3.3 (Equation 4), we derive the exact reverse-KL logit gradient shift when replacing a single-view distribution qidq_{\rm id} with an aligned mean qmeanq_{\rm mean} on decisions where both share the exact same winning action (arg⁡max⁡aqmean(a)=arg⁡max⁡aqid(a)\arg\max_a q_{\rm mean}(a) = \arg\max_a q_{\rm id}(a)): ∂∂za[DKL(p∥qmean)−DKL(p∥qid)]=pa(ra−∑c∈C(xi)pcrc),ra=log⁡qid(a)qmean(a).\frac{\partial}{\partial z_a}\big[D_{\mathrm{KL}}(p\|q_{\rm mean}) - D_{\mathrm{KL}}(p\|q_{\rm id})\big] = p_a\Big(r_a - \sum_{c\in C(x_i)} p_c r_c\Big), \quad r_a = \log\frac{q_{\rm id}(a)}{q_{\rm mean}(a)}. Even when the winning action is fixed, reweighting runner-up commands (as illustrated at Visit 11 in Section 5.2 & Figure 3c, where go to cabinet 2 remains top-1 at 0.20830.2083 while move ladle 1 to cabinet 1 rises nearly sixfold from 0.02030.0203 to 0.11700.1170) reshapes the log ratios rar_a and alters the student's gradient updates even when the winning action stays fixed.
    • Second, we isolate this effect causally on the 800 shared ALFWorld visits (Table 1, Figure 3c, Figure 4a). Across the 800 visits, original and reversed views agree on the top-1 action at 425 visits and disagree at 375 visits (175 where averaging preserves the identity winner and 200 where averaging flips the winner, subset F\mathcal{F}). In Arm N, we apply two-view mean targets exclusively on the ∣N∣=600|\mathcal{N}|=600 visits where the winning action is preserved (425+175425+175) and keep single-view targets qidq_{\rm id} on F\mathcal{F}. Even though all 800 top-1 training labels in Arm N are identical to single-view Arm A, Arm N raises closed-loop task completion from 28/72 (38.9%) to 33/72 (45.8%) (7/2 exclusive wins), compared with 32/72 for changed-winner-only refinement (Arm F) and 36/72 for full refinement (Arm B).
    • Third, in our 240-point counterfactual execution probe (Section 5.2 & Appendix D.2, executing 395 continuation branches per policy: 146 on 96 calibration points and 249 on 144 heldout points), replacing the single-view greedy action aida_{\rm id} with the two-view action ameana_{\rm mean} for a single step under a frozen continuation policy leaves episode return unchanged on 31/34 (91.18%) of heldout switch points under the ID-mean continuation and 28/34 (82.35%) under the SFT continuation. This demonstrates that multi-view supervision operates by shaping the full action distribution across policy optimization rather than through isolated one-step action corrections.
  2. Coupling between teacher view allocation, state coverage, and per-record student exposure (Section 3.2, Equation 2, Section 5.3, Table 2, Figure 4b):

    • In sequential decision distillation, spending a teacher query on a second view of an existing state trades off directly against labeling a new state along the interaction stream.
    • DecisionOPD shows that this trade-off cannot be evaluated solely in terms of teacher queries because expanding state coverage ∣D∣|\mathcal{D}| at a fixed student update budget UU dilutes the mean training exposure Ub/∣D∣Ub/|\mathcal{D}| each cached decision receives. At 200 updates (3,200 presentations) and matched teacher cost (≃1.51M\simeq 1.51\text{M} tokens), 800 two-view visits (Arm B) receive 44 passes per record whereas 1,544 single-view visits (Arm C) receive only 2–32\text{--}3 passes, yielding a 10-task lead for Arm B (36/72 vs. 26/72). Extending student optimization on the cached targets to 400 updates (6,400 presentations, zero new teacher calls) raises exposure on the 1,544-visit pool to 4–54\text{--}5 passes, allowing broader coverage to close the gap in Data Order 43 (35/72 vs. 35/72) and narrow it from +8+8 to +6+6 tasks in Data Order 44 (40/72 vs. 34/72, Table 2 & Figure 4b).

Q2 & W2: Why the primary budget comparison uses a frozen stream, and how the approach performs under online learner-refreshed distillation

  • Why the equal-cost B/C comparison freezes the behavior stream: To test whether spending ≃1.51M\simeq 1.51\text{M} teacher tokens on two views of 800 visits (Arm B) outperforms one view of 1,544 visits (Arm C), both allocations must draw from the same underlying trajectory distribution. If Arm B and Arm C collected trajectories online from their respective evolving policies during training, any performance difference would confound (i) target quality and per-state exposure with (ii) diverging state distributions induced by intermediate student policies. Freezing the shared SFT collection stream ensures that Arms A, B, F, and N share the exact same 800 visits and Arm C extends the exact same stream to 1,544 visits within 0.078%0.078\% teacher-input token cost (1,510,5541{,}510{,}554 vs. 1,511,7261{,}511{,}726 tokens).
  • Controlled online learner-refreshed distillation experiments (Main Text Section 5.5, Table 4, Figure 4c, and Appendix B): To evaluate how multi-view target refinement performs when the current student collects fresh trajectories at the start of each training round, we brought our learner-refreshed collection study into the main text (Section 5.5, Table 4, and Figure 4c). Across two independent initializations (Init. I and Init. II), we compare fixed-pool training against online learner-refreshed collection across three distillation objectives (soft single-view identity, Hard CE on the aligned winner, and soft two-view aligned mean):
    • Under online learner-refreshed collection, soft two-view aligned means outperform both soft single-view identity targets and Hard CE in both initializations: solving 38/72 vs. 33/72 (single-view) and 33/72 (Hard CE) in Init. I (8/3 exclusive wins over single-view) and 41/72 vs. 32/72 (single-view) and 35/72 (Hard CE) in Init. II (11/2 exclusive wins over single-view), raising mean task completion from 45.1% (single-view) and 47.2% (Hard CE) to 54.9% (two-view mean).
    • For the two-view students, switching from a fixed pool to round-by-round collection raised completion from 36 to 38 tasks in Init. I and from 37 to 41 in Init. II. Table 3 compares the training targets within each collection regime. With learner-refreshed collection in Init. II, mean TV was 0.326 for the single-view student and 0.232 for the two-view student. Their top-action consistency was 60.2% and 72.2%, respectively.

Q3 & W3: Reconciling the 200-update results with the high-budget setting (41/72 vs. 33/72), task-set sensitivity, and statistical tests

Rather than being contradictory, these observations define the boundary conditions of the refinement-versus-coverage trade-off, which we now analyze in Section 5.3, Section 5.5, and Section 6:

  1. Interaction between teacher budget scale and student exposure (Table 2, Figure 4b, and Appendix C Table 8):
    • At the ≃1.51M\simeq 1.51\text{M}-token teacher budget and 200 updates, 800 two-view visits (Arm B, 44 passes/record) outperform 1,544 single-view visits (Arm C, 2–32\text{--}3 passes/record) on both evaluation suites (36/72 vs. 26/72 on the main set; 39/72 vs. 36/72 on the historical set, Table 9).
    • When student updates double to 400 on the cached targets (Table 2 & Figure 4b), the 1,544-visit pool receives 4–54\text{--}5 passes/record and improves in both data orders (27→3527\to 35 and 32→3432\to 34), closing the gap in Order 43 (35/72 vs. 35/72) and narrowing it in Order 44 (40/72 vs. 34/72).
    • When the teacher budget doubles to ≃3.15M\simeq 3.15\text{M} tokens at 200 updates (Table 8), single-view supervision covers 3,200 visits (4×4\times the 800-visit pool) and reaches 41/72, outperforming two-view supervision over 1,609 visits (33/72). Together, these regimes show that two-view refinement is advantageous when teacher queries and student optimization steps are tight, while broader state coverage catches up or takes the lead when either student optimization passes or total covered trajectories double.
  2. Sensitivity to evaluation task composition on frozen checkpoints (Table 9):
    • Cross-evaluating identical frozen checkpoints across the two disjoint 72-task sets (Table 9) shows that changing the evaluation task population alone shifts the B−CB-C gap from +10+10 tasks (36/72 vs. 26/72) to +3+3 tasks (39/72 vs. 36/72) and reverses the earlier low-budget pair (38/72 vs. 37/72 on the historical set vs. 28/72 vs. 39/72 on the main set). Highlighting this cross-evaluation in Section 5.5 ensures that conclusions account for evaluation-suite composition.
  3. Paired uncertainty and multi-run consistency (Appendix B, Table 7):
    • On the 72 core tasks, paired family-stratified bootstrap 95% confidence intervals exclude zero for both B−AB-A ([+1.39,+20.83][+1.39, +20.83] pp, 11/3 exclusive wins) and B−CB-C ([+4.17,+23.61][+4.17, +23.61] pp, 13/3 exclusive wins), while two-sided exact McNemar pp-values (0.05740.0574 and 0.02130.0213) adjust to 0.11470.1147 and 0.06380.0638 under Holm correction across three primary contrasts. Across all 9 paired 200-update comparisons in our study—three GPU executions of B−AB-A (+8, +6, +7 tasks), two data-order permutations of B−CB-C (+14, +8 tasks), and four historical fixed/refreshed comparisons (+7, +4, +5, +9 tasks)—two-view refinement improves task completion in every run.
Reviewer_Kg5mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DecisionOPD studies whether teacher-query budget should improve targets for existing decision states or expand state coverage. It represents supervision as categorical distributions over executable actions, aligns distributions from original and reversed action presentations, and distills their mean into a smaller student. The main controlled ALFWorld study uses a frozen behavior stream and compares one-view supervision on 800 visits, two-view supervision on the same 800 visits, and approximately input-token-matched one-view supervision on 1,544 visits. Additional analyses examine argmax-changing versus unchanged targets, execution/device robustness, longer student training, static benchmarks, and a second student family.

Critical strengths
  1. The experiments carefully account for teacher-input-token cost and training exposure, with detailed device-level robustness checks. Same-pool and expanded-pool comparisons separate target refinement from greater state coverage.

  2. Argmax-unchanged interventions suggest that the full target distribution matters beyond correcting top actions. Entropy-matched controls and longer-training follow-ups strengthen the analysis, while less favorable results are reported transparently.

Critical weaknesses
  1. The main evidence uses 72 tasks under largely shared training conditions. Device and data-order checks are useful, but do not establish generalization to new tasks or streams; independent replication on disjoint data and paired uncertainty estimates remain important.

  2. The allocation findings are setting-dependent: historical rankings vary, and static-benchmark gains are small or absent. This limits their generality, although the controlled frozen-stream comparison remains informative; the conclusions should stay within this demonstrated scope.

Questions for authors
  1. Can you report paired confidence intervals, test sidedness, and adjusted p-values for the primary comparisons? This would clarify the strength of the evidence while keeping task-level uncertainty separate from variation across training runs.

  2. Do the main gains persist with a fresh behavior stream, independent initialization, or disjoint evaluation set? A focused replication would help determine whether the allocation advantage is specific to the current stream and task set.

  3. How should readers interpret the different allocation rankings across settings? Please clarify which conclusions the existing evidence supports; a predictive allocation rule would be useful future work, not a requirement for this paper.

4 comments

We thank the Reviewer for the constructive feedback on paired statistical uncertainty, replication across behavior streams and task sets, and interpreting ranking shifts across experimental regimes.

Q1: Paired confidence intervals, test sidedness, and adjusted pp-values for the primary comparisons

In the revised manuscript, Table 1 reports the discordant exclusive task win counts (vs. A (L/R)) for all five core arms on the 72-task benchmark, and Appendix B (Table 7) provides the complete paired uncertainty analysis. We reproduce the full table below:

ContrastSolved (/72)Discordant Wins (L / R only)Net Gain (Percentage Points)95% Paired Bootstrap CI (pp)Two-Sided Exact McNemar ppHolm-Adjusted pp (3 Primary Contrasts)
Primary pre-specified contrasts
B −- A (2-view mean vs. 1-view ID, 800 visits)36 vs. 2811 / 3+8+8 (+11.11+11.11 pp)[+1.39, +20.83][+1.39,\ +20.83]0.05740.05740.11470.1147
B −- C (2-view 800 vs. 1-view 1,544, equal cost)36 vs. 2613 / 3+10+10 (+13.89+13.89 pp)[+4.17, +23.61][+4.17,\ +23.61]0.02130.02130.06380.0638
N −- A (Preserved-argmax mean vs. 1-view ID)33 vs. 287 / 2+5+5 (+6.94+6.94 pp)[+0.00, +15.28][+0.00,\ +15.28]0.17970.17970.17970.1797
Supporting selective-intervention & coverage contrasts
F −- A (Changed-argmax mean vs. 1-view ID)32 vs. 286 / 2+4+4 (+5.56+5.56 pp)[−1.39, +12.50][-1.39,\ +12.50]------
B −- F (Full 800 mean vs. changed-argmax mean)36 vs. 327 / 3+4+4 (+5.56+5.56 pp)[−2.78, +13.89][-2.78,\ +13.89]------
B −- N (Full 800 mean vs. preserved-argmax mean)36 vs. 336 / 3+3+3 (+4.17+4.17 pp)[−2.78, +12.50][-2.78,\ +12.50]------
C −- A (Expanded 1-view 1,544 vs. 1-view 800)26 vs. 284 / 6−2-2 (−2.78-2.78 pp)---------
  • Sidedness and resampling design: To respect the family-balanced benchmark structure (12 heldout tasks per family across all 6 ALFWorld families), each of 10,000 bootstrap replicates (seed 42) resamples 12 tasks with replacement within each family and evaluates all policies on the identical resampled task indices. All McNemar tests are two-sided exact binomial tests on the discordant task pairs, with Holm step-down multiplicity correction across the three primary contrasts (B−AB-A, B−CB-C, N−AN-A).
  • Task-level uncertainty vs. run-to-run variation: Conditional on the evaluated checkpoints, the 95% paired bootstrap intervals exclude zero for B−AB-A ([+1.39,+20.83][+1.39, +20.83] pp) and B−CB-C ([+4.17,+23.61][+4.17, +23.61] pp), while the two-sided exact McNemar pp-values (0.05740.0574 and 0.02130.0213) adjust to 0.11470.1147 and 0.06380.0638 after Holm correction, reflecting the statistical power boundary of a 72-task binary evaluation. As discussed in Appendix B, these paired intervals measure task-level uncertainty for fixed checkpoints, whereas run-to-run optimization variation is evaluated separately through our hardware re-executions, data-order permutations (43 and 44), and independent initializations (I and II).

Q2: Do the main gains persist with a fresh behavior stream, independent initialization, or disjoint evaluation set?

We consolidated three sets of results in Section 5.1, Section 5.5, Table 4, Figure 4c, and Appendix C (Table 9) to show how target refinement behaves across fresh rollouts, independent initializations, and disjoint task sets:

  1. Independent initializations (Init. I and Init. II) and online learner-refreshed streams (Table 4 & Figure 4c):
    • We evaluated single-view identity targets, hard cross-entropy (Hard CE on the aligned winner), and soft two-view aligned mean targets across two earlier independent initializations (Init. I and Init. II) under both fixed-pool collection and online learner-refreshed collection (where new trajectories are collected from the evolving student policy).
    • Soft two-view aligned means improve task completion in all four settings:
      • Fixed 800-decision pool: 29/72 →\to 36/72 in Init. I (11/4 exclusive wins) and 33/72 →\to 37/72 in Init. II (6/2 exclusive wins), averaging 36.5/72 (50.7%) vs. 31.0/72 (43.1%) for single-view identity and 30.5/72 (42.4%) for Hard CE.
      • Learner-refreshed rollouts: 33/72 →\to 38/72 in Init. I (8/3 exclusive wins) and 32/72 →\to 41/72 in Init. II (11/2 exclusive wins), averaging 39.5/72 (54.9%) vs. 32.5/72 (45.1%) for single-view identity and 34.0/72 (47.2%) for Hard CE.
    • Across 1,152 common diagnostic states from these runs (Table 3), two-view training lowers student menu total variation (TV) in all 16 source-by-initialization-by-collection slices (reducing mean TV by 0.064–0.0940.064\text{--}0.094 and increasing argmax consistency by +3.5 to +12.0+3.5\text{ to }+12.0 percentage points).
  2. Repeated hardware executions from the shared initialization (Section 5.1 & Appendix B):
    • Repeating Arms A and B on their original GPUs gives 31/72 vs. 37/72 (+6+6 tasks), and swapping GPU assignments gives 31/72 vs. 38/72 (+7+7 tasks), consistent with the primary 28/72 vs. 36/72 (+8+8 tasks) result.
  3. Disjoint 72-task evaluation sets (Section 5.5 & Appendix C, Table 9):
    • Cross-evaluating the frozen core checkpoints (Arm B and Arm C) on the disjoint 72-task historical development set preserves Arm B's lead (39/72 vs. 36/72, 6/3 exclusive wins), while narrowing the margin from +10+10 tasks on the core set (36/72 vs. 26/72) to +3+3 tasks on the earlier set.
    • Conversely, cross-evaluating an earlier low-budget pair (1,600 single-view visits vs. 824 two-view visits at ≃1.57M\simeq 1.57\text{M} tokens) shifts their ranking from 38/72 vs. 37/72 on the earlier set to 28/72 vs. 39/72 (1/12 exclusive wins) on the core set.
    • In Section 6, we clearly distinguish these regimes: same-pool two-view refinement is replicated across three initializations (Init. I, Init. II, and seed 42), online learner-refreshed rollouts, two disjoint evaluation sets, and a second teacher family (Mistral-Small-3.2-24B, rising from 10/72 to 13/72), whereas the equal-cost coverage comparison (Arm B vs. Arm C) is evaluated on the shared frozen stream.

Q2: Do the main gains persist with a fresh behavior stream, independent initialization, or disjoint evaluation set?

We consolidated three sets of results in Section 5.1, Section 5.5, Table 4, Figure 4c, and Appendix C (Table 9) to show how target refinement behaves across fresh rollouts, independent initializations, and disjoint task sets:

  1. Independent initializations (Init. I and Init. II) and online learner-refreshed streams (Table 4 & Figure 4c):
    • We evaluated single-view identity targets, hard cross-entropy (Hard CE on the aligned winner), and soft two-view aligned mean targets across two earlier independent initializations (Init. I and Init. II) under both fixed-pool collection and online learner-refreshed collection (where new trajectories are collected from the evolving student policy).
    • Soft two-view aligned means improve task completion in all four settings:
      • Fixed 800-decision pool: 29/72 →\to 36/72 in Init. I (11/4 exclusive wins) and 33/72 →\to 37/72 in Init. II (6/2 exclusive wins), averaging 36.5/72 (50.7%) vs. 31.0/72 (43.1%) for single-view identity and 30.5/72 (42.4%) for Hard CE.
      • Learner-refreshed rollouts: 33/72 →\to 38/72 in Init. I (8/3 exclusive wins) and 32/72 →\to 41/72 in Init. II (11/2 exclusive wins), averaging 39.5/72 (54.9%) vs. 32.5/72 (45.1%) for single-view identity and 34.0/72 (47.2%) for Hard CE.
    • Across 1,152 common diagnostic states from these runs (Table 3), two-view training lowers student menu total variation (TV) in all 16 source-by-initialization-by-collection slices (reducing mean TV by 0.064–0.0940.064\text{--}0.094 and increasing argmax consistency by +3.5 to +12.0+3.5\text{ to }+12.0 percentage points).
  2. Repeated hardware executions from the shared initialization (Section 5.1 & Appendix B):
    • Repeating Arms A and B on their original GPUs gives 31/72 vs. 37/72 (+6+6 tasks), and swapping GPU assignments gives 31/72 vs. 38/72 (+7+7 tasks), consistent with the primary 28/72 vs. 36/72 (+8+8 tasks) result.
  3. Disjoint 72-task evaluation sets (Section 5.5 & Appendix C, Table 9):
    • Cross-evaluating the frozen core checkpoints (Arm B and Arm C) on the disjoint 72-task historical development set preserves Arm B's lead (39/72 vs. 36/72, 6/3 exclusive wins), while narrowing the margin from +10+10 tasks on the core set (36/72 vs. 26/72) to +3+3 tasks on the earlier set.
    • Conversely, cross-evaluating an earlier low-budget pair (1,600 single-view visits vs. 824 two-view visits at ≃1.57M\simeq 1.57\text{M} tokens) shifts their ranking from 38/72 vs. 37/72 on the earlier set to 28/72 vs. 39/72 (1/12 exclusive wins) on the core set.
    • In Section 6, we clearly distinguish these regimes: same-pool two-view refinement is replicated across two initializations (Init. I, Init. II), online learner-refreshed rollouts, two disjoint evaluation sets, and a second teacher family (Mistral-Small-3.2-24B, rising from 10/72 to 13/72), whereas the equal-cost coverage comparison (Arm B vs. Arm C) is evaluated on the shared frozen stream.

Thank you again for asking which findings held up when we changed the evaluation tasks. Table 7 now reports the paired uncertainty estimates you requested. When we evaluated the same two policies on a different set of 72 tasks, the lead from spending the teacher budget on a second view narrowed from ten tasks to three (Table 9). Your question helped us explain why the choice of tasks matters to our conclusion.

We will release the corresponding code and experimental logs to support reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.