- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work introduces DecisionOPD, a framework for analysing the tradeoff between refining targets for the same states or extending supervision to new states when distilling a teacher decision-making model into a student one. The underlying problem is that teacher models, when presented with a menu of options, will sometimes predict a different option depending on the order in which they are presented.
Critical strengths
-
The comparison between refinement and coverage is well-controlled and effectively answers the question asked by the authors.
-
The results are presented quite transparently, even in cases that go against the main thesis of the paper.
Critical weaknesses
-
The idea of spending compute to re-evaluate seen states (with reversed choices) is not new (as noted by the authors), so unless I am mistaken, it seems like this work’s contribution is just analyzing this tradeoff under a fixed teacher evaluation budget?
-
The scope is quite limited; the paper only examines one environment, one main teacher/student pair, and one initialization in the main study.
-
The paper mentions exact McNemar tests (with Holm correction), but I could not find these results anywhere.
-
The main results also seem very noisy. For example, Figure 2 shows that, in the high-budget setting, 2 teacher views per decision seems to do worse than 1 under a fixed budget on a different set of 72 tasks than those shown in the main paper. The paper even notices in section A.3 that repeated experiments changed nine task outcomes in each method.
-
Figure 1 shows two line plots with only 2 x-axis ticks each. This should have simply been a table. The “Matched re-execution" shown in the plots is also not sufficiently explained.
-
Most of the appendix also reads like an agent’s research log, including a ton of information that is not relevant to the paper.
-
Perhaps most alarmingly, the “Limitations” section does not list a single limitation. It instead simply lists avenues for future work.
Questions for authors
Despite the explanation regarding the data stream prefix, it seems odd to not make the number of visits in the single-view extended setting double the number of visits of the single-view and two-view mean settings, especially if you are trying to show that two-view mean is superior under the same teacher budget.
Could you include results using exactly double the number of visits, even if this means you have to lower the base number of visits?
5 comments
W3: Exact McNemar test results (with Holm correction)
Thank you for pointing out this omission. We have included the complete paired statistical table in Appendix B (Table 7) (referenced in Section 5.1), reporting discordant exclusive win pairs (L/R only), family-stratified 95% paired bootstrap confidence intervals, two-sided exact binomial McNemar -values, and Holm step-down adjusted -values:
- Arm B vs. Arm A (36/72 vs. 28/72):
11 / 3exclusive wins, net tasks ( pp), 95% bootstrap CI pp, exact McNemar , Holm-adjusted . - Arm B vs. Arm C (36/72 vs. 26/72):
13 / 3exclusive wins, net tasks ( pp), 95% bootstrap CI pp, exact McNemar , Holm-adjusted . - Arm N vs. Arm A (33/72 vs. 28/72):
7 / 2exclusive wins, net tasks ( pp), 95% bootstrap CI pp, exact McNemar , Holm-adjusted . - Supporting contrasts: F vs. A (
32 vs. 28,6/2exclusive wins, CI pp), B vs. F (36 vs. 32,7/3exclusive wins, CI pp), B vs. N (36 vs. 33,6/3exclusive wins, CI pp), and C vs. A (26 vs. 28,4/6exclusive wins).
W1, W2, W6, & W7: Clarifying contribution, rewriting the Appendix, and stating explicit limitations
- Contribution beyond re-evaluating seen states: In Section 1, Section 3.3 (Equation 4), and Section 5.2, we show that our work establishes (i) why multi-view averaging improves sequential policies even when zero top-1 teacher actions change (Arm N, raising completion from 28/72 to 33/72 with identical top-1 labels, whereas one-step greedy action substitutions leave episode return unchanged on of heldout switch points), and (ii) how per-record student exposure governs the trade-off between view refinement and state coverage as student updates scale from 200 to 400.
- **Broader empirical evidence in the main text (Tables 3, 4, 5 & Figure 4c):**Results from our two historical initializations (Init. I and Init. II), online learner-refreshed collection runs (Table 4), 1,152-state menu stability diagnostics (Table 3), Mistral-Small-3.2-24B cross-teacher results (Section 5.6), and PIQA / HellaSwag / ARC-Challenge evaluations (Table 5) into the main text.
- Rewritten Appendix (Appendices A–F) and explicit limitations (Section 6): We removed all research-log text and internal run identifiers from the appendix, restructuring Appendices A–F into six self-contained technical sections (Tables 6–13). In Section 6 (Discussion), we explicitly state our empirical boundaries: the equal-cost coverage comparison (Arm B vs. Arm C) uses one shared frozen stream and one initialization (seed 42 with data-order permutations 43 and 44); interactive evaluations use 72 tasks (with Holm-adjusted McNemar -values in Table 7); cross-evaluating frozen checkpoints on a disjoint 72-task set narrows the gap from 10 tasks to 3 and reverses an earlier low-budget pair; and higher teacher budgets ( tokens) or extended student training (400 updates) shift the balance toward broader single-view coverage.
We thank the Reviewer for noting that our experimental comparisons are well-controlled and transparent, and for the specific feedback on the visit-count ratio (1,544 vs. 1,600 visits), the missing McNemar test table, figure formatting, appendix structure, and the limitations discussion. We address every point below.
Q1 & Q2: Why Arm C uses 1,544 visits under token matching, and comparison with the earlier 1,600-visit single-view checkpoint (Section 5.5 and Appendix C.1, Tables 8 and 9):