- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether on-policy distillation (OPD) improves answer accuracy at the expense of evidence faithfulness. The authors identify a phenomenon called evidence bypass: after distillation, a model becomes more accurate on original documents but is less likely to change its answer when the document's supporting evidence changes. Thus, it proposes a framework called Evidence-Paired OPD, which integrates answer-level margin loss into the original OPD loss and improves evidence following.
Critical strengths
- The observation of evidence bypass is usually ignored in many distillation tasks, which also corresponds to related work on reasoning faithfulness. The motivation of this paper is natural.
- The designed evaluation is complete. The authors use matched counterfactual documents, evidence deletion, paraphrasing, and claim-level support evaluation.
- The empirical findings are promising. On QASPER, standard OPD improves original accuracy from 49.0% to 52.4% while reducing counterfactual accuracy from 44.5% to 39.2%. EP-OPD recovers this degradation partially.
Critical weaknesses
- EP-OPD requires pairs of documents supporting different answers to the same question. However, constructing these pairs involves identifying answer-critical evidence, modifying, updating dependent statements, and validating the edited documents. The paper should report the cost of counterfactual data construction and analyze this tradeoff compared to baselines.
- The ablation study of EP-OPD loss is missing. The loss introduces 1/2 and λ, but there is no ablation study exploring the effect of different λ, just fixing λ=0.5 for all experiments. Also, the explanation for 1/2 is insufficient, which raises concerns about the EP-OPD loss design.
Questions for authors
- The EP-OPD loss equation should be labelled as Equation 3.
2 comments
Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE
Best, AC
We thank the reviewer for the constructive feedback and address the concerns below.
Weakness 1:
Constructing one intervention example uses about 15,000 tokens. Mean evaluation cost ranges from about 810 to 3,891 tokens per response, including claim extraction and evidence-support checks (Appendix D.5). Pair construction includes identifying the answer-critical evidence, editing it, updating dependent statements, and validating the resulting pair.
In the training comparison, 64 pairs count as 128 documents. The matched standard-OPD batch uses two original-document examples per pair, and the comparison matches teacher-scored tokens and optimizer steps. The margin adds four student-side canonical-answer scores per pair, one for each candidate under each document.
Weakness 2:
The complete objective is now Eq. (3):
LEP-OPD=21[LOPD(q,D0)+LOPD(q,D1)]+λLpair.
Each document-level OPD loss is first averaged over its supervised tokens. The factor 1/2 then averages the two document losses so that the two versions contribute equally and the distillation term remains on a per-document scale.
At λ=0.50, CFAcc rises by 4.2 points and CSP by 3.1 points over Augmented OPD, with a 0.2-point reduction in original accuracy. Increasing the weight to 0.75 gives another 0.1 points in CFAcc and lowers bypass by 0.2 points, alongside a further 0.9-point decrease in original accuracy and a 0.2-point decrease in CSP. This trade-off supports the default weight of 0.50 in this experiment (Appendix E.1, Table 8).
Question 1:
The complete EP-OPD objective is labeled Eq. (3) in the revised main text.