acceptodds
Under review as a conference paper at ICLR 2027

Better Answers, Weaker Evidence Control: Evidence Bypass in On-Policy Distillation

Abstract

On-policy distillation (OPD) can improve answer accuracy while weakening responsiveness to supporting evidence. We investigate this separation in scientific-document question answering through matched document interventions that change the supported answer while preserving the question and unrelated context. We identify evidence bypass: a student retains an originally correct answer after the document changes to support a different one. On QASPER with a Qwen3-4B student, standard OPD raises original-document accuracy by 3.4 percentage points, lowers counterfactual accuracy by 5.3 points, and increases bypass by 15.7 points; the same directional pattern appears on PeerQA. Controlled perturbations localize the shift to answer-critical evidence, and bypass grows with the student's pre-distillation closed-book preference for the original answer. A factorized study of teacher targets, privileged information, divergence, and supervision density finds that explicitly supplying the teacher with annotated answer-critical spans yields the largest joint improvement in evidence response and claim support. These findings motivate Evidence-Paired OPD (EP-OPD), which combines on-policy supervision on matched documents supporting different answers to the same question with an answer-level margin that favors each document's supported answer over its paired alternative. Across both datasets and two student sizes, EP-OPD recovers 54–66% of standard OPD's counterfactual-accuracy loss while preserving 80–85% of its original-document accuracy gain. Our results establish evidence responsiveness as a distinct training and evaluation target alongside answer accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Evidence bypass, Context faithfulness, Scientific question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_N5q7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether on-policy distillation (OPD) improves answer accuracy at the expense of evidence faithfulness. The authors identify a phenomenon called evidence bypass: after distillation, a model becomes more accurate on original documents but is less likely to change its answer when the document's supporting evidence changes. Thus, it proposes a framework called Evidence-Paired OPD, which integrates answer-level margin loss into the original OPD loss and improves evidence following.

Critical strengths
  1. The observation of evidence bypass is usually ignored in many distillation tasks, which also corresponds to related work on reasoning faithfulness. The motivation of this paper is natural.
  2. The designed evaluation is complete. The authors use matched counterfactual documents, evidence deletion, paraphrasing, and claim-level support evaluation.
  3. The empirical findings are promising. On QASPER, standard OPD improves original accuracy from 49.0% to 52.4% while reducing counterfactual accuracy from 44.5% to 39.2%. EP-OPD recovers this degradation partially.
Critical weaknesses
  1. EP-OPD requires pairs of documents supporting different answers to the same question. However, constructing these pairs involves identifying answer-critical evidence, modifying, updating dependent statements, and validating the edited documents. The paper should report the cost of counterfactual data construction and analyze this tradeoff compared to baselines.
  2. The ablation study of EP-OPD loss is missing. The loss introduces 1/2 and λ, but there is no ablation study exploring the effect of different λ, just fixing λ=0.5 for all experiments. Also, the explanation for 1/2 is insufficient, which raises concerns about the EP-OPD loss design.
Questions for authors
  1. The EP-OPD loss equation should be labelled as Equation 3.
2 comments

We thank the reviewer for the constructive feedback and address the concerns below.

Weakness 1:

Constructing one intervention example uses about 15,000 tokens. Mean evaluation cost ranges from about 810 to 3,891 tokens per response, including claim extraction and evidence-support checks (Appendix D.5). Pair construction includes identifying the answer-critical evidence, editing it, updating dependent statements, and validating the resulting pair.

In the training comparison, 64 pairs count as 128 documents. The matched standard-OPD batch uses two original-document examples per pair, and the comparison matches teacher-scored tokens and optimizer steps. The margin adds four student-side canonical-answer scores per pair, one for each candidate under each document.

Weakness 2:

The complete objective is now Eq. (3):

LEP-OPD=12[LOPD(q,D0)+LOPD(q,D1)]+λLpair.\mathcal L_{\mathrm{EP\text{-}OPD}} =\frac{1}{2}\left[ \mathcal L_{\mathrm{OPD}}(q,D_0)+ \mathcal L_{\mathrm{OPD}}(q,D_1) \right]+\lambda\mathcal L_{\mathrm{pair}}.

Each document-level OPD loss is first averaged over its supervised tokens. The factor 1/21/2 then averages the two document losses so that the two versions contribute equally and the distillation term remains on a per-document scale.

λ\lambdaOriginal accuracy ↑CFAcc ↑Bypass ↓CSP ↑
0 (Augmented OPD)52.138.527.262.3
0.2552.039.127.562.6
0.5051.942.725.965.4
0.7551.042.825.765.2

At λ=0.50\lambda=0.50, CFAcc rises by 4.2 points and CSP by 3.1 points over Augmented OPD, with a 0.2-point reduction in original accuracy. Increasing the weight to 0.75 gives another 0.1 points in CFAcc and lowers bypass by 0.2 points, alongside a further 0.9-point decrease in original accuracy and a 0.2-point decrease in CSP. This trade-off supports the default weight of 0.50 in this experiment (Appendix E.1, Table 8).

Question 1:

The complete EP-OPD objective is labeled Eq. (3) in the revised main text.

Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

Reviewer_xwePOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether on-policy distillation (OPD) makes students rely less on document evidence. Using matched-document pairs that change one answer-critical fact, the authors show that OPD can raise accuracy on original documents while lowering accuracy on counterfactual ones, and they measure the "evidence bypass", i.e., how often student keeps the original correct answer when the document changes to support a different one. Bypass is stronger when the base student already prefers the closed-book original answer. The authors then vary four OPD design choices and propose Evidence-Paired OPD (EP-OPD), which recovers 54-66% of the counterfactual-accuracy loss while keeping 80-85% of the accuracy gain.

Critical strengths

1. Timely and practical question. OPD is now widely used, and the paper shows that accuracy gains can hide weaker evidence use. This is a useful warning for practitioners who evaluate distillation only by accuracy.

2. Careful experimental design. The counterfactual documents are well controlled, and the main runs are compute-matched across three seeds.

3. Clear presentation. The paper is well written and easy to follow, and Figure 1 conveys the core idea of evidence bypass at a glance.

Critical weaknesses

1. Potentially incremental and missing baselines.

The counterfactual accuracy and bypass are very close to substitution accuracy and memorization ratio (Longpre et al. 2021), cited but are not discussed clearly. Additionally, the insight about instruction-tuning affect context reliance is also investigated in SFT's literature (Goyal et al. Context-Parametric Inversion, ICLR 2025). Moreover, Section 3.3 misses baselines about (1) teacher-only accuracy, because OPD might learn bypass from teacher as well, and (2) verifer-based RL, to show that the effect is specific to distillation only or general post-training methods.

2. Small gains without significance tests.

Although each configuration is run with three seeds, the differences in Table 1 are small, and seed variance could account for part of them. Please report standard deviations or confidence intervals, and ideally a significance test.

3. Teacher and student from the same model family.

Both models are from Qwen3, so they likely share pretraining data and answer priors. This could make bypass stronger than it would be in general. It would be interesting to see whether bypass still appears when the teacher and student come from different families.

4. The scope is narrower than the claims.

The title and abstract present evidence bypass as a general property of OPD, but both datasets are scientific-document QA. I suggest adding at least one dataset outside this domain, such as HotpotQA, NQ, or ConflictBank. If that is not feasible, the title and claims should be narrowed to scientific-document QA.

Questions for authors

See Weakness.

3 comments

We thank the reviewer for the constructive feedback and address the concerns below.

Weakness 1:

[Longpre et al. (2021)] restrict their knowledge-conflict analysis to examples answered correctly under the original context. Our conditional bypass rate is porigp_{\mathrm{orig}} on the corresponding original-correct subset, which retains other responses in the denominator. CFAcc measures edited-answer correctness across the full matched set. Together, the metrics capture distinct outcomes when the evidence changes.

[Goyal et al. (2025)] observe that during instruction fine-tuning, the model exhibits an initial rise followed by a decline in training. We conduct a series of experiments to investigate the associated accuracy–evidence separation under on-policy distillation.

We have report teacher performance, on the QASPER evaluation set, the 30B teacher scores 70.2 original accuracy, 60.9 CFAcc, 10.3 bypass, and 85.6 CSP. The teacher's stronger standalone evidence response motivates examining how that behavior transfers to student-generated trajectories.

We evaluated answer-reward verifier-based RL and on-policy self-distillation (OPSD) under the same QASPER evaluation protocol (Table 1).

MethodOriginal accuracy ↑CFAcc ↑Bypass ↓CSP ↑
Base49.044.520.169.8
Standard OPD52.439.235.857.0
Verifier-based RL52.446.222.970.0
Qwen3-4B OPSD57.746.924.868.1

Verifier-based RL matches standard OPD's original accuracy and improves CFAcc by 1.7 points over Base, with a 2.8-point rise in bypass compared with 15.7 points under standard OPD. Using Qwen3-4B in both roles also improves original accuracy and CFAcc together and lowers bypass by 11.0 points relative to standard OPD. These results and the stronger standalone teacher scores point to how teacher supervision transfers along student-generated trajectories as a factor in the observed trade-off.

Weakness 2:

The primary QASPER–4B interaction is the OPD-induced change in counterfactual accuracy minus the change in original accuracy. It is −8.7 percentage points with a 95% paired bootstrap CI of [−12.4, −5.1] (§4.1 and Appendix B.1). The bootstrap jointly resamples source papers across the two checkpoints and document versions. This interval supports the primary accuracy–evidence separation. The remaining method comparisons are reported as point estimates.

Weakness 3:

On QASPER, we also distill a Qwen3-1.7B-Base student from a Llama-3.1-8B teacher (Appendix F, Table 9).

MetricBaseStandard OPDEP-OPD
Full-set answer-field F121.6426.6726.97
Full-set F1 with abstention mapping21.6437.3038.50
Paired original-document F134.4748.7349.38
Paired counterfactual F126.6539.7741.56

With a Llama-3.1-8B teacher and Qwen3-1.7B-Base student on QASPER, EP-OPD improves paired original-document F1 by 0.65 points, paired counterfactual F1 by 1.79 points, and CSP by 1.71 points over standard OPD (Table 9). Standard OPD also improves counterfactual F1 over initialization (26.65 to 39.77). This experiment supports the benefit of paired supervision across model families. Conditional bypass remains a separate question in this setting.

Weakness 4:

The revised manuscript includes a HotpotQA experiment with an OLMo 32B teacher and an OLMo 7B Instruct student (§6.2, Table 3). This setting tests paired supervision on multi-hop questions grounded in Wikipedia passages.

MetricInitial InstructStandard OPDEP-OPD
Full-set answer F152.8655.9357.34
Supporting-fact F150.0654.3255.21
Joint F131.3934.1135.35
Paired counterfactual F160.8961.5562.15
Paired CSP44.6348.4349.16

On HotpotQA, with an OLMo 32B teacher and OLMo 7B Instruct student, EP-OPD improves full-set answer F1 from 55.93 to 57.34, supporting-fact F1 from 54.32 to 55.21, and joint F1 from 34.11 to 35.35 over standard OPD. Paired counterfactual F1 rises from 61.55 to 62.15 and paired CSP from 48.43 to 49.16 (Table 3). The results extend the mitigation evaluation to multi-hop QA. We ground the evidence-bypass diagnosis in the matched QASPER and PeerQA experiments.associated degradation from evidence that paired supervision improves document-grounded behavior. The new F1 measurements will be reported separately from the original binary correctness and conditional bypass metrics.

Thank you for your response. The teacher-only and verifier-based RL baselines are helpful, as they show the bypass is not inherited from the teacher and is much stronger under distillation.

However, my remaining concerns are unchanged. (1) Significance in Table 1 is still missing, and (2) the cross-family and HotpotQA experiments report F1 and not bypass. So they do not show that evidence bypass occurs outside scientific-document QA, suggesting the generality claim remains unsupported.

I will keep my original score.

We appreciate reviewer's feedback.

We apologize for the misunderstanding regarding statistical significance. To further address statistical reliability, we have added the reported standard deviations to Table 1.

Table 1: QASPER results for Qwen3-4B-Base and its post-trained variants (mean ± SD, %).

MethodOrig. ↑Para. ↑CFAcc ↑Bypass ↓Drop abst. ↑CSP ↑
BASE49.0±0.548.7±1.644.5±1.920.1±0.522.0±0.769.8±0.6
ANS-SFT50.2±0.849.9±0.343.8±0.923.7±2.619.2±1.767.5±0.7
Answer-filtered CoT-SFT51.4±0.651.0±0.842.3±0.829.6±1.916.0±0.564.2±0.8
Standard OPD52.4±0.951.9±0.739.2±0.635.8±1.014.4±0.657.0±0.9
Verifier-based RL52.4±0.7–46.2±0.822.9±0.9–70.0±0.6
On-policy self-distillation57.7±1.1–46.9±2.924.8±2.8–68.1±0.7
EP-OPD (ours)51.9±0.751.6±0.842.7±0.725.9±1.619.1±0.665.4±1.4
Frozen 30B teacher70.2±0.8–60.9±1.510.3±1.4–85.6±2.5

HotpotQA involves multiple supporting facts and multi-step reasoning. We therefore retain the previously reported supporting-fact F1 and joint F1 metrics in Table 3. we compute and report conditional bypass rates for the HotpotQA and cross-family experiments in Table 3 and Table 9, respectively.

Table 3: HotpotQA with OLMo 32B teacher and OLMo 7B Instruct student.

MetricInitial InstructStandard OPDEP-OPD
Full-set answer F152.8655.9357.34
Full-set supporting-fact F150.0654.3255.21
Full-set joint F131.3934.1135.35
Paired counterfactual F160.8961.5562.15
Paired CSP44.6348.4349.16
Paired conditional bypass ↓27.1829.6528.32

On HotpotQA, conditional bypass increases from 27.18% to 29.65% under standard OPD and decreases to 28.32% with EP-OPD.

Table 9: Cross-family QASPER with Llama-3.1-8B teacher and Qwen3-1.7B-Base student.

MetricBaseStandard OPDEP-OPD
Full-set answer-field F121.6426.6726.97
Full-set F1 with abstention mapping21.6437.3038.50
Paired original-document F134.4748.7349.38
Paired counterfactual F126.6539.7741.56
Paired CSP43.4962.0463.75
Paired conditional bypass ↓25.7035.9228.25

In the cross-family setting, standard OPD increases conditional bypass from 25.70% to 35.92%, while EP-OPD reduces it to 28.25%. These additional measurements directly examine evidence bypass across model families and in multi-hop QA.

Reviewer_7A6COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper studies “evidence bypass” in document-grounded on-policy distillation (OPD) for scientific QA, where a distilled student improves accuracy on original documents but more frequently retains the original answer when answer-critical evidence is edited to support a different answer. Using controlled document interventions on QASPER and PeerQA with Qwen3-1.7B-Base and Qwen3-4B-Base students, the authors examine how teacher targets, privileged information, divergence objectives, and entropy-based supervision masking affect answer accuracy and evidence responsiveness. They propose Evidence-Paired OPD (EP-OPD), which combines reverse-KL distillation on paired original and counterfactual documents with an additional margin loss over canonical answers. The authors report that EP-OPD partially recovers counterfactual accuracy and claim support while retaining most of standard OPD’s original-document accuracy gain.

Critical strengths
  1. Distinguishing original-document answer accuracy from responsiveness to supporting evidence is a reasonable evaluation objective.

  2. Matched answer-changing interventions, together with answer-preserving and evidence-deletion controls, provide a potentially useful framework for studying document-grounded behavior.

Critical weaknesses
  1. Incomplete experimental reporting and diagnostics.

    The paper does not report the train/dev/test sizes after filtering and pair validation, making the training schedule difficult to assess, particularly for PeerQA. Results for the irrelevant-insertion control are missing, and Table 3 provides only relative changes rather than complete absolute results for PeerQA and the 1.7B student. Key comparisons lack uncertainty estimates, and Appendix G contains no real dataset intervention examples. The verifier, claim evaluator, prior-scoring procedure, and online teacher used for EMA are insufficiently specified. Without the external teacher’s own performance, it is difficult to tell whether distillation amplifies the failure or transfers an existing teacher weakness. The evaluation-only response format also requires compliance checks, especially for answer-only SFT models. The restriction to small Qwen3 Base students and greedy decoding further limits the conclusions.

  2. Unclear alignment between the auxiliary objective and generation.

    As written, Equation (4) and Algorithm 1 score canonical answers directly under the question–document input, without conditioning on generated reasoning. Evaluation instead asks the model to explain its reasoning before answering. Although the paper mentions a shared answer delimiter, it does not show the complete scoring template or identify the scored token positions. The connection between these two response settings remains unclear. A comparison with reasoning-conditioned answer scoring would help establish whether the direct-answer objective is an appropriate training signal for the behavior being evaluated.

  3. Limited novelty and weak practical justification.

    EP-OPD combines counterfactual augmentation, standard reverse-KL OPD, and a conventional hinge-ranking loss. The paper does not establish a clear advance over related approaches such as SCOTT, DisentQA, and Context-DPO, while the theory offers little insight into mechanisms specific to OPD.

    The gains over paired answer-only SFT are modest, and bypass is slightly worse. Without a paired SFT-plus-margin control, the contribution of OPD remains unclear. EP-OPD also underperforms the pre-distillation base model on all four evidence-sensitive metrics in Table 1.

    The choice of repair is difficult to justify given that the critical-span PI control performs substantially better on all four reported metrics. Pair construction already requires evidence annotations and intervention validation, yet the paper does not adequately compare the resources required by these alternatives. A same-model self-distillation baseline with privileged evidence would also help determine whether a separate large teacher and the paired objective are necessary.

Questions for authors
  1. What are the train/dev/test sizes after filtering and pair validation, and the actual optimizer-step counts for each setting? Please provide the missing results for the irrelevant-insertion control and complete absolute metrics for PeerQA and Qwen3-1.7B, including variability across training seeds. Please also report the 30B teacher’s own performance and show at least one real document-intervention example.

  2. Which models, prompts, or rules are used for the document verifier, automatic claim evaluator, and closed-book prior scoring? What is the online teacher used in the EMA experiments, and how is it updated? Under the evaluation-only structured prompt, how often does each baseline follow the required format, and how are parsing failures, missing citations, and noncompliant responses handled in CSP?

  3. Please provide the exact input template and scored token positions used to compute the canonical-answer scores in Equation (4). As written, these scores do not condition on generated reasoning. How does this compare with scoring the same answers after the student’s generated reasoning, and what supports the choice of direct-answer scoring?

  4. How does paired answer-only SFT with the same margin loss, but without OPD, perform? Since constructing the counterfactual pairs already requires annotated evidence, what practical advantage does EP-OPD offer over the stronger critical-span PI control in Table 2 when annotation and computation costs are taken into account? Could same-model self-distillation with privileged evidence provide comparable benefits without a separate large teacher?

  5. Could you provide results for additional teacher–student pairs, particularly with stronger post-trained students or models from different families? This would help establish whether both the observed degradation and the benefits of EP-OPD extend beyond the current small Qwen3 Base models.

6 comments

We thank the reviewer for the constructive feedback and address the concerns below.

Question 1:

The released QASPER and PeerQA snapshot has the following validated pairs.

Released QASPER dataTrainDevTest
Validated document pairs1,1977199
Distinct source papers6746286
Released PeerQA dataTrainDevTest
Validated document pairs42950100
Distinct source papers2793290

The training pairs include 226 original QASPER questions and 971 questions constructed from training papers. Paired dev and test questions come from the original dataset. The paired splits are disjoint by source paper. The main Qwen3-30B-A3B-Instruct-2507 distillation runs use 76 optimizer steps for each Qwen student size (4B and 1.7B), with a two-epoch configuration. The Llama-3.1-8B to Qwen3-1.7B run uses 76 steps over four epochs (Appendix A.3).

Table 11 gives absolute results for the QASPER–1.7B and PeerQA settings, both with the Qwen3-30B teacher for distillation.

QASPER–1.7B methodOriginal accuracy ↑CFAcc ↑Bypass ↓CSP ↑
Base43.938.424.865.0
Standard OPD46.636.138.654.4
EP-OPD46.138.229.061.9

Standard OPD raises original accuracy by 2.7 points and bypass by 13.8 points while lowering CFAcc by 2.3 points. EP-OPD recovers 2.1 of the 2.3 points lost in CFAcc (91%) and retains 2.2 of the 2.7 points gained in original accuracy (81%).

StudentMethodOriginal accuracy ↑CFAcc ↑Bypass ↓CSP ↑
Qwen3-4BBase46.241.121.067.9
Qwen3-4BStandard OPD48.138.432.360.0
Qwen3-4BEP-OPD47.739.825.065.1
Qwen3-1.7BBase41.236.024.764.1
Qwen3-1.7BStandard OPD43.931.538.853.3
Qwen3-1.7BEP-OPD43.434.329.760.2

On QASPER,4B accuracy under irrelevant insertion is 48.5, 51.6, and 51.2 for Base, standard OPD, and EP-OPD (Table 12). Table 1 reports teacher performance, the 30B teacher scores 70.2 original accuracy, 60.9 CFAcc, 10.3 bypass, and 85.6 CSP when generating independently on the same evaluation set (Table 1). The teacher's stronger standalone evidence response highlights the gap between its own generation and the student's behavior after distillation.

The primary paired interaction on QASPER, 4B model is −8.7 percentage points with a 95% CI of [−12.4, −5.1]. Appendix B.1 specifies 1,000 paired bootstrap draws, with source papers resampled jointly across checkpoints and document versions. This interval quantifies evaluation-sample uncertainty for the primary interaction.

Appendix H presents a sample. The question asks whether S^−3σ\widehat S-3\sigma exceeds the bound of zero. The published measurement 0.101±0.0200.101\pm0.020 gives a lower endpoint of 0.0410.041 and the answer Yes. The edit keeps S^=0.101\widehat S=0.101 and increases σ\sigma to 0.0500.050. The lower endpoint becomes −0.049-0.049 and the supported answer becomes No. A model that repeats Yes after the edit exhibits bypass in this worked example.

Question 2:

Document validation. The construction checks require a valid answer change, local edits, consistent updates to dependent mentions, and removal of residual support for the original answer. A document-only verifier checks the new answer, and held-out pairs receive manual review (Appendix C.1). The released preparation code checks record structure, tokenization, and split separation.

Claim scoring and format. Appendix D.4 gives the templates for the released deepseek-flash judge, which uses temperature zero, disabled thinking, and JSON output. The judge extracts independently checkable claims, including factual statements in the final answer, and checks each relevant claim against its cited blocks and adjacent context. A relevant claim with missing or invalid citations is labeled absent. Irrelevant claims are excluded. The released parser reads the final ANSWER: field when present and otherwise attempts to extract the initial answer segment. The evaluator reports coverage for questions with relevant claims. The QASPER–4B evidence error rises under standard OPD in both the automated evaluation (30.2% to 43.0%) and the blinded expert subset (28.5% to 38.3%, Table 13). Per-method format compliance rates under the submission's evaluation prompt remain unreported.

Prior scoring. We measure the frozen base student's closed-book preference by removing the document and comparing length-normalized scores for the original and counterfactual answers. This preference determines the strata used in the bypass analysis.

EMA teacher. For the QASPER–4B factorized controls, the moving target is a separate Qwen3-4B model with the student's architecture. After optimizer step k+1k+1, it receives the update θˉk+1=μθˉk+(1−μ)θk+1\bar\theta_{k+1}=\mu\bar\theta_k+(1-\mu)\theta_{k+1}, with μ∈0.95,0.995\mu\in\\{0.95,0.995\\}. The target runs in evaluation mode with gradients disabled. Appendix A.4 describes the released initialization and update order.

Question 3:

Equation (4) uses direct-answer teacher forcing under the student model, without conditioning on generated reasoning. For the original Qwen-Base implementation, the input template is: You answer questions about a scientific paper. Use only the supplied paper. Finish with ANSWER: followed by one concise answer. If the evidence is insufficient, answer INSUFFICIENT EVIDENCE. Title: {title} Abstract: {abstract} Question: {question} Paper (paragraph IDs are stable metadata): {document_with_paragraph_IDs} Question: {question} Answer: {canonical_answer} The prefix ends immediately after Answer:. We tokenize the prefix and " " + canonical_answer separately, with add_special_tokens=False, and concatenate their token IDs. The Base student receives this as plain completion text, without a chat-template wrapper. Only the canonical-answer tokens contribute to the score: �0��0� Specifically, for a prefix of length L and an answer of length K, the scored target positions are L,…,L+K−1L,\ldots,L+K-1, using logits at positions L−1,…,L+K−2L-1,\ldots,L+K-2, with zero-based indexing. Prompt tokens, the Answer: delimiter, padding, and any evidence or reasoning text are excluded; no additional EOS token is scored. Both canonical candidates are evaluated under each document, giving the four scores used by the paired margin. Scoring after the student’s generated reasoning would instead compute �3��3� where r is a student-generated reasoning trace. Both candidate answers should share the same reasoning prefix for a given document. This measures answer preference conditional on an already generated trace, whereas Equation (4) measures answer preference directly from the question and document. The rankings need not agree: an incorrect reasoning trace may reinforce an unsupported answer even when the direct-answer score favors the supported one. Our motivation for direct-answer scoring is to impose a controlled answer-level constraint: the paired documents must reverse the ranking of the same two canonical answers, without introducing variation from sampled reasoning or conditioning the auxiliary loss on potentially erroneous traces. It also avoids an additional reasoning-generation step for this auxiliary term. The OPD term separately supervises student-generated trajectories.

Question 4:

This work mainly reveals the evidence bypass phenomenon in OPD and identifies EP-OPD as a promising mitigation strategy. Our experiments show that Acc and CFAcc are hard to reconcile in OPD, and EP-OPD achieves the Pareto frontier of Acc and CFAcc.

MethodOriginal accuracy ↑CFAcc ↑Bypass ↓CSP ↑
Standard OPD52.439.235.857.0
Augmented OPD52.138.527.262.3
Paired answer-only SFT50.542.025.363.8
Paired answer-only SFT + margin51.544.025.165.9
Augmented OPD + answer CE52.540.926.759.7
EP-OPD51.942.725.965.4

We added paired answer-only SFT with the same hinge-ranking objective and margin to the QASPER–4B ablation (Table 7).

Table 7 adds paired answer-only SFT with the same hinge form, answer scores, and margin. It scores 51.5 original accuracy, 44.0 CFAcc, 25.1 bypass, and 65.9 CSP. EP-OPD scores 51.9, 42.7, 25.9, and 65.4, respectively. The control shows that paired answer supervision can improve evidence response without on-policy teacher supervision. EP-OPD recovers about 66% of the QASPER–4B CFAcc loss under standard OPD, while the Base checkpoint still has higher CFAcc and CSP and lower bypass.

We also evaluated on-policy self-distillation with Qwen3-4B in both roles (Table 1). It scores 57.7 original accuracy, 46.9 CFAcc, 24.8 bypass, and 68.1 CSP. Relative to standard OPD, this raises original accuracy by 5.3 points and CFAcc by 7.7 points while reducing bypass by 11.0 points. The 30B critical-span control scores 60.6, 52.6, 14.8, and 76.0, respectively, with the exact annotated span supplied to the teacher.

Question 5: Additional teacher–student settings

The revised manuscript includes a HotpotQA experiment with lager student(OLMo-7B) and teacher(OLMo-32B) (§6.2, Table 3). This setting tests paired supervision on multi-hop questions grounded in Wikipedia passages.

MetricInitial InstructStandard OPDEP-OPD
Full-set answer F152.8655.9357.34
Supporting-fact F150.0654.3255.21
Joint F131.3934.1135.35
Paired counterfactual F160.8961.5562.15
Paired CSP44.6348.4349.16

On QASPER, we also distill a Qwen3-1.7B-Base student from a Llama-3.1-8B teacher (different family) (Appendix F, Table 9).

MetricBaseStandard OPDEP-OPD
Full-set answer-field F121.6426.6726.97
Full-set F1 with abstention mapping21.6437.3038.50
Paired original-document F134.4748.7349.38
Paired counterfactual F126.6539.7741.56

Paper results report evidence bypass across QA and document domains for student models of 1.7B, 4B, 7B paired with a 30B teacher, 1.7B students paired with a 7B teacher, and under both intra-family and cross-family distillation setups.

Weakness 3:

SCOTT uses counterfactual rationales to study rationale–answer consistency. DisentQA uses counterfactual augmentation to separate contextual and parametric answers, and Context-DPO trains preferences for context-faithful answers. Our analysis follows evidence response through on-policy teacher supervision and tests four design choices that change that supervision. The theory explains why one document per question can leave evidence dependence unidentified. This work mainly reveals the evidence bypass phenomenon in OPD and identifies EP-OPD as a promising mitigation strategy. Our experiments show that Acc and CFAcc are hard to reconcile, and EP-OPD achieves the Pareto frontier of Acc and CFAcc.

Reviewer_NTsEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether on-policy distillation of document-grounded scientific QA transfers a student's dependence on supporting evidence or only its answer preferences. The authors introduce evidence bypass: after a single answer-critical edit flips the supported answer. the student retains its originally correct answer. They propose Evidence-Paired OPD, which co-distills matched document pairs supporting different answers under a hinge margin L_pair that ranks each document's supported answer above its paired alternative. Experiments across various datasets and student sizes validate the effectiveness of EP-OPD.

Critical strengths
  1. The paper is well written and easy to follow.

  2. The evidence-bypass phenomenon is novel and interesting, and it carries some value for understanding how OPD works mechanically, which may offer the community a useful insight.

Critical weaknesses
  1. The paper claims to identify a major defect in OPD and a corresponding remedy, yet all of its analysis and experiments are confined to scientific-document QA. It therefore reads more as evidence that evidence bypass arises when OPD is applied to this particular task family, rather than as a general problem — which limits the paper's contribution.

  2. The EP-OPD method proposed in the paper shows no marked improvement over the other baselines on the paper's own custom metrics. It is largely subsumed by simpler paired-document baselines in the paper's own ablation, and neither on-policy distillation nor the paired margin is shown to be necessary. In Table 6 (QASPER-4B), counterfactual documents with no answer objective (Augmented OPD) already recover most of the evidence-response gain.

  3. The paper uses relatively weak student models (1.7B and 4B), and it is unclear whether the claimed evidence-bypass problem persists in a stronger teacher–student setting; the paper offers no discussion on this point.

Questions for authors

See Weaknesses

3 comments

We thank the reviewer for the constructive feedback and address the concerns below.

Weakness 1:

The revised manuscript includes a HotpotQA experiment with an OLMo 32B teacher and an OLMo 7B Instruct student (§6.2, Table 3).

MetricInitial InstructStandard OPDEP-OPD
Full-set answer F152.8655.9357.34
Supporting-fact F150.0654.3255.21
Joint F131.3934.1135.35
Paired counterfactual F160.8961.5562.15
Paired CSP44.6348.4349.16

Standard OPD improves paired counterfactual F1 over the initial student. EP-OPD improves it by a further 0.60 points and raises full-set answer F1 by 1.41 points and joint F1 by 1.24 points over standard OPD. These results extend the benefit of paired supervision to multi-hop QA.

**Weakness 2: ** This work mainly reveals the evidence bypass phenomenon in OPD and identifies EP-OPD as a promising mitigation strategy. Our experiments show that Acc and CFAcc are hard to reconcile, and EP-OPD achieves the Pareto frontier of Acc and CFAcc.

MethodOriginal accuracy ↑CFAcc ↑Bypass ↓CSP ↑
Standard OPD52.439.235.857.0
Augmented OPD52.138.527.262.3
Paired answer-only SFT50.542.025.363.8
Paired answer-only SFT + margin51.544.025.165.9
Augmented OPD + answer CE52.540.926.759.7
EP-OPD51.942.725.965.4

We added paired answer-only SFT with the same hinge-ranking objective and margin to the QASPER–4B ablation (Table 7). Counterfactual exposure alone reduces bypass from 35.8% to 27.2%, while CFAcc falls from 39.2% to 38.5%. Adding the margin to Augmented OPD then raises CFAcc by 4.2 percentage points and CSP by 3.1 points, with a 0.2-point decrease in original accuracy. Adding the margin to paired answer-only SFT also raises CFAcc and CSP, by 2.0 and 2.1 points. The paired ranking signal helps under both training objectives.

Weakness 3: Stronger students and more model families

The HotpotQA run uses lager model with an OLMo 7B Instruct student and an OLMo 32B teacher.

MetricInitial InstructStandard OPDEP-OPD
Full-set answer F152.8655.9357.34
Supporting-fact F150.0654.3255.21
Joint F131.3934.1135.35
Paired counterfactual F160.8961.5562.15
Paired CSP44.6348.4349.16

Paper results report evidence bypass across QA and document domains for student models of 1.7B, 4B, 7B paired with a 30B teacher, 1.7B students paired with a 7B teacher, and under both intra-family and cross-family distillation setups.

Additionally, on QASPER, we also distill a Qwen3-1.7B-Base student from a Llama-3.1-8B teacher (Appendix F, Table 9).

MetricBaseStandard OPDEP-OPD
Full-set answer-field F121.6426.6726.97
Full-set F1 with abstention mapping21.6437.3038.50
Paired original-document F134.4748.7349.38
Paired counterfactual F126.6539.7741.56
Paired CSP43.4962.0463.75

EP-OPD improves paired counterfactual F1 by 1.79 points and CSP by 1.71 points over standard OPD. The initial-to-OPD counterfactual F1 gains in both new settings leave open whether the same conditional bypass increase occurs there.

Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

Thanks for the response, but my concerns remain unresolved. Although the authors have added results on HotpotQA, I still do not understand why OPD needs to be studied on these tasks. Is evidence pass a general problem of OPD across different domains? If it occurs only on these tasks, then why use OPD at all, given that baselines such as Paired answer-only SFT already seem good enough compared with EP-OPD? I think the paper in its current state clearly cannot address these concerns well, so I will keep my original score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback and the AC for overseeing the review process.

We are pleased that the reviewers recognized three aspects of our work: (1) identifying evidence bypass in on-policy distillation (OPD), an overlooked yet important phenomenon; (2) moving beyond answer accuracy to investigate models' responsiveness to supporting evidence, with clear presentation and practical significance; and (3) employing controlled counterfactual document interventions and demonstrating that EP-OPD can partially recover this degradation in OPD-based methods. These three aspects constitute the core empirical contributions of our paper.

In response to the reviewers' concerns, we have substantially revised the manuscript and extended our analysis along three major dimensions:

Expanded experimental validation: We extended our evaluation from scientific-document question answering to HotpotQA and added cross-family distillation experiments involving Llama and Qwen. The new results consistently show that, in both experimental settings, standard on-policy distillation (OPD) exacerbates conditional evidence bypass, while EP-OPD consistently mitigates this degradation, achieving performance comparable to non-OPD methods.

Improved experimental completeness and reliability: We supplemented the manuscript with complete metric results, dataset statistics, and standalone evaluations of the teacher models.

Additional comparisons and analyses of mitigation strategies: We introduced new baselines based on reinforcement learning, on-policy self-distillation, and paired supervised fine-tuning (paired SFT) with the same margin objective. These comparisons further characterize the evidence bypass phenomenon and clarify the role of paired supervision.

The central contribution of our paper is the systematic identification and characterization of evidence bypass as a critical limitation of on-policy distillation (OPD), supported by extensive experiments across diverse settings, together with EP-OPD as a feasible mitigation strategy. We have carefully incorporated the reviewers' feedback and believe that these revisions substantially strengthen the empirical evidence, experimental rigor, and overall contribution of our paper.

We thank the reviewers for their constructive feedback.

We notice that several reviewers question why we adopt OPD given its subpar performance on scientific document tasks and HotpotQA-style reasoning benchmarks. Our clarification is as follows: OPD is a widely adopted post-training paradigm. In many scenarios, such as when high-quality data is scarce, leveraging a teacher model to provide supervision signals is a natural and necessary choice. The core goal of this paper is not to position EP-OPD as a general solution for the aforementioned downstream tasks. Our central contribution lies in identifying and analyzing the evidence bypass failure mode, instead of aiming to boost task performance. This failure pattern is distinctive to the OPD framework and is typically far less pronounced in conventional post-training pipelines. EP-OPD is our proposed improvement targeting this intrinsic flaw inherent to OPD-style post-training methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.