acceptodds
Under review as a conference paper at ICLR 2027

DVSD: Dual-View Self-Distillation for Long-Context Evidence Compression and Answering

Abstract

Long-context reasoning tasks such as question answering and agent memory suffer from poor effective utilization of lengthy inputs, despite continuous expansions to model context windows. A key capability toward solving this problem is enabling models to autonomously compress long contexts into task-relevant compact evidence and leverage the compressed information for downstream tasks. We propose Dual-View Self-Distillation (DVSD), a training framework that jointly optimizes evidence compression and answer generation. A shared self-teacher provides supervision through two complementary views: an answer-privileged view uses reference answers to guide evidence construction without requiring evidence annotations, while an evidence-conditioned view forms answer targets from the question and student-generated evidence, without access to the original context or reference answer. Both views deliver dense token-level guidance along the student's on-policy trajectories, jointly training evidence construction and answer generation. Across three backbone models, DVSD achieves the highest overall scores among all baselines on LongBench v2, RULER, and InfiniteBench. It also generalizes beyond training lengths, improves agent-memory retrieval and test-time learning, and largely preserves the general capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM training, Self-distillation, Long-context reasoning, Long-context compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_Pb8yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DVSD trains a model to generate evidence followed by an answer, using two conditioning views of a shared self-teacher. The evidence teacher receives the original context, question, and reference answer, while the answer teacher receives only the question and student-generated evidence. The two regions are trained with token-level reverse-KL objectives, and the teacher is refreshed between epochs. Experiments evaluate long-context understanding, agent-memory tasks, and general capability retention, alongside ablations of teacher inputs, loss weights, and update strategies.

Critical strengths

1.The supervision design addresses a specific information-dependence problem. Reference answers guide evidence construction, while answer supervision depends on the student's evidence. The approach is straightforward and does not require human evidence annotations. 2. The empirical evaluation and ablations are reasonably comprehensive. Results across three backbones, evaluations beyond the training context length, and ablations of both losses and teacher inputs collectively support the design's usefulness.

Critical weaknesses
  1. The evidence has not been established as the student's actual information bottleneck. Because the student answers with access to the original context, it can bypass its generated evidence. Restricting the answer teacher does not establish that the student relies on evidence, and the ideal conditional-independence assumption does not guarantee this property after training.
  2. The robustness and attribution of the gains remain incomplete. Some improvements over the strongest alternatives are small, without reported multi-seed training uncertainty in the main results. The agent-memory evaluation primarily compares DVSD with the initial models, leaving unclear whether the transfer gains exceed those of competing post-training methods.
Questions for authors
  1. How does answer performance change when generated evidence is removed, shuffled, or replaced? What happens when the original context is removed and only the evidence remains?
  2. Does DVSD improve evidence faithfulness and answer sufficiency under an evaluation independent of the training teacher?
  3. Do the key improvements persist across training seeds, and do matched OPSD or OPSDL baselines show comparable gains on the memory tasks?
5 comments

Thank you for the constructive feedback. We address your concerns below.

Q1/W1: Inference ablations of evidence utilization

We agree that keeping the original long context makes it unclear whether the student utilizes compressed evidence. We therefore conduct two complementary inference ablations on LongBench v2: removing the original context, and perturbing the evidence while retaining the original context.

(1) Evaluating answering with evidence only. We first generate evidence EE from the source context CC and question QQ, then generate a new answer using only QQ and EE. For each backbone, we make two comparisons:

  1. Evidence generation: We use the same answer model to compare accuracy with base-generated and DVSD-generated evidence. This tests whether DVSD generates evidence that better supports answering.
  2. Evidence utilization: We provide the same evidence to the base and DVSD answer models and compare their accuracy. This tests whether DVSD training improves the model’s ability to use the supplied evidence for reasoning and answering.
BackboneAnswer modelBase-generated evidenceDVSD-generated evidence
Qwen3-4B-Instruct-2507Base35.838.0
Qwen3-4B-Instruct-2507DVSD36.041.4
Qwen3-8BBase36.438.8
Qwen3-8BDVSD38.246.3
Llama-3.1-8B-InstructBase23.924.1
Llama-3.1-8B-InstructDVSD31.431.6

All scores are LongBench v2 accuracy (%).

Evidence generation. With the answer model fixed, DVSD-generated evidence yields higher accuracy than base-generated evidence in all six comparisons, with an average gain of 3.08 percentage points. This indicates that DVSD generates evidence that is more useful for answering.

Evidence utilization. With the evidence fixed, the DVSD answer model also outperforms the base model in all six comparisons, with an average gain of 4.65 percentage points. Thus, DVSD improves answering performance even when the supplied evidence is unchanged.

(2) Perturbing evidence while retaining the original context. On Qwen3-4B-Instruct-2507, we run a second answering pass for both the base and DVSD checkpoints using CC, QQ, and evidence generated by the respective checkpoint. We compare original evidence with two perturbations: shuffled evidence, formed by randomly splitting EE into 5–10 segments and shuffling their order; and replaced evidence, constructed by partitioning CC into 100-token chunks and randomly selecting and concatenating these chunks to approximately match the length of the original evidence EE.

BackboneAnswer modelOriginal evidenceShuffled evidenceReplaced evidence
Qwen3-4B-Instruct-2507Base37.435.232.6
Qwen3-4B-Instruct-2507DVSD42.538.238.4

All three conditions retain the original context and question. Relative to the original evidence, DVSD accuracy drops by 4.3 percentage points with shuffled evidence and 4.1 percentage points with replaced evidence. These drops support that the content and organization of the supplied evidence affect answering even when the original context remains available. Together with the evidence-only results, this supports the usefulness of the generated evidence and its influence on answers in the tested inference settings. We have added these inference ablations to Section 4.3 (Tables 3 and 4), with detailed protocols and subset results in Appendix A.7 (Tables 9 and 10) of the revised manuscript.

Q2/W1: Evaluation of evidence quality

We evaluate the quality of evidence generated before and after DVSD training on LongBench v2 across three backbones.

We use DeepSeek-V4.1-Flash, a separate model from the training teachers, to evaluate the generated evidence. The judge model receives the original long context, the question, reference answer, EE, and assigns scores along three dimensions:

  • Relevance measures whether EE captures the information required by the question instead of preserving excessive irrelevant context.
  • Faithfulness measures whether EE retains the facts, entities, temporal attributes, and relational information from the source passage. This helps identify cases where the final answer is correct, but the intermediate evidence misrepresents the source.
  • Sufficiency measures whether EE retains all necessary conditions and reasoning steps for deriving the target answer.

Each dimension is scored on a scale from 0 to 5, with the detailed scoring rubric provided in Appendix A.7 (Figure 7).

Table I. Evidence quality (0–5) on LongBench v2. Avg. denotes the unweighted mean across the three dimensions.

BackboneMethodRelevanceFaithfulnessSufficiencyAvg.
Qwen3-4B-Instruct-2507Base3.443.423.193.35
Qwen3-4B-Instruct-2507DVSD3.623.803.393.60
Qwen3-8BBase2.743.822.382.98
Qwen3-8BDVSD3.533.923.313.59
Llama-3.1-8B-InstructBase1.653.591.422.22
Llama-3.1-8B-InstructDVSD2.454.292.323.02

DVSD improves the overall quality of compressed evidence, with average score gains of 0.25, 0.61, and 0.80 points on the three backbones, respectively. Relevance, faithfulness, and sufficiency improve across all three backbones.

We manually evaluated 20 evidence samples using the same scoring rubric as the DeepSeek judge. The table below compares the mean scores assigned by human evaluation and DeepSeek.

DimensionHuman meanDeepSeek meanMean score difference
Relevance3.903.750.15
Faithfulness4.053.950.10
Sufficiency3.703.450.25
Overall3.883.720.17

Human and DeepSeek mean scores differ by 0.10–0.25 points across the three dimensions (0.17 overall), providing a small-scale human check on the aggregate automatic scores. Together with the evidence-quality comparisons and controlled inference ablations, these results provide additional support for our conclusion that DVSD improves the quality of compressed evidence. We have added the evidence-quality results to Section 4.3 (Table 3) and the scoring rubric and human evaluation to Appendix A.7 (Figure 7 and Table 8) of the revised manuscript.

Q3/W2: Robustness across training seeds and comparisons on agent-memory tasks

(1) Robustness across training seeds. We repeat DVSD training with seeds 43 and 44 in addition to the original seed 42, keeping the other training settings unchanged.

Table III. Long-context performance on LongBench v2.

BackboneSeedEasyHardShortMediumLongOverall
Qwen3-4B-Instruct-25074246.942.849.442.838.944.3
Qwen3-4B-Instruct-25074346.937.651.138.130.641.2
Qwen3-4B-Instruct-25074452.137.648.941.437.043.1
Qwen3-8B4251.644.753.344.742.647.3
Qwen3-8B4355.745.761.743.341.749.5
Qwen3-8B4452.643.457.241.440.746.9
Llama-3.1-8B-Instruct4239.637.946.134.933.338.6
Llama-3.1-8B-Instruct4341.738.346.736.733.339.6
Llama-3.1-8B-Instruct4438.539.252.232.629.639.0

Seed 43 achieves higher overall accuracy than seed 42 for both Qwen3-8B and Llama-3.1-8B-Instruct, while seed 42 remains the best-performing run for Qwen3-4B-Instruct-2507. Across the three seeds, DVSD achieves mean overall accuracies of 42.87%, 47.90%, and 39.07%, respectively. These means remain above the strongest baseline scores reported in Table 1 for the corresponding backbones (41.6%, 44.7%, and 38.0%, respectively). We have added the three-seed results for all three backbones to Appendix A.5 (Table 5) of the revised manuscript.

(2) Matched comparisons on agent-memory tasks. We have added results for the OPSD and OPSDL baselines on MemoryAgentBench across all three backbones.

Table IV. Agent-memory performance (%). LongMemEval refers to the S∗S^* subset; EventQA refers to EventQA-full. Base and DVSD results are taken from Table 2 of the manuscript.

BackboneMethodLongMemEval: Micro AccLongMemEval: Task-Avg AccEventQA: Substring EMEventQA: RecallNLU: Substring EM
Llama-3.1-8B-InstructBase16.6719.0027.033.89.0
Llama-3.1-8B-InstructOPSD18.6721.1926.626.29.0
Llama-3.1-8B-InstructOPSDL11.6714.5228.436.28.0
Llama-3.1-8B-InstructDVSD22.3325.4132.637.412.0
Qwen3-4B-Instruct-2507Base13.3315.5638.035.617.0
Qwen3-4B-Instruct-2507OPSD12.6714.4127.827.462.0
Qwen3-4B-Instruct-2507OPSDL13.3315.5633.432.857.0
Qwen3-4B-Instruct-2507DVSD19.0021.3745.044.460.0
Qwen3-8BBase14.6718.1540.640.033.0
Qwen3-8BOPSD12.3315.1941.240.848.0
Qwen3-8BOPSDL13.3315.1935.435.242.0
Qwen3-8BDVSD16.0020.1943.843.648.0

DVSD outperforms both OPSD and OPSDL on all four LongMemEval and EventQA metrics across all three backbones. Compared with the stronger of OPSD and OPSDL for each backbone and metric, DVSD improves LongMemEval Micro Acc by 2.67–5.67 percentage points and EventQA Substring EM by 2.6–11.6 percentage points. On NLU, DVSD achieves the highest score on Llama-3.1-8B-Instruct (12.0%) and ties OPSD on Qwen3-8B (48.0%). On Qwen3-4B-Instruct-2507, DVSD reaches 60.0%, compared with 62.0% for OPSD and 57.0% for OPSDL. These results are included in Table 2 of the revised manuscript, with the agent-memory evaluation protocol described in Appendix A.4.

Reviewer_y1hWOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces the Dual-View Self-Distillation (DVSD) framework which jointly trains long-context evidence compression and answer generation using two complementary teacher models. Experiments across multiple models and benchmarks show consistent improvements over existing post-training methods in long-context reasoning.

Critical strengths
  • The paper proposes a simple and effective framework.
  • The experimental evaluation is comprehensive.
  • DVSD outperforms competing methods across long-context benchmarks.
Critical weaknesses

[W1] Although the paper aims to improve evidence compression, it does not systematically evaluate whether the generated evidence is relevant, faithful to the original context, or sufficient for answering. Could the authors provide a quantitative evaluation of evidence quality, potentially leveraging the available reference evidence annotations?

[W2]: During answer generation the student retains access to the original long context, making it unclear whether it actually relies on the generated evidence. An inference ablation removing either the original context or the compressed evidence would help establish whether the model genuinely learns to utilize its intermediate evidence.

Questions for authors

Please refer to the weaknesses

2 comments

Thank you for the constructive feedback. We address your concerns below.

W1: Evaluation of evidence quality

We evaluate the quality of evidence generated before and after DVSD training on LongBench v2 across three backbones.

We employ DeepSeek-V4.1-Flash as a judge to assign scores to the compressed evidence EE. The judge model receives the original long context, the question, reference answer, EE, and assigns scores along three dimensions:

  • Relevance measures whether EE captures the information required by the question instead of preserving excessive irrelevant context.
  • Faithfulness measures whether EE retains the facts, entities, temporal attributes, and relational information from the source passage. This helps identify cases where the final answer is correct, but the intermediate evidence misrepresents the source.
  • Sufficiency measures whether EE retains all necessary conditions and reasoning steps for deriving the target answer.

Each dimension is scored on a scale from 0 to 5, with the detailed scoring rubric provided in Appendix A.7 (Figure 7).

Table I. Evidence quality (0-5) on LongBench v2. Avg. denotes the unweighted mean across the three dimensions.

BackboneMethodRelevanceFaithfulnessSufficiencyAvg.
Qwen3-4B-Instruct-2507Base3.443.423.193.35
Qwen3-4B-Instruct-2507DVSD3.623.803.393.60
Qwen3-8BBase2.743.822.382.98
Qwen3-8BDVSD3.533.923.313.59
Llama-3.1-8B-InstructBase1.653.591.422.22
Llama-3.1-8B-InstructDVSD2.454.292.323.02

DVSD improves the overall quality of compressed evidence, with average score gains of 0.25, 0.61, and 0.80 points on the three backbones, respectively. Relevance, faithfulness, and sufficiency improve across all three backbones.

We manually evaluated 20 evidence samples using the same scoring rubric as the DeepSeek judge. The table below compares the mean scores assigned by human evaluation and DeepSeek.

DimensionHuman meanDeepSeek meanMean score difference
Relevance3.903.750.15
Faithfulness4.053.950.10
Sufficiency3.703.450.25
Overall3.883.720.17

Human and DeepSeek mean scores differ by 0.10–0.25 points across the three dimensions (0.17 overall), providing a small-scale human check on the aggregate automatic scores. Together with the evidence-quality comparisons and controlled inference ablations, these results provide additional support for our conclusion that DVSD improves the quality of compressed evidence. We have added the evidence-quality results to Section 4.3 (Table 3) and the scoring rubric and human evaluation to Appendix A.7 (Figure 7 and Table 8) of the revised manuscript.

W2: Inference ablations of evidence utilization

We agree that keeping the original long context makes it unclear whether the student utilizes compressed evidence. We therefore conduct two complementary inference ablations on LongBench v2: removing the original context, and perturbing the evidence while retaining the original context.

(1) Evaluating answering with evidence only. We first generate evidence EE from the source context CC and question QQ, then generate a new answer using only QQ and EE. For each backbone, we make two comparisons:

  1. Evidence generation: We use the same answer model to compare accuracy with base-generated and DVSD-generated evidence. This tests whether DVSD generates evidence that better supports answering.
  2. Evidence utilization: We provide the same evidence to the base and DVSD answer models and compare their accuracy. This tests whether DVSD training improves the model’s ability to use the supplied evidence for reasoning and answering.
BackboneAnswer modelBase-generated evidenceDVSD-generated evidence
Qwen3-4B-Instruct-2507Base35.838.0
Qwen3-4B-Instruct-2507DVSD36.041.4
Qwen3-8BBase36.438.8
Qwen3-8BDVSD38.246.3
Llama-3.1-8B-InstructBase23.924.1
Llama-3.1-8B-InstructDVSD31.431.6

All scores are LongBench v2 accuracy (%).

Evidence generation. With the answer model fixed, DVSD-generated evidence yields higher accuracy than base-generated evidence in all six comparisons, with an average gain of 3.08 percentage points. This indicates that DVSD generates evidence that is more useful for answering.

Evidence utilization. With the evidence fixed, the DVSD answer model also outperforms the base model in all six comparisons, with an average gain of 4.65 percentage points. Thus, DVSD improves answering performance even when the supplied evidence is unchanged.

(2) Perturbing evidence while retaining the original context. On Qwen3-4B-Instruct-2507, we run a second answering pass for both the base and DVSD checkpoints using CC, QQ, and evidence generated by the respective checkpoint. We compare original evidence with two perturbations: shuffled evidence, formed by randomly splitting EE into 5–10 segments and shuffling their order; and replaced evidence, constructed by partitioning CC into 100-token chunks and randomly selecting and concatenating these chunks to approximately match the length of the original evidence EE.

BackboneAnswer modelOriginal evidenceShuffled evidenceReplaced evidence
Qwen3-4B-Instruct-2507Base37.435.232.6
Qwen3-4B-Instruct-2507DVSD42.538.238.4

All three conditions retain the original context and question. Relative to the original evidence, DVSD accuracy drops by 4.3 percentage points with shuffled evidence and 4.1 percentage points with replaced evidence. These drops support that the content and organization of the supplied evidence affect answering even when the original context remains available. Together with the evidence-only results, this supports the usefulness of the generated evidence and its influence on answers in the tested inference settings. We have added these inference ablations to Section 4.3 (Tables 3 and 4), with detailed protocols and subset results in Appendix A.7 (Tables 9 and 10) of the revised manuscript.

Comments on the paper

4 comments

Thank you very much for the careful reading and constructive suggestions. We address your questions as follows.

Q1/W1: Forward-KL comparisons for OPSD and DVSD

We selected reverse KL for our OPSD adaptation because our initial comparisons on this long-context task yielded higher accuracy with reverse KL than with forward KL. To assess the stability of our experimental results across training seeds, we conduct additional runs using seeds 42, 43, and 44 for both OPSD and DVSD under forward and reverse KL on Qwen3-4B-Instruct-2507.

OPSD and DVSD use the same training data, backbone, response format, and generation budgets. For both the forward- and reverse-KL OPSD variants, we retain the stabilization settings used in the official Qwen3-4B non-thinking training script: pointwise KL clipping with a threshold of 10−610^{-6} and gradient-norm clipping with a maximum norm of 0.1. The teacher remains fixed at the initial checkpoint throughout training. These OPSD teacher and stabilization settings are now documented in Appendix A.3 of the revised manuscript.

Table I. LongBench v2 overall accuracy (%). SD denotes the sample standard deviation across three training seeds.

MethodDivergenceSeed 42Seed 43Seed 44Mean ± SD
OPSDReverse KL41.641.239.040.60 ± 1.40
OPSDForward KL37.639.439.238.73 ± 0.99
DVSD (default)Reverse KL44.341.243.142.87 ± 1.56
DVSDForward KL44.542.344.343.70 ± 1.22

Our default DVSD configuration (reverse KL) outperforms forward-KL OPSD in all three seeds, by 6.7, 1.8, and 3.9 percentage points, respectively, giving a mean paired gain of 4.13 percentage points. When both methods use forward KL, DVSD outperforms OPSD in all three seeds, with mean accuracy of 43.70% versus 38.73%, a gain of 4.97 percentage points. Thus, DVSD's advantage in this long-context setting persists when OPSD uses the forward-KL objective preferred in its original study. We have added this divergence ablation to Appendix A.6 (Tables 6 and 7) of the revised manuscript.

Q2/W2: Evidence quality and utilization (Part 1)

We agree that improved answer accuracy alone does not establish the faithfulness or sufficiency of the generated evidence, or whether the student uses it when the original context remains available. We have clarified in Section 3, immediately after Eq. (13), that the conditional-independence assumption describes an ideal answer-teacher target rather than a guaranteed property of the trained student. To address these concerns, we conduct three evaluations on LongBench v2: (1) an external evidence-quality audit with a human check; (2) evidence-only comparisons of evidence sources and answer models; and (3) inference ablations that remove the original context while keeping the evidence fixed, or perturb the evidence while retaining the original context.

(1) Auditing the generated evidence.

We evaluate base-generated and DVSD-generated evidence across three backbones. DeepSeek-V4.1-Flash receives the source context, question, reference answer, and generated evidence, and scores relevance, faithfulness, and sufficiency on a 0–5 scale. Faithfulness assesses whether the evidence is supported by the source; sufficiency assesses whether it contains the information needed to derive the reference answer. The detailed scoring rubric is provided in Appendix A.7 (Figure 7).

BackboneMethodRelevanceFaithfulnessSufficiencyAvg.
Qwen3-4B-Instruct-2507Base3.443.423.193.35
Qwen3-4B-Instruct-2507DVSD3.623.803.393.60
Qwen3-8BBase2.743.822.382.98
Qwen3-8BDVSD3.533.923.313.59
Llama-3.1-8B-InstructBase1.653.591.422.22
Llama-3.1-8B-InstructDVSD2.454.292.323.02

Avg. denotes the unweighted mean across the three dimensions.

DVSD improves the overall quality of compressed evidence under this audit, with average score gains of 0.25, 0.61, and 0.80 points, respectively. Relevance, faithfulness, and sufficiency improve across all three backbones.

We manually evaluated 20 evidence samples using the same scoring rubric as the DeepSeek judge. The table below compares the mean scores assigned by human evaluation and DeepSeek.

DimensionHuman meanDeepSeek meanMean score difference
Relevance3.903.750.15
Faithfulness4.053.950.10
Sufficiency3.703.450.25
Overall3.883.720.17

Human and DeepSeek mean scores differ by 0.10–0.25 points across the three dimensions (0.17 overall), providing a small-scale human check on the aggregate automatic scores. Together with the evidence-quality comparisons and controlled inference ablations, these results provide additional support for our conclusion that DVSD improves the quality of compressed evidence. We have added the evidence-quality results to Section 4.3 (Table 3) and the scoring rubric and human evaluation to Appendix A.7 (Figure 7 and Table 8) of the revised manuscript.

(2) Evaluating answering with evidence only.

We first generate evidence EE from the source context CC and question QQ, then generate a new answer using only QQ and EE. For each backbone, we make two comparisons:

  1. Evidence generation: We use the same answer model to compare accuracy with base-generated and DVSD-generated evidence. This tests whether DVSD generates evidence that better supports answering.
  2. Evidence utilization: We provide the same evidence to the base and DVSD answer models and compare their accuracy. This tests whether DVSD training improves the model’s ability to use the supplied evidence for reasoning and answering.
BackboneAnswer modelBase-generated evidenceDVSD-generated evidence
Qwen3-4B-Instruct-2507Base35.838.0
Qwen3-4B-Instruct-2507DVSD36.041.4
Qwen3-8BBase36.438.8
Qwen3-8BDVSD38.246.3
Llama-3.1-8B-InstructBase23.924.1
Llama-3.1-8B-InstructDVSD31.431.6

All scores are LongBench v2 accuracy (%).

Evidence generation. With the answer model fixed, DVSD-generated evidence yields higher accuracy than base-generated evidence in all six comparisons, with an average gain of 3.08 percentage points. This indicates that DVSD generates evidence that is more useful for answering.

Evidence utilization. With the evidence fixed, the DVSD answer model also outperforms the base model in all six comparisons, with an average gain of 4.65 percentage points. Thus, DVSD improves answering performance even when the supplied evidence is unchanged. These evidence-only comparisons are included in Section 4.3 (Table 3), with detailed protocols and subset results in Appendix A.7 (Table 9) of the revised manuscript.

Q2/W2: Evidence quality and utilization (Part 2)

(3) Removing the original context and perturbing evidence.

We conduct two complementary inference ablations on Qwen3-4B-Instruct-2507. Removing the original context tests whether the generated evidence can support answering on its own; perturbing the evidence tests whether it affects answering when the original context remains available.

For both the base and DVSD checkpoints, we first generate evidence EE from the source context CC and question QQ, then perform a separate answering pass. Each checkpoint uses its own generated evidence. We compare four conditions:

  • Context + original evidence: The answer model receives CC, QQ, and the original EE.
  • Original evidence only: We remove CC while keeping the same QQ, EE, answer model, and other answering settings.
  • Context + shuffled evidence: We retain CC and QQ, randomly split EE into 5–10 segments, and shuffle their order.
  • Context + replaced evidence: We retain CC and QQ, partition CC into 100-token chunks, and randomly select and concatenate chunks to approximately match the length of the original EE.
Answer modelContext + original evidenceOriginal evidence onlyContext + shuffled evidenceContext + replaced evidence
Base37.435.835.232.6
DVSD42.541.438.238.4

All scores are LongBench v2 accuracy (%). The question is included in every condition.

Answering without the original context. Using the same generated evidence, removing CC changes accuracy from 37.4% to 35.8% for the base model and from 42.5% to 41.4% for DVSD. These results show that the generated evidence can support answering without the original context, while access to the context provides an additional benefit. DVSD also outperforms the base model even when DVSD answers from evidence alone, and the base model retains access to both the context and evidence (41.4% vs. 37.4%), showing that DVSD’s overall advantage persists without the original context at answering time.

Evidence influence when the original context remains available. Relative to the original evidence, DVSD accuracy drops by 4.3 percentage points with shuffled evidence and 4.1 percentage points with replaced evidence. These drops indicate that answering is sensitive to the content and organization of the supplied evidence even when the original context remains available.

Together, these experiments support the usefulness of the generated evidence for answering and its influence on answers in the tested inference settings. The evidence-only and context-retaining results are reported in Section 4.3 (Tables 3 and 4), with detailed protocols and subset results in Appendix A.7 (Tables 9 and 10) of the revised manuscript.

We thank all the reviewers and Area Chairs for their time and feedback. During the rebuttal, we added experiments on evidence quality, controlled inference, training stability, and baseline comparisons, and incorporated the results and protocols into the revised manuscript.

1. DVSD improves quality of compressed evidence.

Across all three backbones, DVSD improves relevance, faithfulness, and sufficiency, with average evidence-quality gains of 0.25, 0.61, and 0.80 points on a 0–5 scale (R1/W1). We also added a human evaluation of 20 samples using the same rubric to check the aggregate automatic scores.

2. DVSD improves the model’s ability to answer using compressed evidence.

Our controlled inference experiments (R1/W2) remove the original context and separately vary the evidence source and answer model. With the answer model fixed, DVSD-generated evidence improves accuracy in all six comparisons, averaging 3.08 percentage points. With the evidence fixed, the DVSD answer model also performs better in all six comparisons, averaging 4.65 points. When the original context remains available, shuffling or replacing DVSD’s evidence reduces accuracy by 4.3 and 4.1 points, respectively. These results support improved evidence quality, better answering from identical evidence, and an influence of the supplied evidence on answer accuracy.

3. Contributions of the supervision design and teacher update.

We clarified that the OPSD baseline uses a frozen teacher, while epoch-wise replacement is part of DVSD’s design. With epoch-wise replacement, DVSD achieves 44.3, exceeding OPSD (41.6). DVSD also outperforms OPSD when both use a frozen teacher. With the teacher update schedule held fixed, the loss ablations also favor joint evidence and answer supervision over either loss alone across all three backbones.

4. Additional robustness checks and baseline comparisons on MemoryAgentBench.

We added three training seeds for each backbone (R2/Q3/W2). DVSD’s mean LongBench v2 accuracy exceeds the strongest reported baseline for each backbone. On Qwen3-4B-Instruct-2507, comparisons using the same KL direction show mean gains over OPSD of 2.27 points under reverse KL and 4.97 points under forward KL across three seeds (R3/Q1/W1). We also added OPSD and OPSDL results on MemoryAgentBench for all three backbones (R2/Q3/W2). DVSD outperforms both baselines on all four reported LongMemEval and EventQA metrics across these backbones.

The revised manuscript includes the additional results, detailed evaluation protocols, scoring rubric, and baseline configurations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.