- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DVSD trains a model to generate evidence followed by an answer, using two conditioning views of a shared self-teacher. The evidence teacher receives the original context, question, and reference answer, while the answer teacher receives only the question and student-generated evidence. The two regions are trained with token-level reverse-KL objectives, and the teacher is refreshed between epochs. Experiments evaluate long-context understanding, agent-memory tasks, and general capability retention, alongside ablations of teacher inputs, loss weights, and update strategies.
Critical strengths
1.The supervision design addresses a specific information-dependence problem. Reference answers guide evidence construction, while answer supervision depends on the student's evidence. The approach is straightforward and does not require human evidence annotations. 2. The empirical evaluation and ablations are reasonably comprehensive. Results across three backbones, evaluations beyond the training context length, and ablations of both losses and teacher inputs collectively support the design's usefulness.
Critical weaknesses
- The evidence has not been established as the student's actual information bottleneck. Because the student answers with access to the original context, it can bypass its generated evidence. Restricting the answer teacher does not establish that the student relies on evidence, and the ideal conditional-independence assumption does not guarantee this property after training.
- The robustness and attribution of the gains remain incomplete. Some improvements over the strongest alternatives are small, without reported multi-seed training uncertainty in the main results. The agent-memory evaluation primarily compares DVSD with the initial models, leaving unclear whether the transfer gains exceed those of competing post-training methods.
Questions for authors
- How does answer performance change when generated evidence is removed, shuffled, or replaced? What happens when the original context is removed and only the evidence remains?
- Does DVSD improve evidence faithfulness and answer sufficiency under an evaluation independent of the training teacher?
- Do the key improvements persist across training seeds, and do matched OPSD or OPSDL baselines show comparable gains on the memory tasks?
5 comments
Q2/W1: Evaluation of evidence quality
We evaluate the quality of evidence generated before and after DVSD training on LongBench v2 across three backbones.
We use DeepSeek-V4.1-Flash, a separate model from the training teachers, to evaluate the generated evidence. The judge model receives the original long context, the question, reference answer, , and assigns scores along three dimensions:
- Relevance measures whether captures the information required by the question instead of preserving excessive irrelevant context.
- Faithfulness measures whether retains the facts, entities, temporal attributes, and relational information from the source passage. This helps identify cases where the final answer is correct, but the intermediate evidence misrepresents the source.
- Sufficiency measures whether retains all necessary conditions and reasoning steps for deriving the target answer.
Each dimension is scored on a scale from 0 to 5, with the detailed scoring rubric provided in Appendix A.7 (Figure 7).
Table I. Evidence quality (0–5) on LongBench v2. Avg. denotes the unweighted mean across the three dimensions.
| Backbone | Method | Relevance | Faithfulness | Sufficiency | Avg. |
|---|---|---|---|---|---|
| Qwen3-4B-Instruct-2507 | Base | 3.44 | 3.42 | 3.19 | 3.35 |
| Qwen3-4B-Instruct-2507 | DVSD | 3.62 | 3.80 | 3.39 | 3.60 |
| Qwen3-8B | Base | 2.74 | 3.82 | 2.38 | 2.98 |
| Qwen3-8B | DVSD | 3.53 | 3.92 | 3.31 | 3.59 |
| Llama-3.1-8B-Instruct | Base | 1.65 | 3.59 | 1.42 | 2.22 |
| Llama-3.1-8B-Instruct | DVSD | 2.45 | 4.29 | 2.32 | 3.02 |
DVSD improves the overall quality of compressed evidence, with average score gains of 0.25, 0.61, and 0.80 points on the three backbones, respectively. Relevance, faithfulness, and sufficiency improve across all three backbones.
We manually evaluated 20 evidence samples using the same scoring rubric as the DeepSeek judge. The table below compares the mean scores assigned by human evaluation and DeepSeek.
| Dimension | Human mean | DeepSeek mean | Mean score difference |
|---|---|---|---|
| Relevance | 3.90 | 3.75 | 0.15 |
| Faithfulness | 4.05 | 3.95 | 0.10 |
| Sufficiency | 3.70 | 3.45 | 0.25 |
| Overall | 3.88 | 3.72 | 0.17 |
Human and DeepSeek mean scores differ by 0.10–0.25 points across the three dimensions (0.17 overall), providing a small-scale human check on the aggregate automatic scores. Together with the evidence-quality comparisons and controlled inference ablations, these results provide additional support for our conclusion that DVSD improves the quality of compressed evidence. We have added the evidence-quality results to Section 4.3 (Table 3) and the scoring rubric and human evaluation to Appendix A.7 (Figure 7 and Table 8) of the revised manuscript.
Q3/W2: Robustness across training seeds and comparisons on agent-memory tasks
(1) Robustness across training seeds. We repeat DVSD training with seeds 43 and 44 in addition to the original seed 42, keeping the other training settings unchanged.
Table III. Long-context performance on LongBench v2.
| Backbone | Seed | Easy | Hard | Short | Medium | Long | Overall |
|---|---|---|---|---|---|---|---|
| Qwen3-4B-Instruct-2507 | 42 | 46.9 | 42.8 | 49.4 | 42.8 | 38.9 | 44.3 |
| Qwen3-4B-Instruct-2507 | 43 | 46.9 | 37.6 | 51.1 | 38.1 | 30.6 | 41.2 |
| Qwen3-4B-Instruct-2507 | 44 | 52.1 | 37.6 | 48.9 | 41.4 | 37.0 | 43.1 |
| Qwen3-8B | 42 | 51.6 | 44.7 | 53.3 | 44.7 | 42.6 | 47.3 |
| Qwen3-8B | 43 | 55.7 | 45.7 | 61.7 | 43.3 | 41.7 | 49.5 |
| Qwen3-8B | 44 | 52.6 | 43.4 | 57.2 | 41.4 | 40.7 | 46.9 |
| Llama-3.1-8B-Instruct | 42 | 39.6 | 37.9 | 46.1 | 34.9 | 33.3 | 38.6 |
| Llama-3.1-8B-Instruct | 43 | 41.7 | 38.3 | 46.7 | 36.7 | 33.3 | 39.6 |
| Llama-3.1-8B-Instruct | 44 | 38.5 | 39.2 | 52.2 | 32.6 | 29.6 | 39.0 |
Seed 43 achieves higher overall accuracy than seed 42 for both Qwen3-8B and Llama-3.1-8B-Instruct, while seed 42 remains the best-performing run for Qwen3-4B-Instruct-2507. Across the three seeds, DVSD achieves mean overall accuracies of 42.87%, 47.90%, and 39.07%, respectively. These means remain above the strongest baseline scores reported in Table 1 for the corresponding backbones (41.6%, 44.7%, and 38.0%, respectively). We have added the three-seed results for all three backbones to Appendix A.5 (Table 5) of the revised manuscript.
(2) Matched comparisons on agent-memory tasks. We have added results for the OPSD and OPSDL baselines on MemoryAgentBench across all three backbones.
Table IV. Agent-memory performance (%). LongMemEval refers to the subset; EventQA refers to EventQA-full. Base and DVSD results are taken from Table 2 of the manuscript.
| Backbone | Method | LongMemEval: Micro Acc | LongMemEval: Task-Avg Acc | EventQA: Substring EM | EventQA: Recall | NLU: Substring EM |
|---|---|---|---|---|---|---|
| Llama-3.1-8B-Instruct | Base | 16.67 | 19.00 | 27.0 | 33.8 | 9.0 |
| Llama-3.1-8B-Instruct | OPSD | 18.67 | 21.19 | 26.6 | 26.2 | 9.0 |
| Llama-3.1-8B-Instruct | OPSDL | 11.67 | 14.52 | 28.4 | 36.2 | 8.0 |
| Llama-3.1-8B-Instruct | DVSD | 22.33 | 25.41 | 32.6 | 37.4 | 12.0 |
| Qwen3-4B-Instruct-2507 | Base | 13.33 | 15.56 | 38.0 | 35.6 | 17.0 |
| Qwen3-4B-Instruct-2507 | OPSD | 12.67 | 14.41 | 27.8 | 27.4 | 62.0 |
| Qwen3-4B-Instruct-2507 | OPSDL | 13.33 | 15.56 | 33.4 | 32.8 | 57.0 |
| Qwen3-4B-Instruct-2507 | DVSD | 19.00 | 21.37 | 45.0 | 44.4 | 60.0 |
| Qwen3-8B | Base | 14.67 | 18.15 | 40.6 | 40.0 | 33.0 |
| Qwen3-8B | OPSD | 12.33 | 15.19 | 41.2 | 40.8 | 48.0 |
| Qwen3-8B | OPSDL | 13.33 | 15.19 | 35.4 | 35.2 | 42.0 |
| Qwen3-8B | DVSD | 16.00 | 20.19 | 43.8 | 43.6 | 48.0 |
DVSD outperforms both OPSD and OPSDL on all four LongMemEval and EventQA metrics across all three backbones. Compared with the stronger of OPSD and OPSDL for each backbone and metric, DVSD improves LongMemEval Micro Acc by 2.67–5.67 percentage points and EventQA Substring EM by 2.6–11.6 percentage points. On NLU, DVSD achieves the highest score on Llama-3.1-8B-Instruct (12.0%) and ties OPSD on Qwen3-8B (48.0%). On Qwen3-4B-Instruct-2507, DVSD reaches 60.0%, compared with 62.0% for OPSD and 57.0% for OPSDL. These results are included in Table 2 of the revised manuscript, with the agent-memory evaluation protocol described in Appendix A.4.
Thank you for the constructive feedback. We address your concerns below.
Q1/W1: Inference ablations of evidence utilization
We agree that keeping the original long context makes it unclear whether the student utilizes compressed evidence. We therefore conduct two complementary inference ablations on LongBench v2: removing the original context, and perturbing the evidence while retaining the original context.
(1) Evaluating answering with evidence only. We first generate evidence E from the source context C and question Q, then generate a new answer using only Q and E. For each backbone, we make two comparisons:
All scores are LongBench v2 accuracy (%).
Evidence generation. With the answer model fixed, DVSD-generated evidence yields higher accuracy than base-generated evidence in all six comparisons, with an average gain of 3.08 percentage points. This indicates that DVSD generates evidence that is more useful for answering.
Evidence utilization. With the evidence fixed, the DVSD answer model also outperforms the base model in all six comparisons, with an average gain of 4.65 percentage points. Thus, DVSD improves answering performance even when the supplied evidence is unchanged.
(2) Perturbing evidence while retaining the original context. On Qwen3-4B-Instruct-2507, we run a second answering pass for both the base and DVSD checkpoints using C, Q, and evidence generated by the respective checkpoint. We compare original evidence with two perturbations: shuffled evidence, formed by randomly splitting E into 5–10 segments and shuffling their order; and replaced evidence, constructed by partitioning C into 100-token chunks and randomly selecting and concatenating these chunks to approximately match the length of the original evidence E.
All three conditions retain the original context and question. Relative to the original evidence, DVSD accuracy drops by 4.3 percentage points with shuffled evidence and 4.1 percentage points with replaced evidence. These drops support that the content and organization of the supplied evidence affect answering even when the original context remains available. Together with the evidence-only results, this supports the usefulness of the generated evidence and its influence on answers in the tested inference settings. We have added these inference ablations to Section 4.3 (Tables 3 and 4), with detailed protocols and subset results in Appendix A.7 (Tables 9 and 10) of the revised manuscript.