- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates the relationship between local reconstruction quality and final model output fidelity in Sparse Attention. The authors examine whether Residual Completion can improve the final model’s output distribution by reducing the reconstruction error of a single layer’s Attention Output. Through single-layer interventions on two methods, RESA and Top-K+ϕ, the paper finds a counterexample on the Multi-LexSum task with Qwen3-0.6B in which the Local Reconstruction Gain is positive but Final KL Fidelity instead decreases, and validates this phenomenon on independent Holdout Requests. Further Exact Restoration experiments show that restoring exact Dense Attention at the same layer can improve final fidelity. In addition, the authors attempt to use local reconstruction metrics to screen and disable poorly performing Completion Layers in order to improve the model’s overall performance.
Critical strengths
S1. The research problem is of certain value, revealing the inconsistency between local optimization objectives and global model behavior. This paper questions the assumption that improved local Attention Reconstruction necessarily leads to improved final model output fidelity, and constructs a counterexample through actual model interventions. This finding offers useful insights for designing Sparse Attention and Residual Completion methods based on local reconstruction error.
Critical weaknesses
W1. The paper's presentation has obvious problems, and the allocation of space in the main text is unreasonable. The main text is only about eight pages, yet a large number of important experiments, controlled analyses, and results are placed in the appendix, resulting in insufficient argumentation for the core findings in the main text. For example, the complete Screening Results, sensitivity analyses under different Top-K Budgets, alternative aggregation methods for the Local Reconstruction Metric, and validation results on larger models mainly appear in the appendix. These contents are directly related to the credibility and scope of applicability of the core counterexample, rather than being secondary implementation details. In addition, the main text's description of the experimental protocol and statistical tests is relatively dense, further affecting the clarity of the research storyline.
W2. The validation scope of the core counterexample is relatively narrow, limiting the generality of the research findings. The two most rigorously validated counterexamples in the paper both come from single-layer interventions on Qwen3-0.6B in the Multi-LexSum task. Although the authors provide independent Holdout validation and supplementary experiments on other models, these results are still insufficient to establish the extent to which the inconsistency between Local Reconstruction and Final Fidelity is prevalent. In addition, the main experiments adopt a 65K context and a relatively small Top-K Budget, which constitutes a fairly aggressive sparse setting, and the frequency and severity of this phenomenon in practical deployment remain unclear. Can the authors conduct systematic validation across more model scales, tasks, and more lenient sparse budgets, and further quantify how often this phenomenon occurs under different settings?
Questions for authors
See weaknesses.