acceptodds
Withdrawn from ICLR 2027

Counterexamples to Local Reconstruction Gain as a Proxy for Final Fidelity in Residual Completion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Residual completion augments query-aware sparse attention by estimating the contribution of tokens omitted from the exact sparse computation. We ask whether improving a layer's attention-output reconstruction on the same incoming Q/K/V and selected support necessarily improves the fidelity of the final model output. We study training-free RESA and learned Top-K+ with frozen backbone language models. A prespecified single-layer screen yields two Qwen3-0.6B/Multi-LexSum interventions for which direct-runtime measurements show positive prespecified request-aggregate local reconstruction gain but worse final KL fidelity than the corresponding all-abstain Exact Top-K baseline on both discovery and prompt-token-disjoint holdout requests. Exact restoration at the same layer instead improves final fidelity, showing that the reversal is specific to approximate completion in these cases. In complementary multi-layer experiments, a task-independent local diagnostic often repairs the tested completion estimators, although the repaired models do not consistently outperform Exact Top-K. Together, these results show that better local reconstruction need not translate into better final-model fidelity.

Primary area:
generative models
Keywords:
KV cache, residual completion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Mt7hOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the relationship between local reconstruction quality and final model output fidelity in Sparse Attention. The authors examine whether Residual Completion can improve the final model’s output distribution by reducing the reconstruction error of a single layer’s Attention Output. Through single-layer interventions on two methods, RESA and Top-K+ϕ, the paper finds a counterexample on the Multi-LexSum task with Qwen3-0.6B in which the Local Reconstruction Gain is positive but Final KL Fidelity instead decreases, and validates this phenomenon on independent Holdout Requests. Further Exact Restoration experiments show that restoring exact Dense Attention at the same layer can improve final fidelity. In addition, the authors attempt to use local reconstruction metrics to screen and disable poorly performing Completion Layers in order to improve the model’s overall performance.

Critical strengths

S1. The research problem is of certain value, revealing the inconsistency between local optimization objectives and global model behavior. This paper questions the assumption that improved local Attention Reconstruction necessarily leads to improved final model output fidelity, and constructs a counterexample through actual model interventions. This finding offers useful insights for designing Sparse Attention and Residual Completion methods based on local reconstruction error.

Critical weaknesses

W1. The paper's presentation has obvious problems, and the allocation of space in the main text is unreasonable. The main text is only about eight pages, yet a large number of important experiments, controlled analyses, and results are placed in the appendix, resulting in insufficient argumentation for the core findings in the main text. For example, the complete Screening Results, sensitivity analyses under different Top-K Budgets, alternative aggregation methods for the Local Reconstruction Metric, and validation results on larger models mainly appear in the appendix. These contents are directly related to the credibility and scope of applicability of the core counterexample, rather than being secondary implementation details. In addition, the main text's description of the experimental protocol and statistical tests is relatively dense, further affecting the clarity of the research storyline.

W2. The validation scope of the core counterexample is relatively narrow, limiting the generality of the research findings. The two most rigorously validated counterexamples in the paper both come from single-layer interventions on Qwen3-0.6B in the Multi-LexSum task. Although the authors provide independent Holdout validation and supplementary experiments on other models, these results are still insufficient to establish the extent to which the inconsistency between Local Reconstruction and Final Fidelity is prevalent. In addition, the main experiments adopt a 65K context and a relatively small Top-K Budget, which constitutes a fairly aggressive sparse setting, and the frequency and severity of this phenomenon in practical deployment remain unclear. Can the authors conduct systematic validation across more model scales, tasks, and more lenient sparse budgets, and further quantify how often this phenomenon occurs under different settings?

Questions for authors

See weaknesses.

Reviewer_9WEqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors evaluate effects if different sparse attention mechanisms. However, due to insufficient explanations and introductions in the text, it is difficult to draw further insight into the work.

Critical strengths
  • Studying differences between sparse attention is an important topic.
Critical weaknesses
  • As is, the paper is not readable without going back to the source material of Top-K+ϕ\phi and RESA. If those are going to be the main models studied in the paper, they need to at least be briefly define mathematically so the reader has some idea of the models being studied.

  • Line 39 mentions “abstention” but there is no clear way to know what the authors are speaking of here. This term is subsequently used throughout the paper and leaves the reader guessing as to the definition throughout the text.

  • Equation 2 has a variable pp, and it is said to be the “corresponding all-abstain exact top-k path.” What does this mean and what is pp?


I would suggest that the authors do a complete rewrite of the manuscript from scratch, focusing on a clear introduction of the concepts being studied. The reader should not have to chase down and read multiple source papers in order to understand the basics of what is being presented.

The current form is unreadable. After trying to read the text multiple times, I still cannot understand what is being studied and the significance of the results.

Questions for authors

covered above in "weaknesses"

Reviewer_7SPGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study residual completion in sparse attention, asking whether a correction that improves a layer’s attention-output reconstruction also improves the final model’s agreement with dense attention. They evaluate Top-K+ϕ and RESA with frozen backbones. After screening 36 single-layer interventions, they identify two Qwen3-0.6B/Multi-LexSum cases where local reconstruction improves under the prespecified metric while final KL fidelity worsens. These results are reproduced on prompt-token-disjoint holdout requests using direct-runtime measurements. Replacing sparse attention with dense attention at the same layer instead improves final fidelity. The paper also tests disabling completion at layers with negative local gain on generic calibration data. This often improves the original completion models, but does not consistently outperform Exact Top-K.

Critical strengths

The authors verify that completion and abstention receive identical hidden states, Q/K/V, and selected token IDs at the measured positions, and record the attention outputs actually returned during execution. Together with the fixed interventions, multiplicity correction, and holdout evaluation, this makes the counterexamples considerably more convincing than a comparison of separately collected local and final metrics. The exact-restoration experiment provides a useful control: restoring dense attention at the selected layer improves final fidelity even where approximate completion worsens it. I also appreciate that the authors report unfavorable results, including aggregation sensitivity and failed transfer to PISA-0th, and explicitly distinguish improvement over the original completion model from improvement over Exact Top-K.

Critical weaknesses
  1. The paper establishes that reducing local attention-output error can increase final-output KL, but does not clearly show what this changes about our understanding or evaluation of residual completion. Section 5 describes prior completion methods without identifying a concrete conclusion or design decision that becomes unreliable because of these counterexamples. Sections 4.2 and 6 also leave the cause of the reversal unresolved. Consequently, the paper demonstrates that a failure can occur, but does not explain when to expect it or which existing practice should change. This is my main concern about the significance of the contribution. 
  2. The local-improvement premise of the ϕ holdout counterexample is not established under several alternative reconstruction measures reported by the authors. Table 1 reports positive median-based gain, G=0.345G=0.345, with a 95% interval of [0.244,0.404][0.244,0.404]. However, Table A1 reports row-mean gain of 0.095  [−0.047,0.183]0.095\;[-0.047,0.183], energy-pooled gain of −0.003  [−0.126,0.079]-0.003\;[-0.126,0.079], and absolute error reduction of 0.237  [−0.952,1.116]0.237\;[-0.952,1.116]. All three intervals cross zero. Thus, this experiment supports improvement in the prespecified median-based statistic, but does not establish a reduction in average absolute reconstruction error. The paper acknowledges this distinction, yet does not explain why the median-based statistic is the appropriate proxy to challenge in the first place. This matters because the interpretation could be that completion helps typical steps while occasionally causing large local errors, rather than that an overall reduction in local reconstruction error leads to worse final fidelity. The reported results do not distinguish these explanations.
Questions for authors
  1. Section 5 lists related completion methods, but which specific prior result or method-selection decision would be reconsidered because of your counterexamples? Please identify the relevant paper and its use of the local reconstruction metric. Is the aggregation used there the same as, or meaningfully comparable to, your within-request-median criterion? 
  2. For the ϕ holdout in Table A1, what accounts for the difference between the positive median-based gain and the inconclusive mean-based and absolute-error results? In the existing measurements, are the latter driven by a small number of steps or requests with large reconstruction losses? Showing where these losses occur, and whether the same requests account for the final KL degradation, would help determine what this counterexample demonstrates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.