- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
A purely theoretical paper asking whether accurate next-token prediction on one source tells a sparse-attention router which blocks to keep under distribution shift. In general it does not. Recovery is possible when at least one labeled source breaks the shortcut and a clear gap holds, or when masking preserves the answer, and the paper gives the sample costs. There are no experiments.
Critical strengths
- A clear framing of a question that is usually assumed away, and a useful distinction between refitting the predictor and auditing a frozen one.
Critical weaknesses
- No experiments at all, and this is the main problem. The paper asks a question about LLMs but argues that experiments on a concrete LLM would only test whether an implementation fits its interface; the only figure plots theorem bounds, and the appendix examples are explicitly not empirical evidence. For this question, experiments are not optional. Without a single one we cannot tell whether the failure it proves actually happens in real sparse-attention models, whether key assumptions such as a clear gap between good and bad rules hold, or whether the sample costs are realistic.
- Little guidance for improving models. The results do not tell designers which blocks to keep or how to build a better router, so they offer little help in optimizing sparse-attention models.
Questions for authors
- Can you add some experiments: map an existing sparse-attention method onto your framework and measure the gap on a real model?
- Does the main conclusion hold for a frozen LLM in practice?
- How could your results be used to improve a new sparse-attention model, for example to decide which blocks a router should keep?
Summary of Revisions and New Empirical Results
Dear Area Chair and Reviewers,
Thank you for your careful reading and constructive feedback. We appreciate the positive feedback on framing sparse attention as an identification problem, the theoretical results on compatible worlds, revealing source mixtures, and masking sample complexity, and the distinction between auditing a frozen predictor and refitting one.
In response to your feedback, we have made the following revisions.
Please refer to the latest PDF for the full set of revisions, experimental protocols, and results.