- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how document-localization accuracy affects context interventions in long-context QA. It separately analyzes the gains after correct localization and the losses after incorrect localization. A gold-document control is used to measure the potential improvement. The authors use a small set of attention heads to select documents for attention biasing, document reordering, or pruning. They also use localization accuracy to predict whether an intervention will help or hurt on an unseen model. Experiments report an average improvement of 4.46 percentage points in answer-hit accuracy and 78.1% accuracy in predicting the direction of intervention effects.
Critical strengths
Useful analysis of localization and intervention effectiveness. The paper separates the gains after correct localization from the losses after incorrect localization and includes a gold-document control. This helps distinguish poor document selection from limited intervention effectiveness and provides a useful way to understand why improving localization accuracy does not always improve QA performance.
Critical weaknesses
Unclear additional value of the gate. Section 5.6 notes that the gate mainly helps by rejecting unreliable localizer families. For attention-head configurations, gating increases the mean QA gain from 4.46 to 4.62 percentage points, but the confidence interval for this difference includes zero. However, the paper does not compare the gate with a simple family-based rule, such as using each localizer family’s average QA gain on other models to decide whether to enable its interventions on a new model. Therefore, it is unclear whether collecting localization labels on a new model provides useful information beyond the localizer family itself.
Questions for authors
- Appendix E states that some wrong-target outcomes are imputed, and the reported simulator sign fidelity is 0.807. Could the authors explain how these outcomes were imputed and how the simulator was validated?
- Figure 1 is confusing and difficult to follow. The text is small, and too many boxes, arrows, equations, and numerical results are packed into a single figure. This makes it difficult for readers to identify the main message. I suggest that the authors simplify the layout and increase the font size to make the workflow clearer.
- The shaded region labeled “break-even” in Figure 2 is unclear. What does it represent, and how are its upper and lower bounds determined? Please explain this in the caption.
Thank you for the response, but the main concerns remain. The rebuttal attributes the MuSiQue result to a change in the target definition, where a hit is defined over a sufficient evidence set or reasoning chain, yet no results are reported under this definition. The position-level analysis of the 120-document stress test explains why the frozen threshold rejects positive configurations, but does not resolve the issue. It remains unclear whether a threshold calibrated at 20 to 30 documents can transfer to longer contexts.