- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that traditional relevance labels fail to capture generator-specific passage dependencies. It focuses on answerless bridges, passages that connect a question to its answer without containing the answer string, which may be necessary for some models but dispensable for others. The authors introduce a controlled deletion test to measure this model-specific necessity and evaluate it on HotpotQA, MuSiQue, DROP, and synthetic tasks. Results show that bridge deletion breaks more correct answers than matched control deletion (556 vs. 126 across 1,729 question–model pairs), and bridge necessity varies across models on the same question. Conventional relevance scores poorly predict necessity, while ContextCite attribution performs substantially better (AUC 0.728). Further experiments show increased dependence on answer-bearing notes after post-training and sensitivity of attention-based attribution to aggregation choices.
Critical strengths
A well-motivated intervention. The paper evaluates passage necessity through deletion while holding the question, answer, model, and remaining context fixed. The same-item control, together with robustness checks on control length and alternative controls, strengthens the evidence that answerless bridges contribute to correct multi-hop answers beyond the generic cost of removing text.
Broad empirical investigation. The study covers multiple model families, two multi-hop QA benchmarks, arithmetic reasoning, and controlled synthetic prompts. The authors also examine both model-specific necessity and attribution methods, providing a more comprehensive analysis than a single-benchmark demonstration.
Critical weaknesses
Problem framing and motivation. The introduction does not sufficiently clarify the connection between the limitations of traditional relevance labels and the model-specific necessity of answerless bridges. The title is engaging but does not explicitly convey the paper's core idea: a controlled deletion test for measuring model-specific passage necessity. This makes the motivation and novelty of the work harder to grasp.
Reliability of the necessity labels. The core necessity labels rely on a permissive answer-matching heuristic that accepts arbitrary substring matches (e.g., matching “art” within “cartwheel”) and uses a first-two-word fallback for multi-word answers. Since this scorer determines both sample eligibility and whether deleting a passage breaks an answer, false matches could directly affect the necessity labels and downstream results. The authors should validate the main findings using stricter answer-matching criteria.
Questions for authors
Incremental value of model-specific attribution. ContextCite achieves a cross-model AUC of 0.628, substantially below the model-level necessity-rate baseline (0.779), and its AUC drops to 0.583 after centering attribution scores by model. Can the authors clarify how much attribution contributes beyond models’ overall tendencies to depend on bridges? Does it provide meaningful passage-specific information for distinguishing which models need a given bridge?
Validity of bridge selection. For MuSiQue, the bridge is selected as the first supporting paragraph other than the answer-containing paragraph, even when an item has more than two supporting paragraphs. How often does this selection correspond to the intended intermediate passage in the reasoning chain, and do the results hold under alternative bridge-selection rules?
3 comments
Response to Reviewer tPG7 (2/2)
Q1. What does attribution add beyond each model’s necessity rate?
For the same question and bridge, the model’s necessity rate estimated from its other items is the stronger standalone predictor. Across 1,162 comparisons between models with opposite labels on 125 questions, this baseline reaches 0.762 AUC, compared with 0.636 for ContextCite and 0.583 [0.527, 0.636] after centering the attribution scores within each model. The centered scores retain modest discrimination, and we find limited evidence of an incremental benefit beyond the model-level baseline in this setting.
The clearest use of ContextCite in our experiments is ranking bridges for a given model, as shown in W2. This use follows Cohen-Wang et al. (2024) and Liu et al. (2025), who frame context attribution around how input passages contribute to a particular model response.
The evidence for cross-model dependence comes from the paired deletion experiment. Given the same question and context, removing the bridge can break one model’s answer while another model still answers correctly.
Q2. Does the MuSiQue bridge-selection rule identify the intended intermediate evidence?
Two-hop questions. On all 76 MuSiQue two-hop questions examined, the selected bridge supports the first annotated reasoning step. Together with the HotpotQA bridge questions, these cases account for 1,447 of 1,717 eligible question–model pairs. On this subset, where there is only one annotated non-answer supporting passage, bridge deletion broke 488 answers versus 96 for control deletion. The paired difference was significant in all 11 models.
Three- and four-hop questions. The original rule selects an annotated supporting paragraph other than the answer paragraph. The selected paragraph corresponds to the penultimate annotated step in 22 of 55 questions. To answer your question about alternative rules, we ran a new paired deletion experiment on 187 eligible question–model pairs in the full MuSiQue contexts. With all supporting paragraphs initially present, we separately tested deleting the first-hop paragraph, the penultimate-step paragraph, and the same-item non-supporting control.
| Passage deleted | Answer break rate |
|---|---|
| First-hop supporting paragraph | 17.1% |
| Penultimate-step supporting paragraph | 26.7% |
| Same-item non-supporting control | 9.6% |
The choice of reasoning step affects the size of the deletion effect. Under both selection rules, deleting the supporting paragraph breaks more answers than deleting the non-supporting control. The two-hop subset shows that the main result also holds where bridge selection is unambiguous. We added the new alternative-rule experiment to Appendix J, Table 12.
References
- Cohen-Wang, B., Shah, H., Georgiev, K., & Mądry, A. (2024). ContextCite: Attributing Model Generation to Context. NeurIPS.
- Liu, F., Kandpal, N., & Raffel, C. (2025). AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution. ICLR.
Please engage with the authors during the discussion period.
Dear reviewer,
The authors provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale. (I realize that there is only one business day between when the authors posted their response and when I sent this reminder -- please read it as just a friendly reminder and not an implication that I think you're being unresponsive).
Response to Reviewer tPG7 (1/2)
We are grateful for your thoughtful review and for highlighting how the same-item control strengthens the deletion test, as well as the breadth of our evaluation across models and tasks. In response to your questions, we reran the cross-document experiments with stricter answer matching and tested alternative bridge-selection rules on MuSiQue.
W1. Problem framing and title
Our main contribution is a paired deletion test that measures whether a model needs a particular passage to answer a question correctly. When two models read the same passages and both answer correctly, deleting the same intermediate passage can make one model answer incorrectly while the other still succeeds. The benchmark’s supporting-passage annotation stays the same in both cases.
That observation motivates the title. Answerless Bridges names the evidence we study, and Relevant to Whom? asks whose answer depends on it. The benchmark identifies a passage that supports the annotated reasoning chain. We ask whether the model answering the question depends on that passage in the context it receives. This distinction shapes what we measure, which passages we study, how we test their contribution, and how we evaluate existing scores.
We measure passage necessity for a specific model. For a given question and context, we observe whether removing a passage changes the model’s answer from correct to incorrect. This gives us a way to compare models on the same question and passages.
Answerless bridges let us test dependence on intermediate evidence. A bridge can connect the question to the answer-bearing passage without stating the answer itself. Deleting the bridge while keeping the answer-bearing passage available lets us test whether the model still needs that intermediate step.
The paired deletion test gives us a same-item control. We regenerate the answer after removing the bridge and after separately removing a non-supporting passage from the same context. Comparing the two outcomes helps distinguish the effect of losing the bridge from the effect of shortening the context.
The resulting labels give us a way to evaluate relevance and attribution scores. Among passages already annotated as supporting evidence, we test whether a score ranks the bridges a particular model needs above those it can dispense with. This gives the question in the title a concrete implication for how passage scores are evaluated.
We appreciate your feedback and will revise the introduction to clarify the motivation and contributions of the paper.
W2. Reliability of the necessity labels
To address your concern, we recomputed both decisions using two stricter scoring rules. Under each rule, we determined which question–model pairs were eligible and whether each regenerated answer was correct after deletion. The whole-token rule requires the complete normalized reference answer or a MuSiQue alias to appear as consecutive whole tokens. This prevents
artfrom matchingcartwheeland removes the first-two-word fallback. The second rule requires the normalized first line of the response to match the reference answer exactly.With whole-token scoring, bridge deletion causes more answer breaks on both HotpotQA (273 vs. 50) and MuSiQue (276 vs. 72). The paired difference is significant in all 11 models when pooling the two datasets. The main deletion result holds under both stricter rules, even when each rule is used for both sample eligibility and post-deletion correctness. We added these scoring robustness experiments to the revised manuscript and report the comparison with the original scorer in Appendix J, Table 10.
The attribution result also persists under both scoring rules. ContextCite achieves a pooled AUC of 0.732 [0.687, 0.774] under whole-token scoring and 0.712 under exact first-line scoring. Under each rule, it outperforms every tested question–passage relevance score within each of the 11 models. baseline under whole-token scoring is MiniLM at 0.545 pooled AUC.