acceptodds
Under review as a conference paper at ICLR 2027

Answerless Bridges: Relevant to Whom? One Model Needs Them, Another Does Without

Abstract

Retrievers for language models are trained and evaluated with relevance labels: a passage is relevant when it contains the answer, resembles the question, or is annotated as supporting evidence. Each label reads only the question and the passage, so it gives every model the same verdict and never asks: relevant to whom? Multi-hop questions make this concrete, because a model can depend on an answerless bridge, a passage without the answer that links the question to the passage containing it. For each model, we delete either the bridge or a control passage from the same context and call the bridge necessary when deleting it alone breaks a correct answer. In each of eleven open-weight models on HotpotQA and MuSiQue, bridge deletion breaks more answers than control deletion, yet for most questions evaluated with several models, the same bridge is necessary for some models and dispensable for others. Lexical, dense and cross-encoder relevance scores give these models one value and predict need no better than chance. ContextCite attribution, fitted to each model's responses to context ablations, predicts need in all ten models tested, and on a shared question it separates the models that need the bridge from those that do not, partly through each model's base rate. The same test recovers the operands of DROP arithmetic questions, whose answers appear in no sentence, and on synthetic prompts shown identically to base and instruction-tuned models it shows that post-training strengthens dependence on the passages holding the answer. The answer to "relevant to whom?" is a particular model, whose own responses measure its need.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Interpretability, Retrieval-Augmented Generation, Question Answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tPG7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that traditional relevance labels fail to capture generator-specific passage dependencies. It focuses on answerless bridges, passages that connect a question to its answer without containing the answer string, which may be necessary for some models but dispensable for others. The authors introduce a controlled deletion test to measure this model-specific necessity and evaluate it on HotpotQA, MuSiQue, DROP, and synthetic tasks. Results show that bridge deletion breaks more correct answers than matched control deletion (556 vs. 126 across 1,729 question–model pairs), and bridge necessity varies across models on the same question. Conventional relevance scores poorly predict necessity, while ContextCite attribution performs substantially better (AUC 0.728). Further experiments show increased dependence on answer-bearing notes after post-training and sensitivity of attention-based attribution to aggregation choices.

Critical strengths

A well-motivated intervention. The paper evaluates passage necessity through deletion while holding the question, answer, model, and remaining context fixed. The same-item control, together with robustness checks on control length and alternative controls, strengthens the evidence that answerless bridges contribute to correct multi-hop answers beyond the generic cost of removing text.

Broad empirical investigation. The study covers multiple model families, two multi-hop QA benchmarks, arithmetic reasoning, and controlled synthetic prompts. The authors also examine both model-specific necessity and attribution methods, providing a more comprehensive analysis than a single-benchmark demonstration.

Critical weaknesses

Problem framing and motivation. The introduction does not sufficiently clarify the connection between the limitations of traditional relevance labels and the model-specific necessity of answerless bridges. The title is engaging but does not explicitly convey the paper's core idea: a controlled deletion test for measuring model-specific passage necessity. This makes the motivation and novelty of the work harder to grasp.

Reliability of the necessity labels. The core necessity labels rely on a permissive answer-matching heuristic that accepts arbitrary substring matches (e.g., matching “art” within “cartwheel”) and uses a first-two-word fallback for multi-word answers. Since this scorer determines both sample eligibility and whether deleting a passage breaks an answer, false matches could directly affect the necessity labels and downstream results. The authors should validate the main findings using stricter answer-matching criteria.

Questions for authors

Incremental value of model-specific attribution. ContextCite achieves a cross-model AUC of 0.628, substantially below the model-level necessity-rate baseline (0.779), and its AUC drops to 0.583 after centering attribution scores by model. Can the authors clarify how much attribution contributes beyond models’ overall tendencies to depend on bridges? Does it provide meaningful passage-specific information for distinguishing which models need a given bridge?

Validity of bridge selection. For MuSiQue, the bridge is selected as the first supporting paragraph other than the answer-containing paragraph, even when an item has more than two supporting paragraphs. How often does this selection correspond to the intended intermediate passage in the reasoning chain, and do the results hold under alternative bridge-selection rules?

3 comments

Response to Reviewer tPG7 (1/2)

We are grateful for your thoughtful review and for highlighting how the same-item control strengthens the deletion test, as well as the breadth of our evaluation across models and tasks. In response to your questions, we reran the cross-document experiments with stricter answer matching and tested alternative bridge-selection rules on MuSiQue.

W1. Problem framing and title

Our main contribution is a paired deletion test that measures whether a model needs a particular passage to answer a question correctly. When two models read the same passages and both answer correctly, deleting the same intermediate passage can make one model answer incorrectly while the other still succeeds. The benchmark’s supporting-passage annotation stays the same in both cases.

That observation motivates the title. Answerless Bridges names the evidence we study, and Relevant to Whom? asks whose answer depends on it. The benchmark identifies a passage that supports the annotated reasoning chain. We ask whether the model answering the question depends on that passage in the context it receives. This distinction shapes what we measure, which passages we study, how we test their contribution, and how we evaluate existing scores.

  1. We measure passage necessity for a specific model. For a given question and context, we observe whether removing a passage changes the model’s answer from correct to incorrect. This gives us a way to compare models on the same question and passages.

  2. Answerless bridges let us test dependence on intermediate evidence. A bridge can connect the question to the answer-bearing passage without stating the answer itself. Deleting the bridge while keeping the answer-bearing passage available lets us test whether the model still needs that intermediate step.

  3. The paired deletion test gives us a same-item control. We regenerate the answer after removing the bridge and after separately removing a non-supporting passage from the same context. Comparing the two outcomes helps distinguish the effect of losing the bridge from the effect of shortening the context.

  4. The resulting labels give us a way to evaluate relevance and attribution scores. Among passages already annotated as supporting evidence, we test whether a score ranks the bridges a particular model needs above those it can dispense with. This gives the question in the title a concrete implication for how passage scores are evaluated.

We appreciate your feedback and will revise the introduction to clarify the motivation and contributions of the paper.

W2. Reliability of the necessity labels

To address your concern, we recomputed both decisions using two stricter scoring rules. Under each rule, we determined which question–model pairs were eligible and whether each regenerated answer was correct after deletion. The whole-token rule requires the complete normalized reference answer or a MuSiQue alias to appear as consecutive whole tokens. This prevents art from matching cartwheel and removes the first-two-word fallback. The second rule requires the normalized first line of the response to match the reference answer exactly.

Four-passage testOriginal scorerWhole-token scorerExact first-line scorer
Eligible question–model pairs1,7291,7171,263
Bridge vs. control answer breaks556 vs. 126549 vs. 122433 vs. 93

With whole-token scoring, bridge deletion causes more answer breaks on both HotpotQA (273 vs. 50) and MuSiQue (276 vs. 72). The paired difference is significant in all 11 models when pooling the two datasets. The main deletion result holds under both stricter rules, even when each rule is used for both sample eligibility and post-deletion correctness. We added these scoring robustness experiments to the revised manuscript and report the comparison with the original scorer in Appendix J, Table 10.

The attribution result also persists under both scoring rules. ContextCite achieves a pooled AUC of 0.732 [0.687, 0.774] under whole-token scoring and 0.712 under exact first-line scoring. Under each rule, it outperforms every tested question–passage relevance score within each of the 11 models. baseline under whole-token scoring is MiniLM at 0.545 pooled AUC.

Response to Reviewer tPG7 (2/2)

Q1. What does attribution add beyond each model’s necessity rate?

For the same question and bridge, the model’s necessity rate estimated from its other items is the stronger standalone predictor. Across 1,162 comparisons between models with opposite labels on 125 questions, this baseline reaches 0.762 AUC, compared with 0.636 for ContextCite and 0.583 [0.527, 0.636] after centering the attribution scores within each model. The centered scores retain modest discrimination, and we find limited evidence of an incremental benefit beyond the model-level baseline in this setting.

The clearest use of ContextCite in our experiments is ranking bridges for a given model, as shown in W2. This use follows Cohen-Wang et al. (2024) and Liu et al. (2025), who frame context attribution around how input passages contribute to a particular model response.

The evidence for cross-model dependence comes from the paired deletion experiment. Given the same question and context, removing the bridge can break one model’s answer while another model still answers correctly.

Q2. Does the MuSiQue bridge-selection rule identify the intended intermediate evidence?

Two-hop questions. On all 76 MuSiQue two-hop questions examined, the selected bridge supports the first annotated reasoning step. Together with the HotpotQA bridge questions, these cases account for 1,447 of 1,717 eligible question–model pairs. On this subset, where there is only one annotated non-answer supporting passage, bridge deletion broke 488 answers versus 96 for control deletion. The paired difference was significant in all 11 models.

Three- and four-hop questions. The original rule selects an annotated supporting paragraph other than the answer paragraph. The selected paragraph corresponds to the penultimate annotated step in 22 of 55 questions. To answer your question about alternative rules, we ran a new paired deletion experiment on 187 eligible question–model pairs in the full MuSiQue contexts. With all supporting paragraphs initially present, we separately tested deleting the first-hop paragraph, the penultimate-step paragraph, and the same-item non-supporting control.

Passage deletedAnswer break rate
First-hop supporting paragraph17.1%
Penultimate-step supporting paragraph26.7%
Same-item non-supporting control9.6%

The choice of reasoning step affects the size of the deletion effect. Under both selection rules, deleting the supporting paragraph breaks more answers than deleting the non-supporting control. The two-hop subset shows that the main result also holds where bridge selection is unambiguous. We added the new alternative-rule experiment to Appendix J, Table 12.

References

  • Cohen-Wang, B., Shah, H., Georgiev, K., & Mądry, A. (2024). ContextCite: Attributing Model Generation to Context. NeurIPS.
  • Liu, F., Kandpal, N., & Raffel, C. (2025). AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution. ICLR.
Area_Chair_cqNuOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale. (I realize that there is only one business day between when the authors posted their response and when I sent this reminder -- please read it as just a friendly reminder and not an implication that I think you're being unresponsive).

Reviewer_5NvoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Standard relevance labels (answer containment, question–passage similarity, gold supporting-passage annotation) read only the question and the passage, so they give every model the same verdict; this paper asks “relevant to whom?” and answers it with a per-model deletion label on “answerless bridges” — passages with no answer string that connect the question to the answer passage — where a bridge is necessary if deleting it alone breaks an otherwise-correct answer, benchmarked against deleting a same-item control from the same context, with no thresholds or fitted parameters.

Across 11 open-weight models on HotpotQA/MuSiQue (1,729 rows), bridge deletion breaks 556 answers vs. 126 for the control, yet on 120 of 180 shared items the same bridge is necessary for some models and dispensable for others; query-only relevance scores predict this at chance (AUC 0.45–0.53), while ContextCite attribution reaches 0.728 and graded deletion 0.825.

Three auxiliary settings extend the claim: DROP recovers operand sentences whose answers appear in no sentence, a source-count setting shows writer mass spread over jointly-sufficient sentences, and a synthetic base-vs-instruct comparison on identical prompts shows post-training raises deletion sensitivity 1.4–4× while leaving argmax credit unchanged. Section 7 adds a cautionary result: two aggregation conventions for attention-based attribution pass disjoint model sets, and only the positive-half verdict tracks accuracy.

Critical strengths
  1. Evidence is broad and consistent. Four settings, 11 checkpoints, natural + constructed-answer + synthetic stimuli all point the same way, and the core result holds in 10/11 models and in both corpora separately.
  2. Statistical practice is careful. Threshold-free label, continuous graded-deletion reference, exact McNemar with item-level bootstrap, and readouts graded against an interventional ground truth rather than annotations they could memorize.
Critical weaknesses
  1. The label is conditional on success. Eligibility requires correct-with-context and wrong-closed-book on 4-passage, ≤700-char contexts with gold bridges, so reported rates describe models that already solved the item, not top-20 retrieval where the bridge may be missing or duplicated.
  2. Control design is thin for the weight it carries. One dataset-ordered distractor (11% longer than the bridge) per item gives no control-to-control variability or null distribution over "some other passage."
  3. Cross-model disagreement is confounded. Models differ jointly in scale, family, pretraining data, and accuracy, and the two largest run in 4-bit NF4 while others are unquantized, so Fig. 4d may partly re-express accuracy/quantization; the Qwen2.5-7B ContextCite failure is silently dropped.
Questions for authors
  1. Can you show one downstream use — reranking or retriever training with per-model necessity labels — that beats gold supporting annotations?
  2. Why only one control per item; what do all-distractor deletions or a leave-one-out control distribution give, and what variance does that imply for McNemar?
  3. Is the base→tuned deletion-sensitivity gain genuine context reliance or surface factors (answer length, format compliance, calibration)?
  4. Graded deletion scores teacher-forced log_p(gold); do labels and rankings change if you delete and re-decode, or score the model's own first-pass answer?
3 comments

Response to Reviewer 5Nvo (1/2)

Thank you for the careful review and for highlighting the consistency of the deletion results across settings. In response to your comments, we repeated the deletion test in the full benchmark contexts, varied the distractor control, checked the effect of quantization, and evaluated the graded deletion score using each model’s own answer. The cross-document analyses use the stricter answer scorer described in our general response.

W1. The necessity label is conditional on success

The label measures whether removing a passage breaks an otherwise correct answer for a given model and context. We evaluate it on question–model pairs for which the model answers correctly with context and incorrectly without it. The reported necessity rates therefore describe this population.

To test whether the result persists in longer contexts, we repeated the paired deletion test with all 10 HotpotQA or 20 MuSiQue benchmark paragraphs, including every annotated supporting paragraph and without a passage-length cap. Across 1,313 eligible pairs, bridge deletion broke 426 answers, compared with 84 for control deletion. The paired difference was significant for all 11 models when pooling the datasets. Of the 212 questions eligible for multiple models, 120 received both necessity labels across models. We report the benchmark-level results in Appendix J.

These results show that model-specific bridge dependence persists when the complete benchmark context is supplied. The label measures the effect of removing a passage from that context. If another passage supplies the same evidence, that evidence can remain available after deletion. Whether the retriever supplies the bridge in the first place is a question of retrieval recall.

W2. Sensitivity to the choice of control passage

We used the first distractor in dataset order as a reproducible, same-item control. As the review notes, the control is longer on average than the bridge. The submitted manuscript’s Table 9 reports mean lengths of 65.82 and 59.35 words, respectively. On the 994 pairs where the control was at least as long as the bridge, the graded bridge–control gap was still +1.76 nats [1.40, 2.16].

We tested sensitivity to the choice of distractor by using the other distractor as a control and by comparing the two distractors directly.

Four-passage comparisonAnswer breaksDiscordant pairsPaired test
Bridge vs. original distractor549 vs. 122476 vs. 49Significant in 11/11 models
Bridge vs. second distractor549 vs. 134474 vs. 59Significant in 11/11 models
Original vs. second distractor122 vs. 13478 vs. 90Pooled two-sided p = 0.40, significant in 0/11 models

The graded difference between the two distractors was −0.02 nats [−0.13, +0.09]. In the full benchmark contexts, we also repeated the test over 2,000 draws, choosing one random distractor per item in each draw. The distractor break rate was 0.073 [0.062, 0.085] across draws, compared with 0.324 for the bridge. The pooled McNemar p-value was at most 3.5 × 10⁻⁵⁸ in every draw, and at least 10 of 11 model-level tests were significant in each draw.

The bridge–control difference persists with either fixed distractor and across the randomly sampled controls. The direct distractor comparison shows the amount of variation between the two controls in the four-passage setting.

W3. Cross-model disagreement, quantization, and Qwen2.5-7B

Open-book accuracy and Figure 4d. Figure 4d summarizes how often each model needs a bridge across its eligible items. This rate decreases with open-book accuracy (Spearman ρ = −0.752, p = 0.010). The same-item analysis asks whether two models need the same passage. Pairwise disagreement is 28.3% overall, 27.8% among models whose open-book accuracies are within five percentage points, and 26.1% among unquantized models from the same family. Open-book accuracy is associated with a model’s average necessity rate, and models with similar accuracy still differ in which individual bridges they need.

Quantization. We ran Qwen2.5-14B in both bf16 and NF4. Its labels agreed on 91.7% of the 120 pairs eligible under both settings (κ = 0.833), and the necessity rates were 0.492 and 0.475, respectively. For this checkpoint, changing precision left most individual labels and the overall necessity rate similar.

Qwen2.5-7B. We traced the ContextCite fitting failure to overflow during fp16 ablation passes. With bf16 ablation passes, ContextCite fits all 136 eligible Qwen2.5-7B items. The updated analysis covers 11 models and 1,227 rows. ContextCite achieves a pooled AUC of 0.732 [0.687, 0.774] and outperforms every tested relevance score within each model. The within-model ranking result therefore holds with Qwen2.5-7B included.

Response to Reviewer 5Nvo (2/2)

Q1. Can model-specific necessity improve reranking over gold supporting annotations?

We evaluated this in an offline ranking task among annotated bridges, using the target generator’s deletion-based necessity labels as the outcome. Gold supporting annotations give every candidate the same score, yielding 0.500 AUC. MiniLM, the strongest tested question–passage relevance score, reaches 0.545. ContextCite improves on these scores, as reported in W3, and graded deletion using the reference answer reaches 0.807. When used as positive labels for the target generator, gold annotations have precision 0.295, compared with 0.520 when transferring necessity labels from another model.

The offline comparison shows that model-specific scores distinguish which annotated bridges a generator needs. For n passages, the deletion stage uses one full-context decode and n single-deletion decodes. On one L40S, the experiments took 2.5–17.8 seconds per item for four passages and 9.7–49.5 seconds for full benchmark contexts. These measurements motivate training a generator-specific reranker. Its effect on retrieval or QA performance would require a downstream evaluation on held-out examples.

Q2. Why use one control per item, and how sensitive is McNemar’s test to that choice?

A fixed same-item distractor gives a reproducible paired comparison. W2 reports the results for both available distractors in the four-passage setting, the direct distractor-to-distractor comparison, and the distribution over randomly drawn controls in the full contexts. The McNemar result persists across the sampled control choices.

Q3. Does the base-to-tuned increase in deletion sensitivity reflect reliance on context?

Base and tuned checkpoints see byte-identical prompts and score the same four candidate phrases. The candidate answers and response format are fixed across checkpoints. Deletion sensitivity increases by 1.44× to 4.09×, while source-attribution argmax accuracy changes from 0.731 to 0.773 (p = 0.091). The sensitivity increase also persists on the 173 comparisons where both checkpoints select the correct answer.

To probe calibration, we compared the effect of deleting the queried chain with the effect of deleting a structurally matched chain about an unqueried subject. The control-adjusted increases were +4.06, +5.72, and +1.56 nats for OLMo-2-7B, Qwen2.5-7B, and Qwen2.5-1.5B, respectively, with confidence intervals above zero. Gemma-2-2B gave −1.34 nats [−3.43, +0.48].

For three of the four checkpoint pairs, tuning increased sensitivity to the queried evidence relative to the matched unqueried evidence. This supports an evidence-specific increase in context reliance in the synthetic task.

Q4. What changes with re-decoding or scoring the model’s own answer?

The binary necessity label already uses re-decoding. After each deletion, we generate a new greedy answer and score its correctness. Teacher-forced log probability is used for the graded deletion score.

Using the model’s own full-context answer as the scoring target raises pooled AUC against the binary label to 0.851, compared with the reference-answer result in Q1. Same-item cross-model AUC is nearly identical (0.803 with the reference answer and 0.802 with the model’s own answer). The graded bridge–control contrast is positive with either target (+1.53 and +2.48 nats, respectively). The choice of scoring target affects pooled discrimination, while the same-item cross-model result and the direction of the bridge–control contrast are stable.

Area_Chair_cqNuOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale. (I realize that there is only one business day between when the authors posted their response and when I sent this reminder -- please read it as just a friendly reminder and not an implication that I think you're being unresponsive).

Reviewer_vYYZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper challenges the assumption that passage relevance is a model-agnostic property, focusing on "answerless bridges": supporting passages in multi-hop QA that do not contain the answer but link the question to the passage that does. For each model, the authors keep only questions answered correctly with the full context and incorrectly closed-book, and then delete either the bridge or a non-supporting control passage from the same four-passage context. A bridge is labelled necessary for a model if deleting it alone breaks the correct answer, and dispensable otherwise.

Across eleven open-weight models on HotpotQA and MuSiQue, bridge deletion breaks more answers than control deletion in every model. Findings indicate that some models need the bridge while others do not. Query-based relevance scores (i.e. with TF-IDF, BM25, MiniLM and two cross-encoders) do not separate necessary from dispensable bridges, whereas ContextCite attribution does. Furthermore, the authors test two additional settings.

  • On DROP arithmetic questions, where the needed sentences are known by construction, deletion correctly identifies them.
  • On synthetic prompts given identically to base and instruction-tuned versions of the same models, post-training appears to increase reliance on the passages that contain the answer words.
Critical strengths
  • Asking "relevant to whom?" about in-context passages is a useful step back, as it reframes passage relevance in RAG as a property of the reading model rather than of the passage alone. Showing that the same bridge, in an identical context, is necessary for some models and dispensable for others questions the use of model-agnostic labels for training and evaluating retrievers.
  • The paired same-item deletion test is clean and parameter-free, and it is backed by a good set of robustness checks.
Critical weaknesses
  • The four-passage setting is not realistic for today's RAGs. Every item contains only the answer passage, one bridge and two distractors of at most 700 characters, which is far from realistic RAG contexts with many partially relevant passages. For MuSiQue questions with three or four hops, the remaining supporting paragraphs are dropped, so a correct answer must rely on the model's memory for the missing hops.
    • The authors acknowledge the four-passage scope in the Discussion, but Abstract and Intro state the findings without this qualification, and results are not broken down by hop count.
  • The practical implications for retrieval are not demonstrated. The paper argues that relevance labels should be model-specific, but it does not show that per-model labels improve re-ranking or retriever training, nor how they could be obtained at scale.
  • The source of cross-model variation is not identified. The closed-book filter only rules out answering the full question without context. However, a model may still know the bridge fact itself (in Figure 1, that "Nina" is based on Crime and Punishment), which would make the bridge dispensable for that model. This is consistent with need being model-specific, but it matters for interpreting the result and is not tested.
    • I am also not fully convinced that the labels are stable. A re-run for Llama-3.2-3B changes only 4 of 102 labels (as shown in Appendix), which is reassuring, but it is unclear whether the labels would hold if the passages were reordered or the question paraphrased.
    • Attribution only weakly tells apart the models that need a bridge from those that do not. Once differences in how often each model needs bridges are removed, the AUC drops to 0.558 [0.496,0.618][0.496, 0.618], an interval that includes random chance.
Questions for authors
  • For items where one model needs the bridge and another does not, does the second model know the bridge fact closed-book (e.g. when asked the first-hop question directly)? How much of the disagreement remains after excluding such cases?
  • How stable is a model's necessity label under passage reordering or light paraphrasing of the question?
  • How are results distributed across MuSiQue hop counts, and does necessity differ for questions whose intermediate hops are missing from the context?
  • I believe that expanding the context to 10-20 retrieved passages, with retrieved rather than benchmark distractors, would show whether model-specific dependence persists under more realistic noise, and would bring the findings of the paper closer to more general RAG pipelines.
4 comments

Response to Reviewer vYYZ (1/2)

Thank you for the careful review and for highlighting the value of the same-item deletion test. In response to your comments, we repeated the deletion test in the full HotpotQA and MuSiQue contexts, tested on MuSiQue whether models could answer the bridge step without seeing the supporting passage, and measured label stability under passage reordering and question paraphrasing. These experiments used a stricter answer scorer, which we describe in detail in our general response.

W1. Realism of the four-passage setting

The four-passage setting provides a controlled comparison between deleting the bridge and deleting a distractor from the same item. To test whether the finding holds in longer contexts, we repeated the paired test with all 10 HotpotQA and 20 MuSiQue benchmark paragraphs, including every annotated supporting paragraph, without the original passage-length cap. We added Appendix J to the revised manuscript, where we report the full-context results.

Full-context results. Across 1,313 eligible question–model pairs, bridge deletion broke 426 answers, compared with 84 for control deletion. The paired difference was significant for all 11 models when pooling the two datasets. Of the 212 questions eligible for multiple models, 120 had both necessity labels. On each of these questions, at least one model needed the bridge and at least one could dispense with it. The four-passage and full-context labels agreed on 79.9% of the pairs eligible in both settings.

Hop count and intermediate evidence. The table compares the two context settings by MuSiQue hop count.

MuSiQue hop countFour-passage pairsBridge vs. control breaksFull-context pairsBridge vs. control breaks
2 hops487215 vs. 46308152 vs. 21
3 hops19942 vs. 1714732 vs. 15
4 hops7119 vs. 9407 vs. 3

In the four-passage setting, bridge deletion broke 44.1%, 21.1%, and 26.8% of eligible answers on two-, three-, and four-hop MuSiQue items, respectively. The groups differ in both hop count and available supporting evidence, so these rates cannot isolate the effect of omitting a hop. On the 1,447 HotpotQA and two-hop MuSiQue pairs whose four-passage contexts already contain the complete annotated chain, bridge deletion broke 488 answers versus 96 for control deletion, with a significant paired difference in all 11 models.

W2. Practical implications for retrieval

We evaluated how well different labels and scores identify the annotated bridges a given generator needs. Gold supporting-passage annotations assign the same positive label to every bridge in this candidate set, yielding 0.500 AUC for ranking by necessity. The tested question–passage relevance scores reach 0.437–0.545. ContextCite reaches a pooled AUC of 0.732 and outperforms every tested relevance score within each of the 11 models. On the shared comparisons, precision for identifying necessary bridges is 0.295 for gold supporting labels and 0.520 when transferring necessity labels from another model.

These results motivate using generator-specific necessity labels as supervision for reranking. Given a reference answer, the deletion stage uses one full-context decode and one decode per deleted passage. On one L40S, the experiments took 2.5–17.8 seconds per item for four passages and 9.7–49.5 seconds for the full benchmark contexts. Measuring a downstream retrieval or QA gain would require evaluating a reranker trained with these labels.

Response to Reviewer vYYZ (2/2)

W3. Sources of cross-model variation, label stability, and attribution

Recall of the bridge fact. On MuSiQue, we tested whether models could answer the reasoning step supported by the bridge without seeing the passages, with earlier answers supplied. On questions where models differed in whether they needed the bridge, they answered the isolated step correctly more often when the bridge was dispensable (25.1%) than when it was necessary (13.7%). This is consistent with some models being able to recover the bridge fact without the passage. After excluding all records with a correct answer to the isolated step, pairwise disagreement remained substantial at 35.3%, compared with 33.4% before exclusion. In the main analysis, disagreement also persisted among models whose open-book accuracies were within five percentage points (27.8%, compared with 28.3% overall). Models can therefore differ in whether they need the same bridge even when both fail the isolated-step probe. This variation also appears among models with similar open-book accuracy.

Label stability. We agree that passage order and question wording can affect a model’s answer and, consequently, its bridge-necessity label. However, our new experiments still show that these labels are largely stable under the tested changes. For example, both alternative passage orders yielded over 90% label agreement with the original setting.

We reran the paired deletion test under two alternative passage orders and with paraphrased questions. Across the three checks, we made 3,950 matched label comparisons. For each check, we compared the original and modified labels on question–model pairs eligible in both settings.

Input changeQuestion–model pairs eligible in both settingsLabel agreementCohen’s κ
Second passage order1,47690.7%0.768
Third passage order1,48590.3%0.761
Question paraphrasing98983.5%0.611

When we restricted the analysis to labels that were stable across all three passage orders, 104 of 191 shared questions still had both necessity labels across models. This supports our finding that the same bridge can be necessary for one model and dispensable for another.

Attribution across models. Across 1,162 pairs of models with opposite necessity labels on the same item, ContextCite reaches 0.636 AUC, or 0.583 [0.527, 0.636] after centering scores within each model. A model’s necessity rate estimated from other items gives 0.762 AUC. These results indicate that ContextCite has modest ability to identify which model needs a particular bridge, and that a model’s overall tendency to need bridges is more informative for that comparison. ContextCite’s clearest contribution is ranking bridges for a given model, as shown in W2. The paired deletion test directly shows that different models can depend on the same bridge differently.

Q1. Does the model that can dispense with the bridge know the bridge fact?

Sometimes. The isolated-hop probe in W3, Recall of the bridge fact, shows that models can sometimes answer the intermediate step correctly without seeing the supporting passage. Substantial cross-model disagreement remains after excluding all records with successful isolated-hop recall.

Q2. How stable are the labels under passage reordering or question paraphrasing?

The labels are largely stable under both changes, with higher agreement under passage reordering. The new experiments in W3, Label stability, report agreement for each change and show that cross-model differences persist when the analysis is restricted to labels stable across all three passage orders.

Q3. How do the results vary by hop count and the availability of intermediate evidence?

In the four-passage setting, bridge necessity is lower on three- and four-hop MuSiQue items than on two-hop items. W1, Hop count and intermediate evidence, reports the breakdown in both context settings and shows that the bridge effect also holds when the four-passage context contains the complete annotated chain.

Q4. Does model-specific dependence persist with 10–20 retrieved passages?

Model-specific dependence persists in the full 10- and 20-passage benchmark contexts, as shown in W1, Full-context results. These experiments use benchmark-provided contexts, including HotpotQA’s TF-IDF-retrieved distractors. They address the context-length part of the question. An evaluation using newly retrieved contexts would also address cases where retrieval omits or duplicates the bridge.

Area_Chair_cqNuOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale. (I realize that there is only one business day between when the authors posted their response and when I sent this reminder -- please read it as just a friendly reminder and not an implication that I think you're being unresponsive).

Thank you for the clarifications provided in rebuttal. I am happy with the additional details. I would suggest adding the practical implications for retrieval from the rebuttal to the main body of the paper, and using the corresponding results to better motivate the work. In general, I remain in favour of accepting the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the paper’s generator-specific perspective on passage necessity in RAG (vYYZ), the paired deletion test with a non-supporting control from the same item (vYYZ, tPG7), the care taken in defining and evaluating the necessity labels (5Nvo), and the breadth of the empirical evaluation across model families and tasks (5Nvo, tPG7).

In response to the reviews, we ran additional experiments with stricter answer scoring, full benchmark contexts, alternative controls and bridge-selection rules, passage reordering, and question paraphrasing. These experiments support our original finding that the same answerless bridge can be necessary for one model and dispensable for another when both read the same question and context. We added the key robustness results to Appendix J at the end of the manuscript. They show that the finding persists when every annotated supporting passage is present and give readers a more detailed view of how the deletion effect varies with the bridge-selection rule. We summarize the experimental settings and results below.

Stricter scoring reproduces the main deletion result. The whole-token scorer requires the complete normalized reference answer or a MuSiQue alias to appear as a contiguous sequence of whole tokens. We also tested exact matching on the first response line. We applied each scorer to determine both sample eligibility and correctness after deletion.

Four-passage analysisOriginal scorerWhole-token scorerExact first-line scorer
Eligible question–model rows1,7291,7171,263
Bridge vs. control answer breaks556 vs. 126549 vs. 122433 vs. 93

Under whole-token scoring, the paired bridge–control difference is significant in all 11 models when the two datasets are pooled. Using the other distractor as the control gives 549 vs. 134 answer breaks, with a significant difference in all 11 models. Comparing the two distractors directly gives 122 vs. 134 breaks, with no significant difference in any model.

Model-specific necessity persists in full benchmark contexts. With all 10 HotpotQA or 20 MuSiQue paragraphs present, including every supporting paragraph, bridge deletion causes 426 answer breaks versus 84 for control deletion across 1,313 eligible rows. Of the 212 questions eligible for multiple models, 120 have both necessity labels. On MuSiQue three- and four-hop questions, deleting the first-hop or penultimate-step paragraph breaks 17.1% or 26.7% of answers, respectively, compared with 9.6% for the non-supporting control.

Cross-model disagreement persists under passage reordering and among comparable models. A model’s own label changes across passage orders in 9.1% of comparisons, while models disagree in 27.6% of comparisons on the same rows. Restricting the analysis to labels stable across all three orders still yields 22.3% cross-model disagreement. Disagreement is 26.1% among unquantized models from the same family and 27.8% among models whose open-book accuracies differ by at most five percentage points. In the isolated-hop probe on MuSiQue, models sometimes dispense with a bridge even when they fail to answer its reasoning step in isolation.

Attribution helps identify necessary bridges for a given model. With bf16 ablation passes, ContextCite fits all 11 models, including Qwen2.5-7B. It outperforms every tested question–passage relevance score in each model. The pooled AUC is 0.732 [0.687, 0.774], compared with 0.545 for the strongest tested relevance baseline.

The individual replies provide the hop-count breakdown, the random-control and paraphrase results, and the analyses of closed-book recall, quantization, and attribution across models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.