- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether models that answer multi-hop questions correctly actually depend on every intended reasoning step. It introduces the Behavioral Necessity Rate (BNR), the share of evidence deletions that stop a correct answer from being recovered. On five existing benchmarks, BNR is only 16.6–48.9%, so models often still answer after a step’s evidence is removed. The authors then propose REALHOP, which replaces entities with fictional names, splits some facts into several premises, and adds complete competing paths that lead to wrong answers. On 790 MuSiQue questions, BNR rises from 27.4% to 94.4% while Full accuracy stays at 68.5–83.5%. The method is also applied to FRAMES and LongBench v2, where the accuracy range across 16 models grows from 13.9 to 59.2 under the same multiple-choice format.
Critical strengths
- The audit is carefully controlled. Deleting background text of the same length barely changes accuracy, while deleting gold evidence does (Table 1, Appendix B), so the measured effect comes from the evidence itself rather than from removing text in general. Table 3 shows why the competing paths matter. On the 73-question subset, the deletion rate rises from 23.3% on the original questions to 57.9% after rewriting the evidence without competitors, then to 91.3% after adding both types of competing paths. This shows what the competing paths contribute beyond rewriting alone.
- The experiments are well checked, with confidence intervals, analysis of questions answered correctly in both versions, human validation of sampled items, and a component ablation. This makes the reported numbers reliable. The controls make the main result convincing. On rewritten MuSiQue questions, removing unrelated text of similar length usually preserves the correct answer, while removing a supporting paragraph rarely does (Table 5). The finding also holds when the comparison is restricted to questions the same model answered correctly in both versions (Table 6).
Critical weaknesses
- BNR does not clearly measure what the paper claims. On the original benchmarks, if a model still answers correctly after evidence is removed, the paper counts this as skipping the intended reasoning. But in many of these cases the model is doing nothing wrong: it may already know the answer (the model's own knowledge), use another correct path in the context, or pick the only matching entity in the corpus. The last two are problems with how the datasets were labeled, not reasoning failures, and the paper does not separate these cases, for example by testing the models without any context. On the new data, the high BNR is partly guaranteed by how it is built. The answers are new made-up names that appear only in the inserted evidence, so removing the final-hop passage removes the answer itself. Replacing entities and rewriting the evidence, without adding competing paths, already raises BNR from 23.3% to 57.9%. Since BNR is the main evidence in the paper, this weakens its main claim.
- The paper does not show that the new benchmark tests multi-hop reasoning better, rather than a different skill. Made-up entities remove real-world knowledge, and the competing paths mainly test whether models can avoid distractors. On LongBench, the main evidence is a wider accuracy range and large changes in model ranking, but a wider range does not mean the new ranking is more correct. This version also starts from a made-up graph, so it is a new synthetic task rather than an improved version of LongBench v2.
Questions for authors
- Could the authors use closed-book pass@k to check how often models already know the answers? For answers that survive evidence removal, how many rely on existing knowledge, alternative valid paths, or a single matching entity?
- Could the authors report BNR separately for final and non-final hops, since removing the final hop may also remove the only mention of the answer?
- What evidence shows that the new LongBench ranking better reflects multi-hop reasoning rather than sensitivity to fictional names or distractors? Does it agree with other reasoning benchmarks?
4 comments
Thank you for the detailed response. The additional experiments address most of my concerns, particularly regarding answer deletion and benchmark validity. I still have some concerns about whether the constructed tasks better measure general multi-hop reasoning ability. I appreciate the clarification and have raised my score to 2.
Thank you for clarifying your remaining concern and for raising your score. We agree that general multi-hop reasoning includes answering questions that arise naturally and drawing on real-world knowledge. RealHop addresses a narrower question. When a model answers correctly, which annotated hops does that answer depend on? Final-answer accuracy alone leaves this open, so we remove the evidence for each hop in turn. We will clarify in §8 how this controlled test complements natural multi-hop QA benchmarks.
To see how models distinguish competing answers, we examined the full-context reasoning traces generated by DeepSeek V4 Flash and Qwen 3.7 Max. Claude Opus 5.5 annotated the steps described in each trace. Paired values below are in DeepSeek / Qwen order.
1. Correct traces mostly match a single candidate on source MuSiQue and check relations among candidates on RealHop-MuSiQue. For both datasets, the annotator sees the same types of input (the question, its annotated chain, and the trace, without the passages) and labels how the trace obtained each hop's answer: by relation checking among candidates (it weighs two or more candidates and chooses between them by checking a relation stated in the text), by single-candidate matching (it picks the one entity that fits, without considering alternatives), from memory, or not at all (skipped). Both samples retain questions answered correctly under Full on the initial run and rerun with an available trace. On the source side we randomly sampled 2-hop questions and reran all eligible 3- and 4-hop questions. On RealHop we reran all eligible questions in the stratified 200-question sample.
| Trace label | Source MuSiQue | RealHop-MuSiQue |
|---|---|---|
| Questions analyzed | 387 / 392 | 141 / 119 |
| Hops analyzed | 1,150 / 1,169 | 352 / 291 |
| Hops resolved by relation checking among candidates | 14% / 18% | 80% / 70% |
| Hops resolved by single-candidate matching | 74% / 75% | 20% / 28% |
| Hops filled from memory | 10% / 6% | 0% / 0% |
| Hops skipped | 2% / 1% | 0% / 1% |
| Questions with relation checking at every hop | 1% / 0% | 62% / 43% |
The source sample contains a higher proportion of long chains. Comparing 2-, 3-, and 4-hop questions separately gives higher relation-checking rates on RealHop at every length for both models.
2. On longer source chains, a larger share of hops is filled from memory or skipped. For a closer check across chain lengths on the same source traces, the annotator is additionally given the source passages and labels each hop as read from a passage, filled from memory, or bypassed. A longer chain has more hops that could be bypassed, so fewer answers would have every hop established from the passages even at a constant per-hop rate. As a reference, we compute the expected share with every hop drawn from the passages if hops were independent and the per-hop passage-use rate stayed at the 2-hop level.
| Source chain length | 2 hops | 3 hops | 4 hops |
|---|---|---|---|
| Questions analyzed | 97 / 98 | 204 / 203 | 86 / 91 |
| Hops filled from memory or bypassed | 10% / 4% | 15% / 8% | 18% / 10% |
| Correct answers with every hop from the passages | 82% / 92% | 63% / 77% | 43% / 62% |
| Expected at the 2-hop per-hop rate | – | 72% / 88% | 65% / 85% |
A hop counts as bypassed if the trace never establishes it, for example when shared words lead directly to the final passage. A single-candidate match still counts as passage-based when the trace reads that hop from a passage.
Among correct source answers, the share of hops filled from memory or skipped rises from 10% to 18% for DeepSeek and from 4% to 10% for Qwen. Comparing the per-question fractions for 2-hop and 4-hop items gives a one-sided Mann–Whitney p < 10⁻⁴ for each model.
3. Correct RealHop traces usually check the condition that distinguishes the gold answer from a competitor. Among the correctly answered RealHop-MuSiQue questions in our 200-question sample (correct on both runs, n = 141 / 119), 91% / 92% of traces explicitly check the requirement that separates the gold entity from a similar competitor. On RealHop-LongBench we reran all 50 sampled questions. We saw this check in 12 of 14 correct DeepSeek traces and 8 of 11 correct Qwen traces. In the MuSiQue error group (wrong on both runs, with an available trace), 19/26 and 42/49 traces accepted a candidate matching only part of the relation (73% / 86%). Among valid LongBench reruns judged wrong, the corresponding counts were 22/29 and 29/37 (76% / 78%).
4. RealHop-LongBench rankings agree more closely with the external composite under a matched multiple-choice format. A wider spread of scores alone cannot tell us whether the ranking agrees with independent evaluations. With both LongBench tasks scored in multiple-choice format, Spearman’s correlation with the ten-benchmark composite is 0.54 for source LongBench v2 and 0.83 for RealHop. With open-ended RealHop Avg@3, it is 0.88. The composite averages each model’s available within-benchmark percentile ranks and includes the 15 models covered by at least six benchmarks. The table uses public Epoch AI and ARC Prize scores. The Epoch Capabilities Index is a separate reference.
| External benchmark | n models | Source LongBench v2 (MCQ) | RealHop-LongBench (open-ended Avg@3) |
|---|---|---|---|
| Composite (mean percentile) | 15 | 0.54 | 0.88 |
| Composite (mean z-score) | 15 | 0.62 | 0.88 |
| Epoch Capabilities Index (reference) | 15 | 0.62 | 0.91 |
| ARC-AGI-1 | 11 | 0.63 | 0.83 |
| ARC-AGI-2 | 10 | 0.57 | 0.66 |
| FrontierMath Tiers 1–3 | 15 | 0.35 | 0.78 |
| FrontierMath Tier 4 | 15 | 0.17 | 0.60 |
| OTIS Mock AIME | 15 | 0.29 | 0.56 |
| CritPt | 15 | 0.22 | 0.73 |
| ProofBench | 16 | 0.36 | 0.91 |
| Chess Puzzles | 15 | 0.57 | 0.67 |
| Mystery Game Puzzles | 14 | 0.36 | 0.62 |
| GPQA Diamond | 15 | 0.80 | 0.60 |
Leaving out one benchmark at a time retains the same 15 models. The composite correlation ranges from 0.44 to 0.57 for source MCQ and from 0.84 to 0.90 for RealHop Avg@3. Nine of the ten individual benchmarks correlate more strongly with RealHop Avg@3. GPQA Diamond favors the source ranking (0.80 vs. 0.60). These results describe agreement on this model panel. Some public scores use different reasoning-effort settings from ours.
The trace analysis helps explain what the competing paths change. In the correct RealHop-MuSiQue traces, the models more often compare alternatives against a stated relation, even at the same chain length. The Isolated control in our earlier response gives a behavioral check on that interpretation. With comparable distractor content, connecting the competing facts into paths increases dependence on the annotated evidence. These findings support RealHop as a diagnostic of dependence on the annotated evidence. The trace analysis covers two models and uses LLM annotations of generated text, which gives only indirect evidence about internal computation. We will include the trace analysis in Appendix L.3 and the external comparisons in Appendix M.
We thank the reviewer for pushing us to sharpen the paper's central claim. REALHOP addresses a specific gap in multi-hop QA evaluation: a model can answer a question correctly without depending on all the evidence that the benchmark labels as necessary. Our claim is not that every such answer reflects faulty reasoning. Rather, REALHOP makes the annotated dependencies more consequential for answer recovery and measures their behavioral necessity through targeted evidence deletion. Three results establish the distinction: the original questions often admit alternative answer sources; REALHOP makes even non-final hops necessary; and the resulting long-context evaluation reveals meaningful model differences.
W1/Q1. Why does the original benchmark have low evidence necessity?
The original questions frequently allow correct answers without the targeted support, while entity rebinding substantially reduces closed-book answer recovery. Across the same 790 MuSiQue questions, closed-book pass@3 falls from 51.6–58.9% on the originals to 0.8–3.5% on REALHOP.
We also examined 1,518 original deletion-surviving units using an LLM annotator (Claude Opus 5.5). The remaining-context analysis found:
In a separate rerun analysis on a stratified 200-question sample, the annotator classified 70.6% and 74.1% of the successfully reproduced DeepSeek and Qwen responses, respectively, as consistent with parametric-knowledge use, based on their generated traces.
These are different ways in which final-answer correctness can fail to establish dependence on the annotated chain. That is the evaluation gap BNR is designed to expose.
W2/Q2. Is REALHOP's high BNR just an answer-deletion artifact?
The substantial increase persists under non-final-hop deletion, even though the gold answer string remains in the context in 95.3% of REALHOP cases. On all 790 paired questions, non-final-hop BNR increases from 8.6% to 91.8% (overall: 27.4% to 94.4%; final-hop: 55.8% to 98.0%). This directly addresses the concern that high BNR arises only because the final answer disappears.
The construction ablation also separates answer rebinding from competing-path structure. On 73 questions, Gold-only gives 57.9% overall BNR and 33.0% non-final-hop BNR; with complete competing paths these rise to 91.3% and 87.3%. Rebinding helps prevent direct recall, while connected alternatives increase the importance of the intermediate supports. The gain is therefore not simply a consequence of hiding an unfamiliar final entity.
W3/Q3. What is gained in long-context evaluation?
REALHOP-LongBench provides greater model discrimination, and that discrimination agrees more closely with independent reasoning evaluations. We embed controlled multi-hop evidence chains in LongBench v2 documents, retaining the long-document setting while introducing auditable evidence dependencies. The wider score spread reveals differences that may be compressed in the original evaluation. Across 15 models, the Spearman correlation with a mean-percentile composite of ten external reasoning benchmarks is 0.88 for REALHOP-LongBench Avg@3, versus 0.54 for original LongBench v2 (REALHOP MCQ: 0.83).
The evidence-necessity improvement is also not just an effect of adding distractor text. With comparable distractor content but disconnected competing facts (Isolated), we obtain:
Unlike isolated distractors, complete competing paths introduce alternative connected chains of evidence. The Isolated control shows that this connectivity contributes to evidence necessity beyond distractor volume alone.
Taken together, these results support the paper's central contribution: REALHOP makes benchmark correctness more informative by pairing it with direct tests of dependence on annotated multi-hop evidence. Our paired MuSiQue evaluation preserves the original annotated dependencies across 790 questions and raises BNR from 27.4% to 94.4%; in long contexts, REALHOP yields stronger model discrimination with higher external ranking agreement.