acceptodds
Under review as a conference paper at ICLR 2027

RealHop: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing

16 views

Abstract

Complex questions often require multi-hop reasoning that connects facts distributed across sources or distant regions of a long context through intermediate steps. Benchmarks commonly evaluate this ability with questions built around predefined reasoning chains, treating a correct answer as evidence that the intended composition was used. Yet answer correctness alone leaves open whether success depends on the evidence associated with each intended step: models may instead rely on memorized associations, shorter paths, or partial evidence. We examine this dependence using the Behavioral Necessity Rate (BNR), which measures how often targeted evidence removal prevents answer recovery on initially correct instances. Across five existing benchmarks, panel-mean BNR is only 16.6–48.9%, exposing a substantial gap between annotated structure and observed dependence. Guided by this diagnosis, we introduce RealHop, a diagnose–construct–verify framework that rebinds entities, factorizes selected relations, adds complete competing paths, and places evidence at traceable locations. Structural and semantic checks precede freezing; behavioral interventions follow. On 790 paired MuSiQue questions, RealHop raises panel-mean BNR from 27.4% to 94.4% while retaining high Full accuracy. It also yields high BNR on RealHop-FRAMES and RealHop-LongBench. On 216 long-context questions, the matched multiple-choice spread across 16 models grows from 13.9 to 59.2 points and persists under repeated open-ended evaluation. Together, these results show that a conceptually coherent chain and a correct final answer do not by themselves establish multi-hop reasoning. Verifying that success depends on every intended hop is therefore as fundamental to multi-hop evaluation as measuring answer accuracy itself.

Primary area:
datasets and benchmarks
Keywords:
Multi-hop question answering, long-context reasoning, benchmark construction, evidence necessity, reasoning shortcuts, intervention-based evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HgcZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether models that answer multi-hop questions correctly actually depend on every intended reasoning step. It introduces the Behavioral Necessity Rate (BNR), the share of evidence deletions that stop a correct answer from being recovered. On five existing benchmarks, BNR is only 16.6–48.9%, so models often still answer after a step’s evidence is removed. The authors then propose REALHOP, which replaces entities with fictional names, splits some facts into several premises, and adds complete competing paths that lead to wrong answers. On 790 MuSiQue questions, BNR rises from 27.4% to 94.4% while Full accuracy stays at 68.5–83.5%. The method is also applied to FRAMES and LongBench v2, where the accuracy range across 16 models grows from 13.9 to 59.2 under the same multiple-choice format.

Critical strengths
  • The audit is carefully controlled. Deleting background text of the same length barely changes accuracy, while deleting gold evidence does (Table 1, Appendix B), so the measured effect comes from the evidence itself rather than from removing text in general. Table 3 shows why the competing paths matter. On the 73-question subset, the deletion rate rises from 23.3% on the original questions to 57.9% after rewriting the evidence without competitors, then to 91.3% after adding both types of competing paths. This shows what the competing paths contribute beyond rewriting alone.
  • The experiments are well checked, with confidence intervals, analysis of questions answered correctly in both versions, human validation of sampled items, and a component ablation. This makes the reported numbers reliable. The controls make the main result convincing. On rewritten MuSiQue questions, removing unrelated text of similar length usually preserves the correct answer, while removing a supporting paragraph rarely does (Table 5). The finding also holds when the comparison is restricted to questions the same model answered correctly in both versions (Table 6).
Critical weaknesses
  • BNR does not clearly measure what the paper claims. On the original benchmarks, if a model still answers correctly after evidence is removed, the paper counts this as skipping the intended reasoning. But in many of these cases the model is doing nothing wrong: it may already know the answer (the model's own knowledge), use another correct path in the context, or pick the only matching entity in the corpus. The last two are problems with how the datasets were labeled, not reasoning failures, and the paper does not separate these cases, for example by testing the models without any context. On the new data, the high BNR is partly guaranteed by how it is built. The answers are new made-up names that appear only in the inserted evidence, so removing the final-hop passage removes the answer itself. Replacing entities and rewriting the evidence, without adding competing paths, already raises BNR from 23.3% to 57.9%. Since BNR is the main evidence in the paper, this weakens its main claim.
  • The paper does not show that the new benchmark tests multi-hop reasoning better, rather than a different skill. Made-up entities remove real-world knowledge, and the competing paths mainly test whether models can avoid distractors. On LongBench, the main evidence is a wider accuracy range and large changes in model ranking, but a wider range does not mean the new ranking is more correct. This version also starts from a made-up graph, so it is a new synthetic task rather than an improved version of LongBench v2.
Questions for authors
  • Could the authors use closed-book pass@k to check how often models already know the answers? For answers that survive evidence removal, how many rely on existing knowledge, alternative valid paths, or a single matching entity?
  • Could the authors report BNR separately for final and non-final hops, since removing the final hop may also remove the only mention of the answer?
  • What evidence shows that the new LongBench ranking better reflects multi-hop reasoning rather than sensitivity to fictional names or distractors? Does it agree with other reasoning benchmarks?
4 comments

We thank the reviewer for pushing us to sharpen the paper's central claim. REALHOP addresses a specific gap in multi-hop QA evaluation: a model can answer a question correctly without depending on all the evidence that the benchmark labels as necessary. Our claim is not that every such answer reflects faulty reasoning. Rather, REALHOP makes the annotated dependencies more consequential for answer recovery and measures their behavioral necessity through targeted evidence deletion. Three results establish the distinction: the original questions often admit alternative answer sources; REALHOP makes even non-final hops necessary; and the resulting long-context evaluation reveals meaningful model differences.

W1/Q1. Why does the original benchmark have low evidence necessity?

The original questions frequently allow correct answers without the targeted support, while entity rebinding substantially reduces closed-book answer recovery. Across the same 790 MuSiQue questions, closed-book pass@3 falls from 51.6–58.9% on the originals to 0.8–3.5% on REALHOP.

We also examined 1,518 original deletion-surviving units using an LLM annotator (Claude Opus 5.5). The remaining-context analysis found:

  • 12–13%: Redundant supporting evidence remained.
  • 49–52%: A uniquely identifiable answer candidate remained.
  • 35–39%: The remaining context was insufficient to derive the answer.

In a separate rerun analysis on a stratified 200-question sample, the annotator classified 70.6% and 74.1% of the successfully reproduced DeepSeek and Qwen responses, respectively, as consistent with parametric-knowledge use, based on their generated traces.

These are different ways in which final-answer correctness can fail to establish dependence on the annotated chain. That is the evaluation gap BNR is designed to expose.

W2/Q2. Is REALHOP's high BNR just an answer-deletion artifact?

The substantial increase persists under non-final-hop deletion, even though the gold answer string remains in the context in 95.3% of REALHOP cases. On all 790 paired questions, non-final-hop BNR increases from 8.6% to 91.8% (overall: 27.4% to 94.4%; final-hop: 55.8% to 98.0%). This directly addresses the concern that high BNR arises only because the final answer disappears.

The construction ablation also separates answer rebinding from competing-path structure. On 73 questions, Gold-only gives 57.9% overall BNR and 33.0% non-final-hop BNR; with complete competing paths these rise to 91.3% and 87.3%. Rebinding helps prevent direct recall, while connected alternatives increase the importance of the intermediate supports. The gain is therefore not simply a consequence of hiding an unfamiliar final entity.

W3/Q3. What is gained in long-context evaluation?

REALHOP-LongBench provides greater model discrimination, and that discrimination agrees more closely with independent reasoning evaluations. We embed controlled multi-hop evidence chains in LongBench v2 documents, retaining the long-document setting while introducing auditable evidence dependencies. The wider score spread reveals differences that may be compressed in the original evaluation. Across 15 models, the Spearman correlation with a mean-percentile composite of ten external reasoning benchmarks is 0.88 for REALHOP-LongBench Avg@3, versus 0.54 for original LongBench v2 (REALHOP MCQ: 0.83).

The evidence-necessity improvement is also not just an effect of adding distractor text. With comparable distractor content but disconnected competing facts (Isolated), we obtain:

  • MuSiQue (200 questions): Isolated BNR 71.2% vs. Full BNR 94.8%.
  • REALHOP-LongBench (50 questions): Isolated BNR 26.8% vs. Full BNR 96.6%.

Unlike isolated distractors, complete competing paths introduce alternative connected chains of evidence. The Isolated control shows that this connectivity contributes to evidence necessity beyond distractor volume alone.

Taken together, these results support the paper's central contribution: REALHOP makes benchmark correctness more informative by pairing it with direct tests of dependence on annotated multi-hop evidence. Our paired MuSiQue evaluation preserves the original annotated dependencies across 790 questions and raises BNR from 27.4% to 94.4%; in long contexts, REALHOP yields stronger model discrimination with higher external ranking agreement.

Thank you for the detailed response. The additional experiments address most of my concerns, particularly regarding answer deletion and benchmark validity. I still have some concerns about whether the constructed tasks better measure general multi-hop reasoning ability. I appreciate the clarification and have raised my score to 2.

Thank you for clarifying your remaining concern and for raising your score. We agree that general multi-hop reasoning includes answering questions that arise naturally and drawing on real-world knowledge. RealHop addresses a narrower question. When a model answers correctly, which annotated hops does that answer depend on? Final-answer accuracy alone leaves this open, so we remove the evidence for each hop in turn. We will clarify in §8 how this controlled test complements natural multi-hop QA benchmarks.

To see how models distinguish competing answers, we examined the full-context reasoning traces generated by DeepSeek V4 Flash and Qwen 3.7 Max. Claude Opus 5.5 annotated the steps described in each trace. Paired values below are in DeepSeek / Qwen order.

1. Correct traces mostly match a single candidate on source MuSiQue and check relations among candidates on RealHop-MuSiQue. For both datasets, the annotator sees the same types of input (the question, its annotated chain, and the trace, without the passages) and labels how the trace obtained each hop's answer: by relation checking among candidates (it weighs two or more candidates and chooses between them by checking a relation stated in the text), by single-candidate matching (it picks the one entity that fits, without considering alternatives), from memory, or not at all (skipped). Both samples retain questions answered correctly under Full on the initial run and rerun with an available trace. On the source side we randomly sampled 2-hop questions and reran all eligible 3- and 4-hop questions. On RealHop we reran all eligible questions in the stratified 200-question sample.

Trace labelSource MuSiQueRealHop-MuSiQue
Questions analyzed387 / 392141 / 119
Hops analyzed1,150 / 1,169352 / 291
Hops resolved by relation checking among candidates14% / 18%80% / 70%
Hops resolved by single-candidate matching74% / 75%20% / 28%
Hops filled from memory10% / 6%0% / 0%
Hops skipped2% / 1%0% / 1%
Questions with relation checking at every hop1% / 0%62% / 43%

The source sample contains a higher proportion of long chains. Comparing 2-, 3-, and 4-hop questions separately gives higher relation-checking rates on RealHop at every length for both models.

2. On longer source chains, a larger share of hops is filled from memory or skipped. For a closer check across chain lengths on the same source traces, the annotator is additionally given the source passages and labels each hop as read from a passage, filled from memory, or bypassed. A longer chain has more hops that could be bypassed, so fewer answers would have every hop established from the passages even at a constant per-hop rate. As a reference, we compute the expected share with every hop drawn from the passages if hops were independent and the per-hop passage-use rate stayed at the 2-hop level.

Source chain length2 hops3 hops4 hops
Questions analyzed97 / 98204 / 20386 / 91
Hops filled from memory or bypassed10% / 4%15% / 8%18% / 10%
Correct answers with every hop from the passages82% / 92%63% / 77%43% / 62%
Expected at the 2-hop per-hop rate–72% / 88%65% / 85%

A hop counts as bypassed if the trace never establishes it, for example when shared words lead directly to the final passage. A single-candidate match still counts as passage-based when the trace reads that hop from a passage.

Among correct source answers, the share of hops filled from memory or skipped rises from 10% to 18% for DeepSeek and from 4% to 10% for Qwen. Comparing the per-question fractions for 2-hop and 4-hop items gives a one-sided Mann–Whitney p < 10⁻⁴ for each model.

3. Correct RealHop traces usually check the condition that distinguishes the gold answer from a competitor. Among the correctly answered RealHop-MuSiQue questions in our 200-question sample (correct on both runs, n = 141 / 119), 91% / 92% of traces explicitly check the requirement that separates the gold entity from a similar competitor. On RealHop-LongBench we reran all 50 sampled questions. We saw this check in 12 of 14 correct DeepSeek traces and 8 of 11 correct Qwen traces. In the MuSiQue error group (wrong on both runs, with an available trace), 19/26 and 42/49 traces accepted a candidate matching only part of the relation (73% / 86%). Among valid LongBench reruns judged wrong, the corresponding counts were 22/29 and 29/37 (76% / 78%).

4. RealHop-LongBench rankings agree more closely with the external composite under a matched multiple-choice format. A wider spread of scores alone cannot tell us whether the ranking agrees with independent evaluations. With both LongBench tasks scored in multiple-choice format, Spearman’s correlation with the ten-benchmark composite is 0.54 for source LongBench v2 and 0.83 for RealHop. With open-ended RealHop Avg@3, it is 0.88. The composite averages each model’s available within-benchmark percentile ranks and includes the 15 models covered by at least six benchmarks. The table uses public Epoch AI and ARC Prize scores. The Epoch Capabilities Index is a separate reference.

External benchmarkn modelsSource LongBench v2 (MCQ)RealHop-LongBench (open-ended Avg@3)
Composite (mean percentile)150.540.88
Composite (mean z-score)150.620.88
Epoch Capabilities Index (reference)150.620.91
ARC-AGI-1110.630.83
ARC-AGI-2100.570.66
FrontierMath Tiers 1–3150.350.78
FrontierMath Tier 4150.170.60
OTIS Mock AIME150.290.56
CritPt150.220.73
ProofBench160.360.91
Chess Puzzles150.570.67
Mystery Game Puzzles140.360.62
GPQA Diamond150.800.60

Leaving out one benchmark at a time retains the same 15 models. The composite correlation ranges from 0.44 to 0.57 for source MCQ and from 0.84 to 0.90 for RealHop Avg@3. Nine of the ten individual benchmarks correlate more strongly with RealHop Avg@3. GPQA Diamond favors the source ranking (0.80 vs. 0.60). These results describe agreement on this model panel. Some public scores use different reasoning-effort settings from ours.

The trace analysis helps explain what the competing paths change. In the correct RealHop-MuSiQue traces, the models more often compare alternatives against a stated relation, even at the same chain length. The Isolated control in our earlier response gives a behavioral check on that interpretation. With comparable distractor content, connecting the competing facts into paths increases dependence on the annotated evidence. These findings support RealHop as a diagnostic of dependence on the annotated evidence. The trace analysis covers two models and uses LLM annotations of generated text, which gives only indirect evidence about internal computation. We will include the trace analysis in Appendix L.3 and the external comparisons in Appendix M.

Reviewer_jmw1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether correct answers on multi-hop QA benchmarks actually depend on the annotated supporting evidence. It introduces the Behavioral Necessity Rate (BNR), the rate at which deleting a declared support prevents a correct answer, and RealHop, a pipeline that rebinds entities, adds competing reasoning paths and makes evidence traceable. Across five existing benchmarks, the authors find substantial gaps between annotated reasoning chains and observed evidence dependence. On 790 paired MuSiQue questions, RealHop increases panel-mean BNR from 27.4% to 94.4%, with additional evaluations on FRAMES and LongBench v2 demonstrating stronger evidence dependence, and the LongBench v2 comparison showing wider performance separation across models.

Critical strengths
  • The paper goes beyond answer correctness to examine dependence on individual reasoning supports. On items answered correctly both before and after construction, BNR gains remain 66.6–69.7 points, so the effect is not just a change in which items are answered correctly. Background deletion barely affects accuracy, unlike targeted deletion.

  • RealHop combines entity rebinding, premise decomposition, and competing paths in a traceable construction pipeline. The component ablations demonstrate the importance of structured competitors, while human checks support construction quality. Evaluation across three settings, including repeated trials on 16 models, provides useful evidence of practical applicability beyond a single benchmark.

  • Construction and behavioral checking are well integrated. In the ablation, BNR rises from 57.9% (gold only) to 91.3% (both competitor types). Explicit construction rules, traceable evidence, freezing before evaluation and human checks make the resource easy to audit.

Critical weaknesses

CofCA [1] already investigates inflated multi-hop performance through counterfactual evidence and step-wise reasoning evaluation, but is not discussed. In addition, CRiT-QA already combines fictionalized MuSiQue chains with competing paths, and Plausible Distractors builds near-miss paths. RealHop contributes a distinct evidence-deletion audit and structured competing paths; however, their incremental value relative to such approaches is not sufficiently established. The existing ablations also change competitor structure and added context simultaneously, without a matched simpler-distractor comparison that isolates the benefit of complete competing paths.

[1] Jian Wu, et al. CofCA: A Step-wise Counterfactual Multi-hop QA Benchmark. ICLR, 2025.

Questions for authors
  • How does RealHop differ empirically from existing step-wise counterfactual evaluations such as CofCA? In addition, what are the main contributions of RealHop relative to its closest prior work CRiT-QA?

  • In Section 3.1, the authors define an instance x as (q, C, a*, G). What are the definitions of q, C, a*, and G?

3 comments

Thank you for pointing out the missing CofCA discussion and for suggesting a matched distractor control.

W1/Q1. Relation to CofCA, CRiT-QA, and Plausible Distractors

CofCA, CRiT-QA, and Plausible Distractors are important precedents for counterfactual evidence and alternative paths, and we will discuss them more directly. The key methodological distinction lies in what is evaluated. CofCA examines answers to intermediate subquestions under counterfactual evidence; REALHOP holds the final question fixed and intervenes on individual annotated supports to measure whether the correct final answer depends on each of them. Our contribution is not the isolated use of fictional entities or competing paths, but the combination of traceable construction and hop-level behavioral necessity auditing.

We also evaluated CofCA using publicly available data from GitHub, as the original paper's data link is no longer accessible. We applied the same HotpotQA auditing protocol used in Sec. 3 of our paper, including support-sentence deletion, deterministic short-answer scoring, and the same three models. CofCA obtains BNRs of 57.1% (DeepSeek), 42.4% (Gemini), and 40.2% (Qwen), averaging 46.6%, with mean Full accuracy of 81.1%. For comparison, our audits report 43.5% for FalseCoTQA, 48.9% for Plausible Distractors, and 94.4% for REALHOP-MuSiQue. We will include these results and the evaluation protocol in the revision.

Our within-framework ablation separately examines the construction: on the same 73 questions, Gold-only gives 57.9% overall and 33.0% non-final-hop BNR, compared with 91.3% and 87.3% using complete competing paths. Together with the CofCA audit, this supports the value of combining traceable construction with hop-targeted behavioral necessity testing.

W2. Matched simpler-distractor control

We introduced Isolated, which keeps the competitor statements, relation types, distractor answer entities, positions, and approximately the same context length as Full, while replacing the connecting entities so the statements no longer form complete alternative paths.

For MuSiQue, we sampled 200 eligible questions by hop count (110 two-hop, 61 three-hop, and 29 four-hop) outside the earlier ablation sample. We also evaluated 50 REALHOP-LongBench questions. Results are:

DatasetConditionFull accuracyOverall BNRNon-final BNR
MuSiQueGold-only91.2%61.7%38.5%
MuSiQueIsolated94.6%71.2%54.6%
MuSiQueFull76.7%94.8%91.0%
LongBenchGold-only90.5%14.7%3.3%
LongBenchIsolated80.3%26.8%15.3%
LongBenchFull30.6%96.6%95.8%

Because Full and Isolated differ in which questions are answered correctly, we also compare the MuSiQue questions answered correctly in both conditions. Full retains a 23.3-point advantage in overall BNR (95% CI [19.9, 26.8]) and a 36.6-point advantage on non-final hops ([31.1, 42.2]). The results show a contribution from complete path connectivity beyond extra distractor text or changes in the Full-correct subset.

Q2. Notation

We will define x = (q, C, a*, G) in Section 3.1, where q is the question, C is the provided context, a* is the gold answer (including accepted aliases), and G is the annotated evidence graph recording reasoning steps, dependencies, and supporting evidence.

Thanks for adding the CofCA comparison and the disconnected-path control. The control addresses my concern that the extra distractor text alone might explain the result. I'll keep my positive score.

Thank you for following up and for your positive assessment of our work. We really appreciate your thoughtful comments and suggestions!

Reviewer_xCFeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates the extent to which the accuracy of multi-hop question answering reflects a genuine reliance on supporting evidence. The paper proposes the Behavioral Necessity Rate (BNR), the proportion of questions that become incorrectly answered when specific supporting evidence is removed and reconstruct existing datasets by altering entity names and introducing competing reasoning paths. Through experiments based on MuSiQue, FRAMES, and LongBench, the authors report on BNR values ​​and performance disparities across models, further validating their findings through control experiments involving the removal of background information and component-wise comparisons.

Critical strengths
  1. The controlled experiments support the core findings robustly. On the 790 corresponding questions from MuSiQue, the average BNR increases from 27.4% to 94.4%. This improvement persists even when the analysis is restricted to the same set of questions for which both versions yielded the correct answer. Furthermore, since the correct answers largely remain unchanged after removing background information, the observed effects cannot be attributed merely to differences in the evaluation questions or the general removal of text. This supports the claim that the proposed construction enhances reliance on supporting evidence.

  2. The effectiveness of the proposed data construction method has been empirically verified. Adding "competing paths" to the "Gold-only" baseline increases the average BNR from 57.9% to 91.3%. Notably, the selection of questions did not rely on the outcomes from the "Full/Drop" experiments, and the final dataset underwent human review. These measures minimize the possibility that questions were cherry-picked to favor specific results, thereby clearly demonstrating the role of competing paths.

Critical weaknesses
  1. Insufficient verification of the necessity of individual hops: In MuSiQue, entire paragraphs, rather than just the specific supporting sentences are removed; consequently, it is unclear whether any observed performance degradation is truly attributable to the removal of a specific hop. Furthermore, since the BNR metric represents an average value across hops, it is necessary to examine the proportion of questions for which every single hop is genuinely essential.

  2. Lack of reliability verification for BNR in long-context settings: BNR is calculated based on the results of a single execution, and the reproducibility of the correct answers for any given question has not been verified. Even if the overall accuracy remains similar, the outcomes for individual questions may vary, making it difficult to determine whether a decline in performance is actually due to the removal of the supporting evidence.

Questions for authors
  1. Verification of the intervention method: Does the increase in BNR persist even in a smaller-scale comparison where only the "gold span" of the target hop is removed, while other hops and competing paths are preserved?

  2. Verification of results from repeated runs: For questions answered correctly in the first full-context run on LongBench, what is the success rate (i.e., the proportion of questions still answered correctly) in the second and third full-context runs? Comparing this to the retention rate of correct answers following the removal of specific spans would provide greater confidence in the results concerning long-context performance.

3 comments

Thank you for the specific suggestions. We tested finer-grained deletion and used the existing repeated LongBench trials to examine the effect of prediction variability.

W1/Q1. Minimal evidence deletion and all-hop necessity

For this analysis we used a shared, stratified sample of 200 MuSiQue questions: 110 two-hop, 61 three-hop, and 29 four-hop questions, randomly selected from 708 eligible instances outside the earlier ablation sample. The same questions and target hops were used for original MuSiQue and REALHOP, and for both deletion granularities. We compared whole-paragraph deletion with removing only the minimal gold support. In REALHOP, the latter removes the recorded gold span while keeping surrounding text and competing statements. For original MuSiQue, we identified minimal supporting sentences using Claude Opus 5.5, followed by human review of the annotations.

The results are summarized below:

DatasetParagraph deletion BNRMinimal-support deletion BNR
Original MuSiQue27.6%22.6%
REALHOP95.8%95.3%
REALHOP − Original (pp)+68.2+72.7

The REALHOP–original gap remains 72.7 percentage points with minimal-support deletion, so the effect is not due to removing the surrounding paragraph text.

We also counted questions for which every annotated hop is necessary under the individual deletion tests, rather than averaging over hops. Among Full-correct questions with valid interventions, this fraction is 4.1–5.6% on original MuSiQue and 87.1–89.4% on REALHOP across the three models. We will report results by hop count in the revision.

W2/Q2. Repeated-run reliability

We share the concern that a single Full/Drop pair can be affected by run-to-run variation. Using the three existing open-ended REALHOP-LongBench trials, we selected questions answered correctly in Trial-1 Full. Their mean correctness retention in Full trials 2 and 3 is 46.4% for DeepSeek, 64.6% for Gemini, and 44.8% for Qwen.

With length- and position-matched background deletion, retention is 54.3%, 61.4%, and 49.1%, respectively. With targeted gold-evidence deletion, it is only 2.4%, 1.9%, and 3.5%. Individual predictions are therefore variable, but the loss following targeted deletion is far larger than either ordinary rerunning or matched background deletion. Among questions correct in all three Full trials, targeted-deletion BNR remains 91–98%.

We will report these controls alongside the observed run-to-run variability.

Dear Authors,

Thank you for your response. I have read your response and will maintain my positive score.

Thank you for taking the time to review our response and for your continued support of our work. We really appreciate your helpful feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the author response

Dear Area Chair and Reviewers,

We thank the Area Chair for overseeing the review and the reviewers for their detailed feedback. We have revised the manuscript in response.

1. Clarifying the audit and its relationship to prior work. To clarify how to interpret correct answers that survive evidence deletion, we explain that the Behavioral Necessity Rate (BNR) measures how often a previously correct answer is lost when we hold the final question fixed and remove one annotated support. Our new closed-book and answer-survival analyses examine how models can still answer correctly after deletion. We also expanded the discussion of CofCA, CRiT-QA, and Plausible Distractors and applied our evidence-deletion audit to CofCA to measure the effect on final-answer accuracy.

2. Testing dependence on intermediate evidence. To interpret the high BNR, we used separate controls to examine the roles of answer-string removal, deletion scope, and run-to-run variation. We first examined non-final-hop deletions, which usually leave the answer string in the context. On the 790 paired MuSiQue questions, non-final-hop BNR rises from 8.6% on the source questions to 91.8% on REALHOP. On a matched subset of 200 question pairs, deleting only the minimal supporting evidence yielded an overall BNR of 22.6% on original MuSiQue and 95.3% on REALHOP. In the full 790-pair audit, the share of REALHOP questions that lost the correct answer under every single-hop deletion ranged from 87.1% to 89.4% across the three models. This analysis includes only questions answered correctly with full context and with valid deletion results for all hops. On LongBench, targeted deletion also causes substantially more answer loss than unmodified reruns or matched background deletion.

3. Isolating the contribution of complete competing paths. To test whether the links between distractor facts contribute to evidence dependence, we added the Isolated control, which keeps distractor content comparable while breaking those links. Among MuSiQue questions answered correctly in both conditions, complete paths raise BNR by 23.3 percentage points, indicating that connectivity contributes to dependence on the targeted evidence.

4. Addressing the question of general multi-hop reasoning. HgcZ asked how the differences measured by REALHOP relate to broader multi-hop reasoning. In the correct-answer traces we analyzed from two models, explicit checks of the relations that distinguish plausible candidates appeared more often on REALHOP. We also examined whether the resulting model rankings align with other reasoning evaluations. Under the same multiple-choice format, REALHOP-LongBench’s ranking agrees more closely with an external reasoning composite than the original LongBench v2 ranking does (Spearman 0.83 vs. 0.54). This provides preliminary evidence of alignment with other reasoning evaluations. REALHOP complements natural multi-hop benchmarks by testing how much correct answers depend on annotated intermediate evidence when several candidates are plausible.

Thank you for considering the revised manuscript.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.