acceptodds
Under review as a conference paper at ICLR 2027

FondueQA: A Benchmark for Evaluating Parametric Compensation in Multi-Hop QA

Abstract

Large Language Models (LLMs) show remarkable performance based on parametric knowledge, i.e., their ability to recall facts and associations internalized in a model's weights. However, studying this capability on established multi-hop benchmarks may be complicated by benchmark contamination, as many of these benchmarks have been publicly available long enough to plausibly appear in modern pretraining corpora. As a result, correct answers under missing evidence may reflect either general parametric knowledge or memorization of benchmark-specific instances. To study how models use internal knowledge when retrieval evidence is incomplete, we introduce **FondueQA**, in which every required multi-hop fact is uniquely localized to a designated evidence document. This enables controlled document-level evidence ablations to study parametric compensation under incomplete retrieval. Our empirical evaluation reveals that parametric compensation strongly depends on both entity popularity and the structural role of missing evidence. Generally, models compensate more successfully for facts involving popular entities. On **FondueQA**, we show that bridging evidence can be more useful than the answer-bearing document itself. Moreover, models may recall a missing fact in isolation but fail to use it under partial evidence, particularly when the context contains plausible competing entities, which can reduce Exact Match by up to 15 percentage points.

Primary area:
datasets and benchmarks
Keywords:
Retrieval-Augmented Generation, Multi-Hop Question Answering, Retrieval Evaluation, Parametric Knowledge, Datasets and Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8gcuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FondueQA, a benchmark for studying how language models use internal knowledge to answer questions when retrieval evidence is incomplete.

Critical strengths
  1. The dataset provides a substantial collection of questions across several evidence structures, supporting comparisons under different evidence conditions.
  2. Studying how models compensate for missing evidence offers an interesting perspective on the interaction between internal knowledge and retrieval.
Critical weaknesses
  1. The lack of independent human validation leaves answer correctness, evidence support, and the claimed isolation of required facts insufficiently verified.
  2. Despite evaluating several open model families, the absence of commercial flagship models leaves unclear whether stronger systems would find this benchmark too easy.
  3. Results are mostly presented in figures, making exact comparisons difficult. Please tabulate absolute accuracy across models and evidence conditions, clearly specifying denominators.
  4. The text within the figures is too small, reducing readability and visual clarity. Please enlarge it throughout the paper.
Questions for authors
  1. Could two annotators independently audit a representative subset, such as 20%, and report agreement, error rates, and disagreement resolution?
  2. How do representative commercial flagship models perform under the same evidence conditions, and do the main findings hold for them?
  3. Would a small reinforcement learning study improve a local model toward 80% accuracy on separate test questions under clearly specified evidence conditions?
3 comments

Thank you very much for your thoughtful feedback and for recognizing the careful construction of FondueQA and its potential for studying model behavior under incomplete evidence.

W1 & Q1: Human annotation

We agree that independent human validation is important. We therefore audited 500 randomly sampled questions from the final release. Each question was independently reviewed by two annotators from a pool of six, covering the question, answer, required facts, and complete context.

Overall, 94.3% of the 1,000 individual judgments were Valid, and both annotators judged 89.6% of questions valid. Most flagged concerns involved other issues, like ambiguous questions or incomplete answers. Direct leakage was flagged by at least one annotator in only 8 questions (1.6%) and inferential leakage in 9 (1.8%).

Paired assessmentQuestionsShare
Both judged valid44889.6%
Both identified an error51.0%
Disagreement479.4%

Valid/Error agreement was 90.6%, with Krippendorff’s α = 0.126 (95% bootstrap CI: −0.006 to 0.272). Most judgments were Valid, which contributes to the low chance-corrected agreement despite 90.6% raw agreement. The large majority of questions received valid judgments, and leakage flags were uncommon. These findings support the effectiveness of our fact-isolation procedure. We will include the audit results and agreement statistics in the revised manuscript.

W2 & Q2: Flagship LLMs

We agree that evaluating commercial flagship models would provide a valuable complementary perspective on the generality of our findings. However, the main FondueQA context-conditioned evaluation alone requires approximately 130,000 requests per model, before additional atomic-probe and candidate-competition evaluations. Running the full evaluation on commercial APIs would therefore incur substantial costs that exceed our available monetary budget.

W3 & W4: Tabularization & Presentation

Thank you for pointing out the presentation issues. We will add a table, reporting absolute Exact Match scores and explicit evaluation denominators. This complements the relative comparisons with absolute performance across the key evidence conditions, including No-Gold, All-Gold, Bridge-only, and Target-only. We are also revising the figures throughout the manuscript to improve readability, particularly by increasing font sizes, axis labels, and legends.

Q3: RL baseline

Thank you for the suggestion. For the scope of this work, we focus on evaluating and understanding parametric compensation rather than improving model performance through additional training. We therefore consider the proposed RL experiment beyond the scope of the current submission.

We sincerely thank the reviewer for their constructive feedback and for the opportunity to further strengthen the empirical evaluation and argumentation of our paper. We hope that our responses adequately address the concerns raised and would welcome further discussion of any remaining questions or concerns.

Thank you for the response. My concerns remain unresolved, so I will maintain my original score. In addition, little suggestion, even if you cannot conduct the experiments I requested, I would still encourage you to address my questions through alternative experiments. Otherwise, your response is unlikely to convince me and other reviewers.

Thank you for your follow-up and for encouraging us to explore additional experiments. We have since conducted two further evaluations and would like to share the results.

Additional model evaluation. We agree and appreciate the option for reporting alternative experiments under similar conditions, and evaluate GLM 5.3 as a strong open-weight alternative to commercial flagship models. According to the LLM Stats reasoning leaderboard, it currently ranks 20th overall and third among open-weight models. In the following, we share results obtained within our available compute budget, using gpustack/GLM-5.3-W4A8:

Question typenAll-GoldPartial APartial BNo-Gold
Two-hop Bridge15179.5%60.0%54.1%27.2%
Comparison22386.1%64.4%57.1%53.4%
Contrastive4081.7%58.3%65.0%62.5%
Cross-Property39100.0%94.0%80.3%92.3%

These preliminary results cover 453 questions for which predictions have been computed across all four evidence conditions. The results show that performance varies considerably with the available evidence. Particularly, for two-hop bridge questions, accuracy drops from 79.5% in the All-Gold setting to 27.2% in No-Gold. These findings suggest that even a strong open-weight model struggles when supporting evidence is missing.

RL experiment. Following your suggestion, we also implemented a small-scale GRPO-LoRA study with Gemma-4-E2B and Gemma-4-E4B, training on full- and partial-evidence conditions. Given the limited resources during the rebuttal, we evaluated the models after 68 and 51 training steps, respectively.

We evaluated both models on 986 held-out questions, ensuring that no required facts overlap between the training and held-out sets. Mean partial-evidence EM increased from 34.79% to 36.41% for E2B (+1.62 pp) and from 40.82% to 42.55% for E4B (+1.73 pp).

We take this small-scale experiment indeed as an indicator that training under these evidence conditions can improve an LLM's behavior. Although the focus of our paper is on the evaluation and providing an analysis on the parametric compensation capabilities, we assess the model optimization problem motivated by the RL experiment as a fruitful future direction.


We appreciate that the reviewer agrees that our “substantial collection across evidence structures” supports the investigation of parametric compensation. Moreover, the reviewer’s feedback helped us to improve the paper’s presentation, strengthen its validity through human annotation, additional experiments with a frontier open-source model and an initial RL study highlighting a promising direction for future work, potentially leveraging our proposed benchmark. We hope that we have satisfactorily addressed all of the reviewer's concerns and that the additional results and clarifications provide a stronger basis for reassessing the paper's contributions. We would welcome any further feedback or questions and remain open to additional discussion.

Reviewer_8KBjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FondueQA, a multi-hop QA benchmark in which each required fact is isolated to a single gold document. This enables clean document-level evidence ablations, which the authors use to study how LLMs compensate for missing evidence with their parametric knowledge.

Critical strengths
  1. The benchmark construction is careful and rigorous, with a clear validation and repair pipeline. The dataset is large, and the authors conduct extensive experiments and analyses.

  2. The paper addresses several practical needs. Beyond the analyses in the paper, I see another potential use: with each fact appearing in exactly one document, FondueQA could make it easier to build other controlled experiments, such as editing a single document (e.g. counterfactual edits for knowledge-conflict studies) with less risk of creating conflicts with the rest of the context.

Critical weaknesses
  1. Fact isolation is the core property of the benchmark, but it is mainly verified by GPT-OSS, and the accuracy of these judgments is not reported. A human audit of a random sample, checking whether any other document still supports a removed fact, would make the isolation claim more convincing.

  2. Most questions appear to involve two hops, with only a small fraction involving longer chains. It would be helpful to report the hop-count distribution and whether the findings also hold for longer reasoning chains.

Questions for authors
  1. Does the bridge-only advantage reflect the structural role of the evidence, or differences in how easily the missing facts can be recalled? Could the authors report the atomic recall rates and popularity of the missing facts in each condition, and compare performance on matched subsets? Figure 6 also suggests that several models show little or no bridge-only advantage.

  2. In figure 2, several models answer a substantial fraction of questions correctly without recalling the missing fact. What might explain this? Since many questions use yes/no or multiple-choice formats, could the authors break down the results by answer type to help assess the role of chance?

2 comments

Thank you very much for your thoughtful and constructive feedback, and for recognizing the careful benchmark construction and the potential of FondueQA for further research.

W1: Fact Isolation

We agree that independent human validation is important. We therefore audited 500 randomly sampled questions from the final release. Each question was independently reviewed by two annotators from a pool of six, covering the question, answer, required facts, and complete context.

Overall, 94.3% of the 1,000 individual judgments were valid, and both annotators judged 89.6% of questions valid. Most flagged concerns involved other issues, like ambiguous questions or incomplete answers. Direct leakage was flagged by at least one annotator in only 8 questions (1.6%) and inferential leakage in 9 (1.8%).

Paired assessmentQuestionsShare
Both judged valid44889.6%
Both identified an error51.0%
Disagreement479.4%

Valid/Error agreement was 90.6%, with Krippendorff’s α = 0.126 (95% bootstrap CI: −0.006 to 0.272). Most judgments were valid, which contributes to the low chance-corrected agreement despite 90.6% raw agreement.

The large majority of questions received valid judgments, and leakage flags were uncommon. These findings support the effectiveness of our fact-isolation procedure. We will include the audit results and agreement statistics in the revised manuscript.

W2: Number of hops

Thanks for this important remark. We will include the hop-count distribution in the revised manuscript. The final dataset contains 12.830 two-hop questions, 15 three-hop questions and 218 nested bridge questions.

We would like to highlight that we assess the fact-isolation requirement as a key property of FondueQA for evaluating the multi-hop reasoning capabilities. Given kk required facts, each assigned to a separate gold document, ensuring isolation requires checking each fact against the other gold documents, resulting in k(k−1)k(k-1) support checks (Appendix A.2). As the number of hops increases, satisfying these constraints becomes more challenging because additional connected facts introduce more opportunities for cross-document leakage. By construction, the probability of successfully passing all checks results in a retention rate of 0.15 as shown in Figure 5 across all QA pairs in our setup. Theoretically, the number of valid QA pairs follows a super-exponential decay pk(k−1)p^{k(k-1)} w.r.t. the hop count, where pp is the probability that a single isolation check succeeds. Therefore, even for 3-hop QA pairs, the number of valid pairs drastically shrinks.

We agree with the reviewer that to foster research on longer reasoning chains, one possibility would be to examine further relaxations built on our validation pipeline to increase the number of hops in longer chains, or investigations of other selection heuristics. In this work, we hold on the strict fact isolation criteria which we will point out more clearly in our revised manuscript.

Q1: Atomic recall rates

Following the reviewer’s suggestion, we will add an analysis that restricts the comparison to questions for which both facts are correctly recalled in their respective atomic probes. The Bridge-Only advantage persists in 12 of 13 model configurations, with paired differences ranging from 3.7 to 30.2 percentage points. GPT-OSS-20B is the exception, favoring Target-only by 11.7 points. This suggests that differences in atomic recall success alone do not explain the observed role asymmetries.

We additionally investigate entity popularity using a separate popularity-matched subset, independent of atomic recall. We retain paired questions whose bridge and target entities differ by at most 0.2 in log10(2023 pageviews + 1), yielding 824-832 questions per configuration. Bridge-Only and Target-Only are evaluated on the same questions, with matched retained-document positions and distractors. The table reports absolute EM and paired differences with pointwise 95% confidence intervals from 2,000 question-level bootstrap resamples.

ModelnBridge-Only EM (%)Target-Only EM (%)Difference (pp), 95% CI
DeepSeek-R1-32B83225.627.2-1.6 [-4.8, 1.7]
Gemma-4-E2B83217.19.6+7.5 [4.9, 10.3]
Gemma-4-E4B83222.215.2+7.0 [3.9, 10.1]
Gemma-4-12B83229.920.4+9.5 [5.8, 13.4]
Gemma-4-26B-A4B83233.527.4+6.1 [2.0, 10.3]
Gemma-4-31B83232.946.0-13.0 [-17.6, -8.7]
GPT-OSS-20B83233.927.7+6.2 [2.6, 9.7]
GPT-OSS-120B83247.557.8-10.3 [-14.1, -6.4]
Qwen3-8B82426.317.6+8.8 [5.4, 12.1]
Qwen3-32B82431.529.6+1.9 [-1.6, 5.7]
Qwen3.6-27B83240.750.7-10.0 [-13.7, -6.1]
Qwen3.6-35B-A3B83241.449.4-8.0 [-12.1, -4.0]
Qwen3.8-27B83236.149.4-13.4 [-17.4, -9.3]

On this subset, six configurations favor Bridge-Only with intervals entirely above zero, five favor Target-Only, and two have intervals overlapping zero. We report the recall and popularity controls separately because imposing both restrictions leaves only 4-65 questions per configuration. The popularity analysis therefore does not jointly control for atomic recall. Furthermore, the popularity-matched subset contains a different selection of questions, which should be considered when comparing its results with the full cohort.

Regarding Figure 6, we agree that the role effect is not homogeneous across models. We would like to highlight that we do not claim a universal Bridge-Only advantage, but rather our findings indicate that the relative utility of bridge and target evidence varies across models. We will make this qualification explicit in the revised manuscript.

Q2: Recall of missing facts

Following the reviewer’s suggestion, we will add an analysis of joint atomic-recall and multi-hop outcomes across answer domains. In the following, we distinguish three categories, namely Binary, with exactly two possible answers, Bounded, with a clearly defined but larger set of possible answers (e.g., countries or languages), and Open, without a predefined limit on possible answers (e.g., people, organizations or dates). Across our 13 reasoning-mode configurations, model-macro multi-hop accuracy conditional on failed atomic recall is 65.5% for Binary, 38.8% for Bounded, and 18.0% for Open questions. We additionally examine the answer domains of the atomic probes and their corresponding multi-hop questions jointly.

Atomic-probe domainMulti-hop domainNot Recalled / Correct (%)
OpenBinary55.5
BoundedBinary23.5
OpenBounded28.0
BoundedBounded12.0
BoundedOpen11.0
OpenOpen14.2

These results suggest that differences in answer domains partly explain the observed mismatch. Importantly, the mismatch is not confined to two-choice multi-hop questions. The remaining context may narrow plausible answers or provide related entity information, but differences in relations, popularity, and question composition remain alternative explanations.

We sincerely thank the reviewer for their constructive feedback and for the opportunity to further strengthen the empirical evaluation and argumentation of our paper. We hope that our responses adequately address the concerns raised and would welcome further discussion of any remaining questions or concerns.

Reviewer_D3yrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FondueQA, a multi-hop QA benchmark for investigating how LLMs compensate for missing retrieval evidence using parametric knowledge. The benchmark employs fact isolation and leakage checks to enable controlled removal of supporting evidence. Experiments across four model families show that parametric compensation is influenced by entity popularity, the role of evidence, and the presence of competing entities. The results further show that bridge evidence can be more useful than answer-bearing evidence on FondueQA, and that successful recall of individual facts does not necessarily lead to correct multi-hop answers.

Critical strengths
  1. The proposed FondueQA benchmark employs an explicit fact-isolation design and leakage checks to enable controlled, role-specific evidence removal. This is a useful addition to existing multi-hop QA benchmarks, allowing for more interpretable comparisons under incomplete retrieval.
  2. The experimental findings on FondueQA show that bridge evidence can be more useful than answer-bearing evidence, while successful recall of individual facts does not necessarily lead to correct multi-hop answers. These findings provide useful insights into how evidence structure and factual recall affect parametric compensation.
  3. The proposed framework is formally well-defined and clearly presented, particularly the requirements for controlled evidence removal in Section 3.2.
Critical weaknesses
  1. The fact-isolation procedure needs stronger validation. The paper uses GPT-OSS-120B to check whether each required fact appears only in its designated gold document. However, it is unclear whether the procedure also detects facts that can be logically inferred from other documents. Fact isolation should ensure that missing facts cannot be inferred from the remaining evidence.
  2. The reliability of the automated validation is not sufficiently established. The paper does not report an independent quantitative evaluation of the support and leakage checks. Human evaluation or independent verification is needed to establish the reliability of the constructed evidence gaps.
  3. The results in Figures 3 and 4 need further explanation. The authors show that competing evidence affects parametric compensation under the Same-Gold and Same-Other settings. However, even under Same-Gold, correct answers may result from answer biases rather than successful recovery of missing facts. Further analysis is needed to distinguish these possibilities.
  4. The Bridge-only and Target-only comparison needs to control for factual recall difficulty. The two conditions remove different facts, which may differ in parametric accessibility. Although atomic recall is measured, the paper does not compare the two conditions on questions where both facts are successfully recalled. Such a comparison would help isolate the effect of evidence structure.
Questions for authors
  1. Could an independent audit of a representative sample estimate the rates of unsupported gold facts and residual leakage into questions or other documents?
  2. How can the authors further validate whether the results in Figures 3 and 4 reflect genuine parametric compensation rather than contextual answer biases?
  3. Does the Bridge-only advantage persist when restricting the evaluation to questions for which the same model correctly recalls both required facts in isolation?
  4. Do the observed mismatches between atomic recall and multi-hop success persist under alternative phrasings of the atomic fact questions?
3 comments

Thank you very much for your thoughtful feedback and for recognizing the value of FondueQA's fact-isolation design and its contribution to controlled multi-hop QA evaluation.

W1/W2/Q1: Fact Isolation & Reliability

Our fact-isolation procedure is not limited to explicit leakage (cf. Appendix A.2). The validator is explicitly instructed to consider “ordinary aliases, inverse wording, pronouns with a clear antecedent, and evidence distributed across the supplied sentences” (from the prompt). It also flags weak bypass clues, including a “synonym, paraphrase, geographic implication, alternate valid relation value, or short multi-sentence combination” (from the same prompt). Distractors undergo corresponding semantic checks, explicitly covering cases where “multiple sentences together reveal the fact or make a missing gold document practically unnecessary” (from the distractor-checking prompt). However, these document-wise checks do not establish that arbitrary combinations of remaining documents cannot support an alternative inference chain.

To address this, we conducted an independent human audit to assess whether the released benchmark contains amongst others inferential leakage. Each question was independently reviewed by two annotators from a pool of six, covering the question, answer, required facts, and complete context.

Overall, 94.3% of the 1,000 individual judgments were “Valid”, and both annotators judged 89.6% of questions as valid. Most flagged concerns involved other issues, like ambiguous questions or incomplete answers. Direct leakage was flagged by at least one annotator in only 8 questions (1.6%) and inferential leakage in 9 (1.8%).

Paired assessmentQuestionsShare
Both judged valid44889.6%
Both identified an error51.0%
Disagreement479.4%

Valid/Error agreement was 90.6%, with Krippendorff’s α = 0.126 (95% bootstrap CI: −0.006 to 0.272). Most judgments were valid, which contributes to the low chance-corrected agreement despite 90.6% raw agreement.

The large majority of questions were judged valid, with direct and inferential leakage flagged in only 1.6% and 1.8% of questions, respectively. These findings provide evidence that our fact-isolation procedure largely succeeds in preventing leakage. We will include the audit results and agreement statistics in the revised manuscript.

W3/Q2: Competing evidence

Thank you for highlighting the distinction between genuine parametric compensation and contextual bias. We would like to clarify that the Same-Gold and Same-Other settings are deliberately designed as reasoning traps and their purpose is to investigate whether plausible competing evidence shifts model responses towards or away from the correct answer.

This is precisely the behavior we intended to study. However, we agree that the original presentation did not make this distinction sufficiently explicit and we will clarify it in the revised manuscript. To further illustrate this effect, we will add an analysis to the manuscript separating predictions according to whether the missing bridge fact was successfully recalled in isolation.

The results shown in the following table highlight clear directional shifts at k=19. Among questions where the bridge fact was not successfully recalled in isolation, Same-Gold substantially increases the gold-answer share, while Same-Other increases the share of answers matching its designated incorrect alternative. When the bridge fact was successfully recalled, the same directional shifts remain visible but are considerably smaller.

Bridge probeModeGold answer share (Random / Same-Gold)Designated wrong answer share (Random / Same-Other)
IncorrectStandard33.1% / 54.5%1.5% / 29.7%
IncorrectReasoning55.0% / 68.0%2.1% / 15.7%
CorrectStandard68.6% / 73.4%0.6% / 7.7%
CorrectReasoning86.4% / 89.0%0.05% / 0.94%

W4/Q3: Factual Recall

We agree that differences in the accessibility of bridge and target facts are important to consider. Following the reviewer’s suggestion, we conducted an additional analysis, restricting the comparison to questions where both required facts are recalled in isolation.

On this subset, the Bridge-Only advantage persists in 12 of 13 configurations, with paired differences ranging from 3.7 to 30.2 percentage points. GPT-OSS-20B is the exception, favoring Target-Only by 11.7 points.

We additionally provide a separate popularity-matched analysis, independent of atomic recall. We retain paired questions whose bridge and target entities differ by at most 0.2 in log10(pageviews + 1), yielding 824-832 questions per configuration. Here, six configurations favor Bridge-Only with 95% bootstrap intervals excluding zero, five favor Target-Only, and two have intervals overlapping zero.

We report these controls separately because requiring both successful atomic recall and similar popularity leaves only 4-65 questions per configuration. The popularity-matched analysis therefore does not jointly control for atomic recall. Further remarks are provided in our response to Reviewer 8KBj.

Q4: Alternative phrasings

Based on reviewer 8KBj’s feedback, we will add an analysis across Binary, Bounded, and Open answer domains to discuss where the mismatches between failed atomic recall and multi-hop success arise. The results are consistent with differences in answer requirements contributing to the mismatch between atomic recall and multi-hop correctness. However, the mismatch also persists for Open questions, and these descriptive comparisons do not isolate a causal answer-space effect. In addition, we are currently conducting an evaluation of alternative atomic-probe phrasings, covering 1,000 questions with four paraphrased versions each across five models. We will report these results in the revised manuscript to further address the robustness of atomic recall measurements.

We sincerely thank the reviewer for their constructive feedback and for the opportunity to further strengthen the empirical evaluation and argumentation of our paper. We hope that our responses adequately address the concerns raised and would welcome further discussion of any remaining questions or concerns.

Q4: Alternative phrasings

Following the reviewer's suggestion, we evaluated four alternative questions across five reasoning models. The alternatives were generated by GPT-OSS-120B with instructions to preserve the original question’s meaning, the queried fact and the expected answer type and answer domain. Each alternative was then validated by GPT-OSS-120B in a separate call against the same requirements. We then compare the original atomic question with all five formulations, counting a fact as recalled if at least one formulation is answered correctly.

The table shows the percentage of questions for which atomic recall fails but the corresponding multi-hop question is answered correctly. The evaluated questions and multi-hop predictions remain unchanged. Any reduction therefore reflects cases where an initially failed atomic question is answered correctly under an alternative phrasing.

ModelMain questionsOriginalAny of 5Reduction (pp)
DeepSeek-R1-32B1,39131.6%26.1%5.5
Gemma-4-E4B1,42030.7%27.5%3.2
Gemma-4-31B86039.1%34.4%4.7
GPT-OSS-120B1,42034.9%30.7%4.2
Qwen3.6-35B-A3B1,16036.9%30.4%6.5

The reduction is expected, as models now have five opportunities to recall each fact instead of one. However, 82.3-89.6% of the original mismatch remains. This shows that alternative phrasings explain some of the mismatch, but most of it remains. Given the relatively small reductions across all five configurations, we do not see a strong reason to extend this experiment to all models.

Some models exhausted their 15,000-token reasoning budget on individual atomic probes without producing a final answer. We exclude these cases and restrict the analysis to atomic questions with usable responses for all five formulations. This leaves 562-1,000 atomic questions per configuration. The number of main questions can exceed 1,000 because individual atomic probes may be associated with multiple multi-hop questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.