acceptodds
Under review as a conference paper at ICLR 2027

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

Abstract

Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. To characterise this difficulty systematically, we propose quantitative, sample-level definitions of two evidence-related properties, *scope* and *dispersion*, and further extend this characterisation with *reasoning depth*. To study how these three properties contribute to model failures, we construct *LURCH*, a multi-domain question-answering benchmark in which each question is provided with its answer and supporting evidence, enabling all three properties to be quantified. Controlled experiments on both proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, evidence dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context Reasoning, Model Diagnosis, Long-Context Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cbrhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores reasoning task difficulties over long contexts. To further analyze the difficulty measure, the authors provide 3 properties beyond context length, namely: scope, dispersion and reasoning graph/depth. The authors create a multidoman question answering benchmark called LURCH. Experiments show that alongside context length, these 3 properties are associated with model performance and difficulty estimation.

Critical strengths
  1. The paper has a good motivation and addresses a relevant problem.
  2. The properties are simple, intuitive and easy to interpret.
  3. The authors experiment with 4 LLMs over multiple domains and datasets
Critical weaknesses
  1. The dispersion metric considers only the first and last evidence to calculate the dispersion (from eqn. 3). It does not consider whether there are evidences in between the first and last. This can be a potential issue because answers requiring multiple evidences should be more difficult, but this metric might fail to judge it properly.
  2. The impact of Evidence Dispersion and Reasoning Depth seems minimal. (from Fig.5,6)
  3. The comparison with many existing relevant studies is missing. This would help clarify the differences and contributions of the benchmark.
Questions for authors
  1. Is it possible for the dispersion metric to account for the number of evidences and the spread of it, instead of only the first and last evidence?
  2. Can the authors explain or expand on the behavior of Gemini 3.8 Flash, which is unlike the other evaluated models?
  3. Can the authors compare LURCH with relevant benchmarks: [1] and [2]?

References:

  1. Yang, Zi. "Retrieval or holistic understanding? dolce: Differentiate our long context evaluation tasks." arXiv preprint arXiv:2409.06338 (2024).
  2. Taewhoo Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, and Jaewoo Kang. 2025. ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5497–5512, Albuquerque, New Mexico. Association for Computational Linguistics.
4 comments
AuthorsOpenReview ↗

Response to Reviewer cbrh (1/2)

We thank the reviewer for the thoughtful review, and for finding the motivation strong and the properties simple and interpretable. We address the concerns and answer the questions below.

Weakness 1/Question 1

We agree that Eq. 3 summarises outer evidence span (subtracting total evidence length) rather than explicitly modeling intermediate segments. We also considered these alternative definitions for dispersion: (i) Uniformity compares the evidence layout with N evenly spaced points using Wasserstein-1 distance W1W_1, which here is the mean distance between the sorted segment centres and the sorted reference points, as a fraction of context length. Uniformity is 1−2⋅W11-2 \cdot W_1, which runs from 0 (all evidence at one end) to 1 (evenly spaced). (ii) Count is the number of evidence items. (iii) Position spread is the standard deviation of the positions of all evidence tokens, normalized by context length, with each token weighted equally.

MeasureHow to calculateΔ25→75 (pp)
Dispersion (ours)Gaps between consecutive segments−1.12 [−1.85, −0.38]
UniformityPositions of all segments and their number−0.48 [−1.01, 0.05]
CountNumber of evidence items−1.18 [−1.77, −0.59]
Position spreadPositions of all evidence tokens−1.24 [−2.02, −0.46]

All three alternatives show the same negative relationship with score, and our scope and reasoning-depth estimates remain unchanged across every specification. Using all token positions (Position spread) or segment counts yields contrasts nearly identical to our dispersion metric, while Uniformity shows a weaker effect. We therefore apply the simpler definition in Section 3.

Weakness 2

We thank the reviewer for raising this point. We disagree that the effects of dispersion and reasoning depth are minimal and further clarify our claims and results. Our central claim is that context length alone cannot fully capture long-context reasoning difficulty, and our results show that scope, dispersion, and depth add significant explanatory signal beyond length (χ32=73.0,padj=4.0×10−15\chi_3^2 = 73.0, p_{\text{adj}} = 4.0 \times 10^{-15}). Because baseline accuracy in LURCH is high (~85%), context length itself shifts scores by only −1.66 pp-1.66\text{ pp} from Q25 to Q75 (Table 1). Against this benchmark, reasoning depth produces a comparable interquartile drop overall (−1.60 pp-1.60\text{ pp}) and a larger drop (−3.02pp-3.02\text{pp}) across the 9 tasks with untied depth quartiles (Table 7), while doubling depth incurs more than twice the penalty of doubling length (−2.70 pp-2.70\text{ pp} vs. −1.04 pp-1.04\text{ pp}).  Crucially, these effects also intensify on longer inputs and when multiple demands coincide. Figure 5 and Table 2 show that the impact of dispersion grows with context length: moving from Q25 to Q75 in length nearly doubles the score drop under high dispersion (−2.01 pp-2.01\text{ pp} vs. −1.13 pp-1.13\text{ pp} at low dispersion; interaction −0.88 pp,padj=0.01-0.88\text{ pp}, p_{\text{adj}} = 0.01), and similarly widens the length drop at high depth (−1.82 pp-1.82\text{ pp} vs. −1.16 pp-1.16\text{ pp}). In Figure 6, combining high dispersion with high scope or depth creates 6.5 pp6.5\text{ pp} to 10.0 pp10.0\text{ pp} spreads across difficulty regimes.

AuthorsOpenReview ↗

Response to Reviewer cbrh (2/2)

Weakness 3/Question 3

We thank the reviewer for these references; the revised Related Work now discusses both. DOLCE [1] and ETHIC [2] each quantify how much of the context a question needs. LURCH differs in three ways:

  1. Located evidence. DOLCE fits its parameters to a probing LLM’s answers on randomly sampled spans, and ETHIC counts the chunks GPT-4o judges necessary. LURCH computes scope and dispersion from the token positions of evidence that is benchmark-provided, or retrieved and verified as sufficient.
  2. Amount, position and depth are separated. DOLCE’s λ is closest to our evidence span H(E), which we split into scope and dispersion. Neither work assigns a reasoning-depth score.
  3. Linked to scores within tasks. Those two works stop at a task categorisation (DOLCE) and a new benchmark (ETHIC). LURCH estimates how each property relates to scores at fixed task, length and LLM (RQ1–RQ3).

[1] Yang, Zi. "Retrieval or holistic understanding? dolce: Differentiate our long context evaluation tasks." arXiv preprint arXiv:2409.06338 (2024).

[2] Lee, Taewhoo, et al. "Ethic: Evaluating large language models on long-context tasks with high information coverage." NAACL. 2025.

Question 2

We isolate the cause to the SciTrek[1] benchmark, and dropping it from Figure 4 for Gemini 3.8 Flash gives a scope-dependent interquartile score contrast of −1.54 pp [−2.62, −0.45], down from +1.55 pp

+0.42,+2.68+0.42, +2.68

, which is in line with the other 3 LLMs. Dropping any one of the other 16 tasks leaves it positive (+1.50 to +2.70 pp). Further analysis revealed ambiguity in some of SciTrek’s low-scope questions responsible for incorrect answers. Note that SciTrek provides evidence data, so this is independent of our evidence retrieval pipeline.

Cause: SciTrek's low-scope questions need only the articles' titles and author lists. Many ask how many articles a person wrote (e.g. "How many articles has A authored?"). SciTrek counts only the provided full articles; however, the same names also appear in the articles' reference lists, so the prompt question allows a broader reading. Gemini often takes that reading and counts cited papers. For one question with gold answer 1, it answers 8, listing "the main article" plus "7 cited articles in its references". On the 74 author-count questions, Gemini overcounts in 33.8% of attempts, compared with 5.9–8.6% for the other LLMs. This lowers its accuracy on SciTrek's low-scope questions, which causes the positive relation.

Clarification experiment: To test this, we added one sentence to the 379 SciTrek questions that SciTrek itself marks as not requiring reference lists: Note: “article” means one of the full articles provided above; papers that appear only in reference lists do not count as articles. The articles, evidence and answers are unchanged. We reran all four LLMs with the paper settings:

Gemini 3.8 FlashGPT-5.6 LunaQwen3.8-27BQwen3.5-4B
Accuracy, 379 questions0.786 → 0.8740.819 → 0.8450.865 → 0.8780.848 → 0.858
Over-count rate, author-count questions33.8% → 2.7%7.2% → 2.7%8.6% → 4.1%5.9% → 3.2%
Scope Δ25→75, all 17 tasks (pp)+1.55 → −0.47
−1.54,+0.61−1.54, +0.61

| −3.27 → −3.85 | −1.39 → −1.71 | −5.40 → −5.63 |

Overcounting falls for every LLM, and Gemini’s scope contrast is no longer positive. It does not reach the leave-SciTrek-out estimate, which also removes SciTrek’s other 621 questions. The other LLMs’ scope contrasts move by less than 0.6 pp. Gemini’s dispersion contrast shrinks from −2.19 to −1.47 pp, and its CI now overlaps those of the other LLMs. The reasoning-depth contrasts are unchanged.

In the revised paper: Our main analyses keep SciTrek’s original prompt questions, so that LURCH stays comparable with its source benchmark. Section 5.3 now states this cause where the Gemini contrast is reported, and Appendix H reports the experiment above with RQ1–RQ3 recomputed on the clarified questions.

[1] Li, Miao, et al. "SciTrek: Evaluating and Improving Long-Context Numerical Reasoning over Scientific Articles." TMLR.

Dear Reviewer,

We have carefully addressed your concerns and responded to all of your questions. Should you have any further questions or require additional clarification, we would be more than happy to address them.

Best regards,

The Authors

Dear Reviewer cbrh,

Thank you again for your time and thoughtful feedback on our paper. We have carefully revised the manuscript highlighted in blue in the paper and provided detailed responses to each of your comments. We hope our revisions, clarifications and additional experiments have satisfactorily addressed your concerns.

Thank you for your consideration.

Reviewer_froTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper constructs LURCH, a long-context, multi-domain question-answering benchmark which aims to characterizes the three key factors for model failures in long-context reasoning: evidence scope, evidence dispersion, and reasoning depth. The benchmark is constructed by standardising instances from existing long-context benchmarks by filtering, evidence annotation and building reasoning graphs. An empirical analysis on LURCH reveals many findings, such as 1) all three factors are negatively correlated with scores; 2) length matters more for broad/dispersed evidence but not necessarily reasoning depth; 3) deep reasoning over dispersed evidence is especially difficult.

Critical strengths
  1. The three factors on long-context reasoning performance proposed by the paper are reasonable and supported by experiment results. Judging from Fig. 10, the paper does a decent job on predicting the model performance with the proposed factors. The size of the dataset is also sufficiently large. The authors also provides code, which improves the reproducibility of this paper.

  2. Long context reasoning is an important problem in today's LLM community, and this paper studies an important problem on the in-detail evaluation of model's such ability, which is a timely complement to the current LLM community.

Critical weaknesses
  1. Some concerns exist on the dataset quality of this paper.

a) The benchmark proposed in this paper is a compilation of existing long context benchmarks, many of which is already available for a long time (e.g. HotPotQA). Thus, there is a risk of data contamination; i.e., models have either seen such text from benchmarks in training and can recite from context, or seen specific data curated to optimize performance on such benchmark.

b) As the majority of data only contains 1 or 2 layers of reasoning, and the portion of data with reasoning depth greater than 3 is smaller than 2% (as shown in Fig. 3) and also biased towards a particular dataset (Fig. 8), the results on reasoning depth could be biased. Also, in appendix E, Tab. 4, it is shown that the two annotators only get reasoning depth consistency of 65.6% with spearman coefficient 0.48, which poses a question on the validity of reasoning depth results. There are works that studies much deeper reasoning depth, such as DeepReasonQA [1] and GSM-∞\infty [2].

  1. The position of this paper should be more carefully discussed. While the three factors proposed by this paper is reasonable, similar factors have been discussed in prior works which are not referred to in this paper. For example, GSM-∞\infty and NovelHopQA [3] also points out the negative correlation of reasoning depth and performance; ETHIC [4] also quantifies information coverage as acknowledged in line 75-77; LongPibench [5] also studies "dispersion" of evidence (as "relative position feature level"). Additionally, Holobench [6] "adjusts key factors such as context length, information density, distribution of information, and query complexity", which is very similar to all three factors of this work (and Holobench also compares more detailed distribution than a single dispersion ratio). None of the work discussed above (except ETHIC) is cited by this paper.

References

[1] https://arxiv.org/pdf/2601.12465

[2] https://arxiv.org/abs/2502.05252

[3] https://arxiv.org/pdf/2506.02000

[4] https://arxiv.org/abs/2410.16848

[5] https://arxiv.org/abs/2410.14641

[6] https://arxiv.org/abs/2410.11996

Questions for authors

Is the reported findings consistent on different levels of model reasoning efforts, i.e., can any of the three negative factors on performance be mitigated with longer reasoning process?

6 comments
AuthorsOpenReview ↗

Response to Reviewer froT (1/2)

We thank the reviewer for the constructive feedback and for recognising the importance of the problem, the size of the dataset and the released code. We address the weaknesses and answer the questions below.

Weakness 1a

We agree with this concern and designed LURCH to avoid it in two ways. First, we remove instances that the model answers correctly without the context (Appendix B). Second, our analyses estimate relative difficulty within tasks, not absolute performance. Contamination that affects scores on a whole task is absorbed by GLM’s task fixed effects.

Regarding HotpotQA, we use the extended context, which is different from the original benchmark, and the absolute performance for each model shows no obvious data contamination.

ModelF1EM
Qwen 3.8 27B79.5062.32
Qwen 3.5 4B77.7761.74
GPT-5.6 Luna77.2559.33
Gemini 3.8 Flash77.1859.31

Weakness 1b

We agree with the reviewer that few instances need four or more reasoning steps. We see this as a property of naturally occurring long-context tasks. Synthetic benchmarks such as GSM-∞[1] and DeepReasonQA[2] reach far greater depth and are complementary.

Regarding the annotators, we agree that the agreement between two annotators is not very high. We therefore designed a human audit process, in which both auditors judged both graphs acceptable on every instance where the annotators agreed. Where the depths differed, each auditor preferred the Gemini graphs we use more often than the GPT graphs (2 vs 0 and 14 vs 3; Table 5).

Weakness 2

We thank the reviewer for providing these related papers. We have added the discussion in the Related Work section in the revised version.

Designed vs. measured: HoloBench[3], LongPiBench[4], GSM-∞, NovelHopQA[5] and DeepReasonQA set their attributes by design using their own constructed settings, and designed factors can move together: a NovelHopQA H-hop question is built from H paragraphs, so hop count changes evidence amount and depth at once. LURCH measures all three properties per instance in 17 existing benchmarks and estimates each net of the others, of length and of task.

Grounded in located evidence: DOLCE[6] and ETHIC[7] also characterise existing benchmarks, but from a probe model’s success on sampled sub-contexts (DOLCE) or GPT-4o’s per-chunk judgements (ETHIC). LURCH computes scope and dispersion from located evidence spans, benchmark-provided or retrieved and verified sufficient, not from the evaluated models’ answers.

What is new: RQ1 agrees with these studies on information amount (HoloBench, ETHIC), spacing (LongPiBench) and depth (GSM-∞, NovelHopQA). New are the joint within-task estimates, the stronger length effect when evidence is broad or dispersed (RQ2), and compounding (RQ3). DOLCE’s λ is closest to our evidence span H(E), which we split into scope and dispersion.

[1] Zhou, Yang, et al. "GSM-Infinite: How do your LLMs behave over infinitely increasing context length and reasoning complexity?." arXiv preprint arXiv:2502.05252 (2025).

[2] Peng, Miao, et al. "Incentivizing in-depth reasoning over long contexts with process advantage shaping." ACL 2026.

[3] Maekawa, Seiji, et al. "Holistic reasoning with long-context LMs: A benchmark for database operations on massive textual data." ICLR 2025.

[4] Tian, Runchu, et al. "Distance between relevant information pieces causes bias in long-context LLMs." ACL 2025.

[5] Gupta, Abhay, et al. "NovelHopQA: Diagnosing multi-hop reasoning failures in long narrative contexts." EMNLP. 2025.

[6] Yang, Zi. "Retrieval or holistic understanding? dolce: Differentiate our long context evaluation tasks." arXiv preprint arXiv:2409.06338 (2024).

[7] Lee, Taewhoo, et al. "Ethic: Evaluating large language models on long-context tasks with high information coverage." NAACL. 2025.

AuthorsOpenReview ↗

Response to Reviewer froT (2/2)

Question 1

The findings are consistent across reasoning effort. Higher or lower effort does not remove any of the associations.

All experiments in the paper use reasoning level medium. We re-ran a sample of 500 LURCH questions (all 17 tasks) with GPT-5.6 Luna and Gemini 3.8 Flash at low and high reasoning effort, and refitted the RQ1 model for each LLM at each level.

GPT-5.6 Luna

PropertyMetricLowMediumHigh
Scopeβ̂ (per SD)−0.33 [−0.72, 0.05]−0.39 [−0.78, 0.01]−0.29 [−0.68, 0.11]
Δ25→75 (pp)−3.9 [−8.4, 0.5]−4.4 [−8.7, −0.1]−3.1 [−7.3, 1.1]
Dispersionβ̂ (per SD)−0.15 [−0.46, 0.16]−0.14 [−0.44, 0.16]−0.12 [−0.42, 0.18]
Δ25→75 (pp)−1.7 [−4.9, 1.6]−1.5 [−4.7, 1.7]−1.2 [−4.3, 1.8]
Depthβ̂ (per SD)−0.26 [−0.48, −0.03]−0.17 [−0.38, 0.05]−0.23 [−0.46, 0.01]
Δ25→75 (pp)−4.0 [−7.3, −0.6]−2.5 [−5.7, 0.7]−3.3 [−6.7, 0.1]

Gemini 3.8 Flash

PropertyMetricLowMediumHigh
Scopeβ̂ (per SD)0.40 [−0.09, 0.88]0.33 [−0.15, 0.82]0.26 [−0.22, 0.74]
Δ25→75 (pp)2.8 [−0.7, 6.4]2.2 [−1.1, 5.5]1.8 [−1.6, 5.1]
Dispersionβ̂ (per SD)−0.54 [−1.05, −0.03]−0.47 [−0.95, 0.01]−0.31 [−0.76, 0.13]
Δ25→75 (pp)−2.2 [−4.4, 0.1]−1.8 [−3.7, 0.2]−1.2 [−3.0, 0.6]
Depthβ̂ (per SD)−0.30 [−0.62, 0.01]−0.26 [−0.60, 0.08]−0.25 [−0.57, 0.07]
Δ25→75 (pp)−3.4 [−6.8, 0.1]−2.7 [−6.3, 0.8]−2.7 [−6.2, 0.7]

Consistency. The direction of every association is the same at all three levels for both LLMs, matching the pattern in Figure 4. With 500 questions, the intervals are wide, and most include zero; the table is meant for comparing effort levels, and the full-data estimates are in Table 1 and Figure 4.

Note on Gemini’s scope contrast. It is positive at every effort level. This is the SciTrek question-wording effect explained in the additional analysis in Appendix H. This experiment uses SciTrek’s original prompt questions, as do the main analyses.

Thank you for your response. I think my remaining key concern is that, 94% of the data only has the reasoning depth 1 or 2 as shown in Fig. 2, which makes the depth factor almost binary. However, this paper uses "reasoning depth" as one of the key factors for investigation. The question is, would there be a more descriptive and discriminative factor than depth, for example, the structure (e.g. number of nodes/edges) in the evidence-to-answer reasoning graph? Otherwise, the statement in RQ1 "doubling reasoning depth..." (now line 96) is almost equivalent to say "reasoning depth 1 vs. 2".

Reviewer_2fXMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that context length is an incomplete proxy for long-context difficulty and proposes three sample-level, computable attributes as complements: evidence scope, evidence dispersion, and reasoning depth.
The authors build LURCH, a 6,550-question benchmark drawn from 17 natural long-context sources, each annotated with reference answers, retrieved-and-validated evidence spans, and LLM-generated reasoning graphs, and run controlled GLM analyses over four LLMs (two proprietary, two open-weight).
They report that (i) at fixed length all three attributes correlate negatively with accuracy, with depth doubling costing about as much as length doubling; (ii) lengthening context hurts mainly when evidence is broad or dispersed; and (iii) difficulty accumulates across attributes, so questions that are moderately demanding on all three are harder than questions extreme on one.

Critical strengths
  1. The problem is well motivated and convincingly demonstrated. Figure 1's three equal-length (92k) instances with accuracies of 1.00 / 0.75 / 0.58 make the "length is not difficulty" claim concrete before any statistics are introduced.
  2. The attributes are quantitative and cheap to compute, turning Goldman et al.'s qualitative scope/dispersion concepts into sample-level numbers; the dispersion definition (Equation 3) is a nice touch because it separates "how much evidence" from "where it sits".
  3. LURCH is a genuinely useful resource, not just an eval: evidence spans plus repaired reasoning graphs plus all four attributes support many follow-up studies, and the 17-source, multi-domain coverage is broader than most prior difficulty analyses.
Critical weaknesses
  1. The attributes inherit the errors of an automatic pipeline. Evidence is retrieved by Gemini 2.5 Pro and then expanded to full paragraphs/tables, so scope and dispersion describe the “expanded supporting context”, not a minimal sufficient set; over-retrieval inflates scope and under-retrieval deflates dispersion, and neither error is measurable from the reported numbers.
  2. Incremental explanatory value over task identity is small. Table 6 shows task alone reduces cross-entropy by 4.51% while adding the three properties on top of task + LLM + length buys only a further 0.58%; much of the "attribute effect" may be task-level difficulty in disguise.
  3. Metric heterogeneity is only partially addressed. Mixing EM, token-F1 and LLM-as-judge (Loong) across tasks and then centring by task–LLM mean helps, but different metrics have different floors and variances, which the additive GLM does not model.
Questions for authors
  1. How much would scope and dispersion change under a different expansion rule (no paragraph/table expansion, or a minimal-sufficient-evidence set), and would the RQ1/RQ2 conclusions survive?
  2. Why is scope positively associated with accuracy for Gemini 3.8 Flash? Is this a retrieval-strength artifact, an answerability confound, or a real difference in how that model handles broad evidence?
  3. Can the three attributes be computed without gold evidence at practical cost, so practitioners can apply this characterisation to their own unlabeled long-context data?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2fXM (1/3)

We thank the reviewer for the supportive review and for highlighting the motivation, quantitative attributes and value of our work. We provide our responses below.

Weakness 1 & Question 1:

We agree that scope and dispersion should ideally be calculated based on the minimum sufficient set of evidence. However, annotating the minimum supporting context for long-context reasoning tasks requires domain expertise and is time-consuming. To cover diverse data from multiple domains, we resort to (1) using the provided supporting evidence from existing datasets (HotpotQA and SciTrek, 2663 of the 6550 questions), and (2) constructing the supporting evidence based on Gemini-based retrieval for other source datasets. We need to expand retrieved evidence to ensure the constructed evidence is sufficient to produce correct answers, though it may introduce noise.

First, for about 41% of our dataset (HotpotQA and SciTrek), we use oracle supporting evidence (obtained from existing datasets). There is no evidence expansion effect, as it is based on a minimum sufficient set of evidence.

Second, for the rest of the data, our additional experiments show that our conclusion holds for different expansion strategies. We compare results on supporting evidence based on no expansion, our paragraph/table expansion, and further refinement with reasoning graphs. We approximate the minimum sufficient set of supporting context with evidence cited in reasoning graphs, though not all supporting evidence will always be cited in reasoning traces. We report additional results in the table below. Under all three evidence expansion strategies, scope, dispersion and depth each remain negatively associated with score (RQ1: interquartile contrasts between −0.71 and −2.14 pp, with every 95% CI excluding zero). The RQ2 patterns hold as well: the score drop from added length is 0.68–0.90 pp larger at high scope and 0.73–1.03 pp larger at high dispersion (all CIs exclude zero), while the depth interaction remains inconclusive, as in the paper.

Our expansion (copied from the paper)No expansionGraph-cited
RQ1: context length−1.66 [−2.49, −0.83]−1.62 [−2.44, −0.79]−1.56 [−2.40, −0.72]
RQ1: evidence scope−2.14 [−3.07, −1.20]−1.91 [−2.81, −1.01]−1.11 [−1.97, −0.26]
RQ1: evidence dispersion−1.10 [−1.84, −0.36]−1.39 [−2.09, −0.69]−0.71 [−1.37, −0.04]
RQ1: reasoning depth−1.60 [−2.35, −0.85]−1.62 [−2.37, −0.86]−1.73 [−2.48, −0.98]
RQ2: evidence scope−0.90 [−1.43, −0.37]−0.68 [−1.19, −0.18]−0.78 [−1.29, −0.28]
RQ2: evidence dispersion−0.88 [−1.45, −0.32]−1.03 [−1.55, −0.50]−0.73 [−1.23, −0.24]
RQ2: reasoning depth−0.66 [−1.43, 0.10]−0.60 [−1.37, 0.17]−0.72 [−1.49, 0.05]

Weakness 2

We agree that task identity explains more variance than context length and the three attributes. However, we believe this is expected and does not undermine the explanatory value of the attributes, for three reasons.

(1) LURCH comprises 17 benchmarks and tasks, designed to test different LLM capabilities across diverse difficulty levels. Consequently, task identity is expected to account for substantial variation. Our attributes, in contrast, provide additional explanatory information at the question level.

(2) The attribute effects cannot be task-level difficulty in disguise. Our GLM (Equation 4) includes task fixed effects, meaning that each β_k is estimated only from questions within the same task. Thus, the observed attribute effects cannot simply be attributed to differences in average difficulty across tasks.

(3) The relevant benchmark is context length. In Table 6, adding context length lowers cross-entropy from 0.3954 to 0.3951, while adding the three attributes lowers it further to 0.3928. The attributes therefore add roughly eight times the predictive value of context length. This demonstrates that the attributes provide additional predictive information beyond task identity and context length.

AuthorsOpenReview ↗

Response to Reviewer 2fXM (2/3)

Weakness 3

We agree with the points raised and are encouraged that the reviewer appreciates the steps already taken to address the challenge of combining different metrics into a single score. While task intercepts (αt\alpha_t) in Equation 4 absorb baseline differences, sharing attribute coefficients (βk\beta_k) across tasks does not capture differing guessing floors or score variances across metrics. To verify that pooling metrics does not drive our findings, we refitted the RQ1 GLM separately within each metric family, reporting within-task interquartile contrasts (β\beta per SD, 95% CIs):

Metric (questions, tasks)Length (ref.)ScopeDispersionDepth
All tasks (6,550, 17)−0.11 [−0.17, −0.06]−0.21 [−0.30, −0.12] −0.10 [−0.17, −0.03]−0.10 [−0.15, −0.05] 
F1 (5,515, 11)−0.08 [−0.14, −0.01] −0.21 [−0.31, −0.11] −0.10 [−0.17, −0.03] −0.12 [−0.17, −0.06] 
EM (659, 3)−0.08 [−0.27, 0.12] −0.09 [−0.39, 0.22] −0.12 [−0.35, 0.12] −0.06 [−0.24, 0.12] 
LLM judge, Loong (376, 3)−0.52 [−0.63, −0.41]−0.21 [−0.50, 0.08] −0.91 [−1.61, −0.21] +0.10 [0.00, 0.19] 

Within the homogeneous Token-F1 subset (11 tasks, 84% of questions), all three properties remain negative and statistically significant, closely matching the pooled estimates. In the much smaller Exact Match (3 tasks, n=659n=659) and LLM-as-judge (3 tasks, n=376n=376) subsets, confidence intervals widen due to sample size and metric granularity, though point estimates stay negative for length, scope, and dispersion (reaching significance for Loong dispersion).

Question 2

We isolate the cause to the SciTrek[1] benchmark, and dropping it from Figure 4 for Gemini 3.8 Flash gives a scope-dependent interquartile score contrast of −1.54 pp [−2.62, −0.45] down from +1.55 pp

+0.42,+2.68+0.42, +2.68

, which is in line with the other 3 LLMs. Dropping any one of the other 16 tasks leaves it positive (+1.50 to +2.70 pp). Further analysis revealed ambiguity in some of SciTrek’s low-scope questions responsible for incorrect answers. Note that SciTrek provides oracle evidence, so this is independent of our evidence retrieval pipeline.

Cause: SciTrek's low-scope questions need only the articles' titles and author lists. Many ask how many articles a person wrote (e.g. "How many articles has A authored?"). SciTrek counts only the provided full articles; however, the same names also appear in the articles' reference lists, so the prompt question allows a broader reading. Gemini often takes that reading and counts cited papers. For one question with gold answer 1, it answers 8, listing "the main article" plus "7 cited articles in its references". On the 74 author-count questions, Gemini overcounts in 33.8% of attempts, compared with 5.9–8.6% for the other LLMs. This lowers its accuracy on SciTrek's low-scope questions, which causes the positive relation.

Clarification experiment: To test this, we added one sentence to the 379 SciTrek questions that SciTrek itself marks as not requiring reference lists: Note: “article” means one of the full articles provided above; papers that appear only in reference lists do not count as articles. The articles, evidence and answers are unchanged. We reran all four LLMs with the paper settings:

Gemini 3.8 FlashGPT-5.6 LunaQwen3.8-27BQwen3.5-4B
Accuracy, 379 questions0.786 → 0.8740.819 → 0.8450.865 → 0.8780.848 → 0.858
Over-count rate, author-count questions33.8% → 2.7%7.2% → 2.7%8.6% → 4.1%5.9% → 3.2%
Scope contrast, all 17 tasks (pp)+1.55 → −0.47
−1.54,+0.61−1.54, +0.61

| −3.27 → −3.85 | −1.39 → −1.71 | −5.40 → −5.63 |

Overcounting falls for every LLM, and Gemini’s scope contrast is no longer positive. It does not reach the leave-SciTrek-out estimate, which also removes SciTrek’s other 621 questions. The other LLMs’ scope contrasts move by less than 0.6 pp. Gemini’s dispersion contrast shrinks from −2.19 to −1.47 pp, and its CI now overlaps those of the other LLMs. The reasoning-depth contrasts are unchanged.

In the revised paper: Our main analyses keep SciTrek’s original prompt questions, so that LURCH stays comparable with its source benchmark. Section 5.3 now states this cause where the Gemini contrast is reported, and Appendix H reports the experiment above with RQ1–RQ3 recomputed on the clarified questions.

[1] Li, Miao, et al. "SciTrek: Evaluating and Improving Long-Context Numerical Reasoning over Scientific Articles." TMLR. 

AuthorsOpenReview ↗

Response to Reviewer 2fXM (3/3)

Question 3

Yes. Gold evidence is not required, and supporting evidence can be obtained automatically through retrieval. Apart from HotpotQA and SciTrek, where the benchmark provides oracle evidence. For the other 3,887 questions, we computed all three attributes with the automatic pipeline as described in Sections 4.2 and 4.3. It needs only the question, the context and a reference answer, so it applies as is to any QA data that has answers but no evidence labels.

Cost is about one read of the context per question.

StepModelCalls per questionInput
Evidence retrievalGemini 2.5 ProOne pass over the chunked contextFull context
Sufficiency checkGemini 3.7 Flash3Expanded evidence only
Reasoning graphGemini 3.7 Flash1Expanded evidence and answer

The expanded evidence has a median of 0.93% of the context on these questions, so the last two steps are small. The cost is also paid only once per dataset, because the attributes describe the question and not the evaluated LLM.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Changes in the Revised Version

In response to the reviewers’ comments and questions, we have uploaded a revised version of our submission with changes highlighted in blue.

Below are the specific changes we made:

  1. We expanded the introduction and related work sections to discuss the suggested related papers. (Section 1, line 073-081; Section 2, line 145-162) [Reviewer froT, cbrh]

  2. We updated the analysis and experiments regarding Gemini 3.8 Flash. (Section 5.3, line 412-417, Appendix H, line 1404-1452) [Reviewer 2fXM, cbrh]

  3. We added ablation experiments on LLM reasoning effort. (Section 5.1, line 326-328; Appendix I, line 1454-1507) [Reviewer froT]

  4. We added ablation experiments and discussion on justifying our evidence expansion strategy. (Appendix J, line 1508-1570) [Reviewer 2fXM]

  5. We added ablation experiments on score metric sensitivity. (Appendix K, line 1572-1593) [Reviewer 2fXM]

AuthorsOpenReview ↗

Summary of reviews and rebuttals

We thank the reviewers and the Area Chair for their time. We are grateful to the reviewers for recognizing that the problem in our paper is well motivated and timely, the proposed properties are simple, intuitive and cheap to compute, and the constructed dataset is a useful resource for the LLM community.

We have carefully revised the paper and provided detailed responses to all reviewer comments. We appreciate that Reviewer froT confirmed that the expanded related work discussions, ablation on reasoning efforts and ablation on depth measures address their concerns, and consequently increased their score from 2 to 3. We briefly summarize how our responses and revisions address the main concerns.

  1. Related work extension (Reviewers froT and cbrh ). We have expanded our discussion of related work and added detailed comparisons with the suggested papers in Sections 1 and 2. Unlike these studies, our work measures all three properties at the instance level based on located evidence and independently estimates the effect of each property while controlling for the other two.
  2. Gemini 3.8 Flash anomaly (Reviewers 2fXM and cbrh). We have traced the anomaly down to an ambiguity issue in the question wording of the SciTrek benchmark, which leads Gemini to overcount the number of authors. We retain the original wording to ensure comparability with the source benchmark and add a clarification experiment in Appendix H, demonstrating that our conclusions remain valid when the wording is fixed to eliminate this ambiguity.
  3. LLM reasoning effort ablations (Reviewer froT). We have added ablation experiments evaluating two additional reasoning effort levels (low and high) for proprietary LLMs. The results (Appendix I) demonstrate that our conclusions remain consistent across all reasoning effort levels.
  4. Sensitivity to model performance metrics (Reviewer 2fXM). We have added separate analyses for each performance metric (EM, F1, and LLM-as-a-Judge). The results (Appendix K) confirm that our main conclusions remain unchanged when using each metric independently.
  5. Definition of dispersion (Reviewer cbrh). We conducted additional experiments using three alternative dispersion measures based on the number of evidence items and the positions of all evidence items: count, position spread, and uniformity. The results show that all three alternative measures exhibit a similarly negative relationship with model performance, leaving our main conclusions unchanged.

We sincerely thank all reviewers once again for their thoughtful and constructive feedback, which has helped us substantially improve the quality of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.