- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper explores reasoning task difficulties over long contexts. To further analyze the difficulty measure, the authors provide 3 properties beyond context length, namely: scope, dispersion and reasoning graph/depth. The authors create a multidoman question answering benchmark called LURCH. Experiments show that alongside context length, these 3 properties are associated with model performance and difficulty estimation.
Critical strengths
- The paper has a good motivation and addresses a relevant problem.
- The properties are simple, intuitive and easy to interpret.
- The authors experiment with 4 LLMs over multiple domains and datasets
Critical weaknesses
- The dispersion metric considers only the first and last evidence to calculate the dispersion (from eqn. 3). It does not consider whether there are evidences in between the first and last. This can be a potential issue because answers requiring multiple evidences should be more difficult, but this metric might fail to judge it properly.
- The impact of Evidence Dispersion and Reasoning Depth seems minimal. (from Fig.5,6)
- The comparison with many existing relevant studies is missing. This would help clarify the differences and contributions of the benchmark.
Questions for authors
- Is it possible for the dispersion metric to account for the number of evidences and the spread of it, instead of only the first and last evidence?
- Can the authors explain or expand on the behavior of Gemini 3.8 Flash, which is unlike the other evaluated models?
- Can the authors compare LURCH with relevant benchmarks: [1] and [2]?
References:
- Yang, Zi. "Retrieval or holistic understanding? dolce: Differentiate our long context evaluation tasks." arXiv preprint arXiv:2409.06338 (2024).
- Taewhoo Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, and Jaewoo Kang. 2025. ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5497–5512, Albuquerque, New Mexico. Association for Computational Linguistics.
4 comments
Response to Reviewer cbrh (2/2)
Weakness 3/Question 3
We thank the reviewer for these references; the revised Related Work now discusses both. DOLCE [1] and ETHIC [2] each quantify how much of the context a question needs. LURCH differs in three ways:
- Located evidence. DOLCE fits its parameters to a probing LLM’s answers on randomly sampled spans, and ETHIC counts the chunks GPT-4o judges necessary. LURCH computes scope and dispersion from the token positions of evidence that is benchmark-provided, or retrieved and verified as sufficient.
- Amount, position and depth are separated. DOLCE’s λ is closest to our evidence span H(E), which we split into scope and dispersion. Neither work assigns a reasoning-depth score.
- Linked to scores within tasks. Those two works stop at a task categorisation (DOLCE) and a new benchmark (ETHIC). LURCH estimates how each property relates to scores at fixed task, length and LLM (RQ1–RQ3).
[1] Yang, Zi. "Retrieval or holistic understanding? dolce: Differentiate our long context evaluation tasks." arXiv preprint arXiv:2409.06338 (2024).
[2] Lee, Taewhoo, et al. "Ethic: Evaluating large language models on long-context tasks with high information coverage." NAACL. 2025.
Question 2
We isolate the cause to the SciTrek[1] benchmark, and dropping it from Figure 4 for Gemini 3.8 Flash gives a scope-dependent interquartile score contrast of −1.54 pp [−2.62, −0.45], down from +1.55 pp
, which is in line with the other 3 LLMs. Dropping any one of the other 16 tasks leaves it positive (+1.50 to +2.70 pp). Further analysis revealed ambiguity in some of SciTrek’s low-scope questions responsible for incorrect answers. Note that SciTrek provides evidence data, so this is independent of our evidence retrieval pipeline.
Cause: SciTrek's low-scope questions need only the articles' titles and author lists. Many ask how many articles a person wrote (e.g. "How many articles has A authored?"). SciTrek counts only the provided full articles; however, the same names also appear in the articles' reference lists, so the prompt question allows a broader reading. Gemini often takes that reading and counts cited papers. For one question with gold answer 1, it answers 8, listing "the main article" plus "7 cited articles in its references". On the 74 author-count questions, Gemini overcounts in 33.8% of attempts, compared with 5.9–8.6% for the other LLMs. This lowers its accuracy on SciTrek's low-scope questions, which causes the positive relation.
Clarification experiment: To test this, we added one sentence to the 379 SciTrek questions that SciTrek itself marks as not requiring reference lists: Note: “article” means one of the full articles provided above; papers that appear only in reference lists do not count as articles. The articles, evidence and answers are unchanged. We reran all four LLMs with the paper settings:
| Gemini 3.8 Flash | GPT-5.6 Luna | Qwen3.8-27B | Qwen3.5-4B | |
|---|---|---|---|---|
| Accuracy, 379 questions | 0.786 → 0.874 | 0.819 → 0.845 | 0.865 → 0.878 | 0.848 → 0.858 |
| Over-count rate, author-count questions | 33.8% → 2.7% | 7.2% → 2.7% | 8.6% → 4.1% | 5.9% → 3.2% |
| Scope Δ25→75, all 17 tasks (pp) | +1.55 → −0.47 |
| −3.27 → −3.85 | −1.39 → −1.71 | −5.40 → −5.63 |
Overcounting falls for every LLM, and Gemini’s scope contrast is no longer positive. It does not reach the leave-SciTrek-out estimate, which also removes SciTrek’s other 621 questions. The other LLMs’ scope contrasts move by less than 0.6 pp. Gemini’s dispersion contrast shrinks from −2.19 to −1.47 pp, and its CI now overlaps those of the other LLMs. The reasoning-depth contrasts are unchanged.
In the revised paper: Our main analyses keep SciTrek’s original prompt questions, so that LURCH stays comparable with its source benchmark. Section 5.3 now states this cause where the Gemini contrast is reported, and Appendix H reports the experiment above with RQ1–RQ3 recomputed on the clarified questions.
[1] Li, Miao, et al. "SciTrek: Evaluating and Improving Long-Context Numerical Reasoning over Scientific Articles." TMLR.
Dear Reviewer,
We have carefully addressed your concerns and responded to all of your questions. Should you have any further questions or require additional clarification, we would be more than happy to address them.
Best regards,
The Authors
Dear Reviewer cbrh,
Thank you again for your time and thoughtful feedback on our paper. We have carefully revised the manuscript highlighted in blue in the paper and provided detailed responses to each of your comments. We hope our revisions, clarifications and additional experiments have satisfactorily addressed your concerns.
Thank you for your consideration.
Response to Reviewer cbrh (1/2)
We thank the reviewer for the thoughtful review, and for finding the motivation strong and the properties simple and interpretable. We address the concerns and answer the questions below.
Weakness 1/Question 1
We agree that Eq. 3 summarises outer evidence span (subtracting total evidence length) rather than explicitly modeling intermediate segments. We also considered these alternative definitions for dispersion: (i) Uniformity compares the evidence layout with N evenly spaced points using Wasserstein-1 distance W1, which here is the mean distance between the sorted segment centres and the sorted reference points, as a fraction of context length. Uniformity is 1−2⋅W1, which runs from 0 (all evidence at one end) to 1 (evenly spaced). (ii) Count is the number of evidence items. (iii) Position spread is the standard deviation of the positions of all evidence tokens, normalized by context length, with each token weighted equally.
All three alternatives show the same negative relationship with score, and our scope and reasoning-depth estimates remain unchanged across every specification. Using all token positions (Position spread) or segment counts yields contrasts nearly identical to our dispersion metric, while Uniformity shows a weaker effect. We therefore apply the simpler definition in Section 3.
Weakness 2
We thank the reviewer for raising this point. We disagree that the effects of dispersion and reasoning depth are minimal and further clarify our claims and results. Our central claim is that context length alone cannot fully capture long-context reasoning difficulty, and our results show that scope, dispersion, and depth add significant explanatory signal beyond length (χ32=73.0,padj=4.0×10−15). Because baseline accuracy in LURCH is high (~85%), context length itself shifts scores by only −1.66 pp from Q25 to Q75 (Table 1). Against this benchmark, reasoning depth produces a comparable interquartile drop overall (−1.60 pp) and a larger drop (−3.02pp) across the 9 tasks with untied depth quartiles (Table 7), while doubling depth incurs more than twice the penalty of doubling length (−2.70 pp vs. −1.04 pp). Crucially, these effects also intensify on longer inputs and when multiple demands coincide. Figure 5 and Table 2 show that the impact of dispersion grows with context length: moving from Q25 to Q75 in length nearly doubles the score drop under high dispersion (−2.01 pp vs. −1.13 pp at low dispersion; interaction −0.88 pp,padj=0.01), and similarly widens the length drop at high depth (−1.82 pp vs. −1.16 pp). In Figure 6, combining high dispersion with high scope or depth creates 6.5 pp to 10.0 pp spreads across difficulty regimes.