acceptodds
Under review as a conference paper at ICLR 2027

Scenario Collapse: Measuring Scenario Local Evidence Binding in Long Context Reasoning

Abstract

Large language models can retrieve the right evidence from a long context yet apply its value to the wrong case or document version. We call this failure scenario local binding. We introduce ScenarioCollapseBench, a paired scenario benchmark with 10,132 scenario pairs and deterministic evidence, state, and decision traces, together with found-but-not-bound (FBNB), which measures neighboring scenario value substitution after successful evidence localization. Across ten models, substantial binding errors remain even after the correct evidence is found. The same failure appears in 200 naturally occurring historical versions of UK legislation and paired adaptations of RULER and BABILong. A matched Qwen3-14B thinking ablation shows that additional reasoning reduces, but does not eliminate, the problem. Oracle experiments locate the dominant residual error in value/state construction rather than evidence access. Finally, making scenario local state explicit through scenario specific identifiers, state tables, and verification substantially reduces binding errors and collapse. These results identify scenario-local binding as a distinct reasoning problem and provide a practical design principle for systems that reason across similar cases, versions, or branches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Long Context, Scenario Collapse, Evidence Value Binding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NkapOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I believe this paper should be desk-rejected for the reasons I laid out in my official comment and documented in my AI documentation note. But, my understanding of the paper is that the authors have identified a gap in long-context reasoning evaluation. LLMs can retrieve correct evidence, but may apply it to incorrect cases or document versions (here they are dealing specifically with tasks having multiple documents or "scenarios"). They introduce a benchmark SCB and metric FBNB on which they evaluate 10 LLMs, finding varying performance. I also have opinionated critiques of the writing style, structure and prioritization of dense paragraphs over providing intuition or insightful results.

Critical strengths
  • Authors point out a genuine gap in long-context reasoning evaluations.
Critical weaknesses
  • Violation of the ICLR style guidelines, as documented in my Official Comment: https://openreview.net/forum?id=jwevBT2MGb&noteId=WCNzl2kSsx
  • The authors prioritize providing long, dense paragraphs over giving readers intuition or elevating experiments that would clearly illustrate novel findings rather than redundant, muddled figures like Figure 1:
    • I'd suggest that the authors provide more figures in their paper, for results but also the example from the second paragraph of their introduction could have been made into a motivational figure to provide the reader with better intuition for the gap the authors are pointing out.
    • Table 1 comes across as messy and an "info-dump" rather than a justifiable simplification of the related work on similar benchmarks and how the authors' proposed benchmark is novel, which is what such a table should be aiming to be.
    • Examples would have been nice to include in section 3.2
  • Writing clarity is not up to ICLR standards:
    • A high number of "clause :/; clause" sentences which I associate with LLM-generations. I know the authors disclose the use of AI writing assistants, but I'd encourage them to go through the paper and simplify things and make sure they expressing their message in their own words as much as possible.
    • Re: Section 2. I'm not sure why this isn't just titled "Related Work" in keeping with standard practices

Frankly, I stopped reading after completing section 3. I did the style guide violation investigation after I read section 2. Then, I skimmed the results and later sections, but, due to what I believe to be a desk-reject-worthy violation and the clear use (in my opinion) of minimally edited LLM-generated prose, I did not provide my full effort to critique the paper starting from Section 4. If the paper had just one of: a violation or LLM-generated writing, I would have felt obliged to provide the authors with a more meaningful review to aid them in resubmitting. As is, I frankly felt it was not worth my time.

Questions for authors

Why did you edit the PDF after LaTeX compilation?

Reviewer_6aABOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies failures to bind values to the correct scenario despite citing the correct evidence. It introduces 10,132 scenario pairs, FBNB, and conditional binding metrics. Experiments include natural legal versions, paired benchmark adaptations, and scenario-separation interventions.

Critical strengths

Section 3 provides deterministic evidence, value, state, and decision traces for clear diagnostic evaluation. Appendices D/E/L include independently validated human-written items, natural legal versions, and transfer tests. Tables 4–5 and 11 provide informative oracle, matched-thinking, and structural interventions.

Critical weaknesses
  1. Equation (6) matches errors to neighboring values, but Appendix J also places that value in a local stale memo, confounding cross-scenario attribution.
  2. The authors should quantify local value overlap and check this measurement issue on source-disambiguated items.
  3. Table 3 averages to 55.63% decision accuracy across ten models, whereas sample-weighted Table 12 yields 63.30%, requiring evaluation-set clarification and reaggregation.
Questions for authors
  1. How often does a neighboring gold value also occur in the target scenario’s stale records or distractors?
  2. Do existing items without this source ambiguity exhibit the same errors?
  3. Which instances and weights underlie Tables 3 and 12, and how do they explain the 55.63% versus 63.30% discrepancy?
Reviewer_tMobOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCENARIOCOLLAPSEBENCH, a paired-scenario benchmark that isolates and measures a specific long-context reasoning failure: after correctly localizing evidence within a long input, models often fail to bind the correct value to the correct “local world” (scenario). The authors formalize this problem via field-level and pair-level metrics, including a post-localization binding rate (CondBind) and a found-but-not-bound measure that captures neighboring-scenario value substitution conditional on successful localization. In 10,132 paired instances, it shows significantly residual post-localization binding errors; oracle studies have shown the difficulty related more to value/state construction than accessing evidence, and structural changes (scenario-specific identifiers, state tables, and parser-based validation) can compensate for this problem. The phenomenon replicates on 200 naturally occurring legal version pairs and on paired adaptations of RULER and BABILong.

Critical strengths
  • This paper has clearly identified a new error category, that of scenario-local binding, in which the ability to localize relevant evidence and the ability to keep that evidence tied to the correct local scenario is both independent of its associated localization.
  • Well-defined, auditable metrics (CondBind, FBNB, CS-FBNB) distinguish retrieval/localization from binding, enabling precise diagnosis beyond typical citation- or span-based evaluations.
  • The notion of “scenario collapse” and scoring pairs provide a measurable behavioral signature of failures in multi-world maintenance.
Critical weaknesses
  • Requiring models to provide scenario-specific evidence IDs may implicate localization in the ability to follow and format schemas; although normalization and missing-value handling are described, residual conflation risk remains.
  • The paired design reuses local identifiers across scenarios, which is realistic in many ways (e.g., section numbers in a statute, repeating log/event IDs), but it may amplify one problem relative to other sources of scope entanglement.
  • There is limited exploration of how the location of evidence and context may affect binding errors (e.g., “lost in the middle” vs retrieval).
  • Author provides few breakdowns per field or task family in the main text to determine which attributes or decision schemas are more binding-prone.
  • Additional discussion linking to recent mechanistic work on how thinking/CoT alters retrieval and state representation (e.g., targeted vs broad retrieval regimes) could help contextualize the Qwen thinking ablation.
Questions for authors
  • Do binding errors correlate with the relative position of decisive evidence (e.g., early vs. middle vs. late) after conditioning on successful localization? A stratified analysis could separate residual position effects from pure binding failures.
  • Can you provide a per-attribute or per-task-family breakdown of CondBind and FBNB to identify which fields (e.g., numeric limits vs. categorical flags) most often leak across scenarios?
  • In the human-written tier, do binding errors cluster in instances with higher lexical overlap between scenarios (e.g., near-identical paraphrases) compared to ones with more stylistic divergence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.