acceptodds
Under review as a conference paper at ICLR 2027

StructScope: Can LLMs Maintain Structural Scope in Long Contexts?

Abstract

The rapid growth of structured information across long contexts, such as scientific documents, software artifacts, and harness-managed contexts creates a need for models that can determine not only which information is relevant but also which structural scope makes it admissible. However, existing long-context benchmarks primarily evaluate models' capability to process text information with explicit structured information as fixed background. To bridge this gap, we introduce , a novel benchmark designed to evaluate models' capability in , i.e., applying query-specified scope constraints when selecting and attributing information. Although models show strong structural reasoning when evaluated on compact structural parse trees, our experiments reveal that their performance degrades sharply as the same structures are embedded within longer contexts. Further fine-grained experiments identify scope-constrained selection, particularly within nested structures, as the key bottleneck. Our findings highlight structural scope maintenance as an important dimension of long-context capability, motivating dedicated evaluation, training, and scope-aware context management in harness systems.

Primary area:
datasets and benchmarks
Keywords:
large language models, long-context evaluation, benchmarks, structured documents, structural scope maintenance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CHsSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StructScope, a benchmark for testing whether LLMs can follow structural scope constraints in long contexts. The benchmark uses LaTeX sources from 64 arXiv papers. It extracts heading ownership and reference relations and creates five tasks. These tasks cover scope enumeration, nested exclusion, object retrieval, path attribution, and combinations of these operations.

The paper tests the same structural questions under different context settings, including original LaTeX, length-controlled skeletons, compact parse representations, and parse-assisted contexts. The main finding is that many models perform very well when the structure is compact, but their performance drops when the same structure is distributed across a long document.

Critical strengths
  1. The problem is important and practical. In long documents and agent contexts, the model must not only find relevant information. It must also decide whether the information belongs to the requested scope. This is a useful problem to study.

  2. The benchmark has deterministic answers and scoring. It does not use an LLM judge. The Skeleton-12K, Skeleton-32K, and Skeleton-90K settings give a relatively clean comparison because the structure and answer remain the same while the amount of text changes. I also like that the benchmark can be generated again from new papers.

Critical weaknesses
  1. The main failure may partly be exhaustive retrieval instead of scope maintenance. RA remains strong at long context length, while EN and SE become much worse. SE needs the model to recover two reference sets before applying the exclusion. For some strong models, the SE score is close to what may be expected from combining two EN errors. Therefore, the paper does not fully show that nested exclusion itself is the main problem.

  2. Some comparisons change several factors at the same time. Parse-Only and Original differ in length, representation format, information density, and locality. A length-matched Parse-Only condition is missing. Also, some models have many truncated or unparseable outputs. In these cases, a very low exact-match score does not mean that the model has no structural ability.

Questions for authors
  1. Can the authors report SE accuracy only on examples where the positive-scope and negative-scope reference sets are both recovered correctly? This may show whether the exclusion step itself causes extra errors.

  2. Can the authors add a long Parse-Only condition and a short LaTeX condition? This would help separate context length from representation format and locality.

  3. Can the authors reduce truncation by increasing the output budget and report set exact match, ID recall, and ID-plus-path recall together with query exact match in the main paper?

  4. How often does the benchmark parser give a different result from a full LaTeX engine, for example because of unsupported environments or multi-target references?

1 comment
AuthorsOpenReview ↗

Response to Reviewer CHsS

Thank you for recognizing the importance of scope maintenance and our controlled evaluation design; below, we report additional analyses of retrieval and exclusion errors, context controls, and parser reliability.

1. Conditional SE accuracy (Q1).

Our SE protocol requests the final selected set rather than requiring separate outputs for the positive- and negative-scope reference sets. However, some responses explicitly list both sets in their written reasoning. We conducted a post-hoc analysis of these responses, checking each stated set against the gold and excluding cases with conflicting assignments. For the 102 SE queries in Original-90K, the results are

ModelResponses with both sets explicitly verifiableBoth sets correctFinal answer correct among
Qwen3.8 27B75/1024545/45 (100%)
Gemma-4 31B45/10287/8 (87.5%)

These results support the reviewer’s distinction: within the observable subset, correctly recovering the operand sets is generally followed by successful exclusion. We therefore do not interpret the overall SE decline as evidence that set subtraction itself is the principal bottleneck. Because intermediate sets were not required, these statistics describe a selective subset of responses, rather than a full-dataset estimate of conditional accuracy.

2. Short LaTeX and long-context parsing controls (Q2).

We added Skeleton-Only, a short LaTeX condition retaining heading levels, object definitions, and references while removing plain text. We also report length-matched Long-Parse results.

ConditionMedian input tokensQwen3.8 27B accuracyGemma-4 31B accuracy
Parse-Only5,572100%100%
Skeleton-Only5,72085.3%92.2%
Long-Parse82,04097.1%95.1%

3. Output budget and partial-credit metrics (Q3).

We increased the output budget from 16K to 32K tokens, but observed only a small reduction in truncation. Most truncated responses entered repetitive enumeration or sentence-generation loops and continued until reaching the higher limit. Increasing the budget therefore did not substantially resolve these failures. We will report set exact match, ID recall, and ID-plus-path recall alongside query exact match in the main paper, together with truncation and unparseable-output rates.

4. Parser reliability and engine-backed validation (Q4).

Our source-based parser is not equivalent to LaTeX execution, and discrepancies can affect the benchmark’s gold answers. We therefore examined both specific unsupported constructs and engine-executed structural relations. Across all 470 queries, supplementing our handling of multi-target references changes the gold object sets for 12 queries (2.6%). We additionally implemented an engine-backed audit. In paper 2503.20607, it confirmed two omitted reference relations arising from multi-target commands, changing the gold sets for three of its four queries. In another paper, an exclusion heading appears inside an inactive \iffalse branch: it is recognized by the source parser but not executed by the engine.

Reviewer_dTwfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for evaluating LLM's ablity in maintaining structure in documents. Results demonstrate that existing LLMs have no problem in understanding the given structures but get confused when structures are hidden under long contexts.

Critical strengths
  1. The paper raise a new problem of structure maintainance in long-context, different from other problems in long-context.
  2. The benchmark is curated from real-world latex data.
  3. By comparing performance in different context settings, the results from the benchmark demonstrate that the shortage of existing models in structure maintainance in long contexts.
Critical weaknesses
  1. The data source is restricted in latex of scientific papers which have clear and simple structures.
  2. The benchmark setting iis impractical where using tools is not allowed. I think the problem can be much simpler using coding-based searching.
  3. The proposed settings/problems are of little real-world value, for me, I rarely ask similar questions in the benchmark.
Questions for authors
  1. Why SOTA coding model like Claude Opus 5 achieve only 0 in Original-90K setting.
  2. What's the difference between the proposed benchmark and existing long-context benchmarks in evaluating the ablities of LLMs under long-context?
3 comments
AuthorsOpenReview ↗

Response to Reviewer dTwf (Part1/2)

Thank you for recognizing structural scope maintenance as a distinct long-context challenge; below, we address your concerns about practical relevance and data scope, clarify our differences from existing benchmarks, and report additional experiments investigating Opus 5’s zero score.

1. Why does Opus 5 achieve zero accuracy in Original-90K? (Q1).

Our follow-up experiment indicates that the original zero score is highly related to the reasoning configuration (non-thinking for all the models). On the same 50 Original-90K queries, increasing thinking effort raised query exact-match accuracy from 0/50 to 33/50 (66%). The observed reasoning traces also increased from typically below 1K tokens to approximately 4K tokens. However, increased thinking effort does not fully resolve the difficulty: Opus still fails to produce a completely correct answer on 17 of the 50 queries. Different from Opus 5, the other models still use CoT in the non-thinking mode. We acknowledge that our original presentation did not adequately distinguish reasoning configurations across models. We will report both Opus configurations in the main results, together with their reasoning effort settings and total output lengths.

2. Differences from existing long-context benchmarks (Q2).

StructScope differs from existing benchmarks in both how task-relevant structure is provided and how adherence to that structure is evaluated. The Michelangelo benchmark uses Latent Structure Queries to assess whether models can identify and reason over task-relevant latent structure by interpreting contextual content. StructScope instead makes the strcture explicit through heading-level and reference markers. This reduces the need to infer the underlying structure and focuses evaluation on whether models consistently apply its scope and ownership constraints across a long context.

Broader benchmarks such as LongBench v2 evaluate performance through downstream task answers. Structural scope handling may contribute to these answers, but it is not separately scored. An incorrect answer may result from selecting information from an inapplicable scope or from reasoning incorrectly over correctly selected information; final-answer accuracy alone does not distinguish these causes. StructScope instead makes the results of structural operations themselves the scored outputs. For example, including an object from an excluded branch or returning an incorrect ancestor path directly violates an evaluated constraint. Our contribution is to make structural requirements explicit and verifiable, complementing existing evaluations of long-context reasoning.

AuthorsOpenReview ↗

Response to Reviewer dTwf (Part2/2)

3. Simple structures and data scope (W1).

StructScope evaluates whether models consistently apply explicitly specified structure scope across long contexts. The scope-maintenance challenge is exists regardless of structural complexity: even a shallow, explicitly marked structure requires the model to preserve the scope bundaries. Meanwhile, simple, clear structure also occurs in established agent benchmarks. For example, τ-bench’s retail policy uses a three-level Markdown strucutre. Structural complexity is therefore not a prerequisite for practical relevance. We acknowledge that using only LaTeX papers limits source diversity and extending evaluation to other structure formats remains future work.

4. Practical relevance of the setting without tools (W2).

Our tool-free setting is a practical requirment that also arises within practical tool-assisted setting: maintaining the scope and provenance of information already in context, which the model must interpret before invoking a tool. We illustrate this connection through two examples and have added Fig.10 of two scenarios to the appendix~A.4.

First, when system instructions and user-provided content appear within the same context, the model needs to distinguish their origins to determine which instructions apply. If the user-provided content contains conflicting instructions to system instructions, preserving these source boundaries is necessary for applying instruction priorities, including when deciding whether and how to invoke tools.

Second, when combining results from multiple tool calls, the model needs to preserve the association between each result and its originating call. For example, confusing search results for two different files could cause it to use information from one file when deciding how to modify the other. In both situations, subsequent tool actions depend on correctly interpreting information already received.

These examples illustrate the practical relevance of maintaining information boundaries and provenance. We agree that coding-based parsing and search can simplify our benchmark. But excluding these tools allows us to directly assess whether the model itself can maintain the specified scope and provenance relations across long contexts. The examples above illustrate why tool-free setting is practical and this ability matters in practical tool-assisted applications.

5. Real-world value of the proposed settings and questions (W3).

Users may not explicitly ask these questions, but their requests still require the underlying judgments. For example, “Update this plugin” requires an agent to distinguish Update instructions from Create instructions, including when a tool returns both branches. Incorrect scope handling can then affect subsequent tool calls. A user may also reasonably request these judgments explicitly:

“Before updating the plugin, list the guides referenced by the Update instructions and their full source paths so I can verify what you will use.”

Our probes make such intermediate requirements observable and verifiable. Their practical value lies in diagnosing operations needed for reliable task execution, rather than in how frequently users phrase requests in the benchmark’s exact format.

Thank you for the detailed responses and additional experiments. Overall, the rebuttal has addressed several of my concerns and improved my assessment of the paper. I am willing to raise my rating from 2 to 3, while maintaining some reservations regarding the limited structural complexity and difficulty diversity.

Reviewer_dJUWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes structural scope maintenance (e.g., understanding structures in long context) as a key challenge for agents and thus needs a benchmark. The paper thus extracts arxiv papers, parse their structures, put the paper into agents' context to mimic structured long context, and curate low-level structure-related questions (e.g., which citation appears under Section A while not appears under Section B). Evaluation results on open-source models show low accuracy.

Critical strengths

Comprehensive evaluation across different settings and models. An automated benchmark evaluation pipeline for easy evaluation.

Critical weaknesses
  • The benchmark doesn't reflect realistic long-context scope maintenance challenges, and instead using document structure-related QA as a proxy.
  • The benchmark's questions are kind of bespoke, which seem to barely exist in real-world applications.
Questions for authors

Same as weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dJUW

Thank you for recognizing our comprehensive evaluation and automated evaluation pipeline.

1. How our tasks capture realistic scope-maintenance challenges (W1).

StructScope directly evaluates structural operations required within realistic workflows by making them explicit and verifiable. We acknowledge that the current paper, including Fig.1, may not explain this correspondence sufficiently clearly. We have added an expanded walkthrough of Fig.1 in Appendix~A.4, illustrated in Fig.9. Below, we explain these operations and how our tasks evaluate their scope-maintenance requirements.

Operation A: Reading within an enclosing scope. To update an existing plugin, the model first consults the relevant skill content within a larger collection of instructions organized into sections. The plugin-related cue directs it to ``Plugin Management,'' but locating that heading does not establish which subsequent content belongs to it. The model must read the associated instructions and resource references while maintaining the boundary of this section, including its nested content and stopping before neighbouring sections. The EN task makes this requirement directly testable: it asks the model to enumerate the objects referenced within a specified heading subtree. References included from outside that subtree, or omitted from within it, become observable errors in the returned set.

Operation B: Reading selectively within nested scopes. Within "Plugin Management'', completing an update requires retaining applicable general plugin-related requirements together with update-specific instructions, while excluding other content such as creation-specific instructions. Finding the heading "Update'' alone is therefore insufficient: the model must determine which parts of the enclosing material remain applicable and which belong to the excluded "Create'' branch. This requires maintaining both an enclosing boundary and a nested exclusion boundary as it reads. The SE task explicitly tests this operation by asking the model to select objects referenced within a parent scope while excluding those referenced within a designated descendant scope. Its output directly exposes failures to apply the specified exclusion.

Operations C and D: Retrieving referenced resources and recovering their provenance. After reading an instruction such as ``Follow updating.md'' the model must locate the intended resource and retain its source hierarchy when using its contents. The paths ".codex/ updating.md" and "docs/updating.md" in Fig.1 illustrate why the filename alone is insufficient to establish the resource's identity. Information associated with the wrong enclosing path can be attributed to the wrong source, even when its contents have been successfully retrieved. The RA task tests the corresponding operation by requiring the model to locate referenced object definitions and return their complete ownership paths. Missing ancestors or incorrect branch assignments are thus directly visible in the answer.

StructScope therefore turns the above scope-maintenance challenges within realistic applications into directly verifiable tasks. The document-based questions provide a controlled way to observe these operations, without requiring a downstream execution failure to reveal that a structural constraint was violated.

2. Bespoke questions and practical use (W2).

We agree that our questions are purpose-built, but the judgments they require also arise within real workflows, often as implicit intermediate steps. A user may simply request "Update this plugin'' leaving the model to identify applicable instructions, follow their references, and track resource provenance. The user can also reasonably make these requirements explicit:

"Before updating this plugin, list the guides referenced by the Update instructions and provide their full source paths so I can verify the resources you will use.''

This directly resembles our enumeration-and-attribution task. StructScope standardizes such requirements into controlled, verifiable probes; their diagnostic value comes from exposing operations needed in practical tasks, even when users do not ask about each operation separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.