- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces StructScope, a benchmark for testing whether LLMs can follow structural scope constraints in long contexts. The benchmark uses LaTeX sources from 64 arXiv papers. It extracts heading ownership and reference relations and creates five tasks. These tasks cover scope enumeration, nested exclusion, object retrieval, path attribution, and combinations of these operations.
The paper tests the same structural questions under different context settings, including original LaTeX, length-controlled skeletons, compact parse representations, and parse-assisted contexts. The main finding is that many models perform very well when the structure is compact, but their performance drops when the same structure is distributed across a long document.
Critical strengths
-
The problem is important and practical. In long documents and agent contexts, the model must not only find relevant information. It must also decide whether the information belongs to the requested scope. This is a useful problem to study.
-
The benchmark has deterministic answers and scoring. It does not use an LLM judge. The Skeleton-12K, Skeleton-32K, and Skeleton-90K settings give a relatively clean comparison because the structure and answer remain the same while the amount of text changes. I also like that the benchmark can be generated again from new papers.
Critical weaknesses
-
The main failure may partly be exhaustive retrieval instead of scope maintenance. RA remains strong at long context length, while EN and SE become much worse. SE needs the model to recover two reference sets before applying the exclusion. For some strong models, the SE score is close to what may be expected from combining two EN errors. Therefore, the paper does not fully show that nested exclusion itself is the main problem.
-
Some comparisons change several factors at the same time. Parse-Only and Original differ in length, representation format, information density, and locality. A length-matched Parse-Only condition is missing. Also, some models have many truncated or unparseable outputs. In these cases, a very low exact-match score does not mean that the model has no structural ability.
Questions for authors
-
Can the authors report SE accuracy only on examples where the positive-scope and negative-scope reference sets are both recovered correctly? This may show whether the exclusion step itself causes extra errors.
-
Can the authors add a long Parse-Only condition and a short LaTeX condition? This would help separate context length from representation format and locality.
-
Can the authors reduce truncation by increasing the output budget and report set exact match, ID recall, and ID-plus-path recall together with query exact match in the main paper?
-
How often does the benchmark parser give a different result from a full LaTeX engine, for example because of unsupported environments or multi-target references?
Response to Reviewer CHsS
Thank you for recognizing the importance of scope maintenance and our controlled evaluation design; below, we report additional analyses of retrieval and exclusion errors, context controls, and parser reliability.
1. Conditional SE accuracy (Q1).
Our SE protocol requests the final selected set rather than requiring separate outputs for the positive- and negative-scope reference sets. However, some responses explicitly list both sets in their written reasoning. We conducted a post-hoc analysis of these responses, checking each stated set against the gold and excluding cases with conflicting assignments. For the 102 SE queries in Original-90K, the results are
These results support the reviewer’s distinction: within the observable subset, correctly recovering the operand sets is generally followed by successful exclusion. We therefore do not interpret the overall SE decline as evidence that set subtraction itself is the principal bottleneck. Because intermediate sets were not required, these statistics describe a selective subset of responses, rather than a full-dataset estimate of conditional accuracy.
2. Short LaTeX and long-context parsing controls (Q2).
We added Skeleton-Only, a short LaTeX condition retaining heading levels, object definitions, and references while removing plain text. We also report length-matched Long-Parse results.
3. Output budget and partial-credit metrics (Q3).
We increased the output budget from 16K to 32K tokens, but observed only a small reduction in truncation. Most truncated responses entered repetitive enumeration or sentence-generation loops and continued until reaching the higher limit. Increasing the budget therefore did not substantially resolve these failures. We will report set exact match, ID recall, and ID-plus-path recall alongside query exact match in the main paper, together with truncation and unparseable-output rates.
4. Parser reliability and engine-backed validation (Q4).
Our source-based parser is not equivalent to LaTeX execution, and discrepancies can affect the benchmark’s gold answers. We therefore examined both specific unsupported constructs and engine-executed structural relations. Across all 470 queries, supplementing our handling of multi-target references changes the gold object sets for 12 queries (2.6%). We additionally implemented an engine-backed audit. In paper 2503.20607, it confirmed two omitted reference relations arising from multi-target commands, changing the gold sets for three of its four queries. In another paper, an exclusion heading appears inside an inactive \iffalse branch: it is recognized by the source parser but not executed by the engine.