- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ISCF for converting arbitrarily long unstructured documents into hierarchical trees under limited LLM context. It incrementally updates a global tree using DBR, adaptive masking with HAPE, and unique content ID mapping. Experiments across four domains report improvements over several baselines in accuracy, robustness, and efficiency. However, the supplied implementation and evaluation materials do not fully align with the described method and metrics.
Critical strengths
The paper addresses an important and practical problem by preserving document hierarchy and content fidelity when the input substantially exceeds the LLM context window. Its decomposition into boundary selection, compressed global state, and deterministic content recovery is intuitive and could be useful if implemented and validated as described.
Critical weaknesses
- The supplied implementation does not adequately support two central claims. The DBR workflow does not clearly provide the per node role probability distributions required by Section 2.2, making the reported threshold and activation behavior difficult to verify. The content recovery mechanism also appears to associate spans with structural paths rather than unique content IDs, leaving ambiguity when multiple content nodes share the same path.
- The reported experimental evidence is difficult to reproduce and does not fully align with the supplied evaluation materials. The evaluator does not appear to implement the stated tree edit distance, macro role F1, or Reconstruction Accuracy consistently. The supplement also lacks the claimed datasets, gold annotations, baseline implementations, and experiment scripts. In addition, the reported document lengths, chunk sizes, and average number of model calls are difficult to reconcile.
- Some concepts in this work are conceptually close to existing methods and require clarification in the related work section. Seg2Act (Li et al., EMNLP 2024) formulates document logical structuring as an iterative action generation task that maintains a global context and logical structure. PartPrompt (Mao et al., TPAMI 2025) constructs a global tree and optimizes over the hierarchical structure for long input contexts. AutoChunker (Jain et al., ACL 2025) adopts a bottom-up approach that leverages language models to identify and segregate logical units while preserving document hierarchy through a tree-based representation.
Questions for authors
- What evaluation procedure produced the results in Tables 1 through 5? Please clarify how normalized tree edit distance, macro role F1, and Reconstruction Accuracy were calculated.
- How was the DBR uncertainty calculated in the reported experiments? The paper describes probability distributions over role labels for each node, while the supplementary materials do not clearly demonstrate that these distributions were used.
- How does the proposed mapping guarantee a unique association between each content ID and its original span? The supplementary materials do not clearly explain how multiple content nodes associated with the same structural path are preserved independently. Please provide an example showing exact reconstruction before and after masking and rollback.
- Could the authors clarify the precise technical novelty of ISCF relative to Seg2Act, PartPrompt, and AutoChunker, beyond differences in the task setting or output format?
We thank the reviewer for precise, professional comments and commit to supplying implementation materials.
Q1 — Evaluation pipeline (N-TED / macro role F1 / RecAcc). N-TED = Zhang–Shasha TED / (|gold nodes|+|pred nodes|); macro role F1 is a macro-average over 10 role classes on content nodes (non-span datasets use title-derived structural roles); RecAcc is sentence-level exact-match between reconstruction (after Φ restores masks) and source. Evaluation scripts will be fully released so every number is recomputable.
Q2 — How DBR uncertainty is computed. Consistent with §2.2: under constrained decoding we take logits at role-field positions, normalize to p(role|n_c) over 10 classes in a single forward (temp 0.7), average per-node entropy to get H_c, and compare to τ=1.2. The omission from the supplement is ours — we will add the constrained-decoding logits snippet plus a real sample log (per-node role distribution, H_c, DBR-triggered flag).
Q3 — Content-ID ↔ span uniqueness / multi-node same-path. The primary key is the globally unique content-ID c, Φ: c→span is injective; π is only auxiliary. Multiple nodes under the same π have distinct c's, so Φ restores each span independently, no cross-contamination. We will provide the requested verbatim reconstruction example (before/after masking + rollback).
Q4 — Novelty vs Seg2Act / PartPrompt / AutoChunker. We will add these to Related Work with a mechanism-level table. Our distinction is at the mechanism level, not task/output format: vs Seg2Act — both iterative/stateful, but Seg2Act uses supervised action generation, whereas ISCF's novelty is entropy-driven, unsupervised DBR realignment + checkpoint rollback, an error-control signal absent in action-generation; vs PartPrompt — ISCF's Adaptive Structural Masking explicitly resolves unbounded-structure-vs-finite-context (full skeleton + [SUBTREE_MASK], >60% compression, fully restorable via unique IDs); vs AutoChunker — AutoChunker targets lossless chunking/retrieval, whereas ISCF targets controlled-lossy hierarchical parsing + near-lossless reconstruction with Coverage/Disjointness and global parent assignment. The combination of the three components is the substantive distinction.
Reproducibility. Accepted — we will release a full package (four datasets + gold trees, three baseline scripts, evaluation scripts, HAPE prompt templates) with reference implementations, resolving the "evaluator diverges from text definitions" issue.
Note to all reviewers: the core contributions (stateful iterative construction, entropy-driven DBR, compressed global skeleton, deterministic content-ID reconstruction) and empirical gains are unchanged; we will unify token/sentence units, add hierarchical-correctness/recomputable metrics, release data and annotation protocol, and add a mechanism-level Related Work comparison. We kindly ask reviewers to re-evaluate if these address their concerns.