- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes DocISE for question answering over semi-structured documents. Rather than constructing explicit document trees, it uses learned representations to capture textual, spatial, and hierarchical information. The system combines BM25, dense retrieval, layout fingerprints, a lightweight cross-encoder, and evidence expansion before passing the retrieved context to an LLM. The authors report improvements in accuracy, cost, and latency over existing methods on four benchmarks.
Critical strengths
Reducing the number of LLM calls during retrieval is a reasonable direction, and incorporating layout information into retrieval could be useful for document QA. A lightweight local retrieval pipeline would have practical value if the reported accuracy and efficiency gains can be reproduced.
Critical weaknesses
On page 3 (lines 158–160), the authors define the depth of all non-title atoms as 0. However, the structure consistency loss on page 5 (lines 242–246) requires a child atom to have a greater depth than its parent title. This would exclude ordinary paragraphs and table cells from being children of any title. The problem also appears in Algorithm 1 (Appendix A.2.1), where line 16 only assigns parents to atoms with depth greater than 0. I cannot understand how the model learns the claimed hierarchical relationships between titles and document content under these definitions.
The spatial graph has a similar inconsistency. Both Section 2 and Appendix A.2 explicitly restrict graph edges to atoms on the same page. Yet on page 7 (lines 372–377), the authors claim that the spatial graph can bridge a figure caption and a descriptive paragraph on different pages. The graph as defined cannot support this behavior. I also noticed that the InfoNCE loss on page 5 omits the positive sample from its denominator. With these discrepancies, it is difficult to determine what was actually implemented and trained.
I am also concerned about the fairness of the experimental comparison. DocISE uses GPT-5.5 for answer generation, but the generator configurations of the baselines are unclear. According to Table 3, replacing GPT-5.5 with GPT-4o reduces DocISE's MMDA accuracy from 83.92% to 76.21%, a drop of 7.71 points. This is substantial compared to the reported 10.59-point improvement over MoDora in Table 1. Without controlling for the answer generator, it is hard to attribute the improvement to the proposed retrieval method.
The number of basic inconsistencies and the poor quality of the figures also make me question how carefully the manuscript was reviewed before submission. Figures 3 and 4 contain obvious overlapping text and numerical labels that should have been caught by simply opening the final PDF. Together with the contradictions in the method description, this raises concerns about whether AI-assisted content was adequately verified by the authors. This is particularly concerning given that the AI Use Statement explicitly claims that all AI-assisted work was reviewed and checked for accuracy.
Questions for authors
-
How are parent-child relationships learned for non-title atoms, given that their depth is defined as 0? Could the authors reconcile this with Algorithm 1, line 16? I would also like to understand how the spatial graph retrieves cross-page evidence when its edges are restricted to the same page, and whether the InfoNCE denominator is a typo.
-
Which answer generator was used for MoDora and the other baselines in Table 1? Could the authors provide results under the same generator, either by evaluating DocISE with the baseline generator or rerunning MoDora with GPT-5.5?
-
Were the baseline results reproduced or taken from existing papers? I noticed that TextRAG and QUEST have identical DUDE AIC-Acc scores (55.96), while M3DocRAG and ZenDB have identical MMDA AIC-Acc scores (47.42). Could the authors clarify where these numbers came from and whether all methods were evaluated under the same protocol?
-
Which OCR model was actually used in the experiments? Will the authors release the complete training and evaluation scripts?
-
Figures 3 and 4 contain severe overlaps between text labels, numerical annotations, and plot elements, making some results difficult to read. Could the authors provide corrected figures and confirm that the plotted values are accurate?