acceptodds
Desk rejected at ICLR 2027

DocISE: Implicit Structure Encoder for Semi-Structured Document Question Answering

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Semi-structured documents are ubiquitous in scientific reports, financial state- ments, and technical manuals. Question answering over such documents requires simultaneous understanding of text, tables, charts, and complex hierarchical lay- outs. Existing methods either rely on repeatedly calling large language models (LLM) for structure parsing and retrieval, leading to high costs and latency, or extracting the content of the document and losing layout and hierarchy informa- tion, sacrificing answer accuracy. To address this, we propose DocISE, a sys- tem that achieves state-of-the-art accuracy with minimal computational expense. Our approach introduces three core modules. (1) An Implicit Structure En- coder trained via contrastive learning and a structure consistency loss. This mod- ule jointly embeds hierarchical relationships, spatial positions, and textual con- tent into a dense vector space to capturing document structure. (2) A Hybrid Retrieval Pipeline that leverages BM25, layout fingerprints, and a lightweight cross-encoder to perform high-precision retrieval drastically reducing costs and latency. (3) A Topology-Aware Evidence Expansion mechanism that adaptively retrieves spatially adjacent and structurally related evidence, overcoming the ev- idence omission common in fixed-path retrieval methods. We evaluate DocISE on four benchmarks, and comprehensive evaluations show that DocISE achieves a new state-of-the-art accuracy on the four benchmarks. Crucially, it does so with a 50× reduction in cost and 4× lower query latency compared to the previous state-of-the-art Method. These results demonstrate that DocISE establishes a new optimal trade-off among accuracy, cost, and speed, offering a practical and scal- able path for accurate semi-structured document analysis. Source code is available at https://anonymous.4open.science/r/DocISE.

Primary area:
generative models
Keywords:
Semi-Structured Document QA, Implicit Structure Encoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5Sa4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DocISE for question answering over semi-structured documents. Rather than constructing explicit document trees, it uses learned representations to capture textual, spatial, and hierarchical information. The system combines BM25, dense retrieval, layout fingerprints, a lightweight cross-encoder, and evidence expansion before passing the retrieved context to an LLM. The authors report improvements in accuracy, cost, and latency over existing methods on four benchmarks.

Critical strengths

Reducing the number of LLM calls during retrieval is a reasonable direction, and incorporating layout information into retrieval could be useful for document QA. A lightweight local retrieval pipeline would have practical value if the reported accuracy and efficiency gains can be reproduced.

Critical weaknesses

On page 3 (lines 158–160), the authors define the depth of all non-title atoms as 0. However, the structure consistency loss on page 5 (lines 242–246) requires a child atom to have a greater depth than its parent title. This would exclude ordinary paragraphs and table cells from being children of any title. The problem also appears in Algorithm 1 (Appendix A.2.1), where line 16 only assigns parents to atoms with depth greater than 0. I cannot understand how the model learns the claimed hierarchical relationships between titles and document content under these definitions.

The spatial graph has a similar inconsistency. Both Section 2 and Appendix A.2 explicitly restrict graph edges to atoms on the same page. Yet on page 7 (lines 372–377), the authors claim that the spatial graph can bridge a figure caption and a descriptive paragraph on different pages. The graph as defined cannot support this behavior. I also noticed that the InfoNCE loss on page 5 omits the positive sample from its denominator. With these discrepancies, it is difficult to determine what was actually implemented and trained.

I am also concerned about the fairness of the experimental comparison. DocISE uses GPT-5.5 for answer generation, but the generator configurations of the baselines are unclear. According to Table 3, replacing GPT-5.5 with GPT-4o reduces DocISE's MMDA accuracy from 83.92% to 76.21%, a drop of 7.71 points. This is substantial compared to the reported 10.59-point improvement over MoDora in Table 1. Without controlling for the answer generator, it is hard to attribute the improvement to the proposed retrieval method.

The number of basic inconsistencies and the poor quality of the figures also make me question how carefully the manuscript was reviewed before submission. Figures 3 and 4 contain obvious overlapping text and numerical labels that should have been caught by simply opening the final PDF. Together with the contradictions in the method description, this raises concerns about whether AI-assisted content was adequately verified by the authors. This is particularly concerning given that the AI Use Statement explicitly claims that all AI-assisted work was reviewed and checked for accuracy.

Questions for authors
  1. How are parent-child relationships learned for non-title atoms, given that their depth is defined as 0? Could the authors reconcile this with Algorithm 1, line 16? I would also like to understand how the spatial graph retrieves cross-page evidence when its edges are restricted to the same page, and whether the InfoNCE denominator is a typo.

  2. Which answer generator was used for MoDora and the other baselines in Table 1? Could the authors provide results under the same generator, either by evaluating DocISE with the baseline generator or rerunning MoDora with GPT-5.5?

  3. Were the baseline results reproduced or taken from existing papers? I noticed that TextRAG and QUEST have identical DUDE AIC-Acc scores (55.96), while M3DocRAG and ZenDB have identical MMDA AIC-Acc scores (47.42). Could the authors clarify where these numbers came from and whether all methods were evaluated under the same protocol?

  4. Which OCR model was actually used in the experiments? Will the authors release the complete training and evaluation scripts?

  5. Figures 3 and 4 contain severe overlaps between text labels, numerical annotations, and plot elements, making some results difficult to read. Could the authors provide corrected figures and confirm that the plotted values are accurate?

Reviewer_44k3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses overall the problem of llms and vision language models carrying out question answering over documents that are semi structured and would have various different challenges related to capturing layout and also doing this in a cheap way. They propose this architecture that has different components, such as a flexible router for answering different types of questions based on different types of evidence using different retrievers and also a trained module that would capture layout among others. They show that their method is super cheap and also beats baselines that they have compared against.

Critical strengths

The paper has a really good motivational setup and discuss the problem that It wants to address really clearly and also the problem that is addressing has great importance and many challenges that make it interesting to explore.

Critical weaknesses

Overall how about lad-rag and similar methods that have since come out like MoLoRAG or MMRAG-DocQA ? How do they perform ? My overall concern about this paper is that you’re not comparing against state of the art baselines right now in the field.

For the document representation you’re assuming that the edges are only supposed to capture and support spatial adjacency but how about semantic links and relationships that might be across pages or even a table and its accompanying description that might be in a different page or a section header and all its subsections that are multiple pages after it ?

You’re assuming that documents are actually small but how about documents that are long and would actually introduce their own challenges that are all relevant to the problem you’re trying to address here ? And I don’t mean this just with respect to latency I meant the other challenges that this will introduce, e.g., long context reasoning or retrieval of evidence across multiple pages, …

There are various types of connections or to be more specific, layout-related and semantic connections that you wanna have captured in your contrastive learning loss ? How do you ensure that you have a good coverage with respect to various types of connections that might be a target of questions in document understanding benchmarks ? Have you done any preliminary exploratory analysis of the types of connections you have to have captured or supported ? I’m asking this because based on your explanation you’re mostly capturing local connections and not the global connections, especially the global connections that start from one page and would cover a couple of pages for instance one section that has multiple examples under it but would cover 10 pages after the main title page.

In line 280 you mention that for the text questions, the first filtering steps is based on bm25 but if the questions don’t match lexically with the text of the documents, the evidence would be missed, right ?

And in line 281 you say for the hierarchy aware questions title atoms are identified and then the children are added but by the definition of how you actually parse the documents, here the notion of children is different from the notion of children in a graph that can cover multiple children, you have only spatial heuristics where nodes are adjacent to each other to consider something as a parent and a child, hence, you will miss a lot of the evidence in situations that don’t fit this specific filter. Please correct me if I’m wrong, or provide some detail and elaboration on how you will address such cases that don’t fit your design.

Another question for the expansion method that you explain here, you’re mostly just expanding based on spatial adjecency but how about semantic ones that can actually be more than just one hop away from the node you start from. So based on your design you’re not really solving the problem but just mitigating it by heuristically adding nodes around it which may or may not in fact address the challenges that the question here demands.

Why haven’t you used benchmarks that are actually hard and challenging compared to mpdocvqa or DUDE, for instance MMLongBench-Doc or LongDocURL, MMDOCIR, among many more that I can think of.

You have already provided a good set of baselines but overall my concern is that apart from pushing to keep everything cheap which is a good contribution, other part of the results and experimental setups might not allow the reader to have the right takeaway from the paper given the not enough challenging benchmarks and baselines that you’re using. lad-rag among other specifically would be a good one to check here as it’s actually specifically close to your design regarding contextualization and a neuro-symbolic design that they have similar to yours.

Another question that I have instead of the whole layout-aware retriever that you have trained why not just using ColPali as the retriever component or other similar multimodal retrievers. They overall show a pretty decent performance in various different tasks. Not benchmarking that and simply trying to sell training something from scratch might be a solution to a problem we never had in the literature

Questions for authors

please check weaknesses.

Reviewer_omKqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper identifies key challenges in document QA: scattered information, and ignorance of hierarchical structure. It claims that prior work either fail to retrieve all relevant information, or ignore structure, causing low accuracy, or build structure in an expensive while non-robust way, incurring high cost while struggling with OCR noises. To address these challenges, the paper proposes DocISE, with hybrid embedding and retrieval techniques. The method achieves high accuracy while low cost and latency compared with SOTAs.

Critical strengths
  1. The paper's evaluates many baselines, covering major document QA methods.
  2. The paper identifies critical challenges in document QA, and provides reasonable methods.
Critical weaknesses
  1. The paper claims high accuracy with low accuracy as a major contribution. However, the models used in baselines are not reported, and might not align with the one used in the paper's proposed method. Therefore, the advantages in accuracy might come from models, not the method itself. A fair comparison should use the same methods for all baselines. Also, to support the claim of low cost, the paper needs to report all LLM costs for each method, not only QA.
  2. The paper claims implicit structure encoding as one main contribution. However, it is unclear why other similar implicit structure encoding methods, e.g., LayoutLMv3, SmolDocling, etc.
Questions for authors
  1. What are the average document sizes across datasets used in your experiments? Could you please report at least the average number of pages per doc?
  2. You mentioned " Implicit Structure Encoding" as one of your major contribution. How does it differ from existing pre-trained models that embed text and layout together, such as LayoutLMv3, Smoldocling, etc.? Could you please compare with at least one similar method?
  3. Could you please list the backbone LLM models used for baselines?
  4. Could you please all LLM costs used in your method? For the cost reported in the current paper, is it only QA cost, or also include other calls, e.g., one-time scoring for reranker? Could you also please list all LLM costs of baselines, as well as how they are measured?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.