acceptodds
Withdrawn from ICLR 2027

Structure Before Generation: Invariant Graph Readout with Protected Completion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models often reconstruct document structure by generating serialized graphs, coupling structural decisions to output ordering, identifier copying, and syntax. Direct graph predictors remove this serialization step but do not by themselves specify how an existing prediction can be revised or extended without disturbing already-correct structure. We introduce Invariant Graph Readout, a non-autoregressive framework that combines direct graph prediction from frozen visual representations of a pretrained vision-language model with state-preserving correction and protected completion. A trainable query head predicts typed nodes, bounding boxes, and relations, while operation-specific preservation contracts define which attributes each graph-edit operation may modify and which state it must preserve. Complementary proposals, anchor-preserving selection, and endpoint-consistent attachment recover missing structure without rewriting protected objects. On 600 held-out OracleGraph pages, the complete system achieves a Graph F1 of 0.3988 ± 0.0023 versus 0.3872 ± 0.0072 for Strict JSON Generation (mean ± standard deviation over three training seeds). On a 100-page subset at batch size one on an A100 GPU, it reduces mean end-to-end latency by 22.3× relative to greedy Strict JSON Generation (3.60 vs. 80.14 s per page), whose outputs average approximately 1,157 tokens per page. Under supervised target-domain training on GraphDoc, mean Graph F1 on 1,000 held-out pages improves from 0.0443 for a global-and-local graph readout baseline to 0.1069 across three independently trained pipelines. These results indicate that preservation-aware graph editing built on direct prediction is a practical alternative to autoregressive structural generation, and that contract-guided completion can be reused across document domains under supervised training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, Document structure analysis, Non-autoregressive decoding, Graph edit distance, Structured prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J8UfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies document-graph construction without autoregressive serialization. It proposes Invariant Graph Readout, combining direct graph prediction from frozen VLM visual features with graph-edit contracts that preserve designated state during correction and protected completion. It evaluates graph quality and latency against strict JSON generation on OracleGraph, and instantiates the completion interface with target-domain supervision on GraphDoc.

Critical strengths
  1. The preservation contracts are specified concretely and are actually enforced rather than merely described informally.

  2. The paper evaluates practical system behavior rather than reporting only static graph scores.

  3. The paper is easy to follow.

Critical weaknesses
  1. The methodological novelty appears relatively limited. Non-autoregressive graph prediction is already well established, while the proposed preservation contracts primarily enforce explicit constraints on graph-editing operations.

  2. Despite the framework's complexity, the improvements remain modest. On OracleGraph, Full improves Graph F1 by only 1.16 percentage points over JSON Generation, while Edge F1 actually decreases. Moreover, protected completion contributes only 0.19 points over Guarded Readout. On GraphDoc, the final Graph F1 remains just 0.1069, raising concerns about the framework's practical effectiveness.

  3. Although Section 4.2 describes the 1,000 GraphDoc evaluation pages as held out from training, Appendix D.1 acknowledges that these pages had been exposed during earlier system design. This potentially compromises the independence of the evaluation, particularly since GraphDoc provides the primary evidence for cross-domain reuse. Evaluation on a genuinely untouched test split would strengthen this claim.

  4. The OracleGraph comparison does not adequately control for training resources or baseline strength. Full uses 5,758 fitting sources, compared with 4,700 for Strict JSON Generation. Furthermore, the JSON baseline disables grammar-constrained decoding and JSON repair, with parsing failures treated as empty graphs. Supervision-matched experiments and stronger structured-generation baselines are needed to establish whether the reported gains genuinely arise from the proposed graph-native interface.

Questions for authors
  1. For the GraphDoc pages described in Appendix D.1 as having been exposed during earlier design work, what information from those pages was available during that process? Is there a fully untouched GraphDoc split on which the frozen revised pipelines can be evaluated?

  2. For Strict JSON Generation in Appendix B, what fraction of OracleGraph outputs fail strict parsing, and how would Graph F1 change under grammar-constrained decoding or a validity-preserving JSON decoder? This would determine whether the quality difference in Tables 3 and 4 primarily reflects structural prediction quality or parser-induced empty outputs.

Reviewer_gSyKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Invariant Graph Readout predicts typed document nodes, boxes, and relations from frozen VLM visual features, then applies operations with explicit restrictions on which predicted state may change. Its correction mechanisms separate classification from geometric refinement, calibrate relations, adjust boxes, and delete eligible isolated nodes. Protected completion uses additional proposals, anchor-preserving selection, and relations involving added endpoints. Domain-specific implementations are evaluated on OracleGraph and GraphDoc. On OracleGraph, Full achieves Graph F1 0.3988 versus 0.3872 for Strict JSON Generation and is 22.3 times faster in the reported batch-one timing experiment. On GraphDoc, Full improves from 0.0443 for Base+Local to 0.1069, compared with 0.1039 for a matched-count selection control. The claimed invariance concerns preservation of designated predicted state, not correctness, transformation invariance, or zero-shot domain transfer.

Critical strengths

. The system comparison demonstrates a substantial practical efficiency advantage under a carefully described protocol. Table 8 and Appendix H time image loading through serialization, including online candidate generation and CPU/GPU transfers, on the same GPU without cached features or candidate banks. Full takes 3.60 seconds/page versus 80.14 for JSON, with better Graph F1 on the actual timed outputs. This is stronger evidence than timing only a small prediction head and is relevant to deploying structured document extraction.

  1. The evaluation separates several claims that could otherwise be conflated. The method explicitly guarantees preservation of selected predicted state rather than correctness or monotone F1; 243,000 recorded checks support compliance in the evaluated GraphDoc executions. Three independently trained pipelines, paired document/source resampling, matched-count controls, and the disclosed prior-exposure/fallback history improve interpretability. In particular, GraphDoc's additional 0.3018 percentage-point gain over the matched-count control has positive reported intervals, providing specific evidence for residual reranking within the fixed pipeline.
Critical weaknesses

Equation (2) formalizes immutable designated state, and the execution checks verify that implementations obey this restriction. However, Table 7's controls all retain the applicable protections: they isolate ranking/allocation choices, not whether preservation yields a better correction/completion trade-off than an otherwise matched editor. On OracleGraph, Full adds only 0.1888 Graph-F1 percentage points over Guarded Readout (Section 4.3), while mean latency increases from 0.8354 to 3.5974 seconds/page (Table 8). Learned attachment has no demonstrated positive net effect against the node-only or inheritance-only controls. On GraphDoc, the matched-count control already achieves 0.103872 against Full's 0.106891, so the large gain over Base+Local should not be read as evidence for residual reranking alone. The authors correctly acknowledge these scopes, but the evidence still leaves unclear what useful capability depends on the proposed contracts, especially when protected predictions are incorrect. A matched comparison reporting preserved correct objects, locked-in errors, recovered structure, and overall accuracy would address this central question.

Questions for authors

Evidence for the central contribution: With the same proposals, learned scores, supervision, and budgets, how does protected editing compare with an editor allowed to revise retained objects? Please quantify both avoided damage to correct structure and errors that preservation prevents the system from fixing.

Reviewer_LbooOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Invariant Graph Readout, a non-autoregressive approach to document graph prediction. It introduces Preservation Contracts to constrain graph correction and completion, and evaluates the framework on OracleGraph and GraphDoc.

Critical strengths

The non-autoregressive approach achieves a 22.3× inference speedup over JSON generation on OracleGraph.

Critical weaknesses

1. Limited novelty.

The proposed Preservation Contracts largely formalize standard graph-editing constraints, such as preserving node types during box correction and keeping existing nodes unchanged during completion. These are straightforward engineering choices rather than novel methodological contributions. The paper adds considerable complexity without clearly establishing what is fundamentally new.

2. Marginal gains despite substantial complexity.

On OracleGraph, Full improves Graph F1 only from 0.3969 to 0.3988 over Guarded Readout. On GraphDoc, it improves from 0.1039 to 0.1069 over the matched-count control. These small gains hardly justify the extensive pipeline. Moreover, Full uses additional training data and models, making it difficult to isolate the contribution of the proposed framework. The main speedup comes from non-autoregressive prediction, which is already well established.

Questions for authors
  1. What makes Preservation Contracts fundamentally different from standard graph-editing constraints?
  2. Can the authors justify the added complexity given the marginal improvements over simpler baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.