acceptodds
Withdrawn from ICLR 2027

PubTables-QA: A Multi-Page Table Visual Question Answering for Cross-Page Table Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tables are a central medium for communicating quantitative evidence in scientific documents, yet their compact relational form makes robust interpretation depend on layout and context. However, existing multi-page and multi-table benchmarks may still allow stochastic shortcut search, where models locate local answer-bearing regions without systematically navigating the underlying table structure. To address this gap, we introduce PubTables-QA, a multi-page TableVQA benchmark dataset carefully designed to audit cross-page visual reasoning in authentic academic documents. Built upon the extensive structural table annotations of PubTables- v2, PubTables-QA introduces 1,871 table-based QA pairs, where each question is designed to be unanswerable from a single page alone. This design compels models to reconstruct cross-page table continuity across headers, rows, columns, and cells, or to integrate evidence distributed across multiple tables. PubTables-QA further provides a three-level fine-grained QA taxonomy: Document, Table, and Cell/column levels respectively, ranging from relevant evidence localization and table-level structural understanding, to precise cell/column-level value extraction. Experiments on recent MLLMs, including GPT-4o, Gemini-2.5-Pro, and Gemma-4 reveal that even the strongest models achieve 45.9% overall accuracy. Further, our evidence-aware evaluation confirms that, unlike existing multi-page benchmarks susceptible to stochastic shortcuts, PubTables-QA requires genuine cross-page evidence integration, establishing it as a substantial and unsolved challenge. Code and Dataset are available at https://anonymous.4open.science/r/PubTables-QA-584D.

Primary area:
datasets and benchmarks
Keywords:
Multi-Page Table VQA, Cross-Page Table Reasoning, Hierarchical QA Benchmark, Multimodal Large Language Model, Document Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h2TpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

A well-targeted, falsifiable benchmark, beyond cross-page table QA, the most significant contribution is the oracle single-page analysis, which demonstrates the weaknesses of existing benchmarks; combined with diagnostics along dimensions such as faithfulness and evidence granularity, it gives readers a clear picture of the current models' limitations.

However, the timeliness of the selected models is insufficient, they do not adequately represent the current state of large-model development. Moreover, the paper does not evaluate the performance of agentic systems.

Critical strengths
  1. ORACLE SETTING ANALYSIS is a main contribution, showing that established multi-page benchmarks (MMLongBench, MP-DocVQA, DUDE) improve when handed a single gold page is convincing, quantitative evidence that prior benchmarks are substantially single-page-answerable and that this one is not. This kind of self-critical validity check is rare and materially strengthens the paper's central claim.
  2. Beyond accuracy, the paper measures faithfulness and evidence granularity, make the evaluation more falsifiable.
Critical weaknesses
  1. The set of evaluated models is not up to date, and the paper does not evaluate the performance of agentic systems or tool-integrated approaches.
  2. The evaluation lacks a comparison against human annotations. GPT-5.5 and other models are used to extract answers and judge faithfulness, but no agreement with human labeling is reported.
  3. The corpus is drawn solely from biomedical documents, so it is unclear whether the cross-page challenge and the model rankings generalize to other domains with different table conventions, such as financial statements or legal schedules. The benchmark is also smaller than MP-DocVQA and DUDE, and only 627 of the 1,871 questions are human-written; as a result, several sub-categories have few examples, making the per-category accuracy noisy.
Questions for authors
  1. Given Sec 3.6 shows models converge to 63.7->67.6% when handed located evidence, how much of end-to-end TAA is limited by perception/localization vs reasoning? Pleae report reasoning-given-gold-evidence as a primary metric and provide a per-level (L1/L2/L3) decomposition that separates the two, so readers can see what the benchmark actually measures.
  2. The evaluation should be extended to include more recent models as well as agent- and tool-integrated frameworks.
  3. Can you report human agreement for the extractor and for the faithfulness judge, and a cross-judge sensitivity analysis? How robust are the 45.9% accuracy and 83% faithfulness figures to the judge choice?

If the author provides satisfactory responses to my review comments, I may raise my rating.

Reviewer_oQCoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper claims that existing multi-page and multi-table benchmarks may allow stochastic shortcut search in MLLM-based VQA process. To address this gap, this paper introduces PubTables-QA, whose samples requlre the MLLMs to really integrate information from different page or table segments. Experiments on recent MLLMs demonstrate that they perform not that good over PubTables-QA, establishing the cross-page stochastic shortcut issue as a substantial and unsolved challenge for MLLMs.

Critical strengths
  1. This paper introduces tasks where correct answers can only be derived by integrating split table segments across multiple pages, raising challenges to current MLLMs.
  2. This study provides a structured evaluation across document, table and columns, enabling a diagnostic and comprehensive analysis of model performance.
  3. Experiments reveal that the strongest MLLMs suffer from possible stochastic shortcut search, performing not that good on the proposed PubTables-QA.
Critical weaknesses
  1. The paper lacks a controlled ablation study that tables spanning multiple pages are stitched together at the image level into a single page to eliminate the visual discontinuity.
  2. The dataset construction pipeline relies heavily on MLLM-based judgement, which is likely to introduce algorithmic bias that could inadvertently filter out high-value samples exposing specific architectural flaws.
  3. How do the authors evaluate the generalization of this benchmark's conclusions to domains with vastly different formatting conventions and logical structures, e.g., financial reports or legal documents?
Questions for authors

See weaknesses part.

Reviewer_eisJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PubTables-QA, a benchmark for cross-page visual table question answering built on PubTables-v2, comprising 1,871 question–answer pairs across 296 scientific documents. Questions are designed to require multi-page evidence and cover document-level, table-level, and cell/column-level understanding, supported by structured evidence annotations. The authors evaluate eleven multimodal language models, with the best achieving 45.9% overall accuracy. Additional analyses examine single-page answerability, evidence faithfulness, and the effects of providing page locations, table bounding boxes, and cell values to investigate model limitations.

Critical strengths

1.The research question is clear and of practical significance. The paper distinguishes between "handling long inputs" and "integrating cross-page relationships," addressing a significant blind spot in existing document question answering evaluations.

2.Fine-grained evidence annotation enhances the diagnostic value of the dataset. Page numbers, table bounding boxes, and cell evidence allow the evaluation to further pinpoint the causes of failure.

3.Experiments are relatively comprehensive. The authors not only report leaderboards but also compare full-text versus single-page input, image-free input, different modalities, and performance after gradually increasing evidence.

4.Data construction involves sufficient manual effort. The authors use cross-validation with different models, followed by manual inspection, and supplement with manually written questions. The appendix also reports filtering quantities, validator reliability, and some inter-annotator agreement results, making the data production process relatively transparent.

Critical weaknesses
  1. The author's main proposal need more specification. I think there may exists some inconsistency between the experiments and the proposal. In the single-page experiment, where each evidence page is processed individually and the results are averaged, the ANLS score for PubTables-QA drops from 29.7 to 18.8. However, this decline in the average score does not rule out the possibility that one of the pages may have contained sufficient information to answer the question, while the other pages pulled the average down. The faithfulness metric is not so trustworthy and need to compared with human annotators. 2.The PubTables-v2 dataset used in this paper is drawn primarily from biomedical research papers, which may introduce some distributional bias. It is recommended to incorporate data from additional domains—such as financial reports—to broaden the dataset's scope.
Questions for authors
  1. Can you elaborate on your assertion that "every question structurally requires cross-page reasoning"? I think this propose has not yet been sufficiently proven by experiments. I understand you make sure the evidence are from different pages, but how is this related to the performance in the experiments?
  2. I am curious about the performance using the stronger text reasoning models like gpt-6 for OCR text setting

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.