- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
A well-targeted, falsifiable benchmark, beyond cross-page table QA, the most significant contribution is the oracle single-page analysis, which demonstrates the weaknesses of existing benchmarks; combined with diagnostics along dimensions such as faithfulness and evidence granularity, it gives readers a clear picture of the current models' limitations.
However, the timeliness of the selected models is insufficient, they do not adequately represent the current state of large-model development. Moreover, the paper does not evaluate the performance of agentic systems.
Critical strengths
- ORACLE SETTING ANALYSIS is a main contribution, showing that established multi-page benchmarks (MMLongBench, MP-DocVQA, DUDE) improve when handed a single gold page is convincing, quantitative evidence that prior benchmarks are substantially single-page-answerable and that this one is not. This kind of self-critical validity check is rare and materially strengthens the paper's central claim.
- Beyond accuracy, the paper measures faithfulness and evidence granularity, make the evaluation more falsifiable.
Critical weaknesses
- The set of evaluated models is not up to date, and the paper does not evaluate the performance of agentic systems or tool-integrated approaches.
- The evaluation lacks a comparison against human annotations. GPT-5.5 and other models are used to extract answers and judge faithfulness, but no agreement with human labeling is reported.
- The corpus is drawn solely from biomedical documents, so it is unclear whether the cross-page challenge and the model rankings generalize to other domains with different table conventions, such as financial statements or legal schedules. The benchmark is also smaller than MP-DocVQA and DUDE, and only 627 of the 1,871 questions are human-written; as a result, several sub-categories have few examples, making the per-category accuracy noisy.
Questions for authors
- Given Sec 3.6 shows models converge to 63.7->67.6% when handed located evidence, how much of end-to-end TAA is limited by perception/localization vs reasoning? Pleae report reasoning-given-gold-evidence as a primary metric and provide a per-level (L1/L2/L3) decomposition that separates the two, so readers can see what the benchmark actually measures.
- The evaluation should be extended to include more recent models as well as agent- and tool-integrated frameworks.
- Can you report human agreement for the extractor and for the faithfulness judge, and a cross-judge sensitivity analysis? How robust are the 45.9% accuracy and 83% faithfulness figures to the judge choice?
If the author provides satisfactory responses to my review comments, I may raise my rating.