- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies task scope loss in visual document question answering, where a correct local observation replaces part of the original task. It proposes retaining an initial record of factual requirements, dependencies, candidate values, and source pages throughout rereading planning and answer synthesis. The method uses frozen models with a fixed four-page pool and four reader calls. A factorial comparison evaluates ordinary and focused rereading with and without memory. On 212 questions from 106 held-out MMLongBench-Doc documents, memory improves focused rereading from 44.8% to 49.5% local semantic accuracy. Additional analyses examine reading trajectories, record content, and transfer to LongDocURL.
Critical strengths
-
The main comparison controls the retrieved page pool, reader-call budget, and generation limits, while accounting for additional memory tokens. Paired confidence intervals use document-level resampling. This provides credible evidence for the measured improvement within the evaluated pipeline.
-
The analysis is informative and transparent. The paper reports cases where memory preserves incorrect values, the competitive performance of ordinary rereading, and sensitivity to evaluation choices. These results help identify the circumstances in which retaining earlier observations helps or hurts.
-
The intervention is lightweight and easy to implement. It reuses an existing record without additional training or reader calls, adding roughly 354 input tokens per question for focused rereading. This makes the approach practical for systems with constrained inference budgets.
Critical weaknesses
-
The methodological novelty is limited relative to existing work on persistent reasoning state and evidence organization. G²-Reader [1] maintains a dependency graph of subquestions and intermediate findings, while MARDoc [2] preserves structured evidence and reasoning memories across iterations. The paper needs a clearer account of the additional contribution of its fixed initial-record interface within this literature.
-
The experimental comparison is dominated by ablations and variants of the authors’ pipeline. Strong recent document QA systems are missing from a common evaluation with comparable resource budgets. Ordinary rereading without memory already achieves 49.1%, compared with 49.5% for focused rereading with memory. The main improvement therefore largely restores the performance lost under the focused-reading configuration, leaving a weak empirical case for adopting the proposed combination.
-
The central motivation is insufficiently connected to the evaluation. Task scope loss is illustrated through individual examples, without a systematic measurement of its prevalence or reduction after adding memory. The analysis should identify cases where necessary facts were correctly observed and subsequently omitted, then measure how often memory repairs those cases. This is essential to establishing the paper’s proposed explanation for the accuracy gains.
[1] Yaxin Du et al. G²-Reader: Dual Evolving Graphs for Multimodal Document QA. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 26624–26648, 2026. Conference paper. Full text.
[2] Kaifeng Chen et al. MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA. arXiv preprint arXiv:2606.05749, 2026. Paper. Full text.
Questions for authors
-
Could you explain what is the specific contribution of this work beyond the similar mechanisms of persistent evidence and dependency tracking used in G²-Reader [1] and MARDoc [2]?
-
Do you have results for a competitive document QA agent using the same backbone and evaluation protocol at a comparable inference budget? Such a comparison would help establish the practical value of the approach alongside the internal variants.
-
Among the 19 questions corrected by focused rereading with memory, how many involve necessary facts that were correctly observed earlier and subsequently omitted? What is the frequency of this failure before and after memory retention across the evaluation set?