acceptodds
Withdrawn from ICLR 2027

Keep the Question: Task-Preserving Memory for Visual PDF Reading

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual document question answering often decomposes a question into local reading steps. This decomposition can lose part of the original task: a correct intermediate observation may replace a required comparison, count, or sum. We study this failure as task scope loss and introduce task-preserving memory to connect search-time observations with subsequent reading. An initial record links factual requirements to their prerequisites, candidate values, and source pages. The same record guides follow-up retrieval and is supplied to both the rereading planner and final answer generator, allowing local inspection without discarding previously observed facts. The method uses frozen models and requires no additional pages or reader calls. On 212 questions from 106 held-out MMLongBench-Doc documents, retaining the record improves focused rereading from 44.8% to 49.5% local semantic accuracy, a 4.72-percentage-point gain with a 95% document-bootstrap interval of [0.47, 8.96]. A factorial comparison of memory retention and reading policy, record-content analysis, and a LongDocURL transfer evaluation examine the contribution of retained information. The results show that preserving search-time facts can improve task completion under a fixed page and reader-call budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual document understanding, document question answering, multimodal reasoning, task-preserving memory, long-document understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fkqYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies task scope loss in visual document question answering, where a correct local observation replaces part of the original task. It proposes retaining an initial record of factual requirements, dependencies, candidate values, and source pages throughout rereading planning and answer synthesis. The method uses frozen models with a fixed four-page pool and four reader calls. A factorial comparison evaluates ordinary and focused rereading with and without memory. On 212 questions from 106 held-out MMLongBench-Doc documents, memory improves focused rereading from 44.8% to 49.5% local semantic accuracy. Additional analyses examine reading trajectories, record content, and transfer to LongDocURL.

Critical strengths
  1. The main comparison controls the retrieved page pool, reader-call budget, and generation limits, while accounting for additional memory tokens. Paired confidence intervals use document-level resampling. This provides credible evidence for the measured improvement within the evaluated pipeline.

  2. The analysis is informative and transparent. The paper reports cases where memory preserves incorrect values, the competitive performance of ordinary rereading, and sensitivity to evaluation choices. These results help identify the circumstances in which retaining earlier observations helps or hurts.

  3. The intervention is lightweight and easy to implement. It reuses an existing record without additional training or reader calls, adding roughly 354 input tokens per question for focused rereading. This makes the approach practical for systems with constrained inference budgets.

Critical weaknesses
  1. The methodological novelty is limited relative to existing work on persistent reasoning state and evidence organization. G²-Reader [1] maintains a dependency graph of subquestions and intermediate findings, while MARDoc [2] preserves structured evidence and reasoning memories across iterations. The paper needs a clearer account of the additional contribution of its fixed initial-record interface within this literature.

  2. The experimental comparison is dominated by ablations and variants of the authors’ pipeline. Strong recent document QA systems are missing from a common evaluation with comparable resource budgets. Ordinary rereading without memory already achieves 49.1%, compared with 49.5% for focused rereading with memory. The main improvement therefore largely restores the performance lost under the focused-reading configuration, leaving a weak empirical case for adopting the proposed combination.

  3. The central motivation is insufficiently connected to the evaluation. Task scope loss is illustrated through individual examples, without a systematic measurement of its prevalence or reduction after adding memory. The analysis should identify cases where necessary facts were correctly observed and subsequently omitted, then measure how often memory repairs those cases. This is essential to establishing the paper’s proposed explanation for the accuracy gains.

[1] Yaxin Du et al. G²-Reader: Dual Evolving Graphs for Multimodal Document QA. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 26624–26648, 2026. Conference paper. Full text.

[2] Kaifeng Chen et al. MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA. arXiv preprint arXiv:2606.05749, 2026. Paper. Full text.

Questions for authors
  1. Could you explain what is the specific contribution of this work beyond the similar mechanisms of persistent evidence and dependency tracking used in G²-Reader [1] and MARDoc [2]?

  2. Do you have results for a competitive document QA agent using the same backbone and evaluation protocol at a comparable inference budget? Such a comparison would help establish the practical value of the approach alongside the internal variants.

  3. Among the 19 questions corrected by focused rereading with memory, how many involve necessary facts that were correctly observed earlier and subsequently omitted? What is the frequency of this failure before and after memory retention across the evaluation set?

Reviewer_V7umOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses task scope loss in multi-step visual document QA, a failure mode where models forget the global constraints of the original question (e.g., sums or comparisons) when focusing on narrow, local reading steps. To solve this, the authors propose a straightforward task-preserving memory mechanism that explicitly carries the original question and a structured record of initial observations into all subsequent operations, specifically the local reading planner and the final answer synthesizer. By simply "keeping the question" and its prerequisite facts active across stages, the method prevents the model from discarding previously gathered evidence during local inspections, significantly improving task completion accuracy without requiring any additional page retrievals or computational overhead.

Critical strengths

None

Critical weaknesses

The overall workload and technical contribution of this paper are insufficient for an ICLR submission. Fundamentally, the proposed approach boils down to carrying the original question into subsequent PDF reading stages, which functions more as an ad-hoc prompting trick rather than a principled methodological advancement. Furthermore, both the problem formulation and the experimental evaluation lack clarity and rigor. Consequently, I cannot recommend this paper for acceptance at ICLR.

Questions for authors

None

Reviewer_8CMBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies why AI assistants sometimes lose track of the full question when reading long PDFs. They may correctly extract one detail but forget that the question asks them to compare several items, count them, or combine their values. The authors propose keeping a structured record of what the question requires, what has already been found, and which pages support those observations. This record is passed to later reading and answer-generation steps. Their experiments show that retaining it improves the focused reader’s accuracy without adding more pages or reader calls.

Critical strengths

The paper highlights a practical problem: an assistant can read individual facts correctly and still miss part of the user’s question.

The approach reuses information already collected, requires no model training, and adds little overhead.

The main comparison uses the same pages and number of reader calls, helping isolate the effect of retaining earlier observations.

The authors openly discuss limitations, including cases where an incorrect earlier observation carries through and hurts the final answer.

Critical weaknesses

The evaluation uses only one base model. Other off-the-shelf models report stronger MMLongBench results, so a comparison under the same evaluation conditions would help put the reported improvement in context.

My main question is whether the method still helps stronger models, and whether simpler strategies work just as well, such as keeping the original question alongside each intermediate result or using it to guide retrieval.

The experiments do not sufficiently isolate the method’s added value. Comparisons with plain retained notes, question reminders, and simpler question-guided retrieval would help establish whether the structured memory itself provides a meaningful benefit.

Questions for authors

Does the method still improve accuracy with stronger base models under the same page budget and evaluation setup?

How does it compare with simpler strategies, such as keeping plain notes, appending the original question to each reading step, or using it to guide retrieval?

Can you isolate the benefit of the structured memory by comparing it with plain notes containing the same facts and source pages under a similar token budget?

How many corrected answers actually recover a missing part of the original question, rather than fixing some other type of error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.