- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DocPilot, an interactive agent framework for long visual document question answering. The framework organizes document interaction around six actions: Navigate, Extract, Focus, Solve, Analyze, and Answer. These actions support iterative evidence acquisition, local visual inspection, code-based computation, and answer verification. The authors also construct teacher-generated supervision and train a compact 4B model through two stages of supervised fine-tuning: single-ability training followed by multi-ability trajectory training. Experiments cover multiple backbone models and document QA benchmarks, supplemented by action ablations, training comparisons, and evidence-localization analyses.
Critical strengths
-
The framework demonstrates substantial practical gains across multiple backbones and benchmarks. For example, with Gemini-2.5-Pro, DocPilot improves MMLongBench-Doc accuracy from 58.1 to 72.8 and exceeds the reported DocLens result of 67.6. Evaluating both frontier models and a compact trained model provides useful evidence about the framework’s applicability across model scales.
-
The action ablations and evidence-localization analyses offer meaningful support for the system design. Removing Focus reduces chart and figure accuracy by 7.2 and 11.9 points, respectively, while removing Solve reduces table accuracy by 4.6 points. These results connect particular tools to the capabilities they are intended to support. The additional page-retrieval and element-localization evaluations also help explain the final-answer improvements.
Critical weaknesses
- The positioning against existing trainable document agents is incomplete, and the experiments do not sufficiently isolate the proposed training contribution.
The discussion in lines 087–097 does not adequately situate DocPilot within the broader literature on learning interactive policies for visually rich document understanding. Relevant work extends beyond VRAG-RL [1] and ALDEN [2] to Doc-V* [3], VISOR [4], MM-Doc-R1 [5], and InSight-doc [6]. These methods investigate learned evidence acquisition, document navigation, visual inspection, or iterative information seeking. In particular, VRAG-RL, Doc-V*, VISOR, and InSight-doc employ supervised trajectory learning followed by reinforcement learning, demonstrating that staged training of document interaction policies is already well established. VRAG-RL and InSight-doc also explicitly support fine-grained visual inspection, making them particularly relevant to the paper’s motivation concerning coarse-grained actions.
The issue is not simply missing citations. VRAG-RL and Doc-V* are already cited, but their implications for the proposed research gap are insufficiently discussed. In particular, the description of retrieval-based methods as relying on “static, similarity-driven retrieval” in lines 454–455 does not adequately characterize VRAG-RL’s adaptive interaction policy. ALDEN, VISOR, MM-Doc-R1, and InSight-doc are not discussed, and none of these six methods appears in the main experimental comparison, although a fine-tuned DocDancer baseline is included.
These precedents do not establish that DocPilot’s specific single-ability-to-multi-ability SFT curriculum is redundant: SFT followed by RL is a different training strategy. They do, however, require a more precise account of what is new in the action design, supervision decomposition, and training order. The authors should position these contributions against representative trained document agents and include an appropriate empirical comparison under consistent evaluation settings, with differences in backbone, document access, preprocessing, and inference budget clearly reported.
- Important uncertainties in data provenance and evaluation settings limit the interpretation of generalization and efficiency claims.
The multi-ability training data include MMDocRAG, which uses the MMDocIR document corpus and adapts some of its QA pairs. MMDocIR’s evaluation documents and questions originate from MMLongBench-Doc and DocBench, two principal evaluation benchmarks in this paper. The manuscript does not identify the selected source instances or describe cross-dataset document- and question-level overlap removal. This establishes a concrete need for an overlap audit, rather than proving that the actual training subset is contaminated. Without that information, the trained 4B model’s results cannot confidently be interpreted as generalization to held-out documents and questions.
MADQA is likewise used for trajectory synthesis and evaluation, but the training split and exclusion of the evaluated development questions are not explicitly documented. Its evaluation additionally provides relevant documents through an oracle-guided setting. The paper should clarify whether all baselines receive identical document access and distinguish reasoning over provided documents from end-to-end retrieval across a document collection. These concerns primarily affect the trained-model and cross-document generalization claims; they do not automatically invalidate the framework results with other backbones.
The interpretation of performance and efficiency also needs qualification. DocPilot-4B improves overall MMLongBench-Doc accuracy by 4.2 points, but unanswerable-question accuracy increases by 15.3 points while layout accuracy decreases by 9.8 points. Its FinRAGBench-V chart score also decreases by 6.2 points. Better answerability judgment is valuable, but the results do not demonstrate uniformly stronger multimodal reasoning. Uncertainty estimates for the main training comparisons would further help assess the reliability of the reported gains.
Finally, compact parameter count alone does not establish system efficiency. The paper should specify which checkpoints execute the orchestration and semantic tool modules, and report model calls, token usage, interaction budgets, and end-to-end latency, with document preprocessing costs either included or explicitly amortized.
[1] Wang et al. “VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning.” NeurIPS, 2025. arXiv:2505.22019.
[2] Yang et al. “ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents.” ACL, 2026. arXiv:2510.25668.
[3] Zheng et al. “Doc-V*: Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA.” 2026. arXiv:2604.13731.
[4] Shen et al. “VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning.” 2026. arXiv:2604.09508.
[5] Lin et al. “MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning.” 2026. arXiv:2604.13579.
[6] Li et al. “InSight-doc: Agentic Visual Perception for Long-Document Understanding.” 2026. arXiv:2608.10628.
Questions for authors
-
How does DocPilot differ technically from VRAG-RL, ALDEN etc beyond its expanded action inventory? In particular, how should the research-gap discussion account for existing adaptive navigation and fine-grained visual interaction? Can the authors provide a representative trained-agent comparison under consistent document access, evaluation metrics, and inference budgets?
-
Which exact splits and instances from MMDocRAG and MADQA were used for trajectory synthesis? Please provide document/question identifiers and overlap checks against MMLongBench-Doc, DocBench, and the evaluated MADQA development set. If overlaps are present, how do the results change after excluding them and retraining?
-
How does the proposed curriculum compare with mixed-data SFT using the same combined dataset and a matched training budget? What performance does the original Qwen3.5-4B achieve within the identical DocPilot framework before fine-tuning? These comparisons would help distinguish the effects of additional supervision, curriculum ordering, and orchestration.
-
For MADQA, do all compared systems receive the same oracle-provided documents and comparable inference budgets? Which baseline results were reproduced under this setting, and which were obtained from prior reports?
-
In DocPilot-4B, which model or checkpoint executes each semantic module? What are the total model calls, token usage, and end-to-end latency per question, and how are preprocessing costs accounted for?
4 comments
2. Isolating the training contribution [W1, W2, Q3]
We agree that the original comparison with Multi-only SFT changes both the training data and the training schedule, making it difficult to attribute the gains to curriculum ordering. To address this concern, we added a One-stage Training baseline in Table 3. It mixes the same single- and multi-ability examples in one SFT stage under a matched training budget. The curriculum presents those examples in two stages, from single-ability training to multi-ability training.
| Training setting | MMLongBench-Doc ANS | MMLongBench-Doc UNA | MMLongBench-Doc Overall | FinRAGBench-V Overall |
|---|---|---|---|---|
| Multi-only SFT | 52.3 | 74.4 | 56.8 | 58.4 |
| One-stage Training | 53.4 | 74.0 | 57.7 | 58.6 |
| DocPilot-4B (Single → Multi) | 56.3 | 70.0 | 59.1 | 60.2 |
Under this matched comparison, the curriculum yields modest overall gains of 1.4 and 1.6 percentage points on MMLongBench-Doc and FinRAGBench-V, respectively. On MMLongBench-Doc, accuracy rises from 53.4% to 56.3% on answerable questions and falls from 74.0% to 70.0% on unanswerable questions. This decline may reflect a greater tendency to answer despite insufficient evidence, motivating stronger grounding checks.
3. Data provenance and MADQA protocol [W2, Q2, Q4]
Thank you for raising this concern. Before synthesizing trajectories from the MMDocRAG training and development splits, we excluded 114 questions that overlapped with MMLongBench-Doc and 134 that overlapped with DocBench. For MADQA, we used only the official training split for trajectory synthesis and evaluated on the separate 200-question development split. In the revised manuscript, we will expand Appendix C (Trajectory Synthesis Details) to provide more detail on the source datasets, split selection, and data processing.
4. Action Calls and Inference Latency [W2, Q5]
- Action calls across backbones (Figure 7). Figure 7 reports the average number of action calls per question for each backbone evaluated within DocPilot.
- Tool use under different training strategies (Table 10). On MMLongBench-Doc, curriculum-trained DocPilot-4B uses nearly the same number of calls per question as multi-ability-only fine-tuning (16.40 vs. 16.43). It makes fewer
<Navigate>calls (3.75 → 2.87) and more<Focus>(0.74 → 1.46) and<Solve>calls (0.71 → 0.85), allocating more of its interactions to local inspection and executable reasoning.
Inference latency. We further measured mean inference latency in seconds per question for direct answering and DocPilot. To build DocPilot-4B, we use trajectories generated by Qwen3.5-397B-A17B to transfer reasoning and tool-use behavior to the reasoning-capable Qwen3.5-4B backbone. DocPilot-4B reaches 59.1% accuracy on MMLongBench-Doc, up from 54.9% with direct answering. At inference, the agent spends additional time interacting with the document environment through multiple rounds of evidence gathering, tool execution, and reasoning. Its latency is approximately 5.5× that of direct answering on MMLongBench-Doc and 5.6× on FinRAGBench-V. When both models run within DocPilot, the 4B agent’s mean latency is 73.6% lower than its teacher’s on MMLongBench-Doc and 62.5% lower on FinRAGBench-V.
| Configuration | MMLongBench-Doc (s) | FinRAGBench-V (s) |
|---|---|---|
| Vanilla Qwen3.5-4B | 31.60 | 65.53 |
| DocPilot-4B | 174.93 | 369.27 |
| Vanilla GPT-5.4 | 29.21 | 61.87 |
| DocPilot w/ GPT-5.4 | 151.91 | 333.39 |
| Vanilla Qwen3.5-397B-A17B | 125.68 | 189.53 |
| DocPilot w/ Qwen3.5-397B-A17B | 662.23 | 983.84 |
| Vanilla Gemini-3.1-Pro | 24.80 | 27.48 |
| DocPilot w/ Gemini-3.1-Pro | 126.27 | 146.19 |
Latency by action ablation. We also measured latency for DocPilot with GPT-5.4 on MMLongBench-Doc. The <Navigate> + <Answer> configuration has the lowest reported latency and reaches 64.5% accuracy in Figure 5. Full DocPilot reaches 73.5% with a longer mean inference time, illustrating the accuracy–latency trade-off. The ablation rows report filtered means on their respective evaluation subsets. The full-system row uses the mean from the backbone comparison above.
| DocPilot w/ GPT-5.4 configuration | Mean latency (s) |
|---|---|
<Navigate> + <Answer> | 81.62 |
w/o <Analyze> | 127.30 |
w/o <Focus> | 151.24 |
w/o <Solve> and code execution | 149.65 |
| Full DocPilot w/ GPT-5.4 | 151.91 |
Thank you for the substantial revisions and additional experiments. These address several of my original concerns, particularly the curriculum control, trained-agent comparisons, and reporting of data provenance and latency.
However, my overall assessment remains limited by the modest methodological novelty and contribution. The proposed framework primarily integrates established capabilities and trains their orchestration through a two-stage SFT curriculum. While this integration has practical value, the response does not sufficiently establish a distinct technical advance or broader insight beyond the combination of these components. The added experiments strengthen the empirical support for the system, but do not substantially change my assessment of its contribution. I therefore maintain my original rating.
Dear Reviewer VEzF,
Thank you for carefully considering our revisions and additional experiments. We appreciate your thoughtful feedback throughout the discussion, which has helped us clarify the paper’s contribution and strengthen its empirical evaluation.
Thank you again for your time and effort in reviewing our work.
Best regards,
Authors
1. Positioning and trained-agent comparisons [W1, Q1]
We compare DocPilot with existing document QA methods along four design dimensions. The table summarizes the capabilities of each method in its reported configuration.
Parsed Text at Inference. “Yes” indicates access to OCR or PDF-extracted text, including cached text. “No” indicates image-only access, and “—” indicates insufficient information. For the definitions and scoring criteria of the first four dimensions, please see our response to Reviewer mauh.
Doc‑V* and DocAtlas support stepwise action selection, VRAG‑RL and InSight‑doc support on-demand region inspection, and DocAgent and MACT support recovery after verification. Doc-V* and VRAG-RL train multi-turn policies that use external retrievers at inference. DocPilot brings stepwise action selection, local inspection, and verification-driven recovery into a shared interaction loop. It combines OCR-derived text with visual evidence and uses verification feedback to guide further evidence gathering or recomputation. Through curriculum training, a single 4B model learns to orchestrate these actions end to end.
Shared-backbone comparison. With Qwen2.5-VL-7B-Instruct, DocPilot scores 42.9 on MMLongBench-Doc and 79.8 on SlideVQA, exceeding Doc-V* (GRPO), the strongest baseline in the 7B group, by 0.8 and 2.6 points, respectively. With Qwen3-VL-8B-Instruct, DocPilot reaches 59.5 on MMLongBench-Doc, 1.0 point above InSight-doc (SFT+RL).
Evaluation settings. The MMLongBench-Doc scores for the Qwen3-VL-8B-Instruct baseline and both InSight-doc variants are taken from Table 16 of InSight-doc, using full-document input without a page cap and the high initial resolution setting (r=0.7, 140 DPI). R-f denotes operation without an external page or document retriever. Itr. and Rgn. denote multi-turn and region-level evidence acquisition, respectively. PTA denotes access to OCR-derived, PDF-extracted, or cached text at inference. A dash indicates a missing score.