acceptodds
Under review as a conference paper at ICLR 2027

DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding

Abstract

Multimodal long-document question answering requires models to seek sparse evidence across pages and modalities, understand heterogeneous document content, and reason over the collected evidence. Even advanced vision-language models (VLMs) can overlook sparse evidence and struggle with cross-modal reasoning in a single pass. Existing document agents further introduce multi-step interaction, but some still rely on predefined workflows or coarse-grained actions, limiting the flexibility and granularity of document reasoning. We introduce DocPilot, an agentic document framework that not only supports expressive orchestration but also makes the complex document-solving process learnable by small language models (SLMs). DocPilot first orchestrates the following capabilities: evidence seeking through page-level navigation and fine-grained evidence localization; and multimodal understanding through local visual inspection, code-based numerical reasoning, and cross-modal reasoning. To transfer these capabilities to small models, we propose curriculum-based agentic training that progresses from single-ability learning to adaptive orchestration through supervision from data-grounded reasoning traces and executable interaction trajectories. Across MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA, DocPilot achieves strong gains with frontier VLM backbones, while DocPilot-4B delivers competitive performance against substantially larger models, demonstrating strong effectiveness and generalization across diverse long-document reasoning settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Question Answering, LLM Agents, Agentic Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VEzFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DocPilot, an interactive agent framework for long visual document question answering. The framework organizes document interaction around six actions: Navigate, Extract, Focus, Solve, Analyze, and Answer. These actions support iterative evidence acquisition, local visual inspection, code-based computation, and answer verification. The authors also construct teacher-generated supervision and train a compact 4B model through two stages of supervised fine-tuning: single-ability training followed by multi-ability trajectory training. Experiments cover multiple backbone models and document QA benchmarks, supplemented by action ablations, training comparisons, and evidence-localization analyses.

Critical strengths
  1. The framework demonstrates substantial practical gains across multiple backbones and benchmarks. For example, with Gemini-2.5-Pro, DocPilot improves MMLongBench-Doc accuracy from 58.1 to 72.8 and exceeds the reported DocLens result of 67.6. Evaluating both frontier models and a compact trained model provides useful evidence about the framework’s applicability across model scales.

  2. The action ablations and evidence-localization analyses offer meaningful support for the system design. Removing Focus reduces chart and figure accuracy by 7.2 and 11.9 points, respectively, while removing Solve reduces table accuracy by 4.6 points. These results connect particular tools to the capabilities they are intended to support. The additional page-retrieval and element-localization evaluations also help explain the final-answer improvements.

Critical weaknesses
  1. The positioning against existing trainable document agents is incomplete, and the experiments do not sufficiently isolate the proposed training contribution.

The discussion in lines 087–097 does not adequately situate DocPilot within the broader literature on learning interactive policies for visually rich document understanding. Relevant work extends beyond VRAG-RL [1] and ALDEN [2] to Doc-V* [3], VISOR [4], MM-Doc-R1 [5], and InSight-doc [6]. These methods investigate learned evidence acquisition, document navigation, visual inspection, or iterative information seeking. In particular, VRAG-RL, Doc-V*, VISOR, and InSight-doc employ supervised trajectory learning followed by reinforcement learning, demonstrating that staged training of document interaction policies is already well established. VRAG-RL and InSight-doc also explicitly support fine-grained visual inspection, making them particularly relevant to the paper’s motivation concerning coarse-grained actions.

The issue is not simply missing citations. VRAG-RL and Doc-V* are already cited, but their implications for the proposed research gap are insufficiently discussed. In particular, the description of retrieval-based methods as relying on “static, similarity-driven retrieval” in lines 454–455 does not adequately characterize VRAG-RL’s adaptive interaction policy. ALDEN, VISOR, MM-Doc-R1, and InSight-doc are not discussed, and none of these six methods appears in the main experimental comparison, although a fine-tuned DocDancer baseline is included.

These precedents do not establish that DocPilot’s specific single-ability-to-multi-ability SFT curriculum is redundant: SFT followed by RL is a different training strategy. They do, however, require a more precise account of what is new in the action design, supervision decomposition, and training order. The authors should position these contributions against representative trained document agents and include an appropriate empirical comparison under consistent evaluation settings, with differences in backbone, document access, preprocessing, and inference budget clearly reported.

  1. Important uncertainties in data provenance and evaluation settings limit the interpretation of generalization and efficiency claims.

The multi-ability training data include MMDocRAG, which uses the MMDocIR document corpus and adapts some of its QA pairs. MMDocIR’s evaluation documents and questions originate from MMLongBench-Doc and DocBench, two principal evaluation benchmarks in this paper. The manuscript does not identify the selected source instances or describe cross-dataset document- and question-level overlap removal. This establishes a concrete need for an overlap audit, rather than proving that the actual training subset is contaminated. Without that information, the trained 4B model’s results cannot confidently be interpreted as generalization to held-out documents and questions.

MADQA is likewise used for trajectory synthesis and evaluation, but the training split and exclusion of the evaluated development questions are not explicitly documented. Its evaluation additionally provides relevant documents through an oracle-guided setting. The paper should clarify whether all baselines receive identical document access and distinguish reasoning over provided documents from end-to-end retrieval across a document collection. These concerns primarily affect the trained-model and cross-document generalization claims; they do not automatically invalidate the framework results with other backbones.

The interpretation of performance and efficiency also needs qualification. DocPilot-4B improves overall MMLongBench-Doc accuracy by 4.2 points, but unanswerable-question accuracy increases by 15.3 points while layout accuracy decreases by 9.8 points. Its FinRAGBench-V chart score also decreases by 6.2 points. Better answerability judgment is valuable, but the results do not demonstrate uniformly stronger multimodal reasoning. Uncertainty estimates for the main training comparisons would further help assess the reliability of the reported gains.

Finally, compact parameter count alone does not establish system efficiency. The paper should specify which checkpoints execute the orchestration and semantic tool modules, and report model calls, token usage, interaction budgets, and end-to-end latency, with document preprocessing costs either included or explicitly amortized.

[1] Wang et al. “VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning.” NeurIPS, 2025. arXiv:2505.22019.

[2] Yang et al. “ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents.” ACL, 2026. arXiv:2510.25668.

[3] Zheng et al. “Doc-V*: Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA.” 2026. arXiv:2604.13731.

[4] Shen et al. “VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning.” 2026. arXiv:2604.09508.

[5] Lin et al. “MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning.” 2026. arXiv:2604.13579.

[6] Li et al. “InSight-doc: Agentic Visual Perception for Long-Document Understanding.” 2026. arXiv:2608.10628.

Questions for authors
  1. How does DocPilot differ technically from VRAG-RL, ALDEN etc beyond its expanded action inventory? In particular, how should the research-gap discussion account for existing adaptive navigation and fine-grained visual interaction? Can the authors provide a representative trained-agent comparison under consistent document access, evaluation metrics, and inference budgets?

  2. Which exact splits and instances from MMDocRAG and MADQA were used for trajectory synthesis? Please provide document/question identifiers and overlap checks against MMLongBench-Doc, DocBench, and the evaluated MADQA development set. If overlaps are present, how do the results change after excluding them and retraining?

  3. How does the proposed curriculum compare with mixed-data SFT using the same combined dataset and a matched training budget? What performance does the original Qwen3.5-4B achieve within the identical DocPilot framework before fine-tuning? These comparisons would help distinguish the effects of additional supervision, curriculum ordering, and orchestration.

  4. For MADQA, do all compared systems receive the same oracle-provided documents and comparable inference budgets? Which baseline results were reproduced under this setting, and which were obtained from prior reports?

  5. In DocPilot-4B, which model or checkpoint executes each semantic module? What are the total model calls, token usage, and end-to-end latency per question, and how are preprocessing costs accounted for?

4 comments

1. Positioning and trained-agent comparisons [W1, Q1]

We compare DocPilot with existing document QA methods along four design dimensions. The table summarizes the capabilities of each method in its reported configuration.

MethodFlexible InteractionVisual Evidence GranularityVerification and RecoveryE2E SLM AgentParsed Text at Inference
M3DocRAG0000No
DocAgent1120Yes
SimpleDoc1000Yes
DocLens1110Yes
DocDancer2001Yes
Doc‑V*2001No
MACT1—21—
DocAtlas2101Yes
VRAG‑RL2201No
InSight‑doc2202No
DocPilot2222Yes

Parsed Text at Inference. “Yes” indicates access to OCR or PDF-extracted text, including cached text. “No” indicates image-only access, and “—” indicates insufficient information. For the definitions and scoring criteria of the first four dimensions, please see our response to Reviewer mauh.

Doc‑V* and DocAtlas support stepwise action selection, VRAG‑RL and InSight‑doc support on-demand region inspection, and DocAgent and MACT support recovery after verification. Doc-V* and VRAG-RL train multi-turn policies that use external retrievers at inference. DocPilot brings stepwise action selection, local inspection, and verification-driven recovery into a shared interaction loop. It combines OCR-derived text with visual evidence and uses verification feedback to guide further evidence gathering or recomputation. Through curriculum training, a single 4B model learns to orchestrate these actions end to end.

Shared-backbone comparison. With Qwen2.5-VL-7B-Instruct, DocPilot scores 42.9 on MMLongBench-Doc and 79.8 on SlideVQA, exceeding Doc-V* (GRPO), the strongest baseline in the 7B group, by 0.8 and 2.6 points, respectively. With Qwen3-VL-8B-Instruct, DocPilot reaches 59.5 on MMLongBench-Doc, 1.0 point above InSight-doc (SFT+RL).

MethodR-fItr.Rgn.PTAMMLongBench-DocSlideVQA
Qwen2.5-VL-7B-Instruct
Baseline✓✗✗✗28.055.2
RAG Top-5✗✗✗✗36.162.9
VRAG-RL✗✓✓✗26.6—
DocSeeker✓✗✗✗40.177.1
Doc-V* (SFT)✗✓✗✗39.873.8
Doc-V* (GRPO)✗✓✗✗42.177.2
DocPilot✓✓✓✓42.979.8
Qwen3-VL-8B-Instruct
Baseline✓✗✗✗51.1—
InSight-doc (SFT)✓✓✓✗46.7—
InSight-doc (SFT+RL)✓✓✓✗58.5—
DocPilot✓✓✓✓59.5—

Evaluation settings. The MMLongBench-Doc scores for the Qwen3-VL-8B-Instruct baseline and both InSight-doc variants are taken from Table 16 of InSight-doc, using full-document input without a page cap and the high initial resolution setting (r=0.7r=0.7, 140 DPI). R-f denotes operation without an external page or document retriever. Itr. and Rgn. denote multi-turn and region-level evidence acquisition, respectively. PTA denotes access to OCR-derived, PDF-extracted, or cached text at inference. A dash indicates a missing score.

2. Isolating the training contribution [W1, W2, Q3]

We agree that the original comparison with Multi-only SFT changes both the training data and the training schedule, making it difficult to attribute the gains to curriculum ordering. To address this concern, we added a One-stage Training baseline in Table 3. It mixes the same single- and multi-ability examples in one SFT stage under a matched training budget. The curriculum presents those examples in two stages, from single-ability training to multi-ability training.

Training settingMMLongBench-Doc ANSMMLongBench-Doc UNAMMLongBench-Doc OverallFinRAGBench-V Overall
Multi-only SFT52.374.456.858.4
One-stage Training53.474.057.758.6
DocPilot-4B (Single → Multi)56.370.059.160.2

Under this matched comparison, the curriculum yields modest overall gains of 1.4 and 1.6 percentage points on MMLongBench-Doc and FinRAGBench-V, respectively. On MMLongBench-Doc, accuracy rises from 53.4% to 56.3% on answerable questions and falls from 74.0% to 70.0% on unanswerable questions. This decline may reflect a greater tendency to answer despite insufficient evidence, motivating stronger grounding checks.

3. Data provenance and MADQA protocol [W2, Q2, Q4]

Thank you for raising this concern. Before synthesizing trajectories from the MMDocRAG training and development splits, we excluded 114 questions that overlapped with MMLongBench-Doc and 134 that overlapped with DocBench. For MADQA, we used only the official training split for trajectory synthesis and evaluated on the separate 200-question development split. In the revised manuscript, we will expand Appendix C (Trajectory Synthesis Details) to provide more detail on the source datasets, split selection, and data processing.

4. Action Calls and Inference Latency [W2, Q5]

  • Action calls across backbones (Figure 7). Figure 7 reports the average number of action calls per question for each backbone evaluated within DocPilot.
  • Tool use under different training strategies (Table 10). On MMLongBench-Doc, curriculum-trained DocPilot-4B uses nearly the same number of calls per question as multi-ability-only fine-tuning (16.40 vs. 16.43). It makes fewer <Navigate> calls (3.75 → 2.87) and more <Focus> (0.74 → 1.46) and <Solve> calls (0.71 → 0.85), allocating more of its interactions to local inspection and executable reasoning.

Inference latency. We further measured mean inference latency in seconds per question for direct answering and DocPilot. To build DocPilot-4B, we use trajectories generated by Qwen3.5-397B-A17B to transfer reasoning and tool-use behavior to the reasoning-capable Qwen3.5-4B backbone. DocPilot-4B reaches 59.1% accuracy on MMLongBench-Doc, up from 54.9% with direct answering. At inference, the agent spends additional time interacting with the document environment through multiple rounds of evidence gathering, tool execution, and reasoning. Its latency is approximately 5.5× that of direct answering on MMLongBench-Doc and 5.6× on FinRAGBench-V. When both models run within DocPilot, the 4B agent’s mean latency is 73.6% lower than its teacher’s on MMLongBench-Doc and 62.5% lower on FinRAGBench-V.

ConfigurationMMLongBench-Doc (s)FinRAGBench-V (s)
Vanilla Qwen3.5-4B31.6065.53
DocPilot-4B174.93369.27
Vanilla GPT-5.429.2161.87
DocPilot w/ GPT-5.4151.91333.39
Vanilla Qwen3.5-397B-A17B125.68189.53
DocPilot w/ Qwen3.5-397B-A17B662.23983.84
Vanilla Gemini-3.1-Pro24.8027.48
DocPilot w/ Gemini-3.1-Pro126.27146.19

Latency by action ablation. We also measured latency for DocPilot with GPT-5.4 on MMLongBench-Doc. The <Navigate> + <Answer> configuration has the lowest reported latency and reaches 64.5% accuracy in Figure 5. Full DocPilot reaches 73.5% with a longer mean inference time, illustrating the accuracy–latency trade-off. The ablation rows report filtered means on their respective evaluation subsets. The full-system row uses the mean from the backbone comparison above.

DocPilot w/ GPT-5.4 configurationMean latency (s)
<Navigate> + <Answer>81.62
w/o <Analyze>127.30
w/o <Focus>151.24
w/o <Solve> and code execution149.65
Full DocPilot w/ GPT-5.4151.91

Thank you for the substantial revisions and additional experiments. These address several of my original concerns, particularly the curriculum control, trained-agent comparisons, and reporting of data provenance and latency.

However, my overall assessment remains limited by the modest methodological novelty and contribution. The proposed framework primarily integrates established capabilities and trains their orchestration through a two-stage SFT curriculum. While this integration has practical value, the response does not sufficiently establish a distinct technical advance or broader insight beyond the combination of these components. The added experiments strengthen the empirical support for the system, but do not substantially change my assessment of its contribution. I therefore maintain my original rating.

Dear Reviewer VEzF,

Thank you for carefully considering our revisions and additional experiments. We appreciate your thoughtful feedback throughout the discussion, which has helped us clarify the paper’s contribution and strengthen its empirical evaluation.

Thank you again for your time and effort in reviewing our work.

Best regards,
Authors

Reviewer_rbXDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DocPilot, an interactive agent framework for multimodal long-document question answering. It combines fine-grained tools for evidence seeking, multimodal understanding, and iterative refinement. The authors further introduce curriculum-based training with synthesized trajectories to train a 4B model, demonstrating improvements across multiple document QA benchmarks.

Critical strengths
  1. The paper integrates multiple specialized tools for document navigation, evidence extraction, visual inspection, and numerical reasoning, providing a flexible framework for long-document QA.
  2. The paper evaluates DocPilot across four benchmarks and multiple model backbones, with ablation studies examining the contributions of individual tools and the training strategy.
Critical weaknesses
  1. The curriculum yields only modest improvements (+2.3 and +1.8 points), while the comparison with direct Multi-Ability SFT does not control for training data volume or compute budget. Thus, the benefits of the curriculum itself remain unclear.
  2. MADQA is used for both trajectory synthesis and generalization evaluation, but the paper does not clarify whether the splits are strictly separated, raising potential data leakage concerns.
  3. In Section 1 (Line 106), “A agentic document framework” should be corrected to “An agentic document framework.”
  4. DocPilot primarily combines standard ReAct-based tool orchestration with teacher-generated trajectory distillation and SFT for small models. While the integration is useful, the individual techniques are well established, and the paper provides limited evidence that its specific orchestration and training designs offer substantial methodological advances beyond their combination.
Questions for authors

See the weaknesses.

2 comments

1. Isolating the Effect of Curriculum Ordering [W1]

We agree that the original comparison with Multi-only SFT changes both the training data and the training schedule, making it difficult to attribute the gains to curriculum ordering. To address this concern, we added a One-stage Training baseline in Table 3. It mixes the same single- and multi-ability examples in one SFT stage under a matched training budget. The curriculum presents those examples in two stages, from single-ability training to multi-ability training.

Training settingMMLongBench-Doc ANSMMLongBench-Doc UNAMMLongBench-Doc OverallFinRAGBench-V Overall
Multi-only SFT52.374.456.858.4
One-stage Training53.474.057.758.6
DocPilot-4B (Single → Multi)56.370.059.160.2

Under this matched comparison, the curriculum yields modest overall gains of 1.4 and 1.6 percentage points on MMLongBench-Doc and FinRAGBench-V, respectively. On MMLongBench-Doc, accuracy rises from 53.4% to 56.3% on answerable questions and falls from 74.0% to 70.0% on unanswerable questions. This decline may reflect a greater tendency to answer despite insufficient evidence, motivating stronger grounding checks.

2. MADQA Split Separation [W2]

We used only MADQA’s training split for trajectory synthesis and evaluated on its 200-question development split. The synthesis and evaluation questions are disjoint. We will make this split explicit in the revised manuscript.

3. Typographical Correction [W3]

We will change “A agentic document framework” to “An agentic document framework” in the revised manuscript.

4. Evidence for the Orchestration and Training Designs [W4]

DocPilot exposes navigation, extraction, local inspection, executable reasoning, and verification as distinct actions that the agent can interleave as evidence accumulates. Its curriculum teaches individual actions before training the model to compose them across complete interaction trajectories.

  • Action ablations (Figure 5). With GPT-5.4 fixed, <Navigate> + <Answer> reaches 64.5%, the richer action set without <Analyze> reaches 64.9%, and full DocPilot reaches 73.5%. Removing <Focus> reduces chart and figure accuracy by 7.2 and 11.9 percentage points, respectively. Removing <Solve> reduces table accuracy by 4.6 points.
  • Tool use (Table 8). Compared with direct multi-ability SFT, curriculum-trained DocPilot-4B makes nearly the same number of calls at inference (16.43 → 16.41), with fewer <Navigate> calls (3.75 → 2.87) and more <Focus> (0.74 → 1.46) and <Solve> calls (0.71 → 0.85).
  • Case studies (Figures 10–13). The trajectories illustrate how these actions work together. <Focus> resolves a fine-grained chart detail, <Solve> combines values from a chart and a table, and <Analyze> redirects the agent after a failed computation or prompts it to seek evidence across documents.
  • Shared-backbone comparison. With Qwen2.5-VL-7B-Instruct as the backbone, DocPilot scores 42.9 on MMLongBench-Doc and 79.8 on SlideVQA, exceeding Doc-V* (GRPO), the strongest baseline in this comparison, by 0.8 and 2.6 points, respectively. The table reports the evidence-access settings of each system.
MethodR-fItr.Rgn.PTAMMLongBench-DocSlideVQA
Baseline✓✗✗✗28.055.2
RAG Top-5✗✗✗✗36.162.9
VRAG-RL✗✓✓✗26.6—
DocSeeker✓✗✗✗40.177.1
Doc-V* (SFT)✗✓✗✗39.873.8
Doc-V* (GRPO)✗✓✗✗42.177.2
DocPilot✓✓✓✓42.979.8

R-f denotes operation without an external page or document retriever. Itr. and Rgn. denote multi-turn and region-level evidence acquisition, respectively. PTA denotes access to OCR-derived, PDF-extracted, or cached text at inference. A dash indicates a missing score.

Dear Reviewer rbXD,

We hope this message finds you well. As we approach the end of the discussion period, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,
Authors

Reviewer_mauhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

DocPilot helps models answer questions about long documents containing text, tables, charts, and images. It adds an agent layer that can navigate pages, inspect details, extract evidence, run calculations, and check answers. The authors also train a smaller 4B model to use these capabilities, first teaching individual skills and then teaching it to combine them. The experiments (of the agentic layer on top of existing models) show substantial improvements over the base model.

Critical strengths

DocPilot appears to be an effective agent for document understanding. It brings together capabilities that address practical difficulties in reading long documents, including finding scattered evidence, inspecting small visual details, and checking numerical answers. The substantial improvements across several strong models suggest that the framework has value across different backbones.

I also appreciate the analyses of evidence selection and individual components. These help explain how the system works and support the usefulness of local visual inspection and answer verification.

Critical weaknesses

My main concern is that the central research contribution remains unclear. I would expect an agent with useful tools and repeated opportunities to inspect evidence to outperform the same LLM answering directly. Building such an agent for document understanding is valuable, and DocPilot seems to work well. However, I would like a clearer explanation of what is new about this particular design and what the results teach us beyond the established benefits of tools and iterative reasoning. Comparisons with existing document agents are helpful, but the paper should explain which specific advances account for its advantage.

The smaller model introduces a second narrative that makes the contribution harder to assess. Its results are impressive as a demonstration that a compact model can operate within DocPilot. They do not establish a separate advance in the model’s standalone capabilities, since it is evaluated with the surrounding framework and tools. I would treat this as supporting evidence for the agent framework. Presenting it as another major contribution leaves me uncertain about the paper’s main claim.

Questions for authors

What is the main contribution beyond combining established agent capabilities for document understanding? Which design choices distinguish DocPilot from existing document agents, and which comparisons best establish their value?

What should readers conclude from the human comparison in Table 1? Could you report agent performance on the same subset used for the human evaluation and clarify the evaluation conditions? (agent surpasses human eval on accuracy and f1)

3 comments

DocPilot makes two main contributions. (i) State-conditioned orchestration of fine-grained document actions lenables the agent to adaptively interleave evidence seeking, local visual inspection, and verifiable reasoning within a unified multimodal document environment. (ii) Curriculum training for a compact end-to-end document agent teaches a 4B VLM to execute individual actions and then coordinate them across complete document interactions.

1. Main Contributions and Evidence for the Orchestration Design [W1, Q1]

We first address contribution (i), our state-conditioned orchestration design.

Task motivation. Long visual document QA requires locating evidence across pages and interpreting it across modalities. Reading a page, inspecting a figure, or attempting a calculation can reveal what evidence is needed next.

Design motivation. Fixed pipelines prescribe when retrieval gives way to reasoning, while coarse Search/Read interfaces limit control over local inspection and computation. DocPilot exposes these operations as distinct actions that the agent can interleave as intermediate observations and verification feedback update a shared evidence state. For example, if <Analyze> identifies a missing denominator, the agent can retrieve it, rerun <Solve>, and verify the revised answer.

We compare DocPilot with existing document QA methods along four design dimensions. The table summarizes the capabilities of each method in its reported configuration.

MethodFlexible InteractionVisual Evidence GranularityVerification and RecoveryE2E SLM Agent
M3DocRAG [1]0000
DocAgent [2]1120
SimpleDoc [3]1000
DocLens [4]1110
DocDancer [5]2001
Doc-V* [6]2001
MACT [7]1-21
DocAtlas [8]2101
DocPilot2222

Doc‑V* and DocAtlas support stepwise action selection, and DocAgent and MACT support recovery after verification. The trained SLM baselines in this table rely on auxiliary components at inference, including an external retriever in Doc‑V*, a separate reader model in DocDancer, and auxiliary LLMs in DocAtlas. DocPilot combines stepwise action selection, on-demand subregion inspection, and verification-driven recovery in a shared interaction loop that integrates OCR-derived content and visual evidence. A single 4B model, trained with our curriculum, orchestrates the full loop end to end.

We assign the highest applicable level for each dimension using the definitions below. A dash indicates insufficient detail to assign a level.

Dimension012
Flexible InteractionFixed execution orderAdaptive decisions within a fixed workflowStepwise choice among action types in a shared loop
Visual Evidence GranularityPage- or section-level accessAccess to detected document elementsOn-demand inspection of agent-selected subregions
Verification and RecoveryNo explicit verificationVerification or adjudication over existing evidenceVerification can trigger further evidence gathering or re-execution
End-to-End SLMNo task-specific SLM trainingTask-specific SLM training or with an auxiliary LLM moduleFull-loop training

Which comparisons establish its value? Fig. 5 compares variants that already use tools. With GPT-5.4 fixed, <Navigate> + <Answer> reaches 64.5%, the richer tool set without <Analyze> reaches 64.9%, and full DocPilot reaches 73.5%. This shows that the richer tool set alone recovers little of the full gain. Removing <Focus> reduces chart and figure accuracy by 7.2 and 11.9 points, respectively, showing where local inspection helps. Table 3 provides a complementary comparison against DocLens with Gemini-2.5-Pro. DocPilot selects fewer pages on average (3.0 vs. 3.5) and achieves higher page precision (73.2% vs. 55.1%) and QA accuracy (72.8% vs. 67.6%), despite lower page recall.

2. Curriculum Training for DocPilot-4B [W2]

Design motivation. Fine-grained orchestration is challenging for off-the-shelf compact VLMs. They must execute individual actions, interpret tool outputs, and decide what to do next across multiple steps. On MMLongBench-Doc, vanilla Qwen3.5-4B achieves 35.21% (381/1,082) accuracy with DocPilot, below the 54.9% it achieves when answering directly. Failure modes include exhausting the step budget before returning a valid answer, producing outputs that the framework cannot parse, failed action executions, and incomplete evidence gathering.

This drop suggests that the base small model struggles with the complex skills required for multi-step document QA. These challenges motivate our curriculum (ii), which first teaches the model to execute individual actions and then trains it to coordinate them across complete interaction trajectories. The resulting DocPilot-4B uses a single trained VLM to carry out the full interaction loop introduced in (i), from evidence seeking to verification and answering. (We will update our introduction in the revised manuscript.)

System performance. DocPilot-4B improves over vanilla Qwen3.5-4B by 4.2, 7.5, and 11.0 points on MMLongBench-Doc, FinRAGBench-V, and DocBench, respectively (Table 1).

Training strategyMMLongBench-Doc (ANS)MMLongBench-Doc (UNA)MMLongBench-Doc (Overall)FinRAGBench-V
DocPilot-4B56.370.059.160.2
One-stage Training53.474.057.758.6
Only Multi-ability SFT52.374.456.858.4

Effect of the curriculum. Curriculum training outperforms both one-stage training and direct multi-ability SFT on MMLongBench-Doc and FinRAGBench-V (Table 3). Appendix D.2 shows a shift toward local inspection and code-based reasoning with similar total calls (16.41 vs. 16.43), suggesting better capability coordination.

3. Human Comparison in Table 1 [Q2]

Following the original benchmark paper, we included the published human scores in Table 1 as a reference for task difficulty. We apologize that this presentation gave the impression of a matched human–agent comparison. MMLongBench-Doc [9] report 65.8% accuracy and 66.0% F1 on 238 questions from 29 documents, while DocPilot was evaluated on all 1,082 questions. In the revision, we will label the human row with its subset size and distinguish it from the full-benchmark results.

The benchmark paper specifies the subset size, but we could not identify the exact questions from the paper or the public data release. If we obtain it, we will report DocPilot’s accuracy and F1 on those items using the benchmark’s scoring protocol and clarify the human evaluation conditions where they can be confirmed.


References:

[1] M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding (arXiv 2024)

[2] DocAgent: An Agentic Framework for Multi-Modal Long-Context Document Understanding (EMNLP 2025)

[3] SimpleDoc: Multi-Modal Document Understanding with Dual-Cue Page Retrieval and Iterative Refinement (EMNLP 2025)

[4] DocLens: A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding (ACL 2026)

[5] Towards Agentic Document-Grounded Information Seeking (EMNLP 2026)

[6] Doc-V*: Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA (ACL 2026)

[7] Visual Document Understanding and Reasoning: A Multi-Agent Collaboration Framework with Agent-Wise Adaptive Test-Time Scaling (CVPR 2026)

[8] DocAtlas: Long-Document Understanding as Mutable-State Interaction (NeurIPS 2026)

[9] MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024)

Dear Reviewer mauh,

We hope this message finds you well. As we approach the end of the discussion period, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.