acceptodds
Under review as a conference paper at ICLR 2027

DocTrek: Tree-Guided Agentic Reinforcement Learning via Dense Rewards for Document Understanding

Abstract

Multimodal long-document question answering requires reasoning over hierarchical structures and heterogeneous content across many pages. Existing approaches discard such structures, use the hierarchy only as static retrieval units, or train the policy over flat sequences with trajectory-level feedback. We present DocTrek, which couples a section-structured document tree with token-position dense rewards for agentic reinforcement learning. A parsing–construction–integration pipeline converts raw PDFs into a tree whose section skeleton hosts multimodal elements as typed leaves, providing an addressable RL environment. Within it, we train the navigation-and-answering policy with a reward decomposition that converts page-level gold-evidence coverage into rewards placed at action-end tokens, providing dense credit for evidence retrieval and answer generation. Across four benchmarks, DocTrek improves the average score by 2.3 points over the strongest baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
document visual question answering, retrieval-augmented generation, multi-hop reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CQD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DocTrek, an agentic RL framework for multimodal long-document QA. It constructs a hierarchical document tree with typed multimodal elements and trains an agent to navigate this structure using dense process rewards assigned at action-end token positions. Experiments on four benchmarks demonstrate improvements over existing RAG and agentic systems.

Critical strengths
  1. Formulating multimodal long-document QA as a Markov decision process over a document tree is an interesting idea.
  2. DocTrek surpasses the chosen baselines across four multimodal long-document QA benchmarks.
Critical weaknesses
  1. The experiments do not sufficiently validate the proposed method. Several baselines use different model backbones, making direct comparisons less conclusive. Also, the individual reward components are not adequately ablated, leaving their respective contributions unclear.
  2. The improvement over “w/o Dense Reward” is relatively small (from 0.446 to 0.448 in EM), while average tool calls increase substantially (from 18.43 to 33.20). This raises concerns about whether the modest effectiveness gains are worth the additional inference cost and whether the proposed method achieves a favorable effectiveness-efficiency trade-off.

Minor

  1. Citations should use \citep{} for better readability.
  2. The main paper could be more self-contained. For example, the experimental section should at least explicitly list the names of baselines rather than only describing categories and leaving all the details to the appendix.
Questions for authors
  1. Could the authors provide separate ablations for the two per-turn reward components RretR^{ret} and RpriR^{pri}? Further ablations of the individual shaping terms of RpriR^{pri} and sensitivity analyses of the reward coefficients would also help establish their contributions.
  2. Could the authors clarify whether “w/o Dense Reward” removes the process rewards entirely or simply places them at the final token? The descriptions in the main text and Table 2 caption appear inconsistent, and these two settings test different aspects of the proposed method.
2 comments

Thank you for recognizing the tree-based MDP and the results across four benchmarks. We believe the current evidence supports the combined system and its measured within-system components more strongly than the review's characterization of insufficient validation suggests.

  1. Do differing backbones make the results inconclusive? Table 1 evaluates complete systems under their published configurations and does not isolate backbone or hierarchy. Each baseline's prompts and tool-invocation protocol are closely coupled to the model used in its official implementation; replacing that model alone would change the method's operating conditions rather than faithfully reproduce its best-performing implementation. We therefore compare against the baselines in their official, best-performing configurations. Model size alone is an insufficient account of the pattern: DocTrek (Qwen3-VL-8B) reaches 0.562 average, compared with 0.455 for the 8B MDocAgent and 0.539 for GPT-4o-based DMAP. Against the latter it leads on MMLongBench-Doc (0.501 vs. 0.432), PaperTab (0.422 vs. 0.390), and FetaTab (0.750 vs. 0.725), although DMAP leads on LongDocURL. These are system-level comparisons, not backbone-controlled causal estimates. Separately, the within-DocTrek comparisons use the same main-agent setting: SFT reaches 0.482 accuracy, final-token reward placement 0.488, and action-end placement 0.501; Table 3 further shows 0.486 without cross-page merging and 0.414 without vector retrieval. Therefore the experimental support does not rest solely on comparing 8B with 7B external agents.

  2. Are the reward terms unvalidated? Equation (6) distinguishes the gold-evidence retrieval reward Rret from the heuristic shaping reward Rpri, with coefficients specified in Appendix C. Table 2 keeps these components fixed and changes their token positions, yielding 0.488 → 0.501 accuracy and the retrieval gains in Appendix E; it tests the paper's central claim about action-end credit assignment. The agent's training pipeline requires approximately 120 hours on 24 H20 GPUs. Under this training-time constraint, separately retraining variants for Rret, Rpri, each shaping term, and multiple coefficient settings is beyond the scope of the present study. We will investigate these individual contributions and coefficient sensitivity in future work.

  3. Is the effectiveness–cost trade-off unfavorable? The reviewer is right to distinguish EM (0.446 → 0.448) from accuracy (0.488 → 0.501) and tool calls (18.43 → 33.20). The gain is not restricted to a 0.2-point EM difference: the first ten retrieved evidence leaves improve by 9.13 points in recall and 1.52 points in precision, with improvement already visible at five leaves (Appendix E). This is evidence of better evidence localization at a fixed retrieval cutoff, while the end-task improvement and extra navigation are both reported transparently. It is not a claim that accuracy per call or latency is improved. The 124 s/query online cost is measured on eight H20 GPUs; the offline 13.8 s/page tree construction is reused across questions. Calling the trade-off categorically unfavorable from EM and calls alone overlooks the accuracy, early retrieval quality, and amortized preprocessing reported in the paper.

  4. What exactly does “w/o Dense Reward” test, and can the presentation be clarified? It retains all reward components but puts their accumulated value at the final token, as the Table 2 caption states. Thus it isolates action-end versus final-token placement, not the deletion of process rewards. We will replace the imprecise “terminal, outcome-only” phrase in §5.3 to match the actual experiment. We will also name the baseline methods in §5.1 and use parenthetical citations where appropriate; these presentation edits do not alter the results.

Thank you for the clarifications. I disagree that the official configurations of prior baselines necessarily represent their best-performing settings, as backbone choices are often constrained by model availability at the time of publication. However, given the inclusion of recent baselines and the substantial reproduction costs, I can accept the current comparisons. Also, the lack of ablations for individual reward components still remains a major concern. I understand the computational constraints during rebuttal, but they do not justify the absence of experiments essential to validating the proposed reward design and the effectiveness-efficiency trade-off of the method. Therefore, I will maintain my original rating.

Reviewer_EnYhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DocTrek addresses question answering over long multimodal documents by organizing document elements into a section tree and training an agent to retrieve evidence with dense rewards based on gold evidence pages. It achieves the highest average score among the compared systems across four benchmarks, improving over the strongest baseline by 2.3 points.

Critical strengths

The paper presents an interesting way to combine document structure with agent training. Evaluation on four benchmarks outside the training domains is a useful test of generalization.

Critical weaknesses
  1. The contribution of the document hierarchy is not isolated. Table 3 removes vector retrieval and cross-page merging, but neither experiment tests the section hierarchy itself. A flat representation using the same parsed elements, retrieval models, and reward signals would show how much tree navigation contributes.

  2. The dense-reward gains come with substantially more tool use. In Table 2, accuracy increases from 48.8% to 50.1%, while average tool calls rise from 18.43 to 33.20. Successful calls and backtracking also receive explicit rewards. A comparison at matched inference budgets is needed to determine whether dense rewards improve search efficiency or mainly encourage longer searches.

  3. Zero padding in advantage normalization can favor longer trajectories. Appendix D includes zero returns for completed trajectories when computing group statistics at later token positions. Even when all trajectories receive the same positive terminal reward, this gives positive advantages to tokens in longer trajectories after shorter ones have ended. The paper does not examine how this affects response length or tool use.

Questions for authors
  1. How does DocTrek perform with a flat document representation while keeping the other components fixed?
  2. Does the dense-reward advantage remain under matched tool-call or token budgets?
  3. Does “w/o Dense Reward” move all rewards to the final token, as Table 2 states, or remove process rewards, as §5.3 suggests?
  4. What changes when the bonuses for successful tool calls and backtracking are removed?
  5. How do accuracy and trajectory length change when completed trajectories are excluded from the normalization statistics?
2 comments

Thank you for recognizing the structure–training combination and the cross-domain evaluation. Several concerns distinguish a useful additional causal experiment from what the current controlled comparisons already establish.

  1. What is established about hierarchy without a flat-document baseline? We do not infer the isolated contribution of the section tree from Table 1's cross-system comparisons. Nevertheless, Table 3 is a within-DocTrek structural intervention: with vector retrieval still enabled, leaving cross-page elements unmerged lowers accuracy 0.501 → 0.486 and increases average calls 33.20 → 37.60. This directly contradicts the explanation that the retriever alone accounts for all gains from document organization. Removing vector retrieval separately yields 0.414 accuracy, so the two components are complementary. A same-elements flat-document ablation would further measure the contribution of section hierarchy itself; that narrower quantity is not what the existing cross-page ablation purports to estimate.

  2. Do dense rewards only encourage longer searches? The ablation keeps the reward components and changes where their accumulated rewards are placed: action ends versus the final response token (Table 2 caption). Action-end placement improves accuracy 0.488 → 0.501 and EM 0.446 → 0.448, while calls increase 18.43 → 33.20. Crucially, the first ten evidence leaves are better, not just more numerous: Recall@10 improves 59.72% → 68.85% and Precision@10 9.86% → 11.38% (Appendix E); the gains are also present at k=5. Extra tool use after the first ten retrieved leaves cannot by itself produce these top-ten differences. The reported result is therefore an improvement in early evidence selection alongside more navigation. A matched call/token-budget accuracy curve would answer a separate efficiency question and is not reported; the evidence here should not be mistaken for an accuracy-per-call claim.

  3. Does “w/o Dense Reward” delete the process terms? No. The Table 2 caption gives the actual setting: all reward components remain, and their accumulated value is deposited at the final response token. The treatment isolates the temporal placement of otherwise identical reward terms, precisely the comparison relevant to action-end credit assignment. The phrase “terminal, outcome-only” in §5.3 is an inaccurate shorthand for that setting; we will correct that sentence. The experiment, reward coefficients, and numerical results are unchanged.

  4. Could the tool-use and backtracking bonuses explain the extra calls? Appendix C specifies 0.01 for a successful tool call and 0.02 for backtracking, against 0.10 for each newly covered gold evidence page through expand_node. Evidence-page credit cannot be repeatedly earned for the same page, revisited nodes return no new content, and the main process-reward channel is capped at 0.30 per trajectory (the vector channel has a separate 0.10 cap). The 60-turn limit and a penalty for exhausting the iteration budget further constrain aimless exploration. Thus, repeated calls cannot accumulate unlimited reward. We will conduct separate ablations of each reward component in future work.

  5. Does zero padding in Appendix D invalidate the reward-placement comparison? The mathematical edge case is real: with identical positive terminal rewards, the zero-filled positions of completed trajectories can give a surviving trajectory positive normalized advantage at later positions. But the conclusion that this causes the observed accuracy and early-evidence gains does not follow. Both variants use the same group normalization, and the controlled change is reward placement; the first-ten-leaf precision and recall gains cannot be produced merely by adding leaves beyond position ten. The first-hit rule, capped process reward, no-content revisits, and iteration-exhaustion penalty also constrain incentives for redundant actions, though they do not eliminate every possible length preference. We will make the normalization behavior explicit. Excluding completed trajectories from group statistics defines a different training algorithm; we have not measured its accuracy or trajectory length and will not invent that result. The reported credit-assignment procedure is exactly the one in Appendix D.

Thank you for clarifying how the rewards were assigned. I agree that the agent finds better evidence early in the search. My concern is that reaching those early results can itself take more tool calls, so I still cannot tell whether the agent answers more accurately when both methods have the same search budget. Therefore, I will maintain my original rating

Reviewer_QgidOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DocTrek, which organizes multimodal PDF content using a section-based structure and enables an agent to gather evidence through tree navigation, node expansion, and vector retrieval. During training, it first applies supervised fine-tuning using filtered demonstration trajectories, then optimizes the primary agent via GRPO, and provides dense process rewards (such as evidence page hits) at the end of tool actions. The authors report an average score of 0.562 across four long-document question answering dataset, outperforming the strongest complete baseline in the table (0.539), conduct ablations on training stages, dense rewards, vector retrieval, and cross-page merging on MMLongBench-Doc.

Critical strengths
  1. The system demonstrates highly competitive performance across multiple benchmarks. As shown in Table 1, its average accuracy across four datasets surpasses the strongest baseline by 2.3 percentage points (leading in three of them), fully demonstrating the high research value of this combined approach.
  2. It provides detailed and reproducible ablation studies on the placement of rewards and the roles of core components. For instance, when placing the process reward at the end of actions, the accuracy significantly increases from 0.488 to 0.501, and Recall@10 rises sharply from 59.72% to 68.85%. In addition, the paper presents careful ablation analyses on cross page merging and vector retrieval.
Critical weaknesses
  1. Table 3 shows that removing vector retrieval results in an 8.7 percentage point drop. However, without experiments that maintain the same model, retriever, training data, and budget while merely replacing the section tree with flat pages or flat elements, it remains unclear whether the advantages stem from the tree, the strong retriever, the auxiliary model, or their combination.
  2. Compared to the terminal reward, its accuracy increases by only 1.3 percentage points and Exact Match by 0.2 percentage points, while the average tool calls surge from 18.43 to 33.20. Lacking a comparison of accuracy under the same call budget, it is difficult to determine whether the gains are stable and whether the increased inference cost is justified.
Questions for authors
  1. Can you provide a cross-comparison between "section tree / flat pages or elements" and "action-end rewards / terminal reward" under the same primary model, auxiliary model, retrieval index, training data, and tool-call budget?
  2. How do accuracy and Recall@10 change under the same number of tool calls or the same inference time?
  3. In Table 2, the sum of list_children, expand_node, and backtrack calls for DocTrek is 32.15, while the total number reported as covering six tools is 33.20. How are the remaining approximately 1.05 calls distributed among question decomposition, vector retrieval, and answer generation?
2 comments

Thank you for highlighting the cross-benchmark performance and the reward-placement and retrieval analyses. We address the three questions directly.

  1. Is the gain due to the section tree, the retriever, or the combination? DocTrek's stated contribution is the combination of an addressable document tree and action-end credit assignment, rather than a claim that the full system's margin over another agent equals the isolated effect of hierarchy. The evidence is stronger than a comparison of different backbones alone: in Table 3, disabling cross-page merging while retaining vector retrieval lowers accuracy from 0.501 to 0.486 and increases calls from 33.20 to 37.60. Thus, the improvement cannot be attributed solely to the presence of the vector retriever; a structural decision in the document representation improves both answer quality and navigation effort within our system. Conversely, disabling vector retrieval lowers accuracy to 0.414, confirming that global retrieval is also important. A flat-page/flat-element × reward-placement factorial experiment would identify the isolated section-hierarchy effect and its interaction with reward placement, but its absence does not negate the observed contributions of cross-page structure, retrieval, and the combined system. We will state this distinction explicitly rather than assign the entire Table 1 margin to the tree alone.

  2. Are the additional calls simply buying a marginal answer gain? Table 2 reports the cost honestly: relative to placing the same reward components at the final token, action-end placement changes accuracy from 0.488 to 0.501 and calls from 18.43 to 33.20. However, the result is not merely that more leaves were retrieved. At the same cutoff of the first ten retrieved leaves, Appendix E reports Recall@10 of 68.85% versus 59.72% and Precision@10 of 11.38% versus 9.86%; at five leaves, Recall@5 is 56.83% versus 52.93% and Precision@5 is 17.76% versus 16.65%. More gold evidence appears earlier and with higher purity in the ordered evidence pool. This fixed-evidence-cutoff comparison directly addresses the concern that the retrieval improvement arises only from appending more leaves after the tenth. It does not equate total navigation calls or wall-clock time, so we do not present it as an equal-cost accuracy result. The reported online latency is 124 s/query on eight H20 GPUs; the 13.8 s/page construction is performed once per document and reused across queries (amortized cost 13.8P/Q s for a P-page document with Q questions). A higher call count is a real trade-off, but it does not erase the controlled top-k evidence-quality improvement or the end-task gain.

  3. What accounts for the 1.05-call difference? The three repeatedly used navigation tools sum to 11.67 + 11.85 + 8.63 = 32.15 calls. The remaining three tools—decompose_query, vector_retrieval, and generate_answer—contribute approximately 1.05 calls per question in total. Each question invokes generate_answer once; the approximately 0.05 additional calls on average indicate that query decomposition and vector retrieval are used only occasionally, for difficult questions or when tree navigation does not find sufficient evidence. We report the three navigation tools separately because their repeated use best characterizes the agent's search behavior.

Thanks for the detailed responses and clarifications. The authors clarify the breakdown of tool calls and the retrieval quality at fixed evidence cutoffs, but I remain uncertain about the isolated contribution of the section tree relative to the retriever (especially given that vector retrieval is invoked fewer than 0.05 times per query yet accounts for an 8.7pt accuracy drop in Table 3) and whether the marginal accuracy gain justifies nearly doubling the tool calls without an equal budget comparison. I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.