acceptodds
Under review as a conference paper at ICLR 2027

Lessons from Agentic Post-Training of MLLMs for Multimodal Web Search

Abstract

Multimodal web search requires models to formulate effective queries for knowledge-intensive visual questions, gather relevant evidence from the web, and synthesize grounded answers. We systematically study three aspects of post-training multimodal search agents: evaluation, supervised fine-tuning, and reinforcement learning. First, we quantify the impact of individual evaluation components on measured performance and advocate for a standardized evaluation protocol to enable reliable comparisons across methods. Next, we examine how the distribution of tool calls in the SFT training corpus shapes the student agent's search behavior. We further show empirically that short interaction horizons reduce computational costs while maintaining strong performance, whereas longer search horizons do not consistently improve accuracy. Finally, we investigate branching RL to refine intermediate search decisions using only terminal rewards. This approach generates alternative continuations from shared intermediate states and compares their terminal rewards, enabling more targeted credit assignment to individual search decisions. Our empirical analysis identifies early branching as an effective data-driven heuristic: concentrating branching on a small number of early search decisions yields more useful exploration than random branching and outperforms strategies guided by the model's own uncertainty, measured through token entropy or self-reported confidence. Extensive experiments show that the resulting RL policy improves average accuracy from 59.89% to 63.02% over its SFT initialization and achieves state-of-the-art performance on multiple benchmarks. Together, these findings provide practical guidance for evaluating and post-training multimodal search agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Post-Training, Web Search

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sMYDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the post-training of multimodal large language models (MLLMs) for multimodal web search. The authors systematically study three aspects: (1) evaluation protocol sensitivity, demonstrating that evaluation choices significantly affect measured performance; (2) supervised fine-tuning, analyzing how the tool-call distribution in training trajectories shapes student agent behavior; and (3) reinforcement learning, proposing a branching rollout method that enables step-level credit assignment using only terminal rewards. Experiments are conducted on seven multimodal search benchmarks, and the proposed method achieves competitive results compared to direct-inference and agentic-inference baselines.

Critical strengths
  1. Multimodal web search is an important and actively researched direction. The systematic study of post-training aspects (evaluation, SFT, RL) is timely and provides useful guidance for the community.
  2. The sensitivity analysis of evaluation components (summarizer, judge model, judge prompt) is valuable and highlights a real problem: lack of standardized evaluation protocols makes method comparison difficult. The recommendation for a shared harness with fixed open-source models is practical and actionable.
  3. Seven benchmarks provide broad coverage across different types of multimodal search tasks, and models achieves competitive performances.
Critical weaknesses
  1. The experiments on the evaluation setup lack meaningful insights. They merely demonstrate that different configurations affect the results, but fail to provide any guidance on which setting should be adopted for multimodal search.
  2. Regarding the impact of SFT on model behavior, while the statistical metrics support the authors' claims, a more fine-grained analysis is needed. Specifically, the authors should clarify whether SFT primarily alters the distribution of tool types, or if it changes sequence lengths and reasoning patterns. Additionally, although Claude 4.6 is used as the teacher model, its performance should be reported in Figure 2 for proper reference.
Questions for authors
  1. Authors should include comparisons with more relevant models, such as RedSearcher and Point-Seeker.
5 comments
AuthorsOpenReview ↗

Rebuttal Response

We thank the reviewer for the constructive feedback and for recognizing the importance of our systematic study. We address the concerns below and will incorporate the suggested clarifications and comparisons in the revised manuscript.

W1: Actionable insights from the evaluation sensitivity analysis

We agree that identifying evaluation sensitivity is most useful when accompanied by concrete recommendations. Our experiments provide several actionable insights beyond demonstrating that evaluation configurations affect performance.

First, the choice of summarizer affects not only the reported accuracy but also the agent's search trajectory, as it determines which retrieved evidence is presented to the policy. We observe changes of up to 6.00 percentage points when varying the summarizer alone. Second, judge prompts introduce systematic differences: switching from recall-based to stricter semantic-equivalence judging generally reduces measured accuracy by up to 5.52 points, whereas changing the judge model produces smaller and less consistent effects. These findings suggest that evaluation components should not be treated as interchangeable.

Based on these observations, we recommend a standardized evaluation harness with a fixed open-source summarizer and judge, explicitly specified judge prompts, and matched interaction budgets. In particular, Qwen3-235B-A22B provides a reproducible open-source reference configuration, for which we already report additional results in Appendix A. We retain the Claude Sonnet summarizer and GPT-5-mini judge for our headline results to maintain comparability with existing evaluation conventions.

Importantly, our analysis also shows that increasing the interaction budget does not consistently improve performance. Therefore, we recommend reporting both matched-budget and native-budget evaluations, along with tool calls and inference costs, to distinguish improvements in the search policy from differences in the surrounding evaluation system.

We will revise Section 3.2 to make these practical recommendations more explicit and clarify the rationale for the adopted evaluation configuration.


Q1: Comparisons with REDSearcher and POINTS-Seeker

We thank the reviewer for pointing out these relevant works. We agree that REDSearcher (Chu et al., 2026) and POINTS-Seeker (Liu et al., 2026) are important baselines for multimodal agentic search. We provide their published results alongside ours below.

Comparison on multimodal search benchmarks (accuracy, %):

ModelBC-VLVDRMMSearchMMSearch+FVQASimpleVQALiveVQA
REDSearcher-MM-RL57.20–72.9026.60––79.30
POINTS-Seeker-8B44.40–70.8025.2071.2068.8077.70
Ours (SFT)56.3925.8078.3626.6978.3374.6779.00
Ours (RL)59.8927.8081.8733.9480.0075.6782.00

The REDSearcher and POINTS-Seeker numbers are taken from their respective published results; our numbers are from Table 4. A dash indicates that the corresponding result is not reported.

We will update Table 4 to include both REDSearcher and POINTS-Seeker, with the appropriate citations, and expand the related-work discussion to position our approach relative to these methods.

AuthorsOpenReview ↗

Rebuttal Response

W2: Fine-grained analysis of SFT behavior and teacher reference

We thank the reviewer for this suggestion which makes the manuscript strong. We conducted additional analyses to distinguish the effects of SFT on tool-use distribution, sequence lengths, and reasoning patterns. These findings complement Section 3.3 of main paper by providing a more fine-grained understanding of the behavior transferred from the teacher (Claude 4.6), and we will revise the manuscript accordingly.

First, consistent with Section 3.3, SFT shifts the student's tool-use distribution toward the training corpus. However, some changes in tool frequencies are influenced by the interaction protocol, which requires an image-search operation at the first turn. To isolate learned behavior, we additionally examine decisions where the model has free choice:

Tool-use metricTeacher (Claude 4.6)SFTBase
Web visits (% of research calls)24.3%25.5%13.2%
Queries per text search2.582.201.27
URLs per web visit1.671.551.00

These results further support our observation that SFT transfers the teacher's tool-use behavior, not only in tool selection but also in how tools are invoked, including the number of queries per search and URLs examined per visit.

Second, SFT does not substantially alter per-turn reasoning length. The median reasoning length remains similar (106 tokens for Base vs. 104 for SFT), as does the 90th percentile (199 vs. 200). However, SFT changes final-answer behavior: median answer length increases from 33 to 157 tokens, and 97% of SFT answers follow the required format, compared with 7% for Base. Furthermore, the ratio of excessively long rollout lengths to the median decreases from 3.7× to 1.2×, indicating more consistent sequence lengths.

Third, SFT also affects reasoning patterns, particularly their surface structure. For example, 92% of SFT first-turn reasoning follows the teacher's opening structure, compared with 10% for Base. However, the transfer of underlying reasoning decisions is more limited: on decisions where the teacher and Base differ, SFT moves 45% of the way toward the teacher. Similarly, the tendency to commit to an answer increases from 3.7% (Base) to 20.6% (SFT), compared with 40.5% for the teacher. Only 0.99% of SFT reasoning sentences match teacher trajectories verbatim, suggesting that these changes are not simply due to memorization.

Overall, these findings reinforce our conclusion in Section 3.3 that SFT distills the tool-use behavior represented in the training corpus, while clarifying that this transfer extends beyond tool-type distributions to tool invocation patterns, answer formatting, and reasoning structure, without substantially changing per-turn reasoning length.

The training corpus consists of Claude 4.6-generated trajectories after quality filtering. Thus, the "Corpus" row in Figure 2(b) already represents the teacher's tool-use behavior. We will relabel it as "Claude 4.6 (Training Corpus)" to make this explicit, include Claude 4.6's benchmark performance as a reference in Figure 2(a), and incorporate the additional behavioral analysis into the revised manuscript.

I appreciate the authors’ response, which has clarified all my concerns. I will maintain my original rating.

Reviewer_HuhxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-training of multimodal web-search agents built on Qwen3-VL-30B-A3B and reports four lessons. Swapping the summarizer, judge model or judge prompt shifts accuracy by up to about 6 points. SFT transfers the teacher corpus’s tool-use behavior, with gains that depend on the student model. Longer search horizons do not reliably help. For RL, branching (“vine”) rollouts that fork at early turns produce more groups with mixed rewards than GRPO or entropy- or confidence-guided forking. The resulting recipe (five-turn SFT, then early-branching RLOO) raises average accuracy on seven benchmarks from 59.89% to 63.02%.

Critical strengths
  1. The topic is timely and practical. Showing that harness choices (summarizer, judge model, judge prompt) move accuracy as much as reported method gains is a useful message for the community, and the factorial sweep is paired and well documented. 2, The turn-budget analysis is careful. It separates examples that were cut off by the turn limit from those that weren’t, and counts answers flipped both ways.
  2. The vine branching scheme is simple, and the comparison with GRPO matches the number of trajectories per prompt.
  3. The appendix gives thorough detail on SFT data generation, filtering and training.
Critical weaknesses
  1. The paper looks unfinished. Line 310 still has a placeholder (“by A and B”), so the efficiency argument for T=5 has no numbers. Table 6 lists RL FVQA as 81.87 but Table 4 lists 80.00, and the “+3.54 over GRPO” is computed from the wrong value. A paragraph in Appendix A.1 is duplicated.
  2. The RL stage cannot be reproduced. There is no RL training pool, no hyperparameters, no seeds, and no measured rollout cost. The GRPO-vs-branching comparison appears only in the appendix, on 3 of 7 benchmarks, from a single run.
  3. The credit-assignment claim does not match the estimator. RLOO uses one baseline pooled over all nine leaves, and only the prefix shared by all leaves cancels. Siblings are never compared against their own fork-local mean, so the claimed “targeted credit assignment” is not actually implemented.
  4. Lessons 2–3 have weak controls and limited novelty. Lesson 2 rests on two students, and untrained Gemma (60.08) already beats Ours (SFT) (59.89). The 5- vs 30-turn SFT comparison changes the data and the inference budget together. Related work on tree and branching RL (VinePPO, TreeRL, ARPO and others) is missing.
  5. The main comparison contradicts Lesson 1. Table 4 places published baseline numbers next to the authors’ own harness results, which is exactly the practice the paper warns against. Proprietary agents also score higher (68.00 and 67.77), so the state-of-the-art claim needs qualifying.
Questions for authors
  1. Please give the full RL configuration (training data, hyperparameters, seeds) and the actual rollout cost of vine branching compared with GRPO.
  2. How do models trained with RL using entropy, self-confidence and random branching perform, rather than only the share of groups with mixed rewards? Did you try a fork-local baseline instead of pooling over all nine leaves?
  3. Can you re-run the baselines in your own harness, and report the RL model under the semantic judge?
6 comments
AuthorsOpenReview ↗

Rebuttal Response

W1. Manuscript corrections

Thank you for flagging these issues. Moving from five to thirty turns increased model execution time from 7.32 to 22.15 seconds per question (3.02×) and end-to-end wall-clock latency from 144.77 to 600.14 seconds per question (4.15×). We will replace the placeholders at line 310 with these measurements.

We mistakenly reported the MMSearch score (81.87%) as the FVQA score in Table 6. We will correct the FVQA entry to 80.00% and update the reported gain over GRPO from +3.54 to +1.67 percentage points (80.00% vs. 78.33%). We will also fix the duplicated paragraph in Appendix A.1 in the revised manuscript.


W2 / Q1. RL reproducibility, rollout cost, and benchmark coverage

Thank you for flagging the missing RL details and the limited benchmark coverage in Table 6. We will add the data construction procedure, the training configuration, cost accounting, and include the expanded seven-benchmark comparison in the appendix.

RL training pool and configuration. The pool contains 7,359 image–question–answer prompts from LiveVQA train (3,591), OSVL WebQA (1,499), OSVL demo-1k (996), InfoSeek (418), OSVL Wiki-EN (313), RedSearcher-MM SFT-5k (213), OSVL new-FVQA (122), Vision-DeepResearch SFT-1k (119), and OSVL WikiArt (88). We initialize from SFT checkpoint and train with RLOO without advantage standard-deviation normalization. We use one PPO epoch per batch, clip values of 0.20 and 0.28, dual-clip c=3.0c=3.0, and no KL or entropy term. We use Adam with a learning rate of 10−610^{-6}, 10 linear warmup steps followed by a constant rate, weight decay of 0.01, 64 prompts per step, and a PPO minibatch size of 288 (32 trees x 9 leaves). The reward is 0.9 times GPT-5-mini answer correctness plus 0.1 times format validity. Rollouts use temperature 0.8, top-p=1.0p=1.0, and up to five tool turns followed by a final answer. Vine branching forks the trunk at up to two turn boundaries with width five, yielding up to nine leaves per prompt. We set seed 42 for the data loader, actor, and rollout engine. The reported model is the step-90 checkpoint, trained on 5,760 prompts using eight H200 GPUs.

Measured rollout time and estimated search cost. We compare the first 90 steps of both runs on the same hardware. Vine branching and GRPO both generate 9 trajectories per prompt.

Cost over the first 90 stepsGRPOVine branching
Rollout generation, measured H200 GPU-hours43.547.5
Total training excluding validation, measured H200 GPU-hours98.997.5
Estimated search-API credits458k318k
Estimated search-API cost$344$239

Vine branching used about 9% more rollout GPU time and 31% fewer estimated search-API credits. Shared prefixes allow branches to reuse earlier tool results, including the initial image search. The API figures are derived from rollout statistics and exclude unlogged retries.

Expanded benchmark comparison. We retain the three Table 6 benchmarks, incorporate the FVQA correction described in W1, and add results for the four remaining benchmarks. We will identify the evaluation run and checkpoint used for each result in the revised manuscript.

MethodBC-VLMMSearch+FVQAVDR-BenchMMSearchSimpleVQALive-VQA
GRPO53.8826.2478.3323.6076.6172.0079.00
TreeGRPO57.1423.0777.3325.2073.1071.3379.33
Ours (RL)59.8933.9480.0027.8081.8775.6782.00

The vine checkpoint scores higher than GRPO on all seven benchmarks, with gains ranging from 1.67 to 7.70 percentage points. The expanded table addresses benchmark coverage.

AuthorsOpenReview ↗

Rebuttal Response

W3. Credit assignment under tree-level RLOO

We apologize for the ambiguity in our use of “targeted credit assignment.” We meant that branching samples alternative continuations from the same search history, allowing their terminal rewards to give those continuations different learning signals. Under the policy gradient expression as shown in Appendix E, the contribution from a prefix shared by every leaf cancels when its token gradients enter with identical weights. We agree our wording suggests a stronger claim about assigning credit to individual search decisions, and we will revise it.

For each surviving leaf, RLOO uses the mean reward of all other surviving leaves in the tree as its baseline. With nn leaves and tree mean reward rˉ\bar r, the advantage is Ai=nn−1(ri−rˉ)A_i=\frac{n}{n-1}(r_i-\bar r). Two sibling leaves therefore differ in advantage by nn−1(ri−rj)\frac{n}{n-1}(r_i-r_j), with both advantages centered at the tree level. For a prefix shared by only a subset SS, the aggregate advantage is n∣S∣n−1(rˉS−rˉ)\frac{n\lvert S\rvert}{n-1}(\bar r_S-\bar r), where rˉS\bar r_S is the mean reward in that subset. Such a prefix can still receive a learning signal based on how its descendants fare relative to the full tree.

We will revise the abstract and Lesson 4 to describe this mechanism explicitly. In Appendix E, we will state that the baseline is tree-level, add the subset-prefix derivation, and clarify the conditions for exact cancellation.


W4. Controls and scope of Lessons 2–3 and related work

Lesson 2. We selected two strong open-weight multimodal models, Qwen3-VL and Gemma 4, to examine how the same SFT corpus affects students with different starting performance in our agentic search setup. Before SFT, Qwen3-VL achieves 44.51% accuracy, while Gemma 4 reaches 59.46%. After training on the same SFT corpus, Qwen3-VL improves to 60.11% (+15.60 percentage points), whereas Gemma 4 decreases to 56.19% (−3.27 percentage points).

Qwen3-VL’s tool-use frequencies also shift toward those of the teacher trajectories, which supports our interpretation that SFT transfers search behavior alongside the interaction protocol. We observe student-dependent effects of SFT in both model families. Given the broadly similar training pipelines used by many open-weight models, we expect the finding to extend beyond the two students evaluated here.

Lesson 3. The five- versus thirty-turn SFT comparison asks whether training and deploying a policy with a larger search budget improves accuracy. The two SFT checkpoints, by design, learn from trajectories generated under different turn limits, and we evaluate each checkpoint at its corresponding budget. We use the same image–question pairs, teacher model, and evaluation setup, varying the turn limit during trajectory generation and evaluation. The thirty-turn setting improves MMSearch+ but reduces accuracy on BrowseComp-VL and FVQA. We will clarify that this result reflects the joint effect of training and deploying with a larger budget.

However, in the separate sweep for base Qwen3-VL and GPT-5.2, each checkpoint is fixed and only the inference turn limit varies. This isolates the effect of giving the same policy more search turns and shows that accuracy does not consistently improve as the budget increases.

Related work. We will expand the related work section to discuss vine-based credit assignment in VinePPO [2], on-policy tree search in TreeRL [3], and entropy-guided branching for tool-using agents in ARPO [4]. We will clarify that our contribution is an empirical comparison of where to branch in multimodal web search, evaluating early-turn, entropy-based, self-confidence-based, and random fork selection.

References

  1. Peng et al. (2026). MTA-Agent: An Open Recipe for Multimodal Deep Search Agents, Table 2.
  2. Kazemnejad et al. (2025). VinePPO: Refining Credit Assignment in RL Training of LLMs.
  3. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search (2025).
  4. Agentic Reinforced Policy Optimization (2025).
AuthorsOpenReview ↗

Rebuttal Response

W5 / Q3. Common-harness evaluation and semantic judging

We agree with the reviewer and will revise our state-of-the-art claim to cover the open-source multimodal deep-search agents listed in Table 4. Our RL model achieves the highest reported accuracy on five of seven benchmarks among these models. We used scores from the original papers under their respective evaluation harnesses because some baselines are not openly available.

Following the reviewer's suggestion, we additionally evaluated two recent baselines, Vision-DeepResearch and OpenSearch-VL, under our common evaluation harness. We also evaluated our RL model using the semantic prompt, applying the same GPT-5-mini judge to the saved answers without regenerating trajectories.

Under the common harness, our RL model outperforms both re-evaluated baselines on six of seven benchmarks. Changing from the recall to the semantic judging prompt reduces our model's average accuracy from 63.02% to 60.52%, a decrease of 2.50 percentage points. We will include these results in the revised manuscript and clearly distinguish published scores from common-harness evaluations.

Model and settingBC-VLMMSearch+FVQAVDR-BenchMMSearchSimpleVQALive-VQA
Vision-DeepResearch-30B-A3B (published)53.7028.5074.2037.8069.60-77.60
Vision-DeepResearch-30B-A3B (our harness)53.6016.7471.0032.8062.5769.6771.00
OpenSearch-VL-30B-A3B (published)41.10-73.2033.5068.7074.9067.40
OpenSearch-VL-30B-A3B (our harness)39.6014.4867.6729.0060.8269.6760.00
Ours (RL, recall prompt)59.8933.9480.0027.8081.8775.6782.00
Ours (RL, semantic prompt)56.6432.1378.3324.6078.9573.3379.67
Reviewer_1mVmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies evaluation and post-training choices for multimodal web-search agents. It varies the page summarizer, answer judge, judge prompt, and interaction horizon, studies SFT on teacher-generated trajectories, and explores an early-branching RL rollout scheme. The experiments show that evaluation-stack choices can materially change measured accuracy, that SFT transfers tool-use patterns from its trajectory corpus, and that longer search is not consistently better. The final Qwen3-VL-30B-A3B agent improves from 59.89 to 63.02 average accuracy over seven benchmarks after RL. I see the main contribution as a careful practical study of the training and evaluation stack, with more limited evidence for the independent advantage of the branching method.

Critical strengths
  1. The evaluation study varies summarizers, judges, and judge prompts factorially, with judge settings compared on the same stored predictions. Appendix A reports the full results. The several-percentage-point shifts show why these choices need to be controlled when comparing agent-training methods.

  2. The paper examines trajectory filtering, tool-use distributions, SFT horizons, and inference-time horizon sweeps, then evaluates the final agent on seven benchmarks. Appendix A.1 also reports trained GRPO and TreeGRPO baselines with nine nominal terminal trajectories per prompt. This gives the RL comparison more substance than an SFT-versus-RL result alone.

Critical weaknesses
  1. Early branching has not been evaluated at equal training cost. Appendix A.1 matches nominal trajectory counts but notes that token and tool-call costs can differ. It also needs to state whether the GRPO and TreeGRPO baselines use the same advantage estimates, loss normalization, and optimization settings as the final RLOO method. The text says the objective is unchanged; if the independent-rollout baseline already matches these settings, please make that explicit. Table 3's higher fraction of mixed-reward groups helps explain the sampling behavior, but does not establish learning efficiency at equal cost.

  2. The final results need fuller documentation. The SFT corpus contains 2,911 LiveVQA examples and evaluation uses 300, but the split relationship and image/question overlap checks are not described. This needs clarification, without assuming overlap from the counts alone. RL data and optimization settings are also less complete than the SFT description. Finally, FVQA is 80.00 for the final RL agent in Table 4 and 81.87 for “Ours (RL)” in Table 6. Please explain whether these are different runs or a reporting error. Repeated-run variability would also help interpret the smaller gains.

Questions for authors
  1. Does the independent-rollout baseline in Table 6 already use the same RLOO objective, loss normalization, optimizer, and prompt distribution as the early-branch method? If not, can you provide that matched comparison? Please give generated tokens, tool calls, wall-clock time, and reward-group filtering rates for both. Matching terminal trajectory counts alone does not establish training efficiency.

  2. Please specify the RL prompt corpus, its size, the number of optimization steps, learning rate, batch sizes, and checkpoint-selection procedure. How are the SFT and evaluation LiveVQA splits separated, including near-duplicate images and questions?

  3. Are Tables 4 and 6 evaluating the same RL checkpoint and FVQA protocol? Please reconcile 80.00 and 81.87 and replace the cost placeholders in Section 3.4 with measurements. If repeated runs were conducted, please report their variability for the headline comparison.

  4. Could the horizon conclusion be stated more explicitly within the chosen protocol? The agent must begin with image/crop search and subsequently uses text search or page visits. The fixed-checkpoint inference sweeps are informative, but the SFT comparison changes both training and evaluation horizons. A small crossed comparison would distinguish those effects if the stronger training-horizon claim is retained.

4 comments
AuthorsOpenReview ↗

Rebuttal Response

W1 / Q1. RL baseline matching and training efficiency

We thank the reviewer for emphasizing the importance of objective-matched comparisons and accounting for actual training costs rather than nominal trajectory counts alone.

Optimization objectives and baseline matching. We clarify that GRPO, TreeGRPO, and all branching strategies use the RLOO objective, without advantage standard-deviation normalization. They share the same reward function (0.9 answer correctness + 0.1 format validity), loss normalization, optimizer settings, and training prompt distribution. The key differences lie in rollout construction and advantage grouping: GRPO samples independent trajectories, TreeGRPO uses fork-local reward comparisons, while our method computes tree-level leave-one-out advantages across all surviving leaves. The four branching strategies (random, entropy-based, self-confidence-based, and early-turn branching) additionally share the same vine structure, differing only in fork-point selection. We use one PPO epoch per batch, with nine sampled trajectories or leaves per prompt. Thus, the comparison controls for the optimization objective and training configuration while examining different rollout and reward-grouping strategies. We will make these implementation details explicit in the revised manuscript.

Measured training efficiency. To address the concern that matching terminal trajectory counts does not imply equal computational cost, we measured token usage, actual tool calls, and wall-clock time over the first 90 optimization steps on the same eight-H200 setup.

Metric (first 90 steps)GRPO (independent)Ours (early branching)
Model-generated tokens~45M~37M
Total response tokens (including tool outputs)226M269M (~214M excluding shared prefixes)
Actual tool calls150k104k
Training wall-clock (excluding validation)12.4 h12.2 h
End-to-end wall-clock13.9 h13.8 h
Rollout generation (H200 GPU-hours)43.547.5
Total training (H200 GPU-hours, excluding validation)98.997.5
Estimated search-API credits458k318k

Early branching generates approximately 18% fewer model-generated tokens and executes approximately 31% fewer actual tool calls, primarily because shared prefixes and tool observations are reused rather than regenerated. Although branching requires approximately 9% more rollout GPU time, total training compute and wall-clock time are comparable (97.5 vs. 98.9 H200 GPU-hours; 12.2 vs. 12.4 hours). Estimated search-API usage is also approximately 31% lower. These results demonstrate a trade-off between rollout GPU computation and tool-use costs rather than a uniform reduction in all computational resources.

We agree that matching trajectory counts alone does not establish superior learning efficiency at equal cost. Our measurements show that early-turn branching achieves stronger downstream performance with comparable overall training compute and wall-clock time, while using fewer model-generated tokens and actual tool calls. We will report these measurements and clarify the controlled aspects of each comparison in the revised manuscript.

Reward-group filtering. During training, 45.9% of early-branching groups contain differing terminal rewards, while 54.1% have identical rewards and provide zero group-relative advantage. The corresponding GRPO training-run has 41.8% of informative groups, while 58.2% have zero group-relative advantage. In the controlled 1,000-prompt analysis (Table 3), there are 43.07% informative groups for early branching versus 40.88% for independent rollouts. We will explicitly distinguish these controlled sampling statistics from the training-run measurements.

AuthorsOpenReview ↗

Rebuttal Response

W2 / Q2. RL reproducibility and dataset separation

RL training corpus. The RL prompt pool contains 7,359 image–question–answer examples from nine sources: LiveVQA train (3,591), OSVL WebQA (1,499), OSVL demo-1k (996), InfoSeek (418), OSVL Wiki-EN (313), RedSearcher-MM SFT-5k (213), OSVL new-FVQA (122), Vision-DeepResearch SFT-1k (119), and OSVL WikiArt (88). The LiveVQA training examples are distinct from those used for SFT trajectory generation. Similarly, the OSVL new-FVQA examples are drawn from the training split and do not overlap with the FVQA evaluation set.

Training configuration. RL starts from the SFT checkpoint and uses RLOO without advantage standard-deviation normalization. We train using Adam with a learning rate of 10−610^{-6}, weight decay of 0.01, ten linear warmup steps followed by a constant learning rate, 64 prompts per optimization step, and PPO minibatches of 288 trajectories (32 trees × 9 leaves). We use one PPO epoch per batch, clipping values of 0.20 and 0.28, dual-clip c=3.0c = 3.0, and no KL or entropy regularization term. Rollouts use a sampling temperature of 0.8, top-p=1.0p = 1.0, and a maximum of five tool-enabled turns, followed by a final-answer turn. We set seed 42 for the data loader, actor, and rollout engine. The reported RL model is the step-90 checkpoint, trained on 5,760 prompts using eight H200 GPUs. We will add the complete RL configuration, data provenance, and checkpoint-selection procedure to the appendix.

SFT and evaluation separation. The LiveVQA data used for SFT and evaluation come from different splits. For SFT, we construct a 5,000-example subset from the ONE-Lab/LiveVQA-new dataset (https://huggingface.co/datasets/ONE-Lab/LiveVQA-new), selecting 1,000 image–question pairs whose images originate from arXiv papers, 1,000 pairs whose images originate from YouTube videos, and 3,000 pairs whose images come from other sources. During subset construction, we enforce global uniqueness of both image paths (img_path) and question strings, ensuring that no image path or exact question is reused within the selected subset. The resulting trajectory-generation stage produced 4,981 trajectories, of which 2,911 were retained after quality filtering. In contrast, evaluation uses 300 examples from a separate LiveVQA evaluation split. Thus, the SFT and evaluation examples are drawn from different data splits, and the 300 evaluation examples are not part of the constructed 5,000-example SFT subset. We will clarify the dataset sources, split construction, and deduplication procedure in the revised manuscript.


W2 / Q3. FVQA correction, efficiency measurements, and variability

FVQA discrepancy. Thank you for catching this reporting error. The value of 81.87% in Table 6 was mistakenly copied from the MMSearch result. The correct FVQA accuracy of our final RL checkpoint is 80.00%, as reported in Table 4. Accordingly, its improvement over GRPO on FVQA is +1.67 rather than +3.54 percentage points. We will correct Table 6 and the accompanying text. The discrepancy is not due to different checkpoints or evaluation protocols.

Interaction-budget efficiency. We have also obtained the measurements needed to replace the placeholders in Section 3.4. Increasing the interaction budget from five to thirty turns increases average model execution time from 7.32 to 22.15 seconds per question (3.02×) and end-to-end wall-clock latency from 144.77 to 600.14 seconds per question (4.15×).

These measurements support our conclusion that a larger search budget substantially increases inference cost without providing consistent accuracy improvements across the evaluated benchmarks.


W2 / Q4. Scope of the interaction-horizon conclusion

We agree that the current five- versus thirty-turn SFT comparison changes both the trajectory horizon used during training and the interaction budget used during evaluation. The two SFT checkpoints, by design, learn from trajectories generated under different turn limits, and we evaluate each checkpoint at its corresponding budget. We use the same image–question pairs, teacher model, and evaluation setup, varying the turn limit during trajectory generation and evaluation. We will clarify that this result reflects the joint effect of training and deploying with a larger budget.

Separately, our fixed-checkpoint inference experiments isolate the effect of the evaluation horizon: for base Qwen3-VL and GPT-5.2, we vary the maximum number of interaction turns while keeping the policy, tools, prompts, summarizer, and judge fixed. These experiments show non-monotonic performance as the interaction budget increases.

We will clarify that our findings apply to our search protocol. We will also revise Lesson 3 to emphasize that longer interaction budgets do not consistently improve accuracy, rather than claiming that shorter training horizons are universally superior.

AuthorsOpenReview ↗

Rebuttal Follow-Up

Dear Reviewer,

Thank you again for your valuable feedback. We would be very grateful if you could kindly take a moment to review our responses when convenient. Please let us know if any concerns remain.

Thank you so much for your time and consideration.

Dear Reviewer,

As we approach the end of the discussion phase, we would like to kindly reiterate our request for you to consider the clarifications and responses provided in our rebuttal and reconsider your rating.

We sincerely appreciate your time and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.