acceptodds
Under review as a conference paper at ICLR 2027

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

Abstract

Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search environments often expose retrieved evidence only as text, omitting tool-returned images and reducing visually grounded search to text-only reasoning. Long-horizon interaction also compounds tool-call, response-length, timeout, and budget failures, which can discard salvageable trajectories, waste rollout computation, and disturb policy updates. To address these issues, we introduce WeAgent-Harness, a multimodal agentic harness that supports native text–vision interaction and runtime recovery. Retrieved images receive persistent disk references, allowing the model to inspect, process, and cite them throughout the trajectory. Based on this harness, we develop **WeAgent-MMSearch**, an integrated system spanning data construction, agentic post-training, and multimodal rollout. For data construction, a strong MLLM uses WeAgent-Harness to discover, synthesize, and verify MMSearch-style tasks and collect expert trajectories. During post-training, our Failure-Aware GSPO (FA-GSPO) recovers salvageable abnormal rollouts and filters invalid ones to improve bounded multimodal planning and search. We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image, distinguishing image-retrieval failures from visual-perception failures. Evaluation on VisTarget-Bench and seven public benchmarks shows our agentic post-training yields a 19.22-point average improvement, outperforming similarly-sized open-source models and rivaling those ten times larger.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal search agent, post training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Jb4VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a harness for multimodal agentic search. It implements a file system for retrieved images and supports runtime recovery. The authors also performed sft and rl training. The harness and sft training both yield noticeable improvements. They also proposed some practical tricks for gspo and demonstrated their rl works.

Critical strengths
  1. The experiments are solid, and the benchmark improvements are real.
  2. The harness design is clever: appropriate caching for search, auto-repair from the harness, maintaining a persistent multimodal state, etc.
Critical weaknesses
  1. The individual components in their methodology might not be novel: people usually use caching to save search cost, and previous works have presented similar data collection processes.
  2. The proposed fa-gspo training is strongly coupled with their harness. A model trained using fa-gspo would probably not work very well on a harness without runtime recovery. I'd consider this a relatively minor weakness.
Questions for authors
  1. The comparison with Hermes is presented in Section 7.3. The average benchmark improvement is about 2-3%. Can the authors clarify the differences between the proposed harness and Hermes, and potentially ablate which component in the proposed harness matters the most?
  2. I am interested in knowing the contribution of the in-house data to the post-training. For example, the authors mentioned using 56.2k open-source and 3.3k in-house SFT trajectories. Training the model using only the 56.2k open-source or only the 3.3k in-house trajectories, but not both, might clarify the effectiveness of the proposed data collection pipeline.
1 comment

Dear reviewers,

Will you please check the authors' response and reply it?

Best, AC

Reviewer_H1HpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a new harness named WeAgent-Harness for multimodal agentic search and trains it to achieve good performance using a standard SFT → RL pipeline, a commonly used data construction pipeline, and a GSPO algorithm with runtime failure recovery named FA-GSPO. It also presents a new benchmark named VisTarget-Bench to help reveal whether an agent retrieves the intended visual evidence and uses it correctly after retrieval.

Critical strengths
  1. Good results in Table 1
Critical weaknesses
  1. Technical report-like paper without any new insights or novelty - Multimodal agentic search have already been explored in papers accepted ages ago, e.g., SenseNova-MARS, MM-DeepResearch, etc. Why are these works not cited or compared against? The data construction method is also not novel. The multi-hop data construction approach are largely similar to earlier works, e.g., WebWatcher, OpenSearch-VL, etc. FA-GSPO is essentally GSPO with runtime failure recovery and invalid-trajectory filtering during group normalization, with similar ideas already explored in prior works, e.g., DAPO, AReaL, etc. The training pipeline is also not novel, following the standard SFT -> RL paradigm.
  2. Unfair experimental setup - WeAgent-MMSearch uses different training data, data sizes, training steps, compute budgets, and search/tool pipelines from other methods in Table 1. How is this a fair comparision? How can we know whether the perf gains are simply due to better training data, more compute, or stronger search tools?
Questions for authors
  1. Can we have the actual latency, FLOPs, and number of turns and tool calls comparison for the methods compared in Table 1?
  2. The main difference between VisTarget-Bench and other benchmarks seems to be just the evaluation metrics, i.e., Cite, Match, and Match+Acc. Why not extend existing benchmarks with these metrics instead of introducing another benchmark? What exactly is new about the tasks?
  3. Why is WeAgent-Harness presented as a new contribution? It seems to mainly integrate existing search tools, visual evidence handling, and runtime recovery, which have already been explored in earlier works like MARS and MM-DeepResearch. What exactly is technically new? Dont overly rebrand existing methods if they are essentially the same, as it makes things unnecessarily confusing.
  4. Why use Kimi K2.6 instead of other models to synthesize training tasks and expert trajectories, audit RL labels, and serve as the evaluation judge? Why not GPT, claude, or other models? Will changing these models affect the results? Are there any efforts to account for the effects of different data synthesis and evaluation judge models?
  5. What exact model checkpoint is WeAgent-MMSearch-Base? Is it simply Qwen3-VL-30B-A3B with the harness and no additional training?
  6. Does the sft -> rl training causes degradation in the base model's performance on other datasets and tasks? If the base model is publicly released, e.g., Qwen, we should be able to evaluate this. Also, how does it perform in ood settings, including both ood datasets and ood tasks? Specifically, does it just become a specialized agent that performs well on multimodal search but worse in other settings? If so, whats the point when LLMs' main advantage is their generality?
1 comment

Dear reviewers,

Will you please check the authors' response and reply it?

Best, AC

Reviewer_i4BgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper presents a multimodal search agent system built from four parts.

  1. WeAgent-Harness. Tool-returned images receive persistent references, so the model can re-inspect, process and cite them later in the trajectory. The harness also adds cache-backed search, repair of unambiguous malformed calls, structured errors and budget control.
  2. A data pipeline. Kimi K2.6, acting inside the harness, discovers, synthesizes and verifies multi-hop tasks seeded from Wikipedia and from time-sensitive events (January to July 2026). It also produces expert trajectories. This gives 3.3K in-house SFT trajectories and 4.9K RL prompts, which are combined with open-source data.
  3. FA-GSPO. GSPO augmented with recovery for overlong turns and forced final answers. Trajectories that hit the context limit or time out are excluded from group normalization.
  4. VisTarget-Bench. 150 human-verified tasks, each paired with a held-out target image. Separate metrics distinguish failing to retrieve the image from failing to perceive it.

On Qwen3-VL-30B-A3B, the harness alone raises the 8-benchmark average from 19.13 to 36.75. SFT and RL then reach 51.04 and 55.97, the latter being the reported 19.22-point gain over the harness-only base. Ablations cover image re-feed (Table 2), data composition and the RL objective (Table 3), and the choice of harness for three external models (Table 4).

Critical strengths
  1. Large, well-documented engineering gains for an open 30B-A3B model.

    • The harness alone nearly doubles the average (19.13 to 36.75). The full recipe reaches 55.97, above Qwen3.5-Plus (51.58) and close to Kimi K2.6 with the same harness (58.42, Table 1).
    • The SFT data ablation is informative: raw open-source, then normalized open-source, then adding in-house data gives 44.07, 48.06 and 51.04 (Table 3).
    • The harness comparison holds for three external models (Table 4).
    • Most entries report a mean and standard deviation over three runs.
  2. A useful diagnostic benchmark design. Pairing every VisTarget-Bench question with a held-out target image lets the paper separate retrieval failure from perception failure. Fig. 6 shows that the RL gains come largely from better targeting: the target-match rate given a citation rises from 47.62 after SFT to 87.10 after RL. Few multimodal search evaluations make this distinction.

Critical weaknesses
  1. Three contributions in one paper, which makes it hard to read and the results hard to attribute. This is the main reason my rating is not higher.

    • The paper combines a harness (Sec 3), a post-training recipe of data construction and FA-GSPO (Secs 4 and 6), and a new benchmark (Sec 5). Each gets a short section, and important evidence sits only in the appendix, including the harness comparison (Table 4) and the trigger rates of FA-GSPO's recovery and exclusion rules (Fig. 7).
    • Some parts are heavier than their content. FA-GSPO (Sec 6.2) is rollout-time recovery followed by a filter. An overlong turn is regenerated once, a malformed tool call is returned as an observation, and an exhausted budget forces one final tool-free answer. Trajectories that remain overlong or exceed the wall-clock or context budget, and groups left with fewer than two eligible members, are then dropped, and the remaining advantages go into "the clipped, sequence-level GSPO objective". Yet the paragraphs on eligible-only normalization and the objective introduce a full set of new notation and restate the whole objective (Eq. 1), which makes a simple change hard to follow. A few sentences or a short pseudocode block would be clearer.
    • The contributions are evaluated on each other. VisTarget-Bench comes from the same pipeline as the training data (AI use statement) and keeps only seeds whose target image is reachable through WeAgent-Harness (App. A.1). It is then used to evaluate both the harness and the post-training. Adding the in-house training data raises VisTarget by 9.55 points, more than any other benchmark (Table 3). As a result, it is hard to tell which component drives which gain.
    • Focusing the paper on one contribution, with the others presented as supporting tools or in a separate paper, would make each claim easier to assess.
  2. The evaluation is not independent, and two ablations do not isolate their components. This limits how far the headline gains, and their attribution to individual components, can be trusted.

    • Kimi K2.6 synthesizes the tasks, writes the expert SFT trajectories, audits RL labels and judges every benchmark (AI use statement, Sec 7.1), and no judge-human agreement is reported. It is also not stated whether the search cache, which is filled during data construction (App. A.1), is active during evaluation. No decontamination against the seven public benchmarks is described (App. B).
    • The image re-feed ablation (Table 2) "keeps the WeAgent-Harness interface but drops returned images from later turns". This appears to be the RL checkpoint evaluated with re-feed switched off, a train/test mismatch that would exaggerate the 9.08-point drop.
    • Vanilla GSPO leaves the 8-benchmark average where SFT left it (51.04 to 51.10) but lowers VisTarget from 24.00 to 9.78 ± 5.00, while each recovery or exclusion trigger fires on only 0.8% to 7.0% of trajectories (Fig. 7). Training curves, a statement of whether the three runs in Table 3 are independent trainings, and a baseline that only masks truncated or timed-out samples are needed to attribute FA-GSPO's 4.87-point average gain.
Questions for authors

Please refer to the Critical Weaknesses section for details. My main concern is that the paper attempts to cover too many aspects without adequately explaining or providing sufficient evidence for each. If the authors narrow the scope, clarify the central research contribution, and provide stronger supporting evidence, rather than presenting a broad collection of technical results, I would be willing to reconsider my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.