acceptodds
Withdrawn from ICLR 2027

OmniVL-Guard Pro: A Tool-Augmented Agent for Omnibus Vision-Language Forensics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although multimodal large language models have shown strong potential for remote sensing disaster assessment and reasoning, most existing methods still follow a direct prediction paradigm that maps multimodal inputs to final answers, lacking an explicit and verifiable intermediate reasoning process aligned with pixel-level visual evidence. This limitation reduces the interpretability and reliability of their outputs. To address this issue, we propose the Disaster Vision-Language Reasoner (DisasterVLR), a unified vision-language reasoning model for remote sensing disaster assessment that improves fine-grained change understanding and reliable reasoning through an explicit intermediate thinking process and multi-task balanced reinforcement learning. DisasterVLR is built on two key techniques. First, self-evolving chain-of-thought generation constructs visual-evidence-constrained seed reasoning trajectories through multi-agent collaboration and iteratively expands them to build a high-quality Disaster Assessment and Reasoning (DAR) dataset. Second, ARSPO++ (Adaptive Reward Scaling Policy Optimization Plus) dynamically adjusts optimization signals through intra-task reward mapping and inter-task weight regulation, alleviating training imbalances caused by differences in task difficulty. Experimental results demonstrate that DisasterVLR achieves superior performance across a comprehensive range of disaster-related tasks, exhibiting stronger overall disaster assessment capabilities than existing natural-scene multimodal models and remote sensing vision-language models. Compared with the strongest competing method, DisasterVLR reduces WAPE by – percentage points on damage assessment tasks, including damaged-building counting and damaged-road area estimation, and improves mIoU by points on tasks such as referring expression segmentation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Forensics, Multimodal Large Language Models, Tool-Augmented Agents, Agentic Reinforcement Learning, Forgery Detection and Localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TD9COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OmniVL-Guard Pro, a tool-augmented agent built on Qwen3VL-8B. It handles forgery detection, grounding, and segmentation across text, image, video, and cross-modal inputs, plus real-time event verification. The agent can call web search, cropping, zooming, edge-anomaly screening, face detection, frame extraction, and SAM3. The paper contributes (1) the open-world agent itself, which calls web search, cropping, zooming, edge-anomaly screening, face detection, frame extraction, and SAM3; (2) FSTR, a dataset of tool-use trajectories built by a tree-structured, self-evolving generation pipeline; and (3) Checker-Guided Agentic RL, in which a trained Checker scores whether the reasoning process supports the answer and scales the reward of correct responses accordingly. The authors report gains over prior methods on in-domain and five OOD benchmarks.

Critical strengths
  1. The problem formulation and the CGARL idea. Moving forensics from closed-world prediction to active, tool-based evidence seeking is practically meaningful, and CGARL explicitly separates final-answer correctness from whether the tool trajectory actually supports the answer.
  2. Broad evaluation with some process-level evidence. The paper covers five modalities, a new real-time verification benchmark, five OOD benchmarks, stage-wise and tool-wise ablations, and a human evaluation in which pseudo-success falls from 25.4% to 7.2%.
Critical weaknesses
  1. The central claim that CGARL produces evidence-grounded reasoning is not established. (a) The ablation is confounded: the full model in Table 3 receives an extra RL stage on the additional 30% of data in Drl2D_{rl}^2 plus an extra RtoolminR_{toolmin} reward, compared with "SFT + ARSPO"; without an outcome-only Stage III control on the same data and steps, the gain from 21.64 to 27.88 cannot be attributed to the Checker. (b) The paper's own case studies show the failure mode CGARL is meant to remove. In Case 3, the first zoom call uses exactly the ground-truth box [455, 195, 575, 449] before any evidence is gathered; the subsequent top edge-anomaly box [529, 500, 611, 625] does not overlap it, yet the reasoning claims it "overlaps significantly", and the answer is the initial zoom box. Case 4 (video) and Case 5 (painting) also contain an identical tool call and response (zoom_image on [520, 320, 620, 420], returning [520, 320, 619, 420]) on unrelated inputs. This is consequential because process consistency is the paper's key differentiator from outcome-only tool RL; without it, the remaining contribution is largely an engineering combination of tool use, process supervision, and a data pipeline.
  2. The headline gains are measured against weak or construction-suppressed baselines. The +30.71 on RealFact is computed against tool-free Qwen3VL-235B 's 55.74, but RealFact was filtered with Qwen3VL-235B's parametric memory (As you state at Appendix B.1), so this baseline is suppressed by design. Against tool-using baselines in the appendix, the margin is +5.86 over GPT5.4 ReAct and +6.00 over DEFAME; the Text-Video gain drops from +26.96 to +2.76 against FakeSV-VLM. Moreover, the RealFact split does not state whether claims from the same event are kept in one split, and evaluation uses live web retrieval. This is consequential because the "state-of-the-art" claim reduces to modest margins on a benchmark whose leakage and reproducibility are unverified.
Questions for authors
  • Can you add a control in Table 3 that, after Stage I, continues outcome-only RL on Drl2D_{rl}^2 for the same number of steps as full CGARL, with and without RtoolminR_{toolmin}? This is needed to attribute the gain from 21.64 to 27.88 to the Checker.
  • For the human evaluation in Appendix F.3, how were trajectories sampled, how many annotators judged each one, and were they blinded to model identity and Checker scores? What exactly does the 96.2% agreement measure, and is the pseudo-success rate computed over all trajectories or only correct ones?
  • In Case 3, how did the model choose the exact ground-truth box as its first zoom region without prior evidence, and why does the reasoning claim overlap with an edge-anomaly box that does not overlap it?
  • Why do Case 4 and Case 5 contain an identical zoom_image call and response on unrelated inputs? Are the case-study trajectories raw model outputs, and can you show randomly sampled ones, including failures?
  • How are RealFact examples partitioned across SFT, RL, and testing, and are claims derived from the same underlying event kept in the same split?
  • Can you report RealFact results on a fixed evidence snapshot, and with fact-checking pages or other sources that directly state the verdict excluded, to test whether the model reasons over evidence or copies a retrieved conclusion?
Reviewer_vBFWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

OmniVL-Guard Pro is a tool-augmented multimodal forensic agent built on Qwen3VL-8B. It integrates web retrieval, visual inspection, video frame extraction, and SAM3 segmentation to support forgery detection, localization, segmentation, and event verification. The paper develops a teacher-guided trajectory generation and iterative self-training pipeline, followed by reinforcement learning using a learned Checker that evaluates the evidence consistency of complete trajectories. Experiments evaluate the system on in-domain forensic tasks and five out-of-domain benchmarks, with additional tool ablations and human assessments of trajectory quality.

Critical strengths

1.The system provides a unified interface for classification, localization, segmentation, and event verification across multiple modalities. Combining external retrieval with local visual inspection offers practical value for tasks requiring different types of evidence.

2.The evaluation includes same-tool ReAct baselines, specialized systems, tool ablations, and human trajectory assessments. The reported reduction in human-identified pseudo-success from 25.4% to 7.2% provides useful evidence of improved observable trajectory quality.

Critical weaknesses

1.The methodological novelty is limited. The system combines an existing multimodal backbone, established retrieval and perception tools, and the previously introduced ARSPO optimizer. The main additions concern domain-specific trajectory construction and a learned trajectory evaluator, while their conceptual advancement over established tool-agent training and reward-model supervision remains unclear.

2.The evidence does not fully establish the claimed reasoning faithfulness. Removing explicit hint references from trajectories does not demonstrate that answer-conditioned tool choices have been eliminated. Furthermore, Figure 7 describes two spatially nonoverlapping boxes as significantly overlapping, illustrating that a correct prediction can still accompany an inaccurate interpretation of tool evidence. Human assessments are valuable, but consistency judgments alone do not establish that predictions reliably depend on the acquired evidence.

3.Training and inference budgets complicate attribution of the improvements. The full model receives additional training on D2D_2 and an efficiency reward beyond the Stage-I baseline; an otherwise matched continuation without Checker supervision is not reported. For event verification, repeated inference is used without specifying repetition counts or aggregation, and concrete tool-call limits and costs are unclear. These omissions hinder assessment of the improvements under comparable computational budgets.

Questions for authors

1.How does the proposed trajectory-generation pipeline compare with conventional teacher-generated and filtered tool trajectories when training-set size, tool access, and construction budget are matched?

2.How would the Checker and human evaluators score the inconsistent overlap claim in Figure 7? Can evidence-removal or evidence-substitution tests establish whether the final predictions depend on the retrieved observations?

3.Can you compare Checker-guided RL against continued outcome-only RL using the same D2D_2, training steps, and efficiency reward? Please also specify the inference repetition count, aggregation rule, tool-call limits, and budgets used for the principal baselines.

Reviewer_b1H2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces pseudo-success in tool-augmented reasoning, where an agent arrives at the correct final answer despite an unreliable reasoning process. The authors address this with a tree-structured trajectory generation pipeline and checker-guided reinforcement learning, which jointly optimize answer correctness and consistency between reasoning and evidence. Evaluations across several tasks show improved performance.

Critical strengths

The paper extends MLLMs with tool-augmented reasoning and evaluates them on a broader range of tasks than the OmniVL-Guard baseline. Its ablation studies examine the contribution of different external tools, and results across tasks show that checker-guided training further improves performance.

Critical weaknesses
  1. The paper uses a strong model to label trajectories before training the Checker for Stage 3. It would help to explain why the strong model cannot be used directly as the Checker during RL. The paper also does not make clear how often the Checker provides feedback during training or how that feedback affects the model’s reasoning and behavior.

  2. Section 4.3 introduces a tool-efficiency reward, but its effectiveness is not directly evaluated. An ablation without this reward, along with statistics on tool use, would clarify whether it improves efficiency without compromising evidence quality.

Questions for authors
  1. Does OmniVL-Guard Pro reduce process-level reasoning errors after training? A before-and-after analysis would help establish this. In particular, could the authors compare the types and frequency of reasoning errors in rollouts from the base and trained models, including cases where the final answer is correct despite flawed reasoning? This would clarify whether training improves the reasoning process itself, rather than only final-answer accuracy.

  2. What proportion of RL rollouts receive a Checker-derived reward, and how are the resulting process scores (0, 0.5, and 1) distributed? How do the coverage and score distribution of Checker rewards in the training data affect model performance?

  3. Could the authors ablate the tool-efficiency reward Rtoolmin and report statistics such as the number of tool calls and task success rates? This would help show whether the reward reduces inefficient tool use.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.