acceptodds
Under review as a conference paper at ICLR 2027

Robust-TO: Confidence-Aware Tool Orchestration for Robust Video Understanding

Abstract

Video reasoning language models implicitly assume that every input frame is equally reliable. This leads to what we term the *Blind Trust Problem*: under realistic perturbations such as motion blur, glare, or occlusion, frontier video reasoning models can suffer 15-30%p accuracy drops on real-world embodied benchmarks, while remaining unaware that their visual evidence has been degraded. To address this challenge, we propose Robust-TO, an agentic video understanding framework that explicitly integrates per-frame trustworthiness into every stage of reasoning. Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface. Each tool receives a sub-query derived from the original question and a set of trustworthy frames selected by the reliability-relevance score. It returns evidence in a shared format: a concrete prediction (e.g., a bounding box, motion trajectory, recognized text, or action label), temporal grounding, and a calibrated reliability score. During reasoning, these calibrated scores guide evidence weighting in a three-tier synthesis process (high/medium/low) and define a confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and efficiency. On two video reasoning benchmarks spanning eight tasks, Robust-TO achieves 56.4% average accuracy on clean inputs, surpassing the strongest open-source baseline by 10.6%p and outperforming Gemini-2.5-Pro (46.2%). Under five realistic corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline, while exhibiting the smallest clean-to-corrupted accuracy drop among all compared methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oQYuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Robust-TO studies robust video reasoning under visual corruptions, including motion blur, noise, glare, occlusion, and low light. It is a tool-augmented framework that selects video frames based on visual reliability and query relevance. A host VLM decomposes the original question into atomic sub-queries and routes them to specialized perception tools according to the estimated corruption profile. Tool outputs are assigned confidence scores based on intrinsic tool confidence and input-frame reliability, followed by a three-tier evidence synthesis process. The host VLM is trained with GRPO using rewards for answer correctness, confidence-cost trade-offs, sub-query efficiency, and output formatting. Experiments on UrbanVideo-Bench and VSI-Bench report 56.4% average accuracy on clean videos and 54.3% accuracy on corrupted UrbanVideo-Bench.

Critical strengths
  1. Practical motivation. Robust-TO addresses a meaningful problem, as motion blur, glare, occlusion, and low-light conditions are common in real-world video capture. Explicitly considering evidence reliability is valuable for robust video reasoning.

  2. Unified confidence-aware framework. Robust-TO integrates frame quality assessment, adaptive tool routing, and confidence-aware evidence aggregation.

Critical weaknesses
  1. Unclear backbone. The main experiments rely on Qwen3-VL-7B-Instruct, but this model does not exist.(Maybe 4B or 8B?)
  2. Limited realism and generalization of robustness evaluation. The evaluation relies on synthetic corruptions applied to randomly selected frames, which may not reflect real-world degradation patterns. Experiments on naturally degraded videos and broader video understanding benchmarks (e.g., Video-MME and Video-MME-v2) are missing.
  3. Insufficient isolation of learned tool-routing contributions. Although several ablations are provided, they do not clearly distinguish the effects of predefined corruption-specific routing rules from those of GRPO learning. This makes the contribution of learned corruption-aware tool orchestration unclear.
  4. Inconsistent experimental tables. Several ablation results conflict with the main experiments. For example, Table 21 reports 54.3% corrupted accuracy for the Qwen2.5-VL-7B variant, while Table 2 reports 47.1% for the same model.
  5. Inconsistent efficiency claims. Appendix A claims less than 5% inference overhead on clean videos, but Table 14 reports 2,308 ms for Robust-TO versus 1,826 ms for the base VLM, corresponding to approximately 26.4% additional latency. The authors should clarify the measurement settings and overhead definition.
  6. Potential loss of critical temporal evidence. The reliability-based frame selector may discard degraded frames that contain essential events. The paper does not evaluate whether the method preserves critical evidence when no clean alternative frames are available.
Questions for authors
  1. Why are only four tasks selected from each benchmark, and what criteria determine this selection? Can the authors report full-benchmark results on clean and corrupted videos?

  2. Other questions: see weaknesses.

12 comments
AuthorsOpenReview ↗

response for Reviewer oQYu (1/6)

W1 : Unclear backbone. The main experiments rely on Qwen3-VL-7B-Instruct, but this model does not exist.(Maybe 4B or 8B?)

A : We sincerely thank you for catching this naming error. We confirm that the backbone used in our experiments is Qwen3-VL-8B-Instruct, not the non-existent Qwen3-VL-7B-Instruct. This is a model-naming error rather than an issue with the backbone used in our experiments. In the revision, we have corrected all references to the backbone.

W2 : Limited realism and generalization of robustness evaluation. The evaluation relies on synthetic corruptions applied to randomly selected frames, which may not reflect real-world degradation patterns. Experiments on naturally degraded videos and broader video understanding benchmarks (e.g., Video-MME and Video-MME-v2) are missing.

A: Thank you again for highlighting two distinct questions: generalization beyond our primary benchmarks, and robustness to naturally occurring degradation.

For broader benchmark generalization, we have additionally evaluated Robust-TO on Video-MME and Video-MME-v2 using Qwen3-VL-8B. Under the same Corruption protocol, averaged over five corruption masks at medium severity, Robust-TO improves performance from 58.2 to 62.8 on Video-MME and from 41.5 to 45.2 on Video-MME-v2. Clean performance also improves from 68.4 to 69.6 and from 49.3 to 50.4, respectively. These results extend the evaluation beyond UrbanVideo-Bench and VSI-Bench and support generalization across video content and tasks under the evaluated corruption protocol.

Table R1: Video-MME and Video-MME-v2 generalization (Qwen3-VL-8B).

MethodVideo-MME CleanVideo-MME Corrupted Avg.Video-MME-v2 CleanVideo-MME-v2 Corrupted Avg.
Base VLM68.458.249.341.5
Robust-TO69.662.850.445.2

Existing external-benchmark evaluation. We agree that evaluating robustness under naturally degraded videos is important. We believe that collecting and constructing a large-scale benchmark with naturally occurring degradations would itself constitute a valuable and independent research direction, beyond the scope of this work.

Nevertheless, to assess the broader generalizability of our method beyond the our current corruption protocol, we have additionally evaluated Robust-TO on MVTamperBench, as already reported in Appendix D.1 and Table 11. While MVTamperBench uses an externally defined synthetic tampering protocol rather than naturally occurring degradations, it provides a complementary evaluation under a different corruption setting. Robust-TO achieves an F1 of 0.322 compared to 0.351 for the base VLM, suggesting that its robustness gains do not necessarily transfer to all types of video perturbations.

AuthorsOpenReview ↗

response for Reviewer oQYu (2/6)

W3 : “Insufficient isolation of learned tool-routing contributions. Although several ablations are provided, they do not clearly distinguish the effects of predefined corruption-specific routing rules from those of GRPO learning. This makes the contribution of learned corruption-aware tool orchestration unclear.”

A : Thanks for the helpful comment. We agree that the current ablations don’t yet cleanly separate the effect of predefined corruption-specific routing rules from that of GRPO learning. To address this directly, we’ve added Table R2 and a definition of the agreement rate to the revision. We compare four variants on held-out corrupted UV-Bench with Qwen3-VL-8B: (a) hard-coded rules, (b) prompt-only routing without GRPO, (c) GRPO training without the routing guidelines in the prompt, i.e., the model must learn corruption-aware routing purely from the GRPO reward rather than being guided by the hand-written blur/occlusion/brightness preferences in Sec. 3.3 and Prompt F.2, and (d) full Robust-TO. We also report a learned-vs-rule agreement rate: for each sub-query, we check whether the full model selects the same tool as the hard-coded rule prescribes, and then average this over all sub-queries.

Design of (a) Hard coded rules. The hard coded baseline is a deterministic lookup table built from the two stage routing protocol in Sec. 3.3 and Prompt F.2. It is not free form prompting, and it is not a learned policy. It takes two inputs: (i) the subquery semantic type (spatial, temporal, attribute, action, text) and (ii) the dominant corruption from the averaged disturbance profile d=(dblur,dbright,doccl)\mathbf{d} = (d_{\mathrm{blur}}, d_{\mathrm{bright}}, d_{\mathrm{occl}}) over selected frames. Stage 1 maps the type to a candidate tool set (e.g., spatial to detect_objects or caption_frame; temporal to track_temporal, recognize_action, or caption_frame; text to read_text or caption_frame; action to recognize_action or caption_frame; attribute to detect_objects or caption_frame). Stage 2 applies fixed corruption preferences: under blur, prefer caption_frame over tools that depend on boundaries, such as detect_objects and read_text; under occlusion, prefer recognize_action over track_temporal; under brightness distortion, prefer tools that are robust to illumination; under clean input, use the default tool for that semantic type. Ties go to the lower cost tool in Tab. 8. These rules are fixed before training, are not updated by GRPO, and do not use frame level learned representations. Full Robust-TO instead selects tools via in context reasoning over the disturbance profile and is trained end to end with GRPO. So (a) is a true predefined rule baseline, distinct from the fixed tool and query only baselines in Fig. 4.

Table R2: Routing ablation on held-out corrupted UV-Bench with Qwen3-VL-8B, evaluated at inference after training. All variants use the same selected frames, tool library, and evaluation protocol; only the routing policy differs. Agreement is the percentage of sub-queries on which a variant selects the same tool as the hard-coded rule in (a).

Routing variantCorrupted Acc. (%)Avg. tool callsLatency (ms)Agreement with (a) (%)
(a) Hard-coded rules48.77.102,441100.0
(b) Prompt-only, no GRPO49.96.802,39874.2
(c) GRPO, guidelines removed51.66.502,35263.8
(d) Full Robust-TO54.36.222,30858.9
AuthorsOpenReview ↗

response for Reviewer oQYu (3/6)

Table R2 separates rule-based from learned routing. The hard-coded baseline reaches 48.7%, while full Robust-TO reaches 54.3% (+5.6 points) even though it agrees with the hard-coded rule on only 58.9% of sub-queries. This gap shows that full Robust-TO does not merely reproduce the predefined rules; GRPO training leads it to a substantially different, non-trivial routing policy. Prompt-only without GRPO reaches 49.9%, and GRPO without the guidelines reaches 51.6%, both below full Robust-TO. This pattern indicates that GRPO contributes beyond both hard-coded rules and prompt-level guidance (+1.7 over prompt-only), while the guidelines add a further +2.7 points on top of GRPO (51.6 to 54.3). Full Robust-TO also uses fewer tool calls (6.22 vs. 7.10) and lower latency (2,308 vs. 2,441 ms) than the hard-coded baseline.

The agreement trend is also informative: agreement decreases monotonically from (a) 100.0 to (b) 74.2, (c) 63.8, and (d) 58.9. This ordering is consistent with increasing policy specialization. The hard-coded rule is a deterministic lookup table by construction, so (a) agrees with itself by definition. Prompt-only routing follows the guidelines but lacks reward-driven adaptation. GRPO without guidelines falls back to simpler semantic-type defaults, which happen to align more often with Stage 1 of the lookup table. Full Robust-TO, trained end-to-end with both guidelines and reward, deviates most from the deterministic table, yet achieves the highest accuracy and lowest cost. This makes the agreement rate a measure of policy divergence from the predefined rule, not a measure of correctness.

These results complement Fig. 4, which compares fixed, query-only, and confidence-guided routing, and Tab. 31, which summarizes learned preferences: Table R2 directly quantifies rule-vs-learning agreement, accuracy, and cost. We include the table and the agreement-rate definition in the revised Sec. 4.3 and Appendix D.8.

The full model agrees with the hard-coded rule on 58.9% of sub-query routing decisions, indicating that the learned policy does not simply reproduce the predefined rule. More directly, GRPO (d) improves corrupted accuracy from 49.9% for prompt-only routing (b) to 54.3% for the full model, a gain of 4.4 percentage points. Compared with hard-coded routing (a), the full model (d) improves accuracy by 5.6 points while reducing average tool calls from 7.10 to 6.22. Its 58.9% agreement with the hard-coded rule further shows that the learned policy does not simply reproduce that rule. Together, these results support the additional benefit of GRPO-trained routing under the evaluated corruption conditions.

Reviewer_SPt6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the robustness of video understanding models under visual degradations such as blur, occlusion, and illumination changes. It proposes Robust-TO, a confidence-aware tool orchestration framework that selects frames based on visual quality and query relevance, decomposes questions into sub-queries, and invokes tools for detection, tracking, captioning, action recognition, and OCR. The framework combines intrinsic tool confidence with input-frame reliability, integrates evidence through a three-tier synthesis procedure, and trains the controller using GRPO rewards incorporating answer correctness, tool confidence, computational cost, and sub-query efficiency. The authors evaluate the framework on UrbanVideo-Bench and VSI-Bench and investigate robustness using controlled visual corruptions. The reported results show accuracy improvements over the evaluated baselines, while ablations provide evidence for the contributions of frame selection, confidence handling, and reward design.

Critical strengths
  1. The paper addresses a practically relevant problem through a coherent design centered on evidence reliability. Blur, occlusion, and illumination changes can impair visual perception tools, while their intrinsic confidence may not adequately reflect input degradation. Explicitly incorporating input quality into frame selection, tool-output confidence, and downstream reasoning is a reasonable and potentially useful approach to improving the robustness of video agents.
  2. The empirical evaluation covers multiple tasks, backbone models, and corruption types, with substantial component-level ablations. Beyond reporting final performance, the paper examines frame selection, input-reliability correction, reliability aggregation, and reward design. These experiments provide meaningful evidence that the proposed components contribute to the observed improvements.
Critical weaknesses
  1. The effectiveness of the evidence-conflict handling rule lacks direct validation. Section 3.3 forms a preliminary conclusion from high-reliability evidence and excludes medium-reliability evidence that conflicts with it. This design can suppress noise, but high-reliability evidence can also be incorrect. Discarding evidence solely because it disagrees with the preliminary conclusion may therefore remove information useful for correcting an error. The confidence ablations support the overall value of reliability scoring, but do not directly establish the benefits and scope of this conflict-handling rule. An analysis of conflicting-evidence cases, or a simple comparison with retaining conflicting evidence for joint reasoning, would help assess its reliability.
  2. The effect of frame selection on critical temporal evidence requires further analysis. The experiments support the benefits of quality- and relevance-based selection in improving average accuracy and reducing computation. However, for tasks involving brief events or first-appearance timing, lower-quality frames may contain information unavailable elsewhere in the video. It remains unclear whether selection removes critical evidence in such cases and whether this constitutes a substantial source of failure. A targeted analysis of prediction changes with and without selection would help clarify the trade-off between visual quality and evidence completeness.
Questions for authors
  1. What are the benefits and costs of the current conflict-handling rule in practice? For cases where high- and medium-reliability evidence support different answers, could the authors analyze when discarding conflicting evidence leads to correct or incorrect final judgments? Alternatively, a small-scale comparison with retaining conflicting evidence for joint reasoning would help assess the effectiveness of the current rule.
  2. How do the calibration results in Table 17 support shared reliability thresholds across tools? Table 17 shows that input-reliability correction reduces ECE for each tool. Please clarify how correctness is defined for each tool’s output, how evaluation samples are selected, how confidence bins are constructed, and whether corrupted inputs are included. These details would help assess whether the calibration results support the subsequent evidence-tier assignments.
  3. Does frame selection remove critical evidence in some temporal tasks? Could the authors examine existing evaluation cases that are answered correctly using the full candidate-frame set but incorrectly after selection, and determine whether brief events or first-appearance information were excluded? Reporting the frequency of such cases and representative examples would help clarify the benefits and costs of selection.
  4. Do more recent mainstream multimodal models exhibit similar sensitivity to visual degradation? Could the authors evaluate a more recent mainstream multimodal model, such as GPT-5.5 or a later version, on the same test examples and corruption settings, and report clean accuracy, corrupted accuracy, and the corresponding drop? Please also specify the model version, input settings, and reasoning budget to support an interpretable comparison.
  5. Please clarify several experimental results and the scope of the comparisons. Tables 20 and 21 both identify Qwen2.5-VL-7B + Robust-TO, but report corrupted accuracies of 47.1% and 54.3%, respectively, for the apparent default configuration. Please clarify whether these results use different configurations or evaluation settings. Additionally, do the abstract’s claimed improvements of 10.6 and 5.8 percentage points over the “strongest open-source baseline” include the tool-agent baselines in Tables 1 and 2? The corresponding margins over VideoAgent are 4.1 and 2.1 points. The abstract also claims the smallest clean-to-corrupted performance drop among all compared methods, whereas Table 2 reports a drop of 3.0 points for Robust-TO and 2.7 points for both VideoAgent and Graph-VideoAgent. Please clarify the comparison sets underlying these statements and correct or explain the discrepancies.
5 comments
AuthorsOpenReview ↗

response for Reviewer SPt6 (1/5)

W1: “The effectiveness of the evidence-conflict handling rule lacks direct validation. Section 3.3 forms a preliminary conclusion from high-reliability evidence and excludes medium-reliability evidence that conflicts with it. This design can suppress noise, but high-reliability evidence can also be incorrect. Discarding evidence solely because it disagrees with the preliminary conclusion may therefore remove information useful for correcting an error. The confidence ablations support the overall value of reliability scoring, but do not directly establish the benefits and scope of this conflict-handling rule. An analysis of conflicting-evidence cases, or a simple comparison with retaining conflicting evidence for joint reasoning, would help assess its reliability.”

A: We agree that high-reliability evidence is not necessarily correct, and that prioritizing it over conflicting medium-reliability evidence may occasionally suppress information that could correct an erroneous conclusion. Our three-tier synthesis is designed to balance this trade-off: it prioritizes more reliable evidence to reduce the influence of corruption-induced noise, but does not assume that high-tier evidence represents ground truth. Importantly, evidence reliability is not determined by tool confidence alone. As defined in Eqs. (4)–(5), the confidence score combines intrinsic tool confidence with a conservative worst-K estimate of input reliability. This reduces the likelihood that evidence derived from severely corrupted frames receives high confidence.

To evaluate the conflict-handling rule independently, during the rebuttal, we have additionally compared five evidence synthesis strategies on UrbanVideo-Bench using Qwen3-VL-8B: Full sequential tiered arbitration (ours), Keep-all-MEDIUM joint reasoning, Only-HIGH evidence, Only-MEDIUM evidence, and Only-LOW evidence. We report Clean Acc. (%) and Corrupted Acc. (%) for each strategy below. The Full Robust-TO values are from the paper; the four ablation rows are reported below:

VariantClean Acc. (%)Corrupted Acc. (%)
Full sequential tiered arbitration (ours)57.354.3
Keep-all-MEDIUM joint reasoning56.453.1
Only-HIGH evidence55.852.6
Only-MEDIUM evidence54.250.1
Only-LOW evidence41.536.8

Compared with retaining all medium-reliability evidence for joint reasoning, our tiered arbitration improves accuracy by 0.9 percentage points on clean videos and 1.2 points on corrupted videos. The Only-HIGH variant also performs worse than the full method, indicating that medium-reliability evidence contributes useful information beyond the high tier. These results provide direct evidence that selective evidence arbitration offers a modest but consistent advantage over unrestricted joint reasoning.

AuthorsOpenReview ↗

response for Reviewer SPt6 (2/5)

W2:

“The effect of frame selection on critical temporal evidence requires further analysis. The experiments support the benefits of quality- and relevance-based selection in improving average accuracy and reducing computation. However, for tasks involving brief events or first-appearance timing, lower-quality frames may contain information unavailable elsewhere in the video. It remains unclear whether selection removes critical evidence in such cases and whether this constitutes a substantial source of failure. A targeted analysis of prediction changes with and without selection would help clarify the trade-off between visual quality and evidence completeness.”

A: We sincerely thank you for raising this concern. Preserving critical temporal evidence is indeed important for brief-event and first-appearance reasoning. We clarify how Robust-TO addresses this challenge and provide additional supporting analysis.

  1. Clarification: selection is not a quality-only filter. Our selection is importance-based, corruption-aware, and combination-based: query relevance is measured by sim(ϕ(fi),ψ(q))\mathrm{sim}(\phi(f_i),\psi(q)) with threshold τsim=0.30\tau_{\text{sim}}=0.30; per-frame disturbance d(fi)d(f_i) covers blur, brightness, and occlusion, and the reliability is �0��0� with threshold �1��1�, so a highly relevant but severely degraded frame cannot dominate the input; and the final score is s(fi)s(f_i)=(1-d(fi)d(f_i)) * sim(ϕ(fi),ψ(q))\mathrm{sim}(\phi(f_i), \psi(q))). This produces a compact key-frame set with adaptive K[4,12], together with a retrieve_frames fallback whose usage rises under corruption from 0.08 to 0.24. Our claim is that selection removes severely degraded frames rather than all low-quality frames; the high clean–corrupted frame-selection overlap shows that selection remains largely stable under the evaluated corruptions and that core frames are therefore not highly inconsistent, while in the clean setting this concern does not arise because selection does not remove critical evidence there, and the high overlap further indicates that corrupted videos do not constitute a substantial failure source.

2. Existing case studies show that critical temporal evidence survives. Tab. 26 first detects the bookshelf at f5 before the glare burst rather than at the obscured f11f_{11}; Tab. 28 recovers the landmark order f7→f13→f21→f27f_7 \rightarrow f_{13} \rightarrow f_{21} \rightarrow f_{27} while suppressing f10f_{10}, which has high similarity but high brightness disturbance; and Tabs. 25 and 27 exclude occluded or blurred frames while still recovering the correct evidence.

3. Additional selected-frame consistency analysis. We report selected-frame consistency between clean and corrupted variants of the same video using overlap/Jaccard below, which directly shows how corruption shifts selection and whether query-relevant frames above the reliability threshold are retained under both conditions. In a pilot study on held-out subsets of UrbanVideo and VSI-Bench with Qwen3-VL-8B as the backbone, we observe high overlap:

BenchmarkClean-to-Corrupted JaccardBrief-event / temporally localized subsetObject / scene-centric subset
UrbanVideo0.880.840.91
VSI-Bench0.860.820.90
Benchmark / Task typeSelected-frame Jaccard
UrbanVideo — Landmark Position0.85
UrbanVideo — Action Generation0.83
UrbanVideo — Perception0.92
VSI-Bench — Relative Distance0.87
VSI-Bench — Appearance Order0.81

These results indicate that corruption shifts selection only modestly and that the selected key-frame sets remain highly consistent; as expected, brief-event and temporally localized subsets show slightly lower overlap than object- or scene-centric subsets, but the overlap remains high.

AuthorsOpenReview ↗

response for Reviewer SPt6 (3/5)

Q1: “What are the benefits and costs of the current conflict-handling rule in practice? For cases where high- and medium-reliability evidence support different answers, could the authors analyze when discarding conflicting evidence leads to correct or incorrect final judgments? Alternatively, a small-scale comparison with retaining conflicting evidence for joint reasoning would help assess the effectiveness of the current rule.”

A: We would like to clarify that our arbitration rule does not treat high-tier evidence as ground truth or unconditionally discard conflicting medium-tier evidence. High-tier evidence anchors a preliminary conclusion, while consistent medium-tier evidence is retained. Conflicting medium-tier evidence serves as a conflict signal that reduces confidence and triggers re-checking. When no high-tier evidence is available, the method jointly reasons over medium-tier evidence, using low-tier evidence only as a fallback with explicit residual uncertainty.

The comparison on UrbanVideo-Bench with Qwen3-VL-8B is:

VariantClean Acc. (%)Corrupted Acc. (%)Context (tokens)End-to-end latency (ms)
Full sequential tiered arbitration (ours)57.354.35122,308
Keep-all-MEDIUM joint reasoning56.453.15442,316
Only-HIGH evidence55.852.64482,296
Only-MEDIUM evidence54.250.14802,302
Only-LOW evidence41.536.84322,294

The full arbitration rule achieves the highest accuracy among all tested variants on both clean and corrupted inputs. It outperforms keep-all-MEDIUM joint reasoning by 0.9 and 1.2 percentage points, respectively, supporting the effectiveness of selective arbitration. It also exceeds Only-HIGH by 1.5 and 1.7 points, demonstrating the benefit of incorporating medium-tier evidence through the tiered procedure.

These gains come at comparable computational cost: the full rule uses 512 context tokens versus 544 for keep-all-MEDIUM, with measured end-to-end latency of 2,308 ms versus 2,316 ms. Overall, the comparison supports selective arbitration as an effective practical choice, delivering higher accuracy without additional measured context or latency cost relative to retaining all medium-tier evidence.

Q2: “How do the calibration results in Table 17 support shared reliability thresholds across tools? Table 17 shows that input-reliability correction reduces ECE for each tool. Please clarify how correctness is defined for each tool’s output, how evaluation samples are selected, how confidence bins are constructed, and whether corrupted inputs are included. These details would help assess whether the calibration results support the subsequent evidence-tier assignments.”

A: We thank you for this helpful comment. Table 17 shows that the F correction puts confidences from different tools on a common, better calibrated scale: mean ECE:0.232→0.089\mathrm{mean\ ECE}: 0.232 \rightarrow 0.089. This is why we can apply one fixed HIGH/MEDIUM/LOW threshold set across tools and conditions;

we are not claiming these thresholds are optimal for every individual tool. Appendix D.6 and Sec. 3.3 spell out the protocol: correctness is defined per tool, each successful call contributes 11 sample, failed calls (cj=0)(c_j=0) are excluded from binning but counted as failures, confidence is split into 1010 equal-width bins over [0,1][0,1], ECE is the bin-size-weighted gap between mean confidence and empirical accuracy, ECE=∑b=110nbN∣accb−confb∣\mathrm{ECE}=\sum_{b=1}^{10}\frac{n_b}{N}\left|\mathrm{acc}_b-\mathrm{conf}_b\right|, and we report clean, corrupted, and tier-wise results.

The fixed boundaries on F-corrected confidence are HIGH\mathrm{HIGH} when cj≥0.7c_j \ge 0.7 and d<0.3d < 0.3, LOW\mathrm{LOW} when cj<0.3c_j < 0.3 or d≥0.7d \ge 0.7, and MEDIUM\mathrm{MEDIUM} otherwise, so the evidence tiers rely on the calibrated scale rather than raw tool-specific scores.

Reviewer_KMKdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the "Blind Trust Problem" in existing video reasoning VLMs, where visually degraded frames are treated as equally reliable evidence, potentially leading to incorrect predictions under realistic corruptions such as motion blur, glare, and occlusion. Motivated by the need for robust video understanding, the paper argues for explicitly incorporating frame-level reliability into evidence selection, tool orchestration, and reasoning.

The authors proposes Robust-TO, a confidence-aware tool orchestration framework consisting of three stages: quality-aware frame selection, confidence-guided tool routing, and reliability-aware evidence synthesis. It first estimates frame quality based on blur, brightness, and occlusion, and selects informative frames using a reliability-relevance score. The framework then decomposes questions into atomic sub-queries and routes them to specialized perception tools. then the collected evidence is integrated through a three-tier confidence mechanism. Furthermore, the author introduce a confidence-cost GRPO reward to jointly optimize answer correctness, evidence reliability, and tool-use efficiency.

The proposed framework is built on Qwen2.5-VL-7B and Qwen3-VL-7B. Evaluations are conducted on UrbanVideo-Bench and VSI-Bench, covering eight video reasoning tasks, with additional robustness evaluations under five visual corruption types.

Critical strengths
  • Overall, I appreciate the general motivation of this paper and its identification of the "Blind Trust Problem" in existing video reasoning VLMs. Explicitly considering the reliability of visual evidence is a meaningful direction for improving robustness.
  • The proposed confidence-aware tool orchestration framework is intuitive and well-motivated.
  • The paper provides relatively comprehensive experiments, including evaluations across two base models, multiple corruption types, and ablation studies on key components.
Critical weaknesses

My main concerns lie in the following aspects:

  • The proposed frame reliability estimation is relatively heuristic. It mainly relies on low-level visual statistics, including blur, brightness, and edge density. However, these signals may not accurately reflect the semantic reliability of a frame. For example, a visually bad frame may still contain crucial information for answering the question.
  • The robustness evaluation is somewhat limited. The corrupted videos are mainly generated through synthetic perturbations applied to a subset of frames. which leaves relatively clean frames available for selection. It remains unclear whether the proposed Robust-TO can maintain its advantages under more challenging scenarios. For example, continuously degraded videos where the key evidence appears only in corrupted frames.
Questions for authors

Please see the weaknesses part.

4 comments
AuthorsOpenReview ↗

response for Reviewer KMKd (1/2)

W1: “The proposed frame reliability estimation is relatively heuristic. It mainly relies on low-level visual statistics, including blur, brightness, and edge density. However, these signals may not accurately reflect the semantic reliability of a frame. For example, a visually bad frame may still contain crucial information for answering the question.”

We thank you for this important concern. We clarify that d(f)d(f) is not a semantic reliability score; it is only a parameter-free degradation prior. Table A3 separates visual quality from query relevance to test whether a visually bad but question-critical frame is wrongly discarded:

Table A3: Two-factor decomposition (visual quality vs. relevance to query)

Selection scoreAvg. framesClean Acc.Corrupted Acc.
Visual only s=1−ds=1-d19.846.643.2
Relevance only s=((fi),(q))s=((f_i),(q))22.348.944.8
Joint s=(1−d)((fi),(q))s=(1-d)((f_i),(q))20.750.747.1

Visual-only selection drops the most frames (19.8) yet performs worst (46.6/43.2), matching your concern that quality-only filtering can discard question-critical frames. Relevance-only selection improves (48.9/44.8) but still admits corrupted yet query-relevant frames, which is the Blind Trust Problem in Sec. 3.1 and Fig. 1(b). The joint score is best on both clean (50.7) and corrupted (47.1), improving over visual-only by 4.1 and 3.9 points and over relevance-only by 1.8 and 2.3 points while using fewer frames than relevance-only (20.7 vs. 22.3), so the gain is not from processing more frames.

The product enforces a conjunctive rule: the score is small if either factor is small, and (rel,sim)=(0.55,0.30) prevent high relevance from rescuing badly corrupted frames or high quality from rescuing irrelevant frames. Semantic reliability is handled downstream by cj=cjintrinsic(Fj)c_j = c_j^{\mathrm{intrinsic}}(F_j) with worst-K input reliability in Eqs. (4)–(5), and low-tier evidence is only a fallback with explicit residual uncertainty. Table A3 thus shows that visual quality is not treated as semantic reliability, and a visually bad but query-relevant frame is not admitted merely for being relevant; it must also pass the degradation prior, while downstream confidence and tiering decide how much its evidence can be trusted.

AuthorsOpenReview ↗

response for Reviewer KMKd (2/2)

W2: “The robustness evaluation is somewhat limited. The corrupted videos are mainly generated through synthetic perturbations applied to a subset of frames. which leaves relatively clean frames available for selection. It remains unclear whether the proposed Robust-TO can maintain its advantages under more challenging scenarios. For example, continuously degraded videos where the key evidence appears only in corrupted frames.”

We thank you for this comment, but respectfully disagree with the argument that our robustness evaluation is limited. First, our evaluation targets a practically relevant setting: motion blur, glare, occlusion, and low-light conditions often affect particular moments or regions of a video, leaving other evidence usable. The subset-frame perturbation protocol is designed to model this intermittent degradation. As reviewers SPt6 and oQYu also acknowledge (SPt6: "The paper addresses a practically relevant problem ...", oQYu: "Practical motivation. Robust-TO addresses a meaningful problem, ..., are common in real-world video capture."), these corruption types address meaningful problems in real-world videos.

Second, all methods, including the base VLMs, Video-R1, and Robust-TO, are evaluated on the same corrupted videos and have access to the same remaining clean evidence. Standard VLMs can therefore also benefit from these frames; Robust-TO receives no additional intact evidence unavailable to the baselines. The comparison evaluates how reliably each method uses the available evidence in the presence of corruption.

Finally, we would like to distinguish severe degradation from irreversible information loss. If corruption destroys the information required to answer a question throughout the available video, no method relying solely on that video can reliably recover the answer. This limitation is shared across methods.

Robust-TO mitigates overconfidence through the worst-K input reliability term in Eqs. (4) and (5). By conservatively estimating reliability from the selected frames, this term reduces confidence when the available evidence is severely degraded, preventing a few seemingly clean frames from masking unreliable evidence.

Importantly, Robust-TO can still recover correct answers when corruption is substantial, but the relevant information remains accessible. For example, in the VSI-Bench Relative Distance case (Table 25), a person occludes approximately 55% of the view in frames f8f_8–f11f _{11}, causing the baseline video LLM to incorrectly answer “trash bin.” In contrast, Robust-TO excludes the occluded frames and correctly identifies “kitchen sink” using scene-graph adjacency. Further examples in Tables 27–29 demonstrate its ability to handle blur, glare, and occlusion through reliability-aware frame selection and routing. These case studies, already included in the Appendix, illustrate how Robust-TO handles severe but recoverable visual corruption.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer KMKd,

Thank you again for your detailed and constructive review. As the discussion deadline is approaching, we would like to briefly summarize what we did during the rebuttal and kindly ask you to check whether your concerns have been adequately addressed.

Heuristic frame reliability (W1): We added a two-factor ablation separating visual quality from query relevance. The joint score performs best on both clean and corrupted inputs (50.7 / 47.1), without using more frames. We also clarified that semantic reliability is handled downstream by the worst-K confidence term and three-tier synthesis.

Limited robustness evaluation (W2): We clarified that all methods see the same remaining evidence, and added recoverable-corruption case studies and a paired frame-selection analysis showing critical evidence is lost on only ~0.6% of questions.

We hope these additions address your concerns, and we would be grateful if you could reconsider your recommendation. We are happy to provide further details or experiments if needed.

Best regards, Authors

AuthorsOpenReview ↗

Gentle Follow-up as the Discussion Period Ends in Less Than 3 Hours

Dear Reviewer KMKd,

Thank you once again for your time, effort, and insightful feedback on our manuscript.

As the discussion period ends in less than 3 hours, we wanted to follow up and ask whether our responses and revisions have adequately addressed your concerns. If you find our clarifications satisfactory, we would be deeply grateful if you would consider re-evaluating our work. Of course, if any questions or points remain unclear, we would be more than happy to discuss them further before the deadline.

Thank you very much for your consideration.

Best regards,

The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer SPt6,

Thank you again for your detailed and constructive review. As the discussion deadline is approaching, we would like to briefly summarize our rebuttal and kindly ask you to check whether your concerns have been adequately addressed.

Evidence-conflict handling (W1/Q1): We compared five synthesis strategies. Tiered arbitration performs best on clean and corrupted inputs (57.3 / 54.3) at comparable latency.

Frame selection and critical temporal evidence (W2/Q3): We ran a paired prediction-flip analysis on 1,028 questions. Critical evidence was lost in only ~0.6% of questions.

Shared Thresholds, Calibration, and Evidence Tiers (Q2): We have clarifed the shared thresholds provide a practical unified reliability interface rather than per-tool optimal tuning.

More recent models (Q4): We evaluated GPT-5.5 under the same protocol. It still drops 7.7 points under corruption, so the sensitivity is not specific to the models in our main tables.

Table and abstract inconsistencies (Q5): We corrected the Table 21 typo, narrowed the abstract’s “strongest baseline” and “smallest drop” claims to non-agentic baselines, and reported margins over VideoAgent separately.

We hope these additions address your concerns, and we would be grateful if you could reconsider your rating. We are happy to provide further details or experiments if needed.

Best regards,

Authors

Thanks to the authors for the additional experiments. I appreciate the idea of accounting for frame quality when weighing tool outputs. The arbitration comparison gives me more confidence in the tiered approach, and the frame-selection analysis clarifies how often filtering removed critical temporal evidence on clean inputs.

However, I still have one remaining question about conflict handling. Section 3.3 says that medium-reliability evidence is discarded when it conflicts with the preliminary conclusion, while the rebuttal says that the conflict triggers a re-check. In the original experiments, what exactly did the model re-check, and could the outcome lead it to revise the conclusion drawn from high-reliability evidence?

I will reconsider my rating after I receive your response.

Thank you for your constructive follow-up and for acknowledging our additional experiments and clarifications!

In our original experiments, conflicting medium-tier evidence was not simply discarded. Instead, the model flagged the conflict, lowered its confidence, and re-examined the relevant tool outputs and visual evidence to identify the source of inconsistency. When needed, the framework could also retrieve additional frames or invoke a more robust tool.

Importantly, the high-tier conclusion was provisional and could be revised with lower reported confidence if the rechecked evidence became consistent and high confidence or the input reliability check showed that the original high tier evidence came from degraded frames.

We have clarified this conflict-handling procedure in Sec. 3.3 to avoid further ambiguity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.