acceptodds
Under review as a conference paper at ICLR 2027

OmniAgent: Active Perception Agent for Omnimodal Audio-Video Understanding

Abstract

Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often face challenges in fine-grained cross-modal understanding and have difficulty with multimodal alignment. To address these limitations, we introduce OmniAgent, to the best of our knowledge, the first fully active perception agent that dynamically orchestrates specialized unimodal tools to achieve more fine-grained omnimodal reasoning. Unlike previous works that rely on rigid, static workflows and dense frame-captioning, we demonstrate a paradigm shift from passive response generation to active multimodal inquiry. OmniAgent employs dynamic planning to autonomously orchestrate tool invocation on demand, strategically concentrating perceptual attention on task-relevant cues. Central to our approach is a novel coarse-to-fine audio-guided perception paradigm, which leverages audio cues to localize temporal events and guide subsequent reasoning. Extensive empirical evaluations on three audio-video understanding benchmarks demonstrate that OmniAgent achieves state-of-the-art performance, surpassing leading open-source and closed-source models by substantial margins of 10% - 20% accuracy without training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
OmniLLMs, Audio-Video Understanding, Agent

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_RCaCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OmniAgent, an active perception agent framework for joint audio-video understanding. Experimental results demonstrate that OmniAgent outperforms existing end-to-end multimodal models and agent-based methods across multiple benchmarks. Furthermore, the authors conduct ablation studies on different backbone models, tool combinations, and scheduling strategies to validate the effectiveness of the proposed framework.

Critical strengths

The paper presents a flexible active perception framework for audio-video understanding. The proposed method achieves strong performance across multiple audio-video understanding benchmarks.

Critical weaknesses

Concerns about the fairness of experimental comparisons; Limited methodological novelty.

Questions for authors
  1. OmniAgent integrates several powerful models, including OpenAI o3, Qwen3-VL, Qwen3-Omni, and Gemini 2.5-Flash, whereas some competing methods rely on a single end-to-end model. Could the authors provide further comparisons with existing agent-based methods under matched backbone models, tool access, and inference budgets to demonstrate that the performance improvements primarily arise from the proposed active perception strategy rather than the use of stronger model combinations?

  2. Compared with existing active perception agents and multimodal agent frameworks, what are the fundamental algorithmic innovations of OmniAgent? In particular, could the authors clarify the key differences between the proposed audio-guided event localization mechanism and existing audio event localization and temporal grounding approaches?

  3. Could the authors provide a more comprehensive analysis of inference costs across different benchmarks, including total token consumption, number of model calls, and end-to-end inference latency, along with comparisons against agent-based methods under comparable computational budgets? Could these additional computational costs limit the practical applicability of the framework to long-form video understanding tasks?

Reviewer_QiidOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

OmniAgent is a training-free framework for audio-video understanding that uses a reasoning LLM to dynamically select perception tools and gather evidence. It emphasizes audio-guided temporal localization followed by fine-grained visual inspection. Evaluation on Daily-Omni, OmniVideoBench, and WorldSense shows consistent accuracy improvements, supported by tool and scheduling ablations.

Critical strengths
  1. Effective and practical design. The framework addresses a meaningful limitation of global audio-video processing by selectively inspecting relevant modalities and temporal segments. Its modular, training-free implementation makes the approach practically useful.
  2. Strong empirical results. Improvements across three benchmarks demonstrate broad effectiveness. The tool-removal and scheduling ablations provide supporting evidence that fine-grained perception and adaptive tool selection contribute to performance.
Critical weaknesses

The main remaining concern is insufficient isolation of the framework’s contribution beyond a strong reasoning model equipped with suitable tools. Fixed-order and random scheduling are useful controls, but do not represent a competent general-purpose agent. Without a generic tool-calling baseline using the same o3 model, identical perception tools, and matched inference budgets, the additional benefit of OmniAgent’s specific orchestration and prompting remains unclear.

Questions for authors

Could you provide a controlled comparison against a generic tool-calling/ReAct agent using the same o3 version and reasoning settings, identical tool implementations, and matched inference budgets? Please report accuracy alongside actual tool usage and cost, and clarify which components distinguish OmniAgent from this baseline. I currently recommend 3 (Weak acceptance). Convincing evidence that the proposed framework consistently improves over this controlled baseline would support raising my recommendation to 4 (Clear acceptance).

Reviewer_vofBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OmniAgent is a training-free audio-video QA agent in the ReAct style. It uses an LLM as the "brain" (OpenAI o3 by default), running a Think-Act-Observe-Reflect loop and calling tools that wrap off-the-shelf models: for video, Global QA and Clip QA (both Qwen3-VL); for audio, ASR and Global Caption (Qwen3-Omni) plus Audio QA (Gemini-2.5-Flash); for events, Event List and Event Location (both Gemini-2.5-Flash, prompted to output timestamps from the audio track). The core idea is "audio-guided coarse-to-fine" perception: audio first narrows down the relevant time span, then video is examined in that span at higher FPS and resolution. It scores 82.71 on Daily-Omni, 59.10 on OmniVideoBench, and 61.2 on WorldSense, all above the reported numbers for Qwen3-Omni-30B, Gemini-2.5-Flash, and earlier audio-visual agents. The paper also studies tool-usage patterns, brain-model choice, tool-model choice, and several tool ablations.

Critical strengths
  1. The problem is relevant. End-to-end OmniLLMs do struggle with fine-grained, temporally localized audio-visual reasoning. Using audio as a cheap temporal index for long videos is an intuitive, practical idea.
  2. The system is simple to understand. The appendix gives all agent and tool prompts, which helps with transparency.
  3. The absolute numbers on the three benchmarks are high. The paper also analyzes tool-call distributions across brain models and benchmarks (Fig. 5, 7, 8), which some readers will find informative.
Critical weaknesses
  1. Limited technical novelty; overstated claims. (a) The framework is a standard ReAct loop (Yao et al., 2022) over a toolbox of off-the-shelf model APIs; Eq. (1)–(2) are generic agent equations. Sec. 3.1 frames the problem as sequential decision making that "maximize[s] the information gain," yet there is no learned policy, the state s_t is never defined, and no information-gain quantity is computed or used—the formulation is decorative. (b) The "novel audio-based event localization strategy/algorithm" is, as far as Sec. 3.2 and App. H.5/H.7 show, two prompts asking Gemini-2.5-Flash to list audio segments or return timestamps for a query. There is no algorithmic component: no proposal generation, no scoring, no cross-modal verification, no handling of audio-visual asynchrony. Using audio or ASR timestamps to localize events for video QA has been done before (e.g., the Daily-Omni agent, XGC-AVis, and audio-guided methods the paper itself cites). (c) The claim of being "the first fully active perception agent for omnimodal understanding" is unconvincing: several cited works (XGC-AVis, the Daily-Omni agent, MAGNET; Active Video Perception and DeepVideoDiscovery for video) already use agents with tool calls and iterative evidence collection. The "static/fixed" vs. "active" distinction is never rigorously defined, and no experiment isolates it under controlled conditions (see point 3). (d) Claims that the method "solves the cross-modal alignment problem" (Contributions, Fig. 2, Sec. 3.1) are unsupported. The agent turns each modality into text and lets an LLM reconcile the texts—that sidesteps alignment rather than solving it. No experiment measures alignment (e.g., audio-visual synchrony, sound-source grounding, AV event localization accuracy). Phrases such as "paradigm shift," "genuine fine-grained understanding," and "mirroring human cognitive processes" are likewise unsupported.

  2. The main comparisons are not apples-to-apples, so the reported gains cannot be credited to the proposed design. (a) OmniAgent combines several frontier systems—o3 + Gemini-2.5-Flash + Qwen3-VL + Qwen3-Omni—with up to 30 tool calls per question, yet it is compared with single forward passes of Qwen3-Omni-30B and Gemini-2.5-Flash. Gemini-2.5-Flash is itself one of OmniAgent's tools, so the system strictly contains its strongest baseline. "Without training" makes the result sound cheaper than it is: the method is training-free only because it uses much stronger and much more expensive proprietary models at inference time. (b) Table 4(a) shows that replacing all tools with Gemini-2.5 gives 83.3, above the default configuration, and Fig. 6 shows that with GPT-4o as the brain OmniAgent scores 68.7—below its own tool Gemini-2.5-Flash used alone (72.7) and below Qwen3-Omni-30B (72.08); with o4-mini the score is 75.9. Most of the gain therefore appears to come from o3's reasoning ability plus repeated calls to Gemini (an ensembling or test-time-compute effect), not from the proposed agent design. (c) Key controlled baselines are missing, so the contribution of "active perception" cannot be separated from "a strong reasoning LLM + strong perception APIs": o3 (or the same brain) given uniformly sampled frames plus the full ASR transcript and audio caption in one pass (a static pipeline with the same models); Gemini-2.5-Flash / Gemini-2.5-Pro with CoT and self-consistency or majority voting at matched API cost / number of calls; and the competing agents (XGC-AVis, Daily-Omni agent, DVD) re-run with the same brain (o3) and the same tool models, or OmniAgent run with their backbones. (d) Stronger and more recent end-to-end baselines are absent: Sec. 4.1 says the paper compares with "OpenAI o3," but no o3 row appears in Tables 1–3; Gemini-2.5-Pro is missing; Qwen3.5-Omni and Qwen3.8-Omni are cited in the Introduction (Team 2026a,b) but not compared. (e) Almost all baseline numbers are copied from prior reports (App. A), coming from different frame budgets, prompts, and possibly different evaluation protocols. Many entries are incomplete: Gemini-2.5-Flash* in Table 1 and Qwen3-Omni in Table 3 have only an average, and Table 2 has no agent baselines at all. DVD is a video-only agent evaluated only on Daily-Omni, with a modified FPS and a swapped captioner; App. A says DVD was tested only on Daily-Omni because of cost, while App. C says it was also run on "a randomly sampled subset" of the other benchmarks—yet those accuracies are not reported. (f) The headline claim of a "10%–20%" margin over leading open- and closed-source models (Abstract, Intro) is inconsistent with the tables: on OmniVideoBench the gain over Gemini-2.5-Flash is 6.7 points (59.10 vs. 52.40), and on WorldSense it is 4.7 points over video-SALMONN 2+ (61.2 vs. 56.5) and 7.2 points over Qwen3-Omni (61.2 vs. 54.0). The paper also never says whether the margin is absolute or relative.

  3. The ablations are insufficient and partly contradict the narrative. (a) Reflection is presented as a core component—a "cross-modal consistency check" in Fig. 2, Fig. 3, and Sec. 3.3—but it is never ablated, and its implementation is described only through the system prompt in App. G.1; the same holds for the memory and re-planning modules in Fig. 3. (b) Table 4(b) covers only 3 of the 7 tools and only 4 configurations; ASR, Global Caption, Event List, and Global QA are never ablated. The text calls the drop from removing Audio QA "substantial," but it is 2.5 points (82.7 → 80.2). (c) The "Fixed-order" (66.7) and "Random-choose" (65.2) baselines in Fig. 6 are not specified—which order, how many calls, what answer synthesis—and both score below a single Gemini-2.5-Flash call (72.7), suggesting poorly constructed straw men rather than meaningful controls. (d) All ablations use only Daily-Omni, whose clips are 30s/60s, whereas the paper's main argument concerns localization in long videos; OmniVideoBench (up to 30 min) is the natural place for these ablations and none appear there. The effect of the step budget (max 30, average 6.7) is not studied either. (e) Event localization accuracy—the core claimed contribution—is never evaluated directly (e.g., temporal IoU / recall@k on an audio-visual event grounding benchmark), and there is no analysis of how localization errors propagate to final accuracy, even though App. E names error propagation as a key issue.

  4. The behavioral analyses are partly circular. Finding 1 and Fig. 5/8 present the "audio first, then fine-grained video" pattern as evidence that "empirically validates the efficacy of our algorithmic design," but App. G.1 and G.2 explicitly instruct the brain to "Prefer to listen first, then look" and to "Use audio to find time and content first whenever possible, then inspect the corresponding visuals: from listen to look." The observed pattern is induced by the prompt, not emergent, which also contradicts the claim (Intro) that the decision process is "completely autonomous, decided by the LLM itself." In addition: Finding 2 contradicts itself—ELO is first described as returning "precise timestamps" from "low-cost" audio (Sec. 3.1–3.2), but Finding 2 calls ELO and VGA "computationally intensive tools [that] provide macro-level insights … unable to provide granular, fine-grained details." Fig. 7 shows that 41% of correctly answered WorldSense questions are solved with audio tools alone, suggesting a large share of the benchmark can be solved from transcripts or text shortcuts—this weakens rather than supports the claim of fine-grained cross-modal reasoning; evaluation on benchmarks designed against text shortcuts (e.g., AVUT, Yang et al. 2025c, cited in the paper) or on long audio-video benchmarks (e.g., LVOmniBench, Tao et al. 2026, also cited) would be more convincing. Finding 3 (GPT-4o "modality bias") is qualitative and not quantified.

  5. The interpretability claims rest on post-hoc summaries. The caption of Fig. 1 admits: "Due to the unavailability of the thinking process, we use reflection in the final summary," and the system prompt (G.1) tells the model "Do NOT expose raw tool-call traces or long chain-of-thought." The "transparent, explainable" reasoning in Fig. 1, 4, and 9–12 is therefore a self-reported summary written after the fact by o3 and may not be faithful to the real decision process. All case studies are success cases (and arguably cherry-picked), with no failure or error analysis.

  6. Statistical rigor and evaluation protocol are weak. There are no repeated runs, variance, or confidence intervals, although the authors acknowledge the agent is stochastic (App. E); temperature and o3 reasoning-effort settings are not reported. "Queries interrupted by connectivity issues were re-evaluated in a final testing phase" (App. A)—how many queries were affected, and were re-runs applied uniformly? Selective re-evaluation can bias results. When the 30-step limit is reached, o3 "synthesizes" an answer: how often does this happen, and how accurate are these forced answers? The answer parsing and failure-handling protocol for the multiple-choice format is not described.

Questions for authors

See weekness above

1 comment
Reviewer_vofBOpenReview ↗

No rebuttal for this review.

This article is a relatively large topic, and I raised some detailed questions, nice responce, I will increase his score.

Reviewer_BdWUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OmniAgent, a training-free active perception framework for fine-grained audio-video understanding. It uses a central LLM to dynamically select specialized audio, video, and event perception tools through an iterative Think-Act-Observe-Reflect process. A key component is audio-guided temporal event localization, which identifies relevant timestamps to guide subsequent high-resolution visual analysis and cross-modal reasoning. The framework is evaluated on three benchmarks: Daily-Omni, OmniVideoBench, and WorldSense, achieving accuracies of 82.71%, 59.10%, and 61.20%, respectively. Results outperform several existing omnimodal models and agent-based baselines, while ablation studies investigate the contributions of individual tools, reasoning models, and adaptive tool selection.

Critical strengths
  • OmniAgent demonstrates strong empirical performance compared to existing omnimodal models and agent-based approaches.
  • The idea of using audio cues to first identify relevant temporal segments and then perform fine-grained visual analysis is interesting and practical. It allows the agent to focus on important parts of the video instead of processing the entire input at the same level of detail.
  • Ablations on different perception tools, backbone models, and tool-selection strategies help explain the contribution of different components.
  • The framework combines existing audio and visual models without requiring additional training or explicit joint representation alignment, which makes the approach relatively easy to extend or adapt by replacing individual perception tools with stronger models.
Critical weaknesses
  • Limited evaluation of temporal localization: Although audio-guided event localization is a central contribution, the paper does not directly evaluate timestamp accuracy or localization performance against ground-truth annotations. Such an evaluation would have further supported the effectiveness of the proposed temporal grounding mechanism.
  • Dependence on strong external models: The framework relies on several powerful models, including OpenAI o3, Gemini 2.5-Flash, and Qwen3-VL, which makes it difficult to completely separate the contribution of the agent framework from the capabilities of its underlying models.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.