acceptodds
Withdrawn from ICLR 2027

RTVA: A Framework for Real-Time Visual Agents in Evolving Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Alongside task-oriented visual agents that operate through turn-based action loops, recent proactive visual agents aim to support more natural interaction by continuously perceiving evolving environments and responding in real time. Enabling such agents requires more than low latency: perception, response generation, and LLM reasoning proceed concurrently but at different timescales, making it difficult to keep ongoing interaction aligned with a changing environment. Different tasks also demand distinct patterns of intervention and adaptation. We characterize streaming visual interaction through three temporal boundaries: before response initiation, during response delivery, and when delayed reasoning returns. We introduce RTVA, a framework that decomposes real-time visual interaction along these boundaries into composable components within a dual-system architecture. These components separate content generation from whether, when, and how generated content may influence interaction. To adapt RTVA to task-specific interaction requirements, AutoHarness searches component compositions across these boundaries, then optimizes their behavioral policies. RTVA achieves the strongest overall cross-domain performance in both temporal alignment and interaction quality for real-time visual interaction. We further introduce the Visual In-Flight (VIF) Benchmark, a complementary capability evaluation of whether ongoing responses remain grounded, current, and coherent as visual evidence changes.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Real-Time Visual Agents, Streaming Multimodal Interaction, Agent Harnesses

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9JcuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the challenge of keeping visual agents aligned with changing environments during interaction. It introduces a composable framework for response initiation, in-flight adaptation, and delayed-thought commitment. Besides that, authors also introduce AutoHarness for task-specific optimization and construct VIF for evaluating the performance.

Experiments show improvements across three domains, and ablations support the proposed components. The work offers a useful systems perspective and several meaningful experimental controls. However, the system design lacks sufficient innovation and reads more like a patchwork of disparate components. For instance, I fail to see the necessity of integrating AutoHarness. Perhaps the authors should reflect on how to better tell the story of fast adaptation to changing environments.

Critical strengths
  1. The paper addresses how visual agents can remain aligned with changing environments throughout an ongoing interaction. RTVA provides a concrete design that coordinates response initiation, in-flight adaptation, and delayed-thought commitment.

  2. The paper addresses how visual agents can remain aligned with changing environments throughout an ongoing interaction, providing a concrete design that coordinates response initiation, in-flight adaptation, and delayed-thought commitment.

Critical weaknesses
  1. The paper lacks validation of token probability biases and calibration when System 1 decides whether to speak or remain silent.

  2. The paper does not specify the duration of the test videos and lacks long-duration evaluations of inference lag and error accumulation.

  3. The presentation needs improvement. AutoHarness is tangential to the main topic and offers little novelty, yet occupies considerable space, while the key equations and evaluation metrics are insufficiently explained.

  4. The comparisons primarily involve models without task-specific designs and general-purpose agent frameworks, lacking fair comparisons with closely related methods.

Questions for authors
  1. How does the model used in System 1 differ from a dedicated decision model such as Jev? If System 1 relies on next-token probabilities to choose between “speak” and “silent,” have you evaluated how token preferences affects these decisions?

  2. How long are the video sequences in the evaluation, and how frequently do relevant environmental changes occur? How does the system handle situations where inference repeatedly falls behind these changes? Have you tested whether persistent lag leads to error accumulation during prolonged operation?

  3. Beyond native real-time models, is there any other existing approaches address state information becoming outdated during inference? If so, how does RTVA's performance compare with these methods under comparable experimental conditions?

Reviewer_CA2TOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RTVA organizes streaming visual interaction around response initiation, in-flight adaptation, and validation of delayed reasoning. It combines a foreground interaction loop, asynchronous background reasoning, and shared task memory. AutoHarness searches task-specific configurations, while the proposed VIF benchmark evaluates continue, queue, and revise behaviors under controlled interaction histories. Experiments cover game commentary, procedural guidance, and GUI assistance, with improvements over several existing systems, including a comparison using the same foreground and background models.

Critical strengths
  1. Relevant problem and measurable improvements. Handling changing evidence and outdated background reasoning is practically important. The comparison using the same foreground and background models, together with component ablations, shows improvements in the reported metrics across several domains.
Critical weaknesses
  1. Limited methodological novelty and insufficient comparison with simple alternatives. RTVA's contribution appears to lie mainly in integrating existing components into a configurable framework. Related mechanisms include ongoing-utterance verification in LyraV (Yang et al., 2026) and asynchronous delegation with freshness deadlines in Realtime-Venus (2026). The always-apply ablation shows that filtering helps, but does not establish the semantic gate's benefit over a simple timeout-based filter.

  2. Incomplete validation of real-time interaction benefits. VIF timestamps actions at request time, excludes API wall-clock latency, and permits success despite an earlier incorrect adaptation action. Appendix G reports completion latency but lacks exact first-output timestamps for revisions. These limitations make the results less conclusive about whether corrections reach users promptly and without unnecessary disruption.

Questions for authors
  1. Ablation of semantic validation. Could the authors compare the semantic gate with a simple timeout-based filter that discards results older than a tuned threshold? This would help clarify the contribution of semantic validation beyond simple filtering.

  2. Timing and action scoring. How robust are the VIF gains when actions are timestamped by actual output availability and earlier incorrect adaptation actions count against success?

Reviewer_dt5iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents RTVA, a modular framework for visual agents interacting with changing streams. It separates foreground interaction from asynchronous reasoning and controls three boundaries: response initiation, in-flight revision or queuing, and commitment of delayed thoughts. AutoHarness searches component compositions before tuning their behavior. The paper also introduces the Visual In-Flight (VIF) benchmark to test adaptation during ongoing speech. Evaluation covers game commentary, procedural guidance, and GUI assistance.

Critical strengths
  1. The paper identifies a real failure mode: a response can become outdated after it starts, and background reasoning can become stale before it returns. Separating generation from the decision to apply its output is a useful systems abstraction. Table 2 also shows that blindly committing delayed thoughts can be worse than omitting asynchronous reasoning.
  2. The evaluation spans three different interaction tasks and includes a matched GPT-Realtime/GPT-5.4 comparison. VIF usefully attempts to isolate decisions during an active response rather than judging only response initiation or final answers. The LiveGaming and GUIDE-Bench gains suggest the full framework can improve practical interaction.
Critical weaknesses
  1. VIF only partially captures the claimed real-time problem. Appendix F assigns actions to source-video ticks and explicitly excludes API wall-clock latency. An action can meet the three-second VIF window even if revised speech would reach the listener too late. Appendix G reports substantial revision completion times but lacks exact first-output timestamps. VIF also supplies a reconstructed, already-playing speech prefix instead of requiring the agent to generate that prefix. This is reasonable for isolating adaptation decisions, but cannot establish end-to-end in-flight responsiveness.
  2. VIF reference labels need stronger validation. Appendix E uses GPT-5.4 with complete source timelines and speaker transcripts to annotate CONTINUE, QUEUE, and REVISE. A GPT-5.4 evaluator then maps free-form outputs to those actions. Table 14 checks a sample of the evaluator's labels, not independent agreement on the reference labels or event-onset times. Whether new information should interrupt or follow an utterance can admit multiple reasonable judgments. This matters especially for QUEUE, where RTVA reports its largest VIF improvement.
  3. The thought-commitment ablation uses a weak alternative. In Table 2, 'w/o Gate' always applies delayed thoughts even when intervening events are available. This shows why unconditional commitment fails, not that the Qwen2.5-VL-7B validity gate is superior to simple timeouts or event-based invalidation. RTVA also adds this model and repeated foreground polling to the matched foreground/background setup. Model-call rates, token costs, and actual interruption latency are needed to assess the benefit relative to the system overhead.
  4. Evidence for AutoHarness is mixed. LLM-proposed architecture changes followed by behavioral tuning resemble existing agent-design search methods such as AgentSquare. Figure 2 compares Two-Stage with Joint search only on LiveGaming and appears to show one search run, without variability across seeds. The discovered policies still trail human designs on LiveGaming and GUIDE-Bench. Promoted changes such as event routing and deduplication are useful, but also straightforward task-specific heuristics. It is unclear whether AutoHarness reliably discovers better temporal architectures beyond such choices.
Questions for authors
  1. What is the wall-clock delay from a triggering event to the first audible QUEUE or REVISE update, including generation and playback? How often would a VIF-successful action miss the three-second deadline in real time?
  2. How were VIF reference actions and onset times independently validated? Please report human agreement and confusion between CONTINUE, QUEUE, and REVISE, separately from checking the automatic output evaluator.
  3. How does the commitment gate compare with a simple age cutoff or event-invalidation rule at matched inference cost? Please report model calls, tokens, and end-to-end interruption latency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.