- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of keeping visual agents aligned with changing environments during interaction. It introduces a composable framework for response initiation, in-flight adaptation, and delayed-thought commitment. Besides that, authors also introduce AutoHarness for task-specific optimization and construct VIF for evaluating the performance.
Experiments show improvements across three domains, and ablations support the proposed components. The work offers a useful systems perspective and several meaningful experimental controls. However, the system design lacks sufficient innovation and reads more like a patchwork of disparate components. For instance, I fail to see the necessity of integrating AutoHarness. Perhaps the authors should reflect on how to better tell the story of fast adaptation to changing environments.
Critical strengths
-
The paper addresses how visual agents can remain aligned with changing environments throughout an ongoing interaction. RTVA provides a concrete design that coordinates response initiation, in-flight adaptation, and delayed-thought commitment.
-
The paper addresses how visual agents can remain aligned with changing environments throughout an ongoing interaction, providing a concrete design that coordinates response initiation, in-flight adaptation, and delayed-thought commitment.
Critical weaknesses
-
The paper lacks validation of token probability biases and calibration when System 1 decides whether to speak or remain silent.
-
The paper does not specify the duration of the test videos and lacks long-duration evaluations of inference lag and error accumulation.
-
The presentation needs improvement. AutoHarness is tangential to the main topic and offers little novelty, yet occupies considerable space, while the key equations and evaluation metrics are insufficiently explained.
-
The comparisons primarily involve models without task-specific designs and general-purpose agent frameworks, lacking fair comparisons with closely related methods.
Questions for authors
-
How does the model used in System 1 differ from a dedicated decision model such as Jev? If System 1 relies on next-token probabilities to choose between “speak” and “silent,” have you evaluated how token preferences affects these decisions?
-
How long are the video sequences in the evaluation, and how frequently do relevant environmental changes occur? How does the system handle situations where inference repeatedly falls behind these changes? Have you tested whether persistent lag leads to error accumulation during prolonged operation?
-
Beyond native real-time models, is there any other existing approaches address state information becoming outdated during inference? If so, how does RTVA's performance compare with these methods under comparable experimental conditions?