- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces VeriAct, a framework within an agent loop that explicitly and consequentially integrates the verification of actions and updates its working memory when needed. The motivation centers on the capability of VLMs to handle interactive visual reasoning cases, specifically in terms of performance and efficiency. They claim that agents are not verifying their observations against their expectations, and raise the question of whether this is the root cause for missing changes, repeating actions, or continuing planning from stale thoughts. They evaluate it on a subset of the ARC-AGI-3 benchmark against a baseline without verification.
Critical strengths
The idea of making the model think about its expectations before execution is a compelling tier of the verification modeling, alongside the macro-verification of objective preservation. Additionally, making the verification operational achieves a notable improvement in action efficiency.
Critical weaknesses
- Lack of explanation: The paper assumes preliminary familiarity with ARC's tasks, game mechanics, no proper introduction of the task setup and evaluation metrics.
- Limited baselines: The framework is only evaluated against a "vista-shaped" agent, which lacks a verification phase. What about comparing it against other baselines that include a verification phase (e.g., reflection, retry, self-critique) or non-vision-based agents (text-only)?
- Limited experimental setup (Is the framework robust?):
- Model variety: Why were no other VLMs evaluated besides Qwen3.8? Models differ significantly in scale and in their text/vision encoders.
- Task selection: The evaluation uses only 13 games. Why wasn't the framework evaluated on the full public game set? How do these selected games differ from one another, and what different challenges do they represent?
- Generalizability: Are there other types of games or interactive visual reasoning setups where this framework could be tested?
Questions for authors
- Level progression: How does verification within the framework change (e.g., does it become more challenging) as the levels progress?
- Uncertainty computation: Can the uncertainty somehow be computed internally from the VLM agent (for example, by using its token probabilities)? Furthermore, is the uncertainty report calibrated to this?