acceptodds
Under review as a conference paper at ICLR 2027

FrameTrust: Outcome Agreement for Reliable Video Imagination

Abstract

As video models evolve into implicit world models, autonomous agents can "imagine before acting" - yet visual plausibility does not guarantee predictive reliability, risking irreversible errors. To bridge this gap, we introduce FrameTrust, a training-free visual self-consistency framework that isolates true outcome disagreement from mere path diversity by mapping diverse rollouts to discrete terminal states via task readers. For stochastic environments, a simulator-relative entropy contrast effectively disentangles genuine world randomness from model hallucinations. To mitigate extreme video sampling costs, an intermediate diffusion readout screens divergent futures before all generation steps are spent. These signals drive an intent gate that dynamically routes the agent toward execution, replanning, or a conservative fallback. Across spatial, physical, memory, and interface environments, FrameTrust significantly improves error ranking and reduces denoising work through early screening. Crucially, our gate reduces irreversible GUI execution errors by 59-64% relative to blind imagination without compromising task success, establishing a rigorous foundation for safe, imagination-driven autonomous action.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video generation, outcome agreement, visual self-consistency, uncertainty estimation, multimodal agents, adaptive inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_82KgOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes FrameTrust, which uses the disagreement among several imagined futures from a video generation model to decide whether an agent should trust it before acting. This is aimed to avoid acting on unreliable predictions that may lead to irreversible errors. Instead of comparing pixels, a task reader maps video to a discrete outcome, so only disagreement about the result counts. For stochastic tasks, a simulator-relative entropy contrast separates real task randomness from model error. To reduce time cost, the method reads disagreement early in the denoising process and stops sampling once the vote is stable. An intent gate then decides whether to execute, replan, or take a conservative action. The authors evaluate the method on mazes, physics, cup tracking, GUI transitions, and open-ended video generation. Outcome-level disagreement ranks errors better than the comparison signals. In an AndroidWorld GUI agent, the gate reduces irreversible errors by 59% to 64% relative to acting on a single imagined future, without lowering task success.

Critical strengths
  1. The question is timely and critical. Video models are starting to serve as world models for agents, and a plausible but wrong imagined future can cause severe irreversible errors. The introduction lays out the challenges clearly and persuasively.

  2. Using semantic outcomes instead of pixel-level differences is a good design choice. Different paths that reach the same result no longer count as disagreement, which makes the signal much more meaningful than pixel variance.

  3. The design intuition has a solid mathematical foundation. For instance, the simulator-relative entropy contrast gives a principled way to distinguish real randomness in the task from uncertainty of the model, and it clearly helps on the stochastic physics scenes.

  4. The experiments are well designed and carefully analyzed. They cover both low-level tasks such as Maze and Physics and a high-level GUI agent task, which matches the motivation in the introduction about preventing irreversible errors. The results are promising across generators, and the baselines are well chosen. The early readout also saves a large share of denoising compute with almost no loss in acc.

Critical weaknesses
  1. The simulator-relative contrast needs a simulator to estimate H_sim. For GUI, H_sim is simply set to zero (Table 1). If not misunderstand, real apps can also be random? It is not clear how the contrast would work in the deployment settings that motivate the paper, where no simulator exists.

  2. The concentration bound in Eq. (2) assumes independent seeds and a fixed K. The trajectory ensembling variant produces correlated votes (Appendix G.5), and Algorithm 1 stops adaptively. So the theoretical bound therefore works better as intuition than as a guarantee for the procedure that is actually run.

Questions for authors
  1. Does the bound in Eq. (2) require IID seeds? How does it apply to correlated votes in the ensembling variant and to the adaptive stopping in Algorithm 1?

  2. How would the simulator-relative contrast be used when no simulator is available, as in real GUI or embodied deployment? Is H_sim = 0 justified for AndroidWorld apps?

  3. Appendix H.2 says the conservative branch can end in an alternative plan, a user-assisted decision, or abstention. And the human assistance is counted. Can you report how often each branch is taken in Table 5, including the number of human assists?

  4. Do the three gate branches cover all cases? For example, which branch is taken when the trusted outcome neither matches nor clearly contradicts the intent?

Reviewer_anu2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the reliability of "imagine-before-act" for video world models in agentic settings. FrameTrust is a training-free visual self-consistency framework with four components: (1) task readers map diverse video rollouts onto discrete terminal states, isolating true outcome disagreement from path diversity; (2) a simulator-relative entropy contrast (KL-based, H_sim > 0.1 bits) separates genuine world randomness from model hallucination in stochastic environments; (3) an intermediate diffusion readout at s0 = 0.3 screens divergent futures before full generation, cutting denoising work by 67.5% with parity-level screening accuracy (71.0% vs 71.4%); (4) an intent gate routes the agent among execution, replanning, and conservative fallback. On AUROC-based error ranking across Wan2.2 / CogVideoX1.5-5B / Veo3.1, and on AndroidWorld GUI deployment (60 tasks), the gate reduces irreversible execution errors by 59-64% relative to blind imagination while improving success (+1.6-8.3 pts over blind K=1), and outcome abstraction lifts deterministic-physics AUROC from 0.61 to 0.86.

Critical strengths
  1. The paper cleanly separates true outcome disagreement from path diversity by mapping diverse rollouts to discrete terminal states via task readers - a well-motivated fix to a real failure mode of naive visual self-consistency - and the simulator-relative entropy contrast fills a genuine methodological gap for stochastic environments where vanilla self-consistency is ill-posed.
  2. The AndroidWorld study provides strong end-to-end validation: the intent gate reduces irreversible GUI execution errors by 59-64% relative to blind imagination without compromising task success (68.3/68.9 vs 66.7 blind), and the intermediate diffusion readout reports 67.5% denoising savings with essentially no loss in screening accuracy.
Critical weaknesses
  1. Task readers remain the least-specified component: how they are obtained per environment, the required human/domain engineering across spatial/physical/memory/interface settings, and robustness to reader errors are not fully addressed; reader error propagates into all downstream signals.
  2. The headline 59-64% error reduction rests on a single benchmark (AndroidWorld, 60 tasks) with per-configuration means; error bars, per-task variance, and statistical testing are not reported in the main text, and the fixed thresholds (s0 = 0.3, H_sim > 0.1 bits) lack sensitivity analysis.
Questions for authors
  1. How are task readers instantiated for each of the four environment families, and what is the human effort per task? Can readers be learned or auto-generated?
  2. How sensitive are the results to the readout threshold s0 = 0.3 and to the entropy threshold H_sim = 0.1 bits? Is there a phase transition in screening quality?
  3. Were the 59-64% error reductions statistically tested across the 60 AndroidWorld tasks (e.g., paired bootstrap or McNemar)?
Reviewer_f69gOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies using a video generation model as future predictor for agentic decision making: an action is executed only if multiple rollouts indicate a stable consensus. The framework is empirically verified to be effective.

Critical strengths

Reading disagreement before completion is a great approach, I appreciate the thinking behind it.

Critical weaknesses
  1. The most significant concern is the practicality of the problem studied. Video generation is very expensive, and multiple rollouts, proposed in the paper, makes it even worse. In the case of phone use, does it justify to use a video generation model to perform prediction, given the elements are very structured.

  2. The presentation requires significant improvements. As the authors have claimed using AI to polish the writing, I find it very difficult to follow the presentation, because the overly complicated wording, unnecessary inclusion of technical concepts (e.g., "simulator-relative entropy contrast" is essentially subtracting environment variability from model prediction variability, and "sequential sampling procedure" is essentially stopping when concensus has been reached), and mathematical equations that do not add clarity. Moreover, some key elements in Fig 2 seem AI-generated without re-drawing, judging by their blurry, and noisy appearance. However, this AI used is not included in the AI Use Statement.

  3. The teaser example on deleting a photo from the phone gallery is quite confusing, the context and intent are not fully elaborated. Line 53 in the introduction says the model could imagine a future that the phone pops up a confirmation dialog asking the user to confirm the deleting action and it is claimed that "this is factually incorrect". It is claimed that wrong prediction leads to catastrophic errors, but in this case, what is the catastraphic action? I started to guess that probably the correct action is not to delete the photo at all, but in Fig 2, all rollouts actually point at deleting the photo. Another guess is that the prediction shows this is a dialogue but in fact there isn't, causing the photo to be deleted immediately. But isn't deleting the photo the agent's intention anyways?

  4. It is unclear how the readers perform in actual deployment. In Tab 1, different tasks employ different type of readers. VLM-based reader may itself introduce errors. Moreover, as for the reading disagreement before full diffusion, I wonder how does the programmatic reader evaluate a noisy prediction?

Questions for authors

Please see the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.