acceptodds
Withdrawn from ICLR 2027

Learning via Self-Consistency for Diffusion-based Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video generation models have demonstrated emerging zero-shot capabilities for visual reasoning, perception, and other vision tasks. However, diffusion-based video generation is inherently stochastic, while many downstream vision tasks are deterministic. Motivated by the effectiveness of self-consistency in chain-of-thought reasoning for large language models, we investigate whether self-consistency can similarly improve diffusion-based video reasoning. We first introduce a training-free test-time scaling method that samples multiple video generations and aggregates their predictions through self-consistency. Specifically, we aggregate extracted paths, locations, or masks from multiple rollouts into a consensus prediction. To reduce the inference overhead of multi-rollout generation, we read out predictions early in the denoising trajectory, which preserves consensus quality while reducing denoising steps by more than half. We further propose Rejection Fine-Tuning (RFT) to distill consensus predictions into the video generation model. The resulting model internalizes the benefit of multi-sample consensus and requires only a single generation at inference time, while substantially outperforming the original model. Experiments on three tasks, including maze solving, visual search, and referring segmentation, show that both our self-consistency inference and consensus distillation dramatically improve video-based perception and reasoning, without requiring ground-truth videos or task-specific verification. For visual search, self-consistency raises task accuracy from **48.4%** for a single generation to **99.0%**. The distilled model retains much of the consensus benefit with a single rollout. For maze solving, consensus-based training improves single-generation strict success rate from **72.0%** to **84.0%** with the same inference latency.

Primary area:
generative models
Keywords:
Video Generation, Video Reasoning, Diffusion Model, Self-Consistency, Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4KMQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper transfers chain-of-thought reasoning from LLMs to diffusion-based video reasoning. An extractor turns each generated video into a structured prediction, a consensus is then formed over these predictions, and finally a LoRA student is trained to predict the consensus. All three tasks improve on MiniMax-H3.

Critical strengths

The idea is interesting, and the paper reports the most appropriate diagnostic for confirmation bias.

Critical weaknesses

Weakness 1

One of my major concerns is that the main result on visual search is an artefact of the abstention mechanism. Could the authors explain this in more depth? Please also add a single-rollout abstention baseline, and report accuracy separately for the target-present and target-absent cases.

Weakness 2

Another major concern is that the main segmentation result overstates the contribution of consensus by roughly a factor of two. Is this reasonable? Weakness 3

Scoped to the evidence from two mazes, how much generality across backbones does this paper claim?

Weakness 4

If a second round of consensus RFT is run on the student's own generations, does performance keep improving, or does confirmation bias take over?

Questions for authors

What do a trivial always-abstain predictor and a single-rollout saliency-threshold baseline score on the 100-array search cohort?

Reviewer_ef7nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the application of self-consistency to diffusion-based video reasoning. The authors propose a training-free test-time scaling strategy that aggregates structured predictions from multiple stochastic video generations, including trajectories, spatial locations, and segmentation masks. To reduce the computational overhead associated with multiple rollouts, the framework introduces early-denoising readout, which extracts predictions from intermediate denoising states without completing the entire generation process.

Critical strengths
  • A simple yet effective framework for improving diffusion-based video reasoning. The paper extends self-consistency from language-model reasoning to structured predictions produced by stochastic video generators, providing a unified framework for improving both inference-time predictions and post-training performance. By aggregating task-relevant outputs rather than directly comparing generated videos, the proposed framework accommodates different prediction structures, including paths, points, and segmentation masks. Moreover, the ability to leverage agreement across independently generated samples without requiring ground-truth annotations or external verification provides a practically appealing direction for self-supervised improvement of video generation models.
  • Strong empirical improvements supported by meaningful experimental analyses. The proposed framework demonstrates substantial improvements across three distinct tasks. In particular, self-consistency significantly improves visual search accuracy and maze-solving performance, while consensus-based RFT effectively transfers part of the multi-sample benefit into a single-generation model. The early-denoising readout further reduces the computational overhead associated with multi-rollout inference while largely preserving prediction quality. Additionally, comparisons with random-rollout RFT and verifier-based supervised RFT provide useful evidence that consensus-based sample selection contributes meaningfully to the observed improvements.
Critical weaknesses
  • Limited methodological novelty and insufficient differentiation from established self-consistency techniques. Although the proposed framework achieves encouraging empirical improvements, its central methodology largely adapts established self-consistency and rejection fine-tuning strategies from language-model reasoning to diffusion-based video generation. Specifically, the proposed aggregation mechanisms primarily rely on conventional operations, including majority voting for trajectories, spatial clustering for point predictions, and pixel-wise voting for segmentation masks. Similarly, the consensus-based RFT procedure follows the established paradigm of selecting high-agreement pseudo-labels for subsequent supervised adaptation. While extending these ideas to structured outputs of video generators is practically meaningful, the paper does not sufficiently articulate a fundamentally new methodological principle that is specific to diffusion-based video reasoning. The primary contribution therefore appears to be an effective adaptation and empirical validation of existing techniques rather than a substantial algorithmic advance.
  • Insufficient analysis of consensus reliability and systematic error reinforcement. A central assumption underlying the proposed framework is that agreement across multiple stochastic generations provides a reliable indicator of prediction correctness. However, consensus does not necessarily imply correctness, particularly when independently sampled generations share systematic biases inherited from the same pretrained model. In such cases, multiple rollouts may consistently produce the same incorrect prediction, resulting in a high-confidence but erroneous consensus. When these predictions are subsequently used as pseudo-labels for rejection fine-tuning, the model may reinforce its existing biases rather than acquire improved reasoning capabilities. Although the paper provides empirical evidence that stronger agreement generally improves pseudo-label quality, the selected predictions are not uniformly correct. For example, Appendix B.5 reports that consensus-selected maze trajectories achieve validity rates of 83.3%, 72.2%, and 86.4% for 4×4, 5×5, and 6×6 mazes, respectively. These results demonstrate that agreement is an informative but imperfect proxy for correctness. Furthermore, the paper acknowledges difficulties in extending the approach to more complex reasoning settings, where individual predictions become increasingly diverse. However, the relationship between consensus strength, systematic prediction errors, and downstream self-training performance is not sufficiently characterized.
Questions for authors
  • Could the authors further clarify the methodological contributions that distinguish the proposed framework from existing self-consistency and rejection fine-tuning approaches? In particular, what properties of diffusion-based video generation necessitate fundamentally different treatment beyond task-specific aggregation of structured predictions?
  • How does consensus-based training behave when multiple generations consistently produce the same incorrect prediction? Could the authors provide a more systematic analysis of the relationship between consensus confidence and correctness, particularly on challenging reasoning instances? It would also be useful to investigate whether high-confidence incorrect pseudo-labels lead to systematic error reinforcement during post-training.
Reviewer_KSRHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates self-consistency for diffusion-based video reasoning on maze solving, visual search, and referring expression segmentation. It aggregates predictions extracted from multiple stochastic generations using modal-path selection, spatial grouping of marker locations, and pixel-wise mask voting. To reduce inference cost, it reads predictions from estimated clean latents before denoising finishes. It further uses consensus-selected or rendered targets for rejection fine-tuning, improving single-generation performance without ground-truth task labels in the core target-construction procedure.

Critical strengths

A coherent inference-to-training framework. The paper connects structured-output aggregation, early readout, and consensus-based post-training. Transferring multi-sample improvements into a single-generation student is practically appealing.

Meaningful controlled experiments. The maze comparison controls the layouts, trajectory count, and training recipe between Random RFT and Consensus RFT, with strict validity improving from 72.7% to 84.0%. The segmentation experiments include alternative target-selection methods and an area-controlled baseline, providing evidence beyond simply training longer or shrinking masks.

Critical weaknesses

The novelty claim regarding video aggregation is too broad. Demystifying Video Reasoning already introduces multi-seed latent-space ensembling, while Thinking with Video demonstrates majority voting over repeated video generations. Although the proposed output-space aggregation differs from latent fusion, the general idea of improving video reasoning through aggregation is established. The authors should narrow their “first” claim and clearly identify the additional contribution of task-specific aggregation and consensus-based post-training.

** Handling multiple hypotheses during early readout is unclear.** Early diffusion predictions may contain competing paths, locations, or regions that require subsequent denoising to resolve. (Demystifying Video Reasoning; Video Models Reason Early). The paper applies task extractors and consensus rules directly to these predictions without explaining how such ambiguity is handled. Cross-rollout agreement cannot necessarily resolve hypotheses shared across samples. Pixel voting may retain multiple incompatible regions, while insufficient location agreement is interpreted as target absence.

Limited evidence for generalization and robustness. Most results use one backbone and relatively small cohorts. Maze post-training is evaluated only on 4*4 layouts, while the Wan2.2 consensus experiment contains only two mazes. Search improvements primarily arise from better target-absence decisions, leaving broader reasoning improvements less established. The paper should provide uncertainty estimates over independent inputs, stronger cross-model or difficulty-transfer evaluations, and clarify how consensus thresholds were selected independently of the reported evaluation sets.

Comparisons require clearer attribution and matched evaluation. The 78.4% search result includes counterfactual augmentation and additional training; basic Consensus RFT achieves 69.4%. The independent confirmation results are reported for basic Consensus RFT, but not for the augmented model. An equal-update continuation control would help isolate the contribution of counterfactual data. Likewise, the segmentation consensus score of 0.489 and student score of 0.513 come from different evaluation cohorts. Their juxtaposition in Figure 1 should not be interpreted as demonstrating that the student outperforms its consensus teacher. A unified comparison on the same held-out inputs is needed.

Questions for authors
  1. How are multiple competing hypotheses within one early prediction extracted and resolved? Can the authors evaluate early readout specifically in such cases?
  2. What general principle determines the aggregation rule and threshold for a new task? Can the authors demonstrate transfer of their aggregation strategy beyond the three evaluated tasks with minimal additional design?
  3. On a common held-out set, how do base single-generation, base consensus, and student single-generation predictions compare? Does the counterfactual-augmented search model retain its improvement on the independent confirmation set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.