acceptodds
Withdrawn from ICLR 2027

DAL-CoT: Decision-aligned Latent Reasoning with MLLMs for Video Scene Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) have been applied to video scene segmentation, which requires a model to make multiple boundary decisions over a series of continuous video shots. This calls for the ability to aggregate long-range cues from noisy multimodal inputs, where the model must distinguish genuine boundary signals from numerous conflicting distractors, which poses a unique challenge to Chain-of-thought (CoT). Conventional explicit CoT translates continuous and uncertain visual evidence into an autoregressive sequence of definitive statements, where an incorrect commitment to one boundary decision may directly lead to an overconfident yet incorrect answer. To address this, we propose DAL-CoT, a decision-aligned latent CoT method that encodes explicit CoT reasoning into the model's hidden states. DAL-CoT assigns each candidate shot an observation latent slot for fine-grained reasoning, preventing premature commitment to boundary decisions. During training, an auxiliary decoder reconstructs the explicit reasoning texts from each latent slot, while at inference it is removed. Extensive experiments demonstrate that DAL-CoT significantly outperforms state-of-the-art approaches while substantially reducing the decoding cost of intermediate reasoning compared with explicit CoT.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
decision-aligned, latent chain-of-thought, multimodal large language models, video scene segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XTnZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DAL-CoT addresses video scene segmentation by assigning a latent slot to each requested shot-boundary decision. During training, an auxiliary decoder reconstructs teacher-generated shot-pair observations from the corresponding slot and shared multimodal inputs. The decoder is removed during normal inference. The method is evaluated on MovieNet-318 and zero-shot BBC Planet Earth with two Qwen backbones.

Critical strengths

The accuracy–latency comparison is practically relevant and includes a direct-answer reference. With the same Qwen3-VL backbone, DAL-CoT gains 1.3 AP at approximately 1.04 times answer-only latency, whereas observation-only CoT takes 24.58 seconds versus 2.86 seconds.

Critical weaknesses
  1. Reconstructed text does not establish that the latent states contain the claimed observational reasoning. In Eq. (6), the auxiliary decoder also receives shared multimodal metadata from the full context. It may therefore generate observations largely from those inputs and the slot index. Unique decoded strings and qualitative examples do not rule out this shortcut, nor do attention maps show that slot content is causally necessary. This is consequential because the paper's central contribution is reasoning internalized in decision-aligned latent states, rather than simply auxiliary supervision with indexed prompts.

  2. Table 2 shows that unaligned latent supervision already reaches 66.3 F1/72.7 AP, leaving only 0.5 F1/0.2 AP attributable to decision alignment, without reported run variability. The proposed explanation for explicit-CoT failure is overconfident copying, but the manuscript does not quantify calibration, wrong-verdict copying, or error propagation across decisions. The large commitment ablation establishes degradation under that interface, not its precise cause. Consequently, the evidence is stronger for the overall auxiliary-training recipe than for the claimed attention-collapse and premature-commitment mechanisms.

Questions for authors
  1. What evidence shows that the decoder needs the learned slot content rather than shared multimodal inputs and shot indices to reconstruct observations?

  2. How robust is the decision-alignment gain in Table 2, and what quantitative evidence supports overconfident copying as the cause of the explicit-commitment degradation?

Reviewer_patCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles video scene segmentation using Multimodal Large Language Models (MLLMs). The authors argue that conventional explicit Chain-of-Thought (CoT) suffers from "premature boundary commitment" (where an incorrect verbalized boundary verdict causes overconfident failure in autoregressive decoding), while standard latent CoT methods suffer from "attention collapse" (where attention concentrates disproportionately on the first few latent tokens).To resolve these issues, the authors propose DAL-CoT (Decision-Aligned Latent CoT). The core framework introduces K discrete latent slots corresponding to the K$ candidate shot transitions within a fixed 10-shot focus sequence (selected from a 20-shot multimodal context). During training, an auxiliary decoder (Qwen3-VL-8B) is introduced to reconstruct explicit multi-sentence observational reasoning generated offline by Qwen3-VL-235B without explicit boundary verdicts. The optimization follows a three-stage curriculum (Stage 0: backbone adaptation; Stage 1: decoder warm-up on frozen backbone; Stage 2: joint training). At inference time, the auxiliary decoder is discarded, leaving only the latent slot prefill and direct answer token generation. Experiments are conducted on MovieNet-318 and zero-shot on BBC Planet Earth.

Critical strengths

Practical Inference Efficiency Formulation: The paper appropriately diagnoses the prohibitive inference latency of generating verbose explicit reasoning for dense video shot transitions. Discarding the auxiliary decoder at test time effectively preserves the generation throughput of an answer-only model while maintaining prefill-level latent conditioning. Comprehensive Quality Control for Teacher Rationales: The data collection pipeline (Appendix D & E) thoughtfully designs programmatic heuristics (pointer coverage, temporal directionality, absence of explicit leakage labels) to sanitize observational CoT synthesized by the 235B teacher model.

Critical weaknesses

Critical Weaknesses Marginal Empirical Improvements Relative to Massive Training Complexity:Trivial Metric Gains: In Table 1, under the matched Qwen3-VL-8B backbone, DAL-CoT outperforms the Scene-VLM baseline by only +1.5 F1 and +1.3 AP on MovieNet-318, and +1.2 AP zero-shot on BBC Planet Earth. Under Qwen2.5-VL-7B, the gain is as small as +0.6 F1 and +1.5 AP. In dense temporal boundary detection, such small margins often fall within the variance of random seed initialization and data shuffling, yet no multiple trials, standard deviations, or statistical significance tests (p−values) are reported. Disproportionate Computational & Parameter Footprint: Achieving this modest +1.3 AP gain requires generating 29,038 multi-line CoT descriptions offline using a massive Qwen3-VL-235B model, followed by a heavy three-stage training schedule. In Stage 2, two full 8B models (the backbone and the auxiliary decoder, totaling 16B parameters) must be co-optimized with DeepSpeed ZeRO-3 across multiple epochs. The carbon and compute overhead heavily outweighs the incremental benchmark improvement. Extreme Fragility to Decoder Capacity: Table 5 indicates that when the auxiliary decoder is replaced by a 4B model (Qwen3-VL-4B), the final performance drops to 71.2 AP, which is strictly worse than the baseline Answer-only model without any latent reasoning (71.6 AP in Table 2). This demonstrates that the proposed framework does not possess intrinsic robustness; it collapses unless paired with an auxiliary supervision module of equal scale to the backbone. Flawed Auxiliary Decoder Formulation and Information Shortcut (Ui):In Equation 6, the input fed to the auxiliary decoder is defined as:Ui=[Hmeta(0); Hℐi(−1)]where Hmeta(0) denotes the raw input embeddings of multimodal metadata (frames, subtitles, cast IDs) across all N shots, and Hℐi(−1) is the final backbone representation of slot ii.Severe Confounding: Because the auxiliary decoder is directly provided with the uncompressed multimodal inputs Hmeta(0)H_{meta}^{(0)}, it does not have to rely on Hℐi(−1) to reconstruct fine-grained transition details. The decoder can simply perform cross-attention over Hmeta(0) while using slot i merely as an indexing query pointer.The paper provides no ablation where Hmeta(0) is excluded from Ui (i.e., conditioning the decoder strictly on Hℐi(−1)). Without this control, the central scientific claim that the latent tokens "encode and internalize the complex reasoning process" is fundamentally unsubstantiated. Overstated Core Motivation: "Attention Collapse" vs. Decision Alignment:The paper builds its conceptual thesis on addressing "attention collapse" via decision alignment (Figure 1(b) and Section 3.2). However, examining the ablation results in Table 2 reveals that:DAL-CoT w/o decision-aligned: 66.3 F1 / 72.7 APDAL-CoT (with decision alignment): 66.8 F1 / 72.9 APThe actual empirical gain provided by the central methodological contribution—decision alignment over joint latent tokens—is merely +0.2 AP and +0.5 F1. Staking the entire paper’s narrative and theoretical novelty on solving attention collapse when the quantitative contribution is virtually negligible is an unjustified overclaim.Furthermore, the authors attribute the drastic collapse of Observation-then-commit CoT (59.0 F1 / 50.9 AP) solely to "premature commitment". In reality, autoregressive generation of sequential verdicts across 10 shots inherently suffers from classic autoregressive exposure bias and label bias. A simple non-autoregressive multi-binary classification head or independent per-shot probing would trivially prevent premature verbal commitment without requiring complex multi-stage latent token decoding. Severe Architectural Inflexibility and Lack of Scalability:The method hardcodes a rigid 10-shot focus window (K=10) embedded within a 20-shot context, creating handcrafted text strings (slot(i) = latent || str(i)). Real-world cinematic videos possess highly variable temporal dynamics: scene boundaries occur over spans ranging from 2 shots to over 100 shots. A fixed K=10 sequence length cannot handle continuous streaming video, dynamically sized shot contexts, or boundary decisions near the beginning and ending extremes of videos without unnatural window slicing and edge artifacts. The authors conduct no experiments on varying K (e.g., K∈{5,20,30}) to demonstrate whether the learned representations generalize beyond this rigid 10-slot setup.

Questions for authors

Information Bottleneck in Decoder Conditioning: What is the reconstruction loss and downstream segmentation performance if Hmeta(0) is removed from Equation 6, forcing the auxiliary decoder to reconstruct the observational text solely from the latent hidden state Hℐi(−1)?

Does the latent slot actually retain sufficient visual-linguistic evidence on its own?Statistical Significance and Multiple Runs: Please provide the mean and standard deviation across at least 3 distinct random seeds for Table 1 and Table 2. Is the 0.2 AP difference between DAL-CoT (72.9 AP) and DAL-CoT w/o decision-aligned (72.7 AP) statistically significant under a paired t-test (p<0.05)?

Failure Mode of Smaller Auxiliary Decoders: Why does employing a 4B auxiliary decoder cause the final AP (71.2) to drop below the naive Answer-only baseline (71.6)? What theoretical or empirical factor prevents a smaller model from regularizing the latent slot effectively? Generalization Across Focus Window Lengths (K): How does DAL-CoT perform when the focus window size K deviates from 10 at test time? Does the model require retraining from scratch for each distinct window length, and how does performance scale as K increases to 30 or 50?

Reviewer_H3nJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DAL-CoT, a decision-aligned latent reasoning framework for video scene segmentation. Given multimodal shot sequences containing frames, subtitles, and character identities, the method inserts an indexed latent slot for each candidate scene-boundary decision. During training, an auxiliary decoder reconstructs teacher-generated, observation-only descriptions of the corresponding shot transitions, while the backbone predicts boundary labels.

Critical strengths
  1. Consistent gains with little inference overhead. DAL-CoT improves over matched-backbone Scene-VLM on MovieNet and zero-shot BBC evaluation using two backbones. With Qwen3-VL-8B, it gains 1.3 MovieNet AP while increasing latency from 2.75 to 2.86 seconds.
  2. Useful supervision design. Aligning observation targets with individual boundary decisions is well motivated. The comparison between observation-only and observation-then-commit CoT also provides useful evidence that intermediate output design affects segmentation performance.
Critical weaknesses
  1. Training fairness and gain reliability remain unclear. Section 4.3 describes a two-epoch budget, while DAL-CoT additionally uses Stage-0 adaptation. Matched-training controls are needed to isolate its contribution. Moreover, decision alignment adds only 0.2 AP over unaligned latent supervision, without variability estimates.
  2. The latent-reasoning mechanism is insufficiently validated. The auxiliary decoder receives the full multimodal context, so reconstruction may bypass information encoded in the slots. Unique decoded texts and attention visualizations do not establish that slot content drives boundary predictions. The premature-commitment explanation also lacks quantitative calibration or error-propagation evidence.
Questions for authors
  1. Do all variants receive the same backbone initialization and number of updates? What is the answer-only result under matched training?
  2. How do reconstruction and segmentation performance change when latent states are removed, replaced, or shuffled?
  3. Is the 0.2 AP gain from decision alignment consistent across seeds or statistically significant?
  4. Can you quantify overconfidence and the copying of incorrect intermediate verdicts in observation-then-commit CoT?
  5. What is the practical value of scene-boundary decisions, and how do these improvements benefit downstream applications?
  6. Is “gt: No” for Shot 1 in Figure 1(a) correct? It appears to be a scene boundary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.