- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies action forecasting from Ego4D narration histories. It constructs PDM-Eval by pairing the recorded next event with a high-scoring generated continuation that survives semantic filtering. An RSA interpretation connects this selection procedure to a listener prior and KL-regularized preference learning. PDM-DPO++ adds generated explanations to the preferred and rejected continuations and trains only the language layers of a VLM with DPO. Evaluation measures the difficulty of the constructed comparisons and transfer to TemporalBench and OpenEQA. The reported full-TemporalBench improvements are 1.62% for LLaMA-3.2-11B and 0.86% for Qwen2.5-VL-32B.
Critical strengths
The paper explores a practical route to adapting VLMs through text-only supervision. Updating only language adapters reduces the reported fine-tuning memory and time, and evaluation uses direct answers without generating explanations. The positive results on two backbones and several frame counts provide preliminary evidence that the training signal can transfer beyond the narration data.
Critical weaknesses
-
The conceptual account adds limited insight beyond hard-negative mining and preference learning. The RSA derivation largely re-expresses these operations through a chosen listener model and the standard KL-regularized optimum. Its connection to explanation-based adaptation is weak: the reward ignores the explanation, and Appendix B.4 shows that the optimal explanation distribution conditioned on the outcome equals the reference distribution. In practice, PDM-DPO++ uses a separate model and explicitly asks it to introduce causal errors into rejected explanations. This introduces a source of synthetic reasoning supervision whose benefit is only loosely connected to the derivation. There is also a gap between an unrealized forecast and a physical reasoning error. After someone opens a fridge, taking juice may be a reasonable alternative to the recorded action of taking milk. Observing milk verifies that particular outcome. The rejected-rationale prompt then requires a causal fault to be inserted. This weakens the interpretation as correcting the model's own physical prediction errors.
-
The gains on full TemporalBench are small and consistently lower for the stronger backbone. With one frame, PDM-DPO++ improves LLaMA-3.2-11B by 1.62 points and Qwen2.5-VL-32B by 0.86 points. With two and four frames, Qwen's gains are only 0.09 and 0.55 points, compared with 1.51 and 1.32 points for LLaMA. This raises a scaling concern: the supervision may mainly compensate for gaps in weaker models and become less useful as pretrained capabilities improve.
Questions for authors
-
What does the RSA analysis predict about the benefit of training on explanations beyond ordinary hard-negative DPO? How does the identity in Appendix B.4 connect to the practical use of a separate explanation generator and deliberately flawed rejected rationales?
-
Does selecting the highest-scoring negative improve over randomly selecting from the same generated pool, with filtering, explanation generation, and DPO training held fixed? This would isolate the selection rule from candidate-generation quality.
-
How much of the TemporalBench gain depends on the visual evidence? A control with missing or mismatched frames, or a targeted reversal test such as opening versus closing a door, would help distinguish stronger action priors from improved use of temporal observations.
-
Please recompute confidence intervals for the improvements using paired predictions at the video-group level. Do the gains over zero-shot and answer-only PDM-DPO remain reliable after correcting the sample unit? Any variation across training seeds would also help assess the small effect sizes.
-
Why are the full-TemporalBench gains consistently smaller on Qwen2.5-VL-32B across the frame sweep? What evidence supports the usefulness of this supervision as backbone capability improves? A within-family comparison or an analysis of which errors are corrected would help clarify whether stronger models already capture much of the information supplied by these training examples.
Thank you for your helpful questions. They have clarified our framing and validation and we will make updates in the camera ready.
Q1a. What does the RSA analysis predict about the benefit of training on explanations beyond ordinary hard-negative DPO?
To answer your question specifically, the analysis predicts that explanations add evidence of the form $\log \frac{\pi_0(Z\mid Y^+,H)}{\pi_0(Z\mid Y^-,H)}$, which is non-negative on average when preferred explanations are sampled conditioned on the verified outcome (Eq. 11). More intuitively, it suggests that we should condition reasoning evidence on the verified outcome. Hard-negative DPO on outcomes alone has no such prescription. This motivates our outcome-conditioned explanation sampling, and consistent with the prediction, PDM-DPO++ outperforms answer-only PDM-DPO in all six comparisons (Sec. 4.2.3; intervals in M5). The RSA analysis, and its connection to biological motor control, also prescribes a number of other things (connection to the novel forecasting setting which allows the OOD transfer, the selection rule, etc.) We provide more discussion of these contributions in the response to all reviewers.
Q1b. How does the identity in Appendix B.4 connect to the practical use of a separate explanation generator and deliberately flawed rejected rationales?
Q2. Does selecting the highest-scoring negative improve over randomly selecting from the same generated pool?
M4 of the response to all reviewers explains what Fig. 3 does and does not isolate, and why this training-time comparison is deferred. TLDR; PDM does create more difficult negatives than baseline strategies, as measured by standard uncertainty measures (Brier score, a proper scoring loss).
Q3. How much of the TemporalBench gain depends on the visual evidence (missing or mismatched frames; a targeted reversal test)?
We ran missing- and mismatched-frame controls. Kindly refer to M1 of the response to all reviewers. TLDR; Our method is also using visual evidence.
Q4. Please recompute confidence intervals for the improvements using paired predictions at the video-group level. Do the gains over zero-shot and answer-only PDM-DPO remain reliable after correcting the sample unit? Any variation across training seeds would also help.
- The paired bootstrap in M2 (response to all reviewers) resamples the 2,179 video groups. The gain over zero-shot and most other baselines is reliable (+1.62
+0.73,+2.52), while on the full benchmark we cannot separate PDM-DPO++ from answer-only PDM-DPO (+0.12
−0.78,+1.06). Variation across five training seeds is also in M2 of the response to all reviewers.
Q5. Why are the full-TemporalBench gains consistently smaller on Qwen2.5-VL-32B across the frame sweep? What evidence supports the usefulness of this supervision as backbone capability improves?