acceptodds
Under review as a conference paper at ICLR 2027

PRAGMATIC DYNAMICS MODELS: LEARNING EMBODIED TEMPORAL REASONING FROM FORECASTING IN LANGUAGE

Abstract

Temporal reasoning in embodied settings requires models to predict what will happen next and adapt when those predictions conflict with physical reality. Human motor control implements this through internal forward models that predict possible consequences and compare them with sensory feedback to determine whether adaptation is necessary. To model this process computationally, we introduce pragmatic dynamics models (PDMs), using language as a simulation space for this propose-then-verify planning loop. A generative model proposes possible futures, and the realized physical trajectory verifies what occurred. We then use pragmatic inference—a probabilistic model of how language distinguishes between possible world states—to identify unrealized futures that the model has difficulty distinguishing from reality. This yields a new action forecasting benchmark (PDM-Eval). The same pragmatic account then connects the verification process to preference learning via Direct Preference Optimization (DPO), where both feedback from physical reality and regularization constraints determine how to update based on the pragmatically selected comparison. Our theory also suggests an extension, which uses chain-of-thought explanations of the competing dynamics to further inform the update process. Empirically, our benchmark PDM-Eval produces more difficult comparisons than baseline benchmark creation strategies across five language models. Our proposed DPO-based adaptation algorithm also improves generalization of visual-temporal reasoning to new domains, compared to baselines tested on two vision-language backbones. Gains persist without generating explanations at test time, and only require updates to the language layers of the network, substantially reducing compute costs for fine-tuning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Rational Speech Acts, Pragmatics, Temporal Reasoning, Video-Language Models, Vision-Language Models, Mutlimodal Models, Embodied Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_n8sbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies action forecasting from Ego4D narration histories. It constructs PDM-Eval by pairing the recorded next event with a high-scoring generated continuation that survives semantic filtering. An RSA interpretation connects this selection procedure to a listener prior and KL-regularized preference learning. PDM-DPO++ adds generated explanations to the preferred and rejected continuations and trains only the language layers of a VLM with DPO. Evaluation measures the difficulty of the constructed comparisons and transfer to TemporalBench and OpenEQA. The reported full-TemporalBench improvements are 1.62% for LLaMA-3.2-11B and 0.86% for Qwen2.5-VL-32B.

Critical strengths

The paper explores a practical route to adapting VLMs through text-only supervision. Updating only language adapters reduces the reported fine-tuning memory and time, and evaluation uses direct answers without generating explanations. The positive results on two backbones and several frame counts provide preliminary evidence that the training signal can transfer beyond the narration data.

Critical weaknesses
  1. The conceptual account adds limited insight beyond hard-negative mining and preference learning. The RSA derivation largely re-expresses these operations through a chosen listener model and the standard KL-regularized optimum. Its connection to explanation-based adaptation is weak: the reward ignores the explanation, and Appendix B.4 shows that the optimal explanation distribution conditioned on the outcome equals the reference distribution. In practice, PDM-DPO++ uses a separate model and explicitly asks it to introduce causal errors into rejected explanations. This introduces a source of synthetic reasoning supervision whose benefit is only loosely connected to the derivation. There is also a gap between an unrealized forecast and a physical reasoning error. After someone opens a fridge, taking juice may be a reasonable alternative to the recorded action of taking milk. Observing milk verifies that particular outcome. The rejected-rationale prompt then requires a causal fault to be inserted. This weakens the interpretation as correcting the model's own physical prediction errors.

  2. The gains on full TemporalBench are small and consistently lower for the stronger backbone. With one frame, PDM-DPO++ improves LLaMA-3.2-11B by 1.62 points and Qwen2.5-VL-32B by 0.86 points. With two and four frames, Qwen's gains are only 0.09 and 0.55 points, compared with 1.51 and 1.32 points for LLaMA. This raises a scaling concern: the supervision may mainly compensate for gaps in weaker models and become less useful as pretrained capabilities improve.

Questions for authors
  1. What does the RSA analysis predict about the benefit of training on explanations beyond ordinary hard-negative DPO? How does the identity in Appendix B.4 connect to the practical use of a separate explanation generator and deliberately flawed rejected rationales?

  2. Does selecting the highest-scoring negative improve over randomly selecting from the same generated pool, with filtering, explanation generation, and DPO training held fixed? This would isolate the selection rule from candidate-generation quality.

  3. How much of the TemporalBench gain depends on the visual evidence? A control with missing or mismatched frames, or a targeted reversal test such as opening versus closing a door, would help distinguish stronger action priors from improved use of temporal observations.

  4. Please recompute confidence intervals for the improvements using paired predictions at the video-group level. Do the gains over zero-shot and answer-only PDM-DPO remain reliable after correcting the sample unit? Any variation across training seeds would also help assess the small effect sizes.

  5. Why are the full-TemporalBench gains consistently smaller on Qwen2.5-VL-32B across the frame sweep? What evidence supports the usefulness of this supervision as backbone capability improves? A within-family comparison or an analysis of which errors are corrected would help clarify whether stronger models already capture much of the information supplied by these training examples.

1 comment

Thank you for your helpful questions. They have clarified our framing and validation and we will make updates in the camera ready.

Q1a. What does the RSA analysis predict about the benefit of training on explanations beyond ordinary hard-negative DPO?
To answer your question specifically, the analysis predicts that explanations add evidence of the form $\log \frac{\pi_0(Z\mid Y^+,H)}{\pi_0(Z\mid Y^-,H)}$, which is non-negative on average when preferred explanations are sampled conditioned on the verified outcome (Eq. 11). More intuitively, it suggests that we should condition reasoning evidence on the verified outcome. Hard-negative DPO on outcomes alone has no such prescription. This motivates our outcome-conditioned explanation sampling, and consistent with the prediction, PDM-DPO++ outperforms answer-only PDM-DPO in all six comparisons (Sec. 4.2.3; intervals in M5). The RSA analysis, and its connection to biological motor control, also prescribes a number of other things (connection to the novel forecasting setting which allows the OOD transfer, the selection rule, etc.) We provide more discussion of these contributions in the response to all reviewers.

Q1b. How does the identity in Appendix B.4 connect to the practical use of a separate explanation generator and deliberately flawed rejected rationales?

  • App. B.4 shows that the optimal explanation distribution given an outcome is the reference model's own conditional, $\pi_0(Z\mid Y,H)$, so explanations should be generated conditioned on a given outcome. In our implementation, the generator is always prompted with the outcome (the correct answer for the preferred response and the distractor for the rejected one) to approximate this.
  • In our main runs, a separate frozen text-only model (Qwen3-14B) generates the explanations, because LLaMA is not a reasoning model and self-generation fails often (See M6 in response to all reviewers). However, we also test self-generation on Qwen2.5-VL-32B, which is more consistent with our theory. TLDR; the method still works with self-generation.

Q2. Does selecting the highest-scoring negative improve over randomly selecting from the same generated pool?
M4 of the response to all reviewers explains what Fig. 3 does and does not isolate, and why this training-time comparison is deferred. TLDR; PDM does create more difficult negatives than baseline strategies, as measured by standard uncertainty measures (Brier score, a proper scoring loss).

Q3. How much of the TemporalBench gain depends on the visual evidence (missing or mismatched frames; a targeted reversal test)?
We ran missing- and mismatched-frame controls. Kindly refer to M1 of the response to all reviewers. TLDR; Our method is also using visual evidence.

Q4. Please recompute confidence intervals for the improvements using paired predictions at the video-group level. Do the gains over zero-shot and answer-only PDM-DPO remain reliable after correcting the sample unit? Any variation across training seeds would also help.

  • The paired bootstrap in M2 (response to all reviewers) resamples the 2,179 video groups. The gain over zero-shot and most other baselines is reliable (+1.62
+0.73,+2.52+0.73, +2.52

), while on the full benchmark we cannot separate PDM-DPO++ from answer-only PDM-DPO (+0.12

−0.78,+1.06−0.78, +1.06

). Variation across five training seeds is also in M2 of the response to all reviewers.

  • Please also note that MBA is already a video-level metric: A group counts as correct only if all its questions are, so dependence within a video is built in. We will make sure all Figures / results use group-level intervals / standard errors. We will also add the new results.

Q5. Why are the full-TemporalBench gains consistently smaller on Qwen2.5-VL-32B across the frame sweep? What evidence supports the usefulness of this supervision as backbone capability improves?

  • Smaller gains: Qwen2.5-VL-32B is a harder baseline, since it already has video training and far more parameters than LLaMA. Its zero-shot MBA (29.19 at 2 frames, 32.49 at 4) is comparable to Qwen2-VL-72B's (27.5, 31.2) in Cai et al. (2024).
  • Usefulness as backbones improves: Our results are primarily about efficiency. When compute is not available for full video tuning, language-only tuning can be competitive. Even as backbones improve, specialized, in-house, or smaller backbones will be needed under limited compute (organizational limits, edge deployment, latency).
  • Within-family comparison: Comparing several Qwen sizes would address this. It is infeasible now, because we are running the other experiments the reviewers requested, but we plan to include it in the camera-ready version.
  • Please also see the main response to reviewers, where we discuss other aspects of our setup that put the improvement in MBA in context. TLDR; significant task and data shift, 1 MBA point corresponds to nearly 100 more correct comparisons, etc.
Reviewer_uPDcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats next-action forecasting over Ego4D narrations as a language proxy for embodied temporal reasoning. A frozen forecaster proposes continuations. After the true action Y⁺ is revealed, the highest-likelihood candidate that does not match Y⁺ is kept as Y⁻. These pairs are used to build a benchmark (PDM-Eval) and to fine-tune VLMs with DPO on text only (PDM-DPO). PDM-DPO++ adds explanations that are used only during training. An RSA interpretation connects the two parts. PDM-Eval is harder than random or noun-overlap distractors across five LMs. PDM-DPO++ improves zero-shot TemporalBench MBA on LLaMA-3.2-11B-Vision (15.19→16.81) and Qwen2.5-VL-32B (24.97→25.83). On LLaMA, it also preserves OpenEQA performance and trains at lower cost than video fine-tuning.

Critical strengths
  1. Simple, cheap construction of hard contrastive pairs. The approach does not require counterfactual video synthesis. PDM pairs give the largest Brier error in all 20 comparisons against Random and all 19 available comparisons against Semantic distractors, across five LMs and four elicitation settings. Pairs proposed by a 1B model remain hard for models up to 11B (§4.2.1).
  2. Text-only adaptation that transfers to video temporal QA. Only the language layers are updated, and the method still improves TemporalBench on two backbones. Explanations are used only at training time, yet PDM-DPO++ scores higher than answer-only PDM-DPO in all six reported comparisons (§4.2.3).
Critical weaknesses
  1. The contribution of the selection rule is not directly isolated. No experiment keeps the DPO pipeline fixed and varies only the negatives (e.g., Random, Semantic, or randomly sampled π0 candidates). It is therefore difficult to attribute the downstream gains specifically to the pragmatic selection of Y⁻, as opposed to DPO with self-generated negatives.
  2. Y⁻ is not validated as an incorrect future. Candidates are removed only by automatic filtering (a MiniLM cosine threshold of 0.85 plus lexical rules; Appendix C.1). The paper has no human check that the selected Y⁻ is actually incompatible with the history, rather than another plausible continuation. It also reports no human accuracy on PDM-Eval.
  3. The downstream gains are modest. The TemporalBench improvements are about +1 MBA point (+1.62 on LLaMA, +0.86 on Qwen; Table 1). They come from a single training seed (Appendix C.2) with one observed frame in the main setting (§4.1.2). §4.2.4 also notes that part of the gain may come from stronger action priors. Gains do persist with more frames, but the paper notes that the frame sweep does not fully separate the two explanations.
  4. The benchmark construction is not clearly described across §4.1.1 and Appendix C.1. §4.1.1 states that PDM-Eval is built "using the procedure in Section 3.2" with a fixed LLaMA-3.2-1B-Instruct proposer. Appendix C.1, however, describes this 1B-based setup as a development-time construction that is "an approximation of the propose-and-verify structure in the main text". It is therefore unclear how the construction behind Fig. 3/6/7 differs from Section 3.2.
Questions for authors
  1. What are the results of PDM-DPO/++ with Random, Semantic, or randomly sampled filtered π0 negatives, keeping everything else fixed?
  2. Could the authors provide a small human study on PDM-Eval, checking whether Y⁻ is incorrect and reporting human accuracy?
  3. How exactly does the construction used for Fig. 3/6/7 differ from Section 3.2? What does "the main comparison table" in Appendix C.1 refer to?
  4. In Fig. 8, MC trained on PDM pairs transfers poorly to Random/Semantic. Could stylistic differences between recorded human narrations (Random/Semantic) and model-generated candidates (PDM) contribute to this?
5 comments

Thank you for your helpful questions. They have clarified a few key points in our experimental discussion and we will include new results/discussion in the camera ready

Q1. What are the results of PDM-DPO++ with Random, Semantic, or randomly sampled filtered π₀ negatives (isolating the effect of our selection strategy)?
M4 of the response to all reviewers explains what Fig. 3 does and does not isolate, and why the training-time comparison is deferred. TLDR; PDM does create more difficult negatives than baseline strategies, as measured by standard uncertainty measures (Brier score, a proper scoring loss).

Q2. Can the authors provide a small human study on PDM-Eval, checking whether Y⁻ is incorrect and reporting human accuracy?

  • Y- validity: A human manually verified 100 random samples, each from a different video. All 100 Y- described an action different from the recorded one.
  • Human accuracy: The task is motivated by a biological account of motor forecasting in the human nervous system, so humans are not expected to perform it explicitly, as in traditional annotation tasks. We instead report Brier Skill Scores relative to the 0.25 chance (App. D.1), which can be interpreted as the percentage of variance in the ground truth explained by the predictions.

Q3. How does the construction in Figs. 3, 6 and 7 differ from Sec. 3.2, and what does "the main comparison table" in App. C.1 refer to?

  • Figs. 3, 6, and 7: The pairs use a fixed 1B proposer, independent of the evaluated model, to isolate the distractor strategy from the forecast (App. C.1).
  • The main comparison table in App. C1: In this table, each backbone’s own pre-adaptation checkpoint (π₀) proposes and ranks the candidates and serves as the DPO reference, as in Sec. 3.2. We will edit the text to make this clearer.

Q4. Could stylistic differences between recorded human narrations (Random/Semantic) and model-generated candidates (PDM) contribute to the poor transfer of MC trained on PDM pairs?

  • Plausibly, yes. MC trained on PDM pairs overfits to features specific to PDM's model-generated negatives (Fig 3), and style may be one of them. We did not isolate style from other PDM-specific features (it is unclear how we might do this, but we can brainstorm). Training showed rapid overfitting, as the model diverged before attaining good transfer scores.
  • Why this matters for our method: We tuned MC's hyperparameters before concluding that overfitting was the cause, so the poor transfer is not a tuning failure. That is why PDM-DPO anchors to the reference model, whose KL term limits this drift (Please, also see M3 which discusses theoretical motivations).

Thanks to the authors for the detailed response and the new experiments. The extra seeds and the clarification of the Fig. 3/6/7 setup are helpful. A few concerns remain for me:

  • Q1. Appendix C.1 notes that the Random/Semantic controls do not isolate ranking within a common pool, so I don't think Fig. 3 speaks to Eq. 2 directly. A simple evaluation-only check would help: compare argmax against a random candidate from the same filtered pool. This needs no training.
  • Q2. My question was whether Y−Y^- could still be a plausible continuation, rather than just a different action. Even a small human accuracy check would help show the task is solvable rather than ambiguous.
  • Gains. With one frame, the five-seed mean gain (+1.08) is close to the no-frame gain (+1.07). To me this suggests much of the main setting gain comes from action priors.

I appreciate the effort, but for now I'll keep my score.

Thank you for engaging with our rebuttal and for further clarifying your position. We provide additional experiments and discussion of your questions below.

Q1. Thank you for the suggestion. We ran the evaluation-only check. On the same filtered pool of ten candidates per history (LLaMA, closed-format likelihood scoring), pairing Y⁺ with the argmax candidate yields a mean Brier error of 0.338, against 0.202 for a random selection from the other nine candidates (paired difference +0.136, 95% CI

+0.133,+0.139+0.133, +0.139

, 10,000 bootstrap resamples, same interval when resampling videos). Error decreases monotonically with rank (0.290 at rank 2, 0.217 at rank 5, 0.115 at rank 10). So, within a common pool, selection makes the comparison harder, as Eq. 2 predicts.

Tables are below:

Argmax vs. a random candidate from the same filtered pool.

drawsmean_brier_scorebrier_skill_score
Rank 1 (hardest distractor)0.338−0.35
Random draw from ranks 2–100.202+0.19
Random draw from all 100.215+0.14

Rank 1 - random candidate from pool (paired bootstrap, 10,000 resamples, seed 1759).

resampled_unitdifference95% CI
Histories+0.1361
+0.1328,+0.1393+0.1328, +0.1393

| | Videos (sensitivity) | +0.1361 |

+0.1327,+0.1394+0.1327, +0.1394

|

Brier score by rank of the distractor.

rankmean_brierpair_accuracy (p(Y+) > 0.5)
10.3380.43
20.2900.53
50.2170.68
100.1150.87

Q2. Authors judged 100 rank-1 pairs per video, with ground-truth blinded, and 20 random pairs from other videos as a control. The results show that the task is at least partly unambiguous for humans.

SetJudgeAccuracy (%)95% CI
100 rank-1 pairsHuman73[63.6, 80.7]
100 rank-1 pairsZero-shot model (mean)41.7[32.5, 51.5]
20 random controlsHuman90[69.9, 97.2]
20 random controlsZero-shot model (mean)85[64.0, 94.8]

We would also point out that error correction is actually a critical motivation for the motor planning loop on which our setting is based (Herrick and Yakvenko, 2025). So, there is some expectation that the task we are studying will have some ambiguity for humans. We are happy to make this clearer and also to add these experiments to the text.

Gain: Thank you for your close analysis. We wanted to point out that the limited gain you mention is comparing 1 frame to 0 frames, so this is the least available visual evidence. As such, we expect the least use of visual evidence in this case, and the most use of action priors. However, as we increase visual evidence (2-4 frames) all gains increase as well, showing more learned use of visual evidence. While you note this is not the main setting (i.e., we assume you mean the setting with the most baselines), we would point out that zero-shot is the strongest baseline in that setting, and we do compare with zero-shot for all of our multi-frame experiments, continuing to show performance boosts. We’d also highlight that, even if partly coming from linguistic action priors, it is still quite interesting that the linguistic action priors we learn from forecasting are able to transfer across datasets and tasks.

To your point, we would be more than happy to clarify in the camera ready that part of the gains do come from linguistic action priors. We feel our contributions stand regardless of this nuance. Indeed, our theory is about grounding, which is about both priors and new evidence (both linguistic and visual). We primarily point to the interesting question about visual evidence (in the discussion) as a direction for future work. We’d also be more than happy to prioritize multi-frame experiments in the main text, since they tease this issue out. We primarily chose to do the major baseline comparison on 1 frame for computational efficiency.

Thank you again for your helpful comments. Please, let us know if you have any additional concerns.

Reviewer_yzSWOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Pragmatic Dynamics Models (PDMs), a framework for learning temporal reasoning from language-based action forecasting. Given a history of narrated actions, a forecaster generates candidate futures, and the observed continuation is used to identify highly probable but unrealized outcomes. The authors use these comparisons to construct PDM-Eval, a benchmark for distinguishing observed from counterfactual events, and formulate a preference-learning objective through RSA andDPO. They further extend the approach with generated reasoning traces (PDM-DPO++). Experiments show that PDM-Eval produces more challenging comparisons than random and semantically related distractors, while language-only adaptation provides modest improvements on visual-temporal benchmarks across two VLM backbones, with lower training costs than video-based fine-tuning.

<<< UPDATES >>>

i think there has been some interesting ideas about temporal supervision in this work -- learning from plausible but unrealized future events is very important and have applications in forecasting tasks and can also indicate a cheap way of collecting conterfactual videos. in terms of empirical evidence, i think cross-modal transfer is the strongest contribution here. i think however, better performance on the visual temporal benchmarks does not mean better visual understanding, and very likely it could be some level of linguistic prior about action sequences in the dataset (rather than truly generalizable patterns) that is truly driving the improvements here, which makes the whole work questionable. the experiment controls are also insufficient (no uncertainty estimates etc) especially the gain is quite marginal to me.

after reading authors' rebuttal (that is a lot of hard work, i appreciate!), i think several parts of my questions are addressed. I think there are some weaknesses and questions to be experimentally verified:

(1) authors ack'ed that a same pool random negative training ablation is missing, if i understood correctly. (2) PDM-DPO++ improves over answer-only PDM-DPO by just 0.12 with a CI of [-0.78, +1.06]; authors also ack'ed that they have not separated explanation content from additional supervision length.

i am raising my score to a 3 and lowered my confidence, but hope authors to discuss the above in the final version of this work somehow :)

Critical strengths
  1. An interesting and computationally efficient approach to temporal supervision. Using plausible but unrealized futures as negative examples is a natural way to learn from prediction errors without constructing counterfactual videos. The framework provides a concrete method for generating such comparisons from narrated action sequences, potentially making temporal adaptation substantially cheaper than video-based training.

  2. Encouraging cross-modal transfer with relevant experimental comparisons. The observation that language-only preference optimization improves performance on visual-temporal benchmarks is interesting, particularly because the vision backbone remains frozen. I appreciate the inclusion of multiple training objectives, comparisons with and without reasoning traces, and evaluation across two different VLM backbones. The improvements across several temporal datasets suggest that the proposed supervision can be useful beyond its original language-only training setting.

Critical weaknesses
  1. The empirical gains are modest, and the experiments do not establish improved visual-temporal reasoning. PDM-DPO++ improves overall TemporalBench accuracy from 15.19% to 16.81% on LLaMA and from 24.97% to 25.83% on Qwen. While these improvements are encouraging, the paper reports only a single training seed and does not provide paired statistical tests or confidence intervals for the improvements. More importantly, since adaptation uses only textual narrations, the gains may come from improved linguistic action priors rather than better grounding or temporal reasoning over visual observations. The frame-count experiments do not fully distinguish these explanations, as the authors acknowledge. Text-only, shuffled-frame, or visually counterfactual evaluation controls would help establish whether the adapted models actually make better use of visual-temporal evidence.

  2. The theoretical formulation is more elaborate than the demonstrated algorithmic contribution, and key components are insufficiently isolated. The RSA analysis connects the proposed selection rule to preference optimization, but the resulting update closely resembles standard KL-regularized preference learning over generated hard negatives. It is unclear what the RSA formulation establishes beyond this existing interpretation or which aspects of the theory are essential to the observed improvements. In particular, the benchmark baselines change both the source of negative candidates and the selection strategy, making it difficult to isolate the benefit of selecting the highest-probability incorrect forecast. The implementation also ranks candidates using mean per-token log probability rather than the sequence probability assumed in the derivation. More controlled comparisons are needed to establish the contribution of pragmatic selection independently of hard-negative mining, DPO regularization, and generated explanations.

Questions for authors
  1. I'd like to see evidence that PDM-DPO++ actually improves the model's use of visual-temporal information, rather than just reinforcing language-based action priors. One way to test this would be to compare the gains under correct, shuffled, and absent visual inputs. If the adapted model is truly more sensitive to the observed dynamics, the gap between these conditions should widen.

  2. How much does the pragmatic selection step contribute on its own? A comparison against random negatives drawn from the same candidate pool, and against standard likelihood-based hard-negative mining with the DPO training kept the same, would help clarify this. I also noticed that the implementation scores candidates with length-normalized log probability rather than the sequence probability in Eq. 2. How sensitive are the results to that choice?

  3. Could the authors report paired bootstrap confidence intervals or significance tests for the TemporalBench improvements, ideally with additional training seeds? I'm also curious whether the benefit of the generated reasoning traces can be separated from the extra supervision length and from the use of an external reasoning model. Simpler or non-causal explanation controls might help isolate this.

1 comment

Thank you for your helpful questions. They have clarified our experimental validation and we will include new results/discussion in the camera ready.

Q1. Does PDM-DPO++ improve the model's use of visual-temporal information, rather than only reinforcing language-based action priors?
We ran the suggested comparison of gains under correct, shuffled, and absent frames. Kindly refer to M1 of the response to all reviewers. TLDR; Our method is also using visual evidence.

Q2. How much does pragmatic selection contribute on its own (random negatives from the same pool; standard likelihood-based hard-negative mining), and how sensitive are the results to length-normalized scoring?

  • Random negatives from the same pool: M4 of the response to all reviewers explains what Fig. 3 does and does not isolate, and why the training-time comparison is deferred. TLDR; PDM does create more difficult negatives than baseline strategies, as measured by standard uncertainty measures (Brier score, a proper scoring loss).
  • Standard likelihood-based hard-negative mining: Could you kindly clarify what references this refers to in our transfer setting (language forecasting to held-out vision-language tasks)? As our novel setting has distinct ground-truth availability from more traditional settings, we do not believe likelihood-based hard-negative mining can have been defined in a principled way (a priori to our analysis). For example, it is not even clear (a priori to our analysis) that we should operate in a forecasting setting to achieve this transfer. Please also see M3 of the response to all reviewers.
  • Length-normalized scoring: We score each candidate by its mean per-token log probability, an approximation of the sequence probability in Eq. 2. Candidate lengths still vary under our soft length cap (1.5 times the ground-truth length), and the sequence probability would systematically favor shorter candidates, so we normalize by length, as is standard when comparing sequences of different lengths (e.g., Wu et al., 2016, Lightman et al.). We have yet to measure the sensitivity because it requires re-ranking the candidates and retraining, and we prioritized the experiments requested in the main response. We will add it to the camera-ready version.

Q3. Can the authors report paired bootstrap confidence intervals or significance tests for the TemporalBench improvements, ideally with additional training seeds? Can the benefit of the generated explanations be separated from the extra supervision length and from the use of an external reasoning model?

  • Paired bootstrap intervals and additional seeds: We report both. Kindly refer to M2 of the response to all reviewers. TLDR; The method is consistent across seeds with improvements that are typically statistically significant when compared to other methods.
  • Explanations vs. extra supervision length: Kindly refer to M5 of the response to all reviewers.
  • External reasoning model: Kindly refer to M6 of the response to all reviewers. TLDR; We tested this and still saw improvements.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Shared Response: Part I

Thank you to all three reviewers for their time and helpful suggestions. They recognize that PDM is a simple way to build hard contrastive pairs, and that our text-only training algorithm improves vision-language temporal reasoning on two backbones at much lower cost than video fine-tuning.

Contributions the reviews do not discuss:

  1. While the reviewers correctly recognize that the method is hard-negative mining for RL with preference pairs, this method has not been studied before in our (fairly unique) setting. A priori, it is unclear why hard-negative mining should allow us to train on a language-only forecasting task and observe performance improvements on diverse, held-out temporal reasoning tasks with vision-language data. Our RSA analysis, which connects to a biological model of motor adaptation, explains why it works here and how we should implement it.
  2. The RSA analysis also changes the method, not just its description. While reviewers see it as a restatement of KL-regularized preference learning, our analysis also prescribes the hard-negative (Eq. 2), the method for eliciting reasoning, and formally shows how reasoning adds evidence (Eq. 11). This is why PDM-DPO++ samples reasoning conditioned on each outcome. Our agnostic design also makes it a reusable technique for analyzing reasoning under other verified rewards (more under n8sb Q1).
  3. The gains are harder-won than they appear.
  • MBA credits a video only if all of its ~4.5 binary comparisons are correct, so chance is 9.5% (Cai et al., 2024), and our zero-shot baselines score 15.19 and 24.97. Accounting for total number of videos, each percentage gain corresponds to nearly 100 more correct comparisons.
  • We evaluate with considerable task and data shift, training on a language only forecasting task and testing on other held-out, out-of-distribution vision-language tasks and datasets. Here, most other language-only methods hurt performance.
  • Training uses 54% less memory and 65% less time than video fine-tuning.
  • Qwen2.5-VL-32B was already tuned for video understanding, so our consistent positive gains across these settings (+1.62 and +0.86 full-TemporalBench MBA, i.e. +10.7% and +3.4% relative) are notable.

We now address concerns shared by reviewers.

M1: Is the method exploiting language priors rather than visual evidence?
We ran a new experiment (see Added Exp 1) and found that the method is in fact using visual evidence, so our gains do not come from exploiting language priors alone. Following the suggested test, we evaluated zero-shot and five PDM-DPO++ seeds with frames in order, the same frames shuffled, and no frames. Our findings show that shuffling frames reduces performance, with an even larger reduction after our method (1.79-2.46 vs. 1.29-1.65 MBA). Removing frames further reduces performance, although some improvement over zero-shot remains, suggesting that both visual evidence and learned action priors contribute.

M2: Modest gains, only one seed reported, no significance tests.
We conducted additional experiments with four new seeds (see Added Exp 2) and found consistent improvements over zero-shot across all five seeds and both frame counts. The mean gains were +1.08 ± 0.54 MBA points at one frame and +1.39 ± 0.53 at two frames (mean ± SD), with all individual seeds outperforming zero-shot. Although the absolute gains are modest, MBA is a challenging benchmark where scaling results suggest that doubling model size corresponds to roughly 2-4 MBA points of improvement (Cai et al., 2024), slightly higher than our own gains. Additionally, a paired bootstrap analysis over 2,179 video groups confirms a positive improvement over zero-shot for PDM-DPO++ (+1.62 points, 95% CI

+0.73,+2.52+0.73, +2.52

), while several alternative methods perform significantly worse than zero-shot.

M3: Is the method just hard-negative mining with DPO? What does the theory add?
There are three important contributions over hard-negative mining with KL-regularized DPO:

  1. New setting: We train on language-only forecasting and improve performance on held-out, vision-language reasoning tasks. It is not obvious, nor previously shown, that hard-negative preference learning should transfer across these tasks and modalities.
  2. Concrete algorithm: Despite some similarities in hindsight, our exact algorithm does not exist, because our unique setting makes different inputs and ground-truth available than traditional hard-negative mining. The RSA analysis is needed to make sense of things.
  3. Theoretical justification: Our RSA analysis recovers the DPO optimum, motivates the negative selection rule and reference regularization, and establishes that outcome-conditioned explanations add non-negative evidence in expectation (Eq. 11). This explains why the proposed construction works under substantial task and modality shift, where alternative language-only objectives fail.
AuthorsOpenReview ↗

Shared Response: Part II

M4: Does the selection rule itself, rather than hard-negative DPO in general, drive gains?
As noted in Part I, it is not clear if hard-negative DPO, in general, should even work in our unique setting. However, if novelty is a persistent concern, Fig. 3 does provide partial evidence: PDM produces harder negatives than Random and Semantic constructions in all 20 and 19 matched comparisons, respectively, consistent with the theoretical motivation for our selection rule (Eq. 2). Semantic constructions are also similar to what has been done before in vision-language (see related work, although we emphasize our setting is even more unique than these). We agree that a direct training-time ablation holding DPO fixed would better isolate its contribution. However, our contribution is not only the selection rule, but also the method for evidence generation, the theoretical explanation for modality shift, and the reusability of our framework across learning techniques. Each LLaMA training run also takes approximately 13 hours, and we prioritized the other experiments already shown due to time. We do plan to include currently missing experiments in the camera-ready version. See also Q1 of discussion with Reviewer uPDc for results with additional controls.

M5. Do explanations help through their content, or only by lengthening supervision? (yzSW, n8sb).
PDM-DPO and PDM-DPO++ are built from the same histories and outcomes with the same optimizer settings. The only difference is that PDM-DPO++ responses include an explanation before the action at train-time. PDM-DPO++, when compared to PDM-DPO, scores higher in all six backbone-by-benchmark comparisons (+0.12 to +1.64 points; Sec. 4.2.3), although on LLaMA's full benchmark the difference (+0.12) lies within the paired bootstrap interval (M2).
This comparison adds both explanation content and extra tokens, so it cannot separate the impact of content from the impact of length. A length-matched control with simpler or non-causal explanations, as reviewer yzSW suggests, would do so. It would also test whether conditioning explanations on the verified outcome matters (Eq. 11). However, it needs a new explanation dataset and new training runs, so we prioritized the other requested experiments (five-seed replication, shuffled and absent-frame controls, paired bootstraps). We will include it in the camera-ready version.

M6. Do the gains depend on an external model generating explanations? (yzSW, n8sb)
Not on Qwen, where we can test it (one frame, full TemporalBench). When the model being trained generates its own explanations (iterative self-play, which also re-mines negatives each round), MBA is 25.46 against 24.97 for zero-shot (+0.49), compared with +0.86 using a smaller external text-only reasoner. For LLaMA, we use an external text-only reasoner because self-generation is much less reliable. LLaMA is not a reasoning model, and 19% of its narration sentences failed our validation filter, compared with about 0.7% of pairs with the external reasoner (App. C.1).

AuthorsOpenReview ↗

Added Exp 1: Language Priors

Following the suggested test, we evaluated zero-shot and five PDM-DPO++ seeds with frames in order, the same frames shuffled, and no frames, at 2 and 4 frames (table below). Shuffling uses a random permutation that always differs from the original order, and every model sees the same permutations. "Absent" means a text-only prompt with no image. Note that we did not change any textual aspects of the prompt in this experiment, nor did we generate reasoning, so we are isolating the impact of the visual evidence only.

  • Visual evidence matters: Shuffling lowers MBA by 1.3 to 1.7 points for zero-shot and 1.8 to 2.5 for PDM-DPO++ (seed mean values). Removing frames costs 3.0 to 4.1.
  • The gap between correct and shuffled frames widens after adaptation (with our method), from 1.65 to 2.46 MBA points at 2 frames and from 1.29 to 1.79 at 4, in all 10 seed and frame-count comparisons. Equivalently, the gain over zero-shot falls from +1.39 to +0.58 (2 frames) and from +1.21 to +0.71 (4 frames) when order is destroyed.
  • The gap to absent frames widens only slightly (+0.15 to +0.32), and part of the gain remains without frames (+1.07), so improved action priors account for part of the improvement, as noted in Sec. 4.2.4.
  • All five seeds exceed zero-shot in every condition.
MBA (%)
modelsequentialshuffledabsent
nframes=2nframes=4nframes=2nframes=4nframes=0
zero-shot17.3018.2215.6516.9314.27
seed_4218.9519.6616.1817.8015.63
seed_12318.7419.3116.0917.6115.22
seed_202419.1519.6916.3217.9815.40
seed_3141517.7818.9715.8117.0814.94
seed_175918.8119.5416.7417.7515.50
Mean ± SD18.69 ± 0.5319.43 ± 0.3016.23 ± 0.3417.64 ± 0.3415.34 ± 0.27
Gain over zero-shot1.391.210.580.711.07

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.