acceptodds
Under review as a conference paper at ICLR 2027

Learning Words by Simulating Worlds

Abstract

Result verbs such as open, slice, and fill describe not only what is present in a scene, but how an action transforms one world state into another. We study whether this causal structure provides a privileged learning signal for language models. We train an autoregressive model on interleaved language and visual transition sequences, where each example contains a triple of a pre-condition image, an action description, and a post-condition image. The model is trained with complementary objectives: inferring the action from an observed pre/post transition, predicting the post-condition from the pre-condition and action, and reconstructing the pre-condition from the post-condition and action. Across controlled ablations, visual transition simulation improves grounded word acquisition when it is paired with sufficient language supervision. The gains are concentrated on verbs rather than nouns, and within verbs, on result verbs rather than manner verbs, consistent with the fact that result verbs are defined by observable state changes while manner verbs depend more on motion trajectories. These findings suggest that grounding verb semantics requires more than aligning words with percepts: it requires predictive models of how the world changes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
grounded language learning, lexical acquisition, vision-language modeling, world models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YMeaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether predicting visual changes helps a model learn word meanings. It trains a shared autoregressive model on before-and-after images and action descriptions, mixing action-description prediction with forward and backward visual prediction. The visual backbone is pretrained, while the language components are initialized from scratch. Experiments on synthetic image pairs and real-video frames show improvements in word prediction under several training mixtures.

Critical strengths
  1. The paper compares several training mixtures, sequence orders, and visual losses. These experiments help show when visual prediction benefits word prediction.
  2. The improvements also appear on real-video data. This provides useful evidence that the results are not entirely due to artifacts in the generated image pairs.
Critical weaknesses
  1. The evidence for faster word acquisition is not fully convincing. The nAoA metric measures when each learning curve reaches a threshold relative to its own observed range. Reaching that threshold earlier does not necessarily mean reaching the same level of word-prediction performance earlier.
  2. The unseen-word evaluation is unclear. The paper says that the unseen split contains combinations whose verb or object lemma is held out from training. However, the model uses whole-word tokens with randomly initialized language components. It is therefore important to distinguish new combinations of familiar words from words that never appear in training. Without this distinction, it is difficult to judge what kind of generalization the results demonstrate.
Questions for authors
  1. Do the faster-acquisition results, especially the difference between result and manner verbs, still hold when each word is evaluated using the same absolute surprisal threshold across training conditions?
  2. In the unseen split, are the target words absent from all training text, or are only their verb-object combinations unseen? If words are entirely held out, how are their vocabulary entries handled, and what training signal allows the model to learn their meanings?
Reviewer_9cuUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper proposes a hypothesis (and computational account) for learning action words, i.e., verbs, by learning from triplets of (pre-condition image, action description, post-condition image).
Critical strengths
  1. The paper tackles a very interesting hypothesis: for learning action verbs, we need to learn them from pairing language with state transitions. In particular, by simulating the effects of action descriptions on observations from the world. This is relevant to a broader audience beyond AI: in particular to cognitive scientists and developmental psychologists.
Critical weaknesses
  1. The differences in Table 4, are they simply due to data augmentation from additional training objectives? It seems like most of the training curves are identical up to the first half of training (Figure 2)? The method may still seem useful for increasing the final performance of a system given fixed data -- but it might suggest the mechanism for achieving these gains is due simply to data augmentation. If so, perhaps simulation is not important; merely statistical associations between video events and text descriptions. To address this, Perhaps they can compare in a data-infinite setting where data is not repeated (I suppose the first half of the training is already somewhat, and in this regime there are in fact no differences!). Or compare against other data augmentation strategies, maybe generating rephrasings of the text? I hope the paper can think more deeply about how to address this potential confound.
  2. Given the current writing, it is not clear the implications of the results, and the fields or questions this bears on. Is this work meant to bear on the field of language acquisition in development or cognitive science? If so, it would be very nice if they added more discussion relevant to this topic, concretize their hypothesis as a learning account for humans, and perhaps bring in human data (in the form of the training data being more developmentally plausible, and/or by comparing the trained models to human experimental results). If the paper is meant for the AI audience, it is as written not very clear why the results are important.
Questions for authors

None.

Reviewer_y6W3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates whether modeling action-induced visual state transitions can improve grounded word acquisition, particularly for verbs whose meanings depend on changes in the world. The authors propose an autoregressive world-language model that jointly learns from pre-condition images, action descriptions, and post-condition images through three complementary objectives: action-text prediction, forward visual simulation, and backward visual simulation. The framework uses a pretrained visual backbone with language components initialized from scratch and is evaluated on both real-video data and a synthetic world-model-distilled dataset. Through controlled experiments varying supervision mixtures, visual prediction losses, and training objectives, the authors demonstrate that incorporating visual simulation alongside sufficient language supervision reduces lexical surprisal and accelerates word acquisition compared with language-only training. The improvements are more pronounced for verbs than nouns, particularly for result verbs that describe observable state changes rather than manner verbs that characterize motion trajectories. Backward simulation is found to provide especially strong benefits for verb acquisition speed, while combining forward and backward simulation generally yields the best final lexical predictability.

Critical strengths
  • Insightful research motivation connecting world modeling with grounded word acquisition. This work investigates whether predicting action-induced visual state transitions facilitates word learning, moving beyond conventional static vision-language alignment. Its distinction between result verbs and manner verbs provides a linguistically motivated hypothesis about which semantic categories should benefit from visual simulation. This connection between predictive world modeling and lexical acquisition is conceptually meaningful and offers insights into how different types of words can be grounded in visual experiences.

  • Comprehensive controlled experiments with consistent empirical findings. This work systematically evaluates forward and backward visual simulation, different supervision mixtures, and alternative visual prediction losses across both real-video and synthetic datasets. The results consistently demonstrate that simulation improves verb acquisition more than noun acquisition, with particularly pronounced improvements in acquisition speed for result verbs. The finding that backward simulation can outperform forward simulation in acquisition speed further strengthens the empirical analysis by revealing differences between complementary learning objectives. These experiments provide evidence supporting the paper's central hypothesis rather than merely demonstrating aggregate performance improvements.

Critical weaknesses
  • Insufficient evidence that the improvements arise from genuine causal understanding rather than auxiliary visual supervision. Although the experiments demonstrate improvements in lexical surprisal and neural age of acquisition, these metrics primarily measure the predictability of target words rather than whether the model has learned meaningful causal relationships between actions and state changes. The evaluation provides both pre-condition and post-condition images when predicting action words, potentially allowing the model to exploit visual differences or statistical correlations without learning action-conditioned dynamics. Furthermore, the paper lacks targeted counterfactual evaluations or controls that distinguish causal transition understanding from generic representation improvements induced by auxiliary visual prediction. This limitation weakens the central claim that world simulation, rather than additional visual supervision, is responsible for improved grounded word acquisition.

  • The visual simulation objective does not explicitly enforce action-conditioned state transitions. The proposed simulation loss (Eq. 6) uses InfoNCE to align predicted visual patch embeddings z^i\hat{\mathbf{z}}_i with target embeddings zi\mathbf{z}_i: LSIM=−1∣SSIM∣∑i∈SSIMlog⁡exp⁡(z^i⊤zi/τ)∑j∈SSIMexp⁡(z^i⊤zj/τ).\mathcal{L}_{\mathrm{SIM}} = -\frac{1}{|\mathcal{S}_{\mathrm{SIM}}|} \sum_{i \in \mathcal{S}_{\mathrm{SIM}}} \log \frac{\exp\left(\hat{\mathbf{z}}_i^\top \mathbf{z}_i / \tau\right)}{\sum_{j \in \mathcal{S}_{\mathrm{SIM}}} \exp\left(\hat{\mathbf{z}}_i^\top \mathbf{z}_j / \tau\right)}. Although the model conditions on action descriptions, this objective only encourages correct patch-embedding prediction and does not explicitly require predictions to depend on the specified action. Since pre-condition and post-condition images often share substantial visual content, the model could minimize the loss by exploiting visual continuity and scene appearance rather than learning action-specific transformations. The absence of action-swapping experiments, counterfactual negative samples, or explicit transition-consistency constraints makes it difficult to establish whether the learned representations genuinely capture action-conditioned dynamics. This is particularly consequential because action-conditioned simulation is the central methodological premise of the paper.

  • Limited evaluation of generalization beyond the training distribution. Although the paper introduces seen and unseen test splits, the unseen split contains verb–object combinations in which either the verb or object lemma is held out, making it difficult to isolate compositional generalization from generalization to unseen lexical items. Furthermore, the reported improvements in the seen and unseen splits do not consistently reduce the generalization gap. For example, Table 3 shows that combining forward language modeling with backward simulation yields an unseen–seen pair-surprisal gap of 1.78, compared with 1.72 for the forward language-only baseline. Thus, while simulation improves absolute lexical predictability, the experiments provide limited evidence that it improves generalization to novel action–object compositions. This weakens the broader claim that the proposed framework learns transferable action semantics rather than dataset-specific statistical associations.

Questions for authors
  • Clarification on action-conditioned learning: In Eq. (6), the InfoNCE objective encourages alignment between predicted and target visual embeddings but does not explicitly enforce sensitivity to the action description. Can the authors provide evidence that the model learns action-specific state transformations rather than exploiting visual continuity between pre-condition and post-condition images? For example, how does simulation performance change when action descriptions are shuffled or replaced with incorrect actions?

  • Clarification on the contribution of visual simulation: The improvements in lexical surprisal could arise from additional visual representation learning rather than causal transition modeling. Can the authors compare the proposed simulation objectives against an auxiliary visual prediction baseline that does not require action-conditioned state transitions? Such a comparison would help establish whether the improvements specifically originate from simulation of action effects.

  • Clarification on compositional generalization: Table 3 shows that simulation reduces absolute surprisal on unseen examples but does not consistently reduce the unseen–seen generalization gap. Can the authors clarify whether the proposed framework improves compositional generalization to novel verb–object combinations when both individual words have appeared during training? Separating unseen combinations of known words from examples containing entirely unseen lexical items would help substantiate claims about transferable action semantics.

  • Clarification on forward versus backward simulation: Tables 3 and 5 indicate that backward simulation often provides stronger improvements than forward simulation, particularly in verb acquisition speed. Can the authors explain why predicting a pre-condition from a post-condition and action provides a stronger learning signal? Is this advantage attributable to learning meaningful inverse action dynamics, or could it arise from differences in prediction difficulty, visual ambiguity, or the information available in the conditioning context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.