- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether predicting visual changes helps a model learn word meanings. It trains a shared autoregressive model on before-and-after images and action descriptions, mixing action-description prediction with forward and backward visual prediction. The visual backbone is pretrained, while the language components are initialized from scratch. Experiments on synthetic image pairs and real-video frames show improvements in word prediction under several training mixtures.
Critical strengths
- The paper compares several training mixtures, sequence orders, and visual losses. These experiments help show when visual prediction benefits word prediction.
- The improvements also appear on real-video data. This provides useful evidence that the results are not entirely due to artifacts in the generated image pairs.
Critical weaknesses
- The evidence for faster word acquisition is not fully convincing. The nAoA metric measures when each learning curve reaches a threshold relative to its own observed range. Reaching that threshold earlier does not necessarily mean reaching the same level of word-prediction performance earlier.
- The unseen-word evaluation is unclear. The paper says that the unseen split contains combinations whose verb or object lemma is held out from training. However, the model uses whole-word tokens with randomly initialized language components. It is therefore important to distinguish new combinations of familiar words from words that never appear in training. Without this distinction, it is difficult to judge what kind of generalization the results demonstrate.
Questions for authors
- Do the faster-acquisition results, especially the difference between result and manner verbs, still hold when each word is evaluated using the same absolute surprisal threshold across training conditions?
- In the unseen split, are the target words absent from all training text, or are only their verb-object combinations unseen? If words are entirely held out, how are their vocabulary entries handled, and what training signal allows the model to learn their meanings?