- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies joint learning of image–text matching and unsupervised grammar induction. The approach combines a compound probabilistic context-free grammar (C-PCFG) with a contrastive matching objective between candidate text spans and the visual representations of the images. The model uses grammar to figure out which text spans are most important and gives them higher weight during image matching. In return, the image matching feedback guides the model to learn better grammar. The authors also use visually similar images as negative examples and increase the batch size to strengthen the training.
Experiment covers three tasks: multilingual abstract scene descriptions, grammar induction, and word-order checking. The results show that joint training at start generally improves over staged training, the negative-sampling strategy and batch size greatly affect parsing accuracy and stability, and weighting scores by constituent probabilities improves word-order discrimination compared to other methods.
Critical strengths
-
This paper conduct training process across 5 random seeds and reported both average performance and variance, which could significantly mitigate the training unstability in unsupervised grammar learning.
-
The setting of ablation studies is comprehensive, providing useful insights regarding the training setup.
Critical weaknesses
My main concern for this paper is the limited novelty compared to prior work. The core framework of this paper, including joint optimization of grammar induction and image–text matching and constituent-probability-weighted matching losses, was already introduced in [1]. The main newly introduced extensions are broader empirical evaluations, hard-negative sampling, and batch-size analysis. While these extensions are useful, the paper does not clearly show what substantial new methodological contribution they provide beyond the old setup. I advice the authors to distinguish inherited components from new contributions more explicitly and explain why the additional findings represent a significant advance.
[1] Portelance, Eva, Siva Reddy, and Timothy J. O’Donnell. "Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction models." Journal of Memory and Language 145 (2025): 104672.
Questions for authors
Claim clearly about this paper's newly valuable contribution compared to [1].
3 comments
We thank the reviewer again for asking us to state the contribution precisely. The revised paper is now uploaded, and below we point to where it separates what is inherited from what is new. Line numbers are those in the left margin of the revised PDF, where changes are marked in blue. What we inherit is stated in three places. The introduction attributes the joint objective with equal weights and the trainable semantic encoder to Portelance et al. (2025) in lines 76 to 80, and says in lines 83 to 85 that their work had established the advantages of joint learning in one language and on one synthetic dataset. Section 2 then opens by placing the model in the VC-PCFG family of Zhao and Titov (2020) and naming the two settings in which it differs from theirs (lines 138 to 141). Figure 2 is marked as adapted from Portelance et al. What is new is listed in the same part of the introduction, and each item has its own section.
- French and Chinese (lines 85 to 87). The results are in Section 4, lines 297 to 310.
- Natural images, evaluated on images held out from training, together with the harder negative-sampling regime (lines 87 to 99). The results are in Section 5.1 with Table 1, and in lines 406 to 415 with Table 2.
- The random-image control, which asks whether the gains need the image at all (lines 100 to 107). It is defined in lines 228 to 235. Its results are in lines 311 to 327 for Abstract Scenes and in lines 417 to 431 for MS-COCO, and Section 5.3 studies what the semantic objective teaches the model in place of the image.
- The same control applied to the word-order benchmark (Section 6, lines 484 to 507, with Table 3). The contribution is then stated in one paragraph (lines 127 to 134), and the Discussion says what it changes for earlier results, including those of Portelance et al. on Abstract Scenes (lines 524 to 534). We should correct two things from our first reply. We had announced a table in Section 2 that would separate inherited from new components. For lack of space the separation is made in the text, at the lines given above. We had also quoted 47.2 ± 9.5 as the result of removing the harder negative-sampling regime. That number belongs to a different control, which changes what each span is compared with (lines 452 to 457). The effect of the regime itself is in Table 1 and in lines 344 to 346. The paper is also worded more carefully than our first reply on the central result. With five seeds, the difference between true and random images (1.5 ± 3.8) leaves room for an image contribution of a few points. The paper therefore says that the gains do not require the content of the image, and not that the image contributes nothing (lines 417 to 418), and it limits the conclusion to the VC-PCFG family (lines 532 to 534). We are grateful for the question. It is the reason the paper now says clearly what it adds.
Thank you for the detailed statement. I will raise the rating to 3.
We are very grateful for this review, and for the direct question of what the paper adds to Portelance et al. (2025). Answering it, and running the ablations the other reviews asked for, is what produced point 3 below, which we now take as the contribution of the paper. The revised paper, with all of these changes, will be uploaded by Monday 16 November. The revision also separates, in the introduction and in a table in Section 2, what we inherit from what is new.
We claim no novelty for the components taken from Portelance et al. (2025). These are the compound PCFG, the semantic objective (a contrastive image-text loss weighted by the probability of each constituent), the semantic encoder that maps image features into the shared semantic space, the hyperparameters (including an equal weight on the syntactic and semantic objectives), and the comparison between training both objectives from the start and introducing one of them halfway through.
What the paper adds is the following.
We came to the third point by following the ablations the reviews suggested, and it is the result we most want to stand behind. From Shi et al. (2019), through Zhao and Titov (2020), to Portelance et al. (2025), a gain in parsing from the semantic objective has been offered as evidence that seeing the image helps a learner acquire syntactic structure. Our submission argued the same thing. None of these papers, to our knowledge, trains the model with the image content removed. When we do, the gain remains. It is actually being produced by the weight on the semantic objective and by the choice of negative pairs, not by the images. This changes the reading of a series of published results, and it hands later work a test for grounding that requires only one control run. The control pairs every caption with a random image, revealing that a gain that remains under that pairing did not come from the contents of the image. The paper also explains how the semantic objective can improve parsing performance while the images contribute nothing. The improvement of our model reported in the original submission remains. Joint learning makes grammar induction more accurate and considerably more stable. The novelty is that we can now confidently say which part of the objective is responsible.
We thank the reviewer again. Being asked to state the contribution precisely is what led us to provide a more novel view on the mechanisms behind the actual learning objectives.