acceptodds
Under review as a conference paper at ICLR 2027

Joint Learning For Robust And Efficient Neural Grammar Induction

Abstract

Grammar induction is the process of recovering the latent hierarchical structure of sentences without supervision, a notoriously unstable process, with different seeds converging to different grammars. While recent work on grounding probabilistic context-free grammar induction using vision as a fixed auxiliary signal showed modest improvement, it did not resolve the instability problem. Rather than treating vision as a fixed signal, we present a joint-learning approach that learns syntax and a shared image–text semantic space simultaneously, reducing variance across seeds while increasing the accuracy of the induced grammar. We test the robustness of our approach along three dimensions: across languages (English, French, and Chinese), across datasets (from Abstract Scenes to MS-COCO), and across tasks on a downstream compositional benchmark (COCO-Order task of ARO). The advantages of joint learning replicate across all three dimensions, where we highlight the importance of selecting the appropriate data to avoid collapse onto trivial grammars, introducing a harder negative-sampling regime. Furthermore, using the induced grammar model as an image–text encoder and scorer improves over CLIP on the ARO word-order task by roughly points. Finally, we demonstrate that the quality of a model's induced grammar predicts its word-order accuracy, evidence that more linguistically aligned representations generalise better.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Grammar induction, Visually grounded language learning, Compositional generalization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_gDrYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies joint learning of image–text matching and unsupervised grammar induction. The approach combines a compound probabilistic context-free grammar (C-PCFG) with a contrastive matching objective between candidate text spans and the visual representations of the images. The model uses grammar to figure out which text spans are most important and gives them higher weight during image matching. In return, the image matching feedback guides the model to learn better grammar. The authors also use visually similar images as negative examples and increase the batch size to strengthen the training.

Experiment covers three tasks: multilingual abstract scene descriptions, grammar induction, and word-order checking. The results show that joint training at start generally improves over staged training, the negative-sampling strategy and batch size greatly affect parsing accuracy and stability, and weighting scores by constituent probabilities improves word-order discrimination compared to other methods.

Critical strengths
  1. This paper conduct training process across 5 random seeds and reported both average performance and variance, which could significantly mitigate the training unstability in unsupervised grammar learning.

  2. The setting of ablation studies is comprehensive, providing useful insights regarding the training setup.

Critical weaknesses

My main concern for this paper is the limited novelty compared to prior work. The core framework of this paper, including joint optimization of grammar induction and image–text matching and constituent-probability-weighted matching losses, was already introduced in [1]. The main newly introduced extensions are broader empirical evaluations, hard-negative sampling, and batch-size analysis. While these extensions are useful, the paper does not clearly show what substantial new methodological contribution they provide beyond the old setup. I advice the authors to distinguish inherited components from new contributions more explicitly and explain why the additional findings represent a significant advance.

[1] Portelance, Eva, Siva Reddy, and Timothy J. O’Donnell. "Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction models." Journal of Memory and Language 145 (2025): 104672.

Questions for authors

Claim clearly about this paper's newly valuable contribution compared to [1].

3 comments

We are very grateful for this review, and for the direct question of what the paper adds to Portelance et al. (2025). Answering it, and running the ablations the other reviews asked for, is what produced point 3 below, which we now take as the contribution of the paper. The revised paper, with all of these changes, will be uploaded by Monday 16 November. The revision also separates, in the introduction and in a table in Section 2, what we inherit from what is new.

We claim no novelty for the components taken from Portelance et al. (2025). These are the compound PCFG, the semantic objective (a contrastive image-text loss weighted by the probability of each constituent), the semantic encoder that maps image features into the shared semantic space, the hyperparameters (including an equal weight on the syntactic and semantic objectives), and the comparison between training both objectives from the start and introducing one of them halfway through.

What the paper adds is the following.

  1. The joint-learning result is tested past a single language, on French and Chinese in the Abstract Scenes dataset. These results replicate the trends observed in English and the benefits of joint learning.
  2. It is also tested on natural images and with several visual backbones. On MS-COCO, joint learning reaches 65.4 ± 1.1 in macro span-F1. The same model trained with only the syntactic objective reaches 53.9 ± 8.0, and beats our replication of VC-PCFG (Zhao and Titov, 2020) which reaches 59.2 ± 4.5. Our model is higher in every seed, and reproduces the same trend and learning dynamics observed in Abstract Scenes.
  3. This is the new result, and the one we most want to stand behind. The ablations the reviews prompted show that the gain does not need the image. Pairing every caption at each step with an image drawn at random, so that no caption ever sees its own image, leaves the gain in place at 63.8 ± 4.7. So does keeping the semantic encoder at its random initial weights, with no gradient, at 62.7 ± 2.7. Nothing of the image content is required. The gain mostly comes from the weight on the semantic objective and the choice of sampling negatives pairs. Raising only the weight on Zhao and Titov's semantic objective from 0.01 to 1 takes their model from 58.3 to 63.7, and removing the harder negative sampling regime drops accuracy to 47.2 ± 9.5. To our knowledge this is the first time the image content has been removed from this model and the gain has remained. The revised paper describes the controls in full detail.
  4. COCO-Order, the word-order benchmark we used as a test of grounded composition, is solved better when the image is unavailable. A language model trained only on the captions reaches 99.0, the model trained on randomly paired images reaches 79.3, and our model reaches 63.8.

We came to the third point by following the ablations the reviews suggested, and it is the result we most want to stand behind. From Shi et al. (2019), through Zhao and Titov (2020), to Portelance et al. (2025), a gain in parsing from the semantic objective has been offered as evidence that seeing the image helps a learner acquire syntactic structure. Our submission argued the same thing. None of these papers, to our knowledge, trains the model with the image content removed. When we do, the gain remains. It is actually being produced by the weight on the semantic objective and by the choice of negative pairs, not by the images. This changes the reading of a series of published results, and it hands later work a test for grounding that requires only one control run. The control pairs every caption with a random image, revealing that a gain that remains under that pairing did not come from the contents of the image. The paper also explains how the semantic objective can improve parsing performance while the images contribute nothing. The improvement of our model reported in the original submission remains. Joint learning makes grammar induction more accurate and considerably more stable. The novelty is that we can now confidently say which part of the objective is responsible.

We thank the reviewer again. Being asked to state the contribution precisely is what led us to provide a more novel view on the mechanisms behind the actual learning objectives.

We thank the reviewer again for asking us to state the contribution precisely. The revised paper is now uploaded, and below we point to where it separates what is inherited from what is new. Line numbers are those in the left margin of the revised PDF, where changes are marked in blue. What we inherit is stated in three places. The introduction attributes the joint objective with equal weights and the trainable semantic encoder to Portelance et al. (2025) in lines 76 to 80, and says in lines 83 to 85 that their work had established the advantages of joint learning in one language and on one synthetic dataset. Section 2 then opens by placing the model in the VC-PCFG family of Zhao and Titov (2020) and naming the two settings in which it differs from theirs (lines 138 to 141). Figure 2 is marked as adapted from Portelance et al. What is new is listed in the same part of the introduction, and each item has its own section.

  1. French and Chinese (lines 85 to 87). The results are in Section 4, lines 297 to 310.
  2. Natural images, evaluated on images held out from training, together with the harder negative-sampling regime (lines 87 to 99). The results are in Section 5.1 with Table 1, and in lines 406 to 415 with Table 2.
  3. The random-image control, which asks whether the gains need the image at all (lines 100 to 107). It is defined in lines 228 to 235. Its results are in lines 311 to 327 for Abstract Scenes and in lines 417 to 431 for MS-COCO, and Section 5.3 studies what the semantic objective teaches the model in place of the image.
  4. The same control applied to the word-order benchmark (Section 6, lines 484 to 507, with Table 3). The contribution is then stated in one paragraph (lines 127 to 134), and the Discussion says what it changes for earlier results, including those of Portelance et al. on Abstract Scenes (lines 524 to 534). We should correct two things from our first reply. We had announced a table in Section 2 that would separate inherited from new components. For lack of space the separation is made in the text, at the lines given above. We had also quoted 47.2 ± 9.5 as the result of removing the harder negative-sampling regime. That number belongs to a different control, which changes what each span is compared with (lines 452 to 457). The effect of the regime itself is in Table 1 and in lines 344 to 346. The paper is also worded more carefully than our first reply on the central result. With five seeds, the difference between true and random images (1.5 ± 3.8) leaves room for an image contribution of a few points. The paper therefore says that the gains do not require the content of the image, and not that the image contributes nothing (lines 417 to 418), and it limits the conclusion to the VC-PCFG family (lines 532 to 534). We are grateful for the question. It is the reason the paper now says clearly what it adds.

Thank you for the detailed statement. I will raise the rating to 3.

Reviewer_DqStOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates joint optimization of a compound PCFG and an image–text semantic alignment model. A fixed visual backbone supplies image features, which a trainable semantic encoder maps into a space shared with span representations. The objective combines a text ELBO with a constituent-marginal-weighted image–text loss. Experiments compare joint training from initialization with semantics-first and syntax-first schedules. The study extends earlier joint-learning experiments to translated French and Chinese Abstract Scenes captions, natural MS-COCO images, and several visual backbones. On MS-COCO, hard-negative sampling and larger batches improve parsing accuracy and reduce seed variability. The paper also uses grammar-weighted span scores for COCO-Order, reporting 71.5% accuracy versus 46.8% for CLIP, and examines the association between parsing and word-order performance.

Critical strengths
  1. The paper treats seed instability as a central empirical problem and reports it transparently. Table 1 includes all five seeds, and the authors explicitly disclose their inability to reproduce the published baseline score under their own preprocessing. For DINO-RN50, the change from random negatives/batch size five to hard negatives/batch size twenty improves macro span-F1 from 38.6 ± 24.3 to 65.2 ± 0.5. Appendix A.2 separately varies hardness and batch size, which is an important control and avoids attributing the entire effect to one jointly changed factor.

  2. The evaluation connects induced structure to a downstream scoring task. The marginal and Viterbi scorers outperform the uniform span aggregation in Table 2, and the analysis includes multiple visual backbones and translated languages. The result is suggestive that the learned grammar contains useful structure beyond a high parsing score on one dataset, even though the stronger interpretation of grounded compositional generalization requires additional controls.

Critical weaknesses
  1. The architectural and conceptual distinction from the key prior baseline is not sufficiently established. The paper characterizes Zhao and Titov (2020) as fixed visual grounding and introduces a trainable semantic map as the key difference. However, the original VC-PCFG paper already combines alignment and language-modeling objectives in Section 3.3 and describes projecting pretrained visual features into a matching space in Section 4.2. This does not establish that the implementations are identical, but it makes the precise difference essential. The current manuscript does not specify enough about the new projection or which components of the reproduced baseline are frozen. It also explicitly credits Portelance et al. (2025) for the earlier joint-learning result. The potentially valuable new contribution is therefore the robustness/negative-sampling study and downstream evaluation, which should be separated clearly from inherited model components.

  2. The evaluation does not yet rule out important alternatives to the claimed generalization and efficiency interpretation. Section 3 describes the full MS-COCO collection and Section 6 describes training on approximately 0.6M pairs, but the actual held-out image/caption IDs and their relationship to COCO-Order are not specified. This is a missing protocol detail, not evidence that overlap occurred. The “gold” parses are automatic parser outputs, with right-binarization used in the reported comparison, so preprocessing sensitivity is important. For COCO-Order, a grammar or language model could detect unnatural word orders without using the image; the uniform scorer is not a matched text-only control. Stronger vision-free baselines are especially relevant given Li et al. (2024)(https://aclanthology.org/2024.findings-naacl.70/). Finally, comparing paired-caption counts with CLIP does not account for the visual backbone's pretraining data, compute, or the narrower nature of this evaluation.

Questions for authors
  1. What exactly differs from the original VC-PCFG image projection and joint objective, and from Portelance et al. (2025)? Please specify the semantic encoder architecture and which parameters receive gradients in each reproduced baseline. Would a matched freeze-versus-train ablation with the same loss weights, data, and optimization schedule preserve the result?

  2. Please give the actual training, validation, and test image/caption splits, the checkpoint-selection criterion, and the overlap check against COCO-Order. How is the approximately 0.6M training-pair count reconciled with the held-out evaluation?

  3. How do text-only grammar likelihood, a matched text-only sequence model, and image-permuted versions of the proposed scorer perform on COCO-Order? These controls would show whether correct visual grounding is necessary for the reported improvement, rather than recognition of implausible word order alone.

  4. How sensitive are the parsing gains to gold-tree binarization and tokenization, and can performance on original non-artificial spans or a human-annotated subset be reported? Please also provide a comparison of training time and data that accounts for backbone pretraining before making a broader efficiency claim.

3 comments

We are very grateful for this review. The questions led us to the ablations, and to what we take as the contribution of the paper, that the gain does not come from the images. Every change described here will be in the revised PDF, which we will upload by Monday 16 November. Numbers are macro span-F1 over five seeds unless noted.

Question 1 asks what differs from VC-PCFG and from Portelance et al. (2025), and which parameters are trained. We are especially grateful for this question. It showed that the semantic encoder, which the submission credited for the gain, is not its source, and the ablations below follow from that. VC-PCFG (Zhao and Titov, 2020) already trains its projection and both objectives, so we drop calling it fixed grounding. Our model is the joint-learning model of Portelance et al., with their hyperparameters. Against their released code, their projection is linear where ours is a two-layer MLP, the weight on the semantic objective is 0.01 where ours is 1, and the hidden size is 512 where ours is 768. Only the visual backbone receives no gradient. On seeds 91, 214 and 527 their model scores 58.3. Raising only that weight to 1 gives 63.7 ± 0.7, and our MLP at their weight of 0.01 gives 56.2 ± 4.0. The weight of the semantic objective explains most of the gap, not the semantic encoder.

The same question asks for the semantic encoder frozen. With the loss weights, the data and the schedule unchanged, training it gives 65.4 ± 1.1 and freezing it at its random starting weights gives 62.7 ± 2.7. Training with only the syntactic objective, so the model never sees an image, gives 53.9 ± 8.0.

Once the semantic encoder had been ruled out, we ran one further ablation, giving each caption a random training image at every step, so that no caption is paired with its own image. F1 score is 63.8 ± 4.7, against 65.4 on the true pairs. Every image is mapped to nearly one vector (mean pairwise cosine 0.999), so the gain cannot be credited to the images.

The gain comes from the semantic objective, from its weight, as Question 1 showed, and from the choice of negatives. Changing how the negatives are chosen scores 47.2 ± 9.5, below the syntactic objective alone. Shi et al. (2019) and our earlier submission read such a gain as vision helping syntax, but with no image content left, this is clearly is not evidence of grounding.

Question 2 asks for the splits and the checkpoint rule. We are grateful for it, because it showed that the submitted evaluation used images seen in training. We train on 118,287 images and 590,191 captions and test on the 5,000 COCO-Order images and their 24,957 captions. With no validation set, every run is reported at its final epoch and no seed is dropped. Held out, COCO-Order falls from 71.5 to 64.2 ± 2.1, and parsing from 67.3 ± 3.6 to 65.4 ± 1.1 with ResNet-152 and from 65.2 ± 0.5 to 63.9 ± 3.2 with DINO-RN50.

Question 3 asks how a text-only grammar, a text-only sequence model, and a scorer with permuted images do on COCO-Order.

ScorerP@1 (%)
Ours, true image63.8 ± 2.3 / 42.9
Trained with random image pairings79.3 ± 4.6
Grammar's probability of the caption, no image used97.6 to 97.9
Language model trained on the same captions99.0 ± 0.2

The two scorers that never see an image are the highest, so an implausible word order is recognisable without an image. We withdraw COCO-Order and the CLIP comparison as evidence of grounded composition, and Section 6 studies the benchmark.

Question 4 covers binarisation, tokenisation, annotation and training cost. It gave us a broader evaluation, on the original trees as well as the binarised ones. Binarising adds spans the parser never produced, since a node in the original trees may have more than two children.

BinarisedOriginal
Right-branching58.451.8
Syntactic objective only53.9 ± 8.050.6 ± 6.6
Zhao and Titov59.2 ± 4.556.2 ± 3.7
Ours65.4 ± 1.159.9 ± 0.9

The order is unchanged and ours is higher on every seed, though the lead over Zhao and Titov falls from 6.2 to 3.7 ± 3.5. Our tokenisation maps rare words to an unknown token and cuts captions at 25 words. There are no human-annotated trees, and the paper states this as a limitation. A run takes 34.1 A100 GPU-hours for our model and 23.9 for Zhao and Titov's, on a backbone pretrained on ImageNet (1.28M images). We drop the efficiency claim and remove "Efficient" from the title.

On Li et al. (2024), their LC-PCFG, trained on text alone, reports 67.8 ± 1.2 sentence-level F1, above our 59.9 ± 0.9. The numbers are not directly comparable, since they keep the best four of ten seeds and we report all five, on different splits. Our reimplementation is running and will be added when finished.

We thank the reviewer again for the detailed questions. The revised paper is now uploaded, and to make it easy to check we say below where each question is answered. Line numbers are those in the left margin of the revised PDF, where changes are marked in blue. Question 1 asks what differs from VC-PCFG and from Portelance et al. (2025). The introduction now describes VC-PCFG as a model that projects pretrained image features linearly and adds the semantic objective with a small weight, and it states what we take from Portelance et al. (lines 72 to 80). Section 2 opens with the two differences from VC-PCFG, the weight of the semantic objective and a multilayer semantic encoder in place of a linear map (lines 138 to 141). Section 2.2 states that the vision encoder is never updated and that the semantic encoder is trained (lines 187 to 190), and Section 2.3 gives the weights, 1 against 0.01 (lines 218 to 220). The effect of each difference, changed one at a time, is in lines 426 to 428 and in Appendix A.8 (Table 8), which also lists the embedding widths. The freeze-versus-train ablation asked for in the same question is the frozen-encoder control. It is defined in lines 234 to 235, and its result is in Table 2 and lines 421 to 423. The random-image control that followed from it is defined in lines 230 to 234 and reported in lines 417 to 421 and Appendix A.7. The control on how negatives are chosen is in lines 452 to 457 and Appendix A.11. Question 2 asks for the splits and the checkpoint rule. Section 3 gives the protocol in lines 278 to 282. The 5,000 COCO-Order images and their 24,957 captions are held out, training uses the remaining 118,287 images, and every run is reported at its final checkpoint, with five seeds and none dropped. Lines 406 to 409 say plainly that the first comparison was scored on captions seen in training. In the main text only Table 1 is still trained on all images, and its caption says so. The numbers of the original submission are kept in Appendix A.5, where they are compared with the held-out ones. Question 3 asks for text-only and image-permuted scorers on COCO-Order. They are introduced in lines 481 to 482 and reported in Table 3 (page 10). The table has a column for the image of another item, a column for the mean image embedding, and rows for a language model and for the likelihood of the induced grammar, neither of which sees an image. The discussion is in lines 497 to 507, and all models and scorers are in Appendix A.12 (Table 12). The comparison with CLIP is removed, and the benchmark is no longer offered as evidence of grounded composition. Question 4 covers binarisation, annotation and cost. Every held-out model is now also scored against the original n-ary trees (lines 280 to 282, the n-ary column of Table 2, and lines 412 to 415). Lines 459 to 462 check that the model has not simply become right-branching, and Appendix A.9 gives recall by phrase type on the n-ary trees. Training cost is in Appendix A.13. "Efficient" is removed from the title and the paper makes no efficiency claim. Li et al. (2024) are discussed in lines 528 to 530, where we say that their text-only model is more accurate on MS-COCO than every model we study. We should be open about two things. The first is that the tokenisation details, the pretraining data of the backbone and the absence of human-annotated trees are answered in our first reply but are not stated in this version of the paper. The third is that our reimplementation of the model of Li et al. has not finished, so it is not included. We are grateful for the care that went into this review. Nearly every control in the revised paper goes back to one of its questions.

I appreciate the author's rebuttal, which addressed my concerns; I have decided to raise my score.

Reviewer_WxRzOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of unsupervised grammar induction which is very unstable process from an optimization perspective. Models trained using same data could converge to completely different grammars depending on seeds. The paper shows that jointly learning syntax and visually grounded semantics makes grammar induction more accurate and stable.

Critical strengths

The paper is well written and targets a fundamental and interesting problem of unsupervised grammar induction. Experimental results sufficiently validate the main idea and claims for the paper. The proposed method is validated across three language which shows it's cross-linguistic generalization.

Critical weaknesses

Paper should include more baselines e.g. Table 2 should include a baseline for the grammar induced by Zhao et al. This would help understand how much improvement in grammar leads to downstream performance improvements and further highlight the strengths of the proposed method.

Questions for authors

Please see above.

2 comments

We are very grateful for the positive assessment and for asking us to add the grammar of Zhao and Titov (2020) to Table 2. That comparison, together with the ablations prompted by the other reviews, showed that the gain of joint learning does not come from the images, which we now take as the main finding of the work. The revised PDF, with all of these changes, will be uploaded by Monday 16 November.

We removed the 5,000 COCO-Order images from MS-COCO training, trained every model with five seeds, and report the final checkpoint with no selection among them. The revised Table 2 gives precision at 1, in percent, on these held-out images.

ScorerTrue imageImage of another itemUniform
Joint, DINO-RN5064.2 ± 2.142.16.7
Joint, ResNet-15263.8 ± 2.342.97.4
VC-PCFG (Zhao and Titov, 2020, our replication)67.0 ± 2.648.14.5
Probability of the caption under the induced grammar97.6 to 97.9
Language model trained on the same captions99.0 ± 0.2

The last two rows never consult the image. One is the probability the induced grammar assigns to the word sequence, and the other is an ordinary language model trained on the same captions.

Zhao and Titov's grammar answers the question directly. It parses less accurately than ours (59.2 ± 4.5 against 65.4 ± 1.1 in macro span-F1) and yet scores slightly higher on COCO-Order, so a better grammar does not bring higher word-order accuracy on this task. The last two rows show why the task can be solved from word order alone, in line with what has been found for other word-order benchmarks (Hsieh et al., 2023). Our own scorer does use the image, scoring 64 with the true image and 42 with a mismatched one, but that use of the image is not what the benchmark rewards. The revision presents Table 2 as an analysis of what the benchmark measures, and no longer as evidence of compositional generalisation or as a comparison with CLIP.

We thank the reviewer again. The baseline they asked for was one of the comparisons that led us to a more accurate account of what the model learns.

We thank the reviewer again for the encouraging assessment. The revised paper is now uploaded, and this note says where the baseline you asked for can be found. Line numbers are those in the left margin of the revised PDF, where changes are marked in blue. The grammar of Zhao and Titov (2020) is now in the downstream table, which has become Table 3 (page 10), in the row "VC-PCFG (our repl.)". The table is discussed in Section 6, lines 484 to 507, and every model and scorer is listed in Appendix A.12 (Table 12). You asked for this baseline to see how much a better grammar helps downstream. Table 2 and lines 409 to 412 give the parsing side. On held-out images Zhao and Titov's model reaches 59.2 ± 4.5 and ours 65.4 ± 1.1. Table 3 gives the downstream side, where their model scores 67.0 and ours 63.8. A better grammar therefore does not bring a higher score on this benchmark, and lines 502 to 507 explain why. A language model that never sees an image reaches 99.0, so what the task rewards is the recognition of natural word order. On the more general request for baselines, Table 2 now also has a right-branching baseline, the model trained with the syntactic objective only, and three controls, all defined in Section 2.3 (lines 228 to 239). The most informative of them pairs every caption with a random image. It keeps most of the parsing gain (lines 417 to 425), and this has become the main finding of the paper. All of these numbers are measured on images held out from training (Section 3, lines 278 to 282). The results of the original submission, which used all images, are kept in Appendix A.5. We are grateful for the suggestion, which turned out to matter more than we expected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.