acceptodds
Under review as a conference paper at ICLR 2027

Semantic Role Projection: Structured Visual Embeddings for Compositional Reasoning in Multimodal LLMs

Abstract

Multimodal large language models (MLLMs) rely on learned projectors to translate visual patch features into the language model's token embedding space. We identify a fundamental limitation of standard projectors— semantic role flattening—whereby object identity, attribute, count, and spatial relation information is collapsed into an undifferentiated embedding region, despite the LLM's own text embeddings maintaining clear role-specific subspace structure. We propose the Semantic Role Projector (SRP), which replaces the standard MLP projector with four parallel typed projection heads supervised by role-alignment and inter-role orthogonality losses derived from the LLM's frozen text-embedding subspaces. A lightweight learned router produces soft role assignments per visual patch, and a mixing layer preserves drop-in compatibility with existing MLLM architectures. We prove that SRP provably increases the attention logit gap for role-selective heads under mild spectral assumptions, providing a mechanistic explanation for its compositional advantage. Experiments on CLEVR-CoGenT, CLOSURE, GQA, TallyQA, and VQAv2 demonstrate that SRP achieves state-of-the-art compositional generalization—improving CoGenT Condition B accuracy by +3.6 pp over the best modular baseline (Honeybee) and +5.3 pp over a capacity-matched Mixture-of-Experts-while adding only 3% wall-clock overhead. Ablations confirm that each geometric loss component is essential, and cross-architecture evaluation on Qwen2.5-VL-7B verifies that the gains transfer across LLM backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Compositional Reasoning, Semantic Role Projection, Vision-Language Alignment, Representation Learning, Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TAkmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the Semantic Role Projector (SRP), an MoE-like projector for vision-language models that aims to mitigate semantic role flattening and preserve task-relevant visual information for compositional reasoning. The method introduces role-specific projection heads, with geometric constraints derived from the language model’s contextual text representations and a separate auxiliary counting objective. Experiments on LLaVA-1.5-7B and Qwen-VL-7B report improvements over standard and capacity-matched projector baselines.

Critical strengths
  • The paper is well motivated by semantic role flattening in VLM projectors, a potentially important direction for improving visual reasoning.
  • I appreciate the authors’ focus on representative question types in multimodal reasoning, which are valuable for evaluating VLMs’ practical capabilities.
  • Using specialized projectors to preserve fine-grained information, such as counts and spatial relations, rather than relying on a general semantic representation, is interesting.
Critical weaknesses
  • The construction of the target role geometry is rather coarse. The predefined vocabulary sets may have limited coverage, and direct alignment with a text subspace does not guarantee preservation of task-relevant visual information.
  • Why is Count excluded from the subspace alignment loss? This special treatment is not sufficiently justified.
  • The evaluation is limited to Qwen-VL-7B and LLaVA-1.5-7B. It remains unclear whether the same bottleneck persists in more recent VLMs, or whether basic counting and spatial reasoning errors mainly reflect limitations in training data rather than projector design.
Questions for authors

See in weakness.

Reviewer_eidjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a Semantic Role Projector with four routed heads for object, attribute, count, and relation information. It combines text-subspace alignment, inter-role separation, and counting supervision to improve compositional visual reasoning. Experiments report improvements across synthetic and natural-image benchmarks on two MLLM backbones.

Critical strengths

The projector preserves the existing MLLM token interface and requires no scene annotations at inference, making the approach potentially practical. The reported gains over capacity-matched baselines across two backbones provide encouraging preliminary evidence.

Critical weaknesses
  1. Inconsistent subspace construction. Section 3.3.1 reports 38 relation PCA components from only 23 contextual token embeddings. The centered data matrix has rank at most 22, so the stated procedure cannot produce 38 variance-bearing components. This inconsistency directly affects a core component of the method.
  2. The diagnostic does not establish semantic role flattening. Text probing uses four role classes, when visual labels exclude counting and allow multiple roles per patch. The paper does not explain how these incompatible labeling schemes are reconciled. Consequently, the accuracy gap cannot reliably establish that visual projection destroys role information.
  3. The experiments do not isolate the proposed semantic mechanism. The loss ablations show that auxiliary objectives help, but do not establish that alignment to meaningful linguistic subspaces is responsible. Controls with equivalent supervision and random subspaces are missing.
  4. The alignment operates across insufficiently justified representation spaces. The target subspaces are estimated from layer-10 contextual text representations, but the alignment loss is applied to projector outputs before they enter the LLM. The paper does not demonstrate that this alignment survives subsequent layers or improves the role selectivity of actual attention heads.
  5. Poor presentation of the method. Figure 1 is overloaded with small text and tightly packed panels, making the data flow and supervision pathways difficult to follow. Its schematic clusters and attention scores do not provide empirical evidence for the claimed representation structure.
  6. Insufficient experimental support for the central mechanism. The experiments do not distinguish the benefit of semantic subspace alignment from additional structured supervision. Moreover, the claimed five-seed results are reported without variability estimates. Thus, the reported accuracy gains can not validate the paper’s central explanation.
  7. The orthogonality objective does not enforce orthogonality. Even antiparallel, maximally dependent representations incur zero penalty. This weakens the interpretation of the objective as enforcing distinct role subspaces.
  8. The claimed four-role alignment is not implemented by the stated objective. The count head is excluded from subspace alignment and supervised only through scalar count regression. The estimated count subspace therefore has no explicit role in the training objective, leaving the claimed cardinality geometry unsupported.
  9. The bounding-box noise experiment is insufficiently specified. It is unclear whether corruption occurs during training or evaluation, and why the standard MLP baseline deteriorates when scene annotations are not part of its stated objective.
Questions for authors
  1. How is the relation subspace dimension of 38 obtained from only 23 contextual token embeddings, whose centered matrix has rank at most 22? Please specify the actual PCA samples.
  2. How are overlapping visual role labels and the exclusion of counting handled in the probe evaluation? What makes the reported text and visual accuracies comparable?
  3. Do SRP’s gains persist with supervision-matched baselines and random or shuffled role subspaces? These controls would help distinguish semantic alignment from additional supervision or generic regularization.
  4. How does the one-sided cosine penalty establish orthogonality when antiparallel representations incur no penalty? What role does the count text subspace play?
  5. Is bounding-box corruption applied during training or evaluation? Why does it affect the standard MLP baseline, which does not use the stated scene-graph auxiliary losses?
Reviewer_J5dQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper suggests MLP projectors in MLLMs mix up object, attribute, count and relation information, and proposes SRP with four role-specific projection heads aligned to LLM text embedding subspaces. It reports gains on CLEVR-CoGenT, CLOSURE, GQA and TallyQA with LLaVA-1.5-7B and Qwen-VL-7B.

Critical strengths
  1. The question is interesting and the probing diagnostic is a reasonable start.
  2. The method is drop-in with small overhead.
Critical weaknesses
  1. The comparison is not fair. SRP uses scene graph boxes for role masks plus a Stage 1 pretraining on about 150K images, while the MLP, MoE and Q-Former baselines do not get this supervision. So it is unclear how much of the gain comes from the typed heads and how much from the extra annotation. A baseline with the same scene graph supervision is missing.
Questions for authors
  1. How does an MLP or MoE projector do if it gets the same scene graph supervision and Stage 1 data?
  2. Where is the Honeybee numbers on CoGenT and CLOSURE come from?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.