acceptodds
Under review as a conference paper at ICLR 2027

Gradient Invasion and Alignment Tax in Unified Autoregressive Multimodal Models

Abstract

Unified autoregressive multimodal models train a single transformer and a single softmax over a mixed vocabulary of text subwords and visual codes. Prior theory of modality competition was developed for late fusion networks with separate encoders, and it does not determine the share when the bottleneck is one shared partition function. In this paper, we establish an exact last layer decomposition of the squared hidden gradient into a diagonal term and a cross term. On trained models the cross term is negligible, and the diagonal term therefore carries the share. That term is a Rayleigh quotient and yields a gradient share law in the visual token fraction, the residual energies, and an effective embedding scale. The operator norm provides a scale bound on this Rayleigh factor, and that bound does not replace the factor. Mixing a second discrete code stream into the same softmax raises text cross entropy, which we identify as a foreign code tax. A second text stream at the same mix ratio leaves text cross entropy essentially unchanged, while real visual codes and unstructured foreign codes raise it. A paired class caption does not raise it, so the excess is occupancy of the partition function by a foreign code. End layer isolation and inverse entropy reweighting act on different coordinates of an invasion plane, and the two interventions therefore cannot substitute for one another. The same measurements on public unified checkpoints show that gradient share is present for a joint softmax, a dual head, and a continuous visual head, while the scale channel exists only for a discrete shared softmax.

Primary area:
generative models
Keywords:
Modality competition, Alignment tax, Multimodal autoregressive models, Last layer decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ekPPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies unified autoregressive multimodal models that consist of a single transformer that processes discrete image and text tokens. The abstract and introduction do not succeed in explaining the problem that is studied, what contributions are made, and what kind of experiments were conducted. The remaining sections of the paper remain unclear, and do not present meaningful research results.

Critical strengths
  • none observed
Critical weaknesses
  • Clarity. The abstract and introduction do not succeed in clearly presenting the problem that is addressed in the paper. For example, in the first few lines the notions of bottleneck, diagonal and cross terms are discussed without the necessary context. The rest of the abstract continues with unclear hard-to-follow semi-technical content that is not informative. The same holds for the introduction, related work, and other sections.
Questions for authors

At this point I do not have specific questions to the authors. The paper does unfortunately not succeed in clearly communicating research results.

Reviewer_xu82OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how text and visual tokens compete during training in unified autoregressive multimodal models. It decomposes the last-layer hidden gradient and proposes a gradient-share law based on token proportions, residual energies, and embedding geometry. It also studies the increase in text loss when visual codes are added to training, and compares end-layer isolation with loss reweighting. The experiments include small controlled transformers trained on text and visual codes, synthetic settings, and gradient measurements on public checkpoints.

Critical strengths

The paper asks a useful question: which aspects of multimodal interference come from sharing the backbone, and which come from sharing the output vocabulary. Distinguishing token frequency, gradient magnitude, and gradient direction is helpful, especially when interpreting measurements from models with different output heads.

The controlled experiments are more informative than a simple sweep over the visual-token fraction. The second-text-stream and random-code controls, together with the approximately parameter-matched isolation experiments, are useful attempts to separate possible explanations. The small measured cross terms are also an interesting empirical observation, even though I have concerns about the conclusions drawn from them.

Critical weaknesses
  1. The main gradient-share formula appears to omit a term. Removing the cross term does not remove the text-vocabulary contribution when predicting visual tokens, since both vocabularies still participate in the shared softmax. This makes the derivation of Eq. (4) unclear and raises concerns about the central theoretical result.

  2. The experiments do not clearly separate interference from the shared backbone and interference from the shared softmax. Mixing in visual or random codes also changes the difficulty and nature of the training task, so the observed increase in text loss cannot be attributed to the shared partition function alone.

Questions for authors
  1. In the derivation of Eq. (4), why can the non-target-modality diagonal term be dropped after removing the cross term? Does this require an additional assumption?

  2. Have you compared a joint softmax with a modality-masked softmax using the same data, backbone, and training budget? This would help distinguish interference from the shared normalization from interference in the backbone.

Reviewer_Tn2QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate competition between text and visual tokens in unified autoregressive multimodal models, and when both modalities share a single softmax. Specifically, authors define a "mix alignment tax" as the increase in text loss compared with text-only training. They provide theoretical analysis which characterizes the factors affecting each modality’s gradient contribution and explains why separating modality-specific end layers and adjusting loss weights address different aspects of text-visual token competition. Experiments on small trained transformers and synthetic tasks show that introducing visual codes increases text loss beyond simply reducing text training data, while end-layer isolation and loss reweighting can help mitigate this degradation. Further analysis on larger open-weight multimodal model checkpoints such as Janus-Pro and Qwen3-VL, examines how gradient magnitudes and conflicts vary across different model architectures.

Critical strengths
  1. The paper investigates a relevant problem of modality competition in unified autoregressive multimodal models. The theoretical analysis aims to characterize how token proportions, residual energies, and embedding scales affect gradient contributions, providing a potentially useful perspective on balancing text and visual learning.

  2. The experiments include useful controls to investigate the proposed mechanisms. For instance, fig. 2 compares adding text, visual codes, and random codes at the same mixture ratio to distinguish modality competition from reduced text supervision. Further, the parameter-matched isolation experiment in table 1 shows a reduction in text-loss degradation from 0.320 to 0.234, with visual cross-entropy nearly unchanged. Further gradient measurements on open-weight models are useful to contextualize/relate findings across different architectures.

Critical weaknesses
  1. The current experiments do not sufficiently isolate the contribution of the shared softmax to text degradation. The comparison between text, visual, and random code streams (fig. 2) also changes the prediction task and vocabulary. Similarly, the discrete-versus-continuous comparison in table 2 changes the visual training objective. A controlled comparison using the same visual tokens and prediction objective, but separate versus shared softmax heads, would better establish whether the observed degradation specifically comes from softmax competition.

  2. The practical applicability of reported findings needs stronger validation. Most training experiments use relatively small models (36.5M/63M parameters) and limited text/image datasets, while experiments on larger public checkpoints are primarily gradient diagnostics rather than training interventions. Further, the loss-reweighting experiments mainly report text-loss improvements without fully evaluating the corresponding impact on visual performance. Experiments on more representative multimodal training setups along with reporting both text and visual performance, would strengthen the practical conclusions/takeaways.

Questions for authors

Please see weaknesses above regarding experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.