acceptodds
Under review as a conference paper at ICLR 2027

Compositional Generalization in Autoregressive Models via Logit Composition

Abstract

Composing autoregressive models remains a core challenge in understanding how large language models can combine behaviors or skills learned across tasks. We introduce a new and principled composition strategy for autoregressive systems, inspired by composition methods developed for diffusion models. Under a factorized-conditionals assumption, we show that the resulting composition is projective: each component model preserves control over its own designated subspace of the output distribution avoiding interference between models. This property is further preserved under smooth reparameterizations of the output space, yielding a feature-space theorem. Finally, we show that composition inherits length generalization: if the constituent autoregressive models generalize to longer sequences individually, the composed model does so on the combined task as well. These results provide a principled understanding of when model composition and merging succeed in autoregressive systems and identify conditions under which their interactions remain stable.

Primary area:
learning theory
Keywords:
Autoregressive Models, Inference time composition

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_5HHjOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the composition of autoregressive models, aiming to combine skills learned by different models into a single model. The authors draw substantial inspiration from Bradley et al. (2025), particularly their projective composition framework for diffusion models. They extend this framework to the sequential setting by introducing selective composition and adapting Bradley et al. (2025) probability-space composition principle to autoregressive models.

As in Bradley et al. (2025), the paper establishes sufficient conditions, namely factorization and joint factorizability, under which composition is provably correct. It also provides robustness results showing that, when these conditions are not fully satisfied, composition errors remain controlled under weaker assumptions. The authors connect their framework to model merging methods, empirically validate their approach, and explain some of the conditions under which these methods succeed or fail. In addition, they introduce two diagnostics that directly measure how far real fine-tuned checkpoints deviate from the factorization assumptions, demonstrating the theory’s practical value.

Taken together, these contributions provide a useful and principled basis for understanding when model composition and merging succeed in autoregressive systems, while identifying conditions under which interactions between the composed models remain stable.

Critical strengths
  1. The theoretical analysis is rigorous, and the ideas are sound, insightful, and original. The mathematics is relatively straightforward, although some parts, particularly Definitions 1 and 2, took longer to parse because I was unfamiliar with the notation.

  2. The initial letter-replacement task and trigonometry examples provide clear motivation for the framework, which demostrates the clarity of the writing.

  3. The experiments support the theoretical conclusions.

Critical weaknesses
  1. The common support assumptions seems very important, it is unclear if models without this property if things fail and how they fail. It seems like a good idea to discuss this more.
Questions for authors

None

Reviewer_XSDEOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper extends the projective composition framework of Bradley et al. (2025) from diffusion models to autoregressive models. A base model and k fine-tuned experts are composed at inference time through per-step logit arithmetic: the experts’ log-probabilities are summed, and k−1 copies of the base are subtracted. The authors prove three results: the composition is exact when each expert coincides with the base outside its own region; the result transfers to feature space only under causal (triangular) reparameterizations; and when the assumptions hold approximately, the error degrades quadratically in a defect. Experiments include a synthetic letter-replacement task and a composition of Gemma 2 2B with math and coding experts, which performs on par with weight-space merging.

Critical strengths
  1. The mathematical proofs are sound.
Critical weaknesses
  1. The paper is primarily theoretical, and its empirical support is limited. The method is an inference-time composition rule, yet the LLM evaluation uses only one base model and one pair of experts, and none of the four benchmarks requires combining the two skills, so compositional generalization is not actually tested. The results are also unconvincing: on GSM8K the composition falls 9 points below the math expert (41.2 vs. 50.2) and about 13 points below task arithmetic and TIES, and on average it offers no gain over weight-space merging (33.8 vs. 33.6 and 34.9) while requiring k+1 forward passes per step. The practical significance of the method is therefore unclear.
  2. The theoretical depth is limited. The conclusions of Theorems 1 to 3 follow almost directly from their assumptions, and Theorem 4 is a combination of standard tools.
Questions for authors

Please see 'weaknesses'.

Reviewer_2uwCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The manuscript adapts the projective composition framework to autoregressive models, and cleanly defines the problem setting and neatly presents the theoretical arguments to back the idea, albeit with the experimental results across toy settings not yielding impressive results. Sec. 2 is very well written with each theorem flowing naturally into the next and forms the core of the paper. The authors are honest about the results and position the paper as a more theoretical contribution, which in itself is novel in the context of autoregressive language models, but not a fresh idea in and of itself. The code provided seems well written and provides support for the reliability of the results.

Critical strengths

[S1]: Sec. 2 is clean: I really like the structuring of the core theory sections with the proof sketches making understanding the intuition and idea a lot easier for the reader. For e.g. for Thm. 3, the Line 327 is a great punchline, and likewise the authors do a good job of adapting the composition framework from diffusion models to autoregressive models. I consider the theoretical contributions to be non-trivial in and of themselves. I also like Sec. 2.4 as a clear way of discussing a practical experimental truth in the context of the established framework, and analyzing the implications and the limitations with neat theoretical tools.

[S2]: Transparency: I personally love the fact that the authors are blunt and honest about what works and what doesn't, when it does, and why they think it does not in certain situations. Line 425 for e.g. clearly states experimental parity, and it is refreshing to see non bench-maxxed numbers in a paper. I feel like the results in themselves shed investigative light into the patterns of how the base models and the composed ones perform on different tasks. App. G.2, and Tab. 2 in particular are fascinating to read, and lead to curious outcomes which in my opinion deserve further investigation.

Critical weaknesses

[W1]: Mismatched experimental setup: It is unclear to me how the experimental setup is designed to support the framework proposed by the manuscript. Isn't the idea to see if composed models can do better at tasks that involve multiple skill sets? Why would one expect anything more than a math-focused model to do better on GSM8K or MATH for example? Tab. 2 also clearly highlights this where the math only model does perform quite high at the top. None of the tasks defined in App. G.2.2 evaluate compositional reasoning as suggested in the trigonometry toy example. This renders the entire empirical section moot.

[W2]: Unsupported assumptions and claims: For example, in Cor. 4, Line 1372, the lower bound on oscillation being 4.1×10−34.1 \times 10^{-3} doesn't seem justified, as over a usual ~256k-token vocab, this value would be much much higher. This also implies that the defect being measurable from forward passes is not necessarily true. Also, what is unclear to me is why osc(λ1+λ2)osc(\lambda_1 + \lambda_2) is bounded by \vert \lambda_1 \vert + \vert \lambda_2 \vert$. Likewise, Def. 1 is also step-level but the regions of action are domain specific as specified in Line 169? Further, the independence conditions are never verified as mentioned in Line 1360, so why does one assume these hypothesis remain true? What is also unclear to me is why causality is necessary according to Sec. D.3; providing a non-causal mask that does not work for a specific setting as in Line 1057 is not sufficient evidence that no non-causal mask would work? In general, I feel a lot of the details for certain statements to hold is swept under the rug.

Questions for authors

[Q1]: See [W1] for details. Can you provide results on a setting that truly utilizes multiple skills that need to be composed to get a high score? This means that the base expert models should not have a significant performance by themselves on this task. This would help understand the contribution better from a practical pov.

[Q2]: See [W2] for details. Can you clean up these sections to have a more detailed backing for each of the claims and statements? Why no non-causal map? Why the lower bound on oscillation? Why do the independence conditions hold?

Comments on the paper

2 comments

New experiment: composing two skills in one prompt

We thank the reviewer for noting that, apart from letter replacement, our experiments did not require combining two skills within one generation. As promised, we have now run the experiment with two artificial languages. The revised paper will report it in full.

Task. We invent two languages, A and B, with different alphabets and conflicting grammars (e.g., A puts the verb last, B puts it first). A test text is a few English lines, each tagged [A] or [B], and the model must translate every [A] line into A and every [B] line into B. All references are generated by code, so outputs are checked exactly. Example:

Input:  [A] the strange glove does not surprise the modern brush .
        [B] the sleepy queen yawns .
Target: [A] memeri veki rorako lanato vika niri .
        [B] hygfu jy zujyfuf zydubu .

Models. All models share one prompt and come from Qwen3-4B-Instruct-2507:

Modelon [A] lineson [B] lines
Base pbp_b (fine-tuned to copy)copiescopies
Expert A (fine-tuned from pbp_b)writes Acopies
Expert B (fine-tuned from pbp_b)copieswrites B
Wanted from the compositionwrites Awrites B

No single model can solve it: each expert knows one language, the base none. This is the setting of Theorem 2. Each expert owns a region of the output (its lines) and should behave like the base elsewhere. If it does, the composition p^∝pA pB/pb\hat p \propto p_A\, p_B / p_b (Eq. 2, at every token) reduces to expert A on [A] lines and expert B on [B] lines. We call this ideal behavior the oracle switch.

Results. Exact match of the whole output on 100 test texts (2–4 alternating [A]/[B] lines, with unseen word combinations). Ranges cover two training variants × three seeds.

MethodExact match
Expert A alone, expert B alone, base0%
Product of experts pApBp_A p_B (no division by pbp_b)0–5%
Weight merging: model soup0%
Weight merging: TIES64–85%
Weight merging: task arithmetic76–89%
Experts in sequence (A, then B on A's output)95–100%
Eq. 2 (ours)93–100%
Oracle switch (ideal behavior)92–100%
Joint model trained on both languages100%

In every run, Eq. 2 is within one point of the oracle switch, i.e., it does what Theorem 2 predicts. Running the experts in sequence also works, but needs one generation per expert in a fixed order; Eq. 2 combines them in a single generation.

Why the usual fine-tuning is not enough. Eq. 2 relies on each expert agreeing with the base outside its own lines. On a [B] line, for example, Eq. 2 multiplies expert B's distribution by the ratio pA/pbp_A/p_b. If expert A predicts exactly like the base there, this ratio is 1 for every token and the output is expert B's alone; otherwise expert A pulls the output away from B.

Our first experts were fine-tuned the standard way, with cross-entropy on the target text (which includes copying the lines they do not own). Alone, each was perfect: 100% correct on its own language and 100% faithful copying of the other lines. Yet Eq. 2 composed them correctly on only 1% of texts, against 99% for the oracle switch, and about a third of the words it wrote in [A] lines were B words.

The reason is that cross-entropy only asks the expert to put its top choice on the correct token. When copying, expert B picks the same token as the base, but the rest of its distribution is left free, and fine-tuning on B can leave B words far more probable than under the base. A single model never shows this, because decoding only looks at its top choice. Eq. 2, however, multiplies entire distributions: on an [A] line, the ratio pB/pbp_B/p_b boosts exactly those B words.

Our theory measures this. The defect (Definition 6) is how far the experts' next-token distributions are from the base's outside their own lines, over the whole vocabulary rather than only the top token. It is 0 when the theory applies exactly, and by Theorem 4, Eq. 2 stays close to the oracle switch when it is small. For these experts it was about 29 nats per token, so the theory gave no guarantee, and composition failed.

The fix follows directly. Outside its own lines, we trained each expert to match the base's full next-token distribution, by minimizing the KL divergence KL(pb ∥ pi)\mathrm{KL}(p_b \,\|\, p_i) to the frozen base instead of cross-entropy on the copied token; on its own lines it keeps cross-entropy. One of our two variants also penalizes the defect directly. The defect dropped by a factor of 11 to 36, accuracy on each expert's own language stayed at 95–100%, and Eq. 2 jumped from 1% to 93–100%, matching the oracle switch.

We emphasize that KL divergence-based fine-tuning was also necessary for the other merging methods.

So the condition of our theorems is not a technicality: it decides whether composition works, is measurable from forward passes, and can be enforced cheaply in fine-tuning.

Summary of the revision

We thank all reviewers for their careful reading. We have uploaded a revised PDF. All changes are in blue, and all references (citations, cross-references) are green links, so that blue marks only revised text. We summarize the main changes below.

1. A task that needs two skills within one generation (new App. I; summary in Sec. 3) [2uwC, XSDE, 5HHj]. The submitted benchmarks never require two skills within one answer, so we built a task that does. Two invented languages have disjoint alphabets and conflicting grammars. Each test text has lines tagged [A] or [B], and the model must translate each line into its language. All models come from Qwen3-4B-Instruct: a base model that copies its input, and two experts, each translating its own language's lines and copying the rest. No single model solves the task (0%). With experts trained to match the base's full distribution outside their own lines, our composition reaches 93–100%, within one point of the oracle switch in every run (3 seeds per variant). App. I gives the task with a worked example, the train/test distributions (Table 8), all baselines (Table 9), and a variant trained without knowledge of the partner expert. This also adds a second base model and expert pair, as asked by XSDE. The link to the code for this experiment is provided in App. I.

2. We computed the defect over the full vocabulary (new App. G.3.3, Table 6; App. E.3; abstract) [2uwC]. The defect is the spread, over the whole vocabulary, of an expert's log-ratio log⁡pi−log⁡pb\log p_i - \log p_b outside its own region. Table 3 reported this log-ratio only at the generated token, a lower bound on the defect, which cannot instantiate an upper bound. We now measure the defect itself. Over all 256,000 Gemma 2 tokens it is about 10 nats per step, so the uniform bound Et2/8E_t^2/8 is vacuous at these checkpoints. The spread comes from tokens that no model would ever produce. We removed the 4.1×10−34.1\times10^{-3} instantiation, the addition of realized-token deviations (only subadditivity of the oscillation holds), and the claim that the per-token guarantee is meaningful. The quantity the theorem bounds, the exact per-step divergence between the composition and the on-domain expert, is small (0.03–0.06 nats). The variance form of Prop. 2 weights each token by how likely the composition makes it, and it tracks this divergence within a factor of two.

3. Causality (abstract, contributions, Sec. 2.3, conclusion, App. D.3) [2uwC, 5HHj]. Theorem 3 shows that composition remains projective after a change of coordinates when the map is causal, i.e., when each feature depends only on the tokens generated so far. The submission suggested that causality is necessary, but the example in D.3 only shows that an arbitrary bijection can fail. The paper now states that causality is sufficient, not necessary. The D.3 title and Remark 2 are revised, and D.3 gives a non-causal map (a permutation of positions) for which both conclusions of Thm. 3 hold. Causality remains the natural condition for autoregressive models, because it preserves every next-token conditional. The exact admissible class is stated as open.

4. Common support (sentence after Eq. 2; new App. D.5) [5HHj]. We have clarified why a common vocabulary is required and why this requirement is naturally satisfied in the settings we consider. The ratios pi/pbp_i/p_b in Eq. 2 must be well defined. A softmax gives every token positive probability, so models that share a vocabulary and tokenization, as fine-tunes of a common base do, satisfy the assumption automatically. D.5 also explains how it can fail in practice (numerical underflow, or truncating a component's distribution): a zero in the base gives a token excessive weight, a zero in an expert vetoes it, so truncation should be applied after composing.

5. Regions in Def. 1 (paragraph after Def. 1) [2uwC, 5HHj]. We have clarified that an expert's region of action consists of a set of token positions and may depend on the prompt. In the letter example, an expert's region is the set of positions holding its letter. For a single-domain prompt, the coarsest choice gives every position to that domain's expert, and the condition then says that the other experts behave like the base on that prompt.

Thank you again for the constructive discussion, which has made the paper more accurate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.