- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the composition of autoregressive models, aiming to combine skills learned by different models into a single model. The authors draw substantial inspiration from Bradley et al. (2025), particularly their projective composition framework for diffusion models. They extend this framework to the sequential setting by introducing selective composition and adapting Bradley et al. (2025) probability-space composition principle to autoregressive models.
As in Bradley et al. (2025), the paper establishes sufficient conditions, namely factorization and joint factorizability, under which composition is provably correct. It also provides robustness results showing that, when these conditions are not fully satisfied, composition errors remain controlled under weaker assumptions. The authors connect their framework to model merging methods, empirically validate their approach, and explain some of the conditions under which these methods succeed or fail. In addition, they introduce two diagnostics that directly measure how far real fine-tuned checkpoints deviate from the factorization assumptions, demonstrating the theory’s practical value.
Taken together, these contributions provide a useful and principled basis for understanding when model composition and merging succeed in autoregressive systems, while identifying conditions under which interactions between the composed models remain stable.
Critical strengths
-
The theoretical analysis is rigorous, and the ideas are sound, insightful, and original. The mathematics is relatively straightforward, although some parts, particularly Definitions 1 and 2, took longer to parse because I was unfamiliar with the notation.
-
The initial letter-replacement task and trigonometry examples provide clear motivation for the framework, which demostrates the clarity of the writing.
-
The experiments support the theoretical conclusions.
Critical weaknesses
- The common support assumptions seems very important, it is unclear if models without this property if things fail and how they fail. It seems like a good idea to discuss this more.
Questions for authors
None
New experiment: composing two skills in one prompt
We thank the reviewer for noting that, apart from letter replacement, our experiments did not require combining two skills within one generation. As promised, we have now run the experiment with two artificial languages. The revised paper will report it in full.
Task. We invent two languages, A and B, with different alphabets and conflicting grammars (e.g., A puts the verb last, B puts it first). A test text is a few English lines, each tagged [A] or [B], and the model must translate every [A] line into A and every [B] line into B. All references are generated by code, so outputs are checked exactly. Example:
Models. All models share one prompt and come from Qwen3-4B-Instruct-2507:
No single model can solve it: each expert knows one language, the base none. This is the setting of Theorem 2. Each expert owns a region of the output (its lines) and should behave like the base elsewhere. If it does, the composition p^∝pApB/pb (Eq. 2, at every token) reduces to expert A on [A] lines and expert B on [B] lines. We call this ideal behavior the oracle switch.
Results. Exact match of the whole output on 100 test texts (2–4 alternating [A]/[B] lines, with unseen word combinations). Ranges cover two training variants × three seeds.
In every run, Eq. 2 is within one point of the oracle switch, i.e., it does what Theorem 2 predicts. Running the experts in sequence also works, but needs one generation per expert in a fixed order; Eq. 2 combines them in a single generation.
Why the usual fine-tuning is not enough. Eq. 2 relies on each expert agreeing with the base outside its own lines. On a [B] line, for example, Eq. 2 multiplies expert B's distribution by the ratio pA/pb. If expert A predicts exactly like the base there, this ratio is 1 for every token and the output is expert B's alone; otherwise expert A pulls the output away from B.
Our first experts were fine-tuned the standard way, with cross-entropy on the target text (which includes copying the lines they do not own). Alone, each was perfect: 100% correct on its own language and 100% faithful copying of the other lines. Yet Eq. 2 composed them correctly on only 1% of texts, against 99% for the oracle switch, and about a third of the words it wrote in [A] lines were B words.
The reason is that cross-entropy only asks the expert to put its top choice on the correct token. When copying, expert B picks the same token as the base, but the rest of its distribution is left free, and fine-tuning on B can leave B words far more probable than under the base. A single model never shows this, because decoding only looks at its top choice. Eq. 2, however, multiplies entire distributions: on an [A] line, the ratio pB/pb boosts exactly those B words.
Our theory measures this. The defect (Definition 6) is how far the experts' next-token distributions are from the base's outside their own lines, over the whole vocabulary rather than only the top token. It is 0 when the theory applies exactly, and by Theorem 4, Eq. 2 stays close to the oracle switch when it is small. For these experts it was about 29 nats per token, so the theory gave no guarantee, and composition failed.
The fix follows directly. Outside its own lines, we trained each expert to match the base's full next-token distribution, by minimizing the KL divergence KL(pb∥pi) to the frozen base instead of cross-entropy on the copied token; on its own lines it keeps cross-entropy. One of our two variants also penalizes the defect directly. The defect dropped by a factor of 11 to 36, accuracy on each expert's own language stayed at 95–100%, and Eq. 2 jumped from 1% to 93–100%, matching the oracle switch.
We emphasize that KL divergence-based fine-tuning was also necessary for the other merging methods.
So the condition of our theorems is not a technicality: it decides whether composition works, is measurable from forward passes, and can be enforced cheaply in fine-tuning.