- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies a feedforward composition of five heterogeneous frozen language models. Three source models produce normalized hidden-state vectors, which are linearly projected and averaged into a shared latent representation. This representation is injected into the residual streams of two downstream frozen models, whose representations are combined by a cross-attention classifier. The projection matrices and output module are trained using answer-level cross-entropy, while the language-model weights remain frozen.
With approximately 17.6 million trainable parameters, the paper reports 87.3% accuracy on ARC-Challenge, 82.8% on OpenBookQA, and 67.2% on MMLU, outperforming the reported constituent-model, majority-vote, and single-model learned-head baselines. Its additional contributions are ablations of architectural choices and analyses of gradient flow and component importance.
Critical strengths
1. The reported science-question-answering improvements are substantial enough to merit further investigation.
Sec 4.2 reports improvements of 11.4 percentage points on ARC-Challenge and 6.2 points on OpenBookQA over the best individual constituent. Improvements also remain relative to the reported single-model learned-head baselines. These results make the proposed composition worth studying, even though the current comparisons do not establish which part of the system causes the gains.
2. The ablations provide useful information about the implemented system’s dependencies and failure regime.
Sec 4.4 goes beyond an aggregate accuracy comparison. The injection-coefficient experiment identifies a sharp failure at , rather than suggesting that stronger intervention is uniformly beneficial. These observations help identify which components deserve further investigation and where the architecture is fragile. They support component dependence, although not all of the proposed explanations for that dependence.
Critical weaknesses
1. The novelty is insufficiently established relative to a directly related prior work.
The paper needs a substantive comparison with LMNet [1], which is a highly relevant but missing related work. LMNet already organizes language-model modules into a layer-wise feedforward graph with trainable dense communication and end-to-end optimization from final-task supervision. Importantly, its Section 4.2.2 evaluates training communication edges while freezing the pretrained vertex models. Its Section 6.1 also explicitly allows different pretrained models at different vertices, although the reported implementations emphasize parameter sharing. Thus, neither dense differentiable model graphs nor frozen-backbone communication learning can be treated as new at the level of the general formulation.
There are architectural differences: LMNet communicates dense sequences through stripped transformer modules, whereas this submission pools final-token representations into a shared vector, performs late residual-stream injection, and uses a fixed four-class output head. The plausible contribution is therefore this specific lightweight cross-architecture implementation and its empirical advantages, rather than the general idea of learning communication among language-model nodes. It should be compared empirically.
2. The experimental design does not yet establish the claimed source of improvement or the reliability of all headline conclusions.
The principal learned control is a classifier on the strongest single model, whereas the proposed system has access to several models and modifies downstream computation. Beating that control does not isolate communication from access to multiple representations. Similarly, comparing a task-trained graph against untrained majority voting does not establish that latent communication is superior to learned output-level combination, where additional training is introduced.
The most important missing control is a jointly trained aggregator over the same five models’ unmodified representations, using the same training data and a comparable trainable-parameter budget. A no-injection variant and a supervised output-combination baseline would further distinguish the value of residual intervention from ordinary learned aggregation. These controls could directly test the paper’s central claim.
[1] Language Model Networks: Supervision-Efficient Learning through Dense Communication. ICML 2026/arxiv: 2505.12741.
Questions for authors
Please refer to Critical Weaknesses.