acceptodds
Under review as a conference paper at ICLR 2027

Sabha: Sparse Latent Coordination of Specialist Language Models

Abstract

We introduce Sabha, a modular language model that coordinates separately pre- trained domain specialists through a compact learned interface. Each specialist, or island, is a dense transformer. A router selects a subset of islands, which emit pooled hidden-state messages into a shared latent bus; a Council decoder uses these messages to generate the system’s output. The architecture separates specialist transformer blocks from the coordination interface and does not require one pre- trained island to serve as the output hub. With three 234M-parameter islands and a 25M-parameter Council, the top-2 configuration achieves aggregate multitask cross-entropy of 7.708, compared with 7.721 for a 729M dense baseline and 7.873 for our Branch-Train-Stitch implementation. Its estimated inference FLOPs per token are 74.5% of the dense baseline’s. An individual code-island update reduces system-level code loss by 0.12, with a 0.01 increase on reasoning and a 0.29 de- crease on general text. These findings support sparse whole-model coordination as a practical design point for modular language modeling. They establish neither a general advantage over dense models nor behavioral isolation under arbitrary updates: the quality margin is small, routing underuses one specialist, and the current sequential implementation is slower despite its lower estimated arithmetic cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
modular language models, mixture of experts, compute-efficient inference, latent communication, modular architecture, domain specialization, catastrophic forgetting, sparse routing

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_NtrXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a language model, Sabha, that compresses a pretrained island separately through a compact, learned interface.

Other than that, a router selects a subset of islands per query, pools their hidden states into a shared latent bus via learned queries, and a small Council decoder cross-attends to this bus to produce output.

At last, with three 234M-parameter islands and a 25M-parameter council, Sabha attains slightly better multitask cross-entropy than a comparable 729M dense baseline while estimating 25% fewer FLOPs/token.

Critical strengths

The work advances whole model modularity by introducing a compact council decoder that operates on pooled latent messages from selected, separately pretrained specialist language models, rather than relying on a privileged hub backbone.

The design cleanly separates three axes, where total capacity, per-query compute, and maintenance and studies their interactions at a concrete operating point.

At last, the message interface is a simple, bounded communication mechanism that is easy to reason about and potentially extensible.

Critical weaknesses

There seems to be some issue with the work; the router relies on mean-pooled features from the generalist first layer, introducing asymmetry and mandatory computation even when the generalist is not selected; this partially undermines the premise of symmetric, fully independent specialists.

Causality of routing features and pooled messages is asserted as a requirement but not audited; without a prefix invariance check, reported language model losses could be confounded by inadvertent future leakage through summaries.

Routing appears to underutilise the reasoning island in diagnostics, raising concerns about effective query-adaptive specialisations under the current training and load-balancing scheme.

Despite lower estimated FLOPs, the implementation is much slower wall-clock, highlighting a significant system gap.

Other than the above, no comparison to modern sparse MoE baselines at similar active compute, which is the most directly relevant efficiency-accuracy alternative.

Different sequence lengths, unclear token exposure accounting, and unequal final-stage adaptation.

At last, evaluation is limited to cross-entropy with small held-out sets and single seeds; no task accuracy or robustness metrics are reported.

Questions for authors

The things mentioned in the weaknesses raise some questions as follows:

  1. How exactly are routing summaries and message pools made causal at inference time? Please describe the caching/windowing scheme that guarantees routing features depend only on and provide a prefix-invariance audit.

  2. What are the concerted values for the number of messages and bus width used in the main results? How sensitive are results to these hyperparameters?

  3. Can you report top-1 Sabha-B quality to support the compute table, and provide a quality-compute frontier model by varying K and Council capacity?

  4. Do you have ablations isolating the contributions of the Council decoder vs logit fusion, controlling the parameter count?

  5. Can you include a sparse MoE baseline with similar active FLOPs to Sabha-B and report both loss and throughput?

  6. What is the oracle upper bound with perfect routing, and how far is the Sabha-learned router from this bound?

I would or could raise the rating if the justification seems convincing enough.

2 comments
AuthorsOpenReview ↗

Response to reviewer

Summary

I thank the reviewers for their careful scrutiny and insightful comments. Their concerns about causality, token accounting, routing collapse, and the near-unigram losses were well founded. An audit of the submitted implementation identified defects that invalidate the headline results. While the follow-up experiments provide some limited findings, they do not establish the paper's main claims. We are therefore not submitting a revised paper in this review round. The proposed approach requires further validation and additional experimental results before those claims can be defended.

We withdraw the dense-parity claim (7.708 vs 7.721), the 25% FLOPs advantage, the BTS comparison, the specialist-update isolation result, the original Sabha-A results, and the routing and training-duration diagnostics.

Responses to reviewers

Causality (R2; R3 Q1). The original router summary and message pools read future tokens. Appending a suffix changed Sabha-B logits by up to 0.51 and flipped the top-2 route on every tested prefix. The headline losses are therefore not valid causal language-model losses. The corrected interface restricts pools to positions s <= t and passes the trained 128-token suffix and batch-independence checks. All 18 receiver checkpoints fail strict numerical checks at 256 and 512 tokens at the unchanged 2e-4 tolerance. Long-context causality remains uncertified.

Losses and data provenance (R1; R2). The Council and dense model trained and were evaluated on a token-shuffled multiset whose unigram entropy is 7.700 nats, explaining the near-unigram headline losses. The specialists trained on ordered domain streams. Evaluation sampled training shards; no held-out split was found, and raw sources were not retained. The original "held-out" label was incorrect.

Accounting and fairness (R1; R3). Launch templates imply about 819M tokens per island versus 51M for the dense model; actual consumption cannot be verified because batch and rank settings were not logged. Table 1 listed available corpus tokens. No matched dense rerun was completed. The "25M Council" label also understated trainable capacity: Stage 2 trained 39.0M parameters and Stage 3 trained 222.8M. The top-four-layer unfreezing choice had no ablation.

Routing and efficiency (R1; R2; R3). Hard top-k selection passed no task gradient to the original Sabha-B router; only the balance loss trained it. We cannot separate the proposed balancing-loss explanation from this defect. The follow-up pilot uses a soft gate over all three experts, with no balance loss or hard dispatch. All experts execute. No sparse MoE baseline, trained top-1/top-K frontier, throughput result, or sparse-serving advantage is established.

Follow-up controls and results (R1; R3 Q2-Q6). Single-expert receivers used matched budgets of 5,000 updates and 1.92M targets, with three learner seeds. Expert states improved reasoning CE by 1.53 nats over an embedding-memory control, but receivers remained worse than their parent experts. Current-state-only receivers beat receivers with seven prefix pools by 0.025-0.033 nats across the three domains.

In the joint pilot, joint, lexical and private corrections each had 24,572,419 learned parameters. Joint correction reached 4.068 macro CE versus 4.071 for the adaptive gate and 4.065 for private correction without cross-expert communication. Joint correction failed the predeclared 0.02-nat improvement gate; further seeds stopped under the registered rule. This does not demonstrate a coordination benefit. A label-assisted token oracle reached 3.564 CE, but executes all parents and is not a sparse system.

Uncertainty and task ability (R1; R3). Receiver intervals use 2,000-draw document-group bootstraps for fixed fitted models, measuring evaluation uncertainty rather than expert-pretraining uncertainty. The joint pilot has one neural initialization. Zero-shot math and code were 0/32 for every tested model. Tokenizer roundtrip also broke code formatting: reference programs passed 32/32 before roundtrip and 0/32 afterward. CE improvements do not establish practical competence.

Remaining limitations

The follow-ups are small: context 128 on one Apple M3 Ultra in FP32, with fixed original experts. The 575-document diagnostic set is not certified unseen; expert pretraining exposure is unknown. The 523-document reserve remains unscored because the pilot failed its gate. Tokenizer provenance and original training counts remain incomplete. No revised manuscript or updated figure accompanies this response.

These findings do not support the submitted paper's main claims. We submit this response to correct the record and acknowledge the need for further validation and additional results (underway). Thank you!

Reviewer_NtrXOpenReview ↗

Response to Authors

Thanks for the rebuttal; I read the reviews and acknowledge that you need to work alot before this could become at the level of ICLR level. And all the things you said seem like promises that you will not do in future; it does not seem to be in this cycle of the conference. I have not seen the updated PDF from your side. As said by me and other reviewers, this work needs alot of rewriting and improvisation, which also does not guarantee it will become at the ICLR level. I will ask the authors to reconsider their venue in future before submitting incomplete or lower-quality work to the top-level venue.

Reviewer_5a6eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Sabha, a modular language model built from three separately pretrained decoder-only transformers (islands) of about 234M parameters each, specialized in general text, math, and code. For each input, a router picks two of the three islands. The hidden states of each selected island are pooled into a few message vectors, and a small 25M-parameter decoder (the Council) cross-attends to these messages to predict the output tokens. A second variant fuses the islands' logits directly. The authors compare Sabha with a 729M dense model and a Branch-Train-Stitch-style model using held-out cross-entropy.

Critical strengths
  • None
Critical weaknesses
  • The experiments do not show that the architecture works:
    • The headline gap over dense is small and not significant 0.013 on the held-out cross-entropy
    • All models are far from trained: losses of 7.7–7.9 means these models have ways to go before comparisons became useful
    • The router seems to never work as expected and does not select the Cr Island
    • The paper states that it has not checked whether the router and the pooled messages see future tokens (L193–195), so it is not confirmed that the reported losses are valid language-model losses.

This matters because the central claim, competitive quality at lower cost, rests on a tiny loss gap between barely trained models with an unverified loss.


  • The motivation and presentation are weak:
    • The paper does not explain why whole-model specialists with a small decoder should beat existing options such as MoE, and the related work lists papers with little positioning.
    • Much of the text states what the paper does not show (for example, the abstract ends with 'They establish neither a general advantage over dense models nor behavioral isolation under arbitrary updates'), which makes the actual claims hard to extract.
    • Figure 1 is a block diagram of lettered boxes that adds little beyond the text.

This matters because it makes the contribution hard to assess.

Questions for authors
  • [Suggestion] Can you report any downstream evaluation (for example code or math benchmarks) for Sabha and the dense baseline? Loss differences of this size are hard to interpret on their own.
1 comment
AuthorsOpenReview ↗

Response to reviewer

Summary

I thank the reviewers for their careful scrutiny and insightful comments. Their concerns about causality, token accounting, routing collapse, and the near-unigram losses were well founded. An audit of the submitted implementation identified defects that invalidate the headline results. While the follow-up experiments provide some limited findings, they do not establish the paper's main claims. We are therefore not submitting a revised paper in this review round. The proposed approach requires further validation and additional experimental results before those claims can be defended.

We withdraw the dense-parity claim (7.708 vs 7.721), the 25% FLOPs advantage, the BTS comparison, the specialist-update isolation result, the original Sabha-A results, and the routing and training-duration diagnostics.

Responses to reviewers

Causality (R2; R3 Q1). The original router summary and message pools read future tokens. Appending a suffix changed Sabha-B logits by up to 0.51 and flipped the top-2 route on every tested prefix. The headline losses are therefore not valid causal language-model losses. The corrected interface restricts pools to positions s <= t and passes the trained 128-token suffix and batch-independence checks. All 18 receiver checkpoints fail strict numerical checks at 256 and 512 tokens at the unchanged 2e-4 tolerance. Long-context causality remains uncertified.

Losses and data provenance (R1; R2). The Council and dense model trained and were evaluated on a token-shuffled multiset whose unigram entropy is 7.700 nats, explaining the near-unigram headline losses. The specialists trained on ordered domain streams. Evaluation sampled training shards; no held-out split was found, and raw sources were not retained. The original "held-out" label was incorrect.

Accounting and fairness (R1; R3). Launch templates imply about 819M tokens per island versus 51M for the dense model; actual consumption cannot be verified because batch and rank settings were not logged. Table 1 listed available corpus tokens. No matched dense rerun was completed. The "25M Council" label also understated trainable capacity: Stage 2 trained 39.0M parameters and Stage 3 trained 222.8M. The top-four-layer unfreezing choice had no ablation.

Routing and efficiency (R1; R2; R3). Hard top-k selection passed no task gradient to the original Sabha-B router; only the balance loss trained it. We cannot separate the proposed balancing-loss explanation from this defect. The follow-up pilot uses a soft gate over all three experts, with no balance loss or hard dispatch. All experts execute. No sparse MoE baseline, trained top-1/top-K frontier, throughput result, or sparse-serving advantage is established.

Follow-up controls and results (R1; R3 Q2-Q6). Single-expert receivers used matched budgets of 5,000 updates and 1.92M targets, with three learner seeds. Expert states improved reasoning CE by 1.53 nats over an embedding-memory control, but receivers remained worse than their parent experts. Current-state-only receivers beat receivers with seven prefix pools by 0.025-0.033 nats across the three domains.

In the joint pilot, joint, lexical and private corrections each had 24,572,419 learned parameters. Joint correction reached 4.068 macro CE versus 4.071 for the adaptive gate and 4.065 for private correction without cross-expert communication. Joint correction failed the predeclared 0.02-nat improvement gate; further seeds stopped under the registered rule. This does not demonstrate a coordination benefit. A label-assisted token oracle reached 3.564 CE, but executes all parents and is not a sparse system.

Uncertainty and task ability (R1; R3). Receiver intervals use 2,000-draw document-group bootstraps for fixed fitted models, measuring evaluation uncertainty rather than expert-pretraining uncertainty. The joint pilot has one neural initialization. Zero-shot math and code were 0/32 for every tested model. Tokenizer roundtrip also broke code formatting: reference programs passed 32/32 before roundtrip and 0/32 afterward. CE improvements do not establish practical competence.

Remaining limitations

The follow-ups are small: context 128 on one Apple M3 Ultra in FP32, with fixed original experts. The 575-document diagnostic set is not certified unseen; expert pretraining exposure is unknown. The 523-document reserve remains unscored because the pilot failed its gate. Tokenizer provenance and original training counts remain incomplete. No revised manuscript or updated figure accompanies this response.

These findings do not support the submitted paper's main claims. We submit this response to correct the record and acknowledge the need for further validation and additional results (underway). Thank you!

Reviewer_CNwWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tries to answer the question of can we reuse fully trained domain expert frozen models with auxilliary intelligence (for example in this case through a learned interface, a query level router and a 'council' decoder). The paper claims the contribution to be from architectural perspective compared to MoE or BTS or CALM type architectures with improvements in performance also also reduction of flops compared to baselines.

Critical strengths
  1. Interesting attempt to keep the existing frozen backbones together to make a composite system and the distinction between parameter local update and behavior isolation for modularity and maintenance. I liked how the authors are separating their architecture from existing BTS/CALM type methods.
  2. The broad direction is a reasonable research agenda and is to some degree encouraging reuse of existing capabilities of llms.
Critical weaknesses
  1. My main concern is that there is no key take away in this paper. Almost every result or claim is immediately followed with a disowning sentence or phrase ("not a scaling law", "not a pareto frontier". "no empirical claim of superiority" etc.) even the abstract says the findings establish neither a general advantage over dense models nor behavioral. I was not sure if this is a negative results paper, but it seems not structured as one, since there is, for example, a quality-flops tradeoff results where authors are claiming better performance compared to dense models with only 3/4th the flops compared to the dense model. The conclusion is also essentially a position instead of a conclusion and goes tangential to the main content of the paper.

  2. The numbers in the experiments section in the paper are not always interpretable. They are apples to oranges comparison. The vocab is said to be 32K, so a uniform CE would be ~10ish to start with and the reported models sits at 7.7-8. These are severely undertrained models and given some of them are trained over code or reasoning only its not even clear if they learned any language at all. No training loss curves are shared for readers to get to that understanding.

2.a. The authors said in table 1 that the domain are roughly around 1B, 500M and 200M tokens. Some napkin maths on the following details on the training tells that each island roughly consumes 50K x BS 4 x seqlen 2048 ~ 400M tokens. So table 1 is available tokens, not consumed tokens for training. So across domains, the Sabha islands plus council consumes ~2x the tokens that the dense model has been taken through since its seq len is 1024, assuming if its trained for the same number of steps ! So a possibly undertrained dense model is according to the results and training details beating the main baseline. The numbers in Table 2 are not comparable at all because their training methodology is completely different.

2.b The training details of the council model, and the dense model other than total tokens, is absent, such as what was the data mix etc. This has immense impact on the nats score authors are comparing in the paper.

2.c. It is not explained why exactly the top 4 island layers are unfrozen and why not more or less and why the council is 25M and what the 25M decoder is supposed to be learning and from what corpora.

2.d. The numbers are just point estimates without any repeated seed with uncertainty intervals, the authors already add it in their evaluation section. I am not sure then if we can interpret any of the eval numbers.

  1. There is no MoE baseline even though the authors are actually training this model from scratch. The paper positions itself as an alternative to the conditional computation and MoE specifically on the context of reuse separately trained expert language models. Since everything is trained from scratch anyway a comparison with a minimum one layer sparse MoE with a non linear router might have been a great baseline. 3.b. The authors self claim that the BTS baseline is suboptimal so utility of those comparison is not clear. Further why did the authors choose only BTS, why also not CALM, BTM etc ?

  2. The results make it clear that the central routing mechanism does not work. The reasoning island seems mostly never selected, reducing this to essentially a fixed 2 expert system and any query adaptive specialization is never demonstrated. I have concerns with the load balancing loss this way. This type of uniform load balancing enforcement is perhaps well suited for MoEs because experts are trained jointly from scratch and otherwise there would be expert collapse during training. But here, the islands are pretrained separately on fixed domains already , ideally we would want to enforce domain matched routing not a Switch style term that pushes towards uniform usage of experts. This could be acting against what the domain specialization of this paper is about, and indeed the results shows Sabha seldom selects the reasoning expert, so leading to collapse. Needs some clarity.

  3. There are inconsistencies int he notations. For example the letters in figure 1 does not match Section 3.1 where it is explained and I had a hard time mapping. For example Figure 1 uses B for query router and A for the token context while Section 3.1 uses B for the message interface A for the arbitrer, R for router (in mathcal i think) which seems to be not in the figure at all. Moreover it number of messages or bus width are never defined or motivated. Sabha -A is defined and then mostly never talked about in the rest of the paper.

  4. Some more reported numbers may have some numerical inconsistencies or atleast needs more explanation, for example it seems loss gets worse with more training, Sabha B 7.55 at 10K steps and 7.72 at 50 K steps? In table 4, code island loss is 1.84 while the systems code loss is ~8, if these are comparable evaluations, then it would appear that the coordination is massively degrading the specialist. We need more clarity on the eval protocol here.

  5. Sabha's flop advantage is analytical and does not account for or discuss mitigations of the complexities that might arise for shading such models during inference and the latency hit it would take. The wall clock time is about 4x slower than the dense baseline already, contradicting the claimed benefit.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to reviewers

Summary

I thank the reviewers for their careful scrutiny and insightful comments. Their concerns about causality, token accounting, routing collapse, and the near-unigram losses were well founded. An audit of the submitted implementation identified defects that invalidate the headline results. While the follow-up experiments provide some limited findings, they do not establish the paper's main claims. We are therefore not submitting a revised paper in this review round. The proposed approach requires further validation and additional experimental results before those claims can be defended.

We withdraw the dense-parity claim (7.708 vs 7.721), the 25% FLOPs advantage, the BTS comparison, the specialist-update isolation result, the original Sabha-A results, and the routing and training-duration diagnostics.

Responses to reviewers

Causality (R2; R3 Q1). The original router summary and message pools read future tokens. Appending a suffix changed Sabha-B logits by up to 0.51 and flipped the top-2 route on every tested prefix. The headline losses are therefore not valid causal language-model losses. The corrected interface restricts pools to positions s <= t and passes the trained 128-token suffix and batch-independence checks. All 18 receiver checkpoints fail strict numerical checks at 256 and 512 tokens at the unchanged 2e-4 tolerance. Long-context causality remains uncertified.

Losses and data provenance (R1; R2). The Council and dense model trained and were evaluated on a token-shuffled multiset whose unigram entropy is 7.700 nats, explaining the near-unigram headline losses. The specialists trained on ordered domain streams. Evaluation sampled training shards; no held-out split was found, and raw sources were not retained. The original "held-out" label was incorrect.

Accounting and fairness (R1; R3). Launch templates imply about 819M tokens per island versus 51M for the dense model; actual consumption cannot be verified because batch and rank settings were not logged. Table 1 listed available corpus tokens. No matched dense rerun was completed. The "25M Council" label also understated trainable capacity: Stage 2 trained 39.0M parameters and Stage 3 trained 222.8M. The top-four-layer unfreezing choice had no ablation.

Routing and efficiency (R1; R2; R3). Hard top-k selection passed no task gradient to the original Sabha-B router; only the balance loss trained it. We cannot separate the proposed balancing-loss explanation from this defect. The follow-up pilot uses a soft gate over all three experts, with no balance loss or hard dispatch. All experts execute. No sparse MoE baseline, trained top-1/top-K frontier, throughput result, or sparse-serving advantage is established.

Follow-up controls and results (R1; R3 Q2-Q6). Single-expert receivers used matched budgets of 5,000 updates and 1.92M targets, with three learner seeds. Expert states improved reasoning CE by 1.53 nats over an embedding-memory control, but receivers remained worse than their parent experts. Current-state-only receivers beat receivers with seven prefix pools by 0.025-0.033 nats across the three domains.

In the joint pilot, joint, lexical and private corrections each had 24,572,419 learned parameters. Joint correction reached 4.068 macro CE versus 4.071 for the adaptive gate and 4.065 for private correction without cross-expert communication. Joint correction failed the predeclared 0.02-nat improvement gate; further seeds stopped under the registered rule. This does not demonstrate a coordination benefit. A label-assisted token oracle reached 3.564 CE, but executes all parents and is not a sparse system.

Uncertainty and task ability (R1; R3). Receiver intervals use 2,000-draw document-group bootstraps for fixed fitted models, measuring evaluation uncertainty rather than expert-pretraining uncertainty. The joint pilot has one neural initialization. Zero-shot math and code were 0/32 for every tested model. Tokenizer roundtrip also broke code formatting: reference programs passed 32/32 before roundtrip and 0/32 afterward. CE improvements do not establish practical competence.

Remaining limitations

The follow-ups are small: context 128 on one Apple M3 Ultra in FP32, with fixed original experts. The 575-document diagnostic set is not certified unseen; expert pretraining exposure is unknown. The 523-document reserve remains unscored because the pilot failed its gate. Tokenizer provenance and original training counts remain incomplete. No revised manuscript or updated figure accompanies this response.

These findings do not support the submitted paper's main claims. We submit this response to correct the record and acknowledge the need for further validation and additional results (underway). Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.