acceptodds
Under review as a conference paper at ICLR 2027

Mambura: a Mamba-Transformer Hybrid for Unified Multimodal Understanding and Generation

Abstract

Unified models for image understanding and generation face competing task objectives and the quadratic cost of long-context attention. We introduce Mambura, a state-space model (SSM) architecture that, to our knowledge, is the first Mamba–Transformer hybrid to unify these tasks. Its core Modality-aware Mixture-of-Mamba-Experts (M³) layer combines modality-specific expert projections with a shared state-space core, selecting one expert per token within its modality's expert pool. A small number of attention layers provide global token interactions. Its principal advantage is inference efficiency. Against Qwen3.5-2B, the only baseline that still runs at a 1M-token context, Mambura prefills 2.3× faster at 128K and 15.8× faster at 1M, decodes 4.6× faster at 128K and 1.7× faster at 1M, and holds 26% lower peak memory at 1M. The full-attention unified models exhaust the device much earlier. Activating only 0.39B of its 1.7B parameters per token, Mambura nonetheless outperforms dense Transformer baselines that activate seven to eight billion: it exceeds Chameleon, UniToken and Emu3 both on GenEval, where it reaches 0.70, and on MMMU, and it outperforms Show-o on three of their four shared understanding benchmarks. These results demonstrate the potential of SSM-based hybrid architectures for efficient unified multimodal modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
unified multimodal understanding and generation, Mamba, state space models, hybrid Mamba-Transformer architecture, mixture of experts, modality-aware routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZNx8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Mambura, a unified image understanding and generation model that integrates full-attention layers with Mamba-based layers using a modality-routed state-space core. By employing modality-specific projection experts and hard content routing, the model activates 0.39B parameters per token out of 1.7B total parameters to improve long-context efficiency.

Critical strengths

1.The paper provides a concrete and functional systems demonstration of a hybrid multimodal architecture. Combining a shared recurrent core with modality-specific projections offers a practical path to scaling unified models while controlling per-token compute and cross-modal interference.

2.The study is remarkably transparent regarding model limitations and counter-intuitive findings. Disclosing cases where simpler baselines outperform the primary method and validating the training recipe on alternative backbones adds substantial credibility to the empirical findings.

Critical weaknesses

1.The proposed content-routing mechanism M$^3$ lacks strong empirical justification over simpler architectural alternatives. In controlled ablations, a simpler modality-aware fixed-expert baseline achieves better generation scores and lower training loss than M$^3$. This raises my doubts about the necessity of the added routing complexity.

2.The efficiency benchmark relies on inconsistent evaluation protocols and conflates resource throughput with functional context processing. Benchmark comparisons apply mismatched decoding steps and implementation constraints across models, while long-context tests measure speed and memory usage without proving the model actually uses distant information.

Questions for authors

1.What experimental evidence supports selecting the M$^3$ routing mechanism over the simpler fixed-expert baseline, given that the latter yields better GenEval scores and lower 100K-step training loss?

2.Could you provide a controlled benchmark comparing inference speed under identical decoding steps, input lengths, and cache-handling policies across all models?

3.Do you have empirical results demonstrating that the model effectively retrieves and leverages information from long contexts, or should the long-context claims be restricted to computational throughput?

3 comments

We thank the reviewer for the constructive review.

Weakness 1. Justification of content routing.

Under the main training recipe (λ=0) and a matched 400K-step budget, M³ improves all four understanding benchmarks over the fixed-expert arm (modality-aware only), including an 8.7-point gain on MMBench-dev, while improving GenEval from 0.671 to 0.684 (Table 4). Together with the matched inference comparison in Q1(c), these results support selecting M³ for improved downstream quality at similar decode throughput and peak memory.

Weakness 2. Efficiency protocol and long-context claims.

We agree. We re-measured all models with one protocol (Q2) and limit long-context claims to execution cost (Q3).

Q1. Evidence for selecting M³.

(a) Content routing. At 400K, POPE/MMBench-dev/AI2D/MMMU are 83.1/45.2/32.8/33.4 for M³ versus 80.6/36.5/30.4/31.9 for the fixed-expert arm, and M³ is higher on five of six GenEval categories (Overall 0.684 versus 0.671). Both runs use the same Stage I checkpoint, data, global batch 256, and warm-up followed by a constant learning rate through 400K. M³ uses λ=0; the fixed-expert arm bypasses the router and balancing loss. GenEval uses the same five sampling seeds, with CFG 5.0, temperature 1.0, top-p 0.95 and no top-k. In the earlier λ=0.01 comparison (Table 7), the fixed-expert arm leads at 30K and retains lower generation training loss through 100K, consistent with faster early learning from concentrated training signals.

(b) Modality mask. Removing the mask lowers GenEval from 0.684 to 0.625 for M³ and from 0.671 to 0.601 for the fixed-expert arm, and lowers all four understanding scores in both cases (Table 4). These comparisons favor modality-specific projections under the initialization conditions described in A.7; the fixed-expert versus neither comparison also changes the number of projection groups.

(c) Cost. M³ improves downstream quality at comparable decoding cost on the same hybrid backbone. With matched optimizations at 128K, M³ and the fixed-expert arm decode at 204.0 and 200.3 tok/s, with 8.6 GiB peak memory each. Time to first token is 1.89 versus 1.71 s. Together with Table 4, this shows improved understanding and generation quality with similar decode throughput and memory, accompanied by some additional prefill latency.

Q2. Controlled inference benchmark.

(a) The revised Figure 3 replaces Mambura's fixed-context decode-step evaluations with 49 consecutive autoregressive decoding steps, matching the baseline procedure. One H200, bf16, batch 1, CPU fixed at 2.0 GHz; mean of five trials after warm-up. Every model prefills the same 128K/512K/1M tokens in 4,096-token chunks, except OmniMamba's one-pass implementation. Each step feeds back the greedy token and advances caches and recurrent states where present. Throughput is 49 divided by the wall-clock time of these steps. Baselines keep their released kernels and attention masks: PyTorch SDPA for Show-o and Janus-Pro, flash-linear-attention and causal-conv1d for Qwen3.5-2B, and mamba-ssm for OmniMamba. Mambura's optimizations (kernel fusion, device-resident constants, routing on device, expert-sorted prefill) change how work is dispatched, not what is computed (A.6).

(b) Without our optimizations: with all of them off, against released Qwen3.5-2B, Mambura prefills 2.1/6.9/13.7 times faster at 128K/512K/1M, with 19%/25% lower peak memory at 512K/1M. Decode is 32.0/31.7/28.7 tok/s versus 31.5/31.7/31.7: similar at 128K and 512K, and approximately 10% slower at 1M. Show-o, Janus-Pro and OmniMamba do not complete at 512K or 1M.

(c) With the optimizations, Mambura decodes 6.5/5.6/3.6 times faster than Qwen3.5-2B and prefills 3.5/11.3/22.4 times faster. Figure 3 shows Mambura with and without the optimizations; peak memory varies by less than 1% across configurations.

Q3. Long-context evidence.

No; we restrict the claims to computational cost, as the reviewer suggests. Stage I uses 4,096-token sequences; Stage II has at most 1,241 tokens for understanding and 385 for generation. Figure 3 measures execution cost, not task quality; we have no retrieval or multi-image quality-versus-length results. Long-context training and quality evaluation remain future work.

The revision includes:

  1. Long-training results in Section 4.5 and Table 4: 400K GenEval: M³ 0.684, fixed expert 0.671, content routing only 0.625, neither 0.601.
  2. Common-protocol Figures 3/5, measurement details and operator ablations in Section 4.2/A.6.
  3. Long-context statements in the abstract, introduction and Section 4.2 limited to computational cost, and the planned quality evaluations.

I thank the authors for the additional experiments. However, my three main concerns are only partially addressed.

  • The new 400K-step comparison uses a different recipe (λ=0 v.s. the original λ=0.01). It doesn't show that the earlier fixed-expert advantage reverses on the same trajectory. The "five seeds" are GenEval sampling seeds, not independent training runs, so the +0.013 GenEval gain has no measure of variance. M³ also adds about 10.5% TTFT at 128K. The evidence supports a gain under one recipe, not the general necessity of routing.

  • Implementations are still not matched. Baselines keep their released kernels, and chunked prefill is applied unevenly. More importantly, with the Mambura-specific optimizations disabled, decoding is on par with Qwen3.5-2B at 128K/512K and about 10% slower at 1M. The headline speedups therefore come mainly from engineering, not architecture, and this should be stated clearly in the main text.

  • The model is trained on at most 4K tokens and evaluated with no long-context quality benchmarks. Narrowing the claim to compute cost acknowledges the issue but doesn't resolve it. Running at 1M tokens is not the same as using 1M tokens, and this limits the practical significance of the efficiency results.

Overall, the rebuttal narrows the claims rather than providing the missing evidence. I maintain my score.

We thank the reviewer for the follow-up. We agree with most of it and clarify one point.

1. Routing evidence. We agree on the scope. The 400K comparison uses the final recipe (λ=0) and is not a continuation of the earlier λ=0.01 comparison; A.7 states that the two tables are different runs and not points on one training curve. We do not claim that the earlier ordering reverses on that trajectory, nor that content routing is necessary in general. Our claim is that, under the recipe of the reported model and a matched 400K budget, M³ gives higher downstream quality than one fixed expert per modality. That claim rests mainly on understanding: POPE, MMBench-dev, AI2D and MMMU rise by 2.5, 8.7, 2.4 and 1.5 points. The five seeds are GenEval sampling seeds, not independent training runs. Each arm is a single training run, so training-run variability has not been estimated. The added time to first token at 128K (1.89 versus 1.71 s) is the cost we report for this gain.

2. Source of the speedups. For decoding we agree: the 6.5/5.6/3.6 times decode speedup comes from the operator optimizations. Section 4.2 already states that in eager mode Mambura matches the decode throughput of Qwen3.5-2B at 128K (32 tok/s each), Figure 3 draws both settings at every length, and the abstract labels its numbers as operator-optimized. The prefill and memory advantages remain substantial with all Mambura-specific optimizations disabled: Mambura prefills 2.1/6.9/13.7 times faster than Qwen3.5-2B at 128K/512K/1M and holds 19%/25% lower peak memory at 512K/1M, and peak memory changes by less than 1% when the optimizations are on. On chunking, every model except OmniMamba, including Qwen3.5-2B, prefills in the same 4,096-token chunks; OmniMamba's implementation does not support multi-token continuation, so it prefills in one pass (A.6); it does not enter the comparison with Qwen3.5-2B.

3. Long-context quality. We agree. Running at 1M tokens does not show that the model uses 1M tokens. The revision says so in two places: Section 4.2 states that the sweeps "measure execution cost rather than task quality at those lengths", and the limitations in Section 5 state that the model is trained on at most 4,096 tokens, so Figures 3 and 5 measure "what the architecture costs at 128K to 1M tokens, not what it can do there". Training on long sequences and quality-versus-length evaluation remain future work. Whether a model uses a long context is largely determined by its training data and procedure; the architecture determines what that training and inference cost, and Figure 3 measures this cost. The five attention layers keep direct access to every token in the context, so retrieval is not limited to what the recurrent state retains.

Reviewer_atWgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a unified autoregressive model for image understanding and generation using a Mamba–Transformer hybrid backbone. Its M3 layers combine modality-specific projection experts with a shared recurrent core. A dual visual encoder and joint training recipe support both tasks. The model reports 0.39B activated parameters, a GenEval score of 0.70, and inference efficiency measurements up to one million tokens.

Critical strengths

Improving the inference efficiency of unified multimodal models addresses a practically meaningful problem. The reported generation quality at a small activated parameter budget is promising. The paper also clearly discloses negative ablations and implementation limitations.

Critical weaknesses
  1. The methodological novelty is limited. Mamba–Transformer hybrids, modality-specific projections, and projection-level expert routing have already been explored in Jamba/Samba, Mixture-of-Mamba, and Routing Mamba. The main increment is their integration into unified understanding and generation, with modality-constrained content routing. However, the benefit of this additional routing is not convincingly demonstrated: in Table 4, one fixed expert per modality achieves a higher GenEval score than M3 (0.412 versus 0.333), and retains lower generation training loss through 100K steps. These results leave the necessity of the central design unclear.

  2. The efficiency claims do not sufficiently distinguish inherited architectural benefits from the paper’s own contribution. Reducing attention computation through recurrent layers is an established advantage of Mamba-based hybrids; the paper does not isolate the additional efficiency benefit of M3 over a comparable hybrid backbone without its proposed routing. The measurements also mix sequential autoregressive decoding for baselines with fixed-context decode-step evaluations for Mambura, alongside custom implementation optimizations. Furthermore, the 128K to 1M experiments report runtime without corresponding task-quality results. These gaps leave the practical quality–efficiency advantage unclear and risk overstating the contribution of the proposed method.

Questions for authors
  1. Under the main training settings and a matched budget, does M3 improve downstream quality over one fixed expert per modality?

  2. What task-quality results demonstrate effective use of contexts at 128K tokens or longer?

  3. With a matched hybrid backbone, comparable inference optimizations, and the same sequential decoding protocol, what additional efficiency benefit does M3 provide?

1 comment

We thank the reviewer for the constructive review.

Weakness 1. Novelty and the benefit of routing.

Under the main training recipe (λ=0) and a matched 400K-step budget, M³ improves all four understanding benchmarks over one fixed expert per modality, including an 8.7-point gain on MMBench-dev, while improving GenEval from 0.671 to 0.684 (Table 4). POPE/MMBench-dev/AI2D/MMMU are 83.1/45.2/32.8/33.4 versus 80.6/36.5/30.4/31.9, and M³ is higher on five of six GenEval categories. Removing the modality mask lowers GenEval from 0.684 to 0.625 for M³ and from 0.671 to 0.601 for the fixed-expert arm (modality-aware only), and lowers all four understanding scores in both cases.

In the earlier λ=0.01 comparison (Table 7), the fixed-expert arm leads at 30K and retains lower generation training loss through 100K, consistent with faster early learning from concentrated training signals. The 400K comparison above uses the main λ=0 recipe.

On novelty, we agree that hybrid backbones, modality-specific projections and projection-level routing exist separately. Unlike Mixture-of-Mamba, M³ selects among several experts per modality by content; unlike Routing Mamba, a parameter-free mask confines each modality to its own pool. To our knowledge, Mambura is the first Mamba-Transformer hybrid to serve both understanding and generation, reaching GenEval 0.70 with 0.39B activated parameters per token.

Weakness 2. Efficiency attribution.

We agree that the long-context efficiency comes from the hybrid layout, not from M³ (Q3). Our claim is that this efficiency now reaches a unified understanding-and-generation model: the same-scale unified baselines Show-o and Janus-Pro exhaust the device at 512K. We re-measured all models with one sequential decoding protocol (Q3) and limit long-context claims to execution cost (Q2).

Q1. M³ versus one fixed expert per modality.

Yes, at 400K (Weakness 1). Both runs use the same Stage I checkpoint, data, global batch 256, and warm-up followed by a constant learning rate through 400K. M³ uses λ=0; the fixed-expert arm bypasses the router and balancing loss. GenEval uses the same five sampling seeds, with CFG 5.0, temperature 1.0, top-p 0.95 and no top-k.

Q2. Task quality at 128K or longer.

We have not evaluated task quality at 128K or longer. Our downstream benchmarks establish understanding and generation quality, while Figure 3 separately measures execution cost as context length increases. Stage I uses 4,096-token sequences; Stage II has at most 1,241 tokens for understanding and 385 for generation. We have no retrieval or multi-image quality-versus-length results; long-context training and quality evaluation remain future work.

Q3. Additional efficiency from M³.

(a) M³ improves downstream quality at comparable decoding cost on the same hybrid backbone. With matched optimizations at 128K, M³ and the fixed-expert arm decode at 204.0 and 200.3 tok/s, with 8.6 GiB peak memory each. Time to first token is 1.89 versus 1.71 s. Together with Table 4, this shows improved understanding and generation quality with similar decode throughput and memory, accompanied by some additional prefill latency. The long-context efficiency comes from the hybrid layout: five attention layers among 25, the other 20 with fixed-size recurrent states.

(b) One H200, bf16, batch 1, CPU fixed at 2.0 GHz; mean of five trials after warm-up. Prefill uses 4,096-token chunks, except OmniMamba's one-pass implementation. The 49 autoregressive steps feed back the greedy token and advance caches and recurrent states where present. Baselines keep their released kernels and attention masks: PyTorch SDPA for Show-o and Janus-Pro, flash-linear-attention and causal-conv1d for Qwen3.5-2B, and mamba-ssm for OmniMamba. Mambura's optimizations (kernel fusion, device-resident constants, routing on device, expert-sorted prefill) change how work is dispatched, not what is computed (A.6).

(c) Without our optimizations: with all of them off, against released Qwen3.5-2B, the closest hybrid we measure, Mambura prefills 2.1/6.9/13.7 times faster at 128K/512K/1M, with 19%/25% lower peak memory at 512K/1M. Decode is 32.0/31.7/28.7 tok/s versus 31.5/31.7/31.7: similar at 128K and 512K, and approximately 10% slower at 1M. Show-o, Janus-Pro and OmniMamba do not complete at 512K or 1M.

(d) With the optimizations, Mambura decodes 6.5/5.6/3.6 times faster than Qwen3.5-2B and prefills 3.5/11.3/22.4 times faster. Figure 3 shows Mambura with and without the optimizations; peak memory varies by less than 1% across configurations.

The revision includes:

  1. Long-training results in Section 4.5 and Table 4: 400K GenEval: M³ 0.684, fixed expert 0.671, content routing only 0.625, neither 0.601.
  2. Common-protocol Figures 3/5, measurement details and operator ablations in Section 4.2/A.6.
  3. Explicit limits on long-context claims and the planned quality evaluations.
Reviewer_XDWoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Mambura, a unified model for image understanding and autoregressive image generation with a hybrid backbone of five Transformer layers and twenty Mamba-based M³ layers. M³ shares the recurrent core while assigning modality-specific input, gate, and output projection experts; a modality mask restricts the expert pool and hard top-1 content routing selects one expert within that pool. Understanding uses both VQGAN tokens and SigLIP features, while generation predicts VQ tokens. The model is text-pretrained and then jointly trained on the two multimodal tasks without a separate frozen-backbone alignment stage. The authors report 1.7B total and 0.39B activated parameters per token, GenEval 0.70, competitive results against some earlier unified models, and substantial long-context inference speedups in the evaluated implementations. Ablations investigate modality partitioning, within-modality expert selection, SigLIP features, and load balancing.

Critical strengths
  • A useful demonstration of compact hybrid unified modeling. The model supports both understanding and image generation with a relatively small reported activated parameter count. GenEval 0.70 is close to the 0.73 reported for Janus-Pro-1B and Show-o2, and the reported implementation achieves substantial prefill and decoding speedups at long context lengths (Tables 2–3 and Figure 3). This makes the architecture a potentially useful direction for reducing the inference cost of unified models, even though its understanding quality and practical long-context utility still need stronger validation.
  • Clear reporting of limitations and informative negative results. The paper explicitly explains the lack of task-loss gradients to the router, reports that fixed modality experts outperform M³ at the evaluated ablation budget, and acknowledges initialization/balancing confounds. The shared/private forward-pass decomposition and the additional training and gradient probes make the design understandable. These disclosures are useful for assessing the evidence and studying optimization in sparse recurrent models.
Critical weaknesses
  1. The evidence does not establish the benefit of the central within-modality expert-routing design. In Table 4 (p. 9), the simpler one-fixed-expert-per-modality variant achieves GenEval 0.412 versus 0.333 for M³ and is better in all six generation categories. Appendix A.7 (p. 19) reports that this variant retains a lower generation training loss through 100K steps, but does not provide later downstream results demonstrating a reversal. Moreover, the main model uses unit-weight hard argmax without a straight-through estimator and disables load balancing, so its router receives no gradient (Section 3.2.2; Appendices A.3–A.4). This still produces content-dependent assignments as the hidden representations evolve, but the value of this particular partitioning mechanism over simpler or task-trained alternatives is unproven. The four-arm ablation uses λ=0.01 rather than the final model’s λ=0, and removing the mask also changes exposure to untrained experts and the balancing objective (Appendix A.7, pp. 17–18). The authors disclose these issues, but they leave the contribution of M³ insufficiently isolated. This matters because the paper’s main methodological addition is more elaborate than the strongest simplified variant currently tested.
  2. The principal efficiency claim is not yet connected to a convincing quality–efficiency trade-off for long-context multimodal use. Figures 3 and 5 establish runtime and memory behavior, but the paper does not report understanding, retrieval, or interleaved generation quality at those long context lengths. The timing comparison also uses optimized Mambura kernels against unmodified released baselines, and measures Mambura over 40 decode-step evaluations at a fixed context length versus 49 autoregressive steps for the baselines (Appendix A.6). These disclosed choices do not invalidate the measurements, but prevent clean attribution of the headline ratios to architecture or direct interpretation as an end-to-end serving advantage. The closest cited unified SSM baseline, OmniMamba, is absent from the quantitative comparison. Meanwhile, Table 2 shows substantial understanding gaps to Janus-Pro-1B, including MMBench 49.3 vs. 75.5 and SEED-Bench 46.2 vs. 68.3. Taken together, the results establish an interesting implementation, but do not yet show where it offers a better usable quality–cost trade-off than the closest alternatives. A common decoding protocol and quality measurements on meaningful long-context tasks would materially strengthen the central claim.
Questions for authors
  1. Core design evidence: Do you have downstream results comparing M³ with one fixed expert per modality under the final λ=0 recipe at a representative training budget? Does the larger expert pool eventually improve quality or compute efficiency? A comparison with a router that receives task-loss gradients would also clarify whether the present routing rule is beneficial.

  2. Measurement clarification: What exactly is included in a fixed-context “decode-step evaluation” for Mambura? Does it update both recurrent states and attention caches and perform token selection as in the baseline loop? Please report, if available, a common autoregressive decoding protocol and specify the attention kernels, masks, chunking, cache handling, and operator optimizations. How much of the advantage remains when implementation differences are controlled?

  3. Long-context evidence: What is the maximum training context in Stage II, and is there evidence that the model uses distant image/text information at the lengths benchmarked in Figure 3? A quality-versus-length result on retrieval or multi-image reasoning would help distinguish execution at a given length from useful modeling at that length.

  4. Closest comparisons: Can you provide a common-protocol quality and efficiency comparison with OmniMamba, or with a matched pure-Mamba variant? Appendix A.5 reports GenEval 0.713 and 0.677 for dense Qwen2.5 backbones using the same Stage II recipe; their understanding results and inference costs would also help identify the contribution of the backbone relative to the training recipe.

1 comment

We thank the reviewer for the constructive review.

Weakness 1. Within-modality routing.

Under the final λ=0 recipe and a matched 400K-step budget, M³ improves all four understanding benchmarks over one fixed expert per modality (modality-aware only), including an 8.7-point gain on MMBench-dev, while improving GenEval from 0.671 to 0.684 (Table 4). In the earlier λ=0.01 comparison (Table 7), the fixed-expert arm leads at 30K and retains lower generation training loss through 100K, consistent with faster early learning from concentrated training signals.

Weakness 2. Efficiency and quality.

We re-measured models with a common autoregressive loop (Q2), added OmniMamba (Q4), and limit long-context claims to execution cost (Q3). Mambura reaches GenEval 0.70 with 0.39B activated parameters, versus 0.73 with 1.5B for Janus-Pro-1B, while trailing on perception-heavy understanding benchmarks.

Q1. Core design evidence.

(a) Both runs use the same Stage I checkpoint, data, global batch 256, and warm-up followed by a constant learning rate through 400K. M³ uses λ=0; the fixed-expert arm bypasses the router and balancing loss. GenEval uses the same five sampling seeds, with CFG 5.0, temperature 1.0, top-p 0.95 and no top-k. POPE/MMBench-dev/AI2D/MMMU at 400K: 83.1/45.2/32.8/33.4 versus 80.6/36.5/30.4/31.9.

(b) The larger expert pool improves quality, not compute efficiency. With matched optimizations at 128K, M³ and the fixed-expert arm decode at 204.0 and 200.3 tok/s, with 8.6 GiB peak memory each; time to first token is 1.89 versus 1.71 s.

(c) We tested a router trained by the task loss through a unit-forward weight w=p/sg(p) on the selected projection outputs (A.7). At 30K under matched λ=0 settings, training CE over the final 2K steps is 1.342/1.209 (understanding) and 7.898/7.732 (generation), variant/original. The reported main model retains the original router; this comparison does not establish that task-trained routing is generally inferior.

(d) The revised Table 4 runs all four arms at λ=0 for 400K steps, so the balancing objective no longer differs between arms. Removing the mask lowers GenEval from 0.684 to 0.625 for M³ and from 0.671 to 0.601 for the fixed-expert arm, and lowers all four understanding scores in both cases. The mask comparisons also change access to experts untrained in Stage I and the group count (A.7).

Q2. Measurement clarification.

(a) One H200, bf16, batch 1, CPU fixed at 2.0 GHz; mean of five trials after warm-up. Prefill uses 4,096-token chunks, except OmniMamba's one-pass implementation. Every model, Mambura included, runs the same 49 autoregressive steps, which feed back the greedy token and advance caches and recurrent states where present. Baselines keep their released kernels and attention masks: PyTorch SDPA for Show-o and Janus-Pro, flash-linear-attention and causal-conv1d for Qwen3.5-2B, and mamba-ssm for OmniMamba. Mambura's optimizations (kernel fusion, device-resident constants, routing on device, expert-sorted prefill) change how work is dispatched, not what is computed (A.6).

(b) With Mambura-specific optimizations disabled and baseline implementations unchanged, the prefill and memory advantages remain: against Qwen3.5-2B, Mambura prefills 2.1/6.9/13.7 times faster at 128K/512K/1M, with 19%/25% lower peak memory at 512K/1M. The decode advantage does not: 32.0/31.7/28.7 tok/s versus 31.5/31.7/31.7, similar at 128K and 512K and approximately 10% slower at 1M. Show-o, Janus-Pro and OmniMamba do not complete at 512K or 1M.

(c) The decode speedup comes from the operator optimizations: with them, Mambura decodes 6.5/5.6/3.6 times faster than Qwen3.5-2B and prefills 3.5/11.3/22.4 times faster.

Q3. Long-context evidence.

Stage I uses 4,096-token sequences; Stage II has at most 1,241 tokens for understanding and 385 for generation. These sweeps measure execution cost, not task quality; we have no retrieval or multi-image quality-versus-length results. Long-context training and quality evaluation remain future work.

Q4. Closest comparisons.

(a) Against OmniMamba, Mambura reaches GenEval 0.70 versus 0.12 (our evaluation, its native sampler) and MMMU 33.7 versus 30.6, and trails on POPE, MME-Perception and GQA (published scores). Under the Figure 3 protocol at 128K with our optimizations off, it decodes at 32.0 versus 16.2 tok/s with 8.6 versus 15.6 GiB; OmniMamba prefills faster (2.0 versus 3.1 s).

(b) We have not trained a matched pure-Mamba variant.

(c) Dense Qwen2.5 controls were evaluated on GenEval only, without understanding or inference-cost measurements, so they do not isolate backbone effects from the recipe.

The revision includes the 400K four-arm results (Section 4.5, Table 4), balancing and task-gradient routing controls (A.7), common-protocol Figures 3/5 with operator ablations (Section 4.2, A.6), OmniMamba's GenEval (Table 3), and explicit limits on long-context claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.