- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Mambura, a unified image understanding and generation model that integrates full-attention layers with Mamba-based layers using a modality-routed state-space core. By employing modality-specific projection experts and hard content routing, the model activates 0.39B parameters per token out of 1.7B total parameters to improve long-context efficiency.
Critical strengths
1.The paper provides a concrete and functional systems demonstration of a hybrid multimodal architecture. Combining a shared recurrent core with modality-specific projections offers a practical path to scaling unified models while controlling per-token compute and cross-modal interference.
2.The study is remarkably transparent regarding model limitations and counter-intuitive findings. Disclosing cases where simpler baselines outperform the primary method and validating the training recipe on alternative backbones adds substantial credibility to the empirical findings.
Critical weaknesses
1.The proposed content-routing mechanism M$^3$ lacks strong empirical justification over simpler architectural alternatives. In controlled ablations, a simpler modality-aware fixed-expert baseline achieves better generation scores and lower training loss than M$^3$. This raises my doubts about the necessity of the added routing complexity.
2.The efficiency benchmark relies on inconsistent evaluation protocols and conflates resource throughput with functional context processing. Benchmark comparisons apply mismatched decoding steps and implementation constraints across models, while long-context tests measure speed and memory usage without proving the model actually uses distant information.
Questions for authors
1.What experimental evidence supports selecting the M$^3$ routing mechanism over the simpler fixed-expert baseline, given that the latter yields better GenEval scores and lower 100K-step training loss?
2.Could you provide a controlled benchmark comparing inference speed under identical decoding steps, input lengths, and cache-handling policies across all models?
3.Do you have empirical results demonstrating that the model effectively retrieves and leverages information from long contexts, or should the long-context claims be restricted to computational throughput?
3 comments
I thank the authors for the additional experiments. However, my three main concerns are only partially addressed.
-
The new 400K-step comparison uses a different recipe (λ=0 v.s. the original λ=0.01). It doesn't show that the earlier fixed-expert advantage reverses on the same trajectory. The "five seeds" are GenEval sampling seeds, not independent training runs, so the +0.013 GenEval gain has no measure of variance. M³ also adds about 10.5% TTFT at 128K. The evidence supports a gain under one recipe, not the general necessity of routing.
-
Implementations are still not matched. Baselines keep their released kernels, and chunked prefill is applied unevenly. More importantly, with the Mambura-specific optimizations disabled, decoding is on par with Qwen3.5-2B at 128K/512K and about 10% slower at 1M. The headline speedups therefore come mainly from engineering, not architecture, and this should be stated clearly in the main text.
-
The model is trained on at most 4K tokens and evaluated with no long-context quality benchmarks. Narrowing the claim to compute cost acknowledges the issue but doesn't resolve it. Running at 1M tokens is not the same as using 1M tokens, and this limits the practical significance of the efficiency results.
Overall, the rebuttal narrows the claims rather than providing the missing evidence. I maintain my score.
We thank the reviewer for the follow-up. We agree with most of it and clarify one point.
1. Routing evidence. We agree on the scope. The 400K comparison uses the final recipe (λ=0) and is not a continuation of the earlier λ=0.01 comparison; A.7 states that the two tables are different runs and not points on one training curve. We do not claim that the earlier ordering reverses on that trajectory, nor that content routing is necessary in general. Our claim is that, under the recipe of the reported model and a matched 400K budget, M³ gives higher downstream quality than one fixed expert per modality. That claim rests mainly on understanding: POPE, MMBench-dev, AI2D and MMMU rise by 2.5, 8.7, 2.4 and 1.5 points. The five seeds are GenEval sampling seeds, not independent training runs. Each arm is a single training run, so training-run variability has not been estimated. The added time to first token at 128K (1.89 versus 1.71 s) is the cost we report for this gain.
2. Source of the speedups. For decoding we agree: the 6.5/5.6/3.6 times decode speedup comes from the operator optimizations. Section 4.2 already states that in eager mode Mambura matches the decode throughput of Qwen3.5-2B at 128K (32 tok/s each), Figure 3 draws both settings at every length, and the abstract labels its numbers as operator-optimized. The prefill and memory advantages remain substantial with all Mambura-specific optimizations disabled: Mambura prefills 2.1/6.9/13.7 times faster than Qwen3.5-2B at 128K/512K/1M and holds 19%/25% lower peak memory at 512K/1M, and peak memory changes by less than 1% when the optimizations are on. On chunking, every model except OmniMamba, including Qwen3.5-2B, prefills in the same 4,096-token chunks; OmniMamba's implementation does not support multi-token continuation, so it prefills in one pass (A.6); it does not enter the comparison with Qwen3.5-2B.
3. Long-context quality. We agree. Running at 1M tokens does not show that the model uses 1M tokens. The revision says so in two places: Section 4.2 states that the sweeps "measure execution cost rather than task quality at those lengths", and the limitations in Section 5 state that the model is trained on at most 4,096 tokens, so Figures 3 and 5 measure "what the architecture costs at 128K to 1M tokens, not what it can do there". Training on long sequences and quality-versus-length evaluation remain future work. Whether a model uses a long context is largely determined by its training data and procedure; the architecture determines what that training and inference cost, and Figure 3 measures this cost. The five attention layers keep direct access to every token in the context, so retrieval is not limited to what the recurrent state retains.
We thank the reviewer for the constructive review.
Weakness 1. Justification of content routing.
Under the main training recipe (λ=0) and a matched 400K-step budget, M³ improves all four understanding benchmarks over the fixed-expert arm (modality-aware only), including an 8.7-point gain on MMBench-dev, while improving GenEval from 0.671 to 0.684 (Table 4). Together with the matched inference comparison in Q1(c), these results support selecting M³ for improved downstream quality at similar decode throughput and peak memory.
Weakness 2. Efficiency protocol and long-context claims.
We agree. We re-measured all models with one protocol (Q2) and limit long-context claims to execution cost (Q3).
Q1. Evidence for selecting M³.
(a) Content routing. At 400K, POPE/MMBench-dev/AI2D/MMMU are 83.1/45.2/32.8/33.4 for M³ versus 80.6/36.5/30.4/31.9 for the fixed-expert arm, and M³ is higher on five of six GenEval categories (Overall 0.684 versus 0.671). Both runs use the same Stage I checkpoint, data, global batch 256, and warm-up followed by a constant learning rate through 400K. M³ uses λ=0; the fixed-expert arm bypasses the router and balancing loss. GenEval uses the same five sampling seeds, with CFG 5.0, temperature 1.0, top-p 0.95 and no top-k. In the earlier λ=0.01 comparison (Table 7), the fixed-expert arm leads at 30K and retains lower generation training loss through 100K, consistent with faster early learning from concentrated training signals.
(b) Modality mask. Removing the mask lowers GenEval from 0.684 to 0.625 for M³ and from 0.671 to 0.601 for the fixed-expert arm, and lowers all four understanding scores in both cases (Table 4). These comparisons favor modality-specific projections under the initialization conditions described in A.7; the fixed-expert versus neither comparison also changes the number of projection groups.
(c) Cost. M³ improves downstream quality at comparable decoding cost on the same hybrid backbone. With matched optimizations at 128K, M³ and the fixed-expert arm decode at 204.0 and 200.3 tok/s, with 8.6 GiB peak memory each. Time to first token is 1.89 versus 1.71 s. Together with Table 4, this shows improved understanding and generation quality with similar decode throughput and memory, accompanied by some additional prefill latency.
Q2. Controlled inference benchmark.
(a) The revised Figure 3 replaces Mambura's fixed-context decode-step evaluations with 49 consecutive autoregressive decoding steps, matching the baseline procedure. One H200, bf16, batch 1, CPU fixed at 2.0 GHz; mean of five trials after warm-up. Every model prefills the same 128K/512K/1M tokens in 4,096-token chunks, except OmniMamba's one-pass implementation. Each step feeds back the greedy token and advances caches and recurrent states where present. Throughput is 49 divided by the wall-clock time of these steps. Baselines keep their released kernels and attention masks: PyTorch SDPA for Show-o and Janus-Pro, flash-linear-attention and causal-conv1d for Qwen3.5-2B, and mamba-ssm for OmniMamba. Mambura's optimizations (kernel fusion, device-resident constants, routing on device, expert-sorted prefill) change how work is dispatched, not what is computed (A.6).
(b) Without our optimizations: with all of them off, against released Qwen3.5-2B, Mambura prefills 2.1/6.9/13.7 times faster at 128K/512K/1M, with 19%/25% lower peak memory at 512K/1M. Decode is 32.0/31.7/28.7 tok/s versus 31.5/31.7/31.7: similar at 128K and 512K, and approximately 10% slower at 1M. Show-o, Janus-Pro and OmniMamba do not complete at 512K or 1M.
(c) With the optimizations, Mambura decodes 6.5/5.6/3.6 times faster than Qwen3.5-2B and prefills 3.5/11.3/22.4 times faster. Figure 3 shows Mambura with and without the optimizations; peak memory varies by less than 1% across configurations.
Q3. Long-context evidence.
No; we restrict the claims to computational cost, as the reviewer suggests. Stage I uses 4,096-token sequences; Stage II has at most 1,241 tokens for understanding and 385 for generation. Figure 3 measures execution cost, not task quality; we have no retrieval or multi-image quality-versus-length results. Long-context training and quality evaluation remain future work.
The revision includes: