- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces FlexUMM, a parameter-efficient adaptation framework for unified multimodal models that jointly support understanding and generation. FlexUMM combines token-level capacity routing with sequence-level early exiting. At each routed FFN, a gate selects the original FFN, the FFN augmented with a low-rank residual, or suppression of the FFN update. A separate halting controller selects among predefined exit depths. The pretrained backbone remains frozen while the added adaptation modules are trained. Experiments on Bagel, Janus-Pro, and Harmon report improvements across understanding, generation, and unified evaluation benchmarks, alongside reductions in a compute proxy. Additional analyses examine routing behavior and motivate the separation of shared communication from specialized transformation capacity.
Critical strengths
- Jointly considering task-dependent capacity and input-dependent depth is a sensible direction for unified multimodal models. The combination addresses both the allocation of specialized transformations and the amount of computation devoted to each input.
- The experiments cover three backbones and multiple understanding, generation, and unified benchmarks. On Bagel, the reported gains include MMMU, MMStar, and GenEval. Improving understanding and generation simultaneously is more compelling than demonstrating gains in only one capability.
- The reported additional parameter count is approximately 0.381% for Bagel. The attention-versus-FFN ablations, threshold sweep, and routing analyses provide useful information about the design. The appendix also explicitly acknowledges the distinction between synthetic mechanistic evidence, real-model observations, and compute proxies.
Critical weaknesses
- The compute proxy does not faithfully represent the operations executed, undermining the headline efficiency claim.
Equation (3) defines the specialized route as (F(u)+R(u)), which executes the base FFN plus an additional residual. However, Appendix A.7 assigns nominal costs of 1.0, 0.75, and 0.0 to SHARE, DECOUPLE, and EXIT, respectively. The specialized route is therefore counted as cheaper than the shared route despite performing additional computation. Appendix A.4 further states that the dense base FFN is evaluated even for tokens whose FFN updates are subsequently suppressed.
These limitations are disclosed, but the conclusion nevertheless claims a 6.33% reduction in FLOPs. An allocation proxy with these weights cannot substantiate that claim. The paper should report operation-based FLOPs and measured latency, including controller and adapter overhead, and separately quantify savings from actual sequence termination. This is particularly important because the default Bagel setting has a mean exit depth of 27.841 out of 28, corresponding to only approximately 0.6% fewer executed layers before accounting for overhead.
- The experiments do not sufficiently isolate the benefits of adaptive routing from those of additional post-training and adapter capacity.
Bagel+SFT is a useful baseline, but its trainable parameter budget, training schedule, and data exposure relative to FlexUMM are insufficiently specified. Replacing learned routing with random routing does not establish that dynamic allocation outperforms a well-trained, parameter-matched static adapter.
Stronger controls would include fixed task-specific adapters, shared adapters, depth-only adaptation, and capacity-only adaptation under matched training budgets. The threshold sweep provides a useful full-depth comparison, but its results suggest that the incremental accuracy contribution of early exiting is modest. Moreover, the fixed-depth baselines at depths 25 and 26 are substantially shallower than the default mean depth of 27.841, making them insufficient to establish superiority at matched computation.
- The mechanistic explanation is supported primarily by constructed examples rather than direct measurements on the evaluated backbones.
The gradient-alignment results in Figure 1 and Appendix B.1 come from synthetic objectives that deliberately increase task-private pressure with depth. The shared-normalization experiment explicitly constructs opposing FFN objectives. These experiments demonstrate that the proposed behavior can arise under the chosen construction, but do not establish that it explains the improvements on Bagel, Janus-Pro, or Harmon.
The authors acknowledge this boundary, which is valuable. Nevertheless, some main-text statements present aligned attention gradients and increasingly conflicting FFN gradients as general findings about UMMs. The real Show-o2 analysis measures predicted token-importance overlap rather than component-gradient conflict and uses only four understanding and two generation test examples. Direct measurements on representative task data would provide substantially stronger support.
Questions for authors
-
How are the nominal route costs in Appendix A.7 derived, and why is DECOUPLE assigned a lower cost than SHARE when it executes (F(u)+R(u))? Do these same weights enter the training regularizer?
-
What exact training objective, adapter rank, expert count, routing width, risk-head configuration, and hyperparameters produce the main results?
-
How much of the improvement remains with parameter-matched static adapters trained on the same data for the same number of updates? Can you provide capacity-only, depth-only, and joint variants, together with fixed-depth baselines matched to the adaptive model’s actual compute?
4 comments
2. Adaptive routing versus post-training and adapter capacity (W2, Q3)
Table R1. Controls on Bagel. † = evaluated on a 32-sample diagnostic subset of each benchmark (indicative only); all other rows are full-benchmark results. "–" = not evaluated.
| Variant | Token routing | Depth | Trainable params | MMMU | MathVista | MMStar | GenEval | WISE |
|---|---|---|---|---|---|---|---|---|
| Bagel | – | 28 | 0 | 51.9 | 71.3 | 63.20 | 78.81 | 0.4044 |
| Bagel + SFT (same parameter budget, data and updates; no routing, no early exit) | none | 28 | 0.056B | 52.6 | 73.1 | 68.27 | 78.03 | 0.2274 |
| Random router (Table 3) | random | adaptive | 0.056B | 52.4 | 71.0 | 64.53 | 76.38 | 0.3849 |
| Depth-only (halting and risk heads, no residual banks) | none | adaptive | 0.002B | 52.0 | 65.6 | 68.1 | – | – |
| Capacity-only (, ; Table 2) | learned | 28 | 0.056B | 53.6 | 73.1 | 70.8 | – | – |
| FlexUMM (joint, , ) | learned | adaptive | 0.056B | 54.0 | 73.4 | 70.93 | 82.03 | 0.4292 |
Table R2. Adaptive versus static depth at matched depth on Bagel. The fixed row is reproduced from Table 3. The fixed row is a backbone-only static truncation (no TAFE gates or residual banks) evaluated on the 32-sample diagnostic subset.
| Policy | Mean depth | MMMU | MMStar | MathVista |
|---|---|---|---|---|
| Fixed (Table 3) | 26 | 52.1 | 67.90 | 67.2 |
| Adaptive, (Table 2) | 26.072 | 52.9 | 71.2 | 69.1 |
| Fixed | 27 | 50.0 | 68.1 | 68.75 |
| Full depth, (Table 2) | 28.000 | 53.6 | 70.8 | 73.1 |
| Adaptive, (Table 2) | 27.841 | 54.0 | 70.9 | 73.4 |
- Specification of Bagel+SFT. 0.056B trainable parameters (the same budget as FlexUMM), 1,500 AdamW updates, batch size 1 with gradient accumulation 1, and the same post-training data, with routing and early exit disabled. It is therefore matched in parameters, data and updates. It reaches 52.6 / 73.1 / 68.27 on MMMU / MathVista / MMStar and does not improve generation (GenEval 78.03, WISE 0.2274), against 54.0 / 73.4 / 70.93 and 82.03 / 0.4292 for FlexUMM.
- Static, parameter-matched control (Q3). Bagel+SFT is the static control under a matched budget, and the random-router row keeps the architecture and the 0.056B parameters while removing the learned policy. We refer to Bagel+SFT as a budget-matched SFT control rather than as a task-specific adapter in the architectural sense; it addresses the question of how much improvement remains when allocation is not input-dependent: part of the understanding gain, and none of the generation gain.
- Capacity-only, depth-only and joint (Q3). Capacity-only is in the paper as (): 53.6 / 73.1 / 70.8. Joint is the default: 54.0 / 73.4 / 70.93. Depth-only (halting and risk heads, 0.002B, no residual banks) scores 50.0 / 65.6 / 28.1 on the 32-sample subset; GenEval and WISE were not evaluated for this variant. We use it only as a directional indication that a halting policy without the capacity axis does not account for the gains, and make no full-benchmark claim for it.
- Role of early exit. We agree with the reviewer's reading that the incremental accuracy contribution of early exit is modest; this is the intended division of labor. The capacity axis provides the accuracy gain; adaptive depth lowers the executed depth without giving it back (54.0 / 73.4 / 70.9 at ) and exposes an inference-time budget knob on a single checkpoint.
- Fixed-depth baselines at matched depth (Q3). At about 26 layers, the adaptive policy () is ahead of the static truncation by +0.8 MMMU, +3.3 MMStar and +1.9 MathVista; we present this as a matched-depth comparison of the two reported policies rather than as a controlled decomposition of the depth effect. At the default operating point, lies between the static depths 27 and 28. Against (the same model with exit inactive, ), the adaptive policy is at or above full depth on all three benchmarks (54.0 / 70.9 / 73.4 vs. 53.6 / 70.8 / 73.1) at a lower executed depth. Against , the backbone-only static truncation gives 50.0 / 28.1 / 68.75 on the 32-sample subset, which is a small-sample diagnostic and not a full-benchmark estimate.
3. Mechanistic evidence (W3)
- The main text presents the diagnostic as a hypothesis. Sec. 1 states that the pattern is observed "under the task construction described in the appendix" and that "this is a design hypothesis rather than a direct measurement of every released backbone"; App. B.1 contains an explicit evidence-boundary paragraph. We will carry the same qualifier into the contribution list and Sec. 4.7, so that the scope is stated wherever the diagnostic is referenced.
- Real-model support comes from intervention on Bagel. Routing attention while sharing the FFN falls below the backbone on MMBench-EN, MathVista, MMStar and WISE, whereas routing the FFN while sharing attention yields the gains (Table 3). This is the behavior the hypothesis predicts, tested directly on the evaluated backbone.
- Direct gradient measurement. App. B.1 identifies component-gradient measurement on each released backbone as a separate experiment. We do not report such cosines, and none of the paper's claims about Bagel, Janus-Pro or Harmon relies on them: those claims rest on task quality, routing behavior and the intervention above.
- Show-o2 analysis. The unit of this analysis is the token rather than the image. Each input contributes all of its valid tokens at every one of the 28 layers and, for generation, at every one of the 49 steps, so the generation analysis alone comprises 16,464 sample–step–layer records (), each over all valid tokens; the held-out curves in Figs. 6–7 are computed on the test portion of these records and averaged over two seeds. The paper presents the result as functionality-specific token importance and states that it is "not a direct layerwise gradient-cosine measurement" (App. B.1). The conclusion we draw from it is correspondingly limited: that a fixed sharing pattern may be too restrictive.
4. Training objective and configuration (Q2)
Eq. (6) is the compact form of the objective. Written out, it consists of the expected task loss over candidate exits (Eq. 8), the router loss of Eq. (17) (which contains the depth term of Eq. 10), final-depth hidden-state distillation, and the capacity term of Eq. (9), whose weight is 0 in the deployed configuration. Candidate costs and router inputs are detached (App. A.5.1). The configuration that produces the main results is consolidated below and will be added to App. A.
| Item | Value |
|---|---|
| Trained modules | TAFE routers (utility and subset heads), halting and risk heads, low-rank residual banks |
| Trained parameters | 0.056B on Bagel (+0.381%), 0.035B on Janus-Pro (+0.472%), 0.013B on Harmon (+0.837%) |
| Candidate depths | Bagel {22, 24, 26, 28}; Janus-Pro {21, 24, 27, 30}; Harmon {22, 24, 26, 28} |
| Residual bank (adapter rank) | , zero-initialized; rank |
| Expert count , routing width | residual banks per routed FFN, top-1 routing (). Table 7 is the configuration of all main results; the expert-count labels in App. C.1 will be aligned with it |
| Gate temperature | 0.5 for soft routing in training; hard argmax at evaluation |
| Depth router | , , earliest eligible exit at depth 22 (Bagel) |
| Risk head | Predicts harmful-regret probability and uncertainty ; exit requires with , |
| Quality margin | 0.02 |
| Loss weights | , , , , , , ; quality-prediction weight 1.0 |
| Action costs in Eq. (9) | [SHARE, DECOUPLE, BYPASS] = [1.0, 0.75, 0.0], with weight |
| Optimizer | AdamW, , , , weight decay 0 |
| Learning rates (by optimizer group) | TAFE gates and residual banks ; adapter group ; base group ; the depth-router group is frozen in the joint stage and trained in a separate router stage |
| Updates, batch size | 1,500 updates, packed batch size 1, gradient accumulation 1 |
| Post-training data | The mixture listed in Sec. 4.1, identical for FlexUMM and Bagel+SFT |
Thank you again for your thoughtful and constructive feedback. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. In response to your suggestions, we have incorporated the requested analyses and further clarified our methodology. If you have any further questions or feedback, we would be grateful if you could let us know. We hope these revisions adequately address your concerns, and would greatly appreciate it if you could reconsider your evaluation in light of these updates.
1. Compute proxy (W1, Q1)
Table R3. Operation count and peak memory on Bagel (1× H100 NVL 96GB, batch size 1, native-chat resolution; 32 samples each from MMMU / MathVista / MMStar; generation row from a 16-prompt routing audit). The operation count is (dˉ/28)(1+ρ): executed decoder depth, with the rank-64 residual charged on every routed FFN and no saving credited to BYPASS. Values in parentheses use the full-benchmark mean depth. The controllers are reported by size rather than folded into the normalized count: each depth router has 1,068,547 parameters and the TAFE controller has 1,068,037 (the backbone has 14.6B), and their work scales with sequence length and the number of candidate checks rather than with a fixed per-layer term.