acceptodds
Under review as a conference paper at ICLR 2027

FlexUMM: Flexible Representation Learning for Unified Multimodal Models

Abstract

Unified multimodal models (UMMs) can integrate understanding and generation within a single backbone. However, it is commonly known that these two capabilities can cause conflicting representations, leading to a central question: how representation and computation should be shared across functionalities? Existing UMMs often rely on a largely fixed structure, where different inputs and functionalities follow the same depth and parameter-sharing pattern. We argue that this assumption is too restrictive: more effective unification requires sharing the computation that supports cross-functional communication while specializing the capacity that becomes task- or input-specific. We introduce FlexUMM, a flexible UMM framework that dynamically decides when to preserve shared computation, when to route specialized capacity, and when to stop further processing. Evidence shows that this flexible allocation improves UMMs along two complementary axes. It improves accuracy by reducing harmful competition between functionalities, and it improves efficiency by avoiding unnecessary computation once an input representation is sufficiently mature. Beyond simple architectural unification from the pre-trained models, \method works as a model-agnostic, functional plug-in to make existing UMMs more unified, stronger, and efficient. Code is available at: https://anonymous.4open.science/r/flexumm.

Primary area:
generative models
Keywords:
Unified Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sC3jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FlexUMM, a parameter-efficient adaptation framework for unified multimodal models that jointly support understanding and generation. FlexUMM combines token-level capacity routing with sequence-level early exiting. At each routed FFN, a gate selects the original FFN, the FFN augmented with a low-rank residual, or suppression of the FFN update. A separate halting controller selects among predefined exit depths. The pretrained backbone remains frozen while the added adaptation modules are trained. Experiments on Bagel, Janus-Pro, and Harmon report improvements across understanding, generation, and unified evaluation benchmarks, alongside reductions in a compute proxy. Additional analyses examine routing behavior and motivate the separation of shared communication from specialized transformation capacity.

Critical strengths
  1. Jointly considering task-dependent capacity and input-dependent depth is a sensible direction for unified multimodal models. The combination addresses both the allocation of specialized transformations and the amount of computation devoted to each input.
  2. The experiments cover three backbones and multiple understanding, generation, and unified benchmarks. On Bagel, the reported gains include MMMU, MMStar, and GenEval. Improving understanding and generation simultaneously is more compelling than demonstrating gains in only one capability.
  3. The reported additional parameter count is approximately 0.381% for Bagel. The attention-versus-FFN ablations, threshold sweep, and routing analyses provide useful information about the design. The appendix also explicitly acknowledges the distinction between synthetic mechanistic evidence, real-model observations, and compute proxies.
Critical weaknesses
  1. The compute proxy does not faithfully represent the operations executed, undermining the headline efficiency claim.

Equation (3) defines the specialized route as (F(u)+R(u)), which executes the base FFN plus an additional residual. However, Appendix A.7 assigns nominal costs of 1.0, 0.75, and 0.0 to SHARE, DECOUPLE, and EXIT, respectively. The specialized route is therefore counted as cheaper than the shared route despite performing additional computation. Appendix A.4 further states that the dense base FFN is evaluated even for tokens whose FFN updates are subsequently suppressed.

These limitations are disclosed, but the conclusion nevertheless claims a 6.33% reduction in FLOPs. An allocation proxy with these weights cannot substantiate that claim. The paper should report operation-based FLOPs and measured latency, including controller and adapter overhead, and separately quantify savings from actual sequence termination. This is particularly important because the default Bagel setting has a mean exit depth of 27.841 out of 28, corresponding to only approximately 0.6% fewer executed layers before accounting for overhead.

  1. The experiments do not sufficiently isolate the benefits of adaptive routing from those of additional post-training and adapter capacity.

Bagel+SFT is a useful baseline, but its trainable parameter budget, training schedule, and data exposure relative to FlexUMM are insufficiently specified. Replacing learned routing with random routing does not establish that dynamic allocation outperforms a well-trained, parameter-matched static adapter.

Stronger controls would include fixed task-specific adapters, shared adapters, depth-only adaptation, and capacity-only adaptation under matched training budgets. The threshold sweep provides a useful full-depth comparison, but its results suggest that the incremental accuracy contribution of early exiting is modest. Moreover, the fixed-depth baselines at depths 25 and 26 are substantially shallower than the default mean depth of 27.841, making them insufficient to establish superiority at matched computation.

  1. The mechanistic explanation is supported primarily by constructed examples rather than direct measurements on the evaluated backbones.

The gradient-alignment results in Figure 1 and Appendix B.1 come from synthetic objectives that deliberately increase task-private pressure with depth. The shared-normalization experiment explicitly constructs opposing FFN objectives. These experiments demonstrate that the proposed behavior can arise under the chosen construction, but do not establish that it explains the improvements on Bagel, Janus-Pro, or Harmon.

The authors acknowledge this boundary, which is valuable. Nevertheless, some main-text statements present aligned attention gradients and increasingly conflicting FFN gradients as general findings about UMMs. The real Show-o2 analysis measures predicted token-importance overlap rather than component-gradient conflict and uses only four understanding and two generation test examples. Direct measurements on representative task data would provide substantially stronger support.

Questions for authors
  1. How are the nominal route costs in Appendix A.7 derived, and why is DECOUPLE assigned a lower cost than SHARE when it executes (F(u)+R(u))? Do these same weights enter the training regularizer?

  2. What exact training objective, adapter rank, expert count, routing width, risk-head configuration, and hyperparameters produce the main results?

  3. How much of the improvement remains with parameter-matched static adapters trained on the same data for the same number of updates? Can you provide capacity-only, depth-only, and joint variants, together with fixed-depth baselines matched to the adaptive model’s actual compute?

4 comments

1. Compute proxy (W1, Q1)

  • Scope of the reported quantity. As the reviewer notes, the paper discloses that the reported numbers are a proxy (Sec. 4.2; App. A.4, A.7). The 6.33% in the conclusion is that proxy averaged over the three backbones in Table 6. For consistency, the final version will use the single term "allocation proxy" for this quantity throughout, including the conclusion, and report the operation counts below next to it.
  • Derivation of the nominal costs (Q1). They are nominal, hand-set weights of the allocation proxy that summarize how the policy distributes tokens over routes; they are not derived from operation counts and do not assert that F(u)+Rz(u)F(u)+R_z(u) costs less than F(u)F(u). Counting operations in Eq. (3) gives SHARE = 1 and DECOUPLE = 1+ρ1+\rho with ρ=2r/(3h)=0.2252%\rho=2r/(3h)=0.2252\% (r=64r=64, FFN intermediate width h=18,944h=18{,}944), because the residual is two rank-64 projections rather than a second FFN. This is the ordering cBYPASS≤cSHARED≤cSPECIALIZEDc_{\text{BYPASS}}\le c_{\text{SHARED}}\le c_{\text{SPECIALIZED}} stated with Eq. (9).
  • Do the weights enter the training regularizer (Q1)? No. Eq. (9) gives the general form of the capacity term. In the deployed configuration its weight is λcap=0\lambda_{\text{cap}}=0, so the vector ca=[1.0,0.75,0.0]c_a=[1.0, 0.75, 0.0] for [SHARE, DECOUPLE, BYPASS] contributes nothing to the objective and appears only in the logged allocation proxy.
  • Operation counts, adapter overhead and actual termination. Table R3 reports the operation-based count including the adapter (residual) overhead, with measured peak memory.

Table R3. Operation count and peak memory on Bagel (1× H100 NVL 96GB, batch size 1, native-chat resolution; 32 samples each from MMMU / MathVista / MMStar; generation row from a 16-prompt routing audit). The operation count is (dˉ/28)(1+ρ)(\bar d/28)(1+\rho): executed decoder depth, with the rank-64 residual charged on every routed FFN and no saving credited to BYPASS. Values in parentheses use the full-benchmark mean depth. The controllers are reported by size rather than folded into the normalized count: each depth router has 1,068,547 parameters and the TAFE controller has 1,068,037 (the backbone has 14.6B), and their work scales with sequence length and the number of candidate checks rather than with a fixed per-layer term.

SettingMean depth, subset (full benchmark)Operation count, rel., subset (full benchmark)Peak memory
Bagel, full backbone28.0001.000030,176 MiB
FlexUMM, ηβ=0.80\eta_\beta=0.80, understanding28.000 (28.000)1.0023 (1.0023)31,038 MiB
FlexUMM, ηβ=0.50\eta_\beta=0.50, understanding27.917 (27.841)0.9993 (0.9966)31,038 MiB
FlexUMM, ηβ=0.30\eta_\beta=0.30, understanding26.104 (26.072)0.9344 (0.9332)31,034 MiB
FlexUMM, ηβ=0.50\eta_\beta=0.50, generation22.5680.807831,038 MiB
  • Savings from actual sequence termination. These are given by the depth column. On understanding, the default setting executes 0.57% fewer layers (as the reviewer computes) and ηβ=0.30\eta_\beta=0.30 executes 6.9% fewer; net of the residual overhead, the counts are 0.9966 and 0.9332. dˉ=27.841\bar d=27.841 is the quality-preserving default on understanding, not the limit of the method: the same checkpoint runs at dˉ=26.072\bar d=26.072 without retraining, and in a routing audit of a generation-only checkpoint the default policy exits at d=22d=22 (71.62%) or d=24d=24 (28.38%) on every one of 1,568 router calls, a mean of 22.568, i.e. 19.4% fewer decoder layers. The audit characterizes exit frequency; because it was run on a generation-only checkpoint while GenEval and WISE in Table 6 are reported for the checkpoint evaluated there, we cite the two as separate pieces of evidence rather than as a single paired measurement.
  • Measured latency. Consistent with Sec. 4.2, we make no wall-clock claim. The current implementation is a research wrapper that evaluates the dense base FFN before the action mask and has not been optimized for speed. The only timing available is end-to-end on the 32-sample subsets (1.037 s/item, 0.965 item/s, at full depth; 3.700 s/item, 0.270 item/s, at ηβ=0.50\eta_\beta=0.50); it includes model loading and scoring and was not designed as a latency benchmark, so we report what the policy executes (Table R3) rather than a speedup.

2. Adaptive routing versus post-training and adapter capacity (W2, Q3)

Table R1. Controls on Bagel. † = evaluated on a 32-sample diagnostic subset of each benchmark (indicative only); all other rows are full-benchmark results. "–" = not evaluated.

VariantToken routingDepthTrainable paramsMMMUMathVistaMMStarGenEvalWISE
Bagel–28051.971.363.2078.810.4044
Bagel + SFT (same parameter budget, data and updates; no routing, no early exit)none280.056B52.673.168.2778.030.2274
Random router (Table 3)randomadaptive0.056B52.471.064.5376.380.3849
Depth-only (halting and risk heads, no residual banks)noneadaptive0.002B52.065.668.1––
Capacity-only (ηβ=0.80\eta_\beta=0.80, dˉ=28.000\bar d=28.000; Table 2)learned280.056B53.673.170.8––
FlexUMM (joint, ηβ=0.50\eta_\beta=0.50, dˉ=27.841\bar d=27.841)learnedadaptive0.056B54.073.470.9382.030.4292

Table R2. Adaptive versus static depth at matched depth on Bagel. The fixed d=26d=26 row is reproduced from Table 3. The fixed d=27d=27 row is a backbone-only static truncation (no TAFE gates or residual banks) evaluated on the 32-sample diagnostic subset.

PolicyMean depthMMMUMMStarMathVista
Fixed d=26d=26 (Table 3)2652.167.9067.2
Adaptive, ηβ=0.30\eta_\beta=0.30 (Table 2)26.07252.971.269.1
Fixed d=27d=272750.068.168.75
Full depth, ηβ=0.80\eta_\beta=0.80 (Table 2)28.00053.670.873.1
Adaptive, ηβ=0.50\eta_\beta=0.50 (Table 2)27.84154.070.973.4
  • Specification of Bagel+SFT. 0.056B trainable parameters (the same budget as FlexUMM), 1,500 AdamW updates, batch size 1 with gradient accumulation 1, and the same post-training data, with routing and early exit disabled. It is therefore matched in parameters, data and updates. It reaches 52.6 / 73.1 / 68.27 on MMMU / MathVista / MMStar and does not improve generation (GenEval 78.03, WISE 0.2274), against 54.0 / 73.4 / 70.93 and 82.03 / 0.4292 for FlexUMM.
  • Static, parameter-matched control (Q3). Bagel+SFT is the static control under a matched budget, and the random-router row keeps the architecture and the 0.056B parameters while removing the learned policy. We refer to Bagel+SFT as a budget-matched SFT control rather than as a task-specific adapter in the architectural sense; it addresses the question of how much improvement remains when allocation is not input-dependent: part of the understanding gain, and none of the generation gain.
  • Capacity-only, depth-only and joint (Q3). Capacity-only is in the paper as ηβ=0.80\eta_\beta=0.80 (dˉ=28.000\bar d=28.000): 53.6 / 73.1 / 70.8. Joint is the default: 54.0 / 73.4 / 70.93. Depth-only (halting and risk heads, 0.002B, no residual banks) scores 50.0 / 65.6 / 28.1 on the 32-sample subset; GenEval and WISE were not evaluated for this variant. We use it only as a directional indication that a halting policy without the capacity axis does not account for the gains, and make no full-benchmark claim for it.
  • Role of early exit. We agree with the reviewer's reading that the incremental accuracy contribution of early exit is modest; this is the intended division of labor. The capacity axis provides the accuracy gain; adaptive depth lowers the executed depth without giving it back (54.0 / 73.4 / 70.9 at dˉ=27.841\bar d=27.841) and exposes an inference-time budget knob on a single checkpoint.
  • Fixed-depth baselines at matched depth (Q3). At about 26 layers, the adaptive policy (dˉ=26.072\bar d=26.072) is ahead of the static d=26d=26 truncation by +0.8 MMMU, +3.3 MMStar and +1.9 MathVista; we present this as a matched-depth comparison of the two reported policies rather than as a controlled decomposition of the depth effect. At the default operating point, dˉ=27.841\bar d=27.841 lies between the static depths 27 and 28. Against d=28d=28 (the same model with exit inactive, ηβ=0.80\eta_\beta=0.80), the adaptive policy is at or above full depth on all three benchmarks (54.0 / 70.9 / 73.4 vs. 53.6 / 70.8 / 73.1) at a lower executed depth. Against d=27d=27, the backbone-only static truncation gives 50.0 / 28.1 / 68.75 on the 32-sample subset, which is a small-sample diagnostic and not a full-benchmark estimate.

3. Mechanistic evidence (W3)

  • The main text presents the diagnostic as a hypothesis. Sec. 1 states that the pattern is observed "under the task construction described in the appendix" and that "this is a design hypothesis rather than a direct measurement of every released backbone"; App. B.1 contains an explicit evidence-boundary paragraph. We will carry the same qualifier into the contribution list and Sec. 4.7, so that the scope is stated wherever the diagnostic is referenced.
  • Real-model support comes from intervention on Bagel. Routing attention while sharing the FFN falls below the backbone on MMBench-EN, MathVista, MMStar and WISE, whereas routing the FFN while sharing attention yields the gains (Table 3). This is the behavior the hypothesis predicts, tested directly on the evaluated backbone.
  • Direct gradient measurement. App. B.1 identifies component-gradient measurement on each released backbone as a separate experiment. We do not report such cosines, and none of the paper's claims about Bagel, Janus-Pro or Harmon relies on them: those claims rest on task quality, routing behavior and the intervention above.
  • Show-o2 analysis. The unit of this analysis is the token rather than the image. Each input contributes all of its valid tokens at every one of the 28 layers and, for generation, at every one of the 49 steps, so the generation analysis alone comprises 16,464 sample–step–layer records (12×49×2812\times49\times28), each over all valid tokens; the held-out curves in Figs. 6–7 are computed on the test portion of these records and averaged over two seeds. The paper presents the result as functionality-specific token importance and states that it is "not a direct layerwise gradient-cosine measurement" (App. B.1). The conclusion we draw from it is correspondingly limited: that a fixed sharing pattern may be too restrictive.

4. Training objective and configuration (Q2)

Eq. (6) is the compact form of the objective. Written out, it consists of the expected task loss over candidate exits (Eq. 8), the router loss of Eq. (17) (which contains the depth term of Eq. 10), final-depth hidden-state distillation, and the capacity term of Eq. (9), whose weight is 0 in the deployed configuration. Candidate costs and router inputs are detached (App. A.5.1). The configuration that produces the main results is consolidated below and will be added to App. A.

ItemValue
Trained modulesTAFE routers (utility and subset heads), halting and risk heads, low-rank residual banks
Trained parameters0.056B on Bagel (+0.381%), 0.035B on Janus-Pro (+0.472%), 0.013B on Harmon (+0.837%)
Candidate depthsBagel {22, 24, 26, 28}; Janus-Pro {21, 24, 27, 30}; Harmon {22, 24, 26, 28}
Residual bank (adapter rank)Re(u)=Ue SiLU(De RMSNorm(u))R_e(u)=U_e\,\mathrm{SiLU}(D_e\,\mathrm{RMSNorm}(u)), UeU_e zero-initialized; rank r=64r=64
Expert count EE, routing width kkE=2E=2 residual banks per routed FFN, top-1 routing (k=1k=1). Table 7 is the configuration of all main results; the expert-count labels in App. C.1 will be aligned with it
Gate temperature τ\tau0.5 for soft routing in training; hard argmax at evaluation
Depth routerτdepth=1.0\tau_{\text{depth}}=1.0, ηβ=0.50\eta_\beta=0.50, earliest eligible exit at depth 22 (Bagel)
Risk headPredicts harmful-regret probability r^\hat r and uncertainty u^\hat u; exit requires r^+κu^≤τrisk\hat r+\kappa\hat u\le\tau_{\text{risk}} with κ=0.5\kappa=0.5, τrisk=0.1\tau_{\text{risk}}=0.1
Quality margin δ\delta0.02
Loss weightsλcap=0.0\lambda_{\text{cap}}=0.0, λdepth=0.01\lambda_{\text{depth}}=0.01, λtask=1.0\lambda_{\text{task}}=1.0, λent=0.005\lambda_{\text{ent}}=0.005, λsup=2.0\lambda_{\text{sup}}=2.0, λcal=0.25\lambda_{\text{cal}}=0.25, λrisk=1.0\lambda_{\text{risk}}=1.0; quality-prediction weight 1.0
Action costs cac_a in Eq. (9)[SHARE, DECOUPLE, BYPASS] = [1.0, 0.75, 0.0], with weight λcap=0\lambda_{\text{cap}}=0
OptimizerAdamW, β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, ϵ=10−15\epsilon=10^{-15}, weight decay 0
Learning rates (by optimizer group)TAFE gates and residual banks 5×10−55\times10^{-5}; adapter group 10−510^{-5}; base group 5×10−75\times10^{-7}; the depth-router group is frozen in the joint stage and trained in a separate router stage
Updates, batch size1,500 updates, packed batch size 1, gradient accumulation 1
Post-training dataThe mixture listed in Sec. 4.1, identical for FlexUMM and Bagel+SFT

Thank you again for your thoughtful and constructive feedback. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. In response to your suggestions, we have incorporated the requested analyses and further clarified our methodology. If you have any further questions or feedback, we would be grateful if you could let us know. We hope these revisions adequately address your concerns, and would greatly appreciate it if you could reconsider your evaluation in light of these updates.

Reviewer_iiyvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlexUMM, a framework that jointly learns token-level FFN capacity allocation and sequence-level early exiting within a unified multimodal backbone. The TAFE gate preserves shared attention while enabling specialization through low-rank residuals or bypassing FFN updates. The depth router selects exit points from predefined candidate depths based on input-dependent quality constraints. Experiments on Bagel, Janus-Pro, and Harmon demonstrate overall improvements across understanding, generation, unified benchmarks, and safety evaluations, with only a small increase in trainable parameters.

Critical strengths

Comprehensive and lightweight computation allocation. Token-level capacity allocation and sequence-level depth adaptation are complementary. The framework preserves shared attention while introducing only low-rank residuals, routers, and halting heads. Consistent improvements across Bagel, Janus-Pro, and Harmon suggest that the approach is not limited to a single UMM architecture.

Broad evaluation of task quality and computational efficiency. The paper reports quality–compute trade-offs under different halting thresholds, ablations against fixed-depth execution and random routing, unified evaluations on Unison and Unified-Bench, and safety assessments. These experiments provide relatively broad coverage of the proposed research problem.

Critical weaknesses

Overstated computational efficiency claims. The current token-level BYPASS mechanism does not actually skip the dense FFN computation, while the reported efficiency improvements primarily rely on a computational cost proxy. Moreover, actual early exits appear relatively infrequent. The paper lacks direct hardware-level measurements, including inference latency, throughput, peak memory consumption, and actual FLOPs, making the claimed efficiency benefits insufficiently substantiated.

Questionable conceptual significance. The central motivation of UMMs is to unify different functionalities within a shared parameter space. However, the proposed optimization strategy largely moves toward separating computational capacity across functionalities to mitigate interference. Such specialization is an intuitive and well-established approach to reducing task conflicts, making the observed improvements somewhat expected. Consequently, the methodological contribution appears incremental, and its significance beyond existing parameter-decoupling and adaptive-routing approaches remains unclear. I would welcome further clarification from the authors if there is a stronger conceptual justification or a more fundamental contribution underlying this design.

Questions for authors

Please refer to Weaknesses.

3 comments

1. Efficiency results (W1)

  • What the paper claims. The paper reports a compute proxy and distinguishes it from wall-clock speedup. "Rel. compute" is the allocation proxy defined in App. A.7. Sec. 4.2 states that the savings are "compute proxy savings rather than direct wall-clock speedups, since the current dense implementation still evaluates the base FFN", and App. A.7 states that the action weights "describe allocation, not hardware-measured FLOPs". The 6.33% in the conclusion is this proxy averaged over the three backbones in Table 6.
  • BYPASS. App. A.4 states that, in the current dense implementation, BYPASS masks the FFN update after the base FFN has been evaluated, and that "direct execution savings come from terminating the sequence before later decoder layers". Accordingly, the accounting below credits no saving to BYPASS.
  • Operation-level accounting and peak memory. By Eq. (3), the specialized route costs 1+ρ1+\rho with ρ=2r/(3h)=0.2252%\rho=2r/(3h)=0.2252\% (r=64r=64, FFN intermediate width h=18,944h=18{,}944): the residual is two rank-64 projections, so it adds 0.23% to the FFN it augments. The realized saving is the sequence-level exit. Table R3 gives the resulting counts together with measured peak memory.

Table R3. Operation count and peak memory on Bagel (1× H100 NVL 96GB, batch size 1, native-chat resolution; 32 samples each from MMMU / MathVista / MMStar; generation row from a 16-prompt routing audit). The operation count is (dˉ/28)(1+ρ)(\bar d/28)(1+\rho): executed decoder depth, with the rank-64 residual charged on every routed FFN and no saving credited to BYPASS. Values in parentheses use the full-benchmark mean depth. The controllers are reported by size rather than folded into the normalized count: each depth router has 1,068,547 parameters and the TAFE controller has 1,068,037 (the backbone has 14.6B), and their work scales with sequence length and the number of candidate checks rather than with a fixed per-layer term.

SettingMean depth, subset (full benchmark)Operation count, rel., subset (full benchmark)Peak memory
Bagel, full backbone28.0001.000030,176 MiB
FlexUMM, ηβ=0.80\eta_\beta=0.80, understanding28.000 (28.000)1.0023 (1.0023)31,038 MiB
FlexUMM, ηβ=0.50\eta_\beta=0.50, understanding27.917 (27.841)0.9993 (0.9966)31,038 MiB
FlexUMM, ηβ=0.30\eta_\beta=0.30, understanding26.104 (26.072)0.9344 (0.9332)31,034 MiB
FlexUMM, ηβ=0.50\eta_\beta=0.50, generation22.5680.8078– (not recorded in the audit)
  • Latency and throughput. Consistent with Sec. 4.2, we make no wall-clock claim. The current implementation is a research wrapper that evaluates the dense base FFN before the action mask and has not been optimized for speed. The only timing available is end-to-end on the 32-sample subsets (1.037 s/item, 0.965 item/s, at full depth; 3.700 s/item, 0.270 item/s, at ηβ=0.50\eta_\beta=0.50); it includes model loading and scoring and was not designed as a latency benchmark, so we report what the policy executes (Table R3) rather than a speedup. As App. A.7 notes, the realized latency of the low-rank route depends on sparse or fused kernel support, which is an implementation matter separate from the allocation policy.
  • Early exit is not infrequent on generation. Fig. 4b reports understanding benchmarks only. In a routing audit of a generation-only checkpoint (16 prompts, 1,568 router calls), every call exits early: 71.62% at d=22d=22 and 28.38% at d=24d=24, a mean depth of 22.568/28, i.e. 19.4% fewer decoder layers. The audit characterizes exit frequency; because it was run on a generation-only checkpoint while the GenEval and WISE results in Table 6 are reported for the checkpoint evaluated there, we cite the two as separate pieces of evidence rather than as a single paired measurement.
  • On understanding, the exit frequency is a chosen operating point. The default ηβ=0.50\eta_\beta=0.50 is the quality-preserving setting: 2.6–8.8% of inputs exit early per benchmark (Fig. 4b), and MMMU and MathVista are at their highest (Table 2). The same checkpoint, without retraining, runs at dˉ=26.072\bar d=26.072 with ηβ=0.30\eta_\beta=0.30, where MMMU (52.9) and MMStar (71.2) remain above the full Bagel backbone (51.9, 63.20) and MathVista is 69.1.
  • The main results do not rest on efficiency. FlexUMM improves understanding, generation, unification and safety metrics at no more than full-backbone depth, with +0.381% parameters and +2.9% peak memory. For consistency, the final version will use the single term "allocation proxy" for the Rel. compute quantity throughout and report the operation counts of Table R3 next to it.

2. Conceptual significance (W2)

We agree with the reviewer that the defining goal of a UMM is to serve different functionalities within a shared parameter space, and FlexUMM is designed to preserve it. We address the step that follows architectural unification.

  • From architectural to behavioral unification. Most existing UMM work establishes unification at the architectural level: one backbone, shared tokenization and shared parameters that serve both understanding and generation. Being architecturally unified, however, does not guarantee that a model behaves as a unified model. Recent analyses report that the two functionalities compete for the same representations and that gains in one can come at the expense of the other (Han et al., 2026; Wu et al., 2026c; Ye et al., 2026). FlexUMM studies the question this leaves open: given an architecturally unified backbone, how should computation be shared, specialized and stopped so that the two functionalities cooperate rather than compete? We regard this as a question about the behavior of UMMs, complementary to architecture design, and FlexUMM as a lightweight, model-agnostic way to study it on released checkpoints.

  • The shared parameter space is preserved, not divided. Attention, normalization and the base FFN are the same modules on every route; both SHARED and SPECIALIZED pass through the same F(u)F(u). Specialization is an additive rank-64 residual (+0.381% parameters on Bagel), selected per token from the hidden state rather than assigned per functionality. No pretrained block is split or duplicated, and the shared backbone continues to carry the computation for both functionalities.

  • The measured outcome is behavioral unification. If FlexUMM moved the model toward separation, cross-capability metrics would be expected to stagnate or decline. Instead they show the largest gains. On Unison (Table 4), Mut. Enh. rises from 35.87 to 46.08 (+28.5%) and G.-G. Und. from 33.68 to 37.08 (+10.1%), with Overall 55.59 → 59.83; Unified-Bench Overall rises from 83.46 to 85.05; and on Uni-SafeBench the share of outputs that are safe under both criteria increases (43.96% → 44.86%; Table 5). Understanding and generation also improve together on all three backbones (Table 6), whereas the budget-matched SFT control improves understanding while lowering generation (WISE 0.4044 → 0.2274). Joint improvement of both functionalities, rather than a trade-off between them, is the behavior a unified model is meant to exhibit.

  • The gain is not what generic specialization delivers. Three controls in the submission show that adding specialized capacity does not by itself produce this behavior:

    • Same budget and data, no routing. Bagel+SFT, with the same 0.056B trainable-parameter budget, data and number of updates, raises understanding in part (MMStar 68.27) but lowers generation (WISE 0.4044 → 0.2274). FlexUMM raises both.
    • Same parameters, random policy. Random routing falls below the Bagel backbone on MMBench-EN (74.34 vs. 82.65) and GenEval (76.38 vs. 78.81).
    • Specializing a different component. Routing attention with a shared FFN falls below the backbone on MMStar (62.30 vs. 63.20) and MMBench-EN (81.95 vs. 82.65); routing both sublayers recovers only part of the gain (MMStar 68.90 vs. 70.80).

    More specialization is therefore not monotonically better.

  • Relation to parameter-decoupling and adaptive-routing work. These lines address related but different questions. Multi-task routing learns task- or instance-level sharing while training the network, in conventional multi-task settings. UMM decoupling such as Janus fixes which components are separated as an architectural decision made before training. Adaptive-computation methods (token pruning, Mixture-of-Depths, MoR) primarily target efficiency, adapting token participation or depth but not sharing. FlexUMM instead (i) operates on top of an already unified backbone without changing its architecture, (ii) makes the share/specialize decision per token and per layer, learned from the hidden state, and (iii) couples it with an input-dependent depth policy. Bagel is the informative case: it already contains separate understanding and generation branches by design, and FlexUMM still improves it, which indicates that predefined architectural decoupling does not exhaust the room for behavioral improvement.

  • What the framework offers for further UMM research. Because the policy is learned and its decisions are logged, FlexUMM also shows how a UMM uses its shared capacity: which inputs need how much depth (Fig. 4b; App. B.2, C.2) and where specialization helps or hurts (Table 3). Since it applies without modification to backbones with different sharing structures, the same instrument can be used to study behavioral unification across UMM designs.

Thank you again for your thoughtful and constructive feedback. As the discussion period draws to a close, we would like to kindly follow up to ensure that we have fully addressed your concerns. In response to your suggestions, we have incorporated the requested analyses and further clarified our methodology. If you have any further questions or feedback, we would be grateful if you could let us know. We hope these revisions adequately address your concerns, and would greatly appreciate it if you could reconsider your evaluation in light of these updates.

Reviewer_MFATOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FlexUMM, which aims to improve multimodal understanding, generation, and computational efficiency through dynamic computation allocation. The method retains the pre-trained backbone while employing token-level routing to either select a base feed-forward network (FFN), incorporate a low-rank residual, or skip the FFN update entirely. Additionally, it implements an early-exit mechanism via a sequence-level stopping module, with training restricted solely to the newly added components. Experiments on Bagel, Janus-Pro, and Harmon show improvements in understanding and generation.

Critical strengths
  1. I find the paper’s motivation reasonable: different inputs and capabilities in unified multimodal models may benefit from different computation depths and degrees of parameter sharing.
  2. FlexUMM adopts a lightweight design that improves both understanding and generation, with consistent gains across multiple backbones. Table 1 shows that Bagel’s scores increase from 51.9 to 54.0 on MMMU, from 63.20 to 70.93 on MMStar, and from 78.81 to 82.03 on GenEval.
Critical weaknesses
  1. The independent contribution of dynamic computation allocation remains insufficiently isolated. Although the paper includes SFT, random-routing, and fixed-depth comparisons, these do not fully distinguish the benefits of dynamic allocation from those of the added low-rank modules and training. A fixed-routing adapter baseline with matched training data, training budget, and trainable parameter count would strengthen this analysis. Separately disabling dynamic routing and early exit would further clarify their individual contributions.
  2. The performance and efficiency claims contain inconsistencies. Page 6 claims the best results on all five understanding benchmarks, yet Table 1 reports a lower MathVista score for FlexUMM than for UniCoT (73.4 vs. 83.0). The performance gap between FlexUMM and UniCoT on MathVista (73.4 vs. 83.0) requires further explanation. Moreover, Eq. (3) defines DECOUPLE as adding a residual computation to the base FFN, but Appendix A.7 assigns it a lower proxy cost than SHARE (0.75 vs. 1.0). This accounting does not adequately support the claimed FLOPs reduction.
Questions for authors
  1. Please provide a fixed-routing adapter baseline with matched training data, training budget, and trainable parameter count to isolate the contribution of dynamic computation allocation.

  2. What empirical evidence supports using the per-step flow-matching loss difference between early-exit and full-depth execution to predict changes in final image quality?

  3. Please clarify the inconsistencies highlighted in the Weaknesses section regarding performance claims and computational cost accounting.

4 comments

Thank you for your careful review.

1. Isolating adaptive allocation from added capacity and training (W1, Q1)

Table R1. Controls on Bagel.

VariantToken routingDepthTrainable paramsMMMUMathVistaMMStarGenEvalWISE
Bagel–28051.971.363.2078.810.4044
Bagel + SFT (same parameter budget, data and updates; no routing, no early exit)none280.056B52.673.168.2778.030.2274
Random router (Table 3)randomadaptive0.056B52.471.064.5376.380.3849
Depth-only† (halting and risk heads, no residual banks)noneadaptive0.002B50.065.628.1––
Capacity-only (ηβ=0.80\eta_\beta=0.80, dˉ=28.000\bar d=28.000; Table 2)learned280.056B53.673.170.8––
FlexUMM (joint, ηβ=0.50\eta_\beta=0.50, dˉ=27.841\bar d=27.841)learnedadaptive0.056B54.073.470.9382.030.4292
  • Matched budget, no routing. Bagel+SFT (Table 1) is trained with the same trainable-parameter budget (0.056B), the same post-training data and the same schedule (1,500 AdamW updates, batch size 1) as FlexUMM, with token routing and early exit disabled. It therefore measures what the added capacity and training deliver without dynamic allocation: understanding improves in part (MMMU 52.6, MMStar 68.27), while generation does not (GenEval 78.03 vs. 78.81; WISE 0.2274 vs. 0.4044). Under the same budget, FlexUMM improves both (MMMU 54.0, MMStar 70.93, GenEval 82.03, WISE 0.4292). We refer to this row as a budget-matched SFT control rather than as a fixed-routing adapter in the architectural sense; it addresses the question behind the requested baseline, namely what the same budget delivers when allocation is not input-dependent.
  • Dynamic routing disabled. Two controls remove the learned token routing while keeping the trainable budget. Bagel+SFT (above) removes routing altogether. The random router keeps the modules and the 0.056B trainable parameters but replaces the learned decisions with random ones; it falls below the Bagel backbone on MMBench-EN (74.34 vs. 82.65), GenEval (76.38 vs. 78.81) and WISE (0.3849 vs. 0.4044). The added modules are therefore not sufficient by themselves; the gain depends on how the policy allocates them. As an additional indication, a depth-only variant that removes the residual banks altogether (halting and risk heads only; 0.002B) scores 50.0 / 65.6 / 28.1 on the 32-sample subset, consistent with a halting policy alone not accounting for the gains.
  • Early exit disabled. At ηβ=0.80\eta_\beta=0.80 every input runs all 28 layers (Table 2, dˉ=28.000\bar d=28.000), which leaves token routing as the only adaptive component: 53.6 MMMU / 73.1 MathVista / 70.8 MMStar, against 51.9 / 71.3 / 63.20 for Bagel. The capacity axis accounts for the accuracy gain.
  • Role of adaptive depth. Adaptive depth lowers the executed depth without giving the gain back (54.0 / 73.4 / 70.9 at dˉ=27.841\bar d=27.841) and exposes an inference-time budget knob. At about 26 layers, the adaptive policy (ηβ=0.30\eta_\beta=0.30, dˉ=26.072\bar d=26.072; Table 2) is ahead of the static d=26d=26 truncation (Table 3) by +0.8 MMMU, +3.3 MMStar and +1.9 MathVista. We present this as a matched-depth comparison of the two reported policies rather than as a controlled decomposition of the depth effect.

In short, the two axes are complementary by design: learned token routing is the source of the accuracy gain, and adaptive depth controls how much computation is spent to obtain it.

2. MathVista and the comparison set (W2, Q3)

  • Scope of the statement. The sentence on page 6 refers to the comparison with existing unified models, i.e. the released checkpoints in Table 1 evaluated under the same protocol. Within this set FlexUMM is best on all five understanding benchmarks (on MathVista, 73.4 against 71.3 for the next best, Bagel). Bagel+SFT, Bagel+UniCoT and Bagel+IRG are listed after the released models as post-training or inference-time methods applied to the same backbone, for reference. To make this unambiguous, the final version will name the comparison set in that sentence and mark the best result within each group.
  • Why UniCoT is higher on MathVista (83.0 vs. 73.4). UniCoT adds explicit multi-step chain-of-thought reasoning at inference, which targets the multi-step mathematical problems that are central to MathVista and spends additional decoding steps on them. FlexUMM is a single-pass method that changes how computation is allocated inside the backbone; it adds no reasoning steps.
  • The broader comparison favors FlexUMM. On the other four understanding benchmarks FlexUMM is ahead of UniCoT (MMMU 54.0 vs. 53.1; MMBench-CN 85.87 vs. 80.99; MMBench-EN 86.75 vs. 83.12; MMStar 70.93 vs. 70.00). UniCoT leaves generation slightly below the backbone (GenEval 77.91, WISE 0.4037), whereas FlexUMM raises both (82.03, 0.4292). The two methods act at different levels (inference procedure vs. allocation inside the backbone) and are complementary in design.

3. Cost accounting for DECOUPLE (W2, Q3)

  • What the paper reports. "Rel. compute" is the allocation proxy defined in App. A.7. Sec. 4.2 states that "we report these as compute proxy savings rather than direct wall-clock speedups", and App. A.7 states that the action weights "describe allocation, not hardware-measured FLOPs". The 6.33% in the conclusion is this proxy averaged over the three backbones in Table 6.
  • What the 0.75 weight is. The values 1.0 / 0.75 / 0.0 are nominal, hand-set weights of this allocation proxy; they summarize how the policy distributes tokens over routes. They are not operation counts and do not assert that F(u)+Rz(u)F(u)+R_z(u) costs less than F(u)F(u). They also carry zero weight in the training objective (λcap=0\lambda_{\text{cap}}=0 in the deployed configuration).
  • Operation-level cost. By Eq. (3), DECOUPLE costs 1+ρ1+\rho with ρ=2r/(3h)=0.2252%\rho=2r/(3h)=0.2252\% (r=64r=64, FFN intermediate width h=18,944h=18{,}944), because the residual is two rank-64 projections rather than a second FFN. Counting executed decoder depth and charging this residual on every routed FFN, (dˉ/28)(1+ρ)(\bar d/28)(1+\rho), the operation count relative to Bagel is 1.0023 at full depth, 0.9966 at the default ηβ=0.50\eta_\beta=0.50 (dˉ=27.841\bar d=27.841) and 0.9332 at ηβ=0.30\eta_\beta=0.30 (dˉ=26.072\bar d=26.072) on understanding, and 0.8078 at the mean generation depth of 22.568 observed in the routing audit of point 4. This count is conservative: it credits no saving to BYPASS.
  • Presentation. The final version will report these operation counts next to the proxy and use the single term "allocation proxy" for the Rel. compute quantity throughout, including the conclusion.

4. Flow-matching loss as the depth signal for generation (Q2)

  • How the signal is used. It is a training label only. Each candidate depth is decoded with the native flow head and its loss is compared with the full-depth loss on the same sample, timestep and noise; a candidate is eligible only if it is within a margin δ=0.02\delta=0.02 of full depth (Eq. 15), and the labels are detached. At inference the router reads hidden states and the timestep embedding; no loss is computed.
  • Why it is a reasonable criterion. The flow-matching loss is the objective the generation head is trained and deployed with, so a candidate that matches the full-depth loss in a paired comparison predicts a comparably accurate velocity for that step. We do not rely on this signal alone: the controller is conditioned on the generation timestep, exit additionally requires the risk upper bound r^+0.5 u^≤0.1\hat r+0.5\,\hat u\le 0.1 (Eq. 18), and the final candidate is an unconditional fallback.
  • Image-level evidence. The policy is assessed on final decoded images rather than on the per-step loss. At the benchmark level: GenEval 78.81 → 82.03 and WISE 0.4044 → 0.4292 on Bagel, with consistent gains on Janus-Pro and Harmon (Table 6). At the sample level, Fig. 5 compares outputs produced with the same checkpoint, prompt, random seed and decoding settings, varying only the available candidate depths: the full-depth baseline misses the requested spatial relation, while the adaptive configurations recover it. App. D.2 (Table 18) reports six further detector-verified paired generation wins on color, object-identity and spatial-relation prompts.
  • How often the criterion is exercised. In a routing audit of a generation-only checkpoint (16 prompts, 1,568 router calls), 71.62% of calls exit at d=22d=22 and 28.38% at d=24d=24, with none at 26 or 28 (mean depth 22.568). The criterion is thus exercised on every generation call, not on a rare tail. The audit characterizes exit frequency; because it was run on a generation-only checkpoint while the image metrics above are reported for the checkpoint evaluated in Table 1, we cite the two as separate pieces of evidence rather than as a single paired measurement.

The authors' response has addressed most of my concerns. In particular, they provided additional clarification on the training budget of the SFT baseline, clarified the scope of the MathVista comparison, and distinguished the allocation proxy metric from actual computational cost. So, I decided to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.