- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies inference efficiency in Unified Multimodal Models (UMMs). Its central observation is that image understanding and image generation rely on different layers despite sharing a backbone, so a single pruning path may struggle to preserve both capabilities. The proposed framework, FoldUMM, constructs task-specific graphs using differences between layer representations and applies dynamic programming to select execution paths under a given pruning budget. Local Stitch LoRA modules then recover performance at layer-skipping points. Full pruning and recovery experiments on Janus-Pro and OmniGen2 show better preservation of understanding and generation performance than the evaluated structured pruning baselines. Janus-Pro achieves end-to-end image-generation acceleration, while the gains on OmniGen2 are mainly confined to the backbone. The work offers an approach to UMM efficiency that shares parameters while adapting execution paths to individual tasks.
Critical strengths
-
Understanding and generation share a backbone, but they may not require exactly the same layers. The layer-representation analysis and cross-task path experiments provide empirical support for this observation. This perspective is useful for UMM development: parameter sharing can coexist with task-specific computation, offering a more flexible way to improve efficiency while preserving unified capabilities.
-
The framework first analyzes task-dependent layer redundancy, then searches for task-specific retained-layer paths, and finally repairs the transitions introduced by layer skipping. Each step serves a clear purpose. The introduction adequately explains why the shared backbone is an optimization target and why a single pruning path may be insufficient. Figure 4 also helps connect offline path search with online execution.
-
Beyond comparisons with existing pruning methods, the paper includes baselines with the same recovery procedure, task-specific paths, local versus global path selection, and localized versus full-layer LoRA. These controls help distinguish gains from recovery training, task adaptation, and path search. In particular, the task-specific greedy baseline provides a more direct reference for assessing the contribution of global search.
-
At 20% pruning, Janus-Pro’s image-generation time decreases from 3.94 to 3.16 seconds, while FoldUMM achieves better overall performance retention than the evaluated structured pruning methods. Localized recovery also uses substantially fewer trainable parameters than full-layer LoRA. These results demonstrate a practical performance–efficiency trade-off and suggest a feasible approach to adapting existing UMMs at relatively low training cost.
Critical weaknesses
-
The method estimates skipping costs using representation similarity in the original model and adds these costs across transitions. However, multiple skips jointly alter downstream representations, and recovery training can further affect the relative quality of different paths. Dynamic programming finds the optimal path under the defined cost, but the experiments do not fully establish whether lower search costs consistently predict smaller task-performance losses. The alternative-distance experiments provide some support, but offer limited evidence about the reliability of this proxy.
-
The complete pruning and recovery pipeline is evaluated on Janus-Pro and OmniGen2, but OmniGen2 shows relatively small cross-task path differences and no clear end-to-end generation speedup. The additional BAGEL and Emu3.5 results mainly demonstrate differences in representations or selected layers, without evaluating the complete method’s performance and efficiency. The benchmarks also assess understanding and text-to-image generation separately, rather than image editing or workflows requiring continuous cooperation between understanding and generation. The evidence therefore supports the evaluated settings more strongly than broader preservation of unified capabilities.
-
On Janus-Pro, average retention exceeds 90% for both understanding and generation, yet MM-Vet-V2 drops from 47.1 to 36.4 and GenEval from 0.79 to 0.66. These results indicate substantial degradation in some capabilities. The appendix provides a useful analysis of recovery-data composition, but more detailed results by capability are still missing. For UMMs, which aim to integrate multiple capabilities, it is important to establish which abilities are sacrificed for efficiency and whether those losses affect practical use.
-
Although OmniGen2 achieves higher backbone throughput, its complete image-generation time remains 30.10 seconds, showing that backbone compression does not always address the main bottleneck. With both paths resident, Janus-Pro’s weight memory decreases from 14.9 GB to 13.2 GB, a smaller reduction than single-path pruning achieves. The paper acknowledges these limitations, but its efficiency experiments mainly use single-request settings and provide limited direct evidence about overall throughput and latency under mixed understanding and generation workloads.
Questions for authors
-
How should the difference between understanding and generation redundancy be characterized more precisely? The main text emphasizes that generation lacks continuous redundant regions. However, Table 12 shows that Janus-Pro’s generation path removes contiguous layers at both 10% and 15% pruning, and the removals at 20% also largely concentrate in later layers. These results appear to support different preferred layer locations more directly than a general distinction between concentrated understanding redundancy and fragmented generation redundancy. Could the authors clarify how broadly this characterization holds across pruning rates and models?
-
Can the cross-task path experiments further separate the effects of layer selection and recovery training? When paths are exchanged in Table 2, are the corresponding LoRA modules reused directly or retrained for the target task? If one shared path is fixed and separately recovered using understanding and generation data, how does it compare with two independently selected paths? This comparison would help establish how much of the task conflict comes from layer selection itself and what limitations remain after sufficient adaptation of a shared path.
-
Do the gains from global search and localized recovery persist when trainable parameter counts are strictly matched? Different paths have different numbers of folding points, so using the same LoRA rank does not necessarily provide the same recovery capacity. The comparison between localized and full-layer LoRA also changes both placement and parameter count. Could the authors fix the total trainable parameter budget when comparing global search with task-specific greedy selection, and fold-point recovery with uniformly distributed recovery? This would help clarify whether the gains come from path quality, recovery placement, or adaptation capacity.
-
Can the method retain its overall benefits in tasks requiring understanding and generation to work together? The appendix discusses switching paths at stage boundaries and re-encoding or prefilling the context. Could the authors evaluate a modest image-editing or generation–understanding–revision task and report final task quality, complete execution time, and memory usage? This would more directly establish whether task-specific paths improve practical efficiency while preserving the collaborative capabilities of a UMM.
8 comments
Initial Response to Reviewer MciY: Q2 [2/5]
Q2. Shared-path comparison with task-specific recovery.
Q2: “Can the cross-task path experiments further separate the effects of layer selection and recovery training? When paths are exchanged in Table 2, are the corresponding LoRA modules reused directly or retrained for the target task? If one shared path is fixed and separately recovered using understanding and generation data, how does it compare with two independently selected paths? This comparison would help establish how much of the task conflict comes from layer selection itself and what limitations remain after sufficient adaptation of a shared path.”
In the original paper's Table 2, each transferred path underwent the full healing procedure for the target task, using the same training setup as the corresponding task-specific path. Stitch LoRA was trained for the target task, with no reuse of the source-task adapter. The comparison therefore assesses path selection under consistent target-task recovery.
To separate shared topology from task-specific recovery, we added a joint-cost shared-path baseline using the same DAG-DP search and 20% pruning budget. We construct its cost matrix as
where each is the mean positive cost over eligible non-adjacent edges for task . We evaluate , covering equal weighting and preferences toward either task. Each shared path is healed separately for IU and T2I. We align the recovery parameter budget with the task-specific counterpart by adjusting LoRA rank to account for differences in the number of fold points.
Results for the shared-path baselines are averaged over three training seeds for both IU and T2I. Mean RP averages the Dense-normalized scores across the five reported benchmarks.
Table 1: Task-specific and shared DAG paths under matched recovery budgets.
| Path configuration | MME ↑ | POPE ↑ | GQA ↑ | MMMU-Pro ↑ | GenEval ↑ | Mean RP ↑ |
|---|---|---|---|---|---|---|
| Dense | 1567 | 85.2 | 60.4 | 35.3 | 0.79 | 100.0% |
| Task-specific | 1505 | 90.2 | 56.4 | 32.8 | 0.66 | 94.4% |
| Shared, λ=0.25 | 1293 | 80.2 | 51.6 | 29.2 | 0.57 | 83.4% |
| Shared, λ=0.50 | 1281 | 82.0 | 52.9 | 31.9 | 0.51 | 84.1% |
| Shared, λ=0.75 | 1436 | 84.3 | 55.5 | 31.3 | 0.56 | 88.4% |
Task-specific paths achieve higher scores across all five reported metrics than each tested shared path. The best tested shared path remains 6.0 percentage points below the task-specific paths in Mean RP (88.4% vs. 94.4%). This remaining gap after matched task-specific healing more directly reveals the task conflict arising from layer selection, showing the limitation of using a shared path for both tasks.
Initial Response to Reviewer MciY: Q3 [3/5]
Q3. Matched recovery capacity and placement.
Q3: “Do the gains from global search and localized recovery persist when trainable parameter counts are strictly matched? Different paths have different numbers of folding points, so using the same LoRA rank does not necessarily provide the same recovery capacity. The comparison between localized and full-layer LoRA also changes both placement and parameter count. Could the authors fix the total trainable parameter budget when comparing global search with task-specific greedy selection, and fold-point recovery with uniformly distributed recovery? This would help clarify whether the gains come from path quality, recovery placement, or adaptation capacity.”
We thank Reviewer MciY for suggesting these experiments. We added two parameter-matched ablations to separate search quality from recovery placement, together with training-cost profiling. Performance results are averaged over three random seeds.
1. Search strategy.
On Janus-Pro IU at 20% pruning, DAG-DP uses four fold points at rank 128, and task-specific Greedy uses two at rank 256. Both train 39,976,960 parameters, with the same data and optimization settings and matched update counts.
Table 2: Greedy and DAG-DP search with matched recovery capacity on IU.
| Search method | MME ↑ | POPE ↑ | GQA ↑ | MMMU-Pro ↑ |
|---|---|---|---|---|
| Greedy | 1421.64 | 85.77 | 56.00 | 32.95 |
| DAG-DP | 1505.00 | 90.20 | 56.40 | 32.80 |
The T2I comparison in the original paper's Table 3 already matches the recovery parameter budget, with both methods using rank-128 Stitch LoRA at two fold points. Under this matched budget, FoldUMM achieves 0.66 on GenEval, compared with 0.62 for task-specific Greedy.
With matched recovery capacity, DAG-DP achieves higher scores on three of the four IU benchmarks, supporting the benefit of global folding-path optimization.
2. Recovery placement.
We further provide a fairer comparison of recovery placement by matching the total trainable parameter budget. We fix the FoldUMM path for each task and distribute this budget either at fold points or across all 24 retained layers.
Table 3: Matched parameter budgets for fold-point and all-layer recovery.
| Task | Placement | Locations | Rank allocation | Trainable parameters |
|---|---|---|---|---|
| IU | Fold points | 4 | 4 × 128 | 39,976,960 |
| IU | All retained layers | 24 | 16 × 21 + 8 × 22 | 39,976,960 |
| T2I | Fold points | 2 | 2 × 128 | 19,988,480 |
| T2I | All retained layers | 24 | 8 × 10 + 16 × 11 | 19,988,480 |
Table 4: Performance of fold-point and all-layer recovery under matched budgets.
| Recovery placement | MME ↑ | POPE ↑ | GQA ↑ | MMMU-Pro ↑ | GenEval ↑ |
|---|---|---|---|---|---|
| Fold points | 1505.00 | 90.20 | 56.40 | 32.80 | 0.659 |
| All retained layers | 1507.09 | 88.19 | 57.12 | 33.01 | 0.625 |
Fold-point LoRA achieves comparable IU performance to all-layer recovery and a GenEval score that is 3.4 points higher.
3. Training efficiency.
We also profile both recovery placements under the same parameter budget. Fold-point LoRA reduces training-step time and peak GPU memory as follows:
Table 5: Training efficiency gains of fold-point over all-layer recovery.
| Task | Training-step time reduction | Peak allocated GPU memory reduction |
|---|---|---|
| IU | 11.86% | 7.31% |
| T2I | 14.15% | 8.23% |
Fold-point LoRA provides competitive IU performance and better T2I retention, while offering lower per-step training time and peak memory usage than all-layer recovery.
Initial Response to Reviewer MciY: W1 and Q1 [1/5]
We thank Reviewer MciY for recognizing task-specific paths, the controlled comparisons, and the practical Janus-Pro trade-off. We clarify the scope of the evidence and address your questions about the search proxy, recovery capacity, capability retention, and collaborative deployment.
W1. Evidence for the folding-path search objective.
The path-cost comparison and downstream results jointly support the practical utility of our proxy objective. Under the same IU cost matrix and 20% pruning budget, FoldUMM’s DAG-DP path achieves a cumulative cost of 0.123195, compared with 0.130811 for task-specific ShortGPT, a 5.82% reduction. Together with the corresponding performance improvement in our ablation experiment, this provides empirical evidence that global proxy optimization identifies a more effective folding path.
A more rigorous empirical assessment would repeatedly prune, heal, and evaluate each candidate removal. With N layers and K removals, this requires O(KNTHE), where THE is the representative cost of one run of healing and evaluation. Repeating full healing and evaluation for every candidate would make this search prohibitively expensive under our experimental protocol. Our approach uses a simple cosine-based estimate following common practice in depth reduction [1, 2]:
Tours=Tproxy+THE,Tproxy≪THE,
where Tproxy covers calibration and path search. This avoids repeated expensive recovery during search, while our ablation experiments support the effectiveness of the selected paths. We will add the complexity derivation to the appendix.
Q1. Precise characterization of task divergence.
We thank Reviewer MciY for pointing out this imprecision. We agree that different preferred layer locations more precisely characterizes the evidence, and our original wording overgeneralizes the continuity distinction. Our motivation comes from the original paper’s
Figure 3, where IU exhibits smoother representation-similarity patterns and T2I exhibits more uneven patterns. These dense-model patterns inform path selection, which also depends on the pruning budget and global search objective. Thus, the selected T2I paths can include continuous deep-layer removals.The additional BAGEL and Emu3.5 analyses support task-specific redundancy across backbones, while showing that its depth distribution and continuity vary across models. We will revise the main text to emphasize task-dependent layer preferences and distinguish this general finding from the model-specific patterns observed in the original paper’s
Figure 3.References
[1] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025.
[2] The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.