acceptodds
Under review as a conference paper at ICLR 2027

FoldUMM: Unlocking Efficient Inference in Unified Multimodal Models via Task-Specific Layer Folding

Abstract

Unified Multimodal Models (UMMs) couple image understanding and text-to-image generation within a shared Transformer backbone, but this shared design introduces substantial inference overhead. Through layer-wise redundancy profiling, we observe a structural divergence between the two tasks: understanding exhibits concentrated redundancy in deeper layers, whereas generation is more sensitive to depth reduction and lacks continuous redundant blocks. This task-dependent redundancy makes static layer pruning prone to cross-task degradation when transferred to UMMs. We propose FoldUMM, a task-aware layer-folding framework that realizes a "One Backbone, Multiple Paths" inference paradigm. FoldUMM formulates layer folding as a constrained shortest-path problem on a directed acyclic graph, and dynamic programming identifies task-specific folding paths under a target pruning budget. Lightweight Stitch LoRA modules are further inserted at fold points to repair representation mismatch. Experiments on Janus-Pro and OmniGen2 show that FoldUMM achieves a more favorable performance–efficiency trade-off, preserving substantially higher understanding and generation performance under structured pruning while providing practical inference speedup. Our code is available at https://anonymous.4open.science/r/FoldUMM/.

Primary area:
optimization
Keywords:
Unified Multimodal Models, Efficient Inference, Layer Pruning, Model Compression, Task-Aware Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_MciYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inference efficiency in Unified Multimodal Models (UMMs). Its central observation is that image understanding and image generation rely on different layers despite sharing a backbone, so a single pruning path may struggle to preserve both capabilities. The proposed framework, FoldUMM, constructs task-specific graphs using differences between layer representations and applies dynamic programming to select execution paths under a given pruning budget. Local Stitch LoRA modules then recover performance at layer-skipping points. Full pruning and recovery experiments on Janus-Pro and OmniGen2 show better preservation of understanding and generation performance than the evaluated structured pruning baselines. Janus-Pro achieves end-to-end image-generation acceleration, while the gains on OmniGen2 are mainly confined to the backbone. The work offers an approach to UMM efficiency that shares parameters while adapting execution paths to individual tasks.

Critical strengths
  1. Understanding and generation share a backbone, but they may not require exactly the same layers. The layer-representation analysis and cross-task path experiments provide empirical support for this observation. This perspective is useful for UMM development: parameter sharing can coexist with task-specific computation, offering a more flexible way to improve efficiency while preserving unified capabilities.

  2. The framework first analyzes task-dependent layer redundancy, then searches for task-specific retained-layer paths, and finally repairs the transitions introduced by layer skipping. Each step serves a clear purpose. The introduction adequately explains why the shared backbone is an optimization target and why a single pruning path may be insufficient. Figure 4 also helps connect offline path search with online execution.

  3. Beyond comparisons with existing pruning methods, the paper includes baselines with the same recovery procedure, task-specific paths, local versus global path selection, and localized versus full-layer LoRA. These controls help distinguish gains from recovery training, task adaptation, and path search. In particular, the task-specific greedy baseline provides a more direct reference for assessing the contribution of global search.

  4. At 20% pruning, Janus-Pro’s image-generation time decreases from 3.94 to 3.16 seconds, while FoldUMM achieves better overall performance retention than the evaluated structured pruning methods. Localized recovery also uses substantially fewer trainable parameters than full-layer LoRA. These results demonstrate a practical performance–efficiency trade-off and suggest a feasible approach to adapting existing UMMs at relatively low training cost.

Critical weaknesses
  1. The method estimates skipping costs using representation similarity in the original model and adds these costs across transitions. However, multiple skips jointly alter downstream representations, and recovery training can further affect the relative quality of different paths. Dynamic programming finds the optimal path under the defined cost, but the experiments do not fully establish whether lower search costs consistently predict smaller task-performance losses. The alternative-distance experiments provide some support, but offer limited evidence about the reliability of this proxy.

  2. The complete pruning and recovery pipeline is evaluated on Janus-Pro and OmniGen2, but OmniGen2 shows relatively small cross-task path differences and no clear end-to-end generation speedup. The additional BAGEL and Emu3.5 results mainly demonstrate differences in representations or selected layers, without evaluating the complete method’s performance and efficiency. The benchmarks also assess understanding and text-to-image generation separately, rather than image editing or workflows requiring continuous cooperation between understanding and generation. The evidence therefore supports the evaluated settings more strongly than broader preservation of unified capabilities.

  3. On Janus-Pro, average retention exceeds 90% for both understanding and generation, yet MM-Vet-V2 drops from 47.1 to 36.4 and GenEval from 0.79 to 0.66. These results indicate substantial degradation in some capabilities. The appendix provides a useful analysis of recovery-data composition, but more detailed results by capability are still missing. For UMMs, which aim to integrate multiple capabilities, it is important to establish which abilities are sacrificed for efficiency and whether those losses affect practical use.

  4. Although OmniGen2 achieves higher backbone throughput, its complete image-generation time remains 30.10 seconds, showing that backbone compression does not always address the main bottleneck. With both paths resident, Janus-Pro’s weight memory decreases from 14.9 GB to 13.2 GB, a smaller reduction than single-path pruning achieves. The paper acknowledges these limitations, but its efficiency experiments mainly use single-request settings and provide limited direct evidence about overall throughput and latency under mixed understanding and generation workloads.

Questions for authors
  1. How should the difference between understanding and generation redundancy be characterized more precisely? The main text emphasizes that generation lacks continuous redundant regions. However, Table 12 shows that Janus-Pro’s generation path removes contiguous layers at both 10% and 15% pruning, and the removals at 20% also largely concentrate in later layers. These results appear to support different preferred layer locations more directly than a general distinction between concentrated understanding redundancy and fragmented generation redundancy. Could the authors clarify how broadly this characterization holds across pruning rates and models?

  2. Can the cross-task path experiments further separate the effects of layer selection and recovery training? When paths are exchanged in Table 2, are the corresponding LoRA modules reused directly or retrained for the target task? If one shared path is fixed and separately recovered using understanding and generation data, how does it compare with two independently selected paths? This comparison would help establish how much of the task conflict comes from layer selection itself and what limitations remain after sufficient adaptation of a shared path.

  3. Do the gains from global search and localized recovery persist when trainable parameter counts are strictly matched? Different paths have different numbers of folding points, so using the same LoRA rank does not necessarily provide the same recovery capacity. The comparison between localized and full-layer LoRA also changes both placement and parameter count. Could the authors fix the total trainable parameter budget when comparing global search with task-specific greedy selection, and fold-point recovery with uniformly distributed recovery? This would help clarify whether the gains come from path quality, recovery placement, or adaptation capacity.

  4. Can the method retain its overall benefits in tasks requiring understanding and generation to work together? The appendix discusses switching paths at stage boundaries and re-encoding or prefilling the context. Could the authors evaluate a modest image-editing or generation–understanding–revision task and report final task quality, complete execution time, and memory usage? This would more directly establish whether task-specific paths improve practical efficiency while preserving the collaborative capabilities of a UMM.

8 comments
AuthorsOpenReview ↗

Initial Response to Reviewer MciY: W1 and Q1 [1/5]

We thank Reviewer MciY for recognizing task-specific paths, the controlled comparisons, and the practical Janus-Pro trade-off. We clarify the scope of the evidence and address your questions about the search proxy, recovery capacity, capability retention, and collaborative deployment.

W1. Evidence for the folding-path search objective.

W1: “The method estimates skipping costs using representation similarity in the original model and adds these costs across transitions. However, multiple skips jointly alter downstream representations, and recovery training can further affect the relative quality of different paths. Dynamic programming finds the optimal path under the defined cost, but the experiments do not fully establish whether lower search costs consistently predict smaller task-performance losses. The alternative-distance experiments provide some support, but offer limited evidence about the reliability of this proxy.”

The path-cost comparison and downstream results jointly support the practical utility of our proxy objective. Under the same IU cost matrix and 20% pruning budget, FoldUMM’s DAG-DP path achieves a cumulative cost of 0.123195, compared with 0.130811 for task-specific ShortGPT, a 5.82% reduction. Together with the corresponding performance improvement in our ablation experiment, this provides empirical evidence that global proxy optimization identifies a more effective folding path.

A more rigorous empirical assessment would repeatedly prune, heal, and evaluate each candidate removal. With NN layers and KK removals, this requires O(KN THE)O(KN\,T_{\mathrm{HE}}), where THET_{\mathrm{HE}} is the representative cost of one run of healing and evaluation. Repeating full healing and evaluation for every candidate would make this search prohibitively expensive under our experimental protocol. Our approach uses a simple cosine-based estimate following common practice in depth reduction [1, 2]:

Tours=Tproxy+THE,Tproxy≪THE,T_{\mathrm{ours}}=T_{\mathrm{proxy}}+T_{\mathrm{HE}},\qquad T_{\mathrm{proxy}}\ll T_{\mathrm{HE}},

where TproxyT_{\mathrm{proxy}} covers calibration and path search. This avoids repeated expensive recovery during search, while our ablation experiments support the effectiveness of the selected paths. We will add the complexity derivation to the appendix.


Q1. Precise characterization of task divergence.

Q1: “How should the difference between understanding and generation redundancy be characterized more precisely? The main text emphasizes that generation lacks continuous redundant regions. However, Table 12 shows that Janus-Pro’s generation path removes contiguous layers at both 10% and 15% pruning, and the removals at 20% also largely concentrate in later layers. These results appear to support different preferred layer locations more directly than a general distinction between concentrated understanding redundancy and fragmented generation redundancy. Could the authors clarify how broadly this characterization holds across pruning rates and models?”

We thank Reviewer MciY for pointing out this imprecision. We agree that different preferred layer locations more precisely characterizes the evidence, and our original wording overgeneralizes the continuity distinction. Our motivation comes from the original paper’s Figure 3, where IU exhibits smoother representation-similarity patterns and T2I exhibits more uneven patterns. These dense-model patterns inform path selection, which also depends on the pruning budget and global search objective. Thus, the selected T2I paths can include continuous deep-layer removals.

The additional BAGEL and Emu3.5 analyses support task-specific redundancy across backbones, while showing that its depth distribution and continuity vary across models. We will revise the main text to emphasize task-dependent layer preferences and distinguish this general finding from the model-specific patterns observed in the original paper’s Figure 3.


References

[1] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025.

[2] The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.

AuthorsOpenReview ↗

Initial Response to Reviewer MciY: Q2 [2/5]

Q2. Shared-path comparison with task-specific recovery.

Q2: “Can the cross-task path experiments further separate the effects of layer selection and recovery training? When paths are exchanged in Table 2, are the corresponding LoRA modules reused directly or retrained for the target task? If one shared path is fixed and separately recovered using understanding and generation data, how does it compare with two independently selected paths? This comparison would help establish how much of the task conflict comes from layer selection itself and what limitations remain after sufficient adaptation of a shared path.”

In the original paper's Table 2, each transferred path underwent the full healing procedure for the target task, using the same training setup as the corresponding task-specific path. Stitch LoRA was trained for the target task, with no reuse of the source-task adapter. The comparison therefore assesses path selection under consistent target-task recovery.

To separate shared topology from task-specific recovery, we added a joint-cost shared-path baseline using the same DAG-DP search and 20% pruning budget. We construct its cost matrix as

Cshared(i,j)=λCIU(i,j)sIU+(1−λ)CT2I(i,j)sT2I,C_{\mathrm{shared}}(i,j)=\lambda\frac{C_{\mathrm{IU}}(i,j)}{s_{\mathrm{IU}}}+(1-\lambda)\frac{C_{\mathrm{T2I}}(i,j)}{s_{\mathrm{T2I}}},

where each sts_t is the mean positive cost over eligible non-adjacent edges for task tt. We evaluate λ∈{0.25,0.5,0.75}\lambda\in\{0.25,0.5,0.75\}, covering equal weighting and preferences toward either task. Each shared path is healed separately for IU and T2I. We align the recovery parameter budget with the task-specific counterpart by adjusting LoRA rank to account for differences in the number of fold points.

Results for the shared-path baselines are averaged over three training seeds for both IU and T2I. Mean RP averages the Dense-normalized scores across the five reported benchmarks.

Table 1: Task-specific and shared DAG paths under matched recovery budgets.

Path configurationMME ↑POPE ↑GQA ↑MMMU-Pro ↑GenEval ↑Mean RP ↑
Dense156785.260.435.30.79100.0%
Task-specific150590.256.432.80.6694.4%
Shared, λ=0.25129380.251.629.20.5783.4%
Shared, λ=0.50128182.052.931.90.5184.1%
Shared, λ=0.75143684.355.531.30.5688.4%

Task-specific paths achieve higher scores across all five reported metrics than each tested shared path. The best tested shared path remains 6.0 percentage points below the task-specific paths in Mean RP (88.4% vs. 94.4%). This remaining gap after matched task-specific healing more directly reveals the task conflict arising from layer selection, showing the limitation of using a shared path for both tasks.

AuthorsOpenReview ↗

Initial Response to Reviewer MciY: Q3 [3/5]

Q3. Matched recovery capacity and placement.

Q3: “Do the gains from global search and localized recovery persist when trainable parameter counts are strictly matched? Different paths have different numbers of folding points, so using the same LoRA rank does not necessarily provide the same recovery capacity. The comparison between localized and full-layer LoRA also changes both placement and parameter count. Could the authors fix the total trainable parameter budget when comparing global search with task-specific greedy selection, and fold-point recovery with uniformly distributed recovery? This would help clarify whether the gains come from path quality, recovery placement, or adaptation capacity.”

We thank Reviewer MciY for suggesting these experiments. We added two parameter-matched ablations to separate search quality from recovery placement, together with training-cost profiling. Performance results are averaged over three random seeds.

1. Search strategy.

On Janus-Pro IU at 20% pruning, DAG-DP uses four fold points at rank 128, and task-specific Greedy uses two at rank 256. Both train 39,976,960 parameters, with the same data and optimization settings and matched update counts.

Table 2: Greedy and DAG-DP search with matched recovery capacity on IU.

Search methodMME ↑POPE ↑GQA ↑MMMU-Pro ↑
Greedy1421.6485.7756.0032.95
DAG-DP1505.0090.2056.4032.80

The T2I comparison in the original paper's Table 3 already matches the recovery parameter budget, with both methods using rank-128 Stitch LoRA at two fold points. Under this matched budget, FoldUMM achieves 0.66 on GenEval, compared with 0.62 for task-specific Greedy.

With matched recovery capacity, DAG-DP achieves higher scores on three of the four IU benchmarks, supporting the benefit of global folding-path optimization.

2. Recovery placement.

We further provide a fairer comparison of recovery placement by matching the total trainable parameter budget. We fix the FoldUMM path for each task and distribute this budget either at fold points or across all 24 retained layers.

Table 3: Matched parameter budgets for fold-point and all-layer recovery.

TaskPlacementLocationsRank allocationTrainable parameters
IUFold points44 × 12839,976,960
IUAll retained layers2416 × 21 + 8 × 2239,976,960
T2IFold points22 × 12819,988,480
T2IAll retained layers248 × 10 + 16 × 1119,988,480

Table 4: Performance of fold-point and all-layer recovery under matched budgets.

Recovery placementMME ↑POPE ↑GQA ↑MMMU-Pro ↑GenEval ↑
Fold points1505.0090.2056.4032.800.659
All retained layers1507.0988.1957.1233.010.625

Fold-point LoRA achieves comparable IU performance to all-layer recovery and a GenEval score that is 3.4 points higher.

3. Training efficiency.

We also profile both recovery placements under the same parameter budget. Fold-point LoRA reduces training-step time and peak GPU memory as follows:

Table 5: Training efficiency gains of fold-point over all-layer recovery.

TaskTraining-step time reductionPeak allocated GPU memory reduction
IU11.86%7.31%
T2I14.15%8.23%

Fold-point LoRA provides competitive IU performance and better T2I retention, while offering lower per-step training time and peak memory usage than all-layer recovery.

Reviewer_CBZqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FoldUMM combines task-specific layer-folding paths, DAG-based dynamic programming, and Stitch LoRA recovery to enable efficient image understanding and generation in unified multimodal models.

Critical strengths
  1. The paper identifies different layer-redundancy patterns for understanding and generation, providing a reasonable motivation for task-specific pruning.
  2. Applying Stitch LoRA locally at folding points is a well-motivated recovery strategy.
Critical weaknesses
  1. The method primarily combines existing layer pruning, dynamic programming, and LoRA techniques, offering limited algorithmic novelty.
  2. On OmniGen2, backbone throughput improves by 1.22×, but end-to-end image generation remains approximately 30.10 seconds, while maintaining multiple paths requires more GPU memory than single-path pruning.
  3. Generation performance drops substantially at a 30% pruning rate, limiting applicability under aggressive compression.
Questions for authors
  1. Why is hidden-state cosine distance used as the objective for folding-path search?
  2. What practical deployment benefits does FoldUMM offer for architectures such as OmniGen2, where external generators dominate latency?
  3. How are folding paths selected and switched for composite tasks such as interleaved image-text generation and multi-turn dialogue?
3 comments
AuthorsOpenReview ↗

Initial Response to Reviewer CBZq: W1, Q1, W2/Q2 [1/2]

We thank Reviewer CBZq for recognizing the task-specific redundancy motivation and localized recovery design. We provide evidence and technical justification for your concerns about novelty, deployment benefits, cost selection, and composite tasks.

W1. Contribution of task-specific folding.

W1: “The method primarily combines existing layer pruning, dynamic programming, and LoRA techniques, offering limited algorithmic novelty.”

FoldUMM’s central contribution is to identify and exploit task-specific redundancy for structured layer pruning in UMMs. This observation motivates FoldUMM’s shared-backbone, multiple-path design. Our ablation experiments support the contributions of global search and task-specific path selection, showing how these components address the different redundancy structures of understanding and generation. The work provides both a practical layer-pruning method for UMMs and insight into how understanding and generation use the depth of a shared backbone differently.


Q1. Choice of cosine mismatch.

Q1: “Why is hidden-state cosine distance used as the objective for folding-path search?”

We use cosine mismatch as a simple representation-discrepancy measure following common practice in prior depth-reduction work [1, 2]. It provides a low-cost proxy for task-specific redundancy, allowing us to focus on folding-path optimization. Appendix D.1 compares alternative representation costs under the same experimental settings. These empirical results support cosine mismatch as an effective, inexpensive choice in our setting. Alternative metrics remain valuable, and combining different representation-discrepancy measures is a promising direction for future work.


W2/Q2. Deployment efficiency.

W2: “On OmniGen2, backbone throughput improves by 1.22×, but end-to-end image generation remains approximately 30.10 seconds, while maintaining multiple paths requires more GPU memory than single-path pruning.”

Q2: “What practical deployment benefits does FoldUMM offer for architectures such as OmniGen2, where external generators dominate latency?”

  1. Quality and inference benefits. On OmniGen2, FoldUMM accelerates text generation and preserves higher quality on both IU and T2I than the compared pruning baselines at the same pruning rate. For image generation, backbone time is approximately its TTFT of 34.2 ms, about 0.1% of the 30.10 s total. This architecture limits the end-to-end speedup achievable through backbone pruning. Dedicated diffusion acceleration methods are orthogonal to FoldUMM and can be combined with it to accelerate the external generator.

  2. Weight storage. OmniGen2 exhibits task-specific path differences alongside substantial overlap: five of seven removed layers are shared. This overlap keeps the additional weight storage for task-specific paths small. With shared base weights, BF16 Transformer-block storage decreases from 5.55 GB to 4.78 GB (13.9%), only 0.31 GB (6.9%) above a single path. We will clarify this architecture-dependent trade-off in the main text.


References

[1] ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL, 2025.

[2] The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.

AuthorsOpenReview ↗

Initial Response to Reviewer CBZq: W3, Q3 [2/2]

W3. Quality retention across pruning rates.

W3: “Generation performance drops substantially at a 30% pruning rate, limiting applicability under aggressive compression.”

The greater sensitivity of T2I highlights the challenge of pruning a UMM while preserving both understanding and generation. This motivates task-specific redundancy analysis and the resulting pruning design. The compared structured baselines exhibit severe degradation at lower pruning rates, while FoldUMM provides a more favorable trade-off between quality and efficiency across the evaluated pruning budgets (Section 4.3). Appendix E.3 identifies 20% for IU and 15% for T2I as favorable operating points on Janus-Pro, allowing generation to use a more conservative path within the same shared backbone.


Q3. Path selection in composite workflows.

Q3: “How are folding paths selected and switched for composite tasks such as interleaved image-text generation and multi-turn dialogue?”

FoldUMM selects paths according to the task of each execution stage. For example, image generation, textual reflection, and image revision use the T2I, IU, and T2I paths, respectively. Within each stage, the path and adapter remain fixed, supporting standard KV caching. A task-path change starts with fresh encoding or prefill of the context; multi-turn understanding can retain the IU path across turns. Appendix F.4 describes this stage-wise execution design.

We further evaluated this design on Janus-Pro using 64 interleaved multimodal task executions: image generation, image description, comparison with the original request, and image regeneration. These comprise two T2I stages, two IU stages, and two task-path switches. FoldUMM achieves 1.29× end-to-end speedup and 9.3% lower GPU memory usage than Dense. These results demonstrate practical acceleration despite task-path switching.

Fine-grained token-level switching and cross-path KV reuse offer promising directions for extending FoldUMM, which we plan to explore in future work.


We hope these responses address your concerns. We remain available to provide further clarification.

AuthorsOpenReview ↗

Summary and Follow-up to Reviewer CBZq

Dear Reviewer CBZq,

We hope this message finds you well. With the discussion deadline approaching, we would like to follow up on our rebuttal and ask whether our responses have addressed your concerns.

To address your concerns about FoldUMM's contribution, quality retention, and practical deployment, we have provided the following clarifications and additional experimental evidence:

  1. Contribution and metric choice (W1/Q1). Our core contributions are the identification of task-specific redundancy in UMMs and the development of FoldUMM, a structured layer-pruning method with a shared-backbone, multiple-path design. We chose cosine mismatch as a simple, low-cost representation-discrepancy metric widely used in prior layer-pruning work. Our ablation experiments demonstrate the benefits of task-specific selection and global search. The alternative-cost analysis supports cosine mismatch as a practical choice.

  2. Deployment benefits (W2/Q2). FoldUMM provides faster text generation and lower backbone storage on OmniGen2. Maintaining both task-specific paths still reduces Transformer-block storage by 13.9%.

  3. Pruning sensitivity (W3). Our pruning-rate experiments show that FoldUMM achieves a more favorable quality–efficiency trade-off than the compared structured baselines. The analysis identifies 20% for IU and 15% for T2I as favorable operating points on Janus-Pro, demonstrating the flexibility of task-specific pruning budgets.

  4. Path selection and switching (Q3). The added interleaved multimodal inference experiment achieves 1.29× end-to-end speedup despite path switching. This result demonstrates FoldUMM’s practical deployment benefits in interleaved multimodal scenarios.

We would appreciate knowing whether any point needs further explanation. We remain available for further discussion.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Initial Response to Reviewer LwJv: W1/Q1 and W2/Q3 [1/3]

We thank Reviewer LwJv for recognizing the global-search control and the matched Stitch LoRA comparisons. Below, we address your concerns about architecture-dependent efficiency, task-specific paths, calibration stability, and evaluation details.

W1/Q1. FoldUMM efficiency across UMM architectures.

W1: “The efficiency claim does not survive the decoupled architecture, which is half the experimental scope.”

Q1: “For OmniGen2, what fraction of the 30.10 s generation time is spent in the shared backbone rather than in the external generator? If the backbone accounts for a small share of end-to-end latency, what is the deployment argument for the method on decoupled UMMs?”

  1. Quality retention. We evaluate FoldUMM on Janus-Pro and OmniGen2 to demonstrate its applicability across both UMM paradigms. FoldUMM achieves higher retained quality than the compared pruning baselines at the same 20% pruning rate, retaining 92.6%/96.5% average IU/T2I relative performance on OmniGen2.

  2. Inference acceleration. For OmniGen2 image generation, backbone time is approximately its TTFT of 34.2 ms, about 0.1% of the 30.10 s total, with diffusion dominating the remaining cost. FoldUMM accelerates both evaluated tasks in the natively unified architecture; in the decoupled architecture, substantial end-to-end gains arise in text generation or tasks where the backbone is the bottleneck. Its quality advantage over the compared pruning baselines holds across both architectures. We will make this architecture-dependent scope more explicit in the main text, clarifying when backbone acceleration translates into substantial end-to-end gains.

  3. Deployment benefits. With shared base weights across the two task paths, OmniGen2’s BF16 Transformer-block storage decreases from 5.55 GB to 4.78 GB (13.9%). Together with faster text generation, this provides practical benefits even when the external diffusion generator dominates image-generation latency.


W2/Q3. Task-specific redundancy and path overlap.

W2: “The task-divergence premise is weak at the operating point where every main result is reported.”

Q3: “Table 12 at 20% gives Jaccard 0.5 between the IU and T2I sets. How does that reconcile with the claim that generation redundancy is not clustered late, given that both sets remove layers in the 20–28 band?”

  1. Operating point and task differences. We chose the 20% operating point based on the empirical trade-off between quality and efficiency and used the same pruning budget for comparison across methods. The partial overlap (Jaccard = 0.5) still leaves consequential task-specific differences: applying the IU configuration to T2I reduces GenEval from 0.66 to 0.49 (the original paper’s Table 2).

  2. Similarity patterns and selected paths. Regarding Q3, our description refers to IU’s smoother representation-similarity pattern and T2I’s more uneven pattern in the original paper’s Figure 3. These observations describe dense-model representations; the selected paths also depend on the pruning budget and global search objective. A continuous deep-layer segment can therefore appear in the selected T2I path at 20% pruning. We will clarify this distinction and qualify the continuity claim.

We would also like to clarify a possible misunderstanding or typographical error in your concluding sentence, which appears to reverse the two architectures: Janus-Pro exhibits both stronger task-path sensitivity and end-to-end acceleration.

AuthorsOpenReview ↗

Initial Response to Reviewer LwJv: Q2 [2/3]

Q2. Joint-cost shared DAG experiment.

Q2: “Could you add the baseline that isolates the "multiple paths" claim: a single path selected by the same DAG-DP under the same 20% budget and healed with the same Stitch LoRA, optimized for (a) IU, (b) T2I, and (c) the minimum or mean of the two? Table 2 approximates this by crossing over the task-specific paths, but does not test a path optimized for a joint objective. If a single joint path lands within ~1–2 RP points on Janus, the "multiple paths" contribution shrinks substantially.”

Following your suggestion, we added a joint-cost shared-path baseline using the same DAG-DP search and 20% pruning budget. We construct its cost matrix as

Cshared(i,j)=λCIU(i,j)sIU+(1−λ)CT2I(i,j)sT2I,C_{\mathrm{shared}}(i,j)=\lambda\frac{C_{\mathrm{IU}}(i,j)}{s_{\mathrm{IU}}}+(1-\lambda)\frac{C_{\mathrm{T2I}}(i,j)}{s_{\mathrm{T2I}}},

where each sts_t is the mean positive cost over eligible non-adjacent edges for task tt. We evaluate $\lambda\in { 0.25,0.5,0.75 } $, covering equal weighting and preferences toward either task. Each shared path is healed separately for IU and T2I. We align the recovery parameter budget with the task-specific counterpart by adjusting LoRA rank to account for differences in the number of fold points.

Results for the shared-path baselines are averaged over three training seeds for both IU and T2I. Mean RP averages the Dense-normalized scores across the five reported benchmarks.

Table 1: Performance of task-specific and shared DAG paths.

Path configurationMME ↑POPE ↑GQA ↑MMMU-Pro ↑GenEval ↑Mean RP ↑
Dense156785.260.435.30.79100.0%
Task-specific150590.256.432.80.6694.4%
Shared, λ=0.25129380.251.629.20.5783.4%
Shared, λ=0.50128182.052.931.90.5184.1%
Shared, λ=0.75143684.355.531.30.5688.4%

Task-specific paths achieve higher scores across all five reported metrics than those of each tested shared path. The best tested shared path remains 6.0 percentage points below the task-specific paths in Mean RP (88.4% vs. 94.4%). These results further support task-specific redundancy and the need for distinct task paths to preserve both IU and T2I quality, validating FoldUMM’s central motivation.

AuthorsOpenReview ↗

Initial Response to Reviewer LwJv: Q4/Q5/Q6 [3/3]

Q4. Folding-path stability across calibration samples.

Q4: “The DP edge cost is a task-specific cosine mismatch between hidden states. How stable are the exact selected sets across calibration draws? You report that 64 or 128 samples give "the same qualitative pattern" — do the sets themselves agree?”

The selected paths stabilize with a modest calibration budget. Following your suggestion, we added a calibration-size analysis using 32, 64, and 128 samples drawn from a fixed 256-example calibration pool for each task, with three sampling seeds per task and unchanged search constraints.

At 128 samples, all three seeds produce identical selected sets within each task. Increasing the sample size from 64 to 128 preserves the selected set in five of six comparisons across tasks and seeds, with a mean Jaccard similarity of 0.952. These results support the stability of path selection at this calibration scale. We will include this analysis in the appendix.


Q5. Baseline recovery and latency.

Q5: “ShortGPT and SLEB receive matched Stitch LoRA healing, while Short-LVLM is evaluated with its original SVD compensation. Would Short-LVLM with Stitch LoRA close part of the gap? Conversely, TTFT and Gen. are blank for the healed ShortGPT and SLEB rows — is that a measurement omission, or were those configurations simply not run end to end?”

  1. Recovery comparison. Adding Stitch LoRA to Short-LVLM would evaluate the additional benefit of LoRA recovery on another method. Such a combination could improve performance, but a controlled assessment would require further ablations of SVD compensation and LoRA under matched recovery budgets. This would primarily examine the interaction between recovery methods, while our central question concerns task-specific redundancy and its implications for layer pruning. We retain Short-LVLM’s original SVD-based compensation as a baseline representing an alternative recovery approach to LoRA. Our existing ablation experiments already provide evidence for FoldUMM’s design contributions.

  2. Latency reporting. We measured the healed baselines end to end and observed no statistically significant latency difference from their corresponding unhealed baselines. Stitch LoRA is merged into the retained weights before inference, preserving the execution graph and adding no adapter-specific inference operations. The blank entries avoid repeating these equivalent latency results. We will clarify this reporting convention in the original paper’s Table 1.


Q6. Quality and efficiency at higher pruning rates.

Q6: “At 20% pruning the Janus result is 1.25× wall-clock. What gain do you project at 30–40%, and at what pruning rate does the retention curve cross the 80% IU RP line referred to in the appendix?”

FoldUMM retains 81.7% IU relative performance at 30% pruning (the original paper’s Figure 5), with retention falling below 80% at higher pruning rates in our experiments. The compared structured baselines fall below this threshold at lower pruning rates, demonstrating FoldUMM’s stronger quality retention under aggressive compression. Since FoldUMM’s measured backbone speedup closely approaches the theoretical layer-compute limit at 20% pruning, we expect backbone speedups near 1.43× and 1.67× at 30% and 40%, respectively.


We hope these responses address your concerns. We remain available to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.