acceptodds
Under review as a conference paper at ICLR 2027

Hourglass: Towards Better Unified Multimodal Modeling Through Hierarchical Sharing and Conflict-Aware Gradient Surgery

10 views

Abstract

Unified multimodal models (UMMs) have attracted growing attention due to their unified task interfaces and potential for broader generalization. However, existing unified methods largely follow either partially shared designs with limited cross-task coordination or fully shared architectures that may induce capacity competition and gradient interference. To address these limitations, we propose Hourglass, a unified framework that facilitates beneficial collaboration at both the architectural and optimization levels. Motivated by the semantic complementarity and task heterogeneity of visual understanding and generation, Hourglass employs a hierarchical decoupled-shared-decoupled design, retaining task-specific processing in both shallow and deep stages while introducing a selective intermediate sharing mechanism. This structures a balanced trade-off that promotes beneficial cross-task collaboration while preserving task specialization. Nevertheless, jointly training the shared parameters may expose them to conflicting gradients stemming from the distinct natures of understanding and generation. To mitigate this, we further present CocoGrad, a gradient surgery method that dynamically partitions the shared-parameter coordinates into collaborative and conflicting subspaces, preserving common descent directions while coordinating conflicting updates. Extensive experiments across various visual understanding and generation benchmarks demonstrate that our method achieves state-of-the-art or highly competitive performance, providing a viable and scalable blueprint for UMMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Models, Visual Understanding and Generation, Hierarchical Architecture, Gradient Surgery

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_3J9vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hourglass, a unified multimodal models(UMMs) for visual understanding and generation. First, it introduces a hierarchical decoupled-shared-decoupled architecture, which preserves task-specific processing in shallow and deep layers while enabling selective parameter sharing in intermediate layers. Second, it proposes CocoGrad, a conflict-aware gradient surgery method that partitions shared-parameter gradients into collaborative and conflicting subspaces and adaptively coordinates updates in the conflicting subspace. Experiments on visual understanding, text-to-image generation, and image editing benchmarks demonstrate competitive performance, with ablation studies evaluating the effectiveness of the proposed designs.

Critical strengths

1.The authors employ a decoupled-shared-decoupled model architecture and update the Q and K parameters in the middle layers by jointly training on losses from both understanding and generation tasks—an approach that is both theoretically sound and intriguing.

2.Conflict-Aware Gradient Optimization. The paper introduces CocoGrad to address gradient interference between visual understanding and generation during joint training. Experimental results show that CocoGrad improves performance over vanilla joint optimization and several existing gradient surgery methods on the evaluated benchmarks.

3.Coherent Framework Design. The hierarchical sharing architecture and CocoGrad are complementary, addressing parameter sharing and optimization interference at the architectural and optimization levels, respectively. Their integration provides a coherent framework for jointly modeling visual understanding and generation.

Critical weaknesses

1.The core insight underlying the hierarchical decoupled-shared-decoupled architecture substantially overlaps with prior UMM studies. In particular, Uni-X (ICLR 2026) has already identified stronger gradient conflicts in shallow and deep layers and proposed a two-end-separated, middle-shared architecture. HBridge (CVPR 2026) similarly explores selective intermediate-layer interactions between understanding and generation experts, while DIVA (ICML 2026) investigates representation divergence and the separation of shared and task-specific information in UMMs. However, these closely related works are not adequately discussed or compared against, making it difficult to assess the architectural novelty of Hourglass beyond its specific Q/K sharing implementation.

2.Fair-compute comparisons are unclear. Training tokens, GPU-hours/FLOPs, and data mixtures for understanding and generation are only partially described; some tables contain artifacts, and it is hard to judge efficiency vs. recent strong UMMs (e.g., EMMA, Lance, NextFlow,Emu3.5).

3.The key ideas behind CocoGrad have substantial connections to existing multi-task optimization methods. Coordinate-wise gradient sign agreement has been explored in GradDrop (NeurIPS 2020), while constrained worst-task improvement is closely related to CAGrad (NeurIPS 2021). The proposed method appears to combine existing gradient partitioning and balancing principles rather than introducing a fundamentally new optimization mechanism.

4.Although the paper reports competitive results on visual understanding, text-to-image generation, and image editing benchmarks, these tasks are primarily evaluated independently. The experiments therefore do not directly demonstrate the claimed beneficial collaboration between understanding and generation. Moreover, the first two training stages focus exclusively on generation while freezing the understanding branch, making it difficult to disentangle the effects of hierarchical sharing from those of the training recipe and pretrained initialization.

Reference [1] Uni-X: mitigating modality conflict with a two-end-separated architecture for unified multimodal models [2] Hbridge: H-shape bridging of heterogeneous experts for unified multimodal understanding and generation [3] DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement [4] Just pick a sign: Reducing gradient conflict in deep networks with gradient sign dropout [5] Conflict-Averse Gradient Descent for Multi-task Learning

Questions for authors

1.Why share QK specifically? Do you have diagnostics (e.g., attention map similarity, head importance, CKA) comparing alternatives such as shared V, partial FFN sharing, or shared attention maps with decoupled projections?

2.Could you provide some performance metrics for Hourglass—compared to stronger baselines(e.g. Emu3.5, Lance)—on proprietary benchmarks(e.g. Unison,Uni-MMMU) used by UMMs?

3.Will code/models be released? If not, can you provide more implementation details?In addition,from my personal perspective, wouldn't it be more compelling if a complete model were made accessible—either by being opened up or by providing usable interfaces—so that people could invoke it directly?

4 comments

Thank you for your thoughtful and insightful feedback. We appreciate your recognition of the complementary roles of hierarchical sharing and CocoGrad. Your comments help us better distinguish our contribution from related work and clarify the evidence supporting it. We respond point by point below.

W1: Relationship to Related Works (Uni-X, HBridge, and DIVA)

Thank you for highlighting these closely related works. Although Hourglass and these works look similar in architecture, they differ in the details of that architecture, the rationale behind it, and the objective it serves. The core code implementing the contributions of our paper is available at this link for your inspection Anonymous code. Our implementation is fundamentally different from those of the other methods.

Compared with Uni-X, Hourglass differs in three respects. First, the two branches receive different inputs. In Uni-X, one branch receives text tokens and the other receives visual tokens, whereas in Hourglass, the inputs to both the understanding and generation branches include visual tokens. Second, the intermediate-layer sharing schemes differ: Hourglass shares only the Q/K projections in these layers, whereas Uni-X shares the intermediate layers in full. Third, the methods address gradient conflicts differently. Uni-X mitigates conflicts between the text and visual modalities by sharing only the intermediate layers, whereas Hourglass uses CocoGrad to address conflicts between the two tasks on the shared parameters.

Compared with HBridge, Hourglass differs in two respects. First, the scope of interaction differs: HBridge connects its understanding and generation experts only at selected intermediate layers, whereas Hourglass places tokens from both task paths in a unified sequence for joint self-attention at every layer. Second, their parameter-sharing schemes differ: HBridge aligns the Q/K/V features of its heterogeneous experts for intermediate-layer attention exchange, whereas Hourglass shares only the Q/K projections between its two task paths in the intermediate layers.

Compared with DIVA, Hourglass differs in its approach to architectural sharing. DIVA is a post-training framework applied to existing single-backbone unified models, whereas Hourglass explicitly separates the understanding and generation paths in shallow and deep layers and shares only their Q/K projections in intermediate layers.

Beyond these methodological differences, Table R1 compares the performance of Hourglass with Uni-X, HBridge, and Nexus-Gen + DIVA. Hourglass-7.5B achieves the best reported results on most visual understanding and generation benchmarks. Hourglass-3B also outperforms Uni-X on every metric reported for both models, suggesting that the proposed design is effective across model scales.

Table R1. Comparison with these related works on visual understanding, image generation, and image editing benchmarks.

ModelParamsMME-PMMBenchMMMUGenEvalDPG-BenchImgEdit
Uni-X3B/4.5B1,22862.7-0.8380.303.44
HBridge7B + 4B---0.8385.233.54
Nexus-Gen + DIVA7B1,35574.949.40.8387.873.35
Hourglass3B1,62479.647.40.8783.303.47
Hourglass7.5B1,70988.454.20.8984.233.55

We will add these references to the Related Work section and include their results in Table 2.

W2: Training Data and Compute Details.

Thank you for raising this concern. To facilitate verification and comparison, we have added the Table R2, which reports the dataset pool sizes, actual task sampling ratios, training tokens, hardware, and GPU-hours for the 3B and 7.5B models at each training stage.

Table R2. Training Data and Compute Details.

Training detailPTCTSFT
Dataset samples (U / G / E)0 / 37M / 8M0 / 6M / 1M9M / 5M / 1M
Task sampling ratio (U:G:E)0:4:10:1:16:1:1
Training tokens1.3B0.8B0.2B
Hardware80 × H100 (80 GB)80 × H100 (80 GB)80 × H100 (80 GB)
GPU-hours (3B)10,416.47,564.82,214.6
GPU-hours (7.5B)14,739.813,532.83,321.3

W3: Relationship of CocoGrad to GradDrop and CAGrad.

We thank the reviewer for highlighting these connections. We agree that CocoGrad shares conceptual similarities with existing gradient-manipulation methods. However, it differs in where and how gradient coordination is performed. Table R3 summarizes these distinctions.

CocoGrad has two main advantages. First, it preserves information contained in conflicting gradient components rather than simply discarding them. Second, by separating the two subspaces, it prevents cooperative signals from interfering with conflicting signals, as illustrated in our response to Reviewer jkUk’s Question 2.

More importantly, our numerical analysis identifies a potentially important source of optimization difficulty in joint UMM training: the conflicting components of the understanding and generation gradients account for a substantial proportion of their gradient norms, as shown in Fig. 2(b). This observation motivates our architectural design of hierarchical sharing and localized gradient surgery in the conflicting subspace. To the best of our knowledge, CocoGrad is among the first methods to explicitly analyze and address gradient conflicts between understanding and generation objectives in joint UMM training.

Table R3. Comparison of gradient manipulation methods.

DimensionGradDropCAGradCocoGrad (Ours)
Masking StrategyRandom masking, sampled by dimensionNo masking; operates at the global parameter levelDeterministic selection based on dimension directions
Conflict HandlingDirectly discardsSolves a Quadratic Programming (QP) problem to achieve global compromiseIntroduces a structured gradient update formula Eq.6, achieving subspace weighted coordination

W4: Evidence for beneficial collaboration and the effect of the training recipe.

Thank you for this important observation. The sharing-depth ablation in Table 3 provides evidence beyond the separately reported task benchmarks: with CocoGrad disabled, 12 intermediate shared layers improve both final GenEval (0.63 vs. 0.55) and MME-S (2006 vs. 1976) relative to having no intermediate shared layers. We further examine whether this joint improvement depends on where the sharing occurs. To isolate the hierarchical sharing effect from the training recipe and initialization, the ablation in Figure 5 compares Hourglass without CocoGrad against a partially shared variant and a shared-then-decoupled variant. All variants use the same pretrained initialization, data, and training recipe. Hourglass without CocoGrad achieves the best final GenEval and MME-S results among these variants. Thus, intermediate sharing benefits both generation and understanding relative to the evaluated alternatives.

Q1: Why Q/K specifically ?

Our design rationale is that Q/K primarily parameterize attention routing and may capture relational patterns transferable between understanding and generation, whereas task-specific V and FFN modules can specialize the content extraction and nonlinear feature processing required by each objective. Understanding and generation therefore benefit from sharing some relational modeling while retaining different ways of processing features. To test this hypothesis, we also conducted the ablation experiment to seek empirical evidence. We compared QK-only, QKV-only, and full sharing in the intermediate layers under the same training and evaluation settings. As shown in Table R4, Q/K-only outperforms Q/K/V-only and full sharing on both benchmarks, providing empirical support for keeping V and FFN task-specific.

Table R4. Sharing-component ablation on Hourglass-3B.

Shared componentGenEval ↑\uparrowMME-S ↑\uparrow
Q/K-only0.632006
Q/K/V-only0.611951
full sharing0.551796

Q2: Evaluation on the proprietary benchmark.

Thank you for suggesting comparisons with stronger UMMs on Unison and Uni-MMMU. We evaluated Hourglass-3B and Hourglass-7.5B on both benchmarks and report the completed results below. In attempting to evaluate Emu3.5 and Lance, we identified the following compatibility and resource constraints in applying the Unison and Uni-MMMU evaluation protocols to these models:

  • Input requirements: Some Unison GGU and ME subtasks require multiple input images, while the available Lance understanding interface accepts only one. A score on the supported subset would not be comparable with full-benchmark scores.
  • Memory requirements: We encountered out-of-memory errors when evaluating Emu3.5 on both Unison and Uni-MMMU using eight H100 GPUs.
  • Inference time: Based on measured throughput from the samples we could run, a full Unison evaluation of either Emu3.5 or Lance would require more than 75 hours using eight H100 GPUs; a full Uni-MMMU evaluation of Emu3.5 would require more than 78 hours on the same hardware.

We therefore report our completed Hourglass results alongside the baseline results published in the respective benchmark papers.

Table R5. Results on Unison. IC: Internal Consistency; UGG: Understanding-Guided Generation; GGU: Generation-Guided Understanding; ME: Mutual Enhancement. BAGEL and UniWorld results are reported in the Unison paper; Hourglass results are evaluated by us.

ModelParamsIC (Uni.) ↑UGG (Uni.) ↑GGU (Uni.) ↑ME (Uni.) ↑Overall ↑
BAGEL14B80.367.932.032.553.2
UniWorld19B65.144.926.931.342.1
Lance3B63.2--------
Hourglass3B56.330.524.515.931.9
Hourglass7.5B70.031.526.423.837.9

On Unison, scaling Hourglass from 3B to 7.5B improves all four subtasks and the overall score. Hourglass-7.5B exceeds UniWorld on IC (70.0 versus 65.1), although its overall score remains below UniWorld and BAGEL. These models differ substantially in training-data scale, training strategies, model size, and compute, which may contribute to the performance gaps. We therefore present these results as reference points rather than controlled comparisons.

Table R6. Results on Uni-MMMU. BAGEL and OmniGen2 results are reported in the Uni-MMMU paper; Hourglass results are evaluated by us.

ModelParamsMathSliding ISliding TMaze IMaze TSci. TSci. R
BAGEL14B--0.0/0.05.6/1.20.0/0.00.0/0.057.363.1
OmniGen27B--0.0/0.00.0/0.00.0/0.00.0/0.033.142.0
Lance3B1.430.0/0.00.0/0.00.0/0.00.0/0.0----
Hourglass3B2.140.0/0.00.0/0.00.0/0.00.0/0.028.735.0
Hourglass7.5B6.430.0/0.00.0/0.00.0/0.00.0/0.046.559.2

On Uni-MMMU, Hourglass-7.5B improves over Hourglass-3B on Math, Sci. T, and Sci. R. Both Hourglass models exceed Lance-3B on Math (1.43). On the two science subtasks, Hourglass-7.5B outperforms OmniGen2 but remains below BAGEL. Sliding and maze tasks remain challenging: all listed models score zero on Sliding I and both Maze subtasks, while only BAGEL reports a nonzero result on Sliding T.

These comparisons provide useful reference points, but they are not controlled comparisons because the models differ in parameter count, training data, computational budget, and pretraining setup. If the reviewer has reproduced reliable Emu3.5 or Lance results under comparable evaluation protocols, we would be grateful if those results could be shared as additional reference points for comparison.

Q3: Release of code, models.

The core code supporting the main contributions of this work is available in our Anonymous code. Upon acceptance, we will publicly release the Hourglass model weights and the full codebase, including the model architecture, the CocoGrad implementation, inference scripts, and usage examples for visual understanding, image generation, and image editing. These resources will enable researchers to reproduce our results and directly run and evaluate the complete model.

We hope these clarifications address your concerns. Thank you again for your insightful and constructive comments.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer 3J9v,

With the discussion period coming to a close, we would like to take this opportunity to ask if you have any further questions or concerns regarding our responses that we could help clarify.

We truly appreciate the time and effort you have invested in reviewing our submission. We have carefully reflected on your feedback and done our best to address all the points you raised. Your thoughtful comments and suggestions have been very helpful in improving our work.

If there is anything that remains unclear or requires further discussion, we would be more than happy to provide additional clarification.

Thank you once again for your valuable feedback and consideration.

Reviewer_jkUkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Hourglass, a unified model for visual understanding, image generation, and editing. It shares Q/K projections in intermediate layers while keeping other components task-specific. It also introduces CocoGrad, which separates aligned and conflicting gradient coordinates and adjusts updates only in the conflicting group. Experiments on two model sizes report competitive results across the three tasks.

Critical strengths

1.This paper considers both parameter sharing and gradient interference. This is a sensible approach to balancing understanding and generation in one model.

  1. The evaluation covers two model sizes and three task types. The full-pipeline comparison with and without CocoGrad is particularly useful. It helps isolate CocoGrad's contribution from the training recipe.
Critical weaknesses
  1. The paper studies where to share and how many layers to share. It does not compare QK-only sharing with QKV or FFN sharing. This leaves a central architectural choice supported mainly by intuition.
  2. CocoGrad shows only modest improvements over PCGrad and CAGrad, and the lack of repeated runs or error bars makes it difficult to assess whether these gains are statistically reliable and consistent across runs.
Questions for authors

1.Do you have matched comparisons between QK-only, QKV, and FFN sharing? What evidence supports keeping V and FFN separate? 2. Why does Eq. (8) exclude the gains from the collaborative coordinates? Consider weighted task gradients g1=(2,1)g_1=(2,1) and g2=(1,−2)g_2=(1,-2). Their sum is d0=(3,−1)d_0=(3,-1). With c=1c=1 and ignoring ϵ\epsilon, CocoGrad gives d=(3,0)d=(3,0). The full first-order descent gains, measured by gi⊤dg_i^\top d, change from (5,5)(5,5) to (6,3)(6,3). The local objective improves, while the overall worst-task gain decreases. This does not contradict the convergence result. What justifies optimizing the conflicting subspace without accounting for the gains from the collaborative subspace? 3. How many independent runs were used to compare CocoGrad, PCGrad, and CAGrad? Please report the variation across runs, if available.

3 comments

Thank you for the constructive review and for recognizing our joint treatment of parameter sharing and gradient interference. Below, we address your concerns in detail.

W1 / Q1: Why share Q/K while keeping V and FFN task-specific?

Q/K primarily parameterize attention routing and may capture relational patterns transferable between understanding and generation, whereas task-specific V and FFN modules can specialize the content extraction and nonlinear feature processing required by each objective. To test the hypothesis, we also conducted an ablation study to seek empirical evidence. We compared Q/K-only, Q/K/V-only, and Q/K/V and FFN sharing in the intermediate layers using the same shortened training protocol as the sharing-depth ablation in Table 3, with all other settings held fixed. As shown in Table R1, Q/K-only sharing outperforms Q/K/V-only sharing on GenEval and MME-S, while Q/K/V and FFN sharing performs worse still, suggesting that retaining task-specific V and FFN transformations is beneficial. We will add this experiment to the paper.

Table R1. Sharing-component ablation on Hourglass-3B.

Shared componentGenEval ↑\uparrowMME-S ↑\uparrow
Q/K-only0.632006
Q/K/V-only0.611951
Q/K/V and FFN sharing0.551796

W2 / Q3: Are the gains over PCGrad and CAGrad consistent across independent runs?

Thank you for raising this concern. We independently trained CocoGrad, PCGrad, and CAGrad with three random seeds (40, 41, and 42), using the same training and evaluation settings for all runs. Seed 40 was used for the results originally reported in the paper. Table R2 reports the results of each run, together with the mean and standard deviation. CocoGrad ranks first on both MME-S and GenEval for all three seeds. Its MME-S score is 1412.3 ± 5.9, compared with 1380.3 ± 6.8 for PCGrad and 1350.7 ± 8.5 for CAGrad; its GenEval score is 0.8463 ± 0.0025, compared with 0.8350 ± 0.0040 and 0.8317 ± 0.0050, respectively. These results show that the observed ranking is consistent across the tested seeds. We will add the per-seed results and experimental protocol to the appendix.

Table R2. Performance across three random seeds (40, 41, and 42). Method rankings are consistent across seeds.

MME-S

MethodRun 1Run 2Run 3MeanSTD
CAGrad1357134113541350.78.5
PCGrad1378137513881380.36.8
CocoGrad1419140814101412.35.9

GenEval

MethodRun 1Run 2Run 3MeanSTD
CAGrad0.8370.8310.8270.83170.0050
PCGrad0.8390.8350.8310.83500.0040
CocoGrad0.8460.8440.8490.84630.0025

Q2: Why does Eq. (8) exclude the gains from collaborative coordinates?

We sincerely thank the reviewer for providing this illustrative example. We would like to clarify the design intent behind Eq. (8).

Table R3. Illustration of the difference between global max-min objective and conflict subspace coordination (CocoGrad) using the reviewer's example.

Methodβ\betaCollaborative-subspace gainConflicting-subspace gainFull gain
Global max-min00(6,3)(6,3)(−1,2)(-1,2)(5,5)(5,5)
CocoGrad13\frac{1}{3}(6,3)(6,3)(0,0)(0,0)(6,3)(6,3)

For the reviewer's gradients g1=(2,1)g_1=(2,1) and g2=(1,−2)g_2=(1,-2), Eq. (6) gives d(β)=(3,−1+3β)d(\beta)=(3,-1+3\beta). If the cooperative-subspace gain is included in Eq. (8), the two task gains would be g1⊤d(β)=5+3βg_1^\top d(\beta)=5+3\beta and g2⊤d(β)=5−6βg_2^\top d(\beta)=5-6\beta. Their minimum is maximized where the two gains are equal: 5+3β=5−6β5+3\beta=5-6\beta, giving β=0\beta=0, which means that no correction is applied in the conflicting subspace. Consequently, the conflicting-subspace gain remains (−1,2)(-1,2): Task 1 has a negative first-order gain in this subspace, although this negative contribution is masked by its larger positive gain in the collaborative subspace. Thus, the global objective would be dominated by the aligned but asymmetric cooperative signals among tasks, overlooking the conflicting directions that CocoGrad is specifically designed to handle. In contrast, CocoGrad operates specifically on the conflicting subspace, yielding a conflicting subspace gain of (0,0)(0, 0), which prevents Task 1 from being adversely updated in the conflicting direction. Without this correction, Task 1 would be adversely affected by the negative component in this update, whereas Task 2 would benefit from the opposite direction.

After coordination, the full gain is (6,3)(6, 3), which is positive for both tasks. Under our sign convention, this means that both objectives admit a first-order decrease for a sufficiently small step size. Moreover, Lemma 1 and Appendix B prove that the overall process converges to a first-order stationary point of the aggregated objective at a rate of O(1/T)\mathcal{O}(1/T). The ablation results in Table 4 further provide empirical support for explicitly partitioning the cooperative and conflicting subspaces.

We hope these clarifications address your concerns. Thank you again for your insightful and constructive comments.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer jkUk,

With the discussion period coming to a close, we would like to take this opportunity to ask if you have any further questions or concerns regarding our responses that we could help clarify.

We truly appreciate the time and effort you have invested in reviewing our submission. We have carefully reflected on your feedback and done our best to address all the points you raised. Your thoughtful comments and suggestions have been very helpful in improving our work.

If there is anything that remains unclear or requires further discussion, we would be more than happy to provide additional clarification.

Thank you once again for your valuable feedback and consideration.

Comments on the paper

5 comments

Thank you for the careful review and for recognizing the value of selective Q/K sharing and conflict-aware optimization. We agree that the relationships to Uni-X and MGDA require a clearer discussion, and that the evidence for CocoGrad's practical gains should be presented with appropriate qualifications. We respond point by point below.

W1: Relationship to Uni-X and the Choice of Shared Components.

Thank you for highlighting the close relationship between Uni-X and our work. Although our architecture looks similar to Uni-X, they differ in the details of that architecture, the rationale behind it, and the objective it serves. Compared with Uni-X, Hourglass differs in three respects. First, the two branches receive different inputs. In Uni-X, one branch receives text tokens and the other receives visual tokens, whereas in Hourglass, the inputs to both the understanding and generation branches include visual tokens. Second, the intermediate-layer sharing schemes differ: Hourglass shares only the Q/K projections in these layers, whereas Uni-X shares the intermediate layers in full. Third, the methods address gradient conflicts differently. Uni-X mitigates conflicts between the text and visual modalities by sharing only the intermediate layers, whereas Hourglass uses CocoGrad to address conflicts between the two tasks on the shared parameters. We will add the citation and a direct comparison in the related work.

To test this sharing-component, we also conducted the ablation experiment to seek empirical evidence. We compared QK-only, QKV-only, and full intermediate-layer sharing under the same initialization, data, training schedule, and evaluation protocol. The results are presented in Table R1. QK-only outperforms QKV-only and full sharing on both benchmarks, providing empirical support for keeping V and FFN task-specific.

Table R1. Sharing-component ablation on Hourglass-3B.

Shared componentParamsGenEval ↑\uparrowMME-S ↑\uparrow
QK-only3.06B0.632006
QKV-only3.03B0.611951
full sharing2.53B0.551796

W2: Relationship to MGDA and the Practical Benefit of CocoGrad.

Thank you for this insightful observation. We would like to clarify that the design of CocoGrad is not derived from MGDA. Our gradient measurements indicate substantial optimization interference in UMM joint training: during training, the conflicting parts of the gradients for understanding and generation have a high norm proportion (as shown in Figure 2(b)). This issue motivated our work. To the best of our knowledge, this is the first gradient surgery work targeting the multi-task training problem in UMMs.

We agree that in the limit of c=1c=1 and ignoring ϵ\epsilon, the coordination of CocoGrad in the conflicting subspace has a mathematical connection to the two-task minimum-norm convex combination in MGDA. However, this connection is restricted to the mutually exclusive (conflicting) subspace and does not hold in the collaborative subspace. Furthermore, our ablation study (Table 4) and the explanation provided in our response to Reviewer jkUk Question 2 demonstrate that global reweighting degrades performance, as it injects noise into the naturally consistent descending signals.

Additionally, the supplementary material of CAGrad also notes that both MGDA and CAGrad require solving Quadratic Programming problems. However, MGDA minimum-norm search requires solving an optimization problem at every step, and its Pareto-optimal convergence may be overly conservative, leading to slower learning.

To empirically demonstrate the superiority of our method over MGDA, we conducted the same experiments as in Table 6. The results are shown in Table R2. CocoGrad outperforms MGDA in both computational time and performance. We will include the comparison results with MGDA in the revised version.

Table R2. Comparison of performance and computational time.

MethodMME-S ↑\uparrowGenEval ↑\uparrowTime ↓\downarrow
CAGrad13570.8370.0831
MGDA12880.8040.2342
CocoGrad14190.8460.0617

Regarding your question about uncertainty estimation, we would like to clarify our evaluation protocol and provide a stability analysis for the full-process ablation in Table 7.

First, we clarify the nature of our evaluation metrics:

  1. Generation tasks (e.g., GenEval, ImgEdit): Image generation evaluation is deterministic. With fixed prompts and random seeds, the generated outputs are identical; hence, repeated runs yield exactly the same scores with zero variance.
  2. Understanding tasks (e.g., MME-S, MMB, MMMU): Since the understanding branch uses autoregressive sampling (involving top-k, top-p, and temperature parameters), there are fluctuations across different runs.

When conducting the numerical tests for Table 7, the previously reported values were obtained by taking the maximum value over 3 runs, which facilitates a straightforward comparison of the final model with other models. Now, to alleviate concerns, we present the results of all 3 runs in Table R3. As shown in the table, within the variance range, CocoGrad stably improves performance across all runs. We will add the mean, variance of multiple runs, and the evaluation protocol to the appendix.

Table R3. Stability analysis of the full-process ablation over 3 runs on MME-S, MMB, and MMMU.

MME-S ↑\uparrow

MethodRun 1Run 2Run 3MeanSTD
Hourglass213520852127211626.86
Hourglass without CocoGrad209620692023206336.91

MMBench ↑\uparrow

MethodRun 1Run 2Run 3MeanSTD
Hourglass79.679.379.379.40.17
Hourglass without CocoGrad79.178.879.079.00.15

MMMU ↑\uparrow

MethodRun 1Run 2Run 3MeanSTD
Hourglass47.445.646.846.60.92
Hourglass without CocoGrad44.343.943.243.80.56

W2: Relationship to MGDA and the Practical Benefit of CocoGrad.

Regarding the end-to-end training overhead, we extracted the average time per 100 training steps from our logs across 30,000 steps. The results are shown in Table R4.

First, CocoGrad achieves better performance with a slight time advantage compared to CAGrad. However, the reviewer may also notice that all gradient surgery methods incur a significant time increase compared to Vanilla. We provide the following explanation:

The main training overhead comes from the forward and backward passes, rather than the optimizer updates. To perform gradient surgery, we must compute the gradients for each task separately. In the pretraining framework FSDP, due to the lack of specialized efficient backward propagation tools, we have to perform the backward pass twice for each branch, which leads to the increased time. We firmly believe that this implementation can be greatly optimized in terms of computational time by writing specialized low-level operators. This represents an interesting system-level research direction, but it is beyond the scope of the algorithmic discussion in this paper.

Table R4. Comparison of training time overhead per 100 steps.

MethodVanillaPCGradCAGradCocoGrad
Time (s / 100 steps) ↓\downarrow147254262259

We hope these clarifications address your concerns. Thank you again for your insightful and constructive comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.