- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Hourglass, a unified multimodal models(UMMs) for visual understanding and generation. First, it introduces a hierarchical decoupled-shared-decoupled architecture, which preserves task-specific processing in shallow and deep layers while enabling selective parameter sharing in intermediate layers. Second, it proposes CocoGrad, a conflict-aware gradient surgery method that partitions shared-parameter gradients into collaborative and conflicting subspaces and adaptively coordinates updates in the conflicting subspace. Experiments on visual understanding, text-to-image generation, and image editing benchmarks demonstrate competitive performance, with ablation studies evaluating the effectiveness of the proposed designs.
Critical strengths
1.The authors employ a decoupled-shared-decoupled model architecture and update the Q and K parameters in the middle layers by jointly training on losses from both understanding and generation tasks—an approach that is both theoretically sound and intriguing.
2.Conflict-Aware Gradient Optimization. The paper introduces CocoGrad to address gradient interference between visual understanding and generation during joint training. Experimental results show that CocoGrad improves performance over vanilla joint optimization and several existing gradient surgery methods on the evaluated benchmarks.
3.Coherent Framework Design. The hierarchical sharing architecture and CocoGrad are complementary, addressing parameter sharing and optimization interference at the architectural and optimization levels, respectively. Their integration provides a coherent framework for jointly modeling visual understanding and generation.
Critical weaknesses
1.The core insight underlying the hierarchical decoupled-shared-decoupled architecture substantially overlaps with prior UMM studies. In particular, Uni-X (ICLR 2026) has already identified stronger gradient conflicts in shallow and deep layers and proposed a two-end-separated, middle-shared architecture. HBridge (CVPR 2026) similarly explores selective intermediate-layer interactions between understanding and generation experts, while DIVA (ICML 2026) investigates representation divergence and the separation of shared and task-specific information in UMMs. However, these closely related works are not adequately discussed or compared against, making it difficult to assess the architectural novelty of Hourglass beyond its specific Q/K sharing implementation.
2.Fair-compute comparisons are unclear. Training tokens, GPU-hours/FLOPs, and data mixtures for understanding and generation are only partially described; some tables contain artifacts, and it is hard to judge efficiency vs. recent strong UMMs (e.g., EMMA, Lance, NextFlow,Emu3.5).
3.The key ideas behind CocoGrad have substantial connections to existing multi-task optimization methods. Coordinate-wise gradient sign agreement has been explored in GradDrop (NeurIPS 2020), while constrained worst-task improvement is closely related to CAGrad (NeurIPS 2021). The proposed method appears to combine existing gradient partitioning and balancing principles rather than introducing a fundamentally new optimization mechanism.
4.Although the paper reports competitive results on visual understanding, text-to-image generation, and image editing benchmarks, these tasks are primarily evaluated independently. The experiments therefore do not directly demonstrate the claimed beneficial collaboration between understanding and generation. Moreover, the first two training stages focus exclusively on generation while freezing the understanding branch, making it difficult to disentangle the effects of hierarchical sharing from those of the training recipe and pretrained initialization.
Reference [1] Uni-X: mitigating modality conflict with a two-end-separated architecture for unified multimodal models [2] Hbridge: H-shape bridging of heterogeneous experts for unified multimodal understanding and generation [3] DIVA: Harnessing the Representation Divergence in Unified Multimodal Models for Mutual Reinforcement [4] Just pick a sign: Reducing gradient conflict in deep networks with gradient sign dropout [5] Conflict-Averse Gradient Descent for Multi-task Learning
Questions for authors
1.Why share QK specifically? Do you have diagnostics (e.g., attention map similarity, head importance, CKA) comparing alternatives such as shared V, partial FFN sharing, or shared attention maps with decoupled projections?
2.Could you provide some performance metrics for Hourglass—compared to stronger baselines(e.g. Emu3.5, Lance)—on proprietary benchmarks(e.g. Unison,Uni-MMMU) used by UMMs?
3.Will code/models be released? If not, can you provide more implementation details?In addition,from my personal perspective, wouldn't it be more compelling if a complete model were made accessible—either by being opened up or by providing usable interfaces—so that people could invoke it directly?
4 comments
W3: Relationship of CocoGrad to GradDrop and CAGrad.
We thank the reviewer for highlighting these connections. We agree that CocoGrad shares conceptual similarities with existing gradient-manipulation methods. However, it differs in where and how gradient coordination is performed. Table R3 summarizes these distinctions.
CocoGrad has two main advantages. First, it preserves information contained in conflicting gradient components rather than simply discarding them. Second, by separating the two subspaces, it prevents cooperative signals from interfering with conflicting signals, as illustrated in our response to Reviewer jkUk’s Question 2.
More importantly, our numerical analysis identifies a potentially important source of optimization difficulty in joint UMM training: the conflicting components of the understanding and generation gradients account for a substantial proportion of their gradient norms, as shown in Fig. 2(b). This observation motivates our architectural design of hierarchical sharing and localized gradient surgery in the conflicting subspace. To the best of our knowledge, CocoGrad is among the first methods to explicitly analyze and address gradient conflicts between understanding and generation objectives in joint UMM training.
Table R3. Comparison of gradient manipulation methods.
| Dimension | GradDrop | CAGrad | CocoGrad (Ours) |
|---|---|---|---|
| Masking Strategy | Random masking, sampled by dimension | No masking; operates at the global parameter level | Deterministic selection based on dimension directions |
| Conflict Handling | Directly discards | Solves a Quadratic Programming (QP) problem to achieve global compromise | Introduces a structured gradient update formula Eq.6, achieving subspace weighted coordination |
W4: Evidence for beneficial collaboration and the effect of the training recipe.
Thank you for this important observation. The sharing-depth ablation in Table 3 provides evidence beyond the separately reported task benchmarks: with CocoGrad disabled, 12 intermediate shared layers improve both final GenEval (0.63 vs. 0.55) and MME-S (2006 vs. 1976) relative to having no intermediate shared layers. We further examine whether this joint improvement depends on where the sharing occurs. To isolate the hierarchical sharing effect from the training recipe and initialization, the ablation in Figure 5 compares Hourglass without CocoGrad against a partially shared variant and a shared-then-decoupled variant. All variants use the same pretrained initialization, data, and training recipe. Hourglass without CocoGrad achieves the best final GenEval and MME-S results among these variants. Thus, intermediate sharing benefits both generation and understanding relative to the evaluated alternatives.
Q1: Why Q/K specifically ?
Our design rationale is that Q/K primarily parameterize attention routing and may capture relational patterns transferable between understanding and generation, whereas task-specific V and FFN modules can specialize the content extraction and nonlinear feature processing required by each objective. Understanding and generation therefore benefit from sharing some relational modeling while retaining different ways of processing features. To test this hypothesis, we also conducted the ablation experiment to seek empirical evidence. We compared QK-only, QKV-only, and full sharing in the intermediate layers under the same training and evaluation settings. As shown in Table R4, Q/K-only outperforms Q/K/V-only and full sharing on both benchmarks, providing empirical support for keeping V and FFN task-specific.
Table R4. Sharing-component ablation on Hourglass-3B.
| Shared component | GenEval | MME-S |
|---|---|---|
| Q/K-only | 0.63 | 2006 |
| Q/K/V-only | 0.61 | 1951 |
| full sharing | 0.55 | 1796 |
Q2: Evaluation on the proprietary benchmark.
Thank you for suggesting comparisons with stronger UMMs on Unison and Uni-MMMU. We evaluated Hourglass-3B and Hourglass-7.5B on both benchmarks and report the completed results below. In attempting to evaluate Emu3.5 and Lance, we identified the following compatibility and resource constraints in applying the Unison and Uni-MMMU evaluation protocols to these models:
- Input requirements: Some Unison GGU and ME subtasks require multiple input images, while the available Lance understanding interface accepts only one. A score on the supported subset would not be comparable with full-benchmark scores.
- Memory requirements: We encountered out-of-memory errors when evaluating Emu3.5 on both Unison and Uni-MMMU using eight H100 GPUs.
- Inference time: Based on measured throughput from the samples we could run, a full Unison evaluation of either Emu3.5 or Lance would require more than 75 hours using eight H100 GPUs; a full Uni-MMMU evaluation of Emu3.5 would require more than 78 hours on the same hardware.
We therefore report our completed Hourglass results alongside the baseline results published in the respective benchmark papers.
Table R5. Results on Unison. IC: Internal Consistency; UGG: Understanding-Guided Generation; GGU: Generation-Guided Understanding; ME: Mutual Enhancement. BAGEL and UniWorld results are reported in the Unison paper; Hourglass results are evaluated by us.
| Model | Params | IC (Uni.) ↑ | UGG (Uni.) ↑ | GGU (Uni.) ↑ | ME (Uni.) ↑ | Overall ↑ |
|---|---|---|---|---|---|---|
| BAGEL | 14B | 80.3 | 67.9 | 32.0 | 32.5 | 53.2 |
| UniWorld | 19B | 65.1 | 44.9 | 26.9 | 31.3 | 42.1 |
| Lance | 3B | 63.2 | -- | -- | -- | -- |
| Hourglass | 3B | 56.3 | 30.5 | 24.5 | 15.9 | 31.9 |
| Hourglass | 7.5B | 70.0 | 31.5 | 26.4 | 23.8 | 37.9 |
On Unison, scaling Hourglass from 3B to 7.5B improves all four subtasks and the overall score. Hourglass-7.5B exceeds UniWorld on IC (70.0 versus 65.1), although its overall score remains below UniWorld and BAGEL. These models differ substantially in training-data scale, training strategies, model size, and compute, which may contribute to the performance gaps. We therefore present these results as reference points rather than controlled comparisons.
Table R6. Results on Uni-MMMU. BAGEL and OmniGen2 results are reported in the Uni-MMMU paper; Hourglass results are evaluated by us.
| Model | Params | Math | Sliding I | Sliding T | Maze I | Maze T | Sci. T | Sci. R |
|---|---|---|---|---|---|---|---|---|
| BAGEL | 14B | -- | 0.0/0.0 | 5.6/1.2 | 0.0/0.0 | 0.0/0.0 | 57.3 | 63.1 |
| OmniGen2 | 7B | -- | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 33.1 | 42.0 |
| Lance | 3B | 1.43 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | -- | -- |
| Hourglass | 3B | 2.14 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 28.7 | 35.0 |
| Hourglass | 7.5B | 6.43 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 0.0/0.0 | 46.5 | 59.2 |
On Uni-MMMU, Hourglass-7.5B improves over Hourglass-3B on Math, Sci. T, and Sci. R. Both Hourglass models exceed Lance-3B on Math (1.43). On the two science subtasks, Hourglass-7.5B outperforms OmniGen2 but remains below BAGEL. Sliding and maze tasks remain challenging: all listed models score zero on Sliding I and both Maze subtasks, while only BAGEL reports a nonzero result on Sliding T.
These comparisons provide useful reference points, but they are not controlled comparisons because the models differ in parameter count, training data, computational budget, and pretraining setup. If the reviewer has reproduced reliable Emu3.5 or Lance results under comparable evaluation protocols, we would be grateful if those results could be shared as additional reference points for comparison.
Q3: Release of code, models.
The core code supporting the main contributions of this work is available in our Anonymous code. Upon acceptance, we will publicly release the Hourglass model weights and the full codebase, including the model architecture, the CocoGrad implementation, inference scripts, and usage examples for visual understanding, image generation, and image editing. These resources will enable researchers to reproduce our results and directly run and evaluate the complete model.
We hope these clarifications address your concerns. Thank you again for your insightful and constructive comments.
Follow-up on Our Rebuttal
Dear Reviewer 3J9v,
With the discussion period coming to a close, we would like to take this opportunity to ask if you have any further questions or concerns regarding our responses that we could help clarify.
We truly appreciate the time and effort you have invested in reviewing our submission. We have carefully reflected on your feedback and done our best to address all the points you raised. Your thoughtful comments and suggestions have been very helpful in improving our work.
If there is anything that remains unclear or requires further discussion, we would be more than happy to provide additional clarification.
Thank you once again for your valuable feedback and consideration.
Thank you for your thoughtful and insightful feedback. We appreciate your recognition of the complementary roles of hierarchical sharing and CocoGrad. Your comments help us better distinguish our contribution from related work and clarify the evidence supporting it. We respond point by point below.
Thank you for highlighting these closely related works. Although Hourglass and these works look similar in architecture, they differ in the details of that architecture, the rationale behind it, and the objective it serves. The core code implementing the contributions of our paper is available at this link for your inspection Anonymous code. Our implementation is fundamentally different from those of the other methods.
Compared with Uni-X, Hourglass differs in three respects. First, the two branches receive different inputs. In Uni-X, one branch receives text tokens and the other receives visual tokens, whereas in Hourglass, the inputs to both the understanding and generation branches include visual tokens. Second, the intermediate-layer sharing schemes differ: Hourglass shares only the Q/K projections in these layers, whereas Uni-X shares the intermediate layers in full. Third, the methods address gradient conflicts differently. Uni-X mitigates conflicts between the text and visual modalities by sharing only the intermediate layers, whereas Hourglass uses CocoGrad to address conflicts between the two tasks on the shared parameters.
Compared with HBridge, Hourglass differs in two respects. First, the scope of interaction differs: HBridge connects its understanding and generation experts only at selected intermediate layers, whereas Hourglass places tokens from both task paths in a unified sequence for joint self-attention at every layer. Second, their parameter-sharing schemes differ: HBridge aligns the Q/K/V features of its heterogeneous experts for intermediate-layer attention exchange, whereas Hourglass shares only the Q/K projections between its two task paths in the intermediate layers.
Compared with DIVA, Hourglass differs in its approach to architectural sharing. DIVA is a post-training framework applied to existing single-backbone unified models, whereas Hourglass explicitly separates the understanding and generation paths in shallow and deep layers and shares only their Q/K projections in intermediate layers.
Beyond these methodological differences, Table R1 compares the performance of Hourglass with Uni-X, HBridge, and Nexus-Gen + DIVA. Hourglass-7.5B achieves the best reported results on most visual understanding and generation benchmarks. Hourglass-3B also outperforms Uni-X on every metric reported for both models, suggesting that the proposed design is effective across model scales.
Table R1. Comparison with these related works on visual understanding, image generation, and image editing benchmarks.
We will add these references to the Related Work section and include their results in Table 2.
Thank you for raising this concern. To facilitate verification and comparison, we have added the Table R2, which reports the dataset pool sizes, actual task sampling ratios, training tokens, hardware, and GPU-hours for the 3B and 7.5B models at each training stage.
Table R2. Training Data and Compute Details.