acceptodds
Under review as a conference paper at ICLR 2027

What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

Abstract

Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding. However, existing generation-assisted understanding methods mainly rely on generative supervision or explicit generation and have not systematically explored direct use of native generation-side representations. To address this gap, we systematically analyze what generation-side representations offer for fine-grained understanding, finding that they are more sensitive to fine-grained semantic changes, capture task-relevant changes missed by understanding-side representations, and depend on correct spatial organization for this advantage. Guided by these findings, we design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. It compresses generation-side visual representations into compact \(G\) tokens that preserve fine-grained visual details, enabling task-conditioned retrieval by the understanding pathway. Experiments show that GAFU consistently outperforms SFT baselines trained on the same data across diverse visual understanding tasks and UMM architectures, with maximum relative improvements of 20.00%, 17.94%, and 8.62% on VStar-Bench, OmniSpatial, and VSR-Bench, respectively. Overall, GAFU converts complementary information from the generation pathway into gains in fine-grained visual understanding, providing an effective path toward synergy between generation and understanding in UMMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Models, Fine-Grained Visual Understanding, Understanding–Generation Synergy, Post-Training, Visual Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LpnCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GAFU, which improves the recognition of local details and spatial relationships on the understanding side by reading visual features from the generation side of a unified multimodal model. The method first compresses the generation-side features into 256 tokens and preserves information via reconstruction, spatial constraints and relationship alignment. Then the text states on the understanding side query these features through cross-attention. The authors analyze the sensitivity and complementarity of the two types of representations across six models, and validate the method on three models with five understanding benchmarks. Across all 15 combinations of models and benchmarks, GAFU outperforms SFT trained on the same data, achieving an average improvement of approximately 3.56–5.98 percentage points over the five benchmarks.

Critical strengths
  1. The idea of directly reading generative-side features to assist comprehension is intriguing. The authors analyzed the differences between two types of representations across six models, and incorporated pixel perturbation control and spatial shuffling experiments, providing relatively concrete evidence for method design.
  2. The method outperforms SFT on the same dataset across three different models and five tests, with consistent results. The ablation studies on feature alignment, injection approaches and training objectives are also comprehensive, revealing the impact of each design choice on the final performance.
Critical weaknesses
  1. The coverage of fine-grained tasks is incomplete. The paper mainly tests objects, attributes and spatial relations, and does not report OCR-related results. Text recognition also relies on local details and serves as a valuable scenario for verifying the proposed method. It remains unclear whether the performance gains can be extended to such tasks.
  2. The paper emphasizes the preservation of original understanding capabilities, yet it does not report broader general understanding tests, which is of great interest to me. In addition, the paper does not report the generation performance after fine-tuning, so it is unknown whether the final model retains its original generation capability.
Questions for authors
  1. How does GAFU perform on OCR tasks? Are there test results such as TextVQA, DocVQA or OCRBench? These results will help determine whether the method can handle fine-grained information beyond object attributes and spatial relationships.
  2. How does the fine-tuned model perform on general understanding tests? Does it suffer from capability degradation compared with the original model and SFT using the same data?
  3. Can the same fine-tuned model retain its original generation capability? How are the understanding-side LoRA and newly added modules used during generation, and are there corresponding generation evaluation results?
3 comments

We thank Reviewer LpnC for the careful review and for identifying the need to evaluate OCR coverage, general understanding, and post-fine-tuning generation. We address these points below.

1. OCR Task Performance (Weakness 1 & Question 1)

Using the existing general-purpose training checkpoints, we add an evaluation on OCRBench v2, with the results shown below.

ModelBaseSFTGAFU
BAGEL23.3025.0526.82
Janus-Pro11.7213.5114.74
Lance5.457.718.86

GAFU outperforms SFT by 1.77, 1.23, and 1.15 points on BAGEL, Janus-Pro, and Lance, respectively, while also exceeding Base across all three models. These results extend the observed benefits of generation-side information to text-related fine-grained understanding.

2. General Understanding Performance (Weakness 2 & Question 2)

To assess whether fine-tuning affects general understanding, we evaluate the three models on MMStar, MMMU, and MME-RealWorld Lite, comparing GAFU with the original models and SFT trained on the same data.

ModelMMStar ↑MMMU ↑MME-RealWorld Lite ↑
BAGEL61.1850.9929.23
BAGEL (SFT)62.3950.0629.91
BAGEL (GAFU)62.5451.4630.64
Janus-Pro35.4733.2620.06
Janus-Pro (SFT)36.1331.9726.84
Janus-Pro (GAFU)36.5233.6125.64
Lance24.7228.4726.78
Lance (SFT)25.5026.3726.52
Lance (GAFU)26.3928.4727.05

GAFU improves MMStar performance across all three model families and generally maintains or improves performance on MMMU and MME-RealWorld Lite. The overall results compare favorably with both Base and SFT trained on the same data. These findings indicate that GAFU preserves general understanding capabilities while improving fine-grained visual performance.

3. Generation Capability and Module Usage (Weakness 2 & Question 3)

GAFU modifies understanding-side inputs and adds cross-attention after each Transformer layer while keeping the generation expert unchanged. To assess its effect on generation, we evaluate the same BAGEL understanding checkpoint on G-Edit and ImgEdit with all GAFU modules enabled, including the generation-representation compression encoder, cross-attention injection modules, and understanding-pathway LoRA.

BenchmarkBaseSFTGAFU
G-Edit6.65576.59826.6631
ImgEdit2.791172.794612.79775

GAFU performs nearly identically to Base and SFT on G-Edit and ImgEdit, indicating that it preserves image-editing performance. We will add the above evaluations and the module-usage settings to the revised manuscript.

We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.

Reviewer_LpnCOpenReview ↗

Thanks for your reply

Thank you for the additional experiments and clear explanations. The OCRBench v2, general understanding, and generation results address my main concerns. I will maintain my current score.

We sincerely thank the reviewer for the positive feedback. We greatly appreciate your time and support.

Reviewer_7V6YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GAFU, a framework that improves fine-grained visual understanding by incorporating generation-side representations into the understanding pathway of unified multimodal models. It compresses these representations into G tokens and uses task-conditioned cross-attention and preference learning to encourage their use.

Critical strengths
  1. The paper analyzes the sensitivity, complementarity, and spatial dependence of generation-side representations, providing useful evidence for their potential value in fine-grained understanding.
  2. GAFU consistently outperforms same-data SFT baselines across three models and five benchmarks, demonstrating effectiveness across different architectures and tasks.
Critical weaknesses
  1. The method appears overly complex, involving compression pretraining, additional modules, alignment losses, preference objectives, and several training strategies. This complexity raises concerns about hyperparameter sensitivity and generalizability. The method’s robustness and the tuning effort required to transfer it to new settings remain unclear.

  2. Prior work such as Cambrian-1 already combines features from different visual encoders. In this context, adding VAE features as an additional visual input appears to offer limited methodological novelty.

Questions for authors
  1. How sensitive is GAFU to its loss weights and training strategies? Can the same hyperparameter configuration transfer reliably across models and datasets without extensive tuning?
  2. Could you clarify how your method differs from prior work on multiple visual encoders?
3 comments

We thank Reviewer 7V6Y for recognizing our analysis and consistent gains. Below, we address robustness, hyperparameter transferability, and the distinction from multi-visual-encoder methods.

1. Cross-Setting Transferability (Weakness 1 & Question 1)

GAFU follows a shared core training configuration across models and datasets and outperforms SFT in all 15 model–benchmark combinations (Table 1). Detailed hyperparameters and training schedules are provided in Appendices C and D.2.

2. Hyperparameter Sensitivity (Weakness 1 & Question 1)

To examine whether these gains depend on precisely chosen loss weights, we vary the Stage 2 DPO and KL weights and the Stage 1 relation-alignment and spatial-binding weights to 0.5× and 2× their defaults, changing one weight at a time.

ConfigurationVStarOmniSpatialMMVPAverage
SFT39.2731.9074.0048.39
GAFU default configuration47.1234.5179.6753.77
DPO ×0.546.6035.0375.6752.43
DPO ×247.1235.2377.3353.23
KL ×0.546.6034.2577.0052.62
KL ×247.6434.5176.0052.72
Relation alignment ×0.547.6434.0576.6752.79
Relation alignment ×245.5534.4475.3351.77
Spatial binding ×0.547.6433.6675.3352.21
Spatial binding ×247.1234.9076.3352.78

All tested configurations outperform SFT on each benchmark, with average scores ranging from 51.77 to 53.23, compared with 48.39 for SFT and 53.77 for the default configuration. MMVP is more sensitive, dropping by 2.34–4.34 percentage points from the default, while some settings improve VStar or OmniSpatial. These results suggest that the gains persist across the tested weight range, although the default configuration provides the best overall performance.

3. Training Strategy Sensitivity (Weakness 1 & Question 1)

The ablations in Table 2 examine the effects of training configurations through both alignment objectives and fine-tuning losses. For alignment objectives, reconstruction alone outperforms SFT on all three benchmarks, while spatial binding and relation alignment further improve overall performance.For fine-tuning losses, CE-only outperforms SFT on two benchmarks, while combining DPO and KL yields better results.These results show that simplified configurations still yield gains, while the complete training design further improves performance, providing some support for the need for these strategies and the robustness of the overall method.

We further evaluate the auxiliary training strategies by separately removing U masking and gradient balancing on BAGEL, keeping all other settings unchanged:

ConfigurationVStar-Bench ↑OmniSpatial ↑MMVP ↑
SFT39.2731.9074.00
Without U masking44.5033.4675.00
Without gradient balancing45.0333.4676.33
Full GAFU47.1234.5179.67

Removing either strategy lowers performance, but both variants still outperform SFT on all three benchmarks.This indicates that training strategies affect the magnitude of the gains and that both strategies contribute positively.Meanwhile, GAFU’s advantage over SFT persists when either strategy is removed, indicating a degree of robustness.

We will include the configuration summary and sensitivity results in the revised manuscript.

4. Distinction from Multi-Visual-Encoder Methods (Weakness 2 & Question 2)

We appreciate the comparison with Cambrian-1. GAFU differs in its research question and use of visual evidence:

In terms of motivation, Cambrian-1 investigates which visual representations and design choices best support visual grounding in multimodal LLMs. GAFU instead starts from the unification of understanding and generation in unified multimodal models, exploring whether existing generation capabilities can assist understanding, what information native generation representations contain that the understanding pathway has not fully exploited, and how to use this information for the current task.

Methodologically, Cambrian-1 integrates information from multiple visual encoders through spatially aware feature aggregation. GAFU first systematically analyzes native representations, revealing their sensitivity to fine-grained semantic changes, complementarity, and spatial dependence.Guided by these findings, we design a method that uses instruction-conditioned text states to query generation representations for task-relevant information and a preference objective to reinforce its use, reducing the tendency to ignore the added features.The query and training-objective ablations in Table 2 further support the effectiveness of this analysis-guided retrieval and training design.

We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.

I thank the authors for their rebuttal. I still think the overall pipeline is overly complex, which raises doubts about its generalization performance. Can such a complex pipeline really be used in practice? The complexity also makes it difficult for others to follow.

Of course, this is just a matter of personal taste. I personally prefer simple and effective work, and the assessment of this paper should also take the other reviewers’ views into account. I maintain my original score.

We sincerely thank you for taking the time to review our responses and further highlighting your concerns about complexity, generalization, and practical use. We greatly appreciate these comments and agree that clarity and ease of reuse are important to a method’s value.

We would like to clarify that GAFU does not require every auxiliary strategy to achieve gains: simplified training configurations remain effective, while several objectives and branches are used only during training, leaving a simpler inference pipeline.

1. Simplifying Auxiliary Training Strategies While Retaining the Core Design

GAFU’s core design comprises complementary information preservation and alignment, task-conditioned retrieval, and explicit optimization of evidence utilization. U masking and gradient balancing support this learning process. On BAGEL, we separately remove these two strategies while retaining the core design and keeping all other settings unchanged:

ConfigurationVStar-BenchOmniSpatialMMVP
SFT39.2731.9074.00
Without U masking44.5033.4675.00
Without gradient balancing45.0333.4676.33
Full GAFU47.1234.5179.67

Without U masking, GAFU still improves over SFT by 5.23, 1.56, and 1.00 percentage points; without gradient balancing, the gains remain 5.76, 1.56, and 2.33 percentage points, respectively. These results show that the gains do not depend on every auxiliary strategy: the core design remains effective when either strategy is removed, while the full configuration further improves performance. Furthermore, the shared core training configuration outperforms same-data SFT in all 15 combinations of three models and five benchmarks, providing evidence of applicability across models and tasks.

2. Training Objectives and Branches Do Not All Carry Over to Deployment

Much of GAFU’s complexity arises during training, while its inference pipeline is simpler.

These training components address the difficulty of effectively using complementary information. Prior work identifies underutilization caused by dominant modalities [1]. Similarly, Figure 8 shows close validation CE under matched and mismatched G with CE-only training, indicating limited use of sample-specific evidence. GAFU combines preference learning with KL regularization; the full objective lowers matched-branch CE while curbing mismatched-branch degradation, supporting evidence use and capability preservation. It improves the three benchmark scores over CE-only by 4.19, 1.18, and 14.34 points (Table 2).

The reconstruction decoder, auxiliary prediction heads, matched–mismatched preference comparisons, and frozen reference model are used only during training. During understanding inference, GAFU only needs to extract and compress generation representations, then incorporate them through task-conditioned retrieval.

To quantify deployment costs, we compare GAFU-Full with SFT on a single NVIDIA H20 after merging LoRA. Accuracy improves by 7.85, 2.61, and 5.67 points, with 7.2% higher 32-token latency and 3.03 GiB more peak memory. First-token latency increases from 102 to 145 ms, including one-time feature processing amortized over decoding. These results quantify the deployment trade-off.

We fully respect your preference for simple, effective methods and appreciate your suggestions for improving our presentation. We will more clearly distinguish the core design, auxiliary training strategies, and inference pipeline in the revision to make the method easier to understand and reuse.

We hope these clarifications address your concerns and would be grateful if you could reconsider your assessment in light of the additional experiments and explanations. Thank you again for your time and constructive feedback. We welcome any further questions or suggestions.

[1] Peng et al. Balanced Multimodal Learning via On-the-fly Gradient Modulation. CVPR 2022.

Reviewer_S8bwOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose GAFU. This is a two-stage post-training method to improve detailed visual understanding in Unified Multimodal Models (UMMs). They use the original generation-side visual features to achieve this. First, the paper does an analysis on 2,084 image pairs from VisMin. It shows that generation-side features are more sensitive to small visual details and spatial structures than understanding-side features. Based on this finding, GAFU compresses these generation-side features into small tokens in Stage 1. Then, in Stage 2, it puts them into the understanding pathway using layer-wise cross-attention. The authors further introduce a CE-space pairwise DPO objective to encourage the use of sample-specific generation features, together with KL regularization to prevent degradation of the mismatched-generation branch.

Critical strengths
  1. The feature-level analysis provides useful evidence that generation-side representations capture fine-grained visual details and spatial information that may be overlooked by understanding-side representations.

  2. The GAFU method successfully uses the analysis results to build a practical post-training architecture. It compresses generation-side features and adds them using cross-attention with text queries. Also, the counterfactual DPO objective is a very useful way to make the model focus on the visual evidence of each specific sample.

Critical weaknesses
  1. Although GAFU and SFT use the same downstream fine-tuning data, GAFU additionally uses 1M Cambrian images for Stage 1 training and introduces extra trainable modules. Without a baseline with comparable model capacity and training data, it is difficult to isolate the contribution of generation-side representations.

  2. The paper does not provide any numbers about the computing cost (FLOPs, VRAM, latency) when running both vision pathways during inference.

  3. The ablation study shows that training GAFU with only CE makes the performance worse than the base model on MMVP. Also, CE + DPO makes the performance on OmniSpatial drop a lot. To make the training stable, the authors use many mixed tricks together. Because the authors did not test these tricks one by one, it is difficult to know which trick is actually improving the performance or if this training method can work well for other tasks.

Questions for authors
  1. Can you provide results for an SFT baseline that has the same parameters and training data?

  2. What is the exact inference cost of GAFU compared to the Base and SFT models?

  3. Can you provide ablations on the understanding-representation masking schedule and gradient balancing strategies to clarify their individual contributions?

4 comments

We sincerely thank Reviewer S8bw for recognizing our representation analysis, task-conditioned architecture, and preference objective. We address the remaining concerns below.

1. Controlled Comparisons for Data and Capacity (Weakness 1 & Question 1)

We conduct two additional experiments on BAGEL to examine the effects of the additional image source and added model capacity:

  1. Train GAFU using only downstream training images, replacing the Stage 1 Cambrian images while retaining the architecture and two-stage procedure.
  2. Train a same-architecture SFT baseline without Stage 1 pretraining. It retains GAFU’s generation-side inputs and inference modules, randomly initializes the added modules, and jointly trains the compression encoder, injection modules, and understanding-pathway LoRA using answer CE on the same downstream data. | Configuration | VStar-Bench | OmniSpatial | MMVP | |---|---:|---:|---:| | Native SFT | 39.27 | 31.90 | 74.00 | | Joint SFT with the same architecture | 43.98 | 32.01 | 74.67 | | GAFU, downstream training images only | 47.64 | 33.27 | 79.00 | | Original GAFU | 47.12 | 34.51 | 79.67 |

Without additional Cambrian images, GAFU still outperforms native SFT on all three benchmarks and achieves performance close to the original configuration. With the same image source and inference architecture, it improves over joint SFT by 3.66, 1.26, and 4.33 percentage points, respectively.

These results indicate that the gains cannot be explained solely by the additional image source or the added architectural capacity, and support the value of the two-stage training and evidence-utilization design. We will specify the initialization, trainable modules, and training budget for each control in the revised manuscript.

2. Inference Cost (Weakness 2 & Question 2)

We benchmark BAGEL Base, SFT, GAFU-Sparse, and GAFU-Full on NVIDIA H20 under identical input, precision, and decoding settings, with LoRA adapters merged and projected generation features and cross-attention K/V reused within each request. Costs and benchmark gains over Base are reported below.

ConfigurationFirst-token latency ↓32-token latency ↓Peak memory ↓VStar gain ↑OmniSpatial gain ↑MMVP gain ↑
Base101 ms2.586 s27.33 GiB———
SFT102 ms2.596 s27.33 GiB+2.62+1.11+2.33
GAFU-Sparse140 ms2.668 s29.30 GiB+9.95+3.72+6.33
GAFU-Full145 ms2.782 s30.36 GiB+10.47+3.72+8.00

After merging LoRA, SFT costs nearly the same as Base. GAFU-Full improves over SFT by 7.85, 2.61, and 5.67 points across the three benchmarks, with 7.2% higher 32-token latency and 3.03 GiB more peak memory. Its larger first-token overhead comes from one-time generation-feature processing, amortized over subsequent decoding. Sparse maintains similar accuracy while saving 1.06 GiB of peak memory and reducing 32-token latency by 4.1% relative to Full, offering a practical accuracy–cost trade-off for resource-constrained deployment.

3. Interpreting the CE-only and CE + DPO Ablation Results (Weakness 3)

CE-only retains U masking, which reduces the information available from understanding-side representations, while removing the preference objective. Figure 8 shows limited differentiation between matched and shuffled G, indicating that CE training alone does not sufficiently encourage the use of complementary generation-side features in this configuration. This imbalance helps explain its MMVP drop to 65.33. In contrast, same-architecture joint SFT retains the full understanding input and jointly trains the compression encoder, reaching 74.67.

DPO encourages the model to assign a higher probability to the correct answer under matched G than under mismatched G, thereby promoting the use of generation evidence. Without KL, however, the model can also improve this objective by lowering the correct-answer probability on the mismatched branch. Figure 8 shows a clear increase in the correct-answer CE on the mismatched branch under CE + DPO, indicating a lower probability assigned to the correct answer. This degradation is consistent with the OmniSpatial accuracy drop from 33.33 to 22.24 in Table 2. KL limits the mismatched branch’s output shift relative to the frozen reference model. Adding KL improves performance over CE + DPO by 3.14, 12.27, and 6.34 percentage points across the three benchmarks, respectively, supporting the complementary roles of preference learning and KL regularization.

We will clarify this interaction using the ablation results and validation dynamics in the revised manuscript.

4. Independent Contributions of U Masking and Gradient Balancing (Weakness 3 & Question 3)

We separately remove U masking and gradient balancing on BAGEL, keeping the initialization, training data, budget, and all other settings unchanged. The results are shown below.

ConfigurationVStar-BenchOmniSpatialMMVP
Base36.6530.7971.67
SFT39.2731.9074.00
Full GAFU47.1234.5179.67
Without U masking44.5033.4675.00
Without gradient balancing45.0333.4676.33

Removing U masking decreases the three benchmark scores by 2.62, 1.05, and 4.67 percentage points, respectively. Removing gradient balancing decreases them by 2.09, 1.05, and 3.34 percentage points, respectively, while both variants still outperform Base and SFT on all benchmarks. These results support the distinct roles of the two strategies. U masking encourages the model to learn to use generation-side information in the early stages of training, while gradient balancing controls the optimization strength of the negative branch during preference learning. We will add the above ablations and implementation details to the revised manuscript and clarify the contribution of each strategy.

We appreciate the reviewer’s constructive feedback, which has helped us clarify the sources of the observed performance gains, the practical inference cost of GAFU, and the respective roles of the training strategies.

Thank you for providing the experimental results. The new results largely address my previous concerns.

However, I share the same concern as 7V6Y regarding the novelty of the method. The overall approach primarily combines existing techniques such as feature compression and cross-attention. While the experimental results are promising, I am not entirely convinced that the proposed method is sufficiently innovative beyond existing multi-encoder feature fusion methods.

I hope the authors can further clarify the significant differences between the proposed method and existing work, and more importantly, the motivation behind the specific design choices, rather than just the motivation for using the generative representation.

We sincerely thank you for taking the time to review our responses and for acknowledging our additional experiments. We greatly appreciate your emphasis on methodological novelty and agree that the rationale behind specific design choices deserves a clear explanation. Below, we clarify the differences from existing multi-encoder fusion methods and explain how our analysis and ablations motivate and support these choices.

1. Differences from Multi-Encoder Feature Fusion Methods

We acknowledge that GAFU shares the basic approach of integrating complementary visual information with multi-encoder feature fusion methods such as Cambrian-1 [1], BRAVE [2], and Eagle [3]. However, guided by our systematic representation analysis, GAFU combines explicit spatial relationship preservation, evolving-context-guided retrieval, and explicit optimization of evidence utilization. This joint design advances both the integration and effective use of complementary visual information. The similarities and differences are summarized below.

MethodComplementary visual information integrationExplicit spatial relationship preservationEvolving-context-guided retrievalExplicit optimization of evidence utilization
Cambrian-1✓✓××
BRAVE✓×××
Eagle✓×××
GAFU✓✓✓✓

Here, ✓/× indicates dedicated designs: Cambrian-1 uses spatial aggregation, BRAVE performs task-conditioned extraction before the language model, and Eagle fuses pre-aligned visual experts. GAFU further addresses how complementary information can remain spatially meaningful, relevant to the evolving task context, and effectively used in prediction. We explain the rationale and supporting ablations below.

2. Motivation for Specific Design Choices

Explicit Spatial Relationship Preservation

GAFU trains compressed tokens to predict generation features at corresponding spatial locations, directly supervising content–location correspondence after compression. In comparison, Cambrian-1 uses spatial aggregation, BRAVE uses task-conditioned resampling, and Eagle concatenates spatially aligned features.

Our systematic analysis motivates this design: spatial shuffling reduces the generation-side sensitivity advantage in all 24 model–domain combinations (Figure 1 and Table 5). Its effectiveness is further supported by our ablations: adding spatial binding improves OmniSpatial from 33.14 to 34.51 and MMVP from 75.00 to 79.67 (Table 2).

Evolving-context-guided Retrieval

GAFU uses evolving task states within the understanding backbone to retrieve relevant visual evidence. Compared with spatial aggregation in Cambrian-1 and feature extraction or fusion before the language model in BRAVE and Eagle, this design directly adapts evidence selection to the task context as understanding progresses.

Our ablations support its effectiveness: task-state queries achieve an average score of 53.77, exceeding concatenation, input-level fusion, visual queries, and learnable queries by 3.53, 7.35, 2.95, and 3.01 percentage points, respectively (Table 2).

Explicit Optimization of Evidence Utilization

GAFU explicitly promotes sample-relevant evidence use through matched–mismatched preference learning, with KL regularization limiting degradation under mismatched evidence. Compared with image–text and instruction supervision in Cambrian-1 and Eagle, and captioning/VQA supervision with encoder dropout in BRAVE, GAFU directly optimizes how predictions depend on whether the supplementary evidence matches the sample.

Figure 8 shows similar validation CE under matched and mismatched G with CE-only training, indicating limited evidence use. DPO widens this gap but raises mismatched-branch CE, weakening correct-answer prediction under mismatched evidence; KL curbs this degradation. Our ablations support the joint design: the full objective improves VStar-Bench, OmniSpatial, and MMVP over CE-only by 4.19, 1.18, and 14.34 percentage points, respectively (Table 2).

We fully appreciate the importance you place on methodological novelty and thank you for helping us identify where our explanation needs to be clearer. We will incorporate these comparisons and design rationales into the revision to make our contributions and their supporting evidence clearer.

We hope these clarifications address your remaining concerns and would be grateful if you could reconsider your assessment in light of the additional experiments and explanations. Thank you again for your time and constructive feedback. We welcome any further questions or suggestions.

[1] Tong et al. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. 2024.

[2] Kar et al. BRAVE: Broadening the Visual Encoding of Vision-Language Models. 2024.

[3] Shi et al. Eagle: Exploring the Design Space for Multimodal LLMs with Mixture of Encoders. 2024.

Reviewer_1WGqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores how generation-side features can help with fine-grained visual understanding. It proposes GAFU, which compresses these features and feeds useful information into the understanding pathway based on the task. Experiments on three models and five benchmarks show consistent improvements.

Critical strengths

The motivation and analysis of the proposed method are clear and convincing. The results are consistently promising across three different models.

Critical weaknesses

The practical cost of the method is also insufficiently evaluated. GAFU adds a compression encoder and cross-attention throughout the understanding backbone. Although the limitations acknowledge additional computation and memory, no measured latency or memory comparison is provided. This is particularly relevant because the sparse-layer variant is already close to the full model in Table 2. Reporting the accuracy–cost trade-off would help establish when using generation-side information is worthwhile in practice.

Questions for authors

See the weaknesses.

2 comments

We sincerely thank Reviewer 1WGq for recognizing our motivation, representation analysis, and consistent gains. To address the accuracy–cost trade-off, we have added inference-cost measurements.

We measure the inference costs of BAGEL Base, SFT, GAFU-Sparse, and GAFU-Full on a single NVIDIA H20 under identical input, precision, and decoding settings, merging LoRA and reusing projected generation features and cross-attention K/V within each request. Inference costs and benchmark performance are reported below.

ConfigurationFirst-token latency ↓32-token latency ↓Peak memory ↓VStar gain ↑OmniSpatial gain ↑MMVP gain ↑
Base101 ms2.586 s27.33 GiB———
SFT102 ms2.596 s27.33 GiB+2.62+1.11+2.33
GAFU-Sparse140 ms2.668 s29.30 GiB+9.95+3.72+6.33
GAFU-Full145 ms2.782 s30.36 GiB+10.47+3.72+8.00

Relative to Base, Sparse and Full increase 32-token latency by 3.2% and 7.6%, respectively. Full achieves higher accuracy on VStar-Bench and MMVP by 0.52 and 1.67 points, while matching Sparse on OmniSpatial. Sparse saves 1.06 GiB of peak memory and reduces latency by 4.1% relative to Full, providing an alternative when efficiency is prioritized. The first-token overhead includes one-time generation-feature extraction and compression, amortized over subsequent decoding.

We will include these measurements and the corresponding accuracy comparison in the revision. We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.

We sincerely thank the reviewer for the positive assessment. We hope our responses have addressed your concerns and greatly appreciate your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Response to All Reviewers and the Area Chair

Dear Area Chair and Reviewers,

We sincerely thank you for the time, care, and effort you have devoted to reviewing our work and engaging in the discussion. We greatly appreciate your recognition of how our systematic analysis reveals complementary generation-side information and guides the method design, as well as the consistent gains across models and tasks. We are especially grateful for the thoughtful questions that helped us strengthen our evidence and clarify our contributions.

During the rebuttal, we provided additional experiments and clarifications in four areas:

  1. Methodological Contributions. We clarified GAFU’s joint design of spatial preservation, evolving-context-guided retrieval, and explicit evidence-use optimization, highlighting differences from existing multi-encoder fusion methods in information compression, retrieval, and training objectives, with supporting analysis and ablations. These clarifications make the methodological contributions and their design rationale more explicit.

  2. Controlled Comparisons and Robustness. We added controls for training images and model capacity, loss-weight sensitivity tests, and auxiliary-strategy ablations. The tested GAFU variants retain gains over SFT, helping clarify the contributions of the core design and training strategies.

  3. Practical Costs. We reported latency, memory, and accuracy comparisons for Base, SFT, Sparse, and Full configurations, and distinguished training-only components from the inference pipeline. These results clarify the deployment trade-offs and the simpler inference requirements.

  4. Extended Evaluation and Capability Preservation. We added OCR, general understanding, and image-editing evaluations, showing improved OCR performance, maintained general understanding relative to Base on the tested benchmarks, and nearly unchanged image-editing performance. These results extend the evidence for GAFU’s applicability and capability preservation.

We are deeply grateful for your thoughtful questions and constructive guidance, which have strengthened both our evidence and presentation. We will incorporate these additions into the revision and sincerely hope they assist your consideration of GAFU’s contributions.

Thank you once again to all reviewers and the Area Chair for your valuable time and careful consideration. We sincerely welcome any further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.