- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GAFU, which improves the recognition of local details and spatial relationships on the understanding side by reading visual features from the generation side of a unified multimodal model. The method first compresses the generation-side features into 256 tokens and preserves information via reconstruction, spatial constraints and relationship alignment. Then the text states on the understanding side query these features through cross-attention. The authors analyze the sensitivity and complementarity of the two types of representations across six models, and validate the method on three models with five understanding benchmarks. Across all 15 combinations of models and benchmarks, GAFU outperforms SFT trained on the same data, achieving an average improvement of approximately 3.56–5.98 percentage points over the five benchmarks.
Critical strengths
- The idea of directly reading generative-side features to assist comprehension is intriguing. The authors analyzed the differences between two types of representations across six models, and incorporated pixel perturbation control and spatial shuffling experiments, providing relatively concrete evidence for method design.
- The method outperforms SFT on the same dataset across three different models and five tests, with consistent results. The ablation studies on feature alignment, injection approaches and training objectives are also comprehensive, revealing the impact of each design choice on the final performance.
Critical weaknesses
- The coverage of fine-grained tasks is incomplete. The paper mainly tests objects, attributes and spatial relations, and does not report OCR-related results. Text recognition also relies on local details and serves as a valuable scenario for verifying the proposed method. It remains unclear whether the performance gains can be extended to such tasks.
- The paper emphasizes the preservation of original understanding capabilities, yet it does not report broader general understanding tests, which is of great interest to me. In addition, the paper does not report the generation performance after fine-tuning, so it is unknown whether the final model retains its original generation capability.
Questions for authors
- How does GAFU perform on OCR tasks? Are there test results such as TextVQA, DocVQA or OCRBench? These results will help determine whether the method can handle fine-grained information beyond object attributes and spatial relationships.
- How does the fine-tuned model perform on general understanding tests? Does it suffer from capability degradation compared with the original model and SFT using the same data?
- Can the same fine-tuned model retain its original generation capability? How are the understanding-side LoRA and newly added modules used during generation, and are there corresponding generation evaluation results?
3 comments
Thanks for your reply
Thank you for the additional experiments and clear explanations. The OCRBench v2, general understanding, and generation results address my main concerns. I will maintain my current score.
We sincerely thank the reviewer for the positive feedback. We greatly appreciate your time and support.
We thank Reviewer LpnC for the careful review and for identifying the need to evaluate OCR coverage, general understanding, and post-fine-tuning generation. We address these points below.
1. OCR Task Performance (Weakness 1 & Question 1)
Using the existing general-purpose training checkpoints, we add an evaluation on OCRBench v2, with the results shown below.
GAFU outperforms SFT by 1.77, 1.23, and 1.15 points on BAGEL, Janus-Pro, and Lance, respectively, while also exceeding Base across all three models. These results extend the observed benefits of generation-side information to text-related fine-grained understanding.
2. General Understanding Performance (Weakness 2 & Question 2)
To assess whether fine-tuning affects general understanding, we evaluate the three models on MMStar, MMMU, and MME-RealWorld Lite, comparing GAFU with the original models and SFT trained on the same data.
GAFU improves MMStar performance across all three model families and generally maintains or improves performance on MMMU and MME-RealWorld Lite. The overall results compare favorably with both Base and SFT trained on the same data. These findings indicate that GAFU preserves general understanding capabilities while improving fine-grained visual performance.
3. Generation Capability and Module Usage (Weakness 2 & Question 3)
GAFU modifies understanding-side inputs and adds cross-attention after each Transformer layer while keeping the generation expert unchanged. To assess its effect on generation, we evaluate the same BAGEL understanding checkpoint on G-Edit and ImgEdit with all GAFU modules enabled, including the generation-representation compression encoder, cross-attention injection modules, and understanding-pathway LoRA.
GAFU performs nearly identically to Base and SFT on G-Edit and ImgEdit, indicating that it preserves image-editing performance. We will add the above evaluations and the module-usage settings to the revised manuscript.
We sincerely thank you again for your constructive suggestions. We hope our responses have addressed your concerns, and we look forward to your feedback and further discussion.