- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Attention Interaction Alignment (AIA). By calculating the cross-modal interaction intensity, it can generally reflect the attention pattern between image and text among each layer. And by Stage-level Averaging & Huber loss, the student can learn the attention pattern from the task-specific teacher (generation/understanding models). Both generation and understanding tasks of the UMM can be enhanced during the process, so as to get a better performance in both task's benchmark.
Critical strengths
-
The paper introduces cross-modal Interaction Intensity to characterize the cross-modal attention mass, providing an intuitive way to analyze layer-wise attention interaction patterns in UMMs.
-
The proposed stage-level alignment guides the student to learn the overall interaction trends of the teacher instead of directly matching individual layers. This preserves flexibility in student's layer-wise representations, reducing the risk of overly restrictive alignment.
Critical weaknesses
-
The proposed Interaction Intensity is sensitive to the sequence lengths of different modalities. For example, when , the text attention mass may be relatively low even when the text contains important conditioning information. It would be helpful to provide an analysis of how token ratios affect the measured interaction intensity.
-
The training datasets include BLIP3-o and ShareGPT-4o-Image which highly overlaps with GenEval. This calls into question whether the reported scores (meanwhile, some benefit itself is very marginal, e.g. 0.01 in Janus-Pro) are a true measure of performance or an artifact of the similar data distributions. Could the authors provide additional results excluding potentially overlapping samples?
Questions for authors
Please see the weakness.
Thanks for the authors’ detailed responses. I will increase the rating from 2 to 3.