- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Authors aligned the <|vis_thought|> token representation with a CLIP image embeddings across modalities and languages, and evaluated generalization on OOD dataset.
Critical strengths
Generously, the motivation behind proofs and the approach used is interesting.
Critical weaknesses
It's unclear which of the downstream performance improvements are statistically significant and generalize beyond small models from same family.
Theorem 1 argues that if cross-modal inputs map into same space, if that token captures "true" mutual information, and if the model uses that information, then the model achieves lower Bayes risk (R) if the alignment token was there only for "sequence expansion" (when any of these assumption is not met). Not only are R, the loss, the distribution used, etc not defined, the arguments as for why these assumptions are met is not sound. A.2.1 promises full derivation but contains no intermediate steps. Same problem for Theorem 3.
Similar issues extend to other parts of the paper. In general, the assumptions should be fully and explicitly listed and then convincingly argued why they are true for every theorem if they are to be used.
Questions for authors
What exactly are "Latent Cos" and "Vis Attn %", how are they computed, and how do they support the arguments made?
What exactly is "true" visual mutual information, how is it measured?
Same question for "residual divergence".