- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies task-free repair of corrupted visual tokens for frozen MLLMs. The authors propose a lightweight, identity-initialized residual MLP repairer that is inserted at the visual-language interface. It is trained only on paired clean/corrupted visual representations, without task labels, prompts, or LLM forward/backward passes. Controlled perturbation experiments on LLaVA-v1.5 show that pointwise token fidelity (MSE) does not reliably predict downstream utility. Motivated by this, the authors propose a structured objective that combines Smooth-L1 reconstruction, local token-relation preservation, and channel-statistic calibration. The method is evaluated on Honeybee, LLaVA-v1.5, and Qwen2.5-VL under two corruptions. Under static 2-bit visual-token quantization, it recovers 81.8%, 72.5%, and 52.9% of the POPE degradation, respectively. Under learned image compression (~0.155 bpp), it recovers roughly 29–44% of the MME degradation. Robust and structure-aware objectives generally outperform plain MSE in matched settings.
Critical strengths
1. Interesting problem setting. The paper studies how to mitigate visual corruption without modifying model weights or using any task-specific data. This is a realistic constraint when the LLM is frozen or remotely hosted. The solution is simple and direct, but it still recovers a decent portion of the lost performance (e.g., 72.5% of the POPE drop on LLaVA-v1.5 under 2-bit quantization).
2. A counter-intuitive finding. The paper shows that reconstruction loss with respect to the clean visual tokens doesn't necessarily align with the downstream task loss. For example, MSE-trained repair actually hurts LLaVA (0.794 → 0.660), while Smooth-L1 and structured objectives help. I find this a useful insight for future work on representation repair.
Critical weaknesses
1. Limited novelty in the method.
Training a small module to pull corrupted representations back toward clean ones is a fairly direct idea and not hard to come up with. The losses used (Smooth-L1, local relation preservation, channel-statistic calibration) are mostly existing designs. Also, Kao et al.[1] (ICLR 2025) already used a lightweight module with a surrogate loss that avoids the LLM to adapt compressed visual inputs, so the task-free idea itself isn’t entirely new. The diagnostic experiments show that perturbation direction matters, but the proposed losses don’t really target this, so the connection between the analysis and the method feels a bit loose.
2. The choice of α seems to depend on downstream scores.
The paper says α can be selected with a representation-level criterion only, but never says what that criterion is. Looking at the results, Qwen uses α=1 for MME and α=0.1 for POPE in the compression experiments, and at α=0.1 every loss gives negative gains on MME. This looks a lot like α was picked based on downstream scores. If that’s the case, the main “task-free” claim becomes questionable.
3. Missing comparison with baseline such as Kao et al. (ICLR 2025).
This is the most closely related prior work. It also avoids training through the LLM and uses the same Ballé hyperprior codec, so it could be compared directly in the compression experiments. Without this comparison, it’s hard to tell what the actual advantage of the proposed method is.
[1] Chia-Hao Kao, Cheng Chien, Yu-Jen Tseng, Yi-Hsin Chen, Alessandro Gnutti, Shao-Yuan Lo, Wen-Hsiao Peng, and Riccardo Leonardi. Bridging Compressed Image Latents and Multimodal Large Language Models. In ICLR, 2025
Questions for authors
- How was α chosen in each experiment? Were downstream scores used? Could you report the α selected by a purely representation-level criterion, its results, and α-sensitivity curves?
- Could you add a comparison with Kao et al. (ICLR 2025) under the same setting?
- On LLaVA, MSE repair drops POPE from 0.794 to 0.660, which is worse than no repair. What’s the reason? Is this specific to α=0.5, or is it caused by MSE training itself?