acceptodds
Withdrawn from ICLR 2027

Task-Free Visual-Token Repair for Frozen Multimodal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) rely on an internal interface that maps visual representations into the language model. When these representations are altered by quantization, compression, pruning, or other efficiency constraints, recovering downstream utility would ideally require optimizing them directly against the MLLM's downstream objective. However, this requires task-specific supervision and repeated forward/backward passes through a large MLLM, making end-to-end adaptation expensive and often impractical. We instead study task-free recovery at the visual-token interface. We introduce a lightweight repairer trained only from paired clean and corrupted visual representations, while keeping the visual encoder, multimodal projector, and language model frozen. Training requires neither task labels, prompts, language losses, nor LLM forward passes. Controlled perturbation experiments show that pointwise token fidelity is incomplete: representations with similar reconstruction error can induce substantially different downstream behavior, depending on their structure and direction. We evaluate this approach under practical corruption across Honeybee, LLaVA-v1.5-7B, and Qwen2.5-VL-7B. Under severe static 2-bit visual-token quantization, task-free repair recovers 81.8%, 72.5%, and 52.9% of the POPE degradation on Honeybee, LLaVA-v1.5, and Qwen2.5-VL, respectively, while objectives that preserve richer representation properties can outperform standard pointwise reconstruction. Learned image compression at approximately bpp provides a second, upstream corruption regime: repair recovers downstream utility across all three architectures, while robust and structure-aware objectives outperform MSE in matched settings. Overall, our results show that lightweight, task-free visual-token repair provides a practical alternative to end-to-end MLLM adaptation and helps identify which representation properties should be preserved under different forms of visual corruption.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Visual Token Repair, Representation Corruption, Task-Free Adaptation, Visual Token Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YfHYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies task-free repair of corrupted visual tokens for frozen MLLMs. The authors propose a lightweight, identity-initialized residual MLP repairer that is inserted at the visual-language interface. It is trained only on paired clean/corrupted visual representations, without task labels, prompts, or LLM forward/backward passes. Controlled perturbation experiments on LLaVA-v1.5 show that pointwise token fidelity (MSE) does not reliably predict downstream utility. Motivated by this, the authors propose a structured objective that combines Smooth-L1 reconstruction, local token-relation preservation, and channel-statistic calibration. The method is evaluated on Honeybee, LLaVA-v1.5, and Qwen2.5-VL under two corruptions. Under static 2-bit visual-token quantization, it recovers 81.8%, 72.5%, and 52.9% of the POPE degradation, respectively. Under learned image compression (~0.155 bpp), it recovers roughly 29–44% of the MME degradation. Robust and structure-aware objectives generally outperform plain MSE in matched settings.

Critical strengths

1. Interesting problem setting. The paper studies how to mitigate visual corruption without modifying model weights or using any task-specific data. This is a realistic constraint when the LLM is frozen or remotely hosted. The solution is simple and direct, but it still recovers a decent portion of the lost performance (e.g., 72.5% of the POPE drop on LLaVA-v1.5 under 2-bit quantization).

2. A counter-intuitive finding. The paper shows that reconstruction loss with respect to the clean visual tokens doesn't necessarily align with the downstream task loss. For example, MSE-trained repair actually hurts LLaVA (0.794 → 0.660), while Smooth-L1 and structured objectives help. I find this a useful insight for future work on representation repair.

Critical weaknesses

1. Limited novelty in the method.

Training a small module to pull corrupted representations back toward clean ones is a fairly direct idea and not hard to come up with. The losses used (Smooth-L1, local relation preservation, channel-statistic calibration) are mostly existing designs. Also, Kao et al.[1] (ICLR 2025) already used a lightweight module with a surrogate loss that avoids the LLM to adapt compressed visual inputs, so the task-free idea itself isn’t entirely new. The diagnostic experiments show that perturbation direction matters, but the proposed losses don’t really target this, so the connection between the analysis and the method feels a bit loose.

2. The choice of α seems to depend on downstream scores.

The paper says α can be selected with a representation-level criterion only, but never says what that criterion is. Looking at the results, Qwen uses α=1 for MME and α=0.1 for POPE in the compression experiments, and at α=0.1 every loss gives negative gains on MME. This looks a lot like α was picked based on downstream scores. If that’s the case, the main “task-free” claim becomes questionable.

3. Missing comparison with baseline such as Kao et al. (ICLR 2025).

This is the most closely related prior work. It also avoids training through the LLM and uses the same Ballé hyperprior codec, so it could be compared directly in the compression experiments. Without this comparison, it’s hard to tell what the actual advantage of the proposed method is.

[1] Chia-Hao Kao, Cheng Chien, Yu-Jen Tseng, Yi-Hsin Chen, Alessandro Gnutti, Shao-Yuan Lo, Wen-Hsiao Peng, and Riccardo Leonardi. Bridging Compressed Image Latents and Multimodal Large Language Models. In ICLR, 2025

Questions for authors
  1. How was α chosen in each experiment? Were downstream scores used? Could you report the α selected by a purely representation-level criterion, its results, and α-sensitivity curves?
  2. Could you add a comparison with Kao et al. (ICLR 2025) under the same setting?
  3. On LLaVA, MSE repair drops POPE from 0.794 to 0.660, which is worse than no repair. What’s the reason? Is this specific to α=0.5, or is it caused by MSE training itself?
Reviewer_3gF3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to recover the performance of frozen multimodal LLMs when their visual representations are corrupted. It trains a small residual MLP on paired clean and corrupted visual tokens, without task labels, prompts, language losses, or language-model execution during training. The proposed objectives combine robust reconstruction with local token-relation preservation and channel-statistic calibration.

Experiments cover Honeybee, LLaVA-v1.5, and Qwen2.5-VL under severe visual-feature quantization and learned image compression. The strongest results are the multi-seed POPE improvements under quantization, which recover 81.8%, 72.5%, and 52.9% of the degradation, respectively. Controlled perturbations and objective comparisons further show that representation error alone does not reliably predict downstream performance.

Critical strengths

The paper addresses a useful adaptation setting with a simple, economical method. Training only a roughly 7-8M-parameter repairer on 448 images, while keeping the multimodal backbone frozen and avoiding LLM execution, is appealing when adapting the downstream model is expensive or unavailable. Showing recovery across three distinct visual interfaces and two corruption regimes makes the study more informative than a single-model denoising experiment.

The primary recovery results have meaningful empirical support. The quantization experiments use three repairer-training seeds, a fixed repair strength across models and tasks, and paired bootstrap intervals. The LLaVA comparison is particularly informative: Smooth-L1 substantially outperforms MSE, and the structured objective adds a further 1.12 percentage points on POPE, with a reported confidence interval that excludes zero. The authors also appropriately disclose negative and inconclusive transfer results.

Critical weaknesses

Task-free configuration selection is not demonstrated, despite its importance to the proposed use case. The manuscript establishes that repairer parameters are trained without downstream supervision, but it only states that repair strength can be selected using held-out representations. It does not specify or validate the selection rule used for the reported compression results. For Qwen, the compression experiments use normalized Smooth-L1 with shrinkage at α=1\alpha=1 for MME but normalized Smooth-L1 at α=0.1\alpha=0.1 for POPE, and at α=0.1\alpha=0.1 all four Qwen objectives in Table 8 reduce MME performance. These differences make the selection consequential. Because the paper itself shows that representation fidelity is an imperfect proxy for downstream utility, a representation-only selection procedure needs empirical validation. Otherwise, the results establish task-free parameter training, not a reliable task-free deployment recipe.

The evidence does not sufficiently isolate the contribution of the proposed structural constraints. The matched-distortion diagnostic measures errors before the nonlinear projector, whereas the repair objectives operate after it, so equal input-space distortion need not imply equal distortion at the repair interface. Moreover, the structured objective bundles robust reconstruction, token relations, channel calibration, and sometimes attention weighting, without a systematic component-wise ablation. Most of the LLaVA improvement over MSE already comes from Smooth-L1, structured repair worsens Qwen MME under quantization (1903.6 to 1867.2 in Table 7), and MSE is strongest among the reported Honeybee compression configurations. These findings support objective sensitivity but give a less definitive account of which structural properties are responsible for recovery.

Questions for authors

Selection protocol: How were the objective, checkpoint, and repair strength selected for each reported setting, and were downstream benchmark scores consulted? Please specify a selection rule that uses only the 64 representation-validation images, and report results with the resulting configuration held fixed across downstream tasks.

Component ablation: Could you isolate the effects of robust reconstruction, relation preservation, channel calibration, and attention weighting under matched training and repair strength? Reporting held-out reconstruction errors at the post-projector repair interface alongside downstream scores would also help distinguish better fitting from better preservation of task-relevant structure.

Evaluation details: What does “POPE score” denote, and which POPE sampling setting is evaluated? Also, why does clean LLaVA MME performance differ between Table 6 (1786.52) and Table 2 (1872.9)? Please clarify which evaluation settings account for the difference.

Reviewer_gvRhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper trains a small residual MLP to map corrupted visual tokens toward clean representations while keeping the entire MLLM frozen. Training uses paired representations without task labels or LLM execution.
  • It combines robust reconstruction, token-relation preservation, and channel-statistic matching, motivated by experiments showing that equal reconstruction error can produce different downstream outcomes.
  • Validation covers three MLLMs under simulated 2-bit token quantization and learned image compression, with POPE as the primary benchmark and additional task evaluations.
Critical strengths
  • Low-cost adaptation setting: Training requires only paired visual representations and a small dataset, making the approach useful when downstream LLM training is unavailable or expensive.
  • Consistent primary-benchmark recovery: Three-seed POPE results show recovery across three architectures under severe quantization, supporting the feasibility of representation-only adaptation.
Critical weaknesses
  • Limited conceptual advance: The central approach is paired feature reconstruction using a small adapter and familiar losses. The perturbation diagnostics establish that MSE is insufficient, but provide limited insight into which representation properties reliably preserve downstream utility. The contribution beyond a straightforward adaptation recipe remains modest.
  • Limited and inconsistent generalization: Evaluation primarily matches training and test corruption settings. Cross-task recovery is uneven—for example, Qwen’s quantized MME score decreases after repair—and compression results depend strongly on the objective and repair strength. These results provide insufficient support for broadly reliable visual-token repair.
Questions for authors
  • Clarification: How were the reported compression objectives and repair strengths selected? Were all choices fixed using only the 64-image representation-validation set, without downstream benchmark feedback?
  • Clarification: Was the same repairer evaluated on clean tokens? This control would help distinguish corruption correction from general feature adaptation.
  • Assessment-relevant comparison: How does the proposed repairer compare with a simple affine or linear feature-matching baseline under the same data and selection protocol? Comparable performance would substantially weaken the methodological contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.