acceptodds
Under review as a conference paper at ICLR 2027

Beyond Feature Fidelity: Recovery and Readout Adaptation in Multimodal Large Language Models

Abstract

Feature recovery targets the geometry of a clean visual representation; multimodal prediction depends on how a readout uses that representation. We examine this distinction in controlled systems connecting a shared visual encoder to frozen language-side models through learned readouts. Across four downstream models and nine benchmarks, recovery has mixed effects under a clean-trained readout. Readout adaptation improves the reported utility in most evaluated model–benchmark combinations, yet adapting directly to raw features is often competitive with or better than adapting after explicit recovery. In a GQA control, additional readout training on recovered clean features raises the score from to , while adaptation using recovered clean and corrupted features reaches . This limits attribution of the entire adaptation gain to distribution matching. Separate fidelity analyses characterize task-specific and task-agnostic recovery settings. Together, the results motivate evaluating the representation–readout pair through distinct measurements of feature fidelity, fixed-readout utility, and adapted-readout utility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zdU5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether recovering corrupted visual features toward their clean counterparts ("feature fidelity") helps downstream MLLM performance. It also asks whether adapting the vision–language readout is a separate and more effective lever.

Critical strengths
  1. The 2×2 factorization (Δrec, Δadapt, Δmarg) is clean and easy to read. Measuring Raw-Adapted, and not only Recovered-Adapted, is the right control, and many recovery papers skip it.

  2. Coverage is broad: four MLLM families and nine benchmarks, plus per-corruption, per-severity, checkpoint-sweep and POPE precision/recall breakdowns.

Critical weaknesses
  1. The central phenomenon, DUG, is never demonstrated, and the main recovery module lowers fidelity. DUG is defined as fidelity improving while utility does not, under a fixed readout. But Table 21 shows the task-specific module, which produces every Recovered row in Table 1, makes features worse: Cosine to the clean reference drops from 0.988 to 0.894; MSE rises roughly 12× (0.0019 → 0.022); Norm ratio inflates to 1.13. Of course a module that moves features away from the clean manifold hurts a clean-trained readout. Δrec < 0 in 30/36 cells (all 9 for Qwen2.5-VL and Qwen3.5, 8/9 for LLaVA-1.5). The paper calls this "heterogeneous," but it is simply what a broken denoiser should produce. The only module that improves fidelity is the Open Images one, and its gain is tiny (cosine 0.9829 → 0.9836 on ImageNet). Its fidelity and utility are also measured on different datasets, which the paper concedes. So the title claim ("Beyond Feature Fidelity") is never supported by a matched experiment in which fidelity goes up and utility does not.

  2. The task-specific recovery module is never described. Appendix A.4 covers only the task-agnostic module. For the module behind the main table, the paper gives no architecture, training data, objective weights or reason for degrading fidelity. Tables 14 and 19 add confusion: their "Full" variant (GQA 59.09, POPE 82.37) matches the Open Images numbers in Table 4, not the task-specific ones (58.82 / 80.91). I cannot tell which module is in which table.

  3. The diagnostic system is too degraded to say anything about real MLLMs. Replacing each model's native vision pathway with DINOv3 plus an MLP trained by token regression (under one epoch, see W5) gives near-floor scores, i.e., LLaVA-1.5-7B TextVQA ≈ 10 (native ≈ 58) and VizWiz ≈ 18. At these levels, "readout adaptation helps" mostly means "an undertrained interface improves with more training." The cross-model generality claim is weak because what is being compared is four badly mismatched projectors attached to four LLMs, not four MLLM architectures. I would weigh one experiment at native performance (native encoder, native projector, feature recovery inserted in the native pathway) above all 36 current cells.

  4. Baselines are missing. The paper evaluates no existing recovery method: not DVT, not the Parikh et al. latent-denoising method it cites, and no pixel-space restorer (e.g., NAFNet or Restormer before the encoder). It also has no robust-encoder baseline. A finding of the form "feature recovery doesn't help" needs to show it for recovery methods the community uses, not only for in-house modules, one of which damages fidelity.

Questions for authors
  1. Can you show one matched setting where fidelity improves and utility under the frozen readout does not, with both measured on the same evaluation images?

  2. What is Raw-Frozen on the clean split alone versus the corrupted splits alone, per benchmark? How much does corruption actually hurt this system?

  3. Why does the clean interface train for only 750 steps (under one epoch)? If you train it to convergence, does Δadapt survive?

  4. What data is used for adaptation (the 7,691 samples in Table 11)? Does it share images or corruption families with the evaluation sets? The paper says heavy settings are held out, yet severity 3 appears at test time; please clarify.

3 comments
AuthorsOpenReview ↗

Response to zdU5

W1/Q1. Matched evidence

We added native-CLIP evaluations on four benchmarks. Within each benchmark, fidelity and utility use identical checkpoints, samples, and question weights; the pretrained vision–language projector and LLM remain frozen.

Corrupted inputsΔ cosineΔ MSEΔ accuracy, pp [95% CI]
GQA+0.025140−0.137467−0.178 [−0.384, 0.033]
TextVQA+0.017440−0.107416−0.024 [−0.306, 0.254]
MMStar+0.021128−0.094334+0.033 [−0.955, 1.035]
BLINK+0.013968−0.070951+0.184 [−0.298, 0.683]

Both fidelity metrics improve with intervals excluding zero; no frozen-accuracy gain is statistically detected. These matched results support evaluating fidelity and utility separately.

W2. Recovery module

We apologize for the omitted description and unclear labels. Stage1 is a residual dual-stream VAE: input/output 1024; content hidden/latent 1024/128; nuisance 512/64; decoder hidden 1024; GELU/LayerNorm. The checkpoint at step 8,000 was trained with AdamW, LR 3e−4, effective batch 16. Loss weights: cosine/MSE/Smooth-L1 1/1/0.5; cycle 0.3; relation 0.2; global cosine 0.1; KL 0.005; total correlation 0.02; minimum clean-identity/residual-L2 0.01/0.02. We will specify curriculum schedules.

Training uses paired clean/corrupted DINOv3 features without QA labels.

Tables 1/21 and Table 4's task-specific rows use Stage1; Table 14's Full and Table 19's C5 use Open Images step 12,000. We will label Table 14 as a recovery-configuration comparison.

We added an error decomposition on 256 evaluation images: ΔMSE = mean(r²) − 2mean(r·d), where r=R(z)−z and d=y−z. On corrupted inputs, Stage1's terms are 0.022957 and −0.000092, yielding ΔMSE +0.023049; C5 yields −0.000309. Thus, the measured fidelity loss occurs in the recovery correction, before projection.

W3. Native pathway

We inserted recovery between LLaVA-1.5's native CLIP and pretrained projector. Under the same evaluation conditions, native corrupted-input baselines (GQA/TextVQA/MMStar/BLINK) are 69.355/43.927/33.500/40.873%.

On 779 GQA images, adapting the projector from its pretrained weights while holding recovered features fixed improves accuracy by 0.245 pp [0.067, 0.424] with clean-only adaptation and 0.178 pp [0.022, 0.336] with mixed adaptation. Thus, adapting a pretrained native projector also improves recovered-feature utility.

W4. Public baseline

We added official blind-denoising Restormer before DINOv3 on 779 GQA images (σ=15/25/50). Across three validation-stopped projector starts, PSNR rises 20.234→30.173 dB and frozen accuracy improves 0.322 pp [0.025, 0.619]. Subsequent mixed adaptation changes accuracy by −0.319 pp [−0.495, −0.145], separating restoration benefit from adaptation.

Q2. Clean/corrupted scores

We added separate Raw-Frozen scores for the four systems in Table 1 (Clean/Corrupted, %; Corrupted pools Medium/Heavy).

TaskLLaVA-1.5Qwen2.5-VLQwen3.5OneVision-2
GQA59.330/58.81548.390/48.31054.200/54.05019.790/19.740
TextVQA10.040/9.8877.633/7.61010.787/10.91710.113/10.107
TallyQA54.230/53.89053.930/53.99060.570/60.15533.380/32.465
POPE82.689/82.28975.300/75.20683.456/83.03977.333/77.078
VQAv265.300/65.52061.060/60.72068.720/68.58023.720/23.180
VizWiz17.929/17.78617.288/17.19920.360/20.18611.546/11.175
MMStar27.000/27.80034.733/33.96739.867/39.30035.067/34.133
BLINK36.034/36.16538.506/38.53241.399/41.50425.145/24.750
NaturalBench47.237/46.66448.658/48.34962.855/62.75746.645/46.829

Across 36 settings, corruption effects range from −0.933 to +0.800 pp and vary by task.

Q3. Longer training

The 750 steps set a shared optimization budget. In additional longer-training experiments, we used 64,926 training/7,214 validation images, with validation loss checked every 500 steps. Patience stopping occurs at 31k/32k/37k steps; minimum-validation-loss checkpoints are selected. On corrupted GQA, another 750 mixed-adaptation steps yield −0.082 pp [−0.516, 0.351] for Stage1 and −0.237 pp [−0.379, −0.095] for C5; frozen C5 improves over Raw by 0.193 pp [0.099, 0.290]. This separates recovery gains from adaptation effects after extended initial training.

Q4. Training and overlap

The projector pool contains 72,140 GQA images; Table 11's 7,691-image subset is for data scaling. Adaptation regresses clean native tokens without QA labels. Training uses severity 1/2; heavy evaluation uses severity 3 in the same five corruption families.

Our ID audit found no GQA overlap; TallyQA has 4,919/7,352 overlapping images. On the remaining 2,433 images, adaptation gains are 5.886/6.278/1.915/2.176 pp, with CIs [4.766,6.969]/[5.216,7.374]/[0.874,3.010]/[1.287,3.056], in table model order (clean/medium/heavy average). The positive adaptation effect therefore persists across all four models after excluding recorded projector-training overlaps.

Thank you for the detailed response. The new matched native-CLIP experiments directly address my most fundamental concern: they provide settings in which feature fidelity improves while no statistically detectable frozen-readout accuracy improvement is observed. The native-pathway and public-baseline experiments also make the empirical picture substantially more informative. I therefore plan to raise my score from 1 to 2. However, I still lean toward rejection in this 4-score system. The newly reported clean/corrupted differences are generally below 1 percentage point and sometimes have the opposite sign, so the original diagnostic setting exhibits very little corruption-induced utility degradation. Moreover, the native-projector adaptation gains are only around 0.2 pp, and they disappear or become negative after longer initial training, which substantially weakens the claim that readout adaptation is a separate and effective lever. The Restormer result also shows that conventional restoration can improve frozen utility, while the main Stage1 module still substantially worsens fidelity. Thus, although the rebuttal resolves the strongest evidentiary omission, I do not think it resolves the concerns about effect size, novelty, or the strength and framing of the paper’s central conclusions.

AuthorsOpenReview ↗

Response to zdU5

Thank you for recognizing that the matched native-CLIP experiments address your fundamental concern. Our main contribution is the controlled separation of feature fidelity, frozen-readout utility, and recovery-specific adaptation effects. The raw/recovered × frozen/adapted comparisons distinguish recovery’s contribution from that of additional projector training, including whether adaptation helps recovered features more than raw features.

Two experiments provide complementary evidence. In the matched native-pathway evaluation on 2,400 previously unevaluated GQA images with three seeds, recovery improves feature fidelity without a statistically detectable frozen-readout accuracy gain. Fidelity and accuracy are measured on the same evaluation inputs, retaining the original CLIP encoder and pretrained projector. The earlier Restormer result in the diagnostic pathway provides a complementary outcome: restoration can improve frozen utility. Together, these results underscore the need to measure task benefits directly.

Separately, we examined recovered clean-only projector adaptation in the native pathway, starting from the pretrained projector. With the encoder, recovery module, and language model unchanged, only the projector was trained to align recovered clean-image features with the original projector’s clean visual-token targets. On a fixed 512-image clean validation set, projected-token MSE decreased by 77.6%–81.3% across all three seeds between steps 1 and 750. This demonstrates substantial progress on the token-alignment objective, while the separate corrupted-input GQA evaluation did not establish a reliable adaptation gain. This clean-input training diagnostic is distinct from the matched-input fidelity experiment above; it does not establish downstream-task convergence or rule out undertraining.

We agree that these small task effects do not support a claim of improved robustness. We will clarify the paper’s contribution as controlled evidence distinguishing fidelity improvement, frozen-readout utility, and recovery-specific adaptation effects, and restrict performance claims to the experimental settings supported by the results.

Reviewer_HxFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The present work investigates whether training a representation recovery module to map distribution shifted multimodal samples (e.g. blurred images, noisy image, varied exposure) back to the original data distribution so as to maximize representation fidelity, is beneficial to a multimodal LLM's downstream task. To that end, the authors propose an experimental setup that involves a multimodal feature extractor (here Dinov3), an LLM, an adapter that maps the Dinov3 visual tokens to the LLM's representation space and a recovery module trained to reconstruct the original sample from a domain shifted version. The main goal is to identify in what setup the recovery module is beneficial to the LLM's downstream task. The experiments include freezing the multimodal features and the LLM while training. The recovery module and the adapter are trained separately, the former is frozen in all setups while the adapter is then finetuned in some setups, while only data varies. The main results from the paper are the following:

  • under a frozen adapter, recovery is harmful.
  • fine-tuning the adapter always improves performance.
  • the recovery adapter impact on performance is marginal.
Critical strengths

S1: The paper's experimental setting is clear well designed.

S2: Obtained results are thoroughly explained and commented.

Critical weaknesses

W1: Paper is hard to follow as chosen wordings are rarely used in the literature. We are quite familiar with the literature but have never come across the term "readout" to designate the adapter/projector, "language-side" model is an odd choice of words for LLMs. This is of course not a basis for rejection, but we encourage the authors to stick to commonly used denominations to facilitate reading their work.

W2: Results are not straightforward to interpret. For instance, measuring the impact of the recovery module on different subset of the evaluation set would facilitate formulating conclusions about the experiment: e.g. having separate metrics for the recovered clean data vs recovered shifted data, would make results easier to understand than plain average. (see Q3)

W3: Some figures (e.g. Fig. 4), are hard to read and lack proper descriptions in the caption.

Questions for authors

Q1: Why do authors consider the recovered version of the clean data to train the model? In other words, why is the training set used to evaluate the recovery module ${R_\theta(E(x))} \forall x \in D $ when one could consider E(x)E(x) for clean samples and Rθ(E(x))R_\theta(E(x)) for corrupted samples. This potentially drags down the performance of the recovery (or not impact it at best) on the clean samples. This make interpreting the results somewhat more complicated.

Q2: When it comes to Δrec\Delta_{rec} :

  • We expect RθR_\theta to have a negative impact for the clean samples.
  • We expect RθR_\theta to have a positive impact on corrupted samples.

A negative Δrec\Delta_{rec} could mean that the negative impact on clean samples outweighs the one of corrupted ones, or that both have a negative impact. It would be interesting to have the metrics for the corrupted vs clean samples to better disentangle performance (not only for Δrec\Delta_{rec})?

Q3: Table 1 and 14 have contradiction results. Table 14 show that all recovery variants score higher on GQA than Raw, but Table 1 shows that Recovered-Frozen is below Raw-Frozen. What table is wrong there?

Overall, we recommend rejection but we are open to discussion.

1 comment
AuthorsOpenReview ↗

Response to HxFQ

W1. Terminology

We apologize for the confusing terminology. This was an oversight on our part. In the revision, we will replace “readout” with “vision–language projector” and “language-side model” with “large language model (LLM)” throughout the paper.

W2. Clean/corrupted breakdown

We added separate clean and corrupted results for LLaVA-1.5/GQA; Q2 gives the complete comparison. With recovered features fixed, mixed projector adaptation improves clean and corrupted accuracy by 0.840 and 0.860 percentage points, respectively, with both confidence intervals above zero.

This directly shows where the improvement occurs: the same recovered representations become more useful on both subsets after changing the projector. We will place the clean, corrupted, and aggregate scores side by side in the revised results table so readers can distinguish recovery and adaptation effects.

W3. Figure 4

We apologize for the unclear presentation. We will enlarge labels, improve the layout, use full condition names, and specify the conditions, metric, evaluation subset, and panel meanings in the caption. Panel (b) will present the four-condition utility comparison, with fidelity evidence shown separately.

Q1. Recovery on clean inputs

Applying recovery to both clean and corrupted inputs keeps the feature mapping fixed within the recovered branch:

BranchClean inputsCorrupted inputs
RawE(x)E(x)
RecoveredRθ(E(x))Rθ(E(x))

Adapted projectors share the same clean-trained initialization and train on their respective mappings. Holding this mapping fixed lets us directly measure how changing the projector affects the utility of recovered features.

The added clean-input results show that clean-only adaptation raises clean accuracy from 59.000% to 59.740%, a gain of 0.740 points (95% CI [0.171, 1.283]). Mixed adaptation raises it to 59.840%. This shows that adapting the projector also improves the use of recovered clean features.

Q2. Separating recovery and adaptation

We added the following clean/corrupted breakdown for LLaVA-1.5/GQA in the Table 1 setting:

ConditionClean (%)Corrupted (%)
Raw-Frozen59.33058.815
Recovered-Frozen59.00058.725
Raw-Adapted60.15059.530
Recovered-Adapted (mixed)59.84059.585
Recovered-Adapted (clean-only)59.74059.155

Corrupted pools Medium and Heavy. The paired effects are:

EffectClean, pp [95% CI]Corrupted, pp [95% CI]
Frozen recovery−0.330 [−0.842, 0.184]−0.090 [−0.521, 0.369]
Mixed adaptation of recovered features+0.840 [0.281, 1.384]+0.860 [0.363, 1.340]

The separate recovery estimates identify the contribution of each subset to the aggregate. The adaptation comparison then holds recovered features fixed and shows positive utility gains on both subsets. On corrupted inputs, mixed adaptation also exceeds clean-only continuation by 0.430 points [0.161, 0.696].

Q3. Tables 1 and 14

The two tables use different recovery configurations:

TableRecovery configurationGQA/POPE
Table 1Task-specific Stage158.82/80.91
Table 14, FullOpen Images C5, step 12,00059.09/82.37

The C5 scores match Table 4's Open Images row, and its checkpoint matches Table 19. We will label Table 1 “Recovered-Frozen/Adapted (task-specific)” and Table 14 “Open Images C5 (12,000 steps; frozen vision–language projector).” We will retitle Table 14 “Comparison of recovery configurations” and state the training-data and checkpoint-selection differences.

Reviewer_yHPNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper meticulously study the feature recovery process in multi-modal language models. The proposed controlled system distinguish and ablates the effect of “feature recovery” and “readout adaptation”. With inclusive experiment, the paper reveal the heterogeneous influence of these parts on the final reported utility. The paper concludes an established convention in investigating representation readout and provides insight in such process.

Critical strengths

S1: The paper establishes it’s theoretical analysis framework and study design and the scripting is clear.

S2: The experiments provides clear evidence supporting the claim of the finding that recovery-only utility effects are heterogeneous while readout adaptation improves recovered-input utility.

Critical weaknesses

W1. Is the conclusion limited to the visual encoder. DINOv3, which is a robust vision encoder that the corruption image could be accommodated by the robust pre-trained visual features. While the recovery target distribution (the visual token) is not robust enough when it get’s to corruption.

W2. The metrics numbers differences are modest throughout, there are statistical stability study in the appendix, supporting fair comparison ablation, however, the modest difference does reveal the minor cause the feature recovery and readout adaptation has on the VLM's understanding under visual corruption circumstances.

Questions for authors

Q1: Could the authors provide the baseline scores of their evaluation vanilla model’s inference on these benchmarks.

Q2: Does the study and conclusion applies to other pre-trained vision encoders such as CLIP or SigLip models?

1 comment
AuthorsOpenReview ↗

Response to yHPN

W1. Encoder dependence

We added controlled experiments with LLaVA-1.5's native CLIP encoder and pretrained vision–language projector, covering GQA, TextVQA, MMStar, and BLINK. Within each benchmark, fidelity and utility use the same recovery checkpoints, evaluation samples, and question weights.

The following additional measurements summarize corrupted inputs:

BenchmarkΔ token cosineΔ MSEΔ frozen accuracy, pp [95% CI]
GQA+0.025140−0.137467−0.178 [−0.384, 0.033]
TextVQA+0.017440−0.107416−0.024 [−0.306, 0.254]
MMStar+0.021128−0.094334+0.033 [−0.955, 1.035]
BLINK+0.013968−0.070951+0.184 [−0.298, 0.683]

Recovery improves both fidelity metrics across all four benchmarks, with fidelity-change intervals excluding zero, while no frozen-accuracy gain is statistically detected. These paired measurements extend the fidelity–utility comparison beyond the shared DINOv3 setting.

On GQA, holding recovered features fixed and adapting only the projector improves accuracy by 0.245 percentage points with clean-only adaptation (95% CI [0.067, 0.424]) and 0.178 points with mixed adaptation [0.022, 0.336]. Adaptation starts from the original pretrained projector. Thus, the positive adaptation effect also appears in a native visual pathway, directly supporting the role of the projector in using a fixed recovered representation.

W2. Effect size and significance

Table 1 reports positive adaptation differences in 33/36 model–benchmark combinations, spanning four downstream models and nine benchmarks.

We now report TallyQA results on 2,433 images after excluding images shared with the projector-training pool. All four models retain positive recovered-branch adaptation gains:

ModelAdaptation gain, pp95% CI
LLaVA-1.5+5.886[4.766, 6.969]
Qwen2.5-VL+6.278[5.216, 7.374]
Qwen3.5+1.915[0.874, 3.010]
OneVision-2+2.176[1.287, 3.056]

These are Recovered-Adapted minus Recovered-Frozen differences, averaged across clean, medium, and heavy conditions. For LLaVA-1.5 on corrupted inputs specifically, the gain is 6.062 points [4.937, 7.167].

Together, the broad pattern in Table 1 and these gains demonstrate that projector adaptation can substantially improve the utility of fixed recovered features.

Q1. Native-model baselines

We added unmodified LLaVA-1.5 baselines under the same evaluation conditions:

BenchmarkCorrupted accuracy (%)
GQA69.355
TextVQA43.927
MMStar33.500
BLINK40.873

These preserve the original CLIP encoder, pretrained projector, and LLM.

Q2. Other encoders

The additional CLIP experiments extend the controlled analysis to the original LLaVA-1.5 visual pathway across four benchmarks. They supply two complementary findings: matched fidelity–utility measurements across tasks, and positive projector-adaptation effects on GQA with recovered features fixed.

Comments on the paper

1 comment

Dear Reviewers,

Please read the authors' response. Please evaluate whether your concerns are addressed or if anything needs further attention. Please engage with the authors while we have time.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.