- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether recovering corrupted visual features toward their clean counterparts ("feature fidelity") helps downstream MLLM performance. It also asks whether adapting the vision–language readout is a separate and more effective lever.
Critical strengths
-
The 2×2 factorization (Δrec, Δadapt, Δmarg) is clean and easy to read. Measuring Raw-Adapted, and not only Recovered-Adapted, is the right control, and many recovery papers skip it.
-
Coverage is broad: four MLLM families and nine benchmarks, plus per-corruption, per-severity, checkpoint-sweep and POPE precision/recall breakdowns.
Critical weaknesses
-
The central phenomenon, DUG, is never demonstrated, and the main recovery module lowers fidelity. DUG is defined as fidelity improving while utility does not, under a fixed readout. But Table 21 shows the task-specific module, which produces every Recovered row in Table 1, makes features worse: Cosine to the clean reference drops from 0.988 to 0.894; MSE rises roughly 12× (0.0019 → 0.022); Norm ratio inflates to 1.13. Of course a module that moves features away from the clean manifold hurts a clean-trained readout. Δrec < 0 in 30/36 cells (all 9 for Qwen2.5-VL and Qwen3.5, 8/9 for LLaVA-1.5). The paper calls this "heterogeneous," but it is simply what a broken denoiser should produce. The only module that improves fidelity is the Open Images one, and its gain is tiny (cosine 0.9829 → 0.9836 on ImageNet). Its fidelity and utility are also measured on different datasets, which the paper concedes. So the title claim ("Beyond Feature Fidelity") is never supported by a matched experiment in which fidelity goes up and utility does not.
-
The task-specific recovery module is never described. Appendix A.4 covers only the task-agnostic module. For the module behind the main table, the paper gives no architecture, training data, objective weights or reason for degrading fidelity. Tables 14 and 19 add confusion: their "Full" variant (GQA 59.09, POPE 82.37) matches the Open Images numbers in Table 4, not the task-specific ones (58.82 / 80.91). I cannot tell which module is in which table.
-
The diagnostic system is too degraded to say anything about real MLLMs. Replacing each model's native vision pathway with DINOv3 plus an MLP trained by token regression (under one epoch, see W5) gives near-floor scores, i.e., LLaVA-1.5-7B TextVQA ≈ 10 (native ≈ 58) and VizWiz ≈ 18. At these levels, "readout adaptation helps" mostly means "an undertrained interface improves with more training." The cross-model generality claim is weak because what is being compared is four badly mismatched projectors attached to four LLMs, not four MLLM architectures. I would weigh one experiment at native performance (native encoder, native projector, feature recovery inserted in the native pathway) above all 36 current cells.
-
Baselines are missing. The paper evaluates no existing recovery method: not DVT, not the Parikh et al. latent-denoising method it cites, and no pixel-space restorer (e.g., NAFNet or Restormer before the encoder). It also has no robust-encoder baseline. A finding of the form "feature recovery doesn't help" needs to show it for recovery methods the community uses, not only for in-house modules, one of which damages fidelity.
Questions for authors
-
Can you show one matched setting where fidelity improves and utility under the frozen readout does not, with both measured on the same evaluation images?
-
What is Raw-Frozen on the clean split alone versus the corrupted splits alone, per benchmark? How much does corruption actually hurt this system?
-
Why does the clean interface train for only 750 steps (under one epoch)? If you train it to convergence, does Δadapt survive?
-
What data is used for adaptation (the 7,691 samples in Table 11)? Does it share images or corruption families with the evaluation sets? The paper says heavy settings are held out, yet severity 3 appears at test time; please clarify.
3 comments
Thank you for the detailed response. The new matched native-CLIP experiments directly address my most fundamental concern: they provide settings in which feature fidelity improves while no statistically detectable frozen-readout accuracy improvement is observed. The native-pathway and public-baseline experiments also make the empirical picture substantially more informative. I therefore plan to raise my score from 1 to 2. However, I still lean toward rejection in this 4-score system. The newly reported clean/corrupted differences are generally below 1 percentage point and sometimes have the opposite sign, so the original diagnostic setting exhibits very little corruption-induced utility degradation. Moreover, the native-projector adaptation gains are only around 0.2 pp, and they disappear or become negative after longer initial training, which substantially weakens the claim that readout adaptation is a separate and effective lever. The Restormer result also shows that conventional restoration can improve frozen utility, while the main Stage1 module still substantially worsens fidelity. Thus, although the rebuttal resolves the strongest evidentiary omission, I do not think it resolves the concerns about effect size, novelty, or the strength and framing of the paper’s central conclusions.
Response to zdU5
Thank you for recognizing that the matched native-CLIP experiments address your fundamental concern. Our main contribution is the controlled separation of feature fidelity, frozen-readout utility, and recovery-specific adaptation effects. The raw/recovered × frozen/adapted comparisons distinguish recovery’s contribution from that of additional projector training, including whether adaptation helps recovered features more than raw features.
Two experiments provide complementary evidence. In the matched native-pathway evaluation on 2,400 previously unevaluated GQA images with three seeds, recovery improves feature fidelity without a statistically detectable frozen-readout accuracy gain. Fidelity and accuracy are measured on the same evaluation inputs, retaining the original CLIP encoder and pretrained projector. The earlier Restormer result in the diagnostic pathway provides a complementary outcome: restoration can improve frozen utility. Together, these results underscore the need to measure task benefits directly.
Separately, we examined recovered clean-only projector adaptation in the native pathway, starting from the pretrained projector. With the encoder, recovery module, and language model unchanged, only the projector was trained to align recovered clean-image features with the original projector’s clean visual-token targets. On a fixed 512-image clean validation set, projected-token MSE decreased by 77.6%–81.3% across all three seeds between steps 1 and 750. This demonstrates substantial progress on the token-alignment objective, while the separate corrupted-input GQA evaluation did not establish a reliable adaptation gain. This clean-input training diagnostic is distinct from the matched-input fidelity experiment above; it does not establish downstream-task convergence or rule out undertraining.
We agree that these small task effects do not support a claim of improved robustness. We will clarify the paper’s contribution as controlled evidence distinguishing fidelity improvement, frozen-readout utility, and recovery-specific adaptation effects, and restrict performance claims to the experimental settings supported by the results.
Response to zdU5
W1/Q1. Matched evidence
We added native-CLIP evaluations on four benchmarks. Within each benchmark, fidelity and utility use identical checkpoints, samples, and question weights; the pretrained vision–language projector and LLM remain frozen.
Both fidelity metrics improve with intervals excluding zero; no frozen-accuracy gain is statistically detected. These matched results support evaluating fidelity and utility separately.
W2. Recovery module
We apologize for the omitted description and unclear labels. Stage1 is a residual dual-stream VAE: input/output 1024; content hidden/latent 1024/128; nuisance 512/64; decoder hidden 1024; GELU/LayerNorm. The checkpoint at step 8,000 was trained with AdamW, LR 3e−4, effective batch 16. Loss weights: cosine/MSE/Smooth-L1 1/1/0.5; cycle 0.3; relation 0.2; global cosine 0.1; KL 0.005; total correlation 0.02; minimum clean-identity/residual-L2 0.01/0.02. We will specify curriculum schedules.
Training uses paired clean/corrupted DINOv3 features without QA labels.
Tables 1/21 and Table 4's task-specific rows use Stage1; Table 14's Full and Table 19's C5 use Open Images step 12,000. We will label Table 14 as a recovery-configuration comparison.
We added an error decomposition on 256 evaluation images: ΔMSE = mean(r²) − 2mean(r·d), where r=R(z)−z and d=y−z. On corrupted inputs, Stage1's terms are 0.022957 and −0.000092, yielding ΔMSE +0.023049; C5 yields −0.000309. Thus, the measured fidelity loss occurs in the recovery correction, before projection.
W3. Native pathway
We inserted recovery between LLaVA-1.5's native CLIP and pretrained projector. Under the same evaluation conditions, native corrupted-input baselines (GQA/TextVQA/MMStar/BLINK) are 69.355/43.927/33.500/40.873%.
On 779 GQA images, adapting the projector from its pretrained weights while holding recovered features fixed improves accuracy by 0.245 pp [0.067, 0.424] with clean-only adaptation and 0.178 pp [0.022, 0.336] with mixed adaptation. Thus, adapting a pretrained native projector also improves recovered-feature utility.
W4. Public baseline
We added official blind-denoising Restormer before DINOv3 on 779 GQA images (σ=15/25/50). Across three validation-stopped projector starts, PSNR rises 20.234→30.173 dB and frozen accuracy improves 0.322 pp [0.025, 0.619]. Subsequent mixed adaptation changes accuracy by −0.319 pp [−0.495, −0.145], separating restoration benefit from adaptation.
Q2. Clean/corrupted scores
We added separate Raw-Frozen scores for the four systems in Table 1 (Clean/Corrupted, %; Corrupted pools Medium/Heavy).
Across 36 settings, corruption effects range from −0.933 to +0.800 pp and vary by task.
Q3. Longer training
The 750 steps set a shared optimization budget. In additional longer-training experiments, we used 64,926 training/7,214 validation images, with validation loss checked every 500 steps. Patience stopping occurs at 31k/32k/37k steps; minimum-validation-loss checkpoints are selected. On corrupted GQA, another 750 mixed-adaptation steps yield −0.082 pp [−0.516, 0.351] for Stage1 and −0.237 pp [−0.379, −0.095] for C5; frozen C5 improves over Raw by 0.193 pp [0.099, 0.290]. This separates recovery gains from adaptation effects after extended initial training.
Q4. Training and overlap
The projector pool contains 72,140 GQA images; Table 11's 7,691-image subset is for data scaling. Adaptation regresses clean native tokens without QA labels. Training uses severity 1/2; heavy evaluation uses severity 3 in the same five corruption families.
Our ID audit found no GQA overlap; TallyQA has 4,919/7,352 overlapping images. On the remaining 2,433 images, adaptation gains are 5.886/6.278/1.915/2.176 pp, with CIs [4.766,6.969]/[5.216,7.374]/[0.874,3.010]/[1.287,3.056], in table model order (clean/medium/heavy average). The positive adaptation effect therefore persists across all four models after excluding recorded projector-training overlaps.