- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GLASS, a dense post-training recipe for generative vision encoders. It retains caption prediction while adding masked feature recovery and layout recovery. Caption-to-patch attention selects the most relevant half of the patches as visible context, while the student predicts EMA-teacher features at hidden locations. A separate position-free encoding pass predicts the visible patches’ normalized coordinates. The paper also identifies a failure pattern under anti-semantic masking, where dense representations deteriorate despite relatively stable caption loss. Experiments on GenLIP and OpenVision 2 demonstrate improvements in segmentation, depth estimation, and downstream captioning, while approximately preserving global recognition and aggregate multimodal performance.
Critical strengths
- Clear motivation. Improving dense representations without losing the capabilities of caption-pretrained encoders is a meaningful problem. The paper explicitly considers the interaction between the evidence available for captioning and the targets used for masked recovery.
- Substantial dense gains. Improvements extend across segmentation datasets, depth benchmarks, two encoder families, and multiple model sizes. For GenLIP-L, GLASS improves ADE20K mIoU from 41.0 to 47.2 and Cityscapes from 61.6 to 70.1, while reducing NYUv2 RMSE from 0.507 to 0.438.
- Useful ablations. The component experiments demonstrate complementary contributions from masked feature recovery and layout recovery. Additional comparisons examine masking rules, visible ratios, coordinate versus categorical targets, caption-augmented baselines, and full-backbone DIP adaptation. Training-free retrieval segmentation also supports improvements beyond learned linear probes.
- Informative diagnostics. Appendix C.4 strengthens the observation beyond caption-loss curves and feature similarity. Under anti-semantic masking, the image-sensitivity gap decreases from 1.609 to 0.638 and correct-image retrieval falls from 97.6% to 75.7%, providing evidence of reduced sensitivity to image–caption correspondence.
Critical weaknesses
- Incomplete comparisons. The evaluation focuses on DIP and NeCo but omits generative-feedback approaches such as DIVA [R1] and GenHancer[R2], which provide another route to improving fine-grained visual representations. Although these methods primarily target contrastively pretrained CLIP rather than caption-pretrained encoders, the paper should discuss this distinction and, where feasible, adapt a representative approach to the same backbone for further comparisons. Furthermore, GLASS uses Recap-DataComp-12M, whereas DIP and NeCo follow different COCO-based training recipes. These differences in data, captions, and budgets limit attribution of the observed advantage to GLASS’s objectives. Comparisons under shared data and matched training budgets would strengthen the conclusions.
- Masking mechanism. The experiments convincingly establish that patch selection matters, but do not fully isolate why. Changing the mask simultaneously changes caption evidence, masked-feature reconstruction targets, and the patches supplied to layout recovery. Consequently, the results cannot determine how much degradation comes specifically from increased language-prior reliance versus changes in the difficulty or informativeness of the auxiliary tasks. The image–caption correspondence diagnostic supports the proposed interpretation, but does not establish the causal relationship between language-prior reliance and dense-feature degradation. Experiments without LM supervision, branch-specific masking controls, and variations in caption supervision would help clarify the mechanism.
- Layout interpretation. Removing positional inputs prevents direct coordinate lookup, but does not establish that coordinate regression learns transferable spatial relationships. The position-free branch may exploit semantic location priors, such as sky patches appearing near the top and road patches near the bottom. These priors can improve standard segmentation and depth benchmarks without establishing robustness to unusual layouts. The coordinate-versus-categorical comparison supports the chosen loss, but does not resolve this distinction. Evaluations under controlled translations, flips, or rearranged scenes, together with coordinate-prediction errors, would help distinguish geometric understanding from appearance-based absolute-position prediction.
- Retention evidence. The multimodal results demonstrate approximately preserved performance after separately fine-tuning the downstream projector and language model. This establishes feature utility under adaptation, but provides a narrower assessment of retention than directly evaluating the original generative pathway. Appendix C.4 offers a useful direct diagnostic on GenLIP, although broader original-pathway evaluation remains limited. Multimodal transfer is also tested with only one 1.5B language model. Finally, the aggregate changes of +0.2 and +0.5 points are small, include task-level decreases, and lack training-run uncertainty estimates. The bootstrap intervals in the correspondence diagnostic quantify image-sampling uncertainty rather than variability across training runs.
Refs: [R1]. Diffusion Feedback Helps CLIP See Better. ICLR 2025. [R2]. GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers. ICCV 2025.
Questions for authors
- Comparisons: How do DIP+LM, NeCo+LM, and GLASS compare when trained on the same Recap-DataComp subset with matched update budgets and comparable hyperparameter tuning?
- Masking: What happens when GLASS is trained without LM supervision? Can caption-view masking be varied independently of masked recovery and layout inputs to isolate the proposed text-prior mechanism?
- Layout: Does layout recovery remain beneficial on translated, flipped, or unusually arranged scenes? Can you report coordinate-prediction errors and tests that distinguish semantic position priors from transferable spatial representations?
- Retention: Can you evaluate the original captioning pathway on both encoder families and test transfer with another language-model size? Please also report training-seed variability for the main dense results and the small multimodal performance changes.
4 comments
Response to Reviewer DcN2 (2/3)
W2 & Q2: Ablating LM supervision and separating masking effects
Removing LM preserves strong dense transfer but lowers downstream multimodal performance. We train GenLIP-L/16 with MR + LR only, keeping the default semantic masking and the remaining training settings of Table 4 unchanged. Caption attention still determines patch selection; only the caption loss is removed. We evaluate the frozen encoder with the same dense probes and downstream Qwen2.5-1.5B VLM protocol as full GLASS. The pretrained and full GLASS references below are from Table 4.
| Objectives | PASCAL ↑ | NYUv2 ↓ | ChartQA ↑ | OCR-B ↑ | MME-P ↑ | SQA ↑ |
|---|---|---|---|---|---|---|
| Pretrained | 70.8 | 0.507 | 41.2 | 51.1 | 1258 | 76.1 |
| LM + MR + LR (GLASS) | 78.9 | 0.438 | 42.4 | 51.0 | 1264 | 76.1 |
| MR + LR | 78.4 | 0.425 | 38.9 | 48.6 | 1223 | 73.3 |
Without LM, PASCAL mIoU changes from 78.9 to 78.4 and NYUv2 RMSE improves from 0.438 to 0.425, with both remaining better than the pretrained reference. However, ChartQA, OCR-B, MME-P, and SQA decrease by 3.5, 2.4, 41, and 2.8 points, respectively, and all four fall below the pretrained reference. Under semantic masking, MR and LR thus provide strong dense transfer without the caption loss, while retaining LM helps preserve downstream multimodal performance.
We next test whether the dense degradation under anti-semantic masking requires auxiliary recovery objectives. We compare semantic and anti-semantic LM-only training on GenLIP-L/16, keeping the remaining settings of Table 4 unchanged. We use the same dense probes and the native teacher-forced caption diagnostic in Appendix C.4, with the same 1,000 COCO image–caption pairs and mismatched-image candidates. NLL and the image-sensitivity gap are in nats per target token; Retrieval@1 selects the correct image among six candidates.
| Masking | Objectives | PASCAL ↑ | NYUv2 ↓ | Correct NLL | Retrieval@1 (%) | |
|---|---|---|---|---|---|---|
| Pretrained | -- | 70.8 | 0.507 | 3.956 | 1.609 | 97.6 |
| Semantic | LM | 71.4 | 0.505 | 3.821 | 1.678 | 97.6 |
| Semantic | LM + MR + LR | 78.9 | 0.438 | 3.860 | 1.677 | 97.5 |
| Anti-semantic | LM + MR + LR | 66.0 | 0.521 | 4.014 | 0.638 | 75.7 |
| Anti-semantic | LM | 64.2 | 0.566 | 4.102 | 0.703 | 75.4 |
Anti-semantic LM-only training reduces PASCAL mIoU from the pretrained 70.8 to 64.2 and raises NYUv2 RMSE from 0.507 to 0.566, whereas semantic LM-only yields 71.4 and 0.505. Auxiliary recovery losses are therefore not necessary for the dense degradation to occur. Adding MR and LR under anti-semantic masking partially mitigates it, yielding 66.0 mIoU and 0.521 RMSE.
The LM-only anti-semantic run also has a smaller image-sensitivity gap (0.703 versus 1.678) and lower Retrieval@1 (75.4% versus 97.6%) than semantic LM-only. Dense degradation thus occurs together with reduced image dependence even without auxiliary recovery losses. Under anti-semantic masking, caption-relevant patches are hidden, but teacher-forced text can still make many target tokens predictable. A plausible account is that textual context continues to sustain caption prediction as image-specific conditioning weakens, so caption supervision alone need not preserve visually discriminative features.
We will include these objective ablations in the revised manuscript.
W3 & Q3: Layout recovery and semantic position priors
The geometric supervision in LR refers to its distance-sensitive coordinate-regression objective. Our claim concerns its benefit to dense transfer, supported by within-backbone downstream ablations. We do not claim robustness to controlled layout transformations. Semantic location priors, such as sky typically appearing above roads, are regularities of natural scenes and can provide useful cues for dense prediction. LR aims to enrich patch representations with spatial information, not to eliminate all such priors.
From the training logs, the final coordinate L1 errors are 0.3798 for GenLIP-L/16 and 0.3431 for OpenVision 2-L/14. These are unweighted means of the per-patch sum of absolute horizontal and vertical coordinate errors, with target coordinates normalized to [0,1].
To evaluate transfer beyond segmentation and depth estimation, we compare the pretrained GenLIP-L/16 encoder, LM + MR, and full GLASS on CV-Bench using the same frozen-encoder LLaVA-NeXT-Qwen2.5-1.5B protocol. Its tasks cover object counting, 2D spatial relations, depth ordering, and relative distance. The table reports task accuracies (%) and the official 2D, 3D, and Overall summaries.
| Objectives | Count ↑ | Relation ↑ | Depth ↑ | Distance ↑ | 2D ↑ | 3D ↑ | Overall ↑ |
|---|---|---|---|---|---|---|---|
| Pretrained | 60.02 | 52.61 | 55.83 | 51.16 | 56.24 | 53.50 | 54.87 |
| LM + MR | 57.74 | 51.54 | 53.33 | 50.33 | 54.72 | 51.83 | 53.28 |
| LM + MR + LR (GLASS) | 60.15 | 52.15 | 55.50 | 51.33 | 56.19 | 53.42 | 54.81 |
Response to Reviewer DcN2 (3/3)
Adding LR to LM + MR improves all four tasks by 0.61–2.41 points and raises Overall from 53.28 to 54.81, close to the pretrained 54.87. Each task is within 0.5 points of its pretrained score. This within-backbone comparison supports LR's contribution to explicit spatial-task performance alongside the dense gains in Table 4.
W4 & Q4: Original generative pathway and larger-MLLM transfer
Appendix C.4 directly evaluates the original GenLIP-L/16 captioning pathway, with all parameters fixed and no downstream adaptation. On the same 1,000 COCO image–caption pairs under teacher forcing, correct-pair NLL changes from 3.956 to 3.860 and the image-sensitivity gap from 1.609 to 1.677 nats per target token, while Retrieval@1 changes from 97.6% to 97.5%. These results support preserved image–caption correspondence sensitivity through GenLIP's native pathway.
We additionally evaluate GenLIP-L/16 with Qwen2.5-7B-Instruct under the same LLaVA-NeXT protocol as the 1.5B experiments. The vision encoder is frozen, while a two-layer MLP projector and the language model are trained on LLaVA OneVision data. The 1.5B rows are from Table 2 of our paper; the 7B pretrained row is the published reference from Table 4 of the GenLIP paper, and the GLASS 7B row contains our new measurements. Δ denotes the mean change from the corresponding pretrained encoder within each metric group, with MME-P converted to a percentage of its 2,000-point maximum.
| LLM size | Encoder | ChartQA ↑ | TextVQA ↑ | OCR-B ↑ | SEED-2 ↑ | MME-P ↑ | SQA ↑ | Δ Multimodal | TextCaps ↑ | NoCaps ↑ | Δ Captioning |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1.5B | GenLIP, pretrained | 41.2 | 53.6 | 51.1 | 51.1 | 1258 | 76.1 | -- | 131.4 | 82.6 | -- |
| 1.5B | GenLIP + GLASS | 42.4 | 54.2 | 51.0 | 50.3 | 1264 | 76.1 | +0.2 | 135.4 | 84.1 | +2.8 |
| 7B | GenLIP, pretrained | 52.7 | 62.9 | 59.2 | 59.0 | 1320 | 85.4 | -- | 139.4 | 81.1 | -- |
| 7B | GenLIP + GLASS | 53.4 | 62.9 | 59.0 | 59.1 | 1342 | 85.6 | +0.3 | 142.2 | 83.5 | +2.6 |
With Qwen2.5-7B, GLASS shows approximate retention of downstream multimodal performance (mean change +0.3 points), together with higher captioning scores (+2.6 CIDEr on average). OCR-B decreases by 0.2 points, TextVQA is unchanged, and the other four multimodal scores increase; both captioning scores improve. This matches the 1.5B pattern (+0.2 and +2.8): aggregate multimodal performance near the pretrained encoder, with higher downstream captioning scores. We will include this additional 7B evaluation in the revised manuscript.
W4 & Q4: Training-run variability and small multimodal differences
Our training runs use a fixed random seed of 0, and we report single-run results. Due to the computational cost, we have not repeated encoder post-training or downstream projector/LLM training with independent seeds, and therefore cannot report training-run standard deviations for the dense or multimodal scores.
In the reported runs, GLASS improves dense perception while keeping aggregate multimodal scores close to the pretrained references. The main paper reports mean multimodal changes of +0.2 points for GenLIP and +0.5 for OpenVision 2, alongside small task-level increases and decreases. We describe this pattern, also observed with the 7B LLM, as approximate retention in the evaluated runs, not a statistically significant improvement.
Looking forward to further feedback
Dear Reviewer DcN2:
We appreciate the time and effort you have dedicated to providing insightful review. We would like to kindly follow up on our response and hope it has addressed your concerns.
As the discussion period approaches its conclusion, we would be grateful for any further comments or suggestions. We are happy to clarify anything further during the discussion period.
Thank you again for your valuable insights, and we look forward to any updates you may have.
Best, The Authors
Response to Reviewer DcN2 (1/3)
Thank you for the detailed assessment and for recognizing the dense-perception gains, complementary ablations, and image–caption correspondence diagnostic. We address your comments and questions point by point below.
We have added a shared-data comparison of DIP + LM, NeCo + LM, and GLASS on GenLIP-L/16. The original comparisons follow the baselines' published COCO-based recipes, while GLASS uses Recap-DataComp-12M because both encoder families were pretrained on DataComp. The new comparison uses a fixed 1M-image subset of Recap-DataComp-12M, keeping the experiment practical given DIP's offline pseudo-label generation.
All three methods retain LM supervision, see the same image–caption pairs for one pass, and use the same number of optimizer updates. We evaluate the frozen encoders with the same dense probes and downstream Qwen2.5-1.5B VLM protocol. Online costs are A800 device-hours; DIP's offline cost is L40S device-hours.
GLASS matches NeCo + LM on PASCAL mIoU (74.8) and achieves lower NYUv2 RMSE (0.469 versus 0.488). Compared with DIP + LM, it gains 2.8 PASCAL mIoU points and reduces NYUv2 RMSE from 0.507 to 0.469. GLASS also scores higher than both baselines on all four multimodal and both captioning benchmarks, with these scores remaining close to the pretrained reference.
GLASS uses 51.2 A800 device-hours, 35.8% fewer than NeCo + LM (79.8). DIP + LM uses 37.6 A800 device-hours online and an additional 100.8 L40S device-hours for offline pseudo-label generation; GLASS uses 13.6 more online device-hours but requires no offline stage.
These results show that GLASS's dense–multimodal balance persists under common data exposure and matched optimizer-update counts. We will include this shared-data comparison in the main-text Experiments section of the revised manuscript.
DIP and NeCo are direct dense post-training baselines: their objectives improve patch representations for dense prediction. DIVA (Diffusion Feedback Helps CLIP See Better, ICLR 2025) and GenHancer (Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers, ICCV 2025) instead use generative reconstruction feedback to improve fine-grained perception in contrastively pretrained CLIP. Their primary objective and pretraining setting differ from the dense post-training of caption-pretrained encoders studied here.
To test whether this alternative route transfers to our backbone, we adapted DIVA to GenLIP-L/16. The reproduction includes GenLIP interface adaptation and corrections to image normalization and loss scaling. We evaluate dense prediction and global recognition; the pretrained and GLASS references below are from the main paper.
The adapted DIVA run leaves classification and segmentation close to the pretrained reference: ImageNet top-1 remains 82.8%, PASCAL mIoU changes from 70.8 to 70.9, and Cityscapes mIoU remains 61.6. KITTI RMSE increases from 3.554 to 3.810. In comparison, GLASS reaches 78.9 PASCAL mIoU, 70.1 Cityscapes mIoU, and 3.277 KITTI RMSE. Thus, this adapted generative-feedback baseline does not reproduce GLASS's dense gains under the evaluated GenLIP settings.
We will include this comparison and clarify the objective and backbone distinctions in the revised manuscript.