acceptodds
Withdrawn from ICLR 2027

GLASS: Unlocking Dense Perception in Generative Vision Encoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative vision encoders pretrained with an autoregressive caption objective transfer well to multimodal LLMs, but their patch-level representations lag on dense perception. Caption prediction does not explicitly supervise the geometry or absolute position of individual patches. In our transfer experiments, existing dense post-training recipes improve dense prediction but reduce downstream captioning and multimodal understanding. We explore retaining the caption objective while adding masked feature recovery. Here the mask selects both the recovery targets and the visual evidence available for caption prediction. Hiding the patches receiving the most caption attention produces a failure pattern we term text prior collapse. Features from different images become more similar and dense performance declines, yet the caption loss remains close to the other masking runs and does not reflect the extent of the degradation. We propose Grounded LAyout and Semantic maSking (GLASS), a caption-guided dense post-training recipe. Semantic masking keeps the highest-attention half of the patches visible, and masked feature recovery uses this context to predict full-image teacher features at hidden locations. Layout recovery adds explicit geometric supervision by re-encoding the visible patches without positional information and regressing their normalized coordinates. On GenLIP and OpenVision 2, GLASS improves dense perception and downstream captioning, with near-pretrained ImageNet accuracy and aggregate multimodal scores under the evaluated protocols. Further dense evaluations show gains across model scales.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dense Percaption, Generative Vision Encoder, Vision-Language Representation Learning, Visual Encoder Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DcN2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GLASS, a dense post-training recipe for generative vision encoders. It retains caption prediction while adding masked feature recovery and layout recovery. Caption-to-patch attention selects the most relevant half of the patches as visible context, while the student predicts EMA-teacher features at hidden locations. A separate position-free encoding pass predicts the visible patches’ normalized coordinates. The paper also identifies a failure pattern under anti-semantic masking, where dense representations deteriorate despite relatively stable caption loss. Experiments on GenLIP and OpenVision 2 demonstrate improvements in segmentation, depth estimation, and downstream captioning, while approximately preserving global recognition and aggregate multimodal performance.

Critical strengths
  1. Clear motivation. Improving dense representations without losing the capabilities of caption-pretrained encoders is a meaningful problem. The paper explicitly considers the interaction between the evidence available for captioning and the targets used for masked recovery.
  2. Substantial dense gains. Improvements extend across segmentation datasets, depth benchmarks, two encoder families, and multiple model sizes. For GenLIP-L, GLASS improves ADE20K mIoU from 41.0 to 47.2 and Cityscapes from 61.6 to 70.1, while reducing NYUv2 RMSE from 0.507 to 0.438.
  3. Useful ablations. The component experiments demonstrate complementary contributions from masked feature recovery and layout recovery. Additional comparisons examine masking rules, visible ratios, coordinate versus categorical targets, caption-augmented baselines, and full-backbone DIP adaptation. Training-free retrieval segmentation also supports improvements beyond learned linear probes.
  4. Informative diagnostics. Appendix C.4 strengthens the observation beyond caption-loss curves and feature similarity. Under anti-semantic masking, the image-sensitivity gap decreases from 1.609 to 0.638 and correct-image retrieval falls from 97.6% to 75.7%, providing evidence of reduced sensitivity to image–caption correspondence.
Critical weaknesses
  1. Incomplete comparisons. The evaluation focuses on DIP and NeCo but omits generative-feedback approaches such as DIVA [R1] and GenHancer[R2], which provide another route to improving fine-grained visual representations. Although these methods primarily target contrastively pretrained CLIP rather than caption-pretrained encoders, the paper should discuss this distinction and, where feasible, adapt a representative approach to the same backbone for further comparisons. Furthermore, GLASS uses Recap-DataComp-12M, whereas DIP and NeCo follow different COCO-based training recipes. These differences in data, captions, and budgets limit attribution of the observed advantage to GLASS’s objectives. Comparisons under shared data and matched training budgets would strengthen the conclusions.
  2. Masking mechanism. The experiments convincingly establish that patch selection matters, but do not fully isolate why. Changing the mask simultaneously changes caption evidence, masked-feature reconstruction targets, and the patches supplied to layout recovery. Consequently, the results cannot determine how much degradation comes specifically from increased language-prior reliance versus changes in the difficulty or informativeness of the auxiliary tasks. The image–caption correspondence diagnostic supports the proposed interpretation, but does not establish the causal relationship between language-prior reliance and dense-feature degradation. Experiments without LM supervision, branch-specific masking controls, and variations in caption supervision would help clarify the mechanism.
  3. Layout interpretation. Removing positional inputs prevents direct coordinate lookup, but does not establish that coordinate regression learns transferable spatial relationships. The position-free branch may exploit semantic location priors, such as sky patches appearing near the top and road patches near the bottom. These priors can improve standard segmentation and depth benchmarks without establishing robustness to unusual layouts. The coordinate-versus-categorical comparison supports the chosen loss, but does not resolve this distinction. Evaluations under controlled translations, flips, or rearranged scenes, together with coordinate-prediction errors, would help distinguish geometric understanding from appearance-based absolute-position prediction.
  4. Retention evidence. The multimodal results demonstrate approximately preserved performance after separately fine-tuning the downstream projector and language model. This establishes feature utility under adaptation, but provides a narrower assessment of retention than directly evaluating the original generative pathway. Appendix C.4 offers a useful direct diagnostic on GenLIP, although broader original-pathway evaluation remains limited. Multimodal transfer is also tested with only one 1.5B language model. Finally, the aggregate changes of +0.2 and +0.5 points are small, include task-level decreases, and lack training-run uncertainty estimates. The bootstrap intervals in the correspondence diagnostic quantify image-sampling uncertainty rather than variability across training runs.

Refs: [R1]. Diffusion Feedback Helps CLIP See Better. ICLR 2025. [R2]. GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers. ICCV 2025.

Questions for authors
  1. Comparisons: How do DIP+LM, NeCo+LM, and GLASS compare when trained on the same Recap-DataComp subset with matched update budgets and comparable hyperparameter tuning?
  2. Masking: What happens when GLASS is trained without LM supervision? Can caption-view masking be varied independently of masked recovery and layout inputs to isolate the proposed text-prior mechanism?
  3. Layout: Does layout recovery remain beneficial on translated, flipped, or unusually arranged scenes? Can you report coordinate-prediction errors and tests that distinguish semantic position priors from transferable spatial representations?
  4. Retention: Can you evaluate the original captioning pathway on both encoder families and test transfer with another language-model size? Please also report training-seed variability for the main dense results and the small multimodal performance changes.
4 comments
AuthorsOpenReview ↗

Response to Reviewer DcN2 (1/3)

Thank you for the detailed assessment and for recognizing the dense-perception gains, complementary ablations, and image–caption correspondence diagnostic. We address your comments and questions point by point below.

W1 & Q1: Shared training data and training budgets

We have added a shared-data comparison of DIP + LM, NeCo + LM, and GLASS on GenLIP-L/16. The original comparisons follow the baselines' published COCO-based recipes, while GLASS uses Recap-DataComp-12M because both encoder families were pretrained on DataComp. The new comparison uses a fixed 1M-image subset of Recap-DataComp-12M, keeping the experiment practical given DIP's offline pseudo-label generation.

All three methods retain LM supervision, see the same image–caption pairs for one pass, and use the same number of optimizer updates. We evaluate the frozen encoders with the same dense probes and downstream Qwen2.5-1.5B VLM protocol. Online costs are A800 device-hours; DIP's offline cost is L40S device-hours.

MethodImages (M)Total tokens (B)Online device-h (Offline device-h)PASCAL ↑NYUv2 ↓ChartQA ↑OCR-B ↑MME-P ↑SQA ↑TextCaps ↑NoCaps ↑
Pretrained------70.80.50741.251.1125876.1131.482.6
DIP + LM1.00.2737.6 (100.8)72.00.50740.850.4125075.6127.780.0
NeCo + LM1.01.0079.8 (--)74.80.48838.248.0120167.2126.877.1
GLASS (Ours)1.00.6151.2 (--)74.80.46942.150.9125876.2132.483.0

GLASS matches NeCo + LM on PASCAL mIoU (74.8) and achieves lower NYUv2 RMSE (0.469 versus 0.488). Compared with DIP + LM, it gains 2.8 PASCAL mIoU points and reduces NYUv2 RMSE from 0.507 to 0.469. GLASS also scores higher than both baselines on all four multimodal and both captioning benchmarks, with these scores remaining close to the pretrained reference.

GLASS uses 51.2 A800 device-hours, 35.8% fewer than NeCo + LM (79.8). DIP + LM uses 37.6 A800 device-hours online and an additional 100.8 L40S device-hours for offline pseudo-label generation; GLASS uses 13.6 more online device-hours but requires no offline stage.

These results show that GLASS's dense–multimodal balance persists under common data exposure and matched optimizer-update counts. We will include this shared-data comparison in the main-text Experiments section of the revised manuscript.

W1: Missing generative-feedback approaches

DIP and NeCo are direct dense post-training baselines: their objectives improve patch representations for dense prediction. DIVA (Diffusion Feedback Helps CLIP See Better, ICLR 2025) and GenHancer (Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers, ICCV 2025) instead use generative reconstruction feedback to improve fine-grained perception in contrastively pretrained CLIP. Their primary objective and pretraining setting differ from the dense post-training of caption-pretrained encoders studied here.

To test whether this alternative route transfers to our backbone, we adapted DIVA to GenLIP-L/16. The reproduction includes GenLIP interface adaptation and corrections to image normalization and loss scaling. We evaluate dense prediction and global recognition; the pretrained and GLASS references below are from the main paper.

EncoderIN top-1 ↑PASCAL ↑Cityscapes ↑NYUv2 ↓KITTI ↓
Pretrained82.870.861.60.5073.554
DIVA, adapted82.870.961.60.5293.810
GLASS82.978.970.10.4383.277

The adapted DIVA run leaves classification and segmentation close to the pretrained reference: ImageNet top-1 remains 82.8%, PASCAL mIoU changes from 70.8 to 70.9, and Cityscapes mIoU remains 61.6. KITTI RMSE increases from 3.554 to 3.810. In comparison, GLASS reaches 78.9 PASCAL mIoU, 70.1 Cityscapes mIoU, and 3.277 KITTI RMSE. Thus, this adapted generative-feedback baseline does not reproduce GLASS's dense gains under the evaluated GenLIP settings.

We will include this comparison and clarify the objective and backbone distinctions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer DcN2 (2/3)

W2 & Q2: Ablating LM supervision and separating masking effects

Removing LM preserves strong dense transfer but lowers downstream multimodal performance. We train GenLIP-L/16 with MR + LR only, keeping the default semantic masking and the remaining training settings of Table 4 unchanged. Caption attention still determines patch selection; only the caption loss is removed. We evaluate the frozen encoder with the same dense probes and downstream Qwen2.5-1.5B VLM protocol as full GLASS. The pretrained and full GLASS references below are from Table 4.

ObjectivesPASCAL ↑NYUv2 ↓ChartQA ↑OCR-B ↑MME-P ↑SQA ↑
Pretrained70.80.50741.251.1125876.1
LM + MR + LR (GLASS)78.90.43842.451.0126476.1
MR + LR78.40.42538.948.6122373.3

Without LM, PASCAL mIoU changes from 78.9 to 78.4 and NYUv2 RMSE improves from 0.438 to 0.425, with both remaining better than the pretrained reference. However, ChartQA, OCR-B, MME-P, and SQA decrease by 3.5, 2.4, 41, and 2.8 points, respectively, and all four fall below the pretrained reference. Under semantic masking, MR and LR thus provide strong dense transfer without the caption loss, while retaining LM helps preserve downstream multimodal performance.

We next test whether the dense degradation under anti-semantic masking requires auxiliary recovery objectives. We compare semantic and anti-semantic LM-only training on GenLIP-L/16, keeping the remaining settings of Table 4 unchanged. We use the same dense probes and the native teacher-forced caption diagnostic in Appendix C.4, with the same 1,000 COCO image–caption pairs and mismatched-image candidates. NLL and the image-sensitivity gap are in nats per target token; Retrieval@1 selects the correct image among six candidates.

MaskingObjectivesPASCAL ↑NYUv2 ↓Correct NLLΔimg\Delta_{\mathrm{img}}Retrieval@1 (%)
Pretrained--70.80.5073.9561.60997.6
SemanticLM71.40.5053.8211.67897.6
SemanticLM + MR + LR78.90.4383.8601.67797.5
Anti-semanticLM + MR + LR66.00.5214.0140.63875.7
Anti-semanticLM64.20.5664.1020.70375.4

Anti-semantic LM-only training reduces PASCAL mIoU from the pretrained 70.8 to 64.2 and raises NYUv2 RMSE from 0.507 to 0.566, whereas semantic LM-only yields 71.4 and 0.505. Auxiliary recovery losses are therefore not necessary for the dense degradation to occur. Adding MR and LR under anti-semantic masking partially mitigates it, yielding 66.0 mIoU and 0.521 RMSE.

The LM-only anti-semantic run also has a smaller image-sensitivity gap (0.703 versus 1.678) and lower Retrieval@1 (75.4% versus 97.6%) than semantic LM-only. Dense degradation thus occurs together with reduced image dependence even without auxiliary recovery losses. Under anti-semantic masking, caption-relevant patches are hidden, but teacher-forced text can still make many target tokens predictable. A plausible account is that textual context continues to sustain caption prediction as image-specific conditioning weakens, so caption supervision alone need not preserve visually discriminative features.

We will include these objective ablations in the revised manuscript.

W3 & Q3: Layout recovery and semantic position priors

The geometric supervision in LR refers to its distance-sensitive coordinate-regression objective. Our claim concerns its benefit to dense transfer, supported by within-backbone downstream ablations. We do not claim robustness to controlled layout transformations. Semantic location priors, such as sky typically appearing above roads, are regularities of natural scenes and can provide useful cues for dense prediction. LR aims to enrich patch representations with spatial information, not to eliminate all such priors.

From the training logs, the final coordinate L1 errors are 0.3798 for GenLIP-L/16 and 0.3431 for OpenVision 2-L/14. These are unweighted means of the per-patch sum of absolute horizontal and vertical coordinate errors, with target coordinates normalized to [0,1].

To evaluate transfer beyond segmentation and depth estimation, we compare the pretrained GenLIP-L/16 encoder, LM + MR, and full GLASS on CV-Bench using the same frozen-encoder LLaVA-NeXT-Qwen2.5-1.5B protocol. Its tasks cover object counting, 2D spatial relations, depth ordering, and relative distance. The table reports task accuracies (%) and the official 2D, 3D, and Overall summaries.

ObjectivesCount ↑Relation ↑Depth ↑Distance ↑2D ↑3D ↑Overall ↑
Pretrained60.0252.6155.8351.1656.2453.5054.87
LM + MR57.7451.5453.3350.3354.7251.8353.28
LM + MR + LR (GLASS)60.1552.1555.5051.3356.1953.4254.81
AuthorsOpenReview ↗

Response to Reviewer DcN2 (3/3)

Adding LR to LM + MR improves all four tasks by 0.61–2.41 points and raises Overall from 53.28 to 54.81, close to the pretrained 54.87. Each task is within 0.5 points of its pretrained score. This within-backbone comparison supports LR's contribution to explicit spatial-task performance alongside the dense gains in Table 4.

W4 & Q4: Original generative pathway and larger-MLLM transfer

Appendix C.4 directly evaluates the original GenLIP-L/16 captioning pathway, with all parameters fixed and no downstream adaptation. On the same 1,000 COCO image–caption pairs under teacher forcing, correct-pair NLL changes from 3.956 to 3.860 and the image-sensitivity gap from 1.609 to 1.677 nats per target token, while Retrieval@1 changes from 97.6% to 97.5%. These results support preserved image–caption correspondence sensitivity through GenLIP's native pathway.

We additionally evaluate GenLIP-L/16 with Qwen2.5-7B-Instruct under the same LLaVA-NeXT protocol as the 1.5B experiments. The vision encoder is frozen, while a two-layer MLP projector and the language model are trained on LLaVA OneVision data. The 1.5B rows are from Table 2 of our paper; the 7B pretrained row is the published reference from Table 4 of the GenLIP paper, and the GLASS 7B row contains our new measurements. Δ denotes the mean change from the corresponding pretrained encoder within each metric group, with MME-P converted to a percentage of its 2,000-point maximum.

LLM sizeEncoderChartQA ↑TextVQA ↑OCR-B ↑SEED-2 ↑MME-P ↑SQA ↑Δ MultimodalTextCaps ↑NoCaps ↑Δ Captioning
1.5BGenLIP, pretrained41.253.651.151.1125876.1--131.482.6--
1.5BGenLIP + GLASS42.454.251.050.3126476.1+0.2135.484.1+2.8
7BGenLIP, pretrained52.762.959.259.0132085.4--139.481.1--
7BGenLIP + GLASS53.462.959.059.1134285.6+0.3142.283.5+2.6

With Qwen2.5-7B, GLASS shows approximate retention of downstream multimodal performance (mean change +0.3 points), together with higher captioning scores (+2.6 CIDEr on average). OCR-B decreases by 0.2 points, TextVQA is unchanged, and the other four multimodal scores increase; both captioning scores improve. This matches the 1.5B pattern (+0.2 and +2.8): aggregate multimodal performance near the pretrained encoder, with higher downstream captioning scores. We will include this additional 7B evaluation in the revised manuscript.

W4 & Q4: Training-run variability and small multimodal differences

Our training runs use a fixed random seed of 0, and we report single-run results. Due to the computational cost, we have not repeated encoder post-training or downstream projector/LLM training with independent seeds, and therefore cannot report training-run standard deviations for the dense or multimodal scores.

In the reported runs, GLASS improves dense perception while keeping aggregate multimodal scores close to the pretrained references. The main paper reports mean multimodal changes of +0.2 points for GenLIP and +0.5 for OpenVision 2, alongside small task-level increases and decreases. We describe this pattern, also observed with the 7B LLM, as approximate retention in the evaluated runs, not a statistically significant improvement.

AuthorsOpenReview ↗

Looking forward to further feedback

Dear Reviewer DcN2:

We appreciate the time and effort you have dedicated to providing insightful review. We would like to kindly follow up on our response and hope it has addressed your concerns.

As the discussion period approaches its conclusion, we would be grateful for any further comments or suggestions. We are happy to clarify anything further during the discussion period.

Thank you again for your valuable insights, and we look forward to any updates you may have.

Best, The Authors

Reviewer_9YYoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the GLASS, a dense post-training framework for generative vision encoders that combines caption-guided semantic masking, masked feature recovery, and layout recovery. The method aims to improve spatial representations while preserving captioning and multimodal understanding capabilities. Experiments on GenLIP and OpenVision 2 demonstrate improvements in semantic segmentation and depth estimation, with largely preserved global recognition and multimodal performance.

Critical strengths

The paper addresses an important challenge in generative vision encoders: improving dense perception without sacrificing their original captioning and multimodal understanding capabilities, which is known as a trade-off. The proposed combination of semantic masking and spatial supervision is intuitive and practically relevant.

The authors evaluate two generative encoder families across multiple segmentation, depth estimation, classification, and multimodal benchmarks. The ablation studies provide useful evidence for the contributions of semantic masking, masked feature recovery, and layout recovery.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework primarily combines existing techniques, including attention-guided masking (e.g., AttMask, ECCV 2022), masked feature recovery, and spatial position recovery. Although their integration into generative vision encoders is useful, the methodological novelty beyond these established techniques could be better clarified, such as what's the inspiration of each component, and how are each component be connected, and what's the rationale between them.

  2. Insufficiently controlled comparisons and limited downstream validation.

The authors acknowledge that GLASS and the competing methods use different post-training datasets and computational budgets (e.g., GLASS uses Recap-DataComp-12M, whereas DIP and NeCo follow COCO-based recipes). These differences make it difficult to attribute the improvements solely to the proposed method.

Furthermore, the multimodal evaluation uses a relatively small LLaVA-NeXT-Qwen2.5-1.5B setup, leaving open whether the reported benefits generalize to stronger modern MLLMs. More experiments in terms of these two axises needed to conducted or illustrated.

Questions for authors
  1. Could the authors provide comparisons with the baseline under matched training data and computational budgets to better isolate the benefits of GLASS? Otherwise, it's hard to distinguish the contributions.

  2. Do such improvements extend to larger, more capable MLLMs beyond the evaluated 1.5B setup?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9YYo (1/2)

Thank you for recognizing the clear motivation and dense-perception gains. We address your comments and questions point by point below.

W1: Methodological contribution and component rationale

We agree that the rationale connecting the components should be clearer. GLASS is designed around a shared constraint: adding patch-level supervision while preserving the visual evidence needed by the retained caption objective.

Caption pretraining does not explicitly supervise the geometry of each patch (Section 1). Masked feature recovery can add local supervision, but the mask also determines what the caption objective can see. These two roles can conflict: hiding caption-attended patches yields 66.0 PASCAL mIoU, compared with 78.9 when they remain visible (Table 4), despite similar caption losses (Figure 1(b)).

GLASS therefore keeps the half most attended by the caption visible and uses the remaining locations for feature recovery (Section 3.2). This reverses the direction of AttMask, which hides highly attended regions and has no caption objective to protect. The hidden patches, which receive less caption attention, are supervised with feature-level targets from the full-image teacher at the representation used by dense heads (Section 3.3). The visible patches support both caption prediction and recovery of hidden features, but neither objective explicitly supervises patch coordinates. We therefore re-encode these visible patches without positional input and regress their normalized coordinates (Section 3.4). Unlike DropPos's discrete position classification, our L1 coordinate loss explicitly reflects spatial distance and yields better dense results than the categorical target in Table 5(a).

The component ablation in Table 4 supports these complementary roles: with LM retained, MR gives the larger segmentation gain, LR gives the larger reduction in depth error, and their combination achieves the best results on both tasks among these variants.

Our contribution is the finding that preserving caption-relevant evidence matters for dense post-training, and the coordinated use of visible and hidden patches that this finding motivates. We will revise the contribution statement and add a paragraph at the start of Section 3 to make this connection explicit.

W2 & Q1: Shared training data and training budgets

We agree that the original comparisons are recipe-level. They use the same released backbone checkpoints and downstream protocols, but DIP and NeCo follow their published COCO-based recipes, while GLASS is trained on Recap-DataComp-12M, chosen because both encoder families were pretrained on DataComp. Such comparisons show the outcome of each recipe without separating the objectives from data and budget. We have therefore run a shared-data comparison, with each method's compute reported.

The comparison uses a fixed 1M-image subset of Recap-DataComp-12M, making the shared-data experiment practical while accounting for DIP's offline pseudo-label generation. The frozen encoders are evaluated with the same dense probes and downstream Qwen2.5-1.5B VLM protocol. Metrics follow the paper's evaluation protocols. Online costs are A800 device-hours; DIP's offline cost is L40S device-hours.

MethodImages (M)Total tokens (B)Online device-h (Offline device-h)PASCAL ↑NYUv2 ↓ChartQA ↑OCR-B ↑MME-P ↑SQA ↑TextCaps ↑NoCaps ↑
Pretrained------70.80.50741.251.1125876.1131.482.6
DIP + LM1.00.2737.6 (100.8)72.00.50740.850.4125075.6127.780.0
NeCo + LM1.01.0079.8 (--)74.80.48838.248.0120167.2126.877.1
GLASS (Ours)1.00.6151.2 (--)74.80.46942.150.9125876.2132.483.0

As shown above, GLASS matches NeCo + LM on PASCAL mIoU (74.8) and achieves lower NYUv2 RMSE (0.469 versus 0.488). Compared with DIP + LM, it gains 2.8 PASCAL mIoU points and reduces NYUv2 RMSE from 0.507 to 0.469. GLASS also scores higher than both baselines on all four multimodal and both captioning benchmarks, maintaining performance close to the pretrained encoder.

GLASS uses 51.2 A800 device-hours, 35.8% fewer than NeCo + LM (79.8). DIP + LM uses 37.6 A800 device-hours online and an additional 100.8 L40S device-hours for offline pseudo-label generation; GLASS requires 13.6 more online device-hours but no offline stage.

We will include this shared-data comparison in the main-text Experiments section of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 9YYo (2/2)

W2 & Q2: Do such improvements extend to larger, more capable MLLMs beyond the evaluated 1.5B setup?

We appreciate your interest in how GLASS transfers to larger language models. We have now evaluated GenLIP-L/16 with Qwen2.5-7B-Instruct under the same LLaVA-NeXT protocol.

LLM sizeEncoderChartQA ↑TextVQA ↑OCR-B ↑SEED-2 ↑MME-P ↑SQA ↑Δ MultimodalTextCaps ↑NoCaps ↑Δ Captioning
1.5BGenLIP, pretrained41.253.651.151.1125876.1--131.482.6--
1.5BGenLIP + GLASS42.454.251.050.3126476.1+0.2135.484.1+2.8
7BGenLIP, pretrained52.762.959.259.0132085.4--139.481.1--
7BGenLIP + GLASS53.462.959.059.1134285.6+0.3142.283.5+2.6

The multimodal pattern extends to the 7B language model. Relative to the published pretrained reference, the mean multimodal change is +0.3+0.3 points and the mean captioning change is +2.6+2.6 CIDEr: TextCaps rises from 139.4 to 142.2 and NoCaps from 81.1 to 83.5, OCR-B decreases by 0.2, TextVQA is unchanged, and the other four multimodal scores increase. This matches the 1.5B pattern (+0.2+0.2 and +2.8+2.8): aggregate multimodal performance near the pretrained encoder with higher captioning scores.

Together, the shared-data comparison shows that GLASS's dense--multimodal balance persists under common data exposure, while the 7B evaluation shows that the observed multimodal and captioning pattern is not confined to the original 1.5B setting. These results support the proposed post-training design within the evaluated settings. We will include the additional 7B evaluation in the revised manuscript.

AuthorsOpenReview ↗

Looking forward to further feedback

Dear Reviewer 9YYo:

We appreciate the time and effort you have dedicated to providing insightful review. We would like to kindly follow up on our response and hope it has addressed your concerns.

As the discussion period approaches its conclusion, we would be grateful for any further comments or suggestions. We are happy to clarify anything further during the discussion period.

Thank you again for your valuable insights, and we look forward to any updates you may have.

Best, The Authors

Reviewer_YKGcOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GLASS, a dense post-training framework for generative vision encoders that improves spatial representations while preserving multimodal capabilities. GLASS combines three objectives: autoregressive captioning, masked feature recovery, and layout recovery. Its semantic masking strategy retains caption-relevant patches identified through teacher attention. The authors also identify text prior collapse, where visual representations degrade despite relatively stable captioning loss. Experiments on GenLIP and OpenVision 2 demonstrate consistent improvements in semantic segmentation and depth estimation, while largely preserving ImageNet classification and multimodal understanding performance.

Critical strengths

1. Strong dense perception improvements without sacrificing multimodal capabilities.

GLASS consistently improves segmentation and depth estimation across GenLIP and OpenVision 2 while maintaining near-pretrained ImageNet accuracy and multimodal performance. Unlike DIP and NeCo, it also improves downstream captioning. These results demonstrate a practically valuable balance between dense and semantic representations.

2. Insightful masking analysis and effective component design.

The identification of text prior collapse highlights an important limitation of caption-based training objectives. The image-caption correspondence diagnostic strengthens this observation, while the ablations demonstrate complementary benefits from masked feature recovery and layout recovery.

Critical weaknesses

1. Insufficiently controlled and potentially unfair baseline comparisons

Although GLASS achieves strong results compared with DIP and NeCo, the comparisons are not conducted under matched training conditions. Specifically, GLASS uses 12M images from Recap-DataComp-12M, whereas DIP and NeCo use approximately 4.9M and 5.9M images, respectively. The training objectives, computational budgets, and parameter-update strategies also differ substantially. These discrepancies make it difficult to attribute GLASS's superior dense–multimodal trade-off to its proposed methodology rather than differences in training data or optimization settings.

While Table 3 introduces LM-augmented DIP and NeCo variants, and Appendix C.3 evaluates full-backbone fine-tuning for DIP, these experiments do not resolve the underlying fairness concern. Comparisons under matched training data and computational budgets are necessary to convincingly establish the advantages of GLASS over existing approaches.

2. The conceptual novelty and missing critical baselines for semantic masking

The three main components of GLASS are closely related to established techniques: semantic masking resembles attention-guided masking in AttMask and SyCoCa, masked feature recovery follows the teacher-student distillation paradigm of iBOT, and layout recovery builds upon existing position reconstruction methods such as DropPos. Although their integration for generatively pretrained vision encoders is practically relevant, the paper does not clearly establish a novel mechanism beyond combining these existing approaches.

More importantly, given that prior works such as AttMask and SyCoCa have already demonstrated the effectiveness of attention-guided masking, it is crucial to establish what unique benefits caption-guided semantic masking offers over alternative importance-based masking strategies. However, the current ablations only compare semantic masking against random and anti-semantic masking, which is insufficient to isolate the contribution of caption guidance itself. Comparisons with vision-only self-attention, feature saliency, object-aware masking, or other importance-based masking strategies would be necessary to substantiate the novelty and necessity of the proposed masking design.

Additionally, the proposed text prior collapse is supported by increased cross-image feature similarity, degraded dense performance, and reduced image-caption correspondence sensitivity. While these observations are consistent with greater reliance on language priors, they do not uniquely establish that mechanism. An alternative explanation is that anti-semantic masking simply removes important visual information and leads to representation degradation without necessarily inducing a distinct form of language-prior reliance.

Questions for authors
  1. Can the authors compare GLASS with DIP + LM and NeCo + LM using matched training data and compute budgets?

  2. How does caption-attention-based masking compare with alternative importance measures, such as vision-only self-attention or feature saliency?

  3. Can the authors provide more direct evidence distinguishing increased language-prior reliance from general degradation of visual representations?

  4. Can the authors evaluate whether coordinate recovery improves explicit spatial reasoning or geometric correspondence beyond segmentation and depth estimation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YKGc (1/3)

Thank you for recognizing the dense-perception gains and the value of balancing dense and multimodal capabilities. Our responses are as follows.

W1 & Q1: Insufficiently controlled and potentially unfair baseline comparisons

The original comparisons follow each method's post-training recipe: GLASS uses Recap-DataComp-12M because both encoder families were pretrained on DataComp, while DIP and NeCo use their published COCO-based settings. These comparisons alone do not separate the effects of the method from differences in data and optimization. We have added a shared-data comparison with DIP + LM and NeCo + LM to examine whether GLASS's dense–multimodal balance persists without greater data exposure.

We compare all three methods using GenLIP-L/16 on a fixed 1M-image subset of Recap-DataComp-12M, keeping the comparison practical given DIP's offline pseudo-label generation. The frozen encoders are evaluated with the same dense probes and downstream Qwen2.5-1.5B VLM protocol. Online costs are A800 device-hours; DIP's offline cost is L40S device-hours.

MethodImages (M)Total tokens (B)Online device-h (Offline device-h)PASCAL ↑NYUv2 ↓ChartQA ↑OCR-B ↑MME-P ↑SQA ↑TextCaps ↑NoCaps ↑
Pretrained------70.80.50741.251.1125876.1131.482.6
DIP + LM1.00.2737.6 (100.8)72.00.50740.850.4125075.6127.780.0
NeCo + LM1.01.0079.8 (--)74.80.48838.248.0120167.2126.877.1
GLASS (Ours)1.00.6151.2 (--)74.80.46942.150.9125876.2132.483.0

Under shared training data, GLASS matches NeCo + LM on PASCAL mIoU (74.8) and achieves lower NYUv2 RMSE (0.469 versus 0.488). Compared with DIP + LM, it gains 2.8 PASCAL mIoU points and reduces NYUv2 RMSE from 0.507 to 0.469. GLASS also scores higher than both baselines on all four multimodal and both captioning benchmarks, with these scores remaining close to the pretrained reference. It uses 35.8% fewer online A800 device-hours than NeCo + LM.

These results show that GLASS's dense–multimodal balance persists under common data exposure. We will include this shared-data comparison in the main-text Experiments section of the revised manuscript.

W2: The conceptual novelty and missing critical baselines for semantic masking

Our contribution is to identify how masking can compromise dense representations even when caption supervision is retained, and to design the recovery objectives around preserving caption-relevant evidence. With LM, MR, and LR unchanged, switching from semantic to anti-semantic masking reduces PASCAL mIoU from 78.9 to 66.0, while caption losses remain close (Table 4 and Figure 1(b)). Retaining the caption objective therefore does not by itself protect dense features.

Unlike AttMask, which hides highly self-attended regions for recovery, GLASS keeps the caption-attended half visible to support the retained caption objective. These visible patches also provide context for recovering hidden features. MR supervises the hidden locations with full-image teacher features at the representation used by downstream dense heads.

Neither caption prediction nor feature recovery explicitly supervises patch coordinates. LR therefore re-encodes the same visible patches without positional input and regresses their normalized coordinates. Unlike DropPos's discrete position classification, our L1 coordinate loss explicitly reflects spatial distance. Compared with the categorical target, it improves PASCAL mIoU from 77.6 to 78.9 and reduces NYUv2 RMSE from 0.467 to 0.438 (Table 5(a)).

The component ablation in Table 4 supports these complementary roles: with LM retained, MR gives the larger segmentation gain, LR gives the larger reduction in depth error, and their combination achieves the best results on both tasks among these variants. We will make this design rationale explicit in the contribution statement and Section 3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YKGc (2/3)

W2 & Q2: Compare with vision-only self-attention

We have added a vision-only self-attention comparison on GenLIP-L/16 to test the value of caption guidance beyond general importance-based selection. Vision-only attention provides an image-based importance score, while caption-to-patch attention provides a caption-conditioned relevance score. Both attention-based rules retain the highest-scoring 50% of patches. All three variants use LM, MR, and LR and are trained for one epoch on Recap-DataComp-12M, following the training and evaluation protocol of Table 4 in the main paper.

Visible-patch selectionPASCAL ↑NYUv2 ↓IN Classif. ↑ChartQA ↑OCR-B ↑MME-P ↑SQA ↑
Random73.50.47282.641.651.3125576.0
Vision-only self-attention78.40.44882.641.250.0124274.2
Caption-to-patch attention (Ours)78.90.43882.942.451.0126476.1

Vision-only attention is a strong dense baseline: compared with random selection, it improves PASCAL mIoU from 73.5 to 78.4 and reduces NYUv2 RMSE from 0.472 to 0.448. Its dense scores are close to those of caption-guided selection (78.9 and 0.438), showing that both importance-based rules can provide effective context for dense recovery.

The additional benefit of caption guidance is in multimodal transfer. Compared with vision-only attention, it improves ChartQA by 1.2 points, OCR-B by 1.0, MME-P by 22, and SQA by 1.9. Vision-only attention also scores below random selection on all four reported multimodal benchmarks, with modest differences that are consistent in direction. These results support caption-conditioned selection as a way to retain the dense benefits of importance-based masking while better preserving multimodal feature utility than the evaluated vision-only rule. We will include this patch-selection comparison in the revised manuscript.

Q3: Text-prior reliance versus general visual degradation

Thank you for your question about the interpretation of text prior collapse. In the paper, “text prior collapse” denotes an observed failure pattern: visual representations deteriorate while caption loss changes comparatively little. The Introduction and Appendix C.4 present increased language-prior reliance as an interpretation consistent with these observations.

Appendix C.4 changes the image while holding the caption and its teacher-forced prefixes fixed. After anti-semantic post-training, correct-image NLL changes only slightly, from 3.956 to 4.014, while mismatched-image NLL decreases from 5.566 to 4.652. The reduced correspondence gap is thus driven mainly by captions becoming more probable under unrelated images. Image content has become a weaker constraint on caption prediction.

Anti-semantic masking removes caption-relevant visual evidence while leaving the preceding caption text available for predicting each target token. Since many tokens are predictable from that text, the caption objective can remain comparatively stable even as image-specific features deteriorate. A plausible account is that textual context continues to sustain caption prediction as image-specific conditioning weakens. Visual degradation and reliance on textual context can therefore be connected parts of the same process.

We have added matched LM-only controls to investigate whether auxiliary recovery objectives are necessary for this behavior. We train GenLIP-L/16 with semantic and anti-semantic masking using LM supervision alone, keeping the remaining settings of Table 4 unchanged. Evaluation uses the same dense probes and the native caption diagnostic on the 1,000 COCO pairs of Appendix C.4. NLL is measured in nats per target token; Δimg is mismatched minus correct NLL, and Retrieval@1 selects the correct image among six candidates.

MaskingObjectivesPASCAL ↑NYUv2 ↓Correct NLL ↓Δimg ↑Retrieval@1 (%) ↑
SemanticLM71.40.5053.8211.67897.6
Anti-semanticLM64.20.5664.1020.70375.4

Anti-semantic LM-only training produces both poorer dense representations and reduced image dependence compared with semantic LM-only training. These controls show that the failure pattern already arises under caption supervision alone. Auxiliary recovery objectives are therefore not necessary for its occurrence. We will include these controls in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YKGc (3/3)

Q4: Effect of layout recovery on explicit spatial reasoning

To assess explicit spatial reasoning beyond segmentation and depth estimation, we evaluate GenLIP-L/16 on CV-Bench using the same frozen-encoder LLaVA-NeXT-Qwen2.5-1.5B protocol as in the main paper. Its 2,638 multiple-choice questions cover object counting, 2D spatial relations, depth ordering, and relative distance. We compare the pretrained encoder, LM + MR, and full GLASS, reporting task accuracies (%) and the official 2D, 3D, and Overall summaries.

ObjectivesCount ↑Relation ↑Depth ↑Distance ↑2D ↑3D ↑Overall ↑
Pretrained60.0252.6155.8351.1656.2453.5054.87
LM + MR57.7451.5453.3350.3354.7251.8353.28
LM + MR + LR (GLASS)60.1552.1555.5051.3356.1953.4254.81

Adding LR to LM + MR improves all four task scores, with gains ranging from 0.61 to 2.41 points, and raises Overall from 53.28 to 54.81. The resulting Overall score is close to the pretrained reference of 54.87, with each task within 0.5 points of its pretrained score.

LR adds explicit patch-coordinate supervision to MR's feature-level targets. We will include this spatial-reasoning evaluation in the revised manuscript.

AuthorsOpenReview ↗

Looking forward to further feedback

Dear Reviewer YKGc:

We appreciate the time and effort you have dedicated to providing insightful review. We would like to kindly follow up on our response and hope it has addressed your concerns.

As the discussion period approaches its conclusion, we would be grateful for any further comments or suggestions. We are happy to clarify anything further during the discussion period.

Thank you again for your valuable insights, and we look forward to any updates you may have.

Best, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

To the Area Chair: Summary of Concerns, Responses, and Revision Plan

Dear Area Chair,

Thank you for handling our submission. We have posted point-by-point responses to all three reviewers but have not received follow-up, so we summarize the discussion here to assist your evaluation. All three reviewers recognized the importance of improving dense perception in generative vision encoders without sacrificing their captioning and multimodal capabilities, the dense gains across GenLIP and OpenVision 2, and the complementary contributions shown by the component ablations. Their main concerns were the controlled comparison with existing baselines, the novelty and rationale of the design, and the mechanism behind the masking results. Below we summarize each concern and our response; detailed protocols and tables are provided in the individual reviewer threads.

Summary of concerns and responses

1. Controlled comparison with DIP and NeCo (YKGc W1/Q1; 9YYo W2/Q1; DcN2 W1/Q1)

The original comparisons are recipe-level: GLASS uses Recap-DataComp-12M, while DIP and NeCo follow their published COCO-based recipes. To separate the method from data exposure, we compare DIP + LM, NeCo + LM, and GLASS on GenLIP-L/16 using a fixed 1M-image subset of Recap-DataComp-12M. All three methods see the same image–caption pairs for one pass with the same number of optimizer updates, and each method's compute is reported in device-hours. GLASS matches NeCo + LM on PASCAL mIoU (74.8), achieves the lowest NYUv2 RMSE among the three methods (0.469), and scores higher than both baselines on all four multimodal and both captioning benchmarks. It uses 35.8% fewer online device-hours than NeCo + LM; DIP + LM uses fewer online device-hours but additionally requires offline pseudo-label generation.

2. Novelty and rationale connecting the components (YKGc W2; 9YYo W1)

GLASS is designed around one constraint: adding patch-level supervision while preserving the visual evidence used by the retained caption objective. With LM, MR, and LR unchanged, hiding the caption-attended half of the patches reduces PASCAL mIoU from 78.9 to 66.0 despite similar caption losses. GLASS therefore keeps that half visible, which reverses the direction of AttMask, and supervises the hidden locations with full-image teacher features. Since neither objective supervises patch coordinates, LR re-encodes the visible patches without positional input and regresses their normalized coordinates; this L1 target outperforms the categorical, DropPos-style target (Table 5(a)). Our contribution is the finding that preserving caption-relevant evidence matters for dense post-training, together with the coordinated use of visible and hidden patches that this finding motivates.

3. Caption guidance versus other importance measures (YKGc W2/Q2)

A vision-only self-attention rule that also retains the highest-scoring 50% of patches gives dense results close to caption guidance (78.4 vs. 78.9 PASCAL mIoU; 0.448 vs. 0.438 NYUv2 RMSE). Caption guidance scores higher on all four reported multimodal benchmarks, so its additional benefit lies in preserving multimodal feature utility.

4. Masking mechanism and the role of caption supervision (YKGc Q3; DcN2 W2/Q2)

Removing LM preserves strong dense transfer (78.4 PASCAL mIoU, 0.425 NYUv2 RMSE) but lowers all four reported multimodal scores below those of the pretrained encoder, so retaining LM helps preserve multimodal performance. Under LM-only training, anti-semantic masking still degrades dense features (64.2 vs. 71.4 PASCAL mIoU) and reduces image–caption correspondence sensitivity (Retrieval@1 75.4% vs. 97.6%), showing that auxiliary recovery losses are not necessary for the failure pattern. "Text prior collapse" denotes this observed pattern. Increased reliance on textual context is a plausible account consistent with the diagnostics, and it can occur together with visual degradation as part of the same process.

5. Generative-feedback baselines (DcN2 W1)

DIVA and GenHancer use generative reconstruction feedback to improve fine-grained perception in contrastively pretrained CLIP, which differs in objective and pretraining setting from dense post-training of caption-pretrained encoders. We adapted DIVA to GenLIP-L/16 as a representative of this route; the adaptation and the corrections it required are described in the thread. The adapted DIVA leaves ImageNet accuracy and segmentation close to the pretrained encoder and increases depth error, so it does not reproduce GLASS's dense gains under the evaluated settings.

6. Layout recovery and spatial reasoning (YKGc Q4; DcN2 W3/Q3)

Our claim for LR concerns its benefit to dense transfer. We do not claim robustness to controlled layout transformations, and semantic location priors in natural scenes can serve as useful cues. On CV-Bench, under the same frozen-encoder 1.5B protocol, adding LR to LM + MR improves all four task scores and raises Overall accuracy from 53.28 to 54.81, close to the pretrained 54.87.

7. Retention and larger language models (9YYo W2/Q2; DcN2 W4/Q4)

Appendix C.4 evaluates GenLIP's original captioning pathway without downstream adaptation, where GLASS keeps image–caption correspondence sensitivity close to the pretrained encoder (Retrieval@1 97.5% vs. 97.6%). With Qwen2.5-7B-Instruct under the same LLaVA-NeXT protocol, GLASS changes the mean multimodal score by +0.3 points and the mean captioning score by +2.6 CIDEr relative to the published pretrained reference, matching the 1.5B pattern (+0.2 and +2.8).

We would be grateful if these responses could be taken into account in the discussion. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.