acceptodds
Under review as a conference paper at ICLR 2027

Efficient Dense Visual Representation Learning via Multi-Granularity Supervision

Abstract

Visual representations encode the semantic and visual content of real-world images in a feature space. They are widely used in semantic segmentation, cross-modal retrieval, and Multimodal Large Language Models (MLLMs). However, existing vision encoders, such as CLIP, are predominantly trained via image–text contrastive learning. This approach relies heavily on large-scale web-crawled alt-text as supervision, resulting in low data-efficiency during training. Moreover, the global instance-level contrastive objective limits the model's ability to capture fine-grained semantics and their corresponding relationships. In this paper, we propose TRIDENT, a data-efficient vision encoder pretrained with complementary forms of supervision: (i) instance-level image–text contrastive learning with a fine-tuned LLM as the text encoder to achieve cross-modal alignment and transfer its rich world knowledge to the vision encoder; (ii) region-level feature distillation to capture semantic consistency and spatial structure; and (iii) pixel-level reconstruction to preserve local visual details. Together, these objectives provide complementary global, regional, and pixel-level supervision, enabling richer dense visual representations and more data-efficient pretraining. Trained on only 1B samples, TRIDENT broadly outperforms eight competitive vision encoders trained on 4–40B samples across image–text retrieval, dense prediction, and downstream MLLM benchmarks. Compared with SigLIP2, TRIDENT improves I2T/T2I retrieval by 12.2/18.2 R@1 on Urban1K and 17.8/20.9 R@1 on ShareGPT4V, while achieving gains of 4.8 mIoU on ADE20K and 6.1 mIoU on PASCAL VOC for semantic segmentation. On MLLM downstream tasks, TRIDENT outperforms SigLIP2 on 10 of 11 benchmarks while matching it on MMMU, including gains of 3.4 points on VQAv2, 3.0 points on GQA, and 7.3 points on MMBench-EN. We will release the corresponding code and model checkpoints to facilitate further research in visual representation learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Encoder, Vision Representation Learning, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3u7KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new model, TRIDENT, to address the data inefficiency of vision-language pretraining, and claims that performance gains need not come solely from scaling image-text corpora, but can instead come from extracting more complementary supervision from each pair. The framework trains a shared ViT backbone with three simultaneous objectives at different granularities: instance-level image-text contrastive learning, region-level feature distillation, and pixel-level masked image reconstruction. Pretrained on only 1B sample presentations drawn from a 283M-pair subset of Recap-DataComp-1B, TRIDENT-B/16 and TRIDENT-L/16 outperform eight competitive encoders pretrained on 4–40B samples across image-text retrieval, dense prediction, and compositional understanding.

Critical strengths

strengths:

1. Multi-granularity supervision, not merely multi-objective stacking. The paper supervises the shared ViT backbone at three distinct levels of granularity — instance, region, and pixel — with each level targeting information the others cannot provide: global cross-modal semantics, view-invariant and structural region features, and fine-grained local appearance.

2. Data efficiency is a well-motivated and practically valuable research orientation. With only 1B sample presentations from a 283M-pair subset, the model can outperform eight encoders pretrained on 4–40B samples across retrieval, dense prediction, and compositional understanding.

Critical weaknesses

Weaknesses:

1. Insufficient justification for loss balancing. Equation (11) combines three loss terms with equal weights, despite their substantially different formulations and potentially different numerical scales (e.g., sigmoid contrastive loss over NN pairs, cross-entropy loss over 18 view pairs, and per-patch MSE). The paper does not justify the equal weighting, report the relative magnitudes of the individual loss terms, or provide a sensitivity analysis or comparisons with alternative weighting strategies. Consequently, it remains unclear whether the proposed loss formulation is appropriately balanced or whether certain objectives dominate the optimization process.

2. Insufficient specification of temperature parameters. The paper does not report the values or initialization settings of several important parameters, including the learnable logit scale α/β\alpha/\beta in the contrastive loss, the teacher and student temperatures τT/τS\tau_T/\tau_S in semantic distillation, and TT/TST_T/T_S in structural distillation.

Questions for authors
  1. Could the authors provide further justification for how the three loss terms are balanced to achieve good performance, either through theoretical analysis or empirical evidence, such as sensitivity studies or ablation experiments with different loss weights?

  2. Could the authors provide more details on how the temperature parameters are selected for the contrastive and distillation losses? It would be helpful to include empirical evidence, such as sensitivity analyses or ablation studies, to demonstrate how these parameters affect model performance and training stability.

1 comment

We thank the reviewer for highlighting the need to justify loss weighting and specify several hyperparameters. We address these concerns below.

Response to Q1: Loss balancing.

(i) Loss-weight sensitivity.

We express the training objective as

L=Lcon+λrecLrec+λdisLdis,\mathcal L=\mathcal L_{\mathrm{con}}+\lambda_{\mathrm{rec}}\mathcal L_{\mathrm{rec}}+\lambda_{\mathrm{dis}}\mathcal L_{\mathrm{dis}},

where Ldis=Lsem+Lsd\mathcal L_{\mathrm{dis}}=\mathcal L_{\mathrm{sem}}+\mathcal L_{\mathrm{sd}}, with λrec=λdis=1\lambda_{\mathrm{rec}}=\lambda_{\mathrm{dis}}=1 by default. We vary each weight over {0.5,1,2}\lbrace 0.5,1,2\rbrace while fixing the other at 1, using ViT-B/16 with 100M image presentations and otherwise identical training and evaluation settings. As shown in the table below, moderate variations in either weight yield comparable downstream performance, with no sharp optimum observed among the tested settings. This supports equal weighting as a simple and effective default without requiring precise weight tuning.

(ii) Loss magnitudes and gradient norms.

Under the default equal-weight configuration, once training stabilizes, the contrastive, distillation, and reconstruction losses are approximately 1.0, 5.0, and 0.15, respectively, while their corresponding gradient ℓ2\ell_2 norms on the shared vision backbone are approximately 3.0, 4.0, and 0.3. Thus, the larger numerical value of the distillation loss does not translate into a proportionally larger gradient: contrastive learning and distillation have comparable gradient norms. Reconstruction contributes a smaller auxiliary gradient, while its benefit is supported by the objective ablation in the manuscript.

(iii) Gradient compatibility.

Our gradient analysis further shows that the smoothed pairwise cosine similarities approach zero as training progresses, without substantial persistent negative alignment, as shown in Figure 4 of the manuscript. This indicates approximately orthogonal gradient directions and no evident sustained directional conflict on the shared vision backbone. Together, the weight-sensitivity results, gradient magnitudes, and directional analysis provide empirical support for the default equal-weight configuration.

λ_recλ_disUrban1K I2TUrban1K T2IADE20KVOCVALSEARO
0.5160.0065.8032.8667.4571.0869.39
1160.5066.5033.6867.1771.2369.31
2159.9066.0033.6266.7871.2170.08
10.560.1066.4034.1166.9470.9869.46
1261.1066.8033.5267.5871.4968.92

Response to Q2: Hyperparameters and initialization.

We apologize for omitting these implementation details. We have documented their values and schedules in the revised manuscript.

For contrastive learning, we initialize the learnable logit-scale parameter and bias as α=ln⁡(10)\alpha=\ln(10) and β=−10\beta=-10, following SigLIP. We clamp α\alpha to a maximum of ln⁡(100)\ln(100). After 100M sample presentations, exp⁡(α)≈25.5\exp(\alpha)\approx25.5 and β≈−21.3\beta\approx-21.3, with the learned scale well below the maximum allowed value of 100.

For both semantic and structural distillation, we follow the corresponding defaults in the public DINOv2 implementation. The student temperature is fixed at τS=0.1\tau_S=0.1, while the teacher temperature increases linearly from 0.04 to 0.07 over the first 30% of training steps and remains at 0.07 thereafter. The teacher EMA momentum follows a cosine schedule from 0.992 to 1.0, and the centering momentum is 0.9, with separate centers for the two distillation branches. We adopted these settings directly and did not conduct a separate temperature sensitivity study.

Reviewer_Ni4AOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes TRIDENT, a vision-encoder that is trained by combining three types of loss functions such that its overall training is data-efficient. The three losses encompass contrastive learning, semantic and spatial feature distillation, and pixel-level reconstruction. The model is trained on 1B samples and the resulting vision encoder is evaluated on image-text retrieval, dense prediction, and MLLM benchmarks.

Critical strengths

S1: The evaluation is broad, covering retrieval, dense tasks, and MLLM benchmarks, showing competitive or superior performance to other popular, general-purpose models.

Critical weaknesses

W1: The main motivation revolves around sample efficiency, but the paper's evaluation and narrative does not support this key goal well.

  • The paper does not make clear why sample efficiency matters for VLM training. Typically, sample efficiency is important when obtaining data is expensive. This is not the case for training VLMs, or for any of the loss functions that are being used. If the goal is to reduce training compute, the manuscript does not discuss this aspect.
  • There is no distinction between samples used for training the vision encoder, and those required for the whole pipeline. Most if not all of the compared models train both the image and the text encoder from scratch, but TRIDENT requires a pre-trained LLM that consumed trillions of tokens for its pre-training. Hence, while the sample size is true for the vision encoder, it remains an unfair comparison to other methods.

W2: Experimental comparison to state-of-the-art and inclusion of related work is severely lacking.

  • The paper compares to popular VLMs, but not variants that tackle the same problem, i.e., granular alignment of vision-language. It mentions few in Sec. 2.2 (FG-CLIP, MTV), but fails to compare to them. It also misses many other works, such as FineCLIP [A], FLAIR [B], COSMOS [C]. FLAIR in particular uses 30M samples for the complete vision-language pre-training, and outperforms TRIDENT on Urban1K, and ShareGPT4V-1K.
  • There is a whole line of work on adapting MLLMs to representation tasks, such as VLM2Vec [D], LamRA [E]. These works are much more aligned with TRIDENT's idea because they rely on a pre-trained MLLM that is further adapted to representation tasks. A comparison to these methods is essential.

W3: The technical novelty is limited.

  • TRIDENT combines mostly known learning objectives.
  • The insight that denser supervision is superior to weak, global supervision is not novel, and the resulting sample efficiency rather obvious.

[A] Jing et al., FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding, NeurIPS 2024
[B] Xiao et al., FLAIR: VLM with Fine-grained Language-informed Image Representations, CVPR 2025
[C] Kim et al., COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training, CVPR 2025
[D] Meng et al., VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents, TMLR 2026
[E] Liu et al., LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant, CVPR 2025

Questions for authors

The authors are encouraged to address the questions that arise from the weaknesses to clarify the contributions and significance of their paper. This concerns:

  • What is the importance of sample efficiency for VLMs?
  • How does TRIDENT compare to more aligned methods such as FLAIR, VLM2Vec, and LamRA?
  • What is the technical novelty of TRIDENT compared to existing works?

Minor comments:

  • Having every axis of the radar plot independently scaled makes the plot harder to comprehend as it can be slightly misleading. It is better practice to stick to a single scale across benchmarks where possible.
  • The qualitative analysis in Fig. 5 is not very insightful as the compared approaches are not well known to have spatially aligned features. A comparison to DINOv3 or more granular VLMs would be more interesting. It is also unclear why TRIDENT seems to lose high-frequency information and retains a more semantic latent space.
6 comments

We thank the reviewer for the constructive feedback. We address the efficiency accounting, comparisons with related methods, and technical contribution below.

Response to Q1 and W1-1: Sample efficiency and training cost.

Our motivation is to reduce the training compute required to learn a transferable visual encoder. We agree that sample presentations alone do not capture the full computational cost. For TRIDENT ViT-L/16, text-encoder adaptation requires approximately 1,280 H200 GPU-hours, and visual pretraining requires 2,080 H200 GPU-hours, totaling 3,360 H200 GPU-hours for these two stages.

For comparison, the reported training configuration for OpenCLIP ViT-L/14 on DataComp-1B uses 512 A100 GPUs for 81 hours, corresponding to 41,472 A100 GPU-hours. OpenAI CLIP ViT-L/14 requires 256 V100 GPUs for 12 days, corresponding to 73,728 V100 GPU-hours. Using peak-throughput normalization, these training budgets correspond to approximately 13,100 and 9,300 H200-equivalent GPU-hours for OpenCLIP and OpenAI CLIP, respectively.

Response to Q1 and W1-2: LLM knowledge and representation learning.

Reusing an LLM's world knowledge to learn a stronger visual encoder with fewer image presentations is precisely the approach we advocate in TRIDENT. However, the original Qwen3 is pretrained for language generation, not explicitly optimized to produce discriminative text embeddings for image–text alignment. It therefore does not provide a ready-to-use text embedding model for cross-modal alignment supervision.

To bridge this gap, we perform contrastive adaptation on 50M caption pairs, transforming Qwen3 into a text encoder whose embeddings can provide cross-modal alignment supervision. We then freeze the adapted text encoder and use its embeddings to guide a vision encoder trained from scratch. Our approach thus explicitly invests computation in converting pretrained language knowledge into usable embedding supervision, with the aim of reducing the data and training required to learn transferable visual representations.

Response to Q2 and W2-1: Comparisons with fine-grained VLMs.

We have expanded the comparisons to include FG-CLIP, MTV, FineCLIP, FLAIR, and COSMOS, covering visual representation evaluation and downstream MLLM performance, as shown in tables below. FLAIR and COSMOS achieve stronger retrieval results, while TRIDENT achieves the highest dense-prediction and compositional-understanding scores among these models and leads on 10 of the 11 downstream MLLM benchmarks. Both FLAIR and COSMOS train on 30M images for 32 epochs, corresponding to approximately 0.96B image presentations, comparable to TRIDENT's 1B presentations over 283M unique image–text pairs. Also, their reported multi-caption configurations also yield approximately 30M×32×8=7.68B30\mathrm{M}\times32\times8=7.68\mathrm{B} sampled text-view presentations, compared with 1B full-caption presentations for TRIDENT. Their sub-captions separately emphasize objects, attributes, and scene details, providing multiple explicit alignment targets. FLAIR further uses text-conditioned image pooling, while COSMOS incorporates cross-modality self-distillation. This emphasis on fine-grained image–text matching may help explain their retrieval advantage. TRIDENT aligns global image representations with full-caption embeddings, while its regional distillation and pixel reconstruction provide complementary visual supervision. This design emphasizes transferable visual representations, consistent with its stronger dense-prediction, compositional understanding and downstream MLLM results in our evaluations.

ModelArchData: ImageData: TextUrban1K I2TUrban1K T2ISGPT4V I2TSGPT4V T2IADE20KPASCALVALSEARO
FG-CLIPB/161.612B†≥ 3.224B79.2081.5087.1086.7037.9974.2667.8548.35
MTVB/163.2B≥ 3.2B53.3055.0072.6069.7042.2578.6863.2537.31
FineCLIPB/161.18M†≥ 1.18M61.0058.2079.7080.0039.2475.2068.0862.28
FLAIRB/160.96B7.68B82.8086.1095.3094.9040.5977.6166.6856.39
COSMOSB/160.96B7.68B82.8088.1095.0095.3039.9877.1263.6460.61
TRIDENTB/161B1B72.1077.6088.8089.3042.3678.7272.2474.95

† FG-CLIP and FineCLIP count post-training presentations, excluding pretraining of their CLIP and EVA02-CLIP initializations, respectively.

ModelArchData: ImageData: TextVQAv2GQASEED-IMME-PMME-CPOPEMMB-ENMMStarMMTMMMUCV-Bench
FG-CLIPB/161.612B†≥ 3.224B62.8454.3458.111278.68306.7979.4152.7534.3541.5740.3362.05
MTVB/163.2B≥ 3.2B67.5558.8363.591270.26313.5781.2261.8938.3044.7939.0060.61
FineCLIPB/161.18M†≥ 1.18M67.0458.4462.371238.48294.2980.0458.9338.5644.8039.8960.54
FLAIRB/160.96B7.68B66.5957.5664.181275.52307.1480.3259.8837.7944.6840.4465.16
COSMOSB/160.96B7.68B63.1854.9057.861127.31296.0780.7651.2936.6743.1740.1159.06
TRIDENTB/161B1B68.1358.9366.321280.02314.6481.6461.9440.6845.0040.5663.61

† FG-CLIP and FineCLIP count post-training presentations, excluding pretraining of their CLIP and EVA02-CLIP initializations, respectively.

Response to Q2 and W2-2: Relation to VLM2Vec and LamRA.

TRIDENT learns a visual backbone for downstream MLLMs, whereas VLM2Vec and LamRA adapt complete MLLMs into embedding models. TRIDENT uses an adapted LLM to provide image–text alignment supervision, together with region-level distillation and pixel-level reconstruction, to train a standalone vision encoder, with approximately 300M parameters for ViT-L/16. The resulting encoder produces both global and spatial patch representations, whose extraction requires only the ViT. However, VLM2Vec and LamRA instead use Qwen2-VL-2B and Qwen2-VL-7B, respectively. Producing their global retrieval embeddings requires running both the vision encoder and the language model, even for image-only inputs. This introduces billion-parameter language-model computation into image encoding, which TRIDENT's visual feature extraction avoids. The differences therefore concern both the inference architecture (a standalone ViT versus a full MLLM) and the output interface (CLS and spatial patch features versus a single global retrieval vector). The standard outputs of VLM2Vec and LamRA do not provide the spatial patch features required by our dense-prediction and downstream MLLM protocols. Consequently, they are not directly comparable as interchangeable vision-backbone baselines under these protocols.

Response to Q3 and W3: Technical contribution.

Main contribution.

In this paper, we propose TRIDENT, a data-efficient vision encoder pretrained with complementary forms of supervision: (i) instance-level image–text contrastive learning with a fine-tuned LLM as the text encoder to achieve cross-modal alignment and transfer its rich world knowledge to the vision encoder; (ii) region-level feature distillation to capture semantic consistency and spatial structure; and (iii) pixel-level reconstruction to preserve local visual details. Together, these objectives provide complementary global, regional, and pixel-level supervision, enabling richer dense visual representations and more data-efficient pretraining. Trained on only 1B samples, TRIDENT broadly outperforms eight competitive vision encoders trained on 4–40B samples across image–text retrieval, dense prediction, and downstream MLLM benchmarks. Compared with SigLIP2, TRIDENT improves I2T/T2I retrieval by 12.2/18.2 R@1 on Urban1K and 17.8/20.9 R@1 on ShareGPT4V, while achieving gains of 4.8 mIoU on ADE20K and 6.1 mIoU on PASCAL VOC for semantic segmentation. On MLLM downstream tasks, TRIDENT outperforms SigLIP2 on 10 of 11 benchmarks while matching it on MMMU, including gains of 3.4 points on VQAv2, 3.0 points on GQA, and 7.3 points on MMBench-EN. We will release the corresponding code and model checkpoints to facilitate further research in visual representation learning.

Gradient compatibility.

Figure 4 in the manuscript shows that the smoothed pairwise gradient cosine similarities on the shared vision backbone are positive early in training and gradually approach zero, with no evident sustained negative alignment. This supports the compatibility of joint optimization; the ablations separately establish its empirical benefits.

Theoretical proof.

We have added a proof to the revised appendix that directly analyzes the sigmoid contrastive, distillation cross-entropy, and reconstruction losses. Under a fixed teacher and matched prediction contexts, consider two equally weighted inputs mapped to the same representation. Such a collision forces shared predictions and incurs an excess conditional Bayes risk.

The cross-entropy entropy gap and squared-error decomposition give the respective collision penalties:

Dc=EJS⁡(Ber⁡(η1),Ber⁡(η2)),D_c=\mathbb{E}\operatorname{JS}\bigl(\operatorname{Ber}(\eta_1),\operatorname{Ber}(\eta_2)\bigr),

Dd=∑b∈{sem,sd}EJS⁡(μ1b,μ2b),D_d=\sum_{b\in\lbrace\mathrm{sem},\mathrm{sd}\rbrace}\mathbb{E}\operatorname{JS}(\mu_1^b,\mu_2^b),

Dr=14E∥m1−m2∥22.D_r=\tfrac14\mathbb{E}\lVert m_1-m_2\rVert_2^2.

where ηa\eta_a, μab\mu_a^b, and mam_a are the conditional matching probability, mean teacher distribution, and mean reconstruction target, respectively. For any objective subset A⊆{c,d,r}A\subseteq\lbrace c,d,r\rbrace,

JA=∑j∈AλjDj,J_A=\sum_{j\in A}\lambda_jD_j,

where positive λj\lambda_j absorb the loss weights and normalizations.

Since all DjD_j are nonnegative, the sets of unpenalized collisions, denoted by E(A)\mathcal E(A), satisfy

E({c,d,r})=E({c})∩E({d})∩E({r})⊆E(A).\mathcal E(\lbrace c,d,r\rbrace)=\mathcal E(\lbrace c\rbrace)\cap\mathcal E(\lbrace d\rbrace)\cap\mathcal E(\lbrace r\rbrace)\subseteq\mathcal E(A).

An omitted objective makes this inclusion strict whenever it distinguishes inputs that the retained objectives cannot. If each objective supplies distinctions absent from the other two, joint supervision imposes strictly stronger representation constraints than any single objective or pair. This identifies target complementarity as the condition under which combining the three losses rules out additional information-losing representations.

Reviewer_RsrsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes TRIDENT, combining instance-level contrastive learning, region-level feature distillation, and pixel-level reconstruction for data-efficient visual pretraining. Experiments across retrieval, dense prediction, and MLLM benchmarks demonstrate substantial improvements over several existing vision encoders.

Critical strengths

The method achieves substantial improvements across multiple visual and multimodal benchmarks. The evaluation covers diverse downstream tasks, with ablations supporting the usefulness of combining multiple supervision objectives.

Critical weaknesses

1.The framework mainly combines existing contrastive learning, self-distillation, and reconstruction objectives. The contribution is relatively engineering-oriented, and the ablations do not sufficiently establish its advantages over existing multi-objective pretraining strategies.

2.TRIDENT uses a fine-tuned Qwen3-8B text encoder, while the baselines use different text encoders. This makes it difficult to attribute the large improvements specifically to multi-granularity supervision.

3.The claimed data efficiency is based on sample presentations, without accounting for the additional computational costs of multi-view distillation, reconstruction, and LLM text-encoder training.

Questions for authors

1.Can the authors provide stronger ablations or comparisons with existing multi-objective pretraining methods to establish the specific contribution of the proposed framework?

2.How much of the improvement comes from the Qwen3-8B text encoder? Can comparisons be conducted with matched text encoders and training data?

3.What are the total GPU-hours or FLOPs, including text-encoder training, and how do they compare under matched computational budgets?

3 comments

We thank the reviewer for highlighting the need to clarify the contribution of multi-granularity supervision, disentangle the role of the text encoder, and account for the full computational cost.

Response to Q1: Ablations, comparisons, and theoretical support.

(i) Complete objective ablations.

Figure 3 of the manuscript evaluates all seven nonempty combinations of the three objectives. We pretrain ViT-B/16 for 100M image presentations, varying only the enabled objectives while keeping the remaining training and evaluation settings fixed. Instance-level contrastive learning provides cross-modal alignment. Adding region-level distillation improves Urban1K I2T/T2I R@1 from 50.40/54.70 to 58.80/65.30, and ADE20K/PASCAL mIoU from 25.35/54.89 to 29.94/64.30, supporting its contribution to semantic and spatial representations. Adding pixel-level reconstruction further improves ADE20K/PASCAL by 3.74/2.87 mIoU and MMStar/MMMU by 2.18/2.22 points, demonstrating the additional benefit of preserving local visual details. The full objective achieves the best results on all eight visual metrics and 10 of the 11 downstream MLLM metrics. These controlled comparisons demonstrate the complementary contributions of global, regional, and pixel-level supervision.

(ii) Comparison with existing multi-objective methods.

In the below table, we compares TRIDENT with EVA02, SigLIP2, FG-CLIP, MTV, and AIMv2, covering both joint and staged multi-objective pretraining. It explicitly lists their loss combinations, architectures, and image presentations alongside results on retrieval, dense prediction, and compositional understanding. Within the base-model block, TRIDENT achieves the highest scores on 6 of the 8 metrics, while FG-CLIP leads on Urban1K retrieval. Within the large-model block, TRIDENT achieves the highest scores on all 8 visual metrics. These comparisons demonstrate TRIDENT's broad transfer performance alongside existing multi-objective approaches, complementing the controlled ablations above.

ModelArchDataLoss combinationUrban1K I2TUrban1K T2ISGPT4V I2TSGPT4V T2IADE20KPASCALVALSEARO
EVA02B/168B§Masked feature pretraining → contrastive67.1060.9081.1075.9037.4672.3166.9651.30
SigLIP2B/1640BSigmoid alignment + captioning/grounding + self-distillation + masked feature prediction42.4043.7058.7059.3036.5371.8568.2338.14
FG-CLIPB/161.6B†Global contrastive + regional contrastive + hard-negative loss79.2081.5087.1086.7037.9974.2667.8548.35
MTVB/163.2B‡Contrastive + self-distillation + masked feature prediction + grounding + depth + KoLeo53.3055.0072.6069.7042.2578.6863.2537.31
TRIDENTB/161BContrastive + semantic/structural distillation + pixel reconstruction72.1077.6088.8089.3042.3678.7272.2474.95
AIMv2L/1412BAutoregressive image-patch prediction + text prediction--------34.5664.24----
EVA02L/144B§Masked feature pretraining → contrastive73.5068.8081.3077.4043.0874.4468.7453.24
SigLIP2L/1640BSigmoid alignment + captioning/grounding + self-distillation + masked feature prediction63.9061.3072.6070.6041.0774.9668.4341.31
TRIDENTL/161BContrastive + semantic/structural distillation + pixel reconstruction76.1079.5090.4091.5045.8881.0871.4063.78

§ EVA02 counts the contrastive stage; masked feature pretraining is an additional earlier stage. The arrow denotes sequential training. † FG-CLIP uses pretrained CLIP initialization; 1.6B counts subsequent training presentations, excluding initialization pretraining. ‡ MTV: 100M × 32 epochs. SigLIP2 counts its 40B main-stage examples.

(iii) Gradient interaction analysis.

Figure 4 of the manuscript shows the pairwise cosine similarities between the gradients of the three objectives with respect to the shared vision backbone. The smoothed pairwise cosine similarities are positive early in training and gradually approach zero, indicating increasingly near-orthogonal directions without persistent negative alignment. This analysis supports the optimization compatibility of the three objectives, while the ablations establish the empirical benefits of joint training.

(iv) Theoretical proof.

We have added a proof to the revised appendix that directly analyzes the sigmoid contrastive, distillation cross-entropy, and reconstruction losses. Under a fixed teacher and matched prediction contexts, consider two equally weighted inputs mapped to the same representation. Such a collision forces shared predictions and incurs an excess conditional Bayes risk.

The cross-entropy entropy gap and squared-error decomposition give the respective collision penalties:

Dc=EJS⁡(Ber⁡(η1),Ber⁡(η2)),D_c=\mathbb{E}\operatorname{JS}\bigl(\operatorname{Ber}(\eta_1),\operatorname{Ber}(\eta_2)\bigr),

Dd=∑b∈{sem,sd}EJS⁡(μ1b,μ2b),D_d=\sum_{b\in\lbrace\mathrm{sem},\mathrm{sd}\rbrace}\mathbb{E}\operatorname{JS}(\mu_1^b,\mu_2^b),

Dr=14E∥m1−m2∥22.D_r=\tfrac14\mathbb{E}\lVert m_1-m_2\rVert_2^2.

where ηa\eta_a, μab\mu_a^b, and mam_a are the conditional matching probability, mean teacher distribution, and mean reconstruction target, respectively. For any objective subset A⊆{c,d,r}A\subseteq\lbrace c,d,r\rbrace,

JA=∑j∈AλjDj,J_A=\sum_{j\in A}\lambda_jD_j,

where positive λj\lambda_j absorb the loss weights and normalizations.

Since all DjD_j are nonnegative, the sets of unpenalized collisions, denoted by E(A)\mathcal E(A), satisfy

E({c,d,r})=E({c})∩E({d})∩E({r})⊆E(A).\mathcal E(\lbrace c,d,r\rbrace)=\mathcal E(\lbrace c\rbrace)\cap\mathcal E(\lbrace d\rbrace)\cap\mathcal E(\lbrace r\rbrace)\subseteq\mathcal E(A).

An omitted objective makes this inclusion strict whenever it distinguishes inputs that the retained objectives cannot. If each objective supplies distinctions absent from the other two, joint supervision imposes strictly stronger representation constraints than any single objective or pair. This identifies target complementarity as the condition under which combining the three losses rules out additional information-losing representations.

Response to Q2: Contribution of the text encoder.

To assess whether TRIDENT's gains can be explained by the text encoder alone, the table below includes OpenCLIP and EVA02 variants using the same adapted Qwen3-8B text encoder as TRIDENT. These variants and TRIDENT are trained on the same dataset for 1B image presentations, using base-sized visual encoders with 16×1616\times16 patches. With the same adapted Qwen3-8B text encoder, TRIDENT outperforms both baselines across all eight metrics, improving ADE20K/PASCAL by 5.74/6.46 mIoU over OpenCLIP and 5.45/7.35 mIoU over EVA02. Thus, using the same LLM text encoder and training data does not by itself reproduce TRIDENT's performance.

Figure 3 of the manuscript further isolates the contribution of multi-granularity supervision by keeping the fine-tuned Qwen3-8B text encoder, captions, and remaining training settings fixed. Under this control, the full objective improves ADE20K/PASCAL by 8.33/12.28 mIoU over contrastive-only training. Together, these results indicate that the joint three-loss design is the main driver of the observed improvements, demonstrating the benefits of combining global alignment, regional distillation, and pixel-level reconstruction.

MethodImage encoderText encoderDataUrban1K I2TUrban1K T2ISGPT4V I2TSGPT4V T2IADE20KPASCALVALSEARO
OpenCLIPViT-B/16CLIP Transformer12.8B72.1067.5080.0078.1036.5573.0064.5839.54
OpenCLIPViT-B/16fine-tuned Qwen3-8B1B71.9069.3081.2079.3036.6272.2665.7440.76
EVA02EVA02-B/16CLIP Transformer8B67.1060.9081.1075.9037.4672.3166.9651.30
EVA02EVA02-B/16fine-tuned Qwen3-8B1B68.5062.8083.0076.0036.9171.3767.8952.28
TRIDENTViT-B/16fine-tuned Qwen3-8B1B72.1077.6088.8089.3042.3678.7272.2474.95

Response to Q3: Training cost and efficiency.

We agree that sample presentations alone do not capture the full computational cost. For TRIDENT ViT-L/16, text-encoder adaptation requires approximately 1,280 H200 GPU-hours, and visual pretraining requires 2,080 H200 GPU-hours, totaling 3,360 H200 GPU-hours.

For comparison, the reported training configuration for OpenCLIP ViT-L/14 on DataComp-1B uses 512 A100 GPUs for 81 hours, corresponding to 41,472 A100 GPU-hours. OpenAI CLIP ViT-L/14 requires 256 V100 GPUs for 12 days, corresponding to 73,728 V100 GPU-hours. Using peak-throughput normalization, these training budgets correspond to approximately 13,100 and 9,300 H200-equivalent GPU-hours for OpenCLIP and OpenAI CLIP, respectively.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviewers feedback and authors response

We thank all reviewers for their constructive feedback, which has helped us clarify TRIDENT’s contribution and strengthen its empirical and theoretical support.

Strengths: The reviews highlighted TRIDENT’s strong performance and broad evaluation across retrieval, dense prediction, and downstream MLLM benchmarks. Positive feedback also recognized the complementarity of instance-, region-, and pixel-level supervision and the practical value of data-efficient visual pretraining.

In our rebuttal responses, we provided matched-text-encoder comparisons showing that TRIDENT’s gains extend beyond the adapted Qwen3-8B encoder, together with loss-weight sensitivity results addressing optimization concerns.

The revised manuscript incorporates five additional baselines into Tables 1 and 2. Within the ViT-B block, TRIDENT ranks first on all dense prediction and compositional understanding metrics and 8 of 11 downstream MLLM metrics across the full baseline set, including 10 of 11 when compared only with the five added baselines. FLAIR and COSMOS remain stronger on retrieval. We also clarify image and text training presentations. Gradient analyses indicate no persistent conflict among the objectives, while updated qualitative comparisons illustrate coherent semantic regions alongside discernible local detail.

The revised paper includes three new appendix sections:

  • Appendix A: Additional Implementation Details specifies text-encoder adaptation, contrastive scale and bias, distillation temperatures, and teacher updates.
  • Appendix B: Training Cost reports 3,360 H200 GPU-hours for text-encoder adaptation and visual pretraining of TRIDENT ViT-L/16, excluding upstream language-model pretraining.
  • Appendix C: Why Joint Supervision Imposes Stronger Representation Constraints provides a loss-specific analysis showing that joint supervision rules out additional representation collisions under explicit complementarity assumptions.

We thank the reviewers again for their valuable feedback and look forward to further discussion to improve the quality of our work on TRIDENT.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.