- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new model, TRIDENT, to address the data inefficiency of vision-language pretraining, and claims that performance gains need not come solely from scaling image-text corpora, but can instead come from extracting more complementary supervision from each pair. The framework trains a shared ViT backbone with three simultaneous objectives at different granularities: instance-level image-text contrastive learning, region-level feature distillation, and pixel-level masked image reconstruction. Pretrained on only 1B sample presentations drawn from a 283M-pair subset of Recap-DataComp-1B, TRIDENT-B/16 and TRIDENT-L/16 outperform eight competitive encoders pretrained on 4–40B samples across image-text retrieval, dense prediction, and compositional understanding.
Critical strengths
strengths:
1. Multi-granularity supervision, not merely multi-objective stacking. The paper supervises the shared ViT backbone at three distinct levels of granularity — instance, region, and pixel — with each level targeting information the others cannot provide: global cross-modal semantics, view-invariant and structural region features, and fine-grained local appearance.
2. Data efficiency is a well-motivated and practically valuable research orientation. With only 1B sample presentations from a 283M-pair subset, the model can outperform eight encoders pretrained on 4–40B samples across retrieval, dense prediction, and compositional understanding.
Critical weaknesses
Weaknesses:
1. Insufficient justification for loss balancing. Equation (11) combines three loss terms with equal weights, despite their substantially different formulations and potentially different numerical scales (e.g., sigmoid contrastive loss over pairs, cross-entropy loss over 18 view pairs, and per-patch MSE). The paper does not justify the equal weighting, report the relative magnitudes of the individual loss terms, or provide a sensitivity analysis or comparisons with alternative weighting strategies. Consequently, it remains unclear whether the proposed loss formulation is appropriately balanced or whether certain objectives dominate the optimization process.
2. Insufficient specification of temperature parameters. The paper does not report the values or initialization settings of several important parameters, including the learnable logit scale in the contrastive loss, the teacher and student temperatures in semantic distillation, and in structural distillation.
Questions for authors
-
Could the authors provide further justification for how the three loss terms are balanced to achieve good performance, either through theoretical analysis or empirical evidence, such as sensitivity studies or ablation experiments with different loss weights?
-
Could the authors provide more details on how the temperature parameters are selected for the contrastive and distillation losses? It would be helpful to include empirical evidence, such as sensitivity analyses or ablation studies, to demonstrate how these parameters affect model performance and training stability.
We thank the reviewer for highlighting the need to justify loss weighting and specify several hyperparameters. We address these concerns below.
Response to Q1: Loss balancing.
(i) Loss-weight sensitivity.
We express the training objective as
L=Lcon+λrecLrec+λdisLdis,
where Ldis=Lsem+Lsd, with λrec=λdis=1 by default. We vary each weight over {0.5,1,2} while fixing the other at 1, using ViT-B/16 with 100M image presentations and otherwise identical training and evaluation settings. As shown in the table below, moderate variations in either weight yield comparable downstream performance, with no sharp optimum observed among the tested settings. This supports equal weighting as a simple and effective default without requiring precise weight tuning.
(ii) Loss magnitudes and gradient norms.
Under the default equal-weight configuration, once training stabilizes, the contrastive, distillation, and reconstruction losses are approximately 1.0, 5.0, and 0.15, respectively, while their corresponding gradient ℓ2 norms on the shared vision backbone are approximately 3.0, 4.0, and 0.3. Thus, the larger numerical value of the distillation loss does not translate into a proportionally larger gradient: contrastive learning and distillation have comparable gradient norms. Reconstruction contributes a smaller auxiliary gradient, while its benefit is supported by the objective ablation in the manuscript.
(iii) Gradient compatibility.
Our gradient analysis further shows that the smoothed pairwise cosine similarities approach zero as training progresses, without substantial persistent negative alignment, as shown in Figure 4 of the manuscript. This indicates approximately orthogonal gradient directions and no evident sustained directional conflict on the shared vision backbone. Together, the weight-sensitivity results, gradient magnitudes, and directional analysis provide empirical support for the default equal-weight configuration.
Response to Q2: Hyperparameters and initialization.
We apologize for omitting these implementation details. We have documented their values and schedules in the revised manuscript.
For contrastive learning, we initialize the learnable logit-scale parameter and bias as α=ln(10) and β=−10, following SigLIP. We clamp α to a maximum of ln(100). After 100M sample presentations, exp(α)≈25.5 and β≈−21.3, with the learned scale well below the maximum allowed value of 100.
For both semantic and structural distillation, we follow the corresponding defaults in the public DINOv2 implementation. The student temperature is fixed at τS=0.1, while the teacher temperature increases linearly from 0.04 to 0.07 over the first 30% of training steps and remains at 0.07 thereafter. The teacher EMA momentum follows a cosine schedule from 0.992 to 1.0, and the centering momentum is 0.9, with separate centers for the two distillation branches. We adopted these settings directly and did not conduct a separate temperature sensitivity study.