- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes alignment guided complementary aggregation (ACA) for jointly adapting vision encoders such as CLIP and DINOv3. ACA introduces patch level relational supervision using normalized gram discrepancies, learned importance weights, an alignment gate, and complementary residual variance regularization. Both encoders are jointly optimized and retained for downstream prediction, while auxiliary modules are discarded during inference. Experiments on COCO and imagenet1k show improvements of 0.25 to 0.94 percentage points over concatenation. Additional ImageNet hard ablations demonstrate that the full three-term objective outperforms the tested individual and pairwise loss combinations.
Critical strengths
[S1]. Interesting relational fusion formulation. ACA extends conventional feature concatenation by explicitly modeling agreement and disagreement between patch level representations. Normalized gram matrices provide a dimension agnostic relational comparison, and the auxiliary modules introduce no additional inference computation beyond the retained encoders.
[S2]. Useful experimental and analytical investigation. The paper evaluates multiple encoder configurations and provides a systematic seven subset loss ablation, a direct summation control, and mathematical analysis of the objective. ACA consistently improves over concatenation across the four main settings, although the gains are modest.
Critical weaknesses
[W1]. Insufficiently validated alignment complementarity mechanism. Maximizing Gram residual variance does not necessarily preserve task relevant complementary information. Moreover, Appendix D shows that the auxiliary objective favors a zero alignment gate under a constant gate assumption. Although this does not establish collapse during joint training, the paper lacks gate distribution statistics or targeted interventions demonstrating that meaningful relational information is preserved. The central mechanistic interpretation therefore remains under validated.
[W2]. Limited experimental reliability and confounded comparisons. The reported improvements are relatively small (0.25 to 0.94 pp) and based on single seed experiments without uncertainty estimates. Gram and ACA also use different classifier architectures and dropout settings (Appendix E.4), preventing clean attribution of gains to the proposed objective. The ImageNet hard ablation lacks a matched task only baseline, while the checkpoint-selection protocol is unclear. Furthermore, the nonstandard single label COCO formulation and absence of dense prediction evaluations limit the conclusions about generalization and relational representation quality.
Questions for authors
[Q1]. Controlled baselines: Can the authors evaluate concatenation, Gram alignment, and ACA using identical classifier architectures, dropout, and training configurations, including a task only baseline for ImageNet hard?
[Q2]. Statistical reliability: Can the authors report multi seed results and paired confidence intervals to establish whether the relatively small improvements are consistent?
[Q3]. Gate behavior: Given the constant gate degeneracy identified in Appendix D, can the authors report gate distributions, importance weights, and gram discrepancies throughout training? What evidence demonstrates that the complementary residual captures useful information?
[Q4]. Evaluation protocol: Are checkpoints selected using the same validation splits used for reporting results? Please clarify the ImageNet hard split and justify using the first instance annotation as the COCO classification label.
[Q5]. Reproducibility: What is the exact objective used for the Direct sum baseline? Can the authors also verify the identical retrieval accuracies reported for the additional COCO and Flickr30k experiments in Table 9?
Official comments
After reviewing all reviewers' comment and authors' reply. I decided to keep my score.
I have read the paper and used AI to generate partial reviews. Frankly, I do not have sufficient expertise to assess this submission.