acceptodds
Under review as a conference paper at ICLR 2027

ACA: COUPLING RELATIONAL ALIGNMENT AND COMPLEMENTARY REGULARIZATION FOR VISION ENCODER FUSION

Abstract

Combining multiple vision foundation models has become an important strategy for improving visual representations. A common approach concatenates their [CLS] tokens and trains a task-specific predictor, but does not explicitly constrain patch-level relations. Inspired by DINOv3’s Gram anchoring, which preserves patch relations during self-supervised training, we study relational alignment across encoders. Uniform Gram matching penalizes every pairwise discrepancy, whereas fusion calls for aligning selected relations while retaining differences between representations. We introduce Alignment-guided Complementary Aggregation (ACA), which couples selective relational alignment with regularization of the complementary residual. Global features provide importance weights, while projected patch features determine a shared gate for weighted alignment, a mean-gate penalty, and a variance reward on the weighted complementary residual. Both encoders are jointly trained and retained for prediction. Across four classification settings on COCO and ImageNet-1K, ACA improves over matched-pair concatenation by 0.25–0.94 percentage points and outperforms uniform Gram alignment in three. On ImageNet-Hard, the full objective reaches 69.98% and exceeds all six one- and two-term variants. ACA also exceeds direct summation of the importance, residual, and gate signals by 1.01 points on ImageNet-Hard and 0.67 points on COCO. These results support coupling relational alignment with complementary residual regularization for encoder fusion.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision foundation models, multi-encoder fusion, representation learning, Gram matrix, loss design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CUVJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes alignment guided complementary aggregation (ACA) for jointly adapting vision encoders such as CLIP and DINOv3. ACA introduces patch level relational supervision using normalized gram discrepancies, learned importance weights, an alignment gate, and complementary residual variance regularization. Both encoders are jointly optimized and retained for downstream prediction, while auxiliary modules are discarded during inference. Experiments on COCO and imagenet1k show improvements of 0.25 to 0.94 percentage points over concatenation. Additional ImageNet hard ablations demonstrate that the full three-term objective outperforms the tested individual and pairwise loss combinations.

Critical strengths

[S1]. Interesting relational fusion formulation. ACA extends conventional feature concatenation by explicitly modeling agreement and disagreement between patch level representations. Normalized gram matrices provide a dimension agnostic relational comparison, and the auxiliary modules introduce no additional inference computation beyond the retained encoders.

[S2]. Useful experimental and analytical investigation. The paper evaluates multiple encoder configurations and provides a systematic seven subset loss ablation, a direct summation control, and mathematical analysis of the objective. ACA consistently improves over concatenation across the four main settings, although the gains are modest.

Critical weaknesses

[W1]. Insufficiently validated alignment complementarity mechanism. Maximizing Gram residual variance does not necessarily preserve task relevant complementary information. Moreover, Appendix D shows that the auxiliary objective favors a zero alignment gate under a constant gate assumption. Although this does not establish collapse during joint training, the paper lacks gate distribution statistics or targeted interventions demonstrating that meaningful relational information is preserved. The central mechanistic interpretation therefore remains under validated.

[W2]. Limited experimental reliability and confounded comparisons. The reported improvements are relatively small (0.25 to 0.94 pp) and based on single seed experiments without uncertainty estimates. Gram and ACA also use different classifier architectures and dropout settings (Appendix E.4), preventing clean attribution of gains to the proposed objective. The ImageNet hard ablation lacks a matched task only baseline, while the checkpoint-selection protocol is unclear. Furthermore, the nonstandard single label COCO formulation and absence of dense prediction evaluations limit the conclusions about generalization and relational representation quality.

Questions for authors

[Q1]. Controlled baselines: Can the authors evaluate concatenation, Gram alignment, and ACA using identical classifier architectures, dropout, and training configurations, including a task only baseline for ImageNet hard?

[Q2]. Statistical reliability: Can the authors report multi seed results and paired confidence intervals to establish whether the relatively small improvements are consistent?

[Q3]. Gate behavior: Given the constant gate degeneracy identified in Appendix D, can the authors report gate distributions, importance weights, and gram discrepancies throughout training? What evidence demonstrates that the complementary residual captures useful information?

[Q4]. Evaluation protocol: Are checkpoints selected using the same validation splits used for reporting results? Please clarify the ImageNet hard split and justify using the first instance annotation as the COCO classification label.

[Q5]. Reproducibility: What is the exact objective used for the Direct sum baseline? Can the authors also verify the identical retrieval accuracies reported for the additional COCO and Flickr30k experiments in Table 9?

Reviewer_xqM1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the joint adaptation of complementary vision encoders, aiming to align useful patch-level relations while retaining differences between their representations. It proposes ACA, which uses shared importance weights and gates to couple selective relational alignment with a variance-based regularizer on the remaining differences. Experiments on COCO and ImageNet-1K show improvements over feature concatenation, while ablations support the combined use of the proposed loss components.

Critical strengths
  1. The paper addresses a useful and well-defined problem: aggregating complementary vision foundation models while balancing relational agreement and representational diversity during joint adaptation.

  2. The proposed methodology appears novel within the studied setting. Its coupling of selective patch-level alignment and complementary residual regularization through shared importance weights and gates offers a distinctive approach to encoder fusion.

Critical weaknesses
  1. This paper already has strong conceptual precedents, for example, CaFo [1] also studies combine complementary pretrained representations for better classification, the only diff is CaFo's problem setting is few-shot classification, but can be conceptually generalize to full adaption.

  2. The main experiments did not adapt proper variable control, as baseline and ACA used different classifier widths and dropout rates.

  3. The improvements are marginal comparing to baseline, and only one random seed run is conducted, posing questions to whether ACA is significantly better than baselines. Moreover, ACA also slightly underperforms plain Gram on ImageNet-1K with large encoders.

  4. The fixed-architecture ImageNet-Hard study lacks a task-only baseline. It also lacks fixed-zero, constant, and random-gate comparisons. Consequently, it does not establish whether learning which relations to align is better than simpler regularization.

  5. The motivation emphasises local patch structure and complementary information, but the principal evidence is classification accuracy; there is no direct validation through dense prediction or complementary predictive capabilities. Moreover, the C-RADIOv4/SAM3 extensions test Gram alignment alone, so they do not demonstrate that the complete ACA method generalizes across encoder families.

Suggestion:

The scope of this paper remains on studying classification accuracy of visual backbone, however, a more interesting and impactful direction would be studying how this approach and help enhance the visual feature aggregation for training VLM/VLA, the Cambrian paper has investigated this, but did not propose a specific methodology. Potentially, more room for contributions can be found there.

Questions for authors

please refer to weaknesses

1 comment
Reviewer_k6DvOpenReview ↗

Official comments

After reviewing all reviewers' comment and authors' reply. I decided to keep my score.

I have read the paper and used AI to generate partial reviews. Frankly, I do not have sufficient expertise to assess this submission.

Reviewer_k6DvOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ACA jointly fine-tunes two vision encoders and keeps both at inference. The classifier is an MLP on the concatenated class tokens and the two mean-pooled patch tokens (Eq. 10). The auxiliary loss compares within-encoder patch Gram matrices. Patches are L2-normalized, Gk=F^kF^k⊤G_k=\hat F_k\hat F_k^\top, and X=GA−GBX=G_A-G_B. An MLP on the two class tokens produces a symmetric importance matrix W=σ(vv⊤)W=\sigma(vv^\top). Projected patch features produce a gate mm from their cross-encoder similarity, with a learned temperature and bias. Alignment minimizes the mean of W⊙m⊙X2W\odot m\odot X^2. The complementary residual q=W⊙(1−m)⊙Xq=W\odot(1-m)\odot X is rewarded for high variance, and the mean of mm is penalized (λg=0.01\lambda_g=0.01, λi=0.1\lambda_i=0.1). The importance and gate heads are used only in training.

On four matched CLIP–DINOv3 settings at 224 pixels (Table 1a), ACA minus concatenation is +0.94 and +0.25 on COCO (76.98 and 77.85) and +0.73 and +0.63 on ImageNet-1K (85.37 and 87.99). ACA is above uniform Gram on three of those four rows. On ImageNet-1K L/L, Gram is higher (88.08 vs 87.99). With the architecture fixed, on a class-stratified within-dataset split of ImageNet-Hard (DINOv3-S+ and CLIP-B), the three-term loss reaches 69.98 (Table 3), which is 0.54 above the best single term (alignment, 69.44) and 0.66 above the best two-term subset (alignment + variance, 69.32). C-RADIOv4 and SAM3 are evaluated with Gram alignment only, at +0.09 and +0.37 over concatenation on COCO (Table 1b).

Critical strengths
  • The premise is measured. For CLIP ViT-B/16 and DINOv3-B, 88.63% of unordered off-diagonal patch pairs on ImageNet-Hard and 91.73% on COCO val2017 have squared Gram discrepancy rij<0.1r_{ij}<0.1 (Table 4, Figure 2). Most relations already agree, so a uniform penalty is a blunt instrument, and a gate is a direct response to that histogram.
  • Table 3 holds the encoder pair, the heads, and the coefficients of active terms fixed. Every two-term subset falls below the best single term, and the full objective is best. That interaction is a clearer argument than the headline accuracy.
  • The paper reports the loss to uniform Gram on ImageNet-1K L/L (Table 1a: 88.08 vs 87.99) and states in Section 4.1 that the Gram and ACA runs use different prediction heads. Section 5 states the dual-encoder inference cost and limits the full objective to classification on corresponding patch grids.
  • The direct-sum control is the right negative. Adding the importance, residual, and gate signals without the coupled reductions scores 68.97 on ImageNet-Hard and 76.31 on COCO B/B (Table 5), which is 1.01 and 0.67 points below full ACA (69.98 and 76.98).
Critical weaknesses
  • The improvement over concatenation is too small for the claim. Table 1a gives ACA − concatenation as +0.94, +0.25, +0.73, and +0.63. On COCO, concatenation over the stronger single encoder is already +2.40 (76.04 vs 73.64) and +1.89 (77.60 vs 75.71); ACA then adds 0.94 and 0.25. On ImageNet-1K the ACA step is larger than the concatenation step (+0.73 vs +0.24 at B/B; +0.63 vs +0.44 at L/L), but uniform Gram reaches 88.08 there and ACA 87.99. That reversal is 0.09 points, the same size as the C-RADIOv4 increment in Table 1b. The experiments do not produce a condition for when the gate and the variance term are worth using.
  • Appendix E.3 states that each configuration uses one training seed. Table 11 sets seed 42 for the main runs and seed 0 for the ImageNet-Hard subsets. The COCO L/L gap is 0.25 (Table 1a), the C-RADIOv4 gap is 0.09 (Table 1b), and the ImageNet-1K L/L reversal is 0.09. A single run does not establish differences of that size.
  • The Gram-versus-ACA gaps in Table 1a compare full training configurations. Section 4.1 says the implementations use different prediction heads. Appendix E.4 specifies the B/B heads: Gram uses 3072→3072→C3072\rightarrow3072\rightarrow C with dropout 0.1, ACA uses 3072→2048→C3072\rightarrow2048\rightarrow C with dropout 0.2, and the accuracy difference includes that classifier change. The fixed-head evidence is Table 3, on the ImageNet-Hard split.
  • Section 4.1 turns COCO into 80-way classification with one label per image, “selected from the first instance annotation encountered by the loader.” Annotation order picks the class. The COCO accuracies in Table 1, including 76.98 and 77.85, are computed under that rule. Appendix F.2 uses the same one-label task for the C-RADIOv4 COCO number.
  • ImageNet-Hard uses a within-dataset, class-stratified split (Section 4.1). The paper does not give the split ratio. The 69.98 result is the only fixed-architecture measurement of the three-term interaction, and it is not a score under the public ImageNet-Hard protocol.
  • Table 2 puts a two-encoder, 20-epoch ACA model at 87.99 beside FT-CLIP-L at 88.0 and BEiT-3-L at 87.6. Table 14 shows the recipes differ: FT-CLIP uses 50/30 epochs, batch 2048, layer-wise decay, and EMA; BEiT-3 uses 50 epochs; ACA uses 20 epochs, microbatch 16, and encoder learning rate 10−610^{-6}, with both encoders retained. Section 4.3 correctly calls this a comparison of complete configurations. It does not isolate the relational loss, and matching a one-encoder recipe while paying for two Large encoders is a weak case for the objective.
  • The extensions do not add the missing fact about agreement. Table 1b applies only Gram alignment, at +0.09 (C-RADIOv4 + DINOv3-L) and +0.37 (SAM3 + DINOv3-L), so the gate and the variance term are untested outside CLIP–DINOv3 classification. Section 4.5 changes fusion and Gram together and says these runs do not isolate the Gram term. In Table 9, primary COCO retrieval moves by −0.02 / +0.06 and primary Flickr30k drops by 0.66 / 0.23. The additional COCO and Flickr30k runs each gain 0.49 / 0.37, with total-loss changes +0.0335 and −0.0206, and those two blocks list the same retrieval accuracies (Native 93.89 / 93.26, Fusion+Gram 94.38 / 93.63). DINOv3’s COCO self-supervised total loss moves by −0.02 at B and −0.03 at L (Table 10). Figure 3 shows four qualitative gate panels. The Gram residual is the construction already used for DINOv3 anchoring. I still do not have a characterization of which patch relations should agree.
Questions for authors
  1. Appendix E.3 and Table 11 use one seed (42 on the main runs, 0 on ImageNet-Hard). Please report mean and standard deviation over at least three seeds for every cell in Table 1a. Do the +0.25 COCO L/L gap and the ImageNet-1K L/L order (Gram 88.08, ACA 87.99) remain?
  2. Section 4.1 takes each COCO label from the first instance annotation the loader encounters. Does the ranking of concatenation, Gram, and ACA in Table 1 hold if the label is the largest object, or if the loss is multi-label?
  3. Uniform Gram beats ACA on ImageNet-1K L/L, and Section 4.1 says those runs use different heads. With a matched classifier, on which encoder pair and dataset do the gate and the variance term beat uniform Gram, and which property of the pair predicts that?
  4. Table 3 is the only fixed-head study of the three-term interaction, on a private ImageNet-Hard split with seed 0. Does that ordering (full ACA above every subset) hold on ImageNet-1K with the classifier held fixed? In Table 9, why do the additional COCO and Flickr30k blocks list the same I2T/T2I numbers while the primary Flickr30k run moves the other way?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.