acceptodds
Under review as a conference paper at ICLR 2027

The Coupling Form of Positional Encoding Determines Multi-Teacher Generalization in Feature Upsampling

Abstract

Cross-attention feature upsamplers increase the spatial resolution of vision foundation model (VFM) features. Recent upsamplers such as NAF build queries and keys from two sources: image content encoded by a lightweight image encoder, and pixel positions supplied by a rotary position embedding (RoPE). Training such an upsampler with multiple VFMs as teachers is a natural way to improve its transfer to unseen VFMs, yet it has been reported to yield no clear gains, and the cause is not understood. We show that the cause lies in how position is coupled with content. RoPE rotates queries and keys, so position multiplies content. Different teachers require different shapes of the spatial kernel, but under RoPE only the shared image encoder can produce this difference, so the gradients of the teachers conflict within the first 100 training steps. This conflict is resolved not by optimization that modifies the gradients, but by replacing RoPE with a relative position bias. This bias adds position to the attention score as a separate term and learns the kernel shape instead of the image encoder. By building every scheme on NAF and swapping only the positional encoding, on Cityscapes semantic segmentation, adding a second teacher VFM causes a 0.83 mIoU drop for RoPE compared to its best single-teacher baseline, whereas the schemes that add position to the attention score gain up to 0.57 mIoU. An upsampler that learns from multiple VFMs should add position to the attention score instead of multiplying it with content.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
feature upsampling, positional encoding, multi-teacher learning, generalization, dense prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_G9DCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how positional encoding affects multi-teacher training of feature upsamplers. Using a fixed NAF architecture, it compares fixed and learned-frequency RoPE with scalar and vector additive positional terms and a no-position baseline. Models are trained with DINOv3, SigLIP2, or both, and evaluated on seven VFMs across Cityscapes, PASCAL VOC, and DAVIS. Additive variants yield positive multi-teacher gains on Cityscapes and reduce degradation on DAVIS. Early gradient measurements, learned positional biases, and a PCGrad experiment support the authors’ proposed explanation that positional coupling influences interference between teachers.

Critical strengths
  1. The controlled positional-encoding comparison identifies a useful architectural choice. Scalar and Vector achieve Cityscapes gains of +0.51 and +0.61 mIoU on the five unseen VFMs, whereas fixed RoPE loses 0.86 mIoU (Table S4). Scalar also improves absolute multi-teacher performance from 58.57 to 60.98 mIoU while adding only 1,156 parameters. This provides a practical contribution beyond improving reconstruction of the training teachers.

  2. The gradient analysis offers useful diagnostic evidence beyond downstream scores. Early conflict is concentrated in RoPE’s 3×33\times3 encoder layers: mean cosine −0.29 with conflict on 76% of steps, compared with +0.54 and 0% for the additive schemes (Table 3). The separate examination of late-training conflict is particularly helpful, since negative gradient cosine also occurs in models with positive downstream transfer.

Critical weaknesses
  1. The proposed causal explanation is stronger than the evidence establishes: Scalar separates the attention-score terms, but its loss gradients still depend on the attention distribution; Vector explicitly retains query-dependent positional gradients (Eq. 5; Appendix A). These observations do not invalidate the positional swap, but leave its precise mechanism unresolved. Moreover, Appendix F acknowledges that global PCGrad can miss the identified conflicts because aligned gradients mask them. Its limited recovery therefore does not establish that optimization cannot address the relevant conflict. A targeted intervention would make this central explanatory claim more convincing.

  2. The reliability and scope of the positive multi-teacher benefit remain uncertain: All experiments use seed 0 and one teacher pair. The +0.48/+0.57 Cityscapes gains lack repeat-run validation; evaluations on several VFMs share the same trained upsampler. On DAVIS, both additive variants still have negative multi-teacher gains, and VOC slightly favors RoPE (Tables 1 and S6). These results support a task-dependent finding within NAF. The broader framing also needs to accommodate AnyUp’s reported improvement in multi-backbone generalization under its different architecture (AnyUp, Table 7).

Questions for authors
  1. Does applying PCGrad separately to the parameter group change RoPE’s multi-teacher transfer? This would directly address the conflict location identified in Table 3 and the masking limitation acknowledged in Appendix F.

  2. How stable are the Cityscapes transfer gains and early-gradient ordering across training seeds? A focused repeat of the fixed-RoPE and Scalar single-versus-multi comparison would help establish whether the reported sign reversal is reproducible.

4 comments
AuthorsOpenReview ↗

Response to Reviewer G9DC (1/2), W1 and Q1

Thank you for the careful review and the specific questions. All section and table numbers in our responses refer to the revised paper.

W1. The causal explanation is stronger than the evidence, and a targeted intervention is missing

We ran three targeted interventions, and none of them changed the sign of the gain. The first applies PCGrad using only the gradients of the 3×3 layers, and its result is given in our response to Q1. The second initializes Scalar from RoPE (Scalar (RoPE init.), Table S16), and the gain remains positive. The third keeps the rotation of RoPE (NAF) and adds the same position bias as Scalar (RoPE (NAF) + position bias). The table below shows the gain on Cityscapes for the four combinations of rotation and position bias (Table S17).

SchemeRotationPosition biasGain (mIoU)
None––+0.87
Scalar–✓+0.48
RoPE (NAF)✓–−0.83
RoPE (NAF) + position bias✓✓−1.09

As the table shows, the gain is negative for the two schemes with rotation and positive for the two schemes without it. From the second and third interventions, we conclude that the cause of negative transfer is the coupling form itself. For the argument leading to this conclusion, please see "Separating the coupling form from other factors" in the general response and the "Discussion" paragraph of Appendix G.

Your two points about the gradients of Scalar and Vector are both correct. First, the loss gradient of Scalar depends on the attention distribution, so we have replaced the sentence in §3.3 that overstated what equation 6 shows. Second, Vector has positional gradients that depend on the query, so the explanation based on equation 6 does not directly apply to Vector (Appendix A, right after equation S12). For the replaced sentence and the reason we include Vector in the comparison, please see the second half of "Separating the coupling form from other factors" in the general response.

In response to the comment that the mechanism remains unresolved, we have also revised the sentence in §1 that states the cause. The revised §1 attributes negative transfer to the structure of multiplicative coupling, and describes the early gradient conflict as an outcome of this structure when the two teachers require different spatial kernels. The results with other teacher pairs are consistent with this description. With PE-Spatial + CLIP, whose two teachers require similar spatial kernels, the conflict rate in the 3×3 layers of RoPE (NAF) is 16%, far below the 76% with DINOv3 + SigLIP2 (Table S15). The difference between the gains of RoPE (NAF) and Scalar is also the smallest for PE-Spatial + CLIP among the three teacher pairs (Table S14).

Q1. Does applying PCGrad separately to the parameter group change RoPE's multi-teacher transfer?

No, restricting PCGrad to the 3×3 layers does not change the gain of RoPE (NAF). We applied PCGrad to RoPE (NAF) using only the gradients of the 3×3 layers, both to detect conflicts and to project (RoPE (NAF) + PCGrad (3×3 only), §4.4). The table below shows the number of projections in steps 50 to 99, out of 100 candidates (two per step, one per teacher), and the gain on Cityscapes (Table 1, Table S23).

SchemeGradients used to detect conflictsProjections (of 100)Gain (mIoU)
RoPE (NAF)––−0.83
RoPE (NAF) + PCGradall parameters38−0.68
RoPE (NAF) + PCGrad (3×3 only)3×3 layers74−0.82

As the table shows, detecting conflicts with the gradients of the 3×3 layers alone increases the number of projections from 38 to 74. However, the gain is −0.82 mIoU, essentially unchanged from −0.83 mIoU without PCGrad. The gain thus remains negative, which is consistent with Prediction 3. The conflict itself also remains. With PCGrad (3×3 only), the cosine similarity in the 3×3 layers is −0.26, almost the same as −0.29 without PCGrad (Appendix I). The limitation that aligned gradients mask the conflict in the 3×3 layers does not apply to this setting.

AuthorsOpenReview ↗

Response to Reviewer G9DC (2/2), W2 and Q2

W2. The reliability and scope of the gain are uncertain, and the framing must accommodate AnyUp's result

You are right that the seven evaluation VFMs share the same trained upsampler. We therefore repeated the training of the upsampler itself, with other seeds and with two other teacher pairs. The results with other seeds are given in our response to Q2. With the two other teacher pairs (FRANCA + SigLIP2 and PE-Spatial + CLIP), the gain on Cityscapes is again negative for RoPE (NAF) and positive for Scalar (Table S14). With FRANCA + SigLIP2, the gain is −0.40 mIoU for RoPE (NAF) and +0.13 mIoU for Scalar. With PE-Spatial + CLIP, the gain is −0.23 mIoU for RoPE (NAF) and +0.04 mIoU for Scalar.

Given the DAVIS and VOC results, we now state that the gain turns from negative to positive only on Cityscapes (the third contribution in §1, and §5). The design rule (the last sentence of the abstract, and §5), in contrast, rests not on the sign of the gain but on additive coupling having a higher gain than multiplicative coupling. This ordering holds on Cityscapes and DAVIS (Table 1), and on VOC it holds near class boundaries when compared with RoPE (NAF) (Table 2). For details of the DAVIS and VOC results, please see "VOC, DAVIS, and the scope of the claims" in the general response.

You are also right that AnyUp reports improved generalization from training on several VFMs (AnyUp, Table 7). We had not mentioned this result, and our description of AnyUp in §1 was inaccurate, so we have revised §1 and §2 accordingly. AnyUp builds keys from both the image and the VFM features, so it can produce different attention weights for each VFM. NAF and RaysUp, in contrast, build queries and keys from the image alone, so all teachers share the same attention weights. The analysis and experiments in this paper concern architectures in which all teachers share the same attention weights (§2). In addition, Table 7 of AnyUp compares against a model trained with DINOv2 alone, whereas our gain compares against the best single-teacher model for each evaluation VFM, so the two measure different quantities. The results of AnyUp and ours are therefore compatible.

We also checked whether the positional encoding of RaysUp (RayPE) yields the same result as RoPE (NAF). As in the other comparisons in the paper, we replaced only the positional encoding of NAF with RayPE. The table below shows the gains on Cityscapes and DAVIS (all rows except RayPE are from Table 1).

SchemeCouplingGain on Cityscapes (mIoU)Gain on DAVIS (%)
None–+0.87−1.46
RoPE (NAF)mult.−0.83−2.80
RoPE (learned freq.)mult.−0.13−2.41
RayPEmult.−0.07−2.83
Scalaradd.+0.48−1.43
Vectoradd.+0.57−1.13

As the table shows, the gain of RayPE is comparable to that of RoPE (learned freq.) on Cityscapes and to that of RoPE (NAF) on DAVIS, and it is lower than the gains of Scalar and Vector on both benchmarks. RayPE also multiplies position into the features through a rotation. This result is therefore consistent with the results of the two RoPE schemes. RayPE was trained with seed 0, as in Table 1, and this result is not included in the revised paper.

This comment helped us state the results of prior work correctly in §1 and §2 and make explicit the architecture that this paper addresses. Thank you for raising this point.

Q2. How stable are the Cityscapes gains and the early-gradient ordering across training seeds?

Neither the sign of the Cityscapes gain nor the ordering of the early gradients changes across the three seeds. As you suggested, we retrained the single-teacher and multi-teacher models of RoPE (NAF) and Scalar with two other seeds (1 and 2), and retrained the linear probes. The table below shows the gain on Cityscapes for training with DINOv3 + SigLIP2.

SeedGain of RoPE (NAF) (mIoU)Gain of Scalar (mIoU)
0 (Table 1)−0.83+0.48
1−0.78+0.25
2−0.60+0.32
Mean ± standard deviation−0.74 ± 0.13+0.35 ± 0.12

As the table shows, across the three seeds the gain is −0.74 ± 0.13 mIoU for RoPE (NAF) and +0.35 ± 0.12 mIoU for Scalar, and no seed changes its sign (Appendix F, Table S13). Across the three seeds, the cosine similarity in the 3×3 layers is −0.19 or lower for RoPE (NAF) and +0.54 or higher for Scalar (Table S19).

Please let us know if any questions remain or if there is anything else we should verify.

I thank the authors for their response and the additional experiments. I can say that most of my concerns are addressed, specifically for the matters about PCGrad, multiple seeds, and different teacher pairs. I also need to say that I appreciate the corrections to the claims and the inclusion of a discussion of AnyUp.

My remaining concern is still the generality of the results, as the gains with other teacher pairs are quite small and multi-teacher training seems to still hurt on DAVIS. I think the conclusions should remain specific to the tested setting.

Overall, the additional experiments improve my confidence in the paper, and I am inclined to raise my rating.

AuthorsOpenReview ↗

Response to Reviewer G9DC

Thank you for reading our responses and the revised paper, and for raising your rating. To keep our conclusions specific to the tested setting, we have revised the abstract, §1, and §5 to describe the coupling form as a cause of negative transfer in multi-teacher training. In addition, the limitations in §5 now state that the conclusions are drawn from experiments on NAF using DINOv3 and SigLIP2 as teachers. We are glad that the additional experiments addressed most of your concerns.

Reviewer_v4jLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work looks at feature upsampling for VFM agnostic methods like recently proposed NAF. Feature upsamplers which are model agnostic are a good way to save compute and be able to get high resolution features for scene understanding with limited compute and more efficiently. Building on NAF it identifies the way positional encoding (ROPE) is added to be a major reason of low performance while doing multi teacher training. To solve it, the authors compare SCALAR and VECTOR strategies and show experiments to validate improvements from these. Evaluation is done on Cityscapes, VOC, and DAVIS across seven VFMs, five unseen. Additive schemes transfer better than RoPE on Cityscapes.

Critical strengths
  1. The authors do a well controlled study varying only the positional encoding strategy keeping other things like optimizer, training recipe etc. fixed.
  2. Compared to ROPE, SCALAR improves Cityscapes from 58.57 to 60.98 mIoU, at the cost of only 1,156 parameters.
  3. The authors perform experiments across different setting including unseen VFMs during training and support their explanation with per teacher gradient analysis and a PCGrad experiment.
Critical weaknesses
  1. Can Table 1 be updated with VOC numbers. Not sure why they are not included since in Table 2 the RoPE variants have the highest multi-teacher VOC scores?
  2. While the authors claim the additive embeddings turn multi-teacher training into a gain (Contribution L102-103), the multi-teacher gain is still negative for DAVIS, negative on VOC (computed from Table S6), and actually only holds for Cityscapes, and even there the largest gain is with no positional encoding. I feel the contribution is overstated.
  3. The claim around additive coupling avoiding early gradient conflict would require initializing them with the values similar to ROPE instead of zeros which is done right now.
  4. The margins in some cases are small, it would be good to report the numbers over a few runs with mean and std. Also all experiments use the same teacher pair, to show generality a different pair should be used as an ablation too.
  5. There is discrepancy when comparing with NAF which uses MSE while this paper uses cosine. For fair comparison one should use the same loss setting to remove any variability.
Questions for authors

I hope the authors can address the questions raised in the weaknesses around fair comparison, the claims and discrepancy I have noted. I think the responses will give more confidence in the results and improve the overall paper.

2 comments
AuthorsOpenReview ↗

Response to Reviewer v4jL (1/2), W1–W3

Thank you for recognizing the controlled comparison and for your comments on how our claims match the results. All section and table numbers in our responses refer to the revised paper.

W1. Table 1 lacks VOC, where the RoPE variants have the highest multi-teacher scores

We agree, and we have added VOC to Table 1. We had left VOC out of Table 1 because we assumed that VOC, where coarse object outlines suffice, would show no difference in gain between the coupling forms. However, we had not verified this assumption empirically, and it did not justify omitting the VOC gains from Table 1.

To check whether the negative transfer of multiplicative coupling also occurs on VOC, we measured the gain by distance from the class boundary (Trimap mIoU, §4.2). The table below compares the gain on VOC over all pixels with the gain within 16 pixels of a class boundary (Table 2).

SchemeAll pixels (mIoU)Within 16 pixels of a class boundary (mIoU)
RoPE (NAF)+0.03−0.90
Scalar−0.19−0.05
Vector−0.15+0.27

As the table shows, within 16 pixels of a class boundary, the gain of RoPE (NAF) is 0.85 mIoU lower than that of Scalar and also lower than that of Vector, consistent with the results on Cityscapes. In the same region, the multi-teacher score of Scalar is also higher than that of RoPE (NAF) (Table S12). The negative transfer of multiplicative coupling therefore also occurs on VOC near class boundaries. It does not appear in the overall gain on VOC because pixels within 16 pixels of a class boundary account for only 17.0% of the scored pixels on VOC, compared with 39.4% on Cityscapes (Table 2).

This comment led us to find that the negative transfer of multiplicative coupling is a loss of accuracy near class boundaries. Thank you for raising this point.

W2. The gain is positive only on Cityscapes, and even there the largest gain is with no positional encoding

We agree, and we now state that the gain turns from negative to positive only on Cityscapes (the third contribution in §1, and §5). The design rule, in contrast, rests not on the sign of the gain but on additive coupling having a higher gain than multiplicative coupling. This ordering holds on Cityscapes and DAVIS (Table 1), and on VOC it holds near class boundaries when compared with RoPE (NAF) (Table 2). For details of the DAVIS and VOC results, please see "VOC, DAVIS, and the scope of the claims" in the general response.

On Cityscapes, None, which uses no positional encoding, likely achieves the largest gain because its gradients do not conflict and a second teacher has more room to help (§4.2). Under None, the early-gradient conflict rate is 0%, just as under additive coupling (§4.3, Figure 3). In addition, the single-teacher scores of None are about 6 mIoU lower than those of Scalar (Table S8). However, because None cannot use positional information, its absolute score is the lowest among the five schemes. Its multi-teacher score is 55.62 mIoU, compared with 60.98 mIoU for Scalar (Score column of Table 1). Additive coupling preserves the high score provided by positional encoding while making the gain positive.

W3. The claim that additive coupling avoids early gradient conflict requires initializing the bias with values similar to RoPE

Additive coupling avoids the early gradient conflict even when the position bias is initialized from RoPE instead of zero (§4.3). As you suggested, we initialized the position bias of Scalar with the relative-position-dependent component of the attention score of RoPE (NAF) before training (Scalar (RoPE init.); see the "Scalar initialized from RoPE" paragraph of Appendix G). As Table S16 shows, the cosine similarity in the 3×3 layers of Scalar (RoPE init.) in early training is +0.54, which remains opposite in sign to the −0.29 of RoPE (NAF) (Table 3). The gain of Scalar (RoPE init.) is +0.48 mIoU, the same as that of Scalar initialized to zero.

AuthorsOpenReview ↗

Response to Reviewer v4jL (2/2), W4–W5

W4. The margins are small, so the mean and standard deviation over a few runs and a different teacher pair are needed

Across both multiple seeds and other teacher pairs, the gain is negative for RoPE (NAF) and positive for Scalar. First, we trained RoPE (NAF) and Scalar with three seeds. The table below shows the gain on Cityscapes for training with DINOv3 + SigLIP2.

SeedGain of RoPE (NAF) (mIoU)Gain of Scalar (mIoU)
0 (Table 1)−0.83+0.48
1−0.78+0.25
2−0.60+0.32
Mean ± standard deviation−0.74 ± 0.13+0.35 ± 0.12

As the table shows, across the three seeds the gain is −0.74 ± 0.13 mIoU for RoPE (NAF) and +0.35 ± 0.12 mIoU for Scalar, and no seed changes its sign (Appendix F, Table S13). RoPE (learned freq.) and Vector were trained with two seeds (0 and 1), so Table S13 reports their gains for each seed. Second, with two other teacher pairs (FRANCA + SigLIP2 and PE-Spatial + CLIP), the gain is again negative for RoPE (NAF) and positive for Scalar (Table S14). With FRANCA + SigLIP2, the gain is −0.40 mIoU for RoPE (NAF) and +0.13 mIoU for Scalar. With PE-Spatial + CLIP, the gain is −0.23 mIoU for RoPE (NAF) and +0.04 mIoU for Scalar.

W5. NAF is trained with MSE, so a fair comparison requires the same loss

All five schemes in the paper, including RoPE (NAF), are trained with the cosine distance, so they are compared under the same loss. With the mean squared error used by the original NAF, the SigLIP2 loss is 35 to 52 times the DINOv3 loss and dominates the mean of the two losses (§4.1, Table S3). We therefore use the cosine distance.

Following your comment, we also checked whether the ordering of the schemes changes with the mean squared error, training four schemes (None, RoPE (NAF), Scalar, and Vector) with it (Appendix C). The table below shows the gain on Cityscapes for each loss (Table S4). The models trained with the mean squared error were evaluated only on the two teacher VFMs used in training (DINOv3 and SigLIP2), so both columns report means over these two VFMs.

SchemeCosine distance (mIoU)Mean squared error (mIoU)
None+0.76−2.42
RoPE (NAF)−0.78−4.85
Scalar+0.42−1.95
Vector+0.47−1.76

As the table shows, with the mean squared error the gain is negative for every scheme, including None. These negative gains include a drop caused by the unequal contributions of the two losses, which cannot be disentangled from the drop caused by the coupling form. However, RoPE (NAF) remains the scheme with the lowest gain under both losses.

Please let us know if any questions remain or if there is anything else we should verify.

Reviewer_iDpZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies positional encoding in multi-teacher feature upsampling. On NAF, it compares fixed/learnable-frequency RoPE, scalar/vector additive positional terms, and no explicit positional encoding, using DINOv3 and SigLIP2 as teachers. It attributes RoPE’s negative transfer to early encoder-gradient conflict, examined through gradient measurements, a toy experiment, and PCGrad. On Cityscapes, additive schemes improve over their best single-teacher baselines by 0.48–0.57 mIoU versus −0.83 for standard RoPE; on DAVIS, they reduce, but do not eliminate, negative transfer.

Critical strengths
  1. The positional-swap study yields a useful empirical observation with limited architectural changes: Tables 1 and S3 show Scalar reaching 60.98 Cityscapes mIoU versus RoPE’s 58.57 with only 1,156 additional parameters; Table S4 separately confirms positive gains on the five unseen evaluation VFMs.
  2. The gradient analysis is more informative than a single aggregate cosine measurement: Table 3 localizes early conflict to RoPE’s 3×3 layers (−0.29 cosine, 76% conflict), while Tables S9–S11 explicitly show that stronger late conflict can coexist with better transfer under additive schemes.
  3. The supplement enables inspection of the mechanism: Appendices A–B provide derivations and a matched-initial-distribution toy experiment; Tables S5–S7 report every evaluation VFM separately.
Critical weaknesses
  1. The evidence establishes that positional schemes matter, but does not isolate coupling form as the cause. Equations (6)/(S12) correctly give ∂b/∂θ = 0, yet the encoder’s loss gradient still depends on position through softmax weights and teacher-dependent loss derivatives. Vector also explicitly couples queries with positional vectors (Eq. (5); Appendix A). Moreover, additive schemes start as None, whereas RoPE starts with a different attention function (Appendix C). Thus, these comparisons also change initialization, spatial prior, and parameterization. Matched-initial-kernel controls, a fixed additive bias, and a RoPE-plus-additive-bias intervention would help distinguish the proposed mechanism from these alternatives.
  2. PCGrad’s limited recovery does not rule out an optimization-side explanation. Appendix F explicitly explains that global projection can overlook conflict in 3×3 layers because aligned 1×1 gradients mask it: projections occur on 38% of steps 50–99 for RoPE, whereas Table 3 reports 76% conflict in the 3×3 group. Consequently, −0.68 versus −0.83 mIoU in Table 1 tests one particular intervention, not whether appropriately targeted optimization can resolve the issue. Test group-wise PCGrad on the identified layers and qualify the abstract/Section 5 claim accordingly.
  3. The general design rule exceeds the demonstrated scope and task outcomes. Training uses one teacher pair and one upsampler architecture (Section 4.1), although evaluation spans seven VFMs. On DAVIS, Scalar/Vector still lose 1.43/1.13 J&F relative to their best single-teacher baselines (Table 1). Applying the stated per-VFM gain definition to rounded Table S6 entries gives approximately −0.19/−0.15 mIoU for Scalar/Vector on VOC, versus +0.03/+0.17 for the RoPE variants. These results support a task-dependent advantage, not general resolution of negative transfer. Broaden teacher/architecture coverage or narrow the conclusion.
  4. The stability of small gains and early-gradient patterns is unknown. The Reproducibility Statement specifies seed 0 for all runs, with no training/probe variance or confidence intervals. This matters for the +0.48/+0.57 and −0.13 mIoU gains and the claimed early-conflict distinction (Table 1; Figure 3). Report paired repeats for both upsampler training and probing.
Questions for authors

See the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer iDpZ (1/2), W1–W2

Thank you for the detailed comments and for proposing concrete experiments. All section and table numbers in our responses refer to the revised paper.

W1. The comparisons also change initialization, spatial prior, and parameterization, so they do not isolate the coupling form as the cause

Following your suggestions, we ran two control experiments and concluded that the cause of negative transfer is the coupling form itself. First, with Scalar initialized from RoPE (Scalar (RoPE init.), Table S16), the cosine similarity in the 3×3 layers is +0.54 and remains positive, and the gain is +0.48 mIoU, the same as that of Scalar initialized to zero. Second, we kept the rotation of RoPE (NAF) and added the same position bias as Scalar (RoPE (NAF) + position bias). The table below shows the gain on Cityscapes for the four combinations of rotation and position bias (Table S17).

SchemeRotationPosition biasGain (mIoU)
None––+0.87
Scalar–✓+0.48
RoPE (NAF)✓–−0.83
RoPE (NAF) + position bias✓✓−1.09

As the table shows, the gain is negative for the two schemes with rotation and positive for the two schemes without it. Adding the position bias changes the gain of RoPE (NAF) from −0.83 to −1.09 mIoU, so it remains negative. For the procedure of the two control experiments and the argument leading to this conclusion, please see "Separating the coupling form from other factors" in the general response and Appendix G.

Both of your points about the equations are also correct. First, even though ∂b/∂θ = 0 holds in equations 6 and S12, the gradient of the loss depends on position through the softmax attention weights. We have therefore replaced the sentence in §3.3 that overstated what equation 6 shows. Second, Vector couples the query with the positional vectors in equation 5, so the explanation based on equation 6 does not directly apply to Vector (Appendix A, right after equation S12). For the replaced sentence and the reason we include Vector in the comparison, please see the second half of "Separating the coupling form from other factors" in the general response.

W2. Targeted optimization may resolve the conflict, and the claim in the abstract and §5 should be qualified

PCGrad restricted to the 3×3 layers does not resolve the negative transfer of RoPE (NAF) either. As you suggested, we applied PCGrad to RoPE (NAF) using only the gradients of the 3×3 layers, where Table 3 locates the conflict, both to detect conflicts and to project (RoPE (NAF) + PCGrad (3×3 only)). The table below shows the number of projections in steps 50 to 99, out of 100 candidates (two per step, one per teacher), and the gain on Cityscapes (Table 1, Table S23).

SchemeGradients used to detect conflictsProjections (of 100)Gain (mIoU)
RoPE (NAF)––−0.83
RoPE (NAF) + PCGradall parameters38−0.68
RoPE (NAF) + PCGrad (3×3 only)3×3 layers74−0.82

As the table shows, detecting conflicts with the gradients of the 3×3 layers alone increases the number of projections from 38 to 74. However, the gain is −0.82 mIoU, essentially unchanged from −0.83 mIoU without PCGrad. The conflict itself also remains. With PCGrad (3×3 only), the cosine similarity in the 3×3 layers is −0.26, almost the same as −0.29 without PCGrad (Appendix I). The issue of the gradients of the 1×1 layers masking the conflict in the 3×3 layers does not apply to this setting.

As for the claim in the abstract and §5, we have added "such as PCGrad".

Thank you for suggesting an experiment that targets the conflicting layers directly.

AuthorsOpenReview ↗

Response to Reviewer iDpZ (2/2), W3–W4

W3. The design rule exceeds the demonstrated teacher pairs, architecture, and tasks

Of the two options you suggested, we have broadened the teacher coverage and narrowed the conclusion for the tasks and the architecture.

With two other teacher pairs (FRANCA + SigLIP2 and PE-Spatial + CLIP), the gain is again negative for RoPE (NAF) and positive for Scalar (Table S14). With FRANCA + SigLIP2, the gain is −0.40 mIoU for RoPE (NAF) and +0.13 mIoU for Scalar. With PE-Spatial + CLIP, the gain is −0.23 mIoU for RoPE (NAF) and +0.04 mIoU for Scalar.

We agree regarding DAVIS and VOC, and we now state that the gain turns from negative to positive only on Cityscapes (the third contribution in §1, and §5). The design rule, in contrast, rests not on the sign of the gain but on additive coupling having a higher gain than multiplicative coupling. This ordering holds on Cityscapes and DAVIS (Table 1), and on VOC it holds near class boundaries when compared with RoPE (NAF) (Table 2). For details of the DAVIS and VOC results, please see "VOC, DAVIS, and the scope of the claims" in the general response.

For the architecture, rather than adding another architecture, we have narrowed the scope of the design rule. The design rule applies to upsamplers that build queries and keys from the image alone, so that all teachers share the same attention weights (§2). Besides NAF, RaysUp also has this structure. To check whether the positional encoding of RaysUp (RayPE) yields the same result as RoPE (NAF), we replaced only the positional encoding of NAF with RayPE, as in the other comparisons in the paper. The table below shows the gains on Cityscapes and DAVIS (all rows except RayPE are from Table 1).

SchemeCouplingGain on Cityscapes (mIoU)Gain on DAVIS (%)
None–+0.87−1.46
RoPE (NAF)mult.−0.83−2.80
RoPE (learned freq.)mult.−0.13−2.41
RayPEmult.−0.07−2.83
Scalaradd.+0.48−1.43
Vectoradd.+0.57−1.13

As the table shows, the gain of RayPE is comparable to that of RoPE (learned freq.) on Cityscapes and to that of RoPE (NAF) on DAVIS, and it is lower than the gains of Scalar and Vector on both benchmarks. RayPE also multiplies position into the features through a rotation. This result is therefore consistent with the results of the two RoPE schemes. RayPE was trained with seed 0, as in Table 1, and this result is not included in the revised paper.

W4. The stability of the small gains and the early-gradient patterns under repeated training and probing is unknown

The signs of the gains of Scalar, Vector, and RoPE (learned freq.) and the difference in the early gradients do not change when both the upsampler and the linear probes are retrained.

For the gains, we retrained the single-teacher and multi-teacher models of RoPE (NAF) and Scalar with two other seeds (1 and 2) and retrained the linear probes. The table below shows the gain on Cityscapes for training with DINOv3 + SigLIP2.

SeedGain of RoPE (NAF) (mIoU)Gain of Scalar (mIoU)
0 (Table 1)−0.83+0.48
1−0.78+0.25
2−0.60+0.32
Mean ± standard deviation−0.74 ± 0.13+0.35 ± 0.12

As the table shows, across the three seeds the gain is −0.74 ± 0.13 mIoU for RoPE (NAF) and +0.35 ± 0.12 mIoU for Scalar, and no seed changes its sign (Table S13). We also retrained Vector and RoPE (learned freq.) with another seed (seed 1). The gain of Vector is +0.57 mIoU with seed 0 and +0.41 mIoU with seed 1, and the gain of RoPE (learned freq.) is −0.13 mIoU with seed 0 and −0.16 mIoU with seed 1 (Table S13). The signs of the three gains you mention therefore do not change in any of the retrained runs. For a discussion of the variation in scores across seeds, please see the "Initial weights" paragraph of Appendix F.

For the early gradients, we measured the cosine similarity in the 3×3 layers with three seeds. As Table S19 shows, across all three seeds the cosine similarity is −0.19 or lower for RoPE (NAF) and +0.54 or higher for Scalar.

Please let us know if any questions remain or if there is anything else we should verify.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response (1/4)

We thank the three reviewers for their detailed and thoughtful reviews. The reviewers noted that our controlled comparison, which swaps only the positional encoding, identifies the design choice that determines the success or failure of multi-teacher training [iDpZ, v4jL, G9DC]. They also highlighted three results as strengths. Scalar adds only 1,156 parameters and raises the multi-teacher score on Cityscapes from the 58.57 mIoU of RoPE (NAF) to 60.98 mIoU [iDpZ, v4jL, G9DC]. The gain of Scalar is also positive on the five VFMs unseen during training [iDpZ, G9DC]. The per-teacher gradient analysis locates the conflict in the 3×3 layers of RoPE [iDpZ, G9DC].

Each concern led to an experiment that tests the basis of these strengths. The revised paper adds the following experiments and changes.

  • Two control experiments, Scalar initialized from RoPE and RoPE (NAF) with an added position bias (§4.2, §4.3, Appendix G, Tables S16 and S17)
  • PCGrad applied using only the gradients of the 3×3 layers (§4.4, Table 1, Appendix I, Table S23)
  • Training with other seeds and with two other teacher pairs (Appendix F, Tables S13–S15)
  • VOC results in Table 1, and the gain by distance from the class boundary (§4.2, Table 2, Appendix E)
  • Training with the mean squared error, the loss used in NAF (Appendix C, Tables S3 and S4)
  • Corrections to the description of AnyUp in §1 and §2, to one sentence in §3.3, and to the third contribution and §5

In the revised paper, added and revised text is color-coded by reviewer, as explained at the top of the first page. Below, we first address four concerns shared by two or more reviewers. We then respond to each reviewer point by point in the individual responses.

AuthorsOpenReview ↗

General Response (2/4)

Separating the coupling form from other factors

iDpZ (W1) and v4jL (W3) point out that RoPE (NAF) and Scalar differ in more than just the coupling form. RoPE (NAF) starts training from an attention score that contains the rotation, whereas Scalar starts from the same attention score as None. In addition, Scalar has a position bias, which is a learnable parameter, while RoPE (NAF) does not. This is correct, so we ran two control experiments that varied the initialization and the presence of a position bias, one at a time.

In the first control experiment, we initialized the position bias of Scalar with the relative-position-dependent component of the attention score of RoPE (NAF) before training (Scalar (RoPE init.), Table S16). If the difference in initialization were the cause of the conflict, the gradients of Scalar (RoPE init.) would also conflict. However, the cosine similarity in the 3×3 layers of Scalar (RoPE init.) is +0.54 and remains positive. The initialization procedure and the part of RoPE (NAF) that Scalar (RoPE init.) can reproduce are described in the "Scalar initialized from RoPE" paragraph of Appendix G.

In the second control experiment, we kept the rotation of RoPE (NAF) and added the same position bias as Scalar (RoPE (NAF) + position bias, Table S17). If the presence of a position bias determined the gain, adding the position bias to RoPE (NAF) would make its gain positive, as for Scalar. However, the measured gain of RoPE (NAF) + position bias is −1.09 mIoU and remains negative.

From these two control experiments, we conclude that the cause of negative transfer is neither the absence of positional parameters nor the part of the rotational effect of RoPE (NAF) that is determined by the relative position alone. The cause is the coupling form itself, which multiplies content and position. The argument leading to this conclusion is given in the "Discussion" paragraph of Appendix G.

Apart from the control experiments, iDpZ (W1) and G9DC (W1) raise two points about the equations. The first point is that, even under Scalar, the gradient of the loss that reaches the image encoder depends on position through the softmax attention weights. This is correct. The sentence in §3.3 stating that "there is no pathway through which a feature update by one teacher changes the spatial kernel of the other" overstated what equation 6 shows. We have replaced it with "the shape of the spatial kernel can be changed by updating the position bias b alone, without changing the features of the image encoder" (§3.3, right after equation 6).

The second point concerns Vector. Vector takes the inner product of the query and the positional vectors in equation 5, so the gradient that reaches the image encoder contains the positional vectors. This is correct. The explanation based on equation 6 is for Scalar and does not directly apply to Vector. However, in Vector, as in Scalar, the positional term is added to the content similarity rather than multiplied by it. A requirement on the shape of the spatial kernel therefore reaches the positional vectors of Vector directly, just as it reaches the position bias of Scalar (Appendix A, right after equation S12). In early training, the conflict rate between the gradients of the two teachers in the 3×3 layers of the image encoder is 0% under Vector, as under Scalar (§4.3, Table 3). We include Vector in the comparison not to explain the mechanism, but to verify that the gain of additive coupling does not depend on the expressiveness of the positional term (§3.3). The term that Scalar adds is a single scalar for each relative position and does not depend on the query. The term that Vector adds is the inner product of a vector for each relative position and the query, so it changes with the query. Despite this difference, the gain on Cityscapes is positive for both Scalar and Vector across two seeds (Table S13).

AuthorsOpenReview ↗

General Response (3/4)

PCGrad restricted to the 3×3 layers, and the claim in the abstract and §5

iDpZ (W2) and G9DC (W1, Q1) point out that PCGrad, which detects conflicts using the gradients of all parameters, may not project even when the gradients of the 3×3 layers conflict. We therefore applied PCGrad to RoPE (NAF) using only the gradients of the 3×3 layers, both to detect conflicts and to project (RoPE (NAF) + PCGrad (3×3 only), §4.4).

As Table S23 shows, the number of projections in steps 50 to 99 increased from 38 with PCGrad over all parameters to 74 with PCGrad over the 3×3 layers alone. However, as Table 1 shows, the gain of RoPE (NAF) + PCGrad (3×3 only) is −0.82 mIoU, essentially unchanged from the −0.83 mIoU of RoPE (NAF) without PCGrad. The conflict rate in the 3×3 layers is 74%, almost the same as the 76% without PCGrad. The number of projections and the conflict rate are discussed in the "PCGrad projections" paragraph of Appendix I.

PCGrad that targets the conflicting layers therefore does not resolve the negative transfer of RoPE (NAF). PCGrad changes only the direction of the update. It does not change the structure of multiplicative coupling, in which the features of the image encoder must form the spatial kernels of both teachers at once (§4.4). In response to these comments, we have added "such as PCGrad" to the claim in the abstract and §5.

Reproducibility of the results and other teacher pairs

iDpZ (W3, W4), v4jL (W4), and G9DC (W2, Q2) point out that every model was trained once, with seed 0, and with a single teacher pair, DINOv3 + SigLIP2. We therefore trained with other seeds and with other teacher pairs (Appendix F).

First, to check the reproducibility of the results, we retrained RoPE (NAF) and Scalar with two other seeds (1 and 2), which change the initial weights, and retrained the linear probes. The table below shows the gain on Cityscapes for training with DINOv3 + SigLIP2.

SeedGain of RoPE (NAF) (mIoU)Gain of Scalar (mIoU)
0 (Table 1)−0.83+0.48
1−0.78+0.25
2−0.60+0.32
Mean ± standard deviation−0.74 ± 0.13+0.35 ± 0.12

As the table shows, across all three seeds the gain remains negative for RoPE (NAF) and positive for Scalar (Table S13). We also retrained RoPE (learned freq.) and Vector with another seed (seed 1), and the gain remains negative for RoPE (learned freq.) and positive for Vector (Table S13). In addition, the difference in cosine similarity in the 3×3 layers in early training, negative for RoPE (NAF) and positive for Scalar, was reproduced across all three seeds (Appendix I, Table S19). The variation in scores across seeds is discussed in the "Initial weights" paragraph of Appendix F.

Next, to check whether the same result holds with other teacher pairs, we trained RoPE (NAF) and Scalar with two other teacher pairs. The table below shows the gain on Cityscapes for each teacher pair (seed 0).

Teacher pairGain of RoPE (NAF) (mIoU)Gain of Scalar (mIoU)
DINOv3 + SigLIP2 (Table 1)−0.83+0.48
FRANCA + SigLIP2−0.40+0.13
PE-Spatial + CLIP−0.23+0.04

As the table shows, for every teacher pair the gain is negative for RoPE (NAF) and positive for Scalar (Table S14). In addition, the smaller the difference between the spatial kernels that the two teachers require, the smaller the difference between the gains of RoPE (NAF) and Scalar (Table S15). The teacher pairs are discussed in the "Teacher pairs" paragraph of Appendix F.

The higher gain of additive coupling over multiplicative coupling is therefore not specific to seed 0 or to the teacher pair DINOv3 + SigLIP2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.