- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how positional encoding affects multi-teacher training of feature upsamplers. Using a fixed NAF architecture, it compares fixed and learned-frequency RoPE with scalar and vector additive positional terms and a no-position baseline. Models are trained with DINOv3, SigLIP2, or both, and evaluated on seven VFMs across Cityscapes, PASCAL VOC, and DAVIS. Additive variants yield positive multi-teacher gains on Cityscapes and reduce degradation on DAVIS. Early gradient measurements, learned positional biases, and a PCGrad experiment support the authors’ proposed explanation that positional coupling influences interference between teachers.
Critical strengths
-
The controlled positional-encoding comparison identifies a useful architectural choice. Scalar and Vector achieve Cityscapes gains of +0.51 and +0.61 mIoU on the five unseen VFMs, whereas fixed RoPE loses 0.86 mIoU (Table S4). Scalar also improves absolute multi-teacher performance from 58.57 to 60.98 mIoU while adding only 1,156 parameters. This provides a practical contribution beyond improving reconstruction of the training teachers.
-
The gradient analysis offers useful diagnostic evidence beyond downstream scores. Early conflict is concentrated in RoPE’s encoder layers: mean cosine −0.29 with conflict on 76% of steps, compared with +0.54 and 0% for the additive schemes (Table 3). The separate examination of late-training conflict is particularly helpful, since negative gradient cosine also occurs in models with positive downstream transfer.
Critical weaknesses
-
The proposed causal explanation is stronger than the evidence establishes: Scalar separates the attention-score terms, but its loss gradients still depend on the attention distribution; Vector explicitly retains query-dependent positional gradients (Eq. 5; Appendix A). These observations do not invalidate the positional swap, but leave its precise mechanism unresolved. Moreover, Appendix F acknowledges that global PCGrad can miss the identified conflicts because aligned gradients mask them. Its limited recovery therefore does not establish that optimization cannot address the relevant conflict. A targeted intervention would make this central explanatory claim more convincing.
-
The reliability and scope of the positive multi-teacher benefit remain uncertain: All experiments use seed 0 and one teacher pair. The +0.48/+0.57 Cityscapes gains lack repeat-run validation; evaluations on several VFMs share the same trained upsampler. On DAVIS, both additive variants still have negative multi-teacher gains, and VOC slightly favors RoPE (Tables 1 and S6). These results support a task-dependent finding within NAF. The broader framing also needs to accommodate AnyUp’s reported improvement in multi-backbone generalization under its different architecture (AnyUp, Table 7).
Questions for authors
-
Does applying PCGrad separately to the parameter group change RoPE’s multi-teacher transfer? This would directly address the conflict location identified in Table 3 and the masking limitation acknowledged in Appendix F.
-
How stable are the Cityscapes transfer gains and early-gradient ordering across training seeds? A focused repeat of the fixed-RoPE and Scalar single-versus-multi comparison would help establish whether the reported sign reversal is reproducible.
4 comments
Response to Reviewer G9DC (2/2), W2 and Q2
W2. The reliability and scope of the gain are uncertain, and the framing must accommodate AnyUp's result
You are right that the seven evaluation VFMs share the same trained upsampler. We therefore repeated the training of the upsampler itself, with other seeds and with two other teacher pairs. The results with other seeds are given in our response to Q2. With the two other teacher pairs (FRANCA + SigLIP2 and PE-Spatial + CLIP), the gain on Cityscapes is again negative for RoPE (NAF) and positive for Scalar (Table S14). With FRANCA + SigLIP2, the gain is −0.40 mIoU for RoPE (NAF) and +0.13 mIoU for Scalar. With PE-Spatial + CLIP, the gain is −0.23 mIoU for RoPE (NAF) and +0.04 mIoU for Scalar.
Given the DAVIS and VOC results, we now state that the gain turns from negative to positive only on Cityscapes (the third contribution in §1, and §5). The design rule (the last sentence of the abstract, and §5), in contrast, rests not on the sign of the gain but on additive coupling having a higher gain than multiplicative coupling. This ordering holds on Cityscapes and DAVIS (Table 1), and on VOC it holds near class boundaries when compared with RoPE (NAF) (Table 2). For details of the DAVIS and VOC results, please see "VOC, DAVIS, and the scope of the claims" in the general response.
You are also right that AnyUp reports improved generalization from training on several VFMs (AnyUp, Table 7). We had not mentioned this result, and our description of AnyUp in §1 was inaccurate, so we have revised §1 and §2 accordingly. AnyUp builds keys from both the image and the VFM features, so it can produce different attention weights for each VFM. NAF and RaysUp, in contrast, build queries and keys from the image alone, so all teachers share the same attention weights. The analysis and experiments in this paper concern architectures in which all teachers share the same attention weights (§2). In addition, Table 7 of AnyUp compares against a model trained with DINOv2 alone, whereas our gain compares against the best single-teacher model for each evaluation VFM, so the two measure different quantities. The results of AnyUp and ours are therefore compatible.
We also checked whether the positional encoding of RaysUp (RayPE) yields the same result as RoPE (NAF). As in the other comparisons in the paper, we replaced only the positional encoding of NAF with RayPE. The table below shows the gains on Cityscapes and DAVIS (all rows except RayPE are from Table 1).
| Scheme | Coupling | Gain on Cityscapes (mIoU) | Gain on DAVIS (%) |
|---|---|---|---|
| None | – | +0.87 | −1.46 |
| RoPE (NAF) | mult. | −0.83 | −2.80 |
| RoPE (learned freq.) | mult. | −0.13 | −2.41 |
| RayPE | mult. | −0.07 | −2.83 |
| Scalar | add. | +0.48 | −1.43 |
| Vector | add. | +0.57 | −1.13 |
As the table shows, the gain of RayPE is comparable to that of RoPE (learned freq.) on Cityscapes and to that of RoPE (NAF) on DAVIS, and it is lower than the gains of Scalar and Vector on both benchmarks. RayPE also multiplies position into the features through a rotation. This result is therefore consistent with the results of the two RoPE schemes. RayPE was trained with seed 0, as in Table 1, and this result is not included in the revised paper.
This comment helped us state the results of prior work correctly in §1 and §2 and make explicit the architecture that this paper addresses. Thank you for raising this point.
Q2. How stable are the Cityscapes gains and the early-gradient ordering across training seeds?
Neither the sign of the Cityscapes gain nor the ordering of the early gradients changes across the three seeds. As you suggested, we retrained the single-teacher and multi-teacher models of RoPE (NAF) and Scalar with two other seeds (1 and 2), and retrained the linear probes. The table below shows the gain on Cityscapes for training with DINOv3 + SigLIP2.
| Seed | Gain of RoPE (NAF) (mIoU) | Gain of Scalar (mIoU) |
|---|---|---|
| 0 (Table 1) | −0.83 | +0.48 |
| 1 | −0.78 | +0.25 |
| 2 | −0.60 | +0.32 |
| Mean ± standard deviation | −0.74 ± 0.13 | +0.35 ± 0.12 |
As the table shows, across the three seeds the gain is −0.74 ± 0.13 mIoU for RoPE (NAF) and +0.35 ± 0.12 mIoU for Scalar, and no seed changes its sign (Appendix F, Table S13). Across the three seeds, the cosine similarity in the 3×3 layers is −0.19 or lower for RoPE (NAF) and +0.54 or higher for Scalar (Table S19).
Please let us know if any questions remain or if there is anything else we should verify.
I thank the authors for their response and the additional experiments. I can say that most of my concerns are addressed, specifically for the matters about PCGrad, multiple seeds, and different teacher pairs. I also need to say that I appreciate the corrections to the claims and the inclusion of a discussion of AnyUp.
My remaining concern is still the generality of the results, as the gains with other teacher pairs are quite small and multi-teacher training seems to still hurt on DAVIS. I think the conclusions should remain specific to the tested setting.
Overall, the additional experiments improve my confidence in the paper, and I am inclined to raise my rating.
Response to Reviewer G9DC
Thank you for reading our responses and the revised paper, and for raising your rating. To keep our conclusions specific to the tested setting, we have revised the abstract, §1, and §5 to describe the coupling form as a cause of negative transfer in multi-teacher training. In addition, the limitations in §5 now state that the conclusions are drawn from experiments on NAF using DINOv3 and SigLIP2 as teachers. We are glad that the additional experiments addressed most of your concerns.
Response to Reviewer G9DC (1/2), W1 and Q1
Thank you for the careful review and the specific questions. All section and table numbers in our responses refer to the revised paper.
W1. The causal explanation is stronger than the evidence, and a targeted intervention is missing
We ran three targeted interventions, and none of them changed the sign of the gain. The first applies PCGrad using only the gradients of the 3×3 layers, and its result is given in our response to Q1. The second initializes Scalar from RoPE (Scalar (RoPE init.), Table S16), and the gain remains positive. The third keeps the rotation of RoPE (NAF) and adds the same position bias as Scalar (RoPE (NAF) + position bias). The table below shows the gain on Cityscapes for the four combinations of rotation and position bias (Table S17).
As the table shows, the gain is negative for the two schemes with rotation and positive for the two schemes without it. From the second and third interventions, we conclude that the cause of negative transfer is the coupling form itself. For the argument leading to this conclusion, please see "Separating the coupling form from other factors" in the general response and the "Discussion" paragraph of Appendix G.
Your two points about the gradients of Scalar and Vector are both correct. First, the loss gradient of Scalar depends on the attention distribution, so we have replaced the sentence in §3.3 that overstated what equation 6 shows. Second, Vector has positional gradients that depend on the query, so the explanation based on equation 6 does not directly apply to Vector (Appendix A, right after equation S12). For the replaced sentence and the reason we include Vector in the comparison, please see the second half of "Separating the coupling form from other factors" in the general response.
In response to the comment that the mechanism remains unresolved, we have also revised the sentence in §1 that states the cause. The revised §1 attributes negative transfer to the structure of multiplicative coupling, and describes the early gradient conflict as an outcome of this structure when the two teachers require different spatial kernels. The results with other teacher pairs are consistent with this description. With PE-Spatial + CLIP, whose two teachers require similar spatial kernels, the conflict rate in the 3×3 layers of RoPE (NAF) is 16%, far below the 76% with DINOv3 + SigLIP2 (Table S15). The difference between the gains of RoPE (NAF) and Scalar is also the smallest for PE-Spatial + CLIP among the three teacher pairs (Table S14).
Q1. Does applying PCGrad separately to the parameter group change RoPE's multi-teacher transfer?
No, restricting PCGrad to the 3×3 layers does not change the gain of RoPE (NAF). We applied PCGrad to RoPE (NAF) using only the gradients of the 3×3 layers, both to detect conflicts and to project (RoPE (NAF) + PCGrad (3×3 only), §4.4). The table below shows the number of projections in steps 50 to 99, out of 100 candidates (two per step, one per teacher), and the gain on Cityscapes (Table 1, Table S23).
As the table shows, detecting conflicts with the gradients of the 3×3 layers alone increases the number of projections from 38 to 74. However, the gain is −0.82 mIoU, essentially unchanged from −0.83 mIoU without PCGrad. The gain thus remains negative, which is consistent with Prediction 3. The conflict itself also remains. With PCGrad (3×3 only), the cosine similarity in the 3×3 layers is −0.26, almost the same as −0.29 without PCGrad (Appendix I). The limitation that aligned gradients mask the conflict in the 3×3 layers does not apply to this setting.