- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AnyVIA, a 299K-parameter visible–infrared feature adapter. For each modality, an RGE generates attention queries and keys from the input image and uses frozen encoder features as values for spatial aggregation and upsampling. The two branches are pretrained using shared-prompt mask supervision and within-modality crop consistency, and their features are added downstream. The authors assemble UAVRGBT235K, containing 117,668 image pairs and 7.15M automatically generated masks, and report transfer of a single frozen adapter across different encoders and eight downstream datasets covering five tasks.
Critical strengths
The paper’s practical contribution is an extensively pretrained adapter that can be reused in frozen form. The broad evaluation supports a useful conclusion: the pretrained adapter can transfer across encoders and tasks and improve downstream performance with frozen features.
Critical weaknesses
-
The paper establishes that pretrained AnyVIA works as a whole, but does not clearly explain the benefits of its key architectural choices. The “w/o pretraining” variant in Table 8 uses randomly initialized RGEs. Its performance below Base shows that an untrained adapter can disrupt features, but does not establish why the particular architecture is effective. Table 13 supports image-based guidance, but the channel-folding ablation remains unclear. The text states that all groups receive identical Q/K and that encoder features enter as values without learned projection, yet the “w/o ” variant does not specify the Q/K dimensions, attention heads, or value-channel assignments. It is therefore unclear whether this variant removes only folding or also changes the attention configuration. The results support the usefulness of the pretrained system, but do not fully substantiate the claimed architectural contributions.
-
The ablations do not clearly establish what joint visible–infrared training adds beyond within-modality crop-consistency training. Each RGE processes its own modality, downstream fusion is addition, and constrains each modality against its own high-resolution crop features. In Table 8, removing joint mask supervision still yields 60.68 mIoU and 49.99 mAP, compared with 61.43 and 50.04 for the full model—a difference of only 0.75 and 0.05 percentage points. The separate-supervision results do support joint supervision over separate supervision. However, they do not clearly explain what additional cross-modal capability it provides, or why detection performance barely exceeds the crop-consistency-only variant. Improvements across tasks support transferability, but do not directly establish that those improvements come from exploiting visible–infrared complementarity.
Questions for authors
-
Does NAF in Figure 3 use its original public checkpoint or a checkpoint pretrained on UAVRGBT235K? Is it applied separately to visible and infrared features before addition? Please clarify the checkpoint, input modalities, and output resolution used in this comparison, so that the reported differences can be interpreted correctly.
-
Do the “w/o ” and “Separate supervision” variants in Table 8 retain the same data sampling, crop sampling, and optimization budget as the full model? Please clarify how separate supervision is implemented and whether any settings besides the supervision objective change. Given that adding joint mask supervision improves KUSTDrone mAP by only 0.05 points over crop-consistency-only training, what do the reported results establish about the additional benefit of joint visible–infrared learning?
-
How are the Q/K dimensions, attention heads, and value-channel assignments configured in the “w/o ” variant? Does removing folding change which spatial attention weights are applied to each value channel, or only how the channels are batched? Please specify any other architectural or training changes and clarify whether the performance difference in Table 8 reflects a changed attention operation or variation between separate pretraining runs.
10 comments
Q2: Supervision-ablation settings, implementation, and joint-learning evidence
Do the “w/o ” and “Separate supervision” variants in Table 8 retain the same data sampling, crop sampling, and optimization budget as the full model? Please clarify how separate supervision is implemented and whether any settings besides the supervision objective change. Given that adding joint mask supervision improves KUSTDrone mAP by only 0.05 points over crop-consistency-only training, what do the reported results establish about the additional benefit of joint visible–infrared learning?
Data sampling, crop sampling, and optimization budgets are unchanged; the multi-seed results further quantify the benefit of joint supervision. All settings use the same 117,668 image pairs, sampler, at most 3 objects per image, augmentation, and mask-guided crops, with a global batch size of 64 and 10 pretraining epochs. Learning rates, schedules, and gradient-clipping settings are identical, and all selected checkpoints are from epoch 10.
Implementation of Separate supervision. Both branches pass through the same frozen SAM2 decoder under identical box prompts. Let denote their mask logits and the object annotation. The three settings differ as follows: \begin{array}{llll} \hline \text{Item} & \text{Full (Joint)} & \text{w/o }L_{\mathrm{mask}} & \text{Separate supervision} \\ \hline \text{Mask objective} & \ell(M_v+M_i,Y) & \substack{\text{Mask-related loss} \\ \text{weights set to 0}} & \substack{\ell(M_v,Y)\text{, }\ell(M_i,Y)\text{,} \\ \text{computed separately}} \\ \substack{\text{Crop consistency and} \\ \text{mask-guided crop placement}} & \text{Both retained} & \text{Both retained} & \text{Both retained} \\ \hline \end{array}
All settings retain the same crop-consistency weight and global gradient clipping. Separate supervises each branch's prediction individually. Full updates both branches using the error of their summed prediction for the shared object, directly aligning the pretraining objective with downstream sum fusion. Shared training settings are described in revised Section 3.4 and Appendix B.3.
Additional gains from joint supervision. Revised Section 4.7, Table 9 reports multi-seed means ± sample standard deviations: \begin{array}{lrr} \hline \text{Pretraining setting} & \text{CART mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline L_{\mathrm{crop}}\text{ only} & 60.41 \pm 0.67 & 49.59 \pm 0.37 \\ \text{Separate supervision} & 60.37 \pm 0.81 & 48.97 \pm 0.24 \\ \text{Full (Joint)} & 61.92 \pm 0.74 & 50.39 \pm 0.32 \\ \text{Full }-\text{ }L_{\mathrm{crop}}\text{ only (paired)} & +1.51 \pm 0.66 & +0.80 \pm 0.68 \\ \text{Full }-\text{ Separate (paired)} & +1.55 \pm 0.91 & +1.43 \pm 0.53 \\ \hline \end{array} Every paired difference in both comparisons is positive. Joint mask supervision adds 1.51 mIoU / 0.80 mAP beyond crop consistency, and jointly constraining the summed prediction also outperforms supervising the branches separately.
Q3: Q/K, attention heads, and V-channel assignments without folding
How are the Q/K dimensions, attention heads, and value-channel assignments configured in the “w/o ” variant? Does removing folding change which spatial attention weights are applied to each value channel, or only how the channels are batched? Please specify any other architectural or training changes and clarify whether the performance difference in Table 8 reflects a changed attention operation or variation between separate pretraining runs.
Both variants use the same RGE, 128-dimensional Q/K with four 32-dimensional heads, a neighborhood, RoPE, and output. Host features directly serve as V, without learned V/output projections. Their value-channel assignments differ as follows (zero-based ): \begin{array}{lll} \hline \text{Item} & \text{Full (folded)} & \text{w/o }g_n\text{ (unfolded)} \\ \hline \text{V organization} & \substack{\text{Groups of 128 channels;} \\ \text{zero-pad the last group;} \\ \text{fold groups into batch}} & \substack{\text{Split the native }C\text{ channels} \\ \text{into four contiguous blocks}} \\ \text{V width per head} & \text{32 per group} & C/4 \\ \text{Head assigned to channel }c & \lfloor(c\bmod128)/32\rfloor & \lfloor c/(C/4)\rfloor \\ \substack{\text{Real channels per head} \\ \text{during pretraining, }C=448} & \substack{\text{128 / 128 / 96 / 96,} \\ \text{plus 64 zero-padded channels}} & 112 / 112 / 112 / 112 \\ \text{Adapter parameters} & 0.299\,\mathrm{M} & 0.299\,\mathrm{M} \\ \hline \end{array} With fixed RGE weights and the same image, Q/K are identical, but the assignments above apply different heads' spatial weights to the original channels (revised Section 3.3, Appendix B.2).
Training and results. Data, losses, crop sampling, optimizer, learning-rate schedule, global batch size (64), and pretraining budget (10 epochs) are matched. Revised Section 4.7, Table 9 reports: \begin{array}{lrr} \hline \text{Setting} & \text{CART mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline \text{Full} & 61.92 \pm 0.74 & 50.39 \pm 0.32 \\ \text{w/o }g_n & 61.38 \pm 0.34 & 49.84 \pm 0.14 \\ \text{Full }-\text{ w/o }g_n\text{ (paired)} & +0.54 \pm 0.40 & +0.56 \pm 0.35 \\ \hline \end{array} All six paired downstream differences are positive. The statistics summarize downstream runs for each separately pretrained variant.
Q1: NAF checkpoint, modalities, and output grid in Figure 3
Figure 3 uses NAF's original public weights, frozen throughout, without retraining on UAVRGBT235K. The same NAF weights are applied separately to both modalities: visible features are guided by the visible image, and infrared features by the infrared image. The upsampled features are then summed. \begin{array}{lll} \hline \text{Comparison setting} & \text{NAF in Figure 3} & \text{AnyVIA} \\ \hline \text{Weight source} & \text{Official NAF checkpoint} & \text{UAVRGBT235K pretrained checkpoint} \\ \text{Downstream training status} & \text{Encoder and upsampler frozen} & \text{Encoder and adapter frozen} \\ \text{Guidance image} & \text{Each modality's own image} & \text{Each modality's own image} \\ \text{Modality processing / fusion} & \substack{\text{Upsample separately,}\ \text{then sum}} & \substack{\text{Upsample separately,}\ \text{then sum}} \\ \text{Output grid} & \text{Host }H\times W\text{ }\to\text{ }2H\times2W & \text{Host }H\times W\text{ }\to\text{ }2H\times2W \\ \text{Downstream protocol} & \substack{\text{Matched encoder, task head,} \\ \text{and 20-epoch training}} & \substack{\text{Matched encoder, task head,} \\ \text{and 20-epoch training}} \\ \hline \end{array}
Figure 3 therefore compares the two frozen pretraining schemes under the same downstream settings and output resolution (revised Section 4.3, Figure 3). We additionally retrained NAF separately for each modality on UAVRGBT235K to obtain the NAF+Lcrop@UAV control. It is reported separately from Figure 3's official checkpoint; its eight-benchmark results are in revised Appendix C.10, Table 21.