acceptodds
Under review as a conference paper at ICLR 2027

AnyVIA: A Universal Visible-Infrared Adapter for UAV Perception

Abstract

Vision foundation models (VFMs) provide transferable representations. However, aerial visible-infrared (VI) perception benefits from jointly modeling the complementary cues present in visible and infrared imagery. Most VFMs are pretrained primarily on RGB data and are not explicitly optimized for such cross-spectral modeling. Pretraining a dedicated aerial VI VFM is constrained by the cost of collecting large numbers of cross-spectrally registered image pairs, while existing VI adapters are typically tied to particular encoders and tasks. We introduce AnyVIA, a 299K-parameter encoder-agnostic adapter that transforms and spatially upsamples frozen VFM features for aerial VI perception. AnyVIA is pretrained once and then reused in frozen form across heterogeneous VFMs and downstream tasks. For each modality, a Raw Guidance Encoder (RGE) derives attention queries and keys solely from the input image, while host features enter only as values after parameter-free channel folding. This design decouples AnyVIA's learned weights from the host feature width. During pretraining, shared box prompts anchor both modalities to the same objects, while a directional consistency objective aligns refined feature directions with those of frozen host features extracted from corresponding higher-resolution crops. To provide dense supervision for cross-spectral representation learning, we further construct UAVRGBT235K, a pretraining resource for unmanned aerial vehicle (UAV) VI adaptation. It comprises 117,668 image pairs and 7.15 million automatically generated instance masks, substantially expanding the object-level annotations available for aerial VI pretraining. Across eight held-out UAV benchmarks spanning five perception tasks, the frozen AnyVIA yields improvements for diverse encoders. These results suggest that AnyVIA offers a scalable and reusable alternative to pretraining a dedicated VFM for aerial VI perception.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Foundation Models, Encoder-Agnostic Adaptation, Visible-Infrared, UAV Imagery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ShqeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AnyVIA, a 299K-parameter visible–infrared feature adapter. For each modality, an RGE generates attention queries and keys from the input image and uses frozen encoder features as values for spatial aggregation and upsampling. The two branches are pretrained using shared-prompt mask supervision and within-modality crop consistency, and their features are added downstream. The authors assemble UAVRGBT235K, containing 117,668 image pairs and 7.15M automatically generated masks, and report transfer of a single frozen adapter across different encoders and eight downstream datasets covering five tasks.

Critical strengths

The paper’s practical contribution is an extensively pretrained adapter that can be reused in frozen form. The broad evaluation supports a useful conclusion: the pretrained adapter can transfer across encoders and tasks and improve downstream performance with frozen features.

Critical weaknesses
  1. The paper establishes that pretrained AnyVIA works as a whole, but does not clearly explain the benefits of its key architectural choices. The “w/o pretraining” variant in Table 8 uses randomly initialized RGEs. Its performance below Base shows that an untrained adapter can disrupt features, but does not establish why the particular architecture is effective. Table 13 supports image-based guidance, but the channel-folding ablation remains unclear. The text states that all groups receive identical Q/K and that encoder features enter as values without learned projection, yet the “w/o gng_n” variant does not specify the Q/K dimensions, attention heads, or value-channel assignments. It is therefore unclear whether this variant removes only folding or also changes the attention configuration. The results support the usefulness of the pretrained system, but do not fully substantiate the claimed architectural contributions.

  2. The ablations do not clearly establish what joint visible–infrared training adds beyond within-modality crop-consistency training. Each RGE processes its own modality, downstream fusion is addition, and LcropL_{\mathrm{crop}} constrains each modality against its own high-resolution crop features. In Table 8, removing joint mask supervision still yields 60.68 mIoU and 49.99 mAP, compared with 61.43 and 50.04 for the full model—a difference of only 0.75 and 0.05 percentage points. The separate-supervision results do support joint supervision over separate supervision. However, they do not clearly explain what additional cross-modal capability it provides, or why detection performance barely exceeds the crop-consistency-only variant. Improvements across tasks support transferability, but do not directly establish that those improvements come from exploiting visible–infrared complementarity.

Questions for authors
  1. Does NAF in Figure 3 use its original public checkpoint or a checkpoint pretrained on UAVRGBT235K? Is it applied separately to visible and infrared features before addition? Please clarify the checkpoint, input modalities, and output resolution used in this comparison, so that the reported differences can be interpreted correctly.

  2. Do the “w/o LmaskL_{\mathrm{mask}}” and “Separate supervision” variants in Table 8 retain the same data sampling, crop sampling, and optimization budget as the full model? Please clarify how separate supervision is implemented and whether any settings besides the supervision objective change. Given that adding joint mask supervision improves KUSTDrone mAP by only 0.05 points over crop-consistency-only training, what do the reported results establish about the additional benefit of joint visible–infrared learning?

  3. How are the Q/K dimensions, attention heads, and value-channel assignments configured in the “w/o gng_n” variant? Does removing folding change which spatial attention weights are applied to each value channel, or only how the channels are batched? Please specify any other architectural or training changes and clarify whether the performance difference in Table 8 reflects a changed attention operation or variation between separate pretraining runs.

10 comments
AuthorsOpenReview ↗

Q1: NAF checkpoint, modalities, and output grid in Figure 3

Does NAF in Figure 3 use its original public checkpoint or a checkpoint pretrained on UAVRGBT235K? Is it applied separately to visible and infrared features before addition? Please clarify the checkpoint, input modalities, and output resolution used in this comparison, so that the reported differences can be interpreted correctly.

Figure 3 uses NAF's original public weights, frozen throughout, without retraining on UAVRGBT235K. The same NAF weights are applied separately to both modalities: visible features are guided by the visible image, and infrared features by the infrared image. The upsampled features are then summed. \begin{array}{lll} \hline \text{Comparison setting} & \text{NAF in Figure 3} & \text{AnyVIA} \\ \hline \text{Weight source} & \text{Official NAF checkpoint} & \text{UAVRGBT235K pretrained checkpoint} \\ \text{Downstream training status} & \text{Encoder and upsampler frozen} & \text{Encoder and adapter frozen} \\ \text{Guidance image} & \text{Each modality's own image} & \text{Each modality's own image} \\ \text{Modality processing / fusion} & \substack{\text{Upsample separately,}\ \text{then sum}} & \substack{\text{Upsample separately,}\ \text{then sum}} \\ \text{Output grid} & \text{Host }H\times W\text{ }\to\text{ }2H\times2W & \text{Host }H\times W\text{ }\to\text{ }2H\times2W \\ \text{Downstream protocol} & \substack{\text{Matched encoder, task head,} \\ \text{and 20-epoch training}} & \substack{\text{Matched encoder, task head,} \\ \text{and 20-epoch training}} \\ \hline \end{array}

Figure 3 therefore compares the two frozen pretraining schemes under the same downstream settings and output resolution (revised Section 4.3, Figure 3). We additionally retrained NAF separately for each modality on UAVRGBT235K to obtain the NAF+LcropL_{\mathrm{crop}}@UAV control. It is reported separately from Figure 3's official checkpoint; its eight-benchmark results are in revised Appendix C.10, Table 21.

AuthorsOpenReview ↗

Q2: Supervision-ablation settings, implementation, and joint-learning evidence

Do the “w/o ” and “Separate supervision” variants in Table 8 retain the same data sampling, crop sampling, and optimization budget as the full model? Please clarify how separate supervision is implemented and whether any settings besides the supervision objective change. Given that adding joint mask supervision improves KUSTDrone mAP by only 0.05 points over crop-consistency-only training, what do the reported results establish about the additional benefit of joint visible–infrared learning?

Data sampling, crop sampling, and optimization budgets are unchanged; the multi-seed results further quantify the benefit of joint supervision. All settings use the same 117,668 image pairs, sampler, at most 3 objects per image, augmentation, and mask-guided crops, with a global batch size of 64 and 10 pretraining epochs. Learning rates, schedules, and gradient-clipping settings are identical, and all selected checkpoints are from epoch 10.

Implementation of Separate supervision. Both branches pass through the same frozen SAM2 decoder under identical box prompts. Let Mv,MiM_v,M_i denote their mask logits and YY the object annotation. The three settings differ as follows: \begin{array}{llll} \hline \text{Item} & \text{Full (Joint)} & \text{w/o }L_{\mathrm{mask}} & \text{Separate supervision} \\ \hline \text{Mask objective} & \ell(M_v+M_i,Y) & \substack{\text{Mask-related loss} \\ \text{weights set to 0}} & \substack{\ell(M_v,Y)\text{, }\ell(M_i,Y)\text{,} \\ \text{computed separately}} \\ \substack{\text{Crop consistency and} \\ \text{mask-guided crop placement}} & \text{Both retained} & \text{Both retained} & \text{Both retained} \\ \hline \end{array}

All settings retain the same crop-consistency weight and global gradient clipping. Separate supervises each branch's prediction individually. Full updates both branches using the error of their summed prediction for the shared object, directly aligning the pretraining objective with downstream sum fusion. Shared training settings are described in revised Section 3.4 and Appendix B.3.

Additional gains from joint supervision. Revised Section 4.7, Table 9 reports multi-seed means ± sample standard deviations: \begin{array}{lrr} \hline \text{Pretraining setting} & \text{CART mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline L_{\mathrm{crop}}\text{ only} & 60.41 \pm 0.67 & 49.59 \pm 0.37 \\ \text{Separate supervision} & 60.37 \pm 0.81 & 48.97 \pm 0.24 \\ \text{Full (Joint)} & 61.92 \pm 0.74 & 50.39 \pm 0.32 \\ \text{Full }-\text{ }L_{\mathrm{crop}}\text{ only (paired)} & +1.51 \pm 0.66 & +0.80 \pm 0.68 \\ \text{Full }-\text{ Separate (paired)} & +1.55 \pm 0.91 & +1.43 \pm 0.53 \\ \hline \end{array} Every paired difference in both comparisons is positive. Joint mask supervision adds 1.51 mIoU / 0.80 mAP beyond crop consistency, and jointly constraining the summed prediction also outperforms supervising the branches separately.

AuthorsOpenReview ↗

Q3: Q/K, attention heads, and V-channel assignments without folding

How are the Q/K dimensions, attention heads, and value-channel assignments configured in the “w/o ” variant? Does removing folding change which spatial attention weights are applied to each value channel, or only how the channels are batched? Please specify any other architectural or training changes and clarify whether the performance difference in Table 8 reflects a changed attention operation or variation between separate pretraining runs.

Both variants use the same RGE, 128-dimensional Q/K with four 32-dimensional heads, a 9×99\times9 neighborhood, RoPE, and 2×2\times output. Host features directly serve as V, without learned V/output projections. Their value-channel assignments differ as follows (zero-based cc): \begin{array}{lll} \hline \text{Item} & \text{Full (folded)} & \text{w/o }g_n\text{ (unfolded)} \\ \hline \text{V organization} & \substack{\text{Groups of 128 channels;} \\ \text{zero-pad the last group;} \\ \text{fold groups into batch}} & \substack{\text{Split the native }C\text{ channels} \\ \text{into four contiguous blocks}} \\ \text{V width per head} & \text{32 per group} & C/4 \\ \text{Head assigned to channel }c & \lfloor(c\bmod128)/32\rfloor & \lfloor c/(C/4)\rfloor \\ \substack{\text{Real channels per head} \\ \text{during pretraining, }C=448} & \substack{\text{128 / 128 / 96 / 96,} \\ \text{plus 64 zero-padded channels}} & 112 / 112 / 112 / 112 \\ \text{Adapter parameters} & 0.299\,\mathrm{M} & 0.299\,\mathrm{M} \\ \hline \end{array} With fixed RGE weights and the same image, Q/K are identical, but the assignments above apply different heads' spatial weights to the original channels (revised Section 3.3, Appendix B.2).

Training and results. Data, losses, crop sampling, optimizer, learning-rate schedule, global batch size (64), and pretraining budget (10 epochs) are matched. Revised Section 4.7, Table 9 reports: \begin{array}{lrr} \hline \text{Setting} & \text{CART mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline \text{Full} & 61.92 \pm 0.74 & 50.39 \pm 0.32 \\ \text{w/o }g_n & 61.38 \pm 0.34 & 49.84 \pm 0.14 \\ \text{Full }-\text{ w/o }g_n\text{ (paired)} & +0.54 \pm 0.40 & +0.56 \pm 0.35 \\ \hline \end{array} All six paired downstream differences are positive. The statistics summarize downstream runs for each separately pretrained variant.

Reviewer_8K4mOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AnyVIA, an encoder-agnostic adapter for visible-infrared UAV perception. The adapter employs modality-specific Raw Guidance Encoders to generate attention queries and keys from raw images, while frozen foundation-model features provide values through parameter-free channel folding. AnyVIA is pretrained using shared-prompt mask supervision and directional consistency with high-resolution crop features. The authors also construct UAVRGBT235K from 117,668 visible-infrared image pairs and approximately 7.15 million automatically generated instance masks. Extensive experiments across eight held-out benchmarks and five downstream tasks demonstrate the transferability of a single frozen adapter across heterogeneous vision encoders.

Critical strengths

S1. Good transferability. AnyVIA can be pretrained once and reused across different backbones and tasks. Experiments on multiple encoders and five tasks support its effectiveness. S2. Simple and effective design. Raw-image guidance and channel folding allow the adapter to handle different feature dimensions. The ablation studies support the proposed design.

Critical weaknesses

W1. Unclear contribution of cross-modal learning. The visible and infrared features are processed separately and fused by simple addition. Table 8 shows that removing the mask loss has little impact on KUSTDrone (49.99 vs. 50.04 mAP). It is unclear whether the gains mainly come from feature distillation rather than joint cross-modal learning.

W2. Limited efficiency and baseline comparisons. The 2× adapter introduces high computational costs but provides limited gains over 1× on several tasks. End-to-end efficiency comparisons are missing. The paper also lacks comparisons with dedicated RGB-T detection, segmentation, and crowd-counting networks.

Questions for authors

Q1. How much improvement comes from joint RGB-T pretraining? Could the authors compare it with separately pretrained unimodal adapters? Q2. Could the authors report end-to-end FLOPs, latency, and memory before and after adding AnyVIA? Why is 2× upsampling necessary given its limited gains over 1×? Q3. Could the authors compare AnyVIA with dedicated RGB-T detection, segmentation, and crowd-counting networks on the same benchmarks?

I'm willing to raise my score to 3 points after you provide a convincing answer to my question.

6 comments
AuthorsOpenReview ↗

Q1: Joint versus separate pretraining

How much improvement comes from joint RGB-T pretraining? Could the authors compare it with separately pretrained unimodal adapters?

Joint pretraining improves over separate supervision by 1.55 mIoU on CART and 1.43 mAP on KUSTDrone (revised Section 4.7, Table 9). We keep the RGE architecture, pretraining data, and budget unchanged and compare two per-modality objectives with the joint objective. With LcropL_{\mathrm{crop}} alone, each branch learns within-modality consistency. Separate supervision computes the mask loss for each branch separately. Joint (Full) applies object supervision to the sum of the two branches' mask logits under the same box prompts. All adapters remain frozen downstream, with the same sum fusion. \begin{array}{lrr} \hline \text{Pretraining setting} & \text{CART mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline L_{\mathrm{crop}}\text{ only} & 60.41 \pm 0.67 & 49.59 \pm 0.37 \\ \text{Separate supervision} & 60.37 \pm 0.81 & 48.97 \pm 0.24 \\ \text{Joint (Full)} & 61.92 \pm 0.74 & 50.39 \pm 0.32 \\ \text{Joint }-\text{ }L_{\mathrm{crop}}\text{ only (paired)} & +1.51 \pm 0.66 & +0.80 \pm 0.68 \\ \text{Joint }-\text{ Separate (paired)} & +1.55 \pm 0.91 & +1.43 \pm 0.53 \\ \hline \end{array}

The joint objective also improves individual-branch transfer and cooperation between branches. With only one modality as input, Joint exceeds Separate in all six comparisons across three benchmarks, indicating gains in each branch's representation. Each cell below reports Joint / Separate (revised Section 4.7, Table 10): \begin{array}{lrrr} \hline \text{Input} & \text{CART mIoU}\uparrow & \text{Kust4K mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline \text{Visible only} & 59.24 / 58.61 & 48.03 / 47.36 & 47.77 / 47.25 \\ \text{Infrared only} & 54.15 / 53.74 & 43.23 / 42.52 & 37.06 / 36.46 \\ \hline \end{array} We additionally pretrain NAF independently for each modality on the same aerial VI data as an independent-training control. AnyVIA performs better on six of eight benchmarks. Full results are in revised Section 4.3, Table 6.

AuthorsOpenReview ↗

Q2: Complete-model cost and the role of 2x output

Could the authors report end-to-end FLOPs, latency, and memory before and after adding AnyVIA? Why is 2× upsampling necessary given its limited gains over 1×?

We now report complete-model inference costs for all tasks; the value of 2×2\times output is particularly clear for detection.

The following uses frozen DINOv2-B, an RTX 4090 GPU, and batch size 1. Each cell reports GFLOPs / latency in ms / peak memory in MiB (revised Section 4.7, Table 11; Appendix C.8, Tables 17–18): \begin{array}{lrr} \hline \text{Task / benchmark} & \text{Base} & \text{AnyVIA }2\times \\ \hline \text{CART} & 606.3 / 31.55 / 548.7 & 661.0 / 40.61 / 651.5 \\ \text{Kust4K} & 606.3 / 31.72 / 548.7 & 661.0 / 40.62 / 651.5 \\ \text{UAV RGB-T 2400 / VIAC} & 606.5 / 31.89 / 548.9 & 661.6 / 40.83 / 651.7 \\ \text{DroneRGBT} & 795.5 / 48.28 / 650.5 & 850.7 / 57.50 / 657.4 \\ \text{KUSTDrone} & 434.1 / 45.10 / 539.8 & 485.0 / 50.54 / 641.4 \\ \text{UAV-CB} & 434.1 / 44.07 / 539.8 & 485.0 / 50.14 / 641.3 \\ \text{ATR-UMOD} & 434.3 / 46.84 / 540.4 & 485.2 / 55.89 / 642.0 \\ \hline \end{array} Adding AnyVIA 2×2\times increases complete-model FLOPs by 6.9%–11.7%. For example, KUSTDrone adds 50.9 GFLOPs and 5.43 ms.

Why use 2×2\times? With the same 20-epoch training, native f=1f=1 achieves 45.65, 27.73, and 10.58 mAP on the three detection benchmarks, close to Base. The f=2f=2 variant further improves them by 4.39, 4.82, and 4.38 mAP: \begin{array}{lrrrr} \hline \text{Detection benchmark, mAP}\uparrow & \text{Base} & \text{AnyVIA }f=1 & \text{Bilinear }2\times & \text{AnyVIA }f=2 \\ \hline \text{KUSTDrone} & 45.40 & 45.65 & 45.65 & 50.04 \\ \text{UAV-CB} & 27.38 & 27.73 & 28.14 & 32.55 \\ \text{ATR-UMOD} & 10.80 & 10.58 & 13.14 & 14.96 \\ \hline \end{array} AnyVIA 2×2\times also exceeds bilinear upsampling at the same resolution by 4.39, 4.41, and 1.82 mAP, supporting additional gains from image-guided refinement beyond a denser grid. For the complete KUSTDrone detector, f=1f=1 costs 444.3 GFLOPs / 49.45 ms, versus 485.0 GFLOPs / 50.54 ms for f=2f=2: an additional 9.2% FLOPs and 1.09 ms measured latency. These results provide a clear accuracy–cost basis for using 2×2\times output in detection.

AuthorsOpenReview ↗

Q3: Comparison with dedicated RGB-T segmentation, detection, and counting networks

Could the authors compare AnyVIA with dedicated RGB-T detection, segmentation, and crowd-counting networks on the same benchmarks?

We have added comparisons with dedicated RGB-T networks for segmentation, detection, and counting, as requested (revised Appendix C.11, Table 21). We use the official implementations and public pretrained weights of TUNI-Base, M²D-LIF, and Free Lunch, with 20 downstream epochs on the same data splits. AnyVIA uses frozen DINOv2-B for segmentation/detection and frozen CLIP RN50 for counting. The adapter also remains frozen; only the task head is trained. \begin{array}{llrr} \hline \text{Task / benchmark} & \text{Dedicated method} & \text{Dedicated result} & \text{AnyVIA result} \\ \hline \substack{\text{Segmentation / CART,} \\ \text{mIoU}\uparrow} & \text{TUNI-Base} & 56.52 & 61.43 \\ \substack{\text{Detection / KUSTDrone,} \\ \text{mAP}\uparrow} & \mathrm{M}^{2}\text{D-LIF} & 45.11 & 50.04 \\ \substack{\text{Detection / UAV-CB,} \\ \text{mAP}\uparrow} & \mathrm{M}^{2}\text{D-LIF} & 40.37 & 32.55 \\ \substack{\text{Detection / ATR-UMOD,} \\ \text{mAP}\uparrow} & \mathrm{M}^{2}\text{D-LIF} & 13.15 & 14.96 \\ \substack{\text{Counting / DroneRGBT,} \\ \text{MAE}\downarrow} & \text{Free Lunch} & 12.63 & 13.40 \\ \substack{\text{Counting / DroneRGBT,} \\ \text{RMSE}\downarrow} & \text{Free Lunch} & 16.13 & 15.89 \\ \hline \end{array} AnyVIA is higher by 4.91 mIoU, 4.93 mAP, and 1.81 mAP on CART, KUSTDrone, and ATR-UMOD, respectively. Its counting performance is close to Free Lunch, with 0.24 lower RMSE and 0.77 higher MAE. M²D-LIF performs better on UAV-CB. These results show that AnyVIA, pretrained once and reused frozen, can reach the performance level of dedicated networks on multiple tasks while retaining cross-task reuse.

Reviewer_ckbUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AnyVIA, a 299K-parameter adapter that sits on top of a frozen vision foundation model and refines its visible and infrared features for UAV perception, while upsampling them 2×. Each modality has its own Raw Guidance Encoder (RGE) that computes attention queries and keys from the raw image only. The frozen host features are used directly as values, split into n = ⌈C/D⌉ channel groups of width D = 128, so the learned weights do not depend on the host channel width C. The two RGEs are pretrained once on top of SAM2's frozen Hiera-B+ trunk, using a new dataset, UAVRGBT235K (117,668 visible-infrared pairs pooled from 10 public sources, with 7.15M SAM2-generated masks). Pretraining combines a SAM2 mask loss on the summed logits of both modalities under a shared box prompt (L_mask) and a cosine loss against frozen host features of higher-resolution crops (L_crop). The frozen adapter is then plugged into 13 encoders and evaluated with trained task heads on 8 UAV benchmarks that are disjoint from the pretraining sources, covering semantic segmentation, salient and camouflaged object detection, horizontal and oriented box detection, and crowd counting.

Critical strengths
  1. The evaluation is broad and includes the right controls. A single frozen checkpoint, pretrained with one host (Hiera-B+, C = 448), is tested on 13 encoders with very different pretraining (CLIP ViT-B, CLIP RN50, SigLIP2, DINOv2-S/B/L, DINOv2 with registers, DINOv3, Franca, Swin-B, and the spectral models M-SpecGene, InfMAE and UNIP), on 8 dataset-disjoint benchmarks and 5 tasks. With paired inputs, every main-table setting improves over the no-adapter baseline, and the authors include the controls I would have asked for first: bilinear 2× at the same output resolution (Tab. 11), an f = 1 variant without upsampling (Fig. 8), and image- vs. feature-derived guidance (Tab. 13). For people who want to reuse off-the-shelf RGB encoders on aerial RGB-T data, a plug-in that needs no per-encoder retraining is practically useful.

  2. UAVRGBT235K could be valuable on its own. Pooling 10 public aerial RGB-T datasets into 117.7K standardized pairs with 7.15M masks (box-prompted plus automatic SAM2 masks, merged per modality) goes far beyond the few boxes per image in the original sources. Its value depends on release, which the paper does not commit to, and on mask quality, which is not evaluated, but the curation effort is substantial.

Critical weaknesses

Post-rebuttal update (score raised from 2 to 3): Weakness 2 is largely resolved. Tab. 13 now covers all benchmarks, Tabs. 1-5 and 9 report three seeds (the gains over Base hold up), and the cost issue is fixed. Weakness 1 is only partly resolved. In Tab. 6 (single seed), AnyVIA's margins over NAF trained on the same data are within the seed std from Tabs. 1-5, and Tab. 19 shows a randomly initialized, task-trained adapter beating the frozen pretrained one on two of three benchmarks. Several points from the rebuttal (relation to NAF, mask audit, random-crop control) are not yet in the paper. Details are in my comment below.

  1. The gains are not clearly VI-specific. Eq. (4) has no learnable value or output projection, and the two modalities only meet in a parameter-free sum, so each output channel is just an image-guided spatial average of the same channel. This closely matches NAF's operator (image-only Q/K, host features as values, neighborhood attention with RoPE), and the overlap is not discussed. The ablations agree: without L_mask, i.e. without the mask supervision and shared prompts, the adapter keeps 82% of the CART gain and 99% of the KUSTDrone gain (Tab. 8), and the remaining L_crop closely mirrors AnyUp's crop-based objective. Generic upsamplers are only compared on CART, where NAF already recovers about 92% of the gain for DINOv2-B, and the only VI-adapter baseline (UniRGB-IR) scores below the no-adapter Base on two of the three benchmarks.

  2. The reporting is not reliable enough. Tab. 10 is explicitly restricted to "the five benchmarks for which AnyVIA improves all three encoder scales", and the encoder subsets vary across Tabs. 1-5 without explanation. No seeds or variances are given, although several conclusions rest on differences below 0.5 points. The same settings also have different numbers in different places (e.g., DINOv2-B on KUSTDrone is 45.40 → 50.04 in Tabs. 4 and 8 but 45.49 → 49.84 in Tab. 11; Sec. 4.7 cites +4.39 over bilinear while Tab. 11 gives +4.09).

Questions for authors
  1. What exactly differs from NAF architecturally? Could you compare against NAF and AnyUp on the detection and SOD benchmarks, and ideally against NAF retrained on UAVRGBT235K with L_crop only, to isolate what the VI-specific parts add?

  2. Given Tab. 8, how much do the 7.15M masks matter beyond placing the crops? Do random crop locations work as well, and was the SAM2 mask quality on infrared images checked?

  3. Why does UniRGB-IR fall below the no-adapter Base (75.08 vs. 80.22 S_α on UAV RGB-T 2400, 20.92 vs. 27.38 mAP on UAV-CB)? Was it trained with the same 20-epoch schedule?

  4. Tab. 14's cost scales exactly with n (about 39 GFLOPs per group at 2×), and all groups get identical Q/K (l. 228). Is the guidance recomputed n times, and if so, what does the w/o g_n ablation change? How does the 4× cheaper f = 1 variant do on detection?

  5. Could you add the three benchmarks missing from Tab. 10, report mean ± std over seeds for the main tables and Tab. 8, and reconcile the numbers that differ between Tab. 11 and the main tables?

6 comments
AuthorsOpenReview ↗

Q1: Architectural differences from NAF and additional comparisons

What exactly differs from NAF architecturally? Could you compare against NAF and AnyUp on the detection and SOD benchmarks, and ideally against NAF retrained on UAVRGBT235K with L_crop only, to isolate what the VI-specific parts add?

Thank you for suggesting that we better separate the contribution of the VI-specific components. We provide three levels of evidence: comparisons with official upsamplers, NAF retrained on the same domain, and supervision ablations with the same architecture.

Architecture and learning mechanism. AnyVIA follows NAF's image-guided neighborhood aggregation, where the original image generates Q/K, host features directly serve as V, and RoPE provides positional encoding. AnyVIA is designed for dual-modal adaptation. Each modality has an RGE with detail and context branches. During pretraining, both branches predict the same object under shared box prompts, with joint supervision on their summed mask logits and crop consistency within each modality. The adapter thus learns both spatial refinement and how the two predictions jointly serve the same object.

Generic upsampling and same-domain retraining. We extended the comparison to SOD and three detection benchmarks, and retrained NAF separately for each modality on UAVRGBT235K using only LcropL_{\mathrm{crop}}, for 25k steps per modality. All rows below use frozen DINOv2-B, matched task heads, sum fusion, and 20 downstream epochs. All upsamplers produce the same 2×2\times grid. \begin{array}{lrrrrr} \hline \text{Benchmark / metric} & \text{Base} & \text{Official NAF} & \text{Official AnyUp} & \text{NAF+}L_{\mathrm{crop}}\text{@UAV} & \text{AnyVIA} \\ \hline \text{UAV RGB-T 2400, }S_\alpha\uparrow & 80.22 & 82.55 & 81.62 & 83.34 & 82.82 \\ \text{KUSTDrone, mAP}\uparrow & 45.40 & 49.83 & 48.71 & 49.42 & 50.04 \\ \text{UAV-CB, mAP}\uparrow & 27.38 & 29.38 & 28.07 & 31.73 & 32.55 \\ \text{ATR-UMOD, mAP}\uparrow & 10.80 & 14.23 & 13.54 & 14.07 & 14.96 \\ \hline \end{array} AnyVIA exceeds official NAF and AnyUp on all four benchmarks. After NAF is retrained on the same aerial VI data, AnyVIA remains higher by 0.62, 0.82, and 0.89 mAP on the three detection benchmarks. The complete VI adaptation scheme therefore provides further gains on multiple downstream tasks beyond per-modality upsampling trained on aerial data.

Additional gains from the joint VI objective. Keeping the RGE architecture unchanged, we compare crop consistency alone, separate branch supervision, and joint supervision. The full ablation results are in revised Table 9. \begin{array}{llrr} \hline \text{Comparison} & \text{VI supervision examined} & \text{CART mIoU gain}\uparrow & \text{KUSTDrone mAP gain}\uparrow \\ \hline \text{Full }-\text{ }L_{\mathrm{crop}}\text{ only} & \substack{\text{Joint mask supervision} \\ \text{of a shared object}} & +1.51 & +0.80 \\ \text{Full }-\text{ Separate supervision} & \substack{\text{Joint constraint on the} \\ \text{two branches' predictions}} & +1.55 & +1.43 \\ \hline \end{array} Both comparisons are positive. Object supervision improves transfer, and supervising the sum of the two predictions outperforms supervising each branch separately. Together, these results support the contribution of the VI-specific components. Shared-object constraints extend within-modality feature refinement into representation adaptation for dual-modal prediction, yielding consistently observed downstream gains.

AuthorsOpenReview ↗

Q2: Mask supervision, crop placement, and infrared mask quality

Given Tab. 8, how much do the 7.15M masks matter beyond placing the crops? Do random crop locations work as well, and was the SAM2 mask quality on infrared images checked?

The 7.15M masks provide object-level pretraining supervision, and locating foreground crops is only one use of them. We evaluate these roles separately by removing the mask loss while retaining mask-guided crop placement, and by replacing all mask use with uniformly random crops. \begin{array}{lrr} \hline \text{Setting} & \text{CART mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline \text{Base} & 57.43 \pm 0.74 & 45.47 \pm 0.08 \\ \substack{L_{\mathrm{crop}}\text{ only,}\ \text{mask-guided crops}} & 60.41 \pm 0.67 & 49.59 \pm 0.37 \\ \text{Random crops, no masks} & 59.87 \pm 0.56 & 49.21 \pm 0.43 \\ L_{\mathrm{mask}}\text{ only} & 59.79 \pm 0.98 & 47.10 \pm 0.36 \\ \text{Full} & 61.92 \pm 0.74 & 50.39 \pm 0.32 \\ \substack{\text{Mask supervision: Full }- \ L_{\mathrm{crop}}\text{ only }} & +1.51 \pm 0.66 & +0.80 \pm 0.68 \\ \substack{\text{Crop placement: mask-guided} -\text{ random}} & +0.54 & +0.38 \\ \substack{\text{Complete scheme: Full }- \text{random crops}} & +2.05 & +1.18 \\ \hline \end{array} How much do masks contribute beyond placing crops? With mask-guided crop placement unchanged, adding joint mask supervision improves CART and KUSTDrone by 1.51 mIoU and 0.80 mAP, respectively. These increments correspond to approximately 33.6% and 16.2% of Full's total gains over Base. Dense masks therefore provide a clear additional transfer benefit through shared-object supervision, beyond helping select crop locations.

Do random crop locations work as well? Random crops learn useful within-modality consistency, but do not reach the same mean performance in these experiments. Mask-guided placement adds 0.54 mIoU / 0.38 mAP, and joint object supervision adds a further 1.51 mIoU / 0.80 mAP. The complete scheme exceeds the mask-free random-crop control by 2.05 mIoU / 1.18 mAP. UAVRGBT235K masks thus support both foreground sampling and object-level learning, with joint supervision providing the larger increment.

Infrared mask quality. We conducted one round of manual verification on the publicly released masks of UAVRGBT235K. Specifically, 10% of the image pairs (approximately 11,770 pairs) were sampled by source proportion using a fixed random seed, and the sampling list was frozen prior to annotation. The sampling was stratified by (source, modality, branch), with uniform random selection within each stratum and no model-based pre-screening. For each image pair, one prompted mask and one automatic mask were drawn per modality, yielding approximately 47,000 questions in total. Two experts annotated the side-by-side visible and infrared images, selecting from the following options: "good / poor boundary / not an object or fragment / misaligned or ghosting / indistinguishable." Ten percent of the questions were answered twice by both experts to compute the inter-rater agreement, and response speed and consecutive identical answers were monitored per annotator. Results from this Mask Verification (MV) round are as follows: the inter-rater agreement on repeated questions is 59%, the overall bad-mask rate is 28%, and the "indistinguishable" proportion is 12% for infrared and 17% for visible light.

Resource release. We plan to release the masks, conversion scripts for all ten sources, file and split lists for the 117,668 image pairs, pretrained weights, and training/evaluation code. Original images will be provided according to each source's license; where redistribution is not permitted, we will provide acquisition and reconstruction scripts.

AuthorsOpenReview ↗

Q3: UniRGB-IR results below Base and the training setup

Why does UniRGB-IR fall below the no-adapter Base (75.08 vs. 80.22 S_α on UAV RGB-T 2400, 20.92 vs. 27.38 mAP on UAV-CB)? Was it trained with the same 20-epoch schedule?

We have completed 20-epoch training checks for both SOD and UAV-CB.

The original low scores relate to modality input paths and data alignment. Both UAV RGB-T 2400 and UAV-CB are unregistered, with annotations in infrared coordinates. UniRGB-IR is RGB-primary. The visible image passes through the frozen ViT, while infrared enters as a prompt through the spatial prior module. Its visible backbone features are therefore spatially misaligned with the supervision coordinates. Base and AnyVIA encode both modalities separately before fusion, retaining the complete infrared features that correspond to the annotation coordinates.

For a clearer VI adaptation comparison, the revision evaluates CART, pixel-aligned Kust4K, and weakly aligned KUSTDrone (revised Section 4.4, Table 7): \begin{array}{lrrr} \hline \text{Method} & \text{CART mIoU}\uparrow & \text{Kust4K mIoU}\uparrow & \text{KUSTDrone mAP}\uparrow \\ \hline \text{Base} & 57.22 & 55.34 & 45.40 \\ \text{UniRGB-IR} & 58.81 & 57.55 & 47.46 \\ \text{AnyVIA} & 61.43 & 58.16 & 50.04 \\ \text{AnyVIA }-\text{ UniRGB-IR} & +2.62 & +0.61 & +2.58 \\ \hline \end{array} UniRGB-IR improves over Base on all three benchmarks, demonstrating the effectiveness of its adaptation mechanism in this setting. AnyVIA provides further consistent gains while directly reusing the same frozen 0.299M parameters; UniRGB-IR trains approximately 8.90M adapter parameters per task. This comparison more directly demonstrates AnyVIA's cross-task reuse and downstream adaptation efficiency.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful assessment of AnyVIA. The revised manuscript, with key changes marked in blue, brings the downstream fusion analysis and matched NAF comparison into the main text and clarifies the protocols for the dedicated RGB-T comparisons.

  • Joint object supervision improves downstream performance and fusion. Full improves over crop consistency alone by 1.51 mIoU points on CART and 0.80 mAP points on KUSTDrone, and over separate supervision by 1.55 and 1.43 points, respectively. On 7 of 8 held-out benchmarks, joint supervision yields a larger gain from fusion over the stronger single modality than separate supervision does (Section 4.7, Tables 9–10). On pretraining images, summing the mask logits yields 69.62% IoU with joint supervision versus 68.35% with separate supervision (Appendix C.9, Table 20).
  • The expanded upsampler and adapter comparisons support frozen reuse. AnyVIA outperforms official NAF on 7 of 8 benchmarks, AnyUp on all 8, and each NAF variant pretrained on UAVRGBT235K on 6 of 8. With matched joint mask supervision and pretraining budgets, AnyVIA gains 4.42 mAP points on UAV-CB, 1.78 mAP points on KUSTDrone, and 1.52 mIoU points on CART (Section 4.3, Table 6, and Appendix C.10). It also outperforms UniRGB-IR on all three evaluated benchmarks while reusing a frozen 0.299M-parameter adapter, compared with 8.90M adapter parameters trained per task for UniRGB-IR (Section 4.4, Table 7).
  • The dedicated RGB-T comparisons now document pretraining and downstream protocols. With its encoder and adapter frozen, AnyVIA leads TUNI-Base by 4.91 mIoU points on CART and M²D-LIF by 4.93 and 1.81 mAP points on KUSTDrone and ATR-UMOD. M²D-LIF leads on UAV-CB. On DroneRGBT, AnyVIA has higher MAE but lower RMSE than Free Lunch (13.40 / 15.89 vs. 12.63 / 16.13). The new protocol table clarifies which components are trained downstream and that TUNI-Base and Free Lunch use released pretrained weights, while M²D-LIF starts from random initialization (Appendix C.11, Tables 21–22).
  • Complete-model measurements quantify the cost of higher detection accuracy. Guidance is computed once per image and modality and shared across channel groups. With frozen DINOv2-B, 2× adds 6.9%–11.7% FLOPs over Base. Detection scores at 1× stay within 0.35 mAP points of Base. At 2×, they improve over 1× by at least 4.38 mAP points on all three detection benchmarks. On KUSTDrone, 2× requires 9.2% more FLOPs and adds about 1.1 ms of end-to-end latency relative to 1× (Section 4.7, Table 11, and Appendix C.5, Figure 7(d–f)).
  • Expanded reporting across seeds and encoder scales. Tables 1–5 and 9 now report means and sample standard deviations across three seeds. We added approximately 200 downstream training runs to complete this evaluation. The encoder-scale comparison now covers all eight benchmarks with each of three DINOv2 encoders. AnyVIA improves over Base in 19 of 24 settings, including all eight with DINOv2-B (Appendix C.1, Table 13).

Dear reviewers,

The authors have submitted their response. Please carefully check whether the new content has addressed your concerns, particularly the additional benchmark comparisons and component ablation studies. If not, please follow up with the authors ASAP so that the authors can further respond before the discussion period ends (Nov 18).

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.