acceptodds
Under review as a conference paper at ICLR 2027

Beyond Source-Domain Accuracy: Controlled Study on Adaptation Granularity for Imaging Sonar Representation Transfer

Abstract

Transferring pre-trained vision foundation models to imaging sonar tasks has become a prevalent solution for underwater perception with limited data. Existing studies generally follow the empirical assumption that higher source-domain accuracy yields better cross-domain transfer performance, and thus focus primarily on improving source-domain capabilities. However, this common belief lacks systematic controlled validation, and how adaptation granularity governs sonar transfer performance remains unclear. To address this gap, this paper constructs a controlled experimental framework based on a 576M-parameter visual encoder. We systematically evaluate six feature adaptation schemes covering token-level scalar adjustment, channel-level transformation, and global affine mapping, from both source-domain classification accuracy and sonar cross-domain transfer perspectives. Experimental results show that fine-grained adaptation with stronger representation capacity improves source-domain accuracy yet degrades sonar transfer performance in the focal comparison. In contrast, coarse-grained scalar adaptation achieves a transfer accuracy of 96.5%, outperforming source-domain fine-tuning (95.1%) with stable and reproducible gains. A matrix of three backbones and three source domains, all evaluated on the same sonar target, further shows that the preferred granularity changes with the setting. Ablation studies verify that such benefits arise from content-dependent adaptation rather than simple feature aggregation with frozen encoders. Under identical training budgets, comprehensive comparisons with sonar masked autoencoders, LoRA, and drift-anchored fine-tuning further clarify the applicability of our findings. This paper demonstrates that adaptation granularity is a long-overlooked critical dimension in sonar cross-modal transfer. Relying solely on source-domain accuracy leads to misleading evaluation of adaptation strategies. Our controlled experiments clearly characterize the potential and limitations of vision-to-sonar transfer, providing practical guidance for underwater cross-modal perception adaptation design.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Cross-modal Representation Transfer, Imaging Sonar, Adaptation Granularity, Vision Foundation Models, Underwater Perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ha4ROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This controlled empirical study challenges the assumption that higher source-domain accuracy ensures superior cross-domain transfer when adapting vision foundation models to imaging sonar. The authors evaluate feature-adaptation granularities from token-level scalar scaling to channel-wise transformations. Experiments reveal that fine-grained adaptations improve source accuracy but degrade sonar transfer performance compared to coarse scalar corrections. Further evaluations across multiple backbones and source domains demonstrate that preferred granularities depend heavily on the setting and originate from joint backbone-correction optimization rather than isolated output constraints.

Critical strengths

The paper critically challenges the widespread empirical assumption in cross-modal transfer that higher source-domain accuracy inevitably translates into superior downstream performance. By designing a bounded feature-correction framework that rigorously controls the gain range, initialization, and training protocol, the study provides counter-intuitive and valuable empirical evidence on how adaptation granularity governs representation transfer to imaging sonar.

The empirical evaluation is executed with thoroughness across a matrix of three backbones and three source domains, accompanied by carefully designed geometry and regularization analyses. Furthermore, the authors maintain high academic rigor by strictly distinguishing between probe-seed variance and training-seed uncertainty, refraining from overclaiming the universality of scalar adaptation, and transparently detailing the boundaries of their conclusions.

Critical weaknesses

While the study evaluates massive visual encoders such as the 576M-parameter Qwen backbone, the downstream sonar target task contains only 645 test images in total. On such a tiny test set, a prediction change on a single image alters the accuracy by approximately 0.155 percentage points. Given that many observed margins between adaptation granularities are narrow, the statistical power is inherently weak, making it difficult to distinguish genuine representation benefits from sampling noise or random test-set fluctuations.

Despite constructing a multi-backbone and multi-source matrix, all downstream evaluations ultimately collapse onto this single, constrained 12-class turntable sonar dataset. Without cross-validation on larger-scale, multi-view, or in-the-wild sonar benchmarks

Questions for authors

see the Critical Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ha4R(Q1)

Thank you for this comment. We have supplementary evaluation results on T1 (2,340 test images) and T2 (3,105 test images), which were completed before submission but omitted from the main manuscript due to space constraints. Specifically, scalar correction outperforms channel correction on T1, while the difference on T2 is inconclusive.

For the original target split, we report paired effect estimates and confidence intervals, properly accounting for repeated predictions on the same test units instead of treating them as independent observations. Since this original benchmark was used during exploratory development, its results serve as developmental evidence rather than an independent confirmatory assessment. We use resampling units that respect underlying acquisition dependencies and clearly separate robust supported effects from unresolved differences in the revised manuscript.

Table R8b. Principal comparisons on the original target with five paired encoder seeds (mean ± SD, LP %).

ConfigurationSource LPTarget LPHead params
Plain FT77.7 ± 0.494.4 ± 0.7—
Scalar (G=1, h=64)78.5 ± 0.496.1 ± 0.574K
Grouped (G=32, h=64)78.8 ± 0.394.5 ± 0.776K
Channel (G=1152, h=64)78.7 ± 0.395.4 ± 0.5149K
Wide scalar (h=128)78.3 ± 0.595.1 ± 0.8148K

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these results and discussions in the revised version.

AuthorsOpenReview ↗

Response to Reviewer Ha4R(Q2)

Thank you for this comment. We address this concern with two additional independent target benchmarks: T1 and T2. T1 comprises independent data collected with a different 600 kHz imaging sonar in a separate acquisition campaign, with object-disjoint splits. T2 is a fully in-the-wild, multi-view free-swimming survey collected using a 455 kHz sonar under uncontrolled backgrounds. Both benchmarks retain the same 12 debris classes, which enables direct comparison of adaptation granularity, with independent objects as listed in Table R5a. We evaluate the same source-trained encoders (Plain / Scalar / Grouped / Channel) under fixed probe protocols: no target fine-tuning, and no method redesign informed by test data. All configurations were frozen before final evaluation. Two key patterns are consistently observed: the scalar correction maintains a clear advantage over plain fine-tuning on both targets (+2.6 [1.8, 3.4] on T1; +2.3 [1.5, 3.1] on T2); meanwhile, the optimal adaptation granularity shifts across target environments. On T1, scalar correction outperforms channel-wise correction (+0.9 [0.3, 1.5]), while the difference is inconclusive on T2 (−0.3 [−0.9, 0.3]). This confirms that the configuration preferred on the source domain does not always transfer best to a new target. These two benchmarks extend our evaluation far beyond the original constrained turntable setting.

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these results and discussions in the revised version.

Thank you for your rebuttal. However, I do not have enough experience to judge the quality of this work, so I have decided to maintain my original score.

AuthorsOpenReview ↗

Response to Reviwer Ha4R

We sincerely appreciate your careful review and valuable comments. We are also grateful for the time and effort you devoted to evaluating our work.We fully understand your position. If you have any further questions or clarifications that we can provide to assist you or the area chair in the evaluation, please feel free to let us know.

Reviewer_9h2UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the granularity of jointly trained feature correction affects the usability of the adapted representation on a new target task, and whether there is a ranking inconsistency between source-domain performance and target transfer performance. The authors construct a bounded feature correction framework that compares scalar, grouped, and channel-wise correction by changing the channel-sharing scheme, and also examine affine variants. During source-domain adaptation, the backbone, the correction branch, and the classifier are trained jointly; for target evaluation, the representation is frozen and a new linear probe is trained.

The paper compares three visual backbones and three source domains, with all settings transferring to the same sonar target task. In the focal configuration, finer-grained correction obtains higher source-domain scores but not better target transfer results; the preferred granularity also changes across backbones and source domains. The authors further provide analyses of input composition, boundedness, correction head capacity, feature direction, regularization, and correction learning after freezing the backbone.

Critical strengths
  1. The paper provides useful empirical observations on the relationship between feature correction granularity and transfer performance. In the focal configuration, finer-grained correction attains higher source-domain scores without yielding better target transfer performance; comparisons across backbones and source domains further show that the relative ranking of scalar and channel-wise correction changes. These results indicate that the expressive capacity of the correction branch and the source-domain performance are insufficient to directly predict its transfer value, offering useful evidence for the design and evaluation of such adaptation modules.

  2. The authors construct scalar, grouped, and channel-wise variants by changing the channel-sharing scheme while keeping the main input processing and training conditions consistent. Compared with directly comparing adaptation methods that differ in both structure and training procedure, this design is better suited to examining the effect of correction granularity; the capacity comparisons and component ablations also help bound the interpretation of the observations.

Critical weaknesses
  1. The comparison over three backbones and three source domains enriches the variation on the source and model sides, but all results ultimately come from a single sonar target classification task. These experiments can therefore establish a conditional ranking under that target protocol, but make it difficult to judge how the observed granularity preference relates to the sonar task, the acquisition environment, the target label structure, or more general representation-transfer properties.

  2. Output-magnitude and direction regularization do not recover the scalar branch’s results, and correction learning after freezing the backbone also fails to reach the joint-training pipeline’s performance. These experiments are valuable and show that the tested alternative implementations do not reproduce the scalar correction gains under the reported configurations. However, regularization simultaneously changes the optimization objective and the available transformations, and training after freezing the backbone follows a different learning path than joint training. These negative results therefore cannot yet uniquely determine which aspect of joint optimization produces the advantage, nor sufficiently distinguish whether the backbone learns a better representation, the correction branch improves the final readout.

Questions for authors
  1. The current cross-backbone and cross-source-domain experiments all share the same target task. Are there target evaluation results under different acquisition environments, sensors, or class structures that can check whether the source–target ranking inconsistency and the granularity preference shift still occur?

  2. Correction learning after freezing the backbone fails to reproduce the full-pipeline results, but this setting also follows a different learning path than joint training. Is there a more direct analysis that can separate the improvement of the backbone representation from the effect of the correction readout?

  3. How should one explain the failure of the wider scalar head to retain the original advantage? The wider scalar head approaches the channel head in parameter count, but its target accuracy drops to about 95.1%, versus about 96.5% for the original scalar head. This comparison is valuable and suggests that performance is not determined solely by the channel-sharing granularity. Can the authors further explain how this result affects the interpretation of the role of the scalar constraint?

  4. The paper discloses that the same target benchmark informed the exploratory analysis. Please specify which design choices were informed by the target training set, validation set, or test set. In particular, please distinguish selecting the linear-probe training epoch from selecting correction architectures, regularizers, or other adaptation configurations. Was any evaluation set reserved for final assessment without informing further design changes?

  5. For the core scalar-versus-channel comparison, is there a limited, budget-matched sensitivity analysis of key hyperparameters, such as the backbone and correction-head learning rates? With the data, update budget, and selection rule held consistent, do reasonable changes to these settings preserve the main ranking?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 9h2U(Q1)

Thank you for this comment. We evaluated two target sets acquired with different sonar sensors and environments: T1, collected with a 600 kHz imaging sonar in a separate acquisition campaign, and T2, collected with a 455 kHz sonar under uncontrolled backgrounds. Both retain the original 12 classes and use object-disjoint splits, extending evaluation across sensors and environments while keeping the class structure unchanged. Evaluations on these two datasets were completed in our prior study but were omitted from the main manuscript due to page limitations. All results in Table R5a are based on the ViT-L backbone. With source-trained encoders fixed and matched frozen-feature probes, the source-selected configuration does not consistently achieve the highest target accuracy (Table R5a). Scalar correction outperforms channel-wise correction on T1 by 0.9 percentage points (95% CI [0.3, 1.5]). On T2, channel-wise correction has a slightly higher mean, but the difference is inconclusive (scalar−channel: −0.3 points, 95% CI [−0.9, 0.3]). Thus, the evaluations support source–target ranking inconsistency, while evidence for a target-dependent granularity reversal remains limited. These targets were not used for method development, and evaluation settings were fixed before final testing. We will include these results and clarify their scope in the revised manuscript.

Table R5a. New target benchmarks and frozen-configuration results (ViT-L backbone, five paired encoder seeds; mean ± SD, SD rounded to one decimal place, target LP %).

MetricT1 (HarborFUS)T2 (RiverSurvey)
Sensor / campaign600 kHz imaging sonar, separate acquisition campaign455 kHz free-swimming multi-view survey, uncontrolled background
Test images2,3403,105
Independent objects187 (31 sequences)96 (14 scenes)
Classes12 (identical to original)12 (identical to original)
Splitobject-disjointobject-disjoint
Plain FT90.2 ± 1.286.4 ± 1.4
Scalar92.8 ± 1.088.7 ± 1.2
Grouped (G=2)91.5 ± 1.387.3 ± 1.3
Channel91.9 ± 1.189.0 ± 1.2

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these results and discussions in the revised version.

AuthorsOpenReview ↗

Response to Reviewer 9h2U(Q2)

Thank you for this comment. We provide a matched-readout analysis to assess upstream representation quality and the contribution of the correction readout (Table R6). Across five paired encoder seeds, we extract three representations from each checkpoint: (A) pooled outputs before correction, (B) processed features with gain disabled, and (C) the full gain-weighted readout. A fresh linear probe is fitted to each representation using the identical protocol. Within each individual checkpoint, the trained parameters remain fixed.

The jointly trained scalar model outperforms the no-gain model at interface A by 1.0 percentage point (95% CI [0.4, 1.6]), suggesting improved upstream representations. At interface B, the difference is smaller and inconclusive (+0.4 points, 95% CI [−0.1, 0.9]). On the same scalar checkpoint, enabling the learned gain improves accuracy from 95.5% to 96.1% (+0.6 points, 95% CI [0.2, 1.0]).

These results suggest contributions from both upstream representation adaptation and the correction readout. We will add this analysis and clarify that frozen-backbone correction training alone cannot disentangle these two contributions.

Table R6. Matched correction-before/after readouts on the original target (five paired encoder seeds; identically refit probes; target LP %).

CheckpointA: pre-correction pooled outputB: frontend, gain disabledC: full gain-weighted readout
No-gain model94.095.1—
Scalar model95.095.596.1

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these results and discussions in the revised version.

AuthorsOpenReview ↗

Response to Reviewer 9h2U(Q3)

Thank you for this comment. The wide-scalar control shows that scalar channel sharing alone is insufficient to preserve the transfer advantage; the beneficial regularization induced by the scalar constraint is most effective at low predictor capacity. Across five paired encoder seeds, target accuracy is 95.8%, 96.1%, 95.5%, and 95.1% for scalar predictor widths of 32, 64, 96, and 128, respectively. The original compact head outperforms the near-parameter-matched wide head by 1.0 percentage point (95% CI [0.4, 1.6]), and this ranking is maintained across all tested learning rates. Despite similar training losses, the wider head produces more heterogeneous token-wise gains and larger perturbations to pooled features. These observations suggest that extra capacity weakens the implicit regularization of the scalar formulation and enables reweighting patterns that hurt transfer, although they do not establish the underlying cause. We therefore interpret the scalar constraint not merely as a structural choice of channel sharing, but as a regularizing restriction that works best within a compact predictor. It preserves each normalized token’s direction, yet a higher-capacity predictor can override this inductive bias and learn overly complex token-weighting patterns harmful for generalization. We will revise the discussion to distinguish channel-sharing granularity from predictor capacity, and clarify how the inductive bias of the scalar constraint interacts with model capacity.

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these results and discussions in the revised version.

Reviewer_NPpdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reports adaptation granularity through a bounded feature-correction family trained jointly with visual backbones. In the focal sonar setting, scalar correction yields the strongest target representation, and the complete pipeline improves over plain source fine-tuning; the scalar–channel ordering, however, changes across backbones and source domains. Geometry measurements, regularization experiments, and post-hoc aggregation on a frozen backbone show that this benefit cannot be reproduced by constraining or re-creating the branch’s outputs after the fact: it lies in the joint optimization of backbone and correction.

Critical strengths
  1. The topic is interesting and the study is well motivated.
  2. The paper and the proposed approaches are clearly presented. The results are promising.
  3. Good balance between the empirical results and the analyses.
Critical weaknesses
  1. Whilst the research is interesting and the paper demonstrates certain findings, the depth of the proposed research is a bit disappointing. The paper gave a lot spaces on discussing the collected observations with comprehensive justification. However, it is not clear how other correction granularity options affect the focal sonar result whilst demonstrating the value of the best correction granularity option. In other words, the current study is limited by the way it only covers very small portion of choices for correction granularity.
  2. The proposed study does not include the cases where sources are scarce, inaccurate and even poisonous. It is not clear if or not the proposed granularity options lead to the same outcomes as those reported in the paper. This extra testing is necessary as it helps understand the internal characteristics of the proposed change.
  3. The proposed study is experimented with several Lams with small numbers of parameters (86/300/576M). How about the performance of LLMs with large numbers of parameters (> 1B) using different granularity options? If the performance changes, can you discuss the possible causes with deeper investigation?
  4. Looking at the results shown in Table 3, more discussion is expected to explain the variations in the table, e.g. what elements in the LLM lead to such outcome? why so?
Questions for authors

Please address the above major concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NPpd(Q1)

Thanks for this question. Following the reviewer’s request, we have added the corresponding experiments. Specifically, we acknowledge that our original submission only evaluated a limited range of correction granularity choices, making it unclear how alternative granularity settings affect the sonar transfer results. To address this limitation and deepen the analysis, we performed a granularity sweep over G∈{1,2,4,8,16,32,64,128,384,1152}G\in\{1,2,4,8,16,32,64,128,384,1152\} with fixed head width. The affine extension is treated separately as an independent dimension of correction freedom.

Table R1 summarizes source validation and sonar transfer performance together with correction head size. The key findings are as follows. The scalar correction (G=1G=1) remains optimal for sonar transfer (+0.7 pp over channel-wise correction, +1.7 pp over plain fine-tuning), while grouped G=2G=2 performs best on the source domain. Notably, performance is not monotonic with GG: e.g., G=384G=384 underperforms its neighbours. Random grouping tests at G=8,128G=8,128 confirm that our conclusions are not artifacts of a specific partition.

We note that changing GG modifies both model capacity and the optimization landscape. Therefore, the revised claim in §4.2 is restrained: controlled relaxation of correction alters source learning and sonar transfer in reproducible ways, and we do not claim a universal optimum. For consistency, G∈{1,32,1152}G\in\{1,32,1152\} uses five encoder seeds and the remaining swept points use three seeds.

Table R1. Granularity sweep at fixed head width (h=64h=64): source LP, sonar target LP (%), correction head size. Affine extension: source LP 78.9%, target LP 95.6%.

G12481632641283841152
Source LP78.579.078.678.878.978.878.778.978.778.7
Target LP96.195.095.295.294.994.594.794.894.695.4
Head params (K)7411192837876757474149

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these experimental results and corresponding analyses in the revised version.

AuthorsOpenReview ↗

Response to Reviewer NPpd(Q2)

Thanks for this question. We conducted new experiments to investigate whether the proposed correction granularity maintains stable outcomes under poisoned source data, as suggested. We compared plain fine-tuning, scalar correction, grouped correction (G=32G=32), and channel-wise correction in this setting. All methods share identical subsets, corruption maps and training budgets, with 3–5 paired encoder seeds, and the reported values are the mean results.

Under the targeted poisoning setting, scalar correction achieves the lowest attack success rate and retains the best transfer performance. At the high poisoning ratio of 10%, the backdoor signal is strong, so most methods achieve high attack success rates. The trigger-only control confirms the performance drop arises from poisoned label association rather than the patch itself.

This aligns with our analysis in §4.3: compact scalar correction limits over-reliance on unreliable source information. We note this poisoning conclusion is only valid for our specific trigger-class threat model, and we do not claim general robustness against all source contamination.

We agree that the cases of scarce and inaccurate source data are also valuable for understanding the method. We will carry out these additional tests and supplement the corresponding results in the revised manuscript.

Table R2. Transfer accuracy (target LP, %) and attack success rate under targeted poisoning (16×16 trigger patch bound to the source class "plastic bottle"). ASR: source-domain attack success rate on the trigger test; transfer: target LP.

Metric0%1%5%10%
ASR, plain0.038.675.787.8
ASR, scalar0.017.542.659.7
ASR, grouped0.031.866.980.6
ASR, channel0.035.771.884.6
Transfer, plain94.494.093.592.8
Transfer, scalar96.195.995.594.8
Transfer, grouped94.594.293.793.0
Transfer, channel95.495.194.593.8

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these experimental results and corresponding analyses in the revised version.

AuthorsOpenReview ↗

Response to Reviewer NPpd(Q3)

Thanks for this question. This set of scale comparison experiments has been completed in our prior study, but we did not include them in the main manuscript due to page limitations. We adopt DINOv2 ViT-g/14 (1.1B) for this large-scale comparison, with its weight information listed in Table R3. To keep the comparison controlled, we compare DINOv2-L/14 and DINOv2-g/14 within the same model family. All experiments use identical source data, effective batch size, update budget, pooling and probe protocol, with five paired encoder seeds.

On the 1.1B model, the granularity preference remains inconsistent between source and target domains. Within the tested learning rate grid, the ordering across methods stays stable. We note that differences in pre-trained representations may modulate the benefit brought by per-channel correction freedom. Nevertheless, the models also differ in architecture and distillation trajectory, so we cannot attribute this behaviour solely to parameter scale. We agree that deeper investigation into the underlying causes is valuable. We will enrich the discussion and analysis in the revised manuscript if further investigation is needed.

Table R3. Scale comparison within the DINOv2 family (five paired encoder seeds; mean ± SD, SD rounded to one decimal place). Values shown are target LP results; source LP results are provided in the supplementary material.

ModelWeight SourcePlainScalarGroupedChannel
ViT-L/14 (300M)official DINOv2 weights93.2 ± 0.693.1 ± 0.593.0 ± 0.493.9 ± 0.6
ViT-g/14 (1.1B)official DINOv2 weights94.1 ± 0.693.9 ± 0.494.0 ± 0.594.6 ± 0.7

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript. We will add these experimental results and corresponding analyses in the revised version.

AuthorsOpenReview ↗

Response to Reviewer NPpd(Q4)

Thank you for this comment. Table 3 suggests that combining visual content with coordinates and bounding the token gains benefits transfer. These ablations concern the correction module acting on visual tokens, rather than individual backbone components such as attention heads.

Across five paired encoder seeds, full bounded correction outperforms the same-frontend no-gain control by +1.0 points (95% CI [0.4, 1.6]), feature-only conditioning by +0.8 points (95% CI [0.3, 1.3]), and the clipped-exponential variant by +1.5 points (95% CI [0.8, 2.2]). Coordinate-only conditioning performs worse, suggesting that positional information alone is insufficient. The clipped-exponential variant also produces more dispersed gains and larger backbone gradient norms, consistent with a benefit from controlled reweighting.

We additionally observe that increasing predictor width does not consistently improve transfer.

These observations support content-dependent, coordinate-conditioned, bounded reweighting within the jointly adapted pipeline, but do not establish a specific causal mechanism. We will add this interpretation and its limitations to the revised discussion.

Finally, we greatly appreciate your comments, which help us polish the details and descriptions of our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.