acceptodds
Under review as a conference paper at ICLR 2027

SonarLLM: A Native Sonar–Optical Multimodal Large Language Model for Underwater Perception

Abstract

Reliable underwater perception requires complementary sensing under variable visibility. Optical cameras capture appearance and semantics but degrade rapidly with turbidity, whereas imaging sonar preserves geometry while exhibiting distinct range–azimuth structure and acoustic artifacts. Existing MLLMs, built primarily on optical encoders, are therefore ill-suited to model sonar or adaptively exploit sonar–optical complementarity. We propose SonarLLM, a sonar–optical MLLM that treats sonar as a native perceptual modality. It combines a sonar-specific encoder, modality-specific physics-aware feature enhancement, and reliability-aware hierarchical fusion to align acoustic structure with optical semantics and dynamically adjust their contributions as sensing quality changes. We also introduce SonarBench, a paired benchmark that spans four tasks—recognition, counting, visual question answering, and captioning—and, across the benchmark, three input settings: sonar-only, optical-only, and fusion. By fixing the scene and sonar observation while varying optical degradation, SonarBench enables controlled measurement of cross-modal complementarity. SonarLLM achieves 72.0% macro accuracy across sonar-only recognition, counting, and VQA, outperforming the strongest baseline by 34.4 percentage points, and 68.7% under fusion, exceeding the best baseline by 25.1 points. For recognition and counting, the fusion-over-optical gain grows from 6.0 to 36.0 points as turbidity increases, indicating the increasing complementary value of sonar under controlled optical degradation. Together, these results show that robust heterogeneous perception depends not only on adding sonar, but on representing and weighting it according to its sensing characteristics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Model, Underwater Perception and Understanding, Sonar-Optical

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kcTMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SonarLLM adapts a multimodal language model to underwater sonar and optical inputs using a dedicated sonar encoder, modality-specific feature enhancement, and reliability-supervised fusion. Its staged training combines sonar representation learning, semantic alignment, and instruction tuning. The paper evaluates sonar and fusion tasks and uses controlled optical degradation to analyze the fusion gate.

Critical strengths
  1. The dedicated acoustic pathway addresses sonar-specific processing rather than treating sonar as an ordinary RGB image. The staged adaptation and modality-specific processing form a coherent approach to an important underwater perception problem.
  2. The paired degradation evaluation keeps the scene and sonar input fixed while changing the optical input. Together with gate and feature ablations, this is useful evidence about the intended fusion behavior.
Critical weaknesses
  1. The interaction between range compensation and the following normalization remains unclear. Equation (4) applies a positive scalar gain to each token and then channel-wise LayerNorm; the supplied acoustic implementation follows that order. Apart from the normalization epsilon, LayerNorm cancels such scaling, so the claimed attenuation compensation needs a different mechanism or direct evidence that it survives this operation.
  2. Reliability supervision and the main turbidity stress test use the same synthetic degradation generator. As the paper acknowledges, these results establish adaptation within that controlled family. This limits the demonstrated scope of the fusion benefit; broader reliability under natural turbidity or unmodeled sensor failures remains open.
Questions for authors
  1. What range-dependent information remains after LayerNorm, and what happens when only the range-gain component is removed?
  2. Can the gate retain its advantage on naturally degraded observations or a degradation generator excluded from training?
3 comments
AuthorsOpenReview ↗

Response to Reviewer kcTM(Q1)

Thank you for this valuable question. After layer normalization, the distance-dependent information retained includes the relative contrast of echoes, speckle statistics, echo waveform morphology, multipath and sidelobe structures, as well as range-varying noise distributions. These are relative structural features that do not rely on the absolute amplitude of the signal.

If we remove the range-gain component only, only the constant-scale contribution from this saturated gain term is discarded, while all distance-related structural features remain intact. We sincerely appreciate this comment, which helps us refine our presentation. We will clarify this mechanism in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer kcTM(Q2)

Thank you for your question. The gate maintains its advantage under naturally degraded observations or when the degradation generator is excluded from training.

Specifically, the test set used in our paper is constructed from held-out real sonar-optical acquisition sequences from RGBS50 and UMOD:

Sec. 4.1: "Evaluation samples come from held-out RGBS50 and UMOD sequences (25 subsets, 150 QA instances each; zero training overlap confirmed by a sequence–frame audit, Appendix C)"

From the perspective of the entire test set, different samples preserve the diverse targets, backgrounds, and sonar-optical discrepancies from various real collected scenes. Therefore, it enables a valid evaluation of the model's degradation adaptation capability on "naturally degraded observations".

Finally, we sincerely thank you for this question, and we will further refine the description in the revised manuscript to make this point clearer.

Dear Reviewer kcTM,

Thank you again for your valuable comments on our submission. We would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them).

Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know.

If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_Xf25OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SonarLLM studies underwater language understanding using optical images and imaging sonar. It extends Qwen3-VL-8B with a dedicated sonar encoder that incorporates acoustic geometry. Separate feature enhancement modules address modality-specific degradation. Adaptive gating and hierarchical feature injection connect both streams to a shared language model. A staged training procedure combines sonar representation learning with cross-modal alignment and instruction tuning. The paper also introduces SonarBench for recognition and counting alongside VQA and captioning. Its paired evaluation holds scene content and sonar observations fixed while synthetically varying optical quality. The authors report improved sonar understanding and increasing fusion benefits as optical observations deteriorate.

Critical strengths
  1. The dedicated sonar pathway explicitly incorporates acoustic geometry into the visual representation. Table 2 reports substantial improvements across the main perception tasks. Table 3 also supports the value of the sonar stem and polar positional encoding. These findings provide a credible case for dedicated sonar adaptation within a language model.
  2. Holding scene content and sonar observations fixed provides a useful framework for studying sensor complementarity. Section 4.3 reports recognition benefits over the stronger individual sensor. The analysis also identifies tasks where fusion provides less benefit. This offers meaningful insight into when additional sensing contributes under controlled optical degradation.
Critical weaknesses
  1. AGFM scores features from both modalities, but its Stage-III gate target depends only on optical degradation severity. The target does not explicitly reflect sonar quality. Evaluation also uses the training degradation family. The authors correctly describe the gate as a supervised degradation proxy in Sections 3.5 and 4.3. The demonstrated contribution is therefore adaptation to a known optical reliability shift. Reliability estimation when acoustic quality also changes remains outside the established scope.
  2. Equation (4) multiplies each sonar token by a positive scalar range gain before LayerNorm. If normalization operates across each token’s channels, this gain cancels when the epsilon term is ignored. This is a conditional concern about the stated formulation rather than a verified implementation failure. The reported Acoustic-VFE ablation supports the complete module but does not establish the contribution of its range gain. The claimed attenuation-compensation mechanism therefore remains insufficiently substantiated.
  3. The LoRA control in Section 4.1 matches instruction tuning but does not match the preceding sonar adaptation and alignment stages. It appropriately tests whether instruction tuning alone reproduces the gains. However the overall comparison also includes the effects of additional domain training. Section 4.4 provides useful component and adaptation ablations. These offer partial separation but do not fully attribute the headline improvement to the architecture independently of the training curriculum.
  4. Hierarchical alignment in Section 3.5 operates on pooled representations. Figure 4 supports improved correspondence after the combined alignment stage. It does not isolate how the hierarchical alignment term contributes to complementary fusion beyond the other objectives in that stage. The independent token streams can still retain modality-specific information. The remaining concern is therefore the limited explanation of hierarchical alignment’s distinct contribution rather than evidence that alignment removes complementary information.
Questions for authors

None.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Xf25(Q1)

Thank you for this thoughtful analysis. In our bimodal optical-acoustic fusion setting, the two sensors are inherently complementary, as stated in Section 1: “The two sensors are thus complementary---appearance and semantics on one side, stable geometry on the other.” Motivated by this complementarity, we design the gating mechanism to modulate one modality using control signals derived from the other, such that the model can adaptively adjust the fusion strength between modalities. Our experimental results verify the effectiveness of this design.

Replacing AGFM by equal weighting at inference, the accuracy advantage is 0.0/0.2/2.7 points under clear/turbid/heavy conditions, and clear-to-heavy degradation is 0.7 points with AGFM versus 3.4 with equal weighting. (a) Sonar weight increases with optical degradation. (b) Under the same checkpoint, AGFM's advantage over equal weighting grows from 0.0 to 2.7 points. We next probe AGFM's internal response on 150 independent RGBS50 scenes at five degradation levels (750 observations). Mean sonar weight rises monotonically with severity (Fig. 3(a): 0.491 to 0.795; Spearman ρ=0.749; 95.3% of scenes non-decreasing). Because Stage III explicitly supervises the gate with degradation strength, we read this as verification of the intended internal response rather than emergent reliability estimation. Quality-aware weighting matters most for robustness: Mean Fusion and w/o gate loss reduce static fusion accuracy by only 1.3/0.9 points but increase the clear-to-heavy decline from 1.3 to 3.8/3.0 points, supporting AGFM---with the same-checkpoint comparison in Fig. 3---as a quality-aware adaptation mechanism.

It is worth noting that this work represents the first in our series of studies on this topic. We are currently investigating a follow-up scheme that jointly estimates and fuses the reliability of both optical and sonar modalities, and we will present these more comprehensive results in future publications.

We appreciate your valuable question. We will refine the relevant descriptions in the revised manuscript to clarify this point further.

AuthorsOpenReview ↗

Response to Reviewer Xf25(Q2)

Thank you for this careful examination of Equation (4). We agree with your judgment that this is a conditional concern regarding the formulation, rather than an implementation error.

We would like to clarify that the attenuation-compensation mechanism corresponds to Equation (4) as a whole. The scalar range gain is only a minor sub-component within this full mechanism, and the main compensation effect arises from its other components. As you observed, this positive scalar gain applied before token-wise channel LayerNorm is largely canceled by normalization when the ϵ\epsilon term is neglected. This aligns with our design intention: we do not rely on this scaling term to serve as the core of attenuation compensation.

Equation (4) consists of two decoupled correction branches. The reverberation suppression branch Fsc[i]=Fs[i]−σ(gr)WsCsF_s^c[i] = F_s[i] - \sigma(g_r)W_s C_s performs cross-channel learnable subtractive correction. The correction offsets produced by projection matrix WsW_s vary across channels, so this branch is immune to the amplitude cancellation effect of LayerNorm and acts as the primary contributor to attenuation compensation.

The questioned range gain G(ri)=(ri/rmin)2⋅e2αc(ri−rmin)G(r_i) = (r_i/r_{\text{min}})^2\cdot e^{2\alpha_c(r_i-r_{\text{min}})} is derived as the analytical inverse of the sonar attenuation term (1/r2)e−2αphy⋅r(1/r^2)e^{-2\alpha_{\text{phy}}\cdot r} provided in Appendix A. Here, αc\alpha_c is a learnable coefficient deliberately distinguished from the physical attenuation coefficient αphy\alpha_{\text{phy}} (Sec. 3.3). Its role is to inject a distance-aware physical prior into the optimization landscape, rather than preserving feature amplitude after normalization.

To support this explanation, we have added a dedicated ablation experiment: removing this range gain only introduces marginal constant-scale differences, confirming that its individual impact is limited. Accordingly, the overall attenuation-compensation mechanism remains valid and sufficient even without this term.

In the revised manuscript, we will rewrite the description of Equation (4) to clearly differentiate the roles of the two branches, explicitly state that the range gain serves merely as an auxiliary optimization prior, and define the boundary of the attenuation-compensation mechanism to resolve this ambiguity.

AuthorsOpenReview ↗

Response to Reviewer Xf25(Q3)

We appreciate the reviewer’s careful evaluation. While the present ablation studies cannot fully disentangle the performance gains attributable to the architecture and the training curriculum, all ablation variants and the paired LoRA baseline in Table 3 are trained under the same fixed four-stage progressive training curriculum.

The newly introduced architectural components contain 882.4 million parameters, together with the 349.2 million-parameter LoRA adapter, yielding 1.2316 billion tunable parameters in total, which accounts for 12% of the overall model size (10.3 billion parameters). Most of these parameters are inherited from pre-trained weights: the 736.9 million-parameter sonar tower is initialized with pre-trained weights, and the DINOv2-L prior encoder remains frozen throughout training. Only 25.1 million parameters correspond to randomly initialized, non-LoRA fusion modules optimized from scratch, constituting 2.0% of the newly added tunable parameters and 0.24% of the global total parameters. Breakdown: AGFM has 3.9 million parameters (0.32% of new tunable parameters), and Acoustic-VFE has 6.65 million parameters (0.54% of new tunable parameters); together they make up 0.86% of the newly added tunable parameters.

Under identical experimental conditions, removing any proposed module leads to performance degradation for the corresponding modality or fusion metric (Table 3):

  • Removing polar coordinate encoding: fusion metric drops by 13.5 (82.1 → 68.6), and the performance gap between clean and highly shifted samples widens from 1.3 to 7.1;
  • Removing the sonar stem: fusion metric drops by 8.2 (73.9), and the sonar macro-average metric drops by 7.5 (58.3);
  • Removing the DeepStack multi-layer injection structure: sonar metric drops by 6.2 (59.6), and fusion metric drops by 3.2;
  • Removing gating supervision: both fusion and sonar metrics drop by 0.9, and the performance gap widens from 1.3 to 3.0;
  • Removing Acoustic-VFE: sonar metric drops by 3.8 (62.0), while the vision metric only experiences a minor drop of 0.4.

We further construct a baseline to isolate the contribution of the training curriculum. Using Qwen3-VL-8B, we perform LoRA fine-tuning on the same 635k instruction data without enabling the sonar pathway. Under this setting, the sonar accuracy reaches 37.5% (zero-shot: 32.1%), and the fusion accuracy is 32.3% (zero-shot: 41.2%). The sonar metric rises from 32.1 to 72.0, with an overall gain of 39.9 points. Among this total gain, only 5.4 points (13.5%) come from data adaptation, and the remaining 34.4 points (86.5%) are contributed by our proposed architecture.

In summary, under the training configuration adopted in this work, the core driver of performance improvement is the newly proposed architecture rather than the training curriculum.

Finally, we thank you for this valuable question. We will improve the presentation in the revised manuscript to elaborate on this issue more clearly.

Reviewer_yhZcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SonarLLM treats imaging sonar as a native modality in a sonar–optical MLLM. It combines a sonar-specific encoder with range–azimuth positional encoding, modality-specific feature enhancement, reliability-aware gated fusion, dual-stream DeepStack injection, and a four-stage curriculum. It also introduces SonarBench, a paired benchmark that fixes scene, question, and sonar observation while degrading only the optical image. Reported results show 72.0% sonar-only and 68.7% fusion macro accuracy, with fusion-over-optical gains growing from 6.0 to 36.0 points as turbidity increases.

Critical strengths
  1. The problem framing is strong and well motivated: sonar and optics fail under different conditions, so treating sonar as an auxiliary RGB image is inadequate. The paper makes a convincing case for sonar-native representation.
  2. SonarBench’s controlled intervention is a useful evaluation idea. Holding the scene and sonar fixed while degrading optics isolates complementarity better than standard paired-sensor benchmarks.
  3. The empirical gains are large and consistent across recognition, counting, VQA, and captioning. The fact that much larger general MLLMs do not close the sonar-only gap supports the value of domain-native representation.
Critical weaknesses

1.The “reliability-aware” claim is weaker than stated. The gate is supervised toward a known degradation parameter eta, so AGFM learns a degradation proxy rather than an unsupervised reliability estimate. At inference no eta is available, and real turbidity is not a single scalar. 2.AGFM’s practical benefit is modest. Under the same checkpoint, its advantage over equal weighting is only 0.0/0.2/2.7 points under clear/turbid/heavy conditions. The main robustness benefit is reduced degradation sensitivity, not large static gains. 3.Evaluation is controlled but narrow: 25 subsets with 150 instances each, using the same degradation generator as training. This may overstate generalization to natural scattering, dynamic particles, sonar-specific failures, and temporal misalignment. 4.The “scale is not the missing ingredient” claim is somewhat overstated. Larger baselines lack sonar-native pretraining, so the comparison shows that scale alone does not substitute for domain representation—not that scale is generally unimportant. 5.LLM-judge evaluation raises concerns, especially using Qwen3.8-Max while the backbone is Qwen3-VL. The Kimi-K3 agreement check on 300 outputs helps, but the judged set is small and does not fully rule out same-family bias. 6.Method complexity and cost are significant: four training stages, paired data for alignment, added parameters, 78.1% higher prefill latency, and 45.1% lower decoding throughput. Deployment value needs clearer justification. 7.The ablation split uses rule-based scoring, while the main results use a semantic judge. This is reasonable for diagnostics, but it makes absolute ablation values hard to compare with main-table claims.

Questions for authors

1.How well does AGFM behave when eta is unavailable and optical degradation is not drawn from the training generator? Can you show gate behavior on naturally turbid or out-of-generator pairs? 2.What is the exact fairness protocol for larger baselines? Do they receive the same prompts, image resolution, modality order, and instruction data, and are they evaluated with and without LoRA controls? 3.Can you report AGFM’s contribution when the gate is trained without direct eta supervision, e.g., using only contrastive or entropy objectives? This would test whether reliability is learned rather than supervised. 4.How many evaluation instances support the captioning GOOD rate, and what is inter-judge agreement for captioning specifically? 5.Does the sequence-frame audit exclude near-duplicate frames from the same dive/sequence across training and evaluation, not just exact frames? 6.How sensitive are the conclusions to the Polar PE and Sonar Stem ablations being retrained front-end variants rather than strict inference-time removals?

7 comments
AuthorsOpenReview ↗

Response to Reviewer yhZc (Q1)

Response to Reviewer yhZc (Q1)

Thank you for this valuable question. We address the two sub-questions as follows:

1. Regarding the first sub-question

First, AGFM does not require η\eta during inference. η\eta is only adopted in the training stage to construct the gate supervision signal. At inference time, the model directly predicts fusion weights from the input sonar and optical features, as detailed in Sec. 3.4 and Sec. 3.5.

Sec. 3.4: "AGFM reweights rather than merges the two modality sequences: the reweighted optical and sonar tokens form the initial multimodal context at the final level" Sec. 3.5: "BCE gate supervision toward πs(η)=clamp(0.5+0.4η,0,1)…\pi_s(\eta) = \text{clamp}(0.5 + 0.4\eta, 0, 1)\dots making gsg_s a supervised degradation proxy rather than an emergent reliability estimate"

Second, our test set already satisfies your condition that optical degradation is not drawn from the training generator. Our evaluation set is built from held-out real sonar-optical sequences from RGBS50 and UMOD:

Sec. 4.1: "Evaluation samples come from held-out RGBS50 and UMOD sequences (25 subsets, 150 QA instances each; zero training overlap confirmed by a sequence–frame audit, Appendix C)"

While the turbid condition of individual samples is processed via the generator, the full test set preserves diverse targets, backgrounds, and sonar-optical discrepancies from real collected scenes, which enables valid evaluation of degradation generalization under natural conditions.

2. Regarding the second sub-question

The gate behavior under real and controlled optical degradation is analyzed in Sec. 4.3 and Fig. 3. Again, our held-out RGBS50 & UMOD test set supports this evaluation. Across 150 independent RGBS50 scenes with 5 degradation levels (750 observations in total), the mean sonar weight rises from 0.491 to 0.795, and 95.3% of scenes show a non-decreasing trend:

Sec. 4.3: "Mean sonar weight rises monotonically with severity (Fig. 3(a): 0.491 to 0.795; Spearman ρ=0.749\rho = 0.749; 95.3% of scenes non-decreasing)"

A fixed-checkpoint ablation replacing AGFM with equal-weight fusion shows a 2.7-point accuracy drop under heavy degradation; the performance gap (clear → heavy) expands from 0.7 to 3.4 points. The repeated-training ablation in Table 3 further confirms this pattern: full AGFM only suffers a 1.3-point drop, versus 3.8 points for equal-weight fusion.

Key conclusion: The gate dynamically adjusts the contribution of sonar and optical modalities under real-world optical quality variation, and this adaptive weighting improves fusion robustness.

Thanks again for this question. We will improve the corresponding description in the paper.

AuthorsOpenReview ↗

Response to Reviewer yhZc (Q2)

Thank you very much for this question. We have designed a rigorous fairness protocol to guarantee fair comparisons across all models (as in Sec. 4).

Sec. 4: "Unless otherwise specified, all models share task prompts, input protocols, and deterministic decoding"

More specifically, for all larger baselines, we adopt identical task prompts, decoding strategies, modality order and modality labels for multimodal inputs across all comparison experiments. Second, the modality order and labels for fusion inputs are fixed. All baselines receive paired sonar-optical inputs via their native multi-image interface (as in Sec. 4.1):

Sec. 4.1: "For fusion, all baselines use their native multi-image interface with fixed modality order and labels."

This fairness protocol ensures that comparisons between larger baselines and our method are fair by design.

Regarding the LoRA versus non-LoRA settings, we have strictly evaluated both in our paper, using the same Qwen3-VL-8B backbone as the control group (as in Sec. 4.1):

Sec. 4.1: "Qwen3-VL-8B+LoRA uses the same Stage-IV data and LoRA configuration as SonarLLM, testing whether instruction tuning alone explains the gains."

Both settings are evaluated and reported in Table 2 of our main paper. The results are reproduced below:

ModelSonar MacroOptical MacroFusion Macro
Qwen3-VL-8B32.137.441.2
Qwen3-VL-8B+LoRA37.536.932.3
SonarLLM72.050.968.7

Thanks again for this question. We will improve the description in the paper accordingly.

AuthorsOpenReview ↗

Response to Reviewer yhZc (Q3)

Thank you for your question. For your first question, since the gate loss is the only supervision term in Equation (9) of Appendix A that uses η\eta, the w/o gate loss ablation in Table 3 of Sec. 4.4 already provides the exact setting you asked for, i.e., "without direct eta supervision (the gate then receives only contrastive, hierarchical-alignment, and entropy objectives)". The results from this table are shown below:

VariantFusion MacroClear-to-heavy fusion drop
Full model82.1%1.3 points
w/o gate loss81.2%3.0 points
Mean Fusion, retrained independently80.8%3.8 points

After removing the gate loss, the static fusion macro-average only drops slightly from 82.1% to 81.2% (−0.9 points), while the clear-to-heavy performance decline expands from 1.3 to 3.0 points. For reference, the independently retrained Mean Fusion under the same setting achieves 80.8% with a 3.8-point drop.

Sec. 4.4: "Quality-aware weighting matters most for robustness: Mean Fusion and w/o gate loss reduce static fusion accuracy by only 1.3/0.9 points but increase the clear-to-heavy decline from 1.3 to 3.8/3.0 points, supporting AGFM---with the same-checkpoint comparison in Fig. 3---as a quality-aware adaptation mechanism."

For your second question, the reliability signal is primarily learned intrinsically, while the η\eta supervision further strengthens it. As demonstrated by Table 3 in Sec. 4.4 cited above, the gate remains functional even without η\eta supervision. In other words, the gate can learn roughly reasonable modality weighting from contrastive/entropy objectives and task losses on its own, and the benefits brought by η\eta supervision concentrate on stability under heavy degradation.

key conclusion: The reliability weighting of AGFM is naturally learned from training objectives and the gate architecture, and η\eta supervision further improves robustness under severe optical degradation.

Thanks again for this question. We will improve the description in the paper accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.