acceptodds
Under review as a conference paper at ICLR 2027

AnchorSLU: Cross-Lingual Semantic Anchoring for Low-Resource Speech Understanding

Abstract

Multilingual speech foundation models have advanced rapidly, yet large resource-dependent performance gaps persist when target-language supervision is limited. Standard cross-entropy (CE) provides only *class-level supervision*: semantically different utterances within the same class receive the same target, leaving exact sentence-level cross-lingual correspondence unused. We introduce ANCHORSLU, which converts this correspondence into *representation-level supervision*. For each target utterance, we synthesize sentence-matched reference speech in eight high-resource languages, encode these reference views offline with a frozen speech backbone, and aggregate selected-layer representations into sentence-specific semantic anchors. During adaptation, the model learns from natural target speech using target-only CE and multi-layer anchor alignment; inference requires only the adapted target model. Across 20 low-resource SIB-Fleurs languages, ANCHORSLU improves Qwen2.5-Omni-7B over matched target-only CE by **4.39** points in Accuracy and **5.04** points in Macro-F1. On 60-intent Speech-MASSIVE, it improves Accuracy by **3.19** points and Macro-F1 by **3.52** points under substantially richer target supervision. The benefit further extends to a substantially larger speech backbone. Controlled comparisons show that removing exact sentence correspondence largely eliminates the benefit even when class identity is preserved, while direct TTS augmentation and output-level distillation remain weaker. Together, these results establish exact sentence-level cross-lingual correspondence as an effective source of representation-level supervision beyond class labels, suggesting that sentence-matched multilingual supervision may also benefit broader speech understanding settings where such correspondence is available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech Understanding, Low-Resource Speech, Multilingual Speech, Cross-Lingual Transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JDaEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AnchorSLU, a way of leveraging cross-lingual representation to supervise the training of low resource speech models. The idea improves upon standard cross-entropy training which only provides class-level supervision, in a way such that semantically different utterances within the same class receive the same target. AnchorSLU on the other hand uses sentence-level correspondence. Specifically, they synthesize reference speech across high-resource languages for each sentence in the target language. Then they encode the references in a frozen speech model. Representations from selected layers are used to construct semantic anchors that are specific to each sentence. During training, an adaptor model learns from speech in the target language, using normal cross-entropy plus alignment to the anchors. The authors evaluated their method on 20 low-resource languages and show general improvement against conventional cross-entropy-based training. They also performed controlled studies which indicate that the exact sentence correspondence is the primary reason fro the performance gains.

Critical strengths
  1. the design is cost-efficient since all alignment-related work is done before deployment. the data creation, encoding and fitting is offline. so that only the adapted model is actually needed for inference.
  2. The evaluation is quite comprehensive, covering 20 languages across different scripts and 2 tasks. This shows the applicability of the method in different scenarios.
Critical weaknesses
  1. The controlled ablation studies are less convincing. For example, the difference between Projected SupCon and AnchorSLU is very small, within normal seed variance. The authors claim that "This comparison isolates the advantage of consolidating multilingual realizations of each sentence into a single semantic anchor, enabling direct target-to-anchor alignment without introducing cross-sentence negatives." which would be more convincing with repetition runs.
  2. It's not surprising that AnchorSLU improves upon target-only CE since the proposed method has seen more training data in the form of translated speech. A better baseline should still see the translated data during training, maybe as a text representation, ablating the TTS component. This would illustrate better that what's causing the improvement is the alignment method, instead of simply seeing more representations of the data.
Questions for authors
  1. From figure 5(b), it seems like over half of the improvement gain is already there when using one language as reference. Has the authors done further analysis into what the multilingual alignment module does?
  2. How were the layers chosen to get the representations from? What happens if only the final layer representation is used to create the anchors?
  3. The t-SNE plots are illustrative and they are easy to read, but can any quantitative metric be provided to show the alignment under AnchorSLU across different low-resource languages?
7 comments
AuthorsOpenReview ↗

Replies to Reviewer JDaE – Review 1

Summarized Review 1: Multi-Seed Evaluation of Projected SupCon

Thank you for your suggestion to further verify the stability of our experimental results. We agree that evaluating Projected SupCon and AnchorSLU across multiple random seeds would make the comparison more convincing. We therefore conducted paired experiments using three independent random seeds (41, 42, and 43) on the fixed 10-language SIB-Fleurs diagnostic panel. Both methods use the same multilingual reference speech, selected layers, projection space, target training data, optimization budget, and checkpoint-selection rule to ensure a fair comparison.

1. Multi-seed comparison

The results are shown below:

SeedProjected SupCon (Acc / F1)AnchorSLU (Acc / F1)ΔAcc / ΔF1
4158.87 / 54.3060.05 / 56.47+1.18 / +2.17
4259.64 / 55.2560.54 / 56.53+0.90 / +1.28
4359.34 / 55.4260.52 / 56.67+1.18 / +1.25
Mean ± SD59.28±0.39 / 54.99±0.6060.37±0.28 / 56.56±0.10+1.09 / +1.57

Results are percentages, with gains in percentage points. Mean ± SD denotes the mean and sample standard deviation across three runs. The Seed 42 results match Table 5.

AnchorSLU outperforms Projected SupCon in both Accuracy and Macro-F1 across all three seeds, with average gains of 1.09 and 1.57 percentage points, respectively. AnchorSLU also shows lower variation across seeds, particularly in Macro-F1, where its standard deviation is 0.10 compared with 0.60 for Projected SupCon. These results show that the performance advantage is not limited to a single run and follows the same positive trend across all three seeds.

2. Target-side optimization

AnchorSLU also requires fewer target-side optimization steps. As reported in Section 5.3 and Appendix D.5, AnchorSLU reaches its selected checkpoint in 212 steps on average, compared with 260 steps for Projected SupCon. The median stopping steps are 360 and 440, respectively, representing a reduction of approximately 18%. Thus, in our original diagnostic experiment, AnchorSLU achieves better classification performance with fewer target-side training steps.

Under the same reference information and representation preprocessing, AnchorSLU aggregates sentence-matched multilingual representations into fixed semantic anchors, allowing direct target-to-anchor alignment without cross-sentence negatives. The additional multi-seed results further support the effectiveness of this supervision strategy. In future work, we plan to extend the evaluation to more random seeds and language settings and further investigate the effects of anchor aggregation and negative samples.

We will include these multi-seed results in the revised manuscript and clarify the conclusions supported by this comparison.

AuthorsOpenReview ↗

Replies to Reviewer JDaE – Review 2

Summarized Review 2: Additional Reference Data vs. Alignment Supervision

Thank you for your suggestion regarding additional reference data. We agree that it is important to distinguish whether the improvements come from exposure to more cross-lingual reference information or from representation-level alignment supervision. Beyond the data augmentation and distillation baselines already included in Section 5.3, we conducted two additional experiments using the same sentence-matched reference texts in eight languages, without requiring TTS:

  • Text-Augmented CE: We jointly train on natural target speech and reference texts in eight languages using classification CE, while matching the training exposure to natural target speech. This evaluates the benefit of directly adding translated text as training data.
  • Text-K8: We completely remove TTS and extract task-conditioned representations through the frozen model's text pathway. We then construct multi-layer semantic anchors using the same projection and alignment procedure as AnchorSLU, providing representation-level supervision for natural target speech.

1. Experimental results

We evaluated these methods on the fixed 10-language SIB-Fleurs diagnostic panel, alongside the existing baselines in Tables 4 and 5:

MethodAccuracy (%)Macro-F1 (%)
Target-only CE55.3551.25
Exposure-matched TTS-CE57.7553.38
Reference KD57.1151.15
Text-Augmented CE56.4853.32
Text-K858.5454.98
AnchorSLU60.5456.53

2. Analysis

The results show that simply adding translated text does not reproduce the gains of AnchorSLU. Text-Augmented CE improves Accuracy and Macro-F1 over Target-only CE by only 1.13 and 2.07 percentage points, while AnchorSLU achieves gains of 5.19 and 5.28 points, respectively. The existing Exposure-matched TTS-CE and Reference KD baselines also remain below AnchorSLU.

More importantly, without TTS, Text-K8 still improves Accuracy by 3.19 points and Macro-F1 by 3.73 points over Target-only CE, showing that exact sentence-level cross-lingual correspondence can provide effective alignment supervision through text representations. The full speech-reference AnchorSLU further outperforms Text-K8 by 2.00 points in Accuracy and 1.55 points in Macro-F1.

These results show that AnchorSLU's improvements come not only from access to additional cross-lingual information, but also from how exact sentence-level correspondence is used for representation-level supervision. Both text and speech references provide useful semantic anchors, while the full speech-reference configuration achieves the best performance. This further supports our central idea of using sentence-level cross-lingual correspondence to complement class-level CE supervision.

We will include these additional text-based comparisons and analyses in the revised manuscript to clarify the respective roles of reference information and alignment supervision.

AuthorsOpenReview ↗

Replies to Reviewer JDaE – Review 3

Summarized Review 3: Contribution of Multilingual Aggregation

Thank you for your question about the contribution of multilingual aggregation. We agree that a single sentence-matched reference already provides substantial gains. To distinguish the roles of sentence-level anchoring and multi-reference aggregation, we conducted two paired comparisons across three training seeds on the 20-language SIB-Fleurs panel.

1. Single-reference supervision

K=1 assigns one randomly selected reference language to each sentence in advance, with the same assignments used across training seeds. Reference-augmented CE trains on natural target speech and the corresponding reference speech using classification CE, without anchor supervision, and receives twice as many training steps as AnchorSLU. AnchorSLU instead uses the same references to construct sentence-level semantic anchors. Models are adapted separately for each target language.

SeedTarget-only CEReference-augmented CE (K=1)AnchorSLU (K=1)
4154.80/51.0856.65/52.8158.51/54.89
4255.73/51.5755.86/51.6158.73/54.74
4355.92/51.5056.90/51.6658.08/54.31
Mean ± SD55.48±0.60 / 51.39±0.2756.47±0.54 / 52.03±0.6858.44±0.33 / 54.65±0.30

Both tables report Accuracy / Macro-F1 (%). Mean ± SD denotes the mean and sample standard deviation across three training seeds.

With the same single-reference information, AnchorSLU improves over Reference-augmented CE by 1.97 percentage points in Accuracy and 2.62 points in Macro-F1 on average, with gains on both metrics in every run. Even with twice as many training steps, reference-augmented CE remains below single-reference anchoring. This shows that the gains depend not only on access to reference data, but also on how these references provide sentence-level representation supervision.

2. Benefits of multi-reference aggregation

Using the same eight-language reference pool, we further compare one reference per sentence with aggregation over all eight references:

SeedAnchorSLU (K=1)AnchorSLU (K=8)ΔAcc / ΔF1 (pp)
4158.51/54.8959.98/56.82+1.47 / +1.93
4258.73/54.7460.32/56.72+1.59 / +1.98
4358.08/54.3159.32/55.74+1.24 / +1.43
Mean ± SD58.44±0.33 / 54.65±0.3059.87±0.51 / 56.43±0.60+1.43 / +1.78

Expanding from K=1 to K=8 further improves Accuracy by 1.43 percentage points and Macro-F1 by 1.78 points, with positive gains across all three training seeds. Thus, the strong gains from a single reference do not remove the value of multilingual aggregation: sentence-level anchoring provides effective supervision, while aggregating multiple cross-lingual references further strengthens its effect.

We will include these two controlled comparisons in the revised manuscript to clarify the respective roles of sentence-level anchoring and multi-reference aggregation.

Reviewer_tuTQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AnchorSLU, a technique for improving SLU performance on low-resource languages. The proposed technique generates semantically parallel speech-to-speech data across languages, which are used to align the model's representations, with the goal of transferring knowledge from high-resource to low-resource languages. Positive results on SLU benchmarks (SIB-FLEURS and Speech-Massive) and representation analyses suggest the effectiveness of the method.

Critical strengths
  1. The paper investigates why semantic anchoring is effective, and shows how it leads to stronger alignment. The ablations are thorough, and lead to interesting findings.
Critical weaknesses
  1. Missing baselines: the method relies on constructing parallel S2S data for alignment, which requires parallel ASR transcripts and a pre-trained TTS model. In that case, why not just run cascaded inference (like ASR -> Text understanding or ASR -> MT to high resource -> Text understanding)? If the e2e method is justified through better performance, then the results should show it.
  2. Limited novelty: there has been multiple previous works on semantic representations of speech [1,2,3,4] (which are not cited nor mentioned in the related work), and they have been applied to many tasks, including multilingual SLU on a similar Speech-SIB100 dataset [4].

[1] Duquenne, Paul-Ambroise, Holger Schwenk, and Benoît Sagot. "SONAR: sentence-level multimodal and language-agnostic representations." arXiv preprint arXiv:2308.11466 (2023).

[2] Chen, Mingda, et al. "BLASER: A text-free speech-to-speech translation evaluation metric. Proc. ACL 2023.

[3] Duquenne, Paul-Ambroise, et al. "Speechmatrix: A large-scale mined corpus of multilingual speech-to-speech translations." Proc. ACL 2023.

[4] Janeiro, João Maria, et al. "Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech." arXiv preprint arXiv:2603.16606 (2026).

Questions for authors
  1. I am assuming that in Table 2, the "frozen" model under "Matched Qwen2.5-Omni-7B comparison" corresponds to the original Qwen2.5-Omni-7B model, and that "Target-only CE" refers to Qwen2.5-Omni-7B model fine-tuned with CE loss? If so, this presentation is not easy for readers to understand. It would be better to name the model Qwen2.5-Omni-7B, and add indented rows representing the modifications on top of it (+Target-only CE or +AnchorSLU).
  2. Why not use real speech/text for SIB-Fleurs? Since SIB-Fleurs is sourced from FLEURS, which is n-way parallel across languages, one could retrieve the semantically aligned utterances, instead of relying on MT and TTS.
7 comments
AuthorsOpenReview ↗

Replies to Reviewer tuTQ – Review 1

Summarized Review 1: Comparison with Cascaded SLU Baselines

Thank you for suggesting additional cascaded baselines. We have added both ASR → Text Understanding and ASR → MT → Text Understanding systems, evaluating each before and after target-task adaptation.

1. Baseline selection

Our ASR-based comparisons draw on PolySpeech-100 [1], which evaluates Whisper- and Qwen3-ASR-based cascades with Qwen text models, and Fleurs-SLU [3], which includes ASR-based multilingual SLU pipelines. We also include the Parakeet-TDT-0.6B-v2 → Qwen2.5-7B-Instruct configuration used by Ugan et al. [2] as an additional literature-based comparison.

For the translation-based cascade, we reuse Whisper-large-v3 and NLLB-200-distilled-600M from a cascaded baseline evaluated in UniSS [4]. We replace its final speech generation stage with Qwen3-4B-Instruct-2507 for topic classification.

2. Evaluation setup

All systems are evaluated on the same 20-language SIB-Fleurs test set. Cascade adaptation trains the text understanding backbone with classification CE, while AnchorSLU uses CE and semantic anchor alignment. All values below are Accuracy / Macro-F1 (%).

MethodBaseline (Acc / F1)After Adaptation (Acc / F1)
Whisper-v3 → Qwen3-4B50.78 / 45.6455.30 / 50.76
Qwen3-ASR → Qwen3-4B49.82 / 45.7254.10 / 49.80
Parakeet-v2 → Qwen2.5-7B34.00 / 33.1444.27 / 39.46
Whisper-v3 → NLLB-200 → Qwen3-4B52.72 / 47.4256.07 / 50.49
AnchorSLU—59.87 / 56.43

3. Results

Target-task adaptation improves all four cascades. Nevertheless, AnchorSLU exceeds the best adapted cascade for each metric by 3.80 percentage points in Accuracy and 5.67 points in Macro-F1. Compared specifically with the translation-based cascade, the gains are 3.80 and 5.94 points, respectively.

4. Discussion

We believe that errors passed through intermediate transcripts and translations contribute to this gap. In the evaluated cascades, the text backend receives target-task adaptation, but the ASR and MT stages do not receive supervision from the final SLU objective. Adding MT raises Accuracy from 55.30% to 56.07%, but slightly lowers Macro-F1 from 50.76% to 50.49%, showing that translation does not improve both metrics in this setting.

AnchorSLU instead adapts directly on natural target speech with sentence-level anchor supervision and predicts task labels without additional ASR, MT, or TTS components at inference. We plan to further examine ASR errors, translation quality, and sample-level predictions to study how intermediate errors affect classification.

We will include these baseline configurations, training and evaluation protocols, and results in the revised manuscript.

References

[1] PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects. KDD 2026.

[2] Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026. IWSLT 2026.

[3] Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding. COLM 2025.

[4] UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice. ICLR 2026.

AuthorsOpenReview ↗

Replies to Reviewer tuTQ – Review 2

Summarized Review 2: Comparison with Prior Speech Representation Methods

Thank you for pointing out these important related works. We agree that SONAR [1], BLASER [2], SpeechMatrix [3], and Omnilingual SONAR [4] have made valuable contributions to cross-lingual speech representations. SONAR and Omnilingual SONAR learn general-purpose multilingual sentence embeddings, BLASER uses speech representations for translation evaluation, and SpeechMatrix focuses on parallel speech mining. Omnilingual SONAR also demonstrates downstream speech understanding through Spectrum, including on Speech-SIB100. We will discuss these studies in our revised Related Work section.

1. Comparison with prior work

The following table summarizes their differences from AnchorSLU:

MethodPrimary ObjectiveCross-lingual SemanticsSpeech–Speech CorrespondenceSpeech UnderstandingTask-conditioned AlignmentMulti-layer Anchoring
SONAR [1]Sentence Embeddings✓✗✗✗✗
BLASER [2]Translation Evaluation✓✓✗✗✗
SpeechMatrix [3]Parallel Speech Mining✓✓✗✗✗
Omnilingual SONAR [4]Embeddings & Downstream Transfer✓✗✓✗✗
AnchorSLULow-resource SLU Adaptation✓✓✓✓✓

Here, Speech–Speech Correspondence refers to directly using matched speech for training, matching, or evaluation. Task-conditioned Alignment uses task-prompt representations, while Multi-layer Anchoring applies semantic supervision at multiple internal layers.

Our key contribution is how sentence-level cross-lingual information is used for target-model adaptation. Unlike SONAR-based methods, which align speech embeddings with text-teacher representations, AnchorSLU constructs fixed semantic anchors from sentence-matched multilingual speech using the same frozen backbone as the target model. These task-conditioned, multi-layer anchors jointly supervise target adaptation with classification CE. Reference-side computation is performed offline, and inference uses only the adapted target model.

2. Evidence from controlled experiments

We further support this design through controlled experiments on the 10-language SIB-Fleurs diagnostic panel:

ComparisonAccuracy GainMacro-F1 Gain
Class-matched Anchoring+4.37+4.81
Exposure-matched TTS-CE+2.79+3.15
Text-K8 Anchoring+2.00+1.55
Final-layer-only Alignment+3.37+3.43
Projected SupCon+0.90+1.28

All gains are in percentage points and are computed within their matched settings. Class-matched Anchoring, Exposure-matched TTS-CE, and Projected SupCon use the original results in Tables 4 and 5. Text-K8 and Final-layer-only are additional experiments; the final-layer comparison reports the mean gain across three runs.

These comparisons support the value of exact sentence-level correspondence and task-conditioned, multi-layer representation supervision for low-resource SLU. Our contribution is therefore not the first use of cross-lingual semantic representations, but a method for turning them into task-conditioned supervision for low-resource SLU adaptation.

We will add the missing references and clarify these technical distinctions and supporting experiments in the revised manuscript.

References

[1] SONAR: Sentence-Level Multimodal and Language-Agnostic Representations. arXiv, 2023.

[2] BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric. ACL, 2023.

[3] SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations. ACL, 2023.

[4] Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech. arXiv, 2026.

AuthorsOpenReview ↗

Replies to Reviewer tuTQ – Review 3

Summarized Review 3: Model and Adaptation Settings in Table 2

Thank you for this helpful suggestion. Your understanding is correct. The Frozen, Target-only CE, and AnchorSLU settings in Table 2 are all based on Qwen2.5-Omni-7B. Specifically, Frozen refers to the original pretrained model without target-task adaptation. Target-only CE uses LoRA adaptation with classification CE only, while AnchorSLU uses LoRA adaptation with both classification CE and semantic anchor alignment loss.

Both Target-only CE and AnchorSLU are independently initialized from the same pretrained model and use identical target-language training data, LoRA configurations, optimization budgets, checkpoint-selection rules, and evaluation protocols to ensure a fair comparison.

We agree that organizing the table by backbone and adaptation method will make the results clearer. Following your suggestion, we will revise Table 2 to use Qwen2.5-Omni-7B as the main row, with indented subrows for different adaptation methods. We will also clarify the relationship between these settings in the table note.

Original:

Matched Qwen2.5-Omni-7B comparison

Model / MethodSIB-Fleurs (Acc / F1)Speech-MASSIVE (Acc / F1)
Frozen45.02 / 41.9735.82 / 35.89
Target-only CE55.48 / 51.3967.20 / 60.92
AnchorSLU59.87 / 56.4370.39 / 64.44

Revised:

Model / MethodSIB-Fleurs (Acc / F1)Speech-MASSIVE (Acc / F1)
Qwen2.5-Omni-7B\color{blue}{\text{Qwen2.5-Omni-7B}}45.02 / 41.9735.82 / 35.89
    +Target-only CE\color{blue}{\text{+Target-only CE}}55.48 / 51.3967.20 / 60.92
    +AnchorSLU\color{blue}{\text{+AnchorSLU}}59.87 / 56.4370.39 / 64.44
Reviewer_UQH4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a representation-level supervision method to enhance spoken language understanding (SLU) for low-resource languages in speech foundation models. The work is motivated by the limitations of conventional target-label-based supervision, which often fails to capture comprehensive semantic representations. To address this, the proposed method leverages sentence-matched references across multiple high-resource languages, utilizing aggregated representations from selected layers to guide the model toward a uniform, cross-lingual semantic space. Consequently, this alignment significantly boosts overall SLU accuracy. Evaluated using Qwen2.5-Omni-7B and Qwen3-Omni, experimental results demonstrate that representation-level supervision consistently outperforms traditional target-only, cross-entropy-based supervision.

Critical strengths

The authors argue that traditional, target-only cross-entropy loss is too coarse to effectively capture the complex semantic representations required for spoken language understanding (SLU). To address this limitation, this paper introduces representation-level supervision as a complementary loss function to guide speech foundation models toward richer semantic learning. Although the approach requires synthesizing sentence-matched text across eight high-resource languages to construct the training data, it demands no such processing during inference, ensuring that deployment remains free from external data synthesis dependencies. Experimental results on Qwen2.5-Omni and Qwen3-Omni demonstrate that this hybrid training objective significantly enhances SLU performance.

Critical weaknesses

The current experimental evaluation is insufficient to demonstrate the generalizability of the proposed method across speech foundation models. Although the method has been applied to Qwen2.5-Omni and Qwen3-Omni—showing clear improvements over the label-only cross-entropy baseline—the performance levels of both baselines in Table 2 remain relatively low. To robustly verify the efficacy of the proposed approach, it is highly desirable to evaluate it on models with stronger baseline performance, such as MiniCPM-o 2.6 or LLaMA-3.1-8B-Omni. Without these critical comparisons, it is difficult to draw definitive conclusions regarding the broader effectiveness of the proposed method.

Questions for authors
  1. In lines 102–103, the statement appears to be inaccurate. Could the authors please clarify why increasing language coverage alone cannot eliminate resource-dependent disparities in multilingual spoken language understanding?

  2. Regarding Equation (3), although the text mentions in the Appendix that "Pool" denotes average pooling, it is necessary to include a brief note defining this operation within the main text. This addition will significantly improve the readability and flow of the paper.

  3. Regarding the sentence-matched aggregation, have the authors experimented with dropping a subset of languages for each sample during training? It would be insightful to know how variations in language density per sample affect the overall performance.

  4. In Table 3, while the proposed method shows improvements when using a larger backbone, the "Frozen" baseline performance for this larger model is relatively low. It would be far more compelling to see a performance comparison using a top-tier baseline model. The authors are encouraged to evaluate their method by fine-tuning a stronger architecture, such as MiniCPM-o 2.6.

7 comments
AuthorsOpenReview ↗

Replies to Reviewer UQH4 – Review 1

Summarized Review 1: Evaluation on Stronger Speech Backbones

Thank you for raising Weakness 1 and Q4. We understand these concerns as asking whether AnchorSLU remains effective on stronger speech backbones and whether its gains hold over task-adapted baselines. We have added Frozen results to clarify the original comparisons and conducted a paired evaluation of Target-only CE and AnchorSLU on MiniCPM-o 2.6.

1. Adapted baselines in the original manuscript

The MiniCPM-o 2.6 and LLaMA-3.1-8B-Omni results in Table 2 are obtained after target-task CE adaptation, rather than from Frozen models. Below, we present the additional Frozen results alongside the original CE results. Both columns use the 20-language SIB-Fleurs panel and report Accuracy / Macro-F1 (%).

BackboneFrozenTarget-only CE (Table 2)
MiniCPM-o 2.641.81 / 29.5156.78 / 52.53
LLaMA-3.1-8B-Omni41.16 / 40.0651.59 / 47.65

After CE adaptation, MiniCPM-o 2.6 achieves 56.78% Accuracy and 52.53% Macro-F1, exceeding the corresponding Qwen2.5-Omni-7B results of 55.48% and 51.39% on the same panel.

Similarly, the Qwen3-Omni gains in Table 3 are measured against the adapted CE baseline: Accuracy increases from 64.88% to 67.83%, and Macro-F1 from 61.03% to 64.23%, yielding gains of 2.95 and 3.20 percentage points. The Frozen rows show the starting performance before adaptation, rather than serving as the primary baselines for measuring AnchorSLU's gains.

2. Additional evaluation on MiniCPM-o 2.6

Following your suggestion, we further evaluated AnchorSLU on MiniCPM-o 2.6. This experiment uses the 10-language SIB-Fleurs panel. We construct sentence-matched anchors at layers 8, 12, 16, and 20 using the original task-prompt token mean pooling rule. CE and AnchorSLU use the same target training data, training budget, checkpoint selection based on validation CE, and evaluation protocol.

MethodAccuracy (%)Macro-F1 (%)ΔAcc / ΔF1 vs. CE
Target-only CE56.8552.37—
AnchorSLU59.4155.29+2.56 / +2.92

AnchorSLU improves over the matched Target-only CE baseline by 2.56 percentage points in Accuracy and 2.92 points in Macro-F1. This provides direct evidence of its applicability to another speech foundation model and shows that its gains are not limited to comparisons with weaker Frozen baselines.

We will clarify the adaptation settings in Tables 2 and 3 and add the MiniCPM-o 2.6 results, reference configuration, and training and evaluation protocols to the revised manuscript.

AuthorsOpenReview ↗

Replies to Reviewer UQH4 – Review 2

Summarized Review 2: Language Coverage and SLU Performance

Thank you for asking us in Q1 to clarify Lines 102–103. Our intention is not to suggest that broader language coverage cannot reduce performance gaps, but to emphasize that covering more languages does not necessarily lead to comparable speech understanding performance across languages.

Language coverage describes the range of languages a model can process. Actual SLU performance also depends on the amount and quality of training data, acoustic and linguistic differences, and the availability of task-specific supervision. Broader pretraining coverage does not ensure sufficient training data or task supervision for every language.

Figure 1 provides a corresponding observation. Under the same frozen Qwen2.5-Omni-7B backbone, task, and prompt settings, the eight high-resource reference languages achieve 83.2% average Accuracy, compared with 45.0% for the 20 low-resource target languages, a gap of 38.2 percentage points. This motivates using sentence-level semantic correspondence from reference languages to provide additional representation-level supervision for low-resource target languages.

We will revise Lines 102–103 to distinguish language coverage from actual understanding performance and avoid an overly strong claim.

Original:

Increasing model scale and language coverage alone therefore does not eliminate resource-dependent disparities in multilingual speech understanding.

Revised:

Broader language coverage and larger models\color{blue}{\text{Broader language coverage and larger models}} do not necessarily lead to comparable speech\color{blue}{\text{do not necessarily lead to comparable speech}} understanding performance across languages,\color{blue}{\text{understanding performance across languages,}} particularly when training data and task-specific\color{blue}{\text{particularly when training data and task-specific}} supervision are uneven.\color{blue}{\text{supervision are uneven.}}

AuthorsOpenReview ↗

Replies to Reviewer UQH4 – Review 3

Summarized Review 3: Task-Prompt Representation Pooling

Thank you for suggesting in Q2 that we clarify the Pool operation. We agree that defining it directly in the main text will improve the readability of Equation (3) and the method description.

Pool denotes mean pooling over task-prompt token hidden states at each selected layer. The span begins after the audio-end token and ends before the first subsequent user-message end token. It includes only task-prompt tokens, excluding the system message, special boundary tokens, assistant prefix, and generated answer. Target and reference speech use the same task prompt and pooling rule to obtain task-conditioned sentence-level representations.

We will add the mathematical definition and token range after Equation (3), while retaining the implementation details in Appendix A.2. The original description remains unchanged; the Revised block shows the added text.

Original:

Given task prompt pp, we pool the task-prompt hidden states after the audio input and before answer generation at layer l∈Sl \in \mathcal{S}, yielding the task-conditioned reference representation.

Revised:

Here, Pool denotes mean pooling over the task-prompt token hidden states.\color{blue}{\text{Here, Pool denotes mean pooling over the task-prompt token hidden states.}}

Let Tp denote the task-prompt token positions between the audio-end token\color{blue}{\text{Let } \mathcal{T}_p \text{ denote the task-prompt token positions between the audio-end token}} and the first subsequent user-message end token. For the hidden states at layer l,\color{blue}{\text{and the first subsequent user-message end token. For the hidden states at layer } l\text{,}}

\color{blue}{\text{Pool(Hₗ) = (∑ₜ∈Tₚ Hₗ,ₜ) / |Tₚ|}}

The system message, special boundary tokens, assistant prefix, and generated answer\color{blue}{\text{The system message, special boundary tokens, assistant prefix, and generated answer}} are excluded. The same pooling rule is used for target and reference speech.\color{blue}{\text{are excluded. The same pooling rule is used for target and reference speech.}}

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers UQH4, tuTQ, and JDaE

We sincerely thank reviewers UQH4, tuTQ, and JDaE for their careful reading and constructive feedback. Their comments have helped us clarify AnchorSLU's contribution and strengthen the evidence for using sentence-level cross-lingual correspondence to support low-resource spoken language understanding.

In response, we have added stronger-backbone and cascaded SLU comparisons, controlled experiments with text and natural-speech references, and multi-seed analyses of Projected SupCon, reference language density, multilingual aggregation, and layer selection. We also provide quantitative cross-lingual retrieval results, discuss related work, and clarify the pooling definition and adaptation settings in the main tables.

Below, we organize our responses by reviewer and theme, addressing each concern with the relevant clarifications and experimental results. We will incorporate these additions into the revised manuscript.

AuthorsOpenReview ↗

Summary of Concerns and Responses

We thank reviewers UQH4, tuTQ, and JDaE for their careful review and constructive discussion. Reviewer UQH4 confirmed that the additional experiments addressed the concern about effectiveness across speech backbones. We summarize the main concerns and our responses below.

  1. Contribution and relation to prior work (tuTQ). We discussed SONAR, BLASER, SpeechMatrix, and Omnilingual SONAR and clarified our contribution: turning sentence-level cross-lingual representations into supervision for low-resource target-task adaptation. AnchorSLU constructs fixed, task-conditioned anchors using a frozen copy of the same backbone. Controlled comparisons support the value of exact sentence matching and representation-level supervision beyond class labels and direct data augmentation. Inference uses only the adapted target model.

  2. Stronger baselines and cross-backbone evaluation (UQH4, tuTQ). We clarified that the main gains are measured against task-adapted CE baselines. On the 10-language panel, AnchorSLU improves over matched CE on MiniCPM-o 2.6 by 2.56 percentage points in Accuracy and 2.92 in Macro-F1. We also evaluated four ASR and ASR–MT cascades before and after task adaptation on the same 20-language test set. AnchorSLU outperforms all four adapted cascades on both metrics.

  3. Reference information and fair comparisons (tuTQ, JDaE). We added text augmentation, text-anchor, and natural-speech reference controls. Both text and natural-speech anchors provide useful supervision, while directly adding reference text or speech as classification data remains below AnchorSLU. With the same reference speech, layers, and projection space, AnchorSLU also outperforms Projected SupCon on both metrics in all three runs, with average gains of 1.09 percentage points in Accuracy and 1.57 in Macro-F1. These results support the benefit of anchor supervision beyond access to additional reference information.

  4. Multilingual aggregation and layer selection (UQH4, JDaE). One reference language per sentence already improves over CE; aggregating all eight languages improves further in each of three runs. For the single-reference setting, the language is sampled separately for each sentence and remains fixed during training. A separate three-run ablation also shows that multi-layer anchoring outperforms final-layer-only anchoring. These comparisons support the additional value of multilingual aggregation and multi-layer supervision.

  5. Quantitative alignment evidence (JDaE). We evaluated sentence retrieval on test speech from all 20 target languages, using the same natural-speech reference gallery and fixed projections fitted on training references. When retrieval candidates are restricted to the same class, R@1 increases from 41.63% with CE to 63.53% with AnchorSLU. This provides quantitative evidence of finer sentence-level correspondence beyond class labels, complementing the t-SNE analysis.

We also clarified the distinction between language coverage and understanding performance, the Pool operation in Equation (3), and the model and adaptation settings in the tables. We will include the additional results and clarifications in the revised manuscript. We hope this summary supports the Area Chair’s and reviewers’ further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.