- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AnchorSLU, a way of leveraging cross-lingual representation to supervise the training of low resource speech models. The idea improves upon standard cross-entropy training which only provides class-level supervision, in a way such that semantically different utterances within the same class receive the same target. AnchorSLU on the other hand uses sentence-level correspondence. Specifically, they synthesize reference speech across high-resource languages for each sentence in the target language. Then they encode the references in a frozen speech model. Representations from selected layers are used to construct semantic anchors that are specific to each sentence. During training, an adaptor model learns from speech in the target language, using normal cross-entropy plus alignment to the anchors. The authors evaluated their method on 20 low-resource languages and show general improvement against conventional cross-entropy-based training. They also performed controlled studies which indicate that the exact sentence correspondence is the primary reason fro the performance gains.
Critical strengths
- the design is cost-efficient since all alignment-related work is done before deployment. the data creation, encoding and fitting is offline. so that only the adapted model is actually needed for inference.
- The evaluation is quite comprehensive, covering 20 languages across different scripts and 2 tasks. This shows the applicability of the method in different scenarios.
Critical weaknesses
- The controlled ablation studies are less convincing. For example, the difference between Projected SupCon and AnchorSLU is very small, within normal seed variance. The authors claim that "This comparison isolates the advantage of consolidating multilingual realizations of each sentence into a single semantic anchor, enabling direct target-to-anchor alignment without introducing cross-sentence negatives." which would be more convincing with repetition runs.
- It's not surprising that AnchorSLU improves upon target-only CE since the proposed method has seen more training data in the form of translated speech. A better baseline should still see the translated data during training, maybe as a text representation, ablating the TTS component. This would illustrate better that what's causing the improvement is the alignment method, instead of simply seeing more representations of the data.
Questions for authors
- From figure 5(b), it seems like over half of the improvement gain is already there when using one language as reference. Has the authors done further analysis into what the multilingual alignment module does?
- How were the layers chosen to get the representations from? What happens if only the final layer representation is used to create the anchors?
- The t-SNE plots are illustrative and they are easy to read, but can any quantitative metric be provided to show the alignment under AnchorSLU across different low-resource languages?
7 comments
Replies to Reviewer JDaE – Review 2
Summarized Review 2: Additional Reference Data vs. Alignment Supervision
Thank you for your suggestion regarding additional reference data. We agree that it is important to distinguish whether the improvements come from exposure to more cross-lingual reference information or from representation-level alignment supervision. Beyond the data augmentation and distillation baselines already included in Section 5.3, we conducted two additional experiments using the same sentence-matched reference texts in eight languages, without requiring TTS:
- Text-Augmented CE: We jointly train on natural target speech and reference texts in eight languages using classification CE, while matching the training exposure to natural target speech. This evaluates the benefit of directly adding translated text as training data.
- Text-K8: We completely remove TTS and extract task-conditioned representations through the frozen model's text pathway. We then construct multi-layer semantic anchors using the same projection and alignment procedure as AnchorSLU, providing representation-level supervision for natural target speech.
1. Experimental results
We evaluated these methods on the fixed 10-language SIB-Fleurs diagnostic panel, alongside the existing baselines in Tables 4 and 5:
| Method | Accuracy (%) | Macro-F1 (%) |
|---|---|---|
| Target-only CE | 55.35 | 51.25 |
| Exposure-matched TTS-CE | 57.75 | 53.38 |
| Reference KD | 57.11 | 51.15 |
| Text-Augmented CE | 56.48 | 53.32 |
| Text-K8 | 58.54 | 54.98 |
| AnchorSLU | 60.54 | 56.53 |
2. Analysis
The results show that simply adding translated text does not reproduce the gains of AnchorSLU. Text-Augmented CE improves Accuracy and Macro-F1 over Target-only CE by only 1.13 and 2.07 percentage points, while AnchorSLU achieves gains of 5.19 and 5.28 points, respectively. The existing Exposure-matched TTS-CE and Reference KD baselines also remain below AnchorSLU.
More importantly, without TTS, Text-K8 still improves Accuracy by 3.19 points and Macro-F1 by 3.73 points over Target-only CE, showing that exact sentence-level cross-lingual correspondence can provide effective alignment supervision through text representations. The full speech-reference AnchorSLU further outperforms Text-K8 by 2.00 points in Accuracy and 1.55 points in Macro-F1.
These results show that AnchorSLU's improvements come not only from access to additional cross-lingual information, but also from how exact sentence-level correspondence is used for representation-level supervision. Both text and speech references provide useful semantic anchors, while the full speech-reference configuration achieves the best performance. This further supports our central idea of using sentence-level cross-lingual correspondence to complement class-level CE supervision.
We will include these additional text-based comparisons and analyses in the revised manuscript to clarify the respective roles of reference information and alignment supervision.
Replies to Reviewer JDaE – Review 3
Summarized Review 3: Contribution of Multilingual Aggregation
Thank you for your question about the contribution of multilingual aggregation. We agree that a single sentence-matched reference already provides substantial gains. To distinguish the roles of sentence-level anchoring and multi-reference aggregation, we conducted two paired comparisons across three training seeds on the 20-language SIB-Fleurs panel.
1. Single-reference supervision
K=1 assigns one randomly selected reference language to each sentence in advance, with the same assignments used across training seeds. Reference-augmented CE trains on natural target speech and the corresponding reference speech using classification CE, without anchor supervision, and receives twice as many training steps as AnchorSLU. AnchorSLU instead uses the same references to construct sentence-level semantic anchors. Models are adapted separately for each target language.
| Seed | Target-only CE | Reference-augmented CE (K=1) | AnchorSLU (K=1) |
|---|---|---|---|
| 41 | 54.80/51.08 | 56.65/52.81 | 58.51/54.89 |
| 42 | 55.73/51.57 | 55.86/51.61 | 58.73/54.74 |
| 43 | 55.92/51.50 | 56.90/51.66 | 58.08/54.31 |
| Mean ± SD | 55.48±0.60 / 51.39±0.27 | 56.47±0.54 / 52.03±0.68 | 58.44±0.33 / 54.65±0.30 |
Both tables report Accuracy / Macro-F1 (%). Mean ± SD denotes the mean and sample standard deviation across three training seeds.
With the same single-reference information, AnchorSLU improves over Reference-augmented CE by 1.97 percentage points in Accuracy and 2.62 points in Macro-F1 on average, with gains on both metrics in every run. Even with twice as many training steps, reference-augmented CE remains below single-reference anchoring. This shows that the gains depend not only on access to reference data, but also on how these references provide sentence-level representation supervision.
2. Benefits of multi-reference aggregation
Using the same eight-language reference pool, we further compare one reference per sentence with aggregation over all eight references:
| Seed | AnchorSLU (K=1) | AnchorSLU (K=8) | ΔAcc / ΔF1 (pp) |
|---|---|---|---|
| 41 | 58.51/54.89 | 59.98/56.82 | +1.47 / +1.93 |
| 42 | 58.73/54.74 | 60.32/56.72 | +1.59 / +1.98 |
| 43 | 58.08/54.31 | 59.32/55.74 | +1.24 / +1.43 |
| Mean ± SD | 58.44±0.33 / 54.65±0.30 | 59.87±0.51 / 56.43±0.60 | +1.43 / +1.78 |
Expanding from K=1 to K=8 further improves Accuracy by 1.43 percentage points and Macro-F1 by 1.78 points, with positive gains across all three training seeds. Thus, the strong gains from a single reference do not remove the value of multilingual aggregation: sentence-level anchoring provides effective supervision, while aggregating multiple cross-lingual references further strengthens its effect.
We will include these two controlled comparisons in the revised manuscript to clarify the respective roles of sentence-level anchoring and multi-reference aggregation.
Replies to Reviewer JDaE – Review 1
Summarized Review 1: Multi-Seed Evaluation of Projected SupCon
Thank you for your suggestion to further verify the stability of our experimental results. We agree that evaluating Projected SupCon and AnchorSLU across multiple random seeds would make the comparison more convincing. We therefore conducted paired experiments using three independent random seeds (41, 42, and 43) on the fixed 10-language SIB-Fleurs diagnostic panel. Both methods use the same multilingual reference speech, selected layers, projection space, target training data, optimization budget, and checkpoint-selection rule to ensure a fair comparison.
1. Multi-seed comparison
The results are shown below:
Results are percentages, with gains in percentage points. Mean ± SD denotes the mean and sample standard deviation across three runs. The Seed 42 results match Table 5.
AnchorSLU outperforms Projected SupCon in both Accuracy and Macro-F1 across all three seeds, with average gains of 1.09 and 1.57 percentage points, respectively. AnchorSLU also shows lower variation across seeds, particularly in Macro-F1, where its standard deviation is 0.10 compared with 0.60 for Projected SupCon. These results show that the performance advantage is not limited to a single run and follows the same positive trend across all three seeds.
2. Target-side optimization
AnchorSLU also requires fewer target-side optimization steps. As reported in Section 5.3 and Appendix D.5, AnchorSLU reaches its selected checkpoint in 212 steps on average, compared with 260 steps for Projected SupCon. The median stopping steps are 360 and 440, respectively, representing a reduction of approximately 18%. Thus, in our original diagnostic experiment, AnchorSLU achieves better classification performance with fewer target-side training steps.
Under the same reference information and representation preprocessing, AnchorSLU aggregates sentence-matched multilingual representations into fixed semantic anchors, allowing direct target-to-anchor alignment without cross-sentence negatives. The additional multi-seed results further support the effectiveness of this supervision strategy. In future work, we plan to extend the evaluation to more random seeds and language settings and further investigate the effects of anchor aggregation and negative samples.
We will include these multi-seed results in the revised manuscript and clarify the conclusions supported by this comparison.