acceptodds
Under review as a conference paper at ICLR 2027

A Fixed-Text Diagnostic for Selector-Aligned Source-Acoustic Prompting in Speech-to-Speech Translation

Abstract

Speech-to-speech translation can preserve target words while changing delivery. We examine a source-acoustic policy that measures source speech to select a target-language prompt. Its selector-aligned score uses the same four acoustic summaries. We introduce a counterfactual protocol that fixes the text-to-speech (TTS) input, renderer, candidate bank, decoding, and utterance identifier while changing only prompt policy. We test 160 read-speech utterances from the Few-shot Learning Evaluation of Universal Representations of Speech (FLEURS) validation set across English-to-German, English-to-French, English-to-Dutch, and English-to-Turkish. We compare source-acoustic prompting with function-label-only prompting, one recorded shuffled-source allocation, and three recorded random allocations. On the selector-aligned score, source-acoustic prompting exceeds the recorded shuffled-source allocation but is inconclusive against the function-label and random controls. Against function labels, the score comparing generated speech with held-out target references has a lower point estimate in all four language pairs. Lower values mean less acoustic similarity. The English-to-Dutch interval remains inconclusive. The duration score is higher for English-to-German but lower in the other three pairs, so it does not show a stable timing benefit. A stricter automatic audit does not use the selector's four stored summary values when it scores the outputs. In that audit, only the pooled pitch contrast against shuffled source is positive, and we do not treat that isolated contrast as standalone evidence. Energy, duration, and pause remain inconclusive, and some Dutch contrasts still have no paired support. On the 80-item support-screened WavLM follow-up, frozen embeddings are lower against function labels and inconclusive against shuffled source; 29 of its IDs overlap the pilot, so it is not an independent-cohort replication. The predeclared positive rule requires both contrasts to be above zero and is not met. Together, these automatic checks show movement in the selector's preferred signal, not reliable improvement in held-out target speech.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech-to-speech translation, speech synthesis, prompt selection, expressive speech, automatic evaluation, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tydHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies source-acoustic prompt selection for speech-to-speech translation under a controlled fixed-text setting. The authors keep the translated TTS input, renderer, prompt bank, decoding configuration, and utterance identity fixed, and vary only the prompt-selection policy. They compare a source-acoustic selector against function-label, shuffled-source, and random controls on 160 FLEURS utterances across four language pairs. The main finding is that source-acoustic prompting improves a selector-aligned acoustic score relative to one shuffled-source control, but this improvement does not consistently carry over to target-reference, duration, stricter summary-exclusion, or frozen-WavLM diagnostics.
The paper therefore argues that improvements on a metric closely aligned with the selector should not by themselves be interpreted as evidence of source-to-target delivery preservation.

Critical strengths
  1. Careful counterfactual experimental design. The fixed-text protocol isolates prompt-policy changes substantially better than an end-to-end S2ST comparison would. The use of paired utterance-level comparisons and several controls makes the empirical observation considerably easier to interpret.
  2. Transparent treatment of negative and inconclusive results. The paper is careful about unsupported contrasts, heterogeneous language-pair behavior, overlapping follow-up data, and the limitations of automatic metrics. This makes the empirical claims appropriately restrained rather than overstated.
Critical weaknesses
  1. The paper lacks a convincing evaluation of actual delivery preservation. All conclusions are based on automatic acoustic metrics or WavLM similarity, without any human perceptual evaluation. This makes it difficult to tell whether the observed metric differences correspond to meaningful differences in perceived delivery.

  2. The main positive result is somewhat built into the evaluation setup. The source-acoustic selector and the source-summary metric use the same four acoustic features, so an improvement on this selector-aligned metric is not very surprising. Most of the more independent target-side measurements are instead negative or inconclusive.

  3. The experimental scope is too limited for the broader methodological claim. The study evaluates one simple selector on small read-speech subsets with a single fixed synthesis setup. The follow-up is also not an independent replication because 29 of the 80 utterances overlap with the pilot, making it hard to assess how well the conclusion generalizes.

Questions for authors
  1. Why is similarity to the held-out target reference an appropriate proxy for preserving the source utterance's delivery?

  2. Do the authors have any human evaluation showing that the reported automatic metrics correlate with perceived delivery preservation?

  3. Would the same conclusion hold with a stronger source-conditioned selector or a different feature set?

  4. Since fixed TTS input does not guarantee identical generated lexical content, have the authors checked ASR consistency across prompt policies?

1 comment

The selector-aligned score uses the same four acoustic summaries as prompt selection: mean pitch, RMS energy, voiced-frame ratio, and pause count. Its positive contrast against one shuffled-source allocation is therefore selector-aligned evidence. Independent readouts do not show a consistent target-side gain: target-reference and WavLM similarities are lower against function-label and random policies, while the shuffled-source WavLM contrast is inconclusive. Duration effects change sign across language pairs.

The stricter summary-exclusion audit removes the selector's stored scalar fields from its score inputs. Its pooled shuffled-source pitch contrast uses 72 paired IDs, includes no English–Dutch items, and has no supported pair-level interval excluding zero. The 80-item WavLM follow-up overlaps the pilot on 29 IDs, so it is not an independent replication.

The paper reports no listener study, speaker-disjoint evaluation, or test of a stronger selector or different feature family. Its target-reference score is an automatic comparison with paired held-out FLEURS waveforms; the paper does not validate it as a measure of perceived delivery. Generated words are unchecked, and the submission reports no prompt-wise transcripts. It provides no result on whether ASR outputs remain consistent across policies or whether another selector preserves the measured result.

Reviewer_33e1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a fixed-text prompt-policy protocol that should separate the prompt effect from a cascaded speech-to-speech translation system (composed of ASR, MT--or a direct ST, and TTS models). The experiments are done in four languages (English to German, French, Dutch, and Turkish) and on a single model (WavLM), on a very small number of samples.

Critical strengths

I'm sorry but I cannot find any.

Critical weaknesses

I am very familiar with the field but I really struggled to understand even the main research questions of this paper. It seems a fully AI-generated paper (a technical report, adapted to an ICLR submission):

  • The paper is very difficult to follow, starting from the abstract (that is extremely long), many sentences are disconnected from each other (e.g. "Against function labels, the score comparing generated speech with held-out target references has a lower point estimate in all four language pairs. Lower values mean less acoustic similarity. The English-to-Dutch interval remains inconclusive.").
  • The paper also refers to its contributions as a "pilot", and indeed it is as it analyzes 40 items from FLEURS on 4 languages, which I don't think can even reach 15-20 minutes per language. Therefore, it is 0 representative and very distant to the standard ST test set durations (from 2 hours onwards).
  • The experimental settings are mixed with the methodology ("protocol") section, with illegible random seeds frequently mentioned in this and the results section.
  • The experiments are done on a single model, Wav-LM, and the results are never elaborated on the broader research question: it is never clearly explained or empirically demonstrated how the proposed "fixed-text prompt-policy protocol" can effectively separate "prompt effects from ASR, MT, renderer, and variation in the TTS input text" from the results.
  • The related works are non existing, the paper has 9 references in total, and neither the Wav-LM model (used in all experiments) has been cited.
  • WavLM is a speech-to-text model, but this paper should be theoretically analyzing speech-to-speech translation, therefore a TTS is needed, but, apart from the Introduction, no TTS is ever mentioned nor used throughout the experiments.
Questions for authors
  • Why is the problem being analyzed relevant for the S2ST community?
  • How your results support the prompt effects separation?
  • Since the paper is about speech-to-speech translation, where is the analysis of the TTS?
  • Why selecting 40 items from FLEURS?
  • Why selecting the 4 languages analyzed? Are they significant for some specific reasons?
  • Why is only a model being used? Why WavLM?
  • What is the total duration (in hours) of the data used for the experiments?
  • Where is the figure of Figure 1?
3 comments

The sample is 160 read-speech FLEURS utterances: 40 IDs per language pair across English–German, English–French, English–Dutch, and English–Turkish. The paper reports these deterministic subsets but gives no rationale for the four language pairs or the 40-item sample size. It also reports no total audio-hours figure, and no duration manifest is provided from which to calculate one.

The protocol fixes the translated TTS input string, renderer checkpoint, candidate bank, decoding, and seed across prompt policies, isolating prompt choice conditional on those inputs. Generated waveforms are not checked for identical lexical content, so the design does not rule out prompt-dependent word errors. Its relevance to speech-to-speech translation is a prompt-policy diagnostic within a translation-to-TTS cascade; it does not evaluate end-to-end translation or direct-audio systems. TTS is the fixed renderer; the paper scores its prompt-conditioned outputs but does not present a standalone TTS-system benchmark.

The supplied PDF contains Figure 1 on page 3. The paper uses one frozen WavLM Base+ revision as an automatic readout and does not compare multiple WavLM models. Its reference list does not cite WavLM, although the protocol names it. The protocol and analysis settings also need clearer organization.

This is not a rebuttal and seems completely AI generated.

The empirical question is whether the selector-aligned pitch score yields a target-side gain; our prior reply did not answer it. The positive strict-summary-exclusion pitch contrast is pooled over 72 paired IDs, contains no English–Dutch examples, and no supported language-pair interval excludes zero. The 80-item WavLM follow-up overlaps the pilot on 29 IDs and does not meet the positive rule. We therefore limit the conclusion to a diagnostic finding: a selector-aligned score can improve without a consistent target-side gain.

Reviewer_3YoJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a fixed-text audit of source-acoustic prompt selection for speech-to-speech translation. On 160 FLEURS items across four language pairs, the selector improves its aligned metric against a shuffled-source control but shows no consistent target-side benefit. Additional acoustic audits and a partially overlapping WavLM follow-up support a cautious interpretation of these results.

Critical strengths

The paired design holds text, renderer, and generation settings fixed, helping isolate prompt-policy effects. The paper addresses an important evaluation issue: improvements on selector-aligned metrics do not establish successful delivery transfer. The authors transparently report uncertainty, missing metric support, language-level heterogeneity, and cohort overlap.

Critical weaknesses

Target-reference validity is unclear. FLEURS provides separately recorded readings of parallel texts, not recordings validated for cross-lingual preservation of emotion, emphasis, or intent. Lower reference similarity may therefore reflect speaker or reading-style differences rather than poorer delivery preservation. The findings establish metric disagreement more clearly than meaningful delivery differences. The tested heuristic has limited demonstrated practical relevance. The paper does not sufficiently motivate retrieving target-language prompts instead of directly conditioning multilingual TTS on source audio. This natural baseline, if supported by the renderer, would help establish the relevance of the diagnostic. Output lexical fidelity is unchecked. Fixed TTS input does not guarantee identical spoken content. Omissions, repetitions, or truncation could affect duration, pauses, and WavLM similarity, complicating delivery-related interpretations. Supporting evidence remains limited. The strict audit’s positive shuffled-source pitch contrast uses only 72 paired items and excludes Dutch entirely. The 80-item follow-up overlaps the pilot on 29 IDs and is not an independent replication. Reproducibility is incomplete. The paper states that the supplementary package cannot independently reproduce the reported scores. Public sample IDs, prompt assignments, exact renderer settings, and reconstruction instructions would strengthen the contribution.

Questions for authors

How are reference recordings selected, and are conclusions robust to alternative recordings? Does the renderer support source audio directly as a cross-lingual prompt? Do the findings persist after checking lexical fidelity and restricting comparisons to common valid support?

1 comment

The held-out FLEURS target waveform is an automatic acoustic comparator, not a validated reference for cross-lingual emotion, emphasis, or intent. The paper reports no reading-style equivalence check, so lower target-reference similarity is an acoustic-score difference under the tested renderer and sample, not evidence of delivery transfer.

The fixed-text intervention holds the TTS input string, candidate bank, renderer checkpoint, decoding, and generation seed constant while changing prompt selection. It does not verify that generated speech contains the same words across policies. No lexical-fidelity audit is reported, so omissions, repetitions, or truncation remain possible explanations for acoustic-score changes.

The summary-exclusion audit gives one positive pooled pitch contrast against shuffled source profiles, based on 72 paired IDs; it includes no English–Dutch items, and no supported pair-level interval excludes zero. The 80-item WavLM follow-up shares 29 pilot IDs and fails its predeclared positive rule. These results do not establish a four-language target-side benefit or independent replication.

The experiment does not compare with direct source-audio conditioning. The submission describes a narrow prompt-selection audit, and no practical advantage over direct source conditioning is established. The paper points to an anonymous archive, but the available record does not verify archived IDs, prompts, renderer settings, or instructions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.