acceptodds
Under review as a conference paper at ICLR 2027

Real-Speech Anchoring for Data-Efficient S2S Post-Training

Abstract

Speech-to-speech (S2S) assistants must adapt to how a request is spoken and reply with appropriate prosody, yet their training supervision rarely carries this information: human-recorded dialogue is scarce and misaligned with assistant behavior, while scalable synthetic dialogue bounds every speech realization by the synthesizer. We call this shortfall the Non-Semantic Information Gap and trace it to the unit both paradigms share, the complete dialogue. At the level of a single exchange, one verified real recording suffices when kept in the role whose realization is the supervision signal, with only its counterpart synthesized. RAST (Real-speech Anchored S2S Training) applies this principle in two directions: user-side anchoring (RAST-U) pairs real input cues with condition-appropriate responses, and assistant-side anchoring (RAST-G) places recorded prosody in the generation targets. We instantiate RAST as 284,660 exchanges spanning 2,914 hours, and introduce RASTBench, a transcript-controlled benchmark that scores response audio directly and separates completing the request from adapting to the speech condition. Against Pure-TTS controls that differ only in the provenance of the anchored endpoint, retaining real speech improves RASTBench by 4.72, 4.52, and 8.81 points for U, G, and their mixture, all statistically significant, with gains concentrated on condition-adaptation criteria. The two directions yield complementary capability profiles, their mixture achieves the strongest response adaptation, and a quarter of the corpus recovers most of the full-data gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech-to-Speech Language Models, Dataset, Benchmark, Multimodal post-training, NLP

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DTqdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that S2S post-training data lacks realistic non-semantic information (the “Non-Semantic Information Gap”). It proposes RAST, which builds each two-turn exchange around one verified real recording and synthesizes only the other side. RAST-U keeps real user speech, RAST-G keeps real (voice-converted) assistant speech, and RAST-Mix pools both. The corpus has 284,660 exchanges (2,914 h) mined from YouTube. The authors also introduce RASTBench, a transcript-controlled benchmark scored from response audio by an LLM judge. Fine-tuning Kimi-Audio-7B, RAST improves RASTBench over Pure-TTS controls by 4.7 (U), 4.5 (G), and 8.8 (Mix) points, and a quarter of the data recovers most of the gains.

Critical strengths
  1. Clean paired-control design. Each RAST configuration has a Pure-TTS control that changes only the provenance of one endpoint and keeps texts, counterparts, and training horizon fixed. This is the right way to isolate the value of real speech, and it is stronger than typical data papers.
  2. Audio-native, transcript-controlled evaluation. RASTBench scores response audio within same-transcript contrast groups and separates task completion from condition-specific adaptation. The judge is calibrated against expert labels (88% agreement).
Critical weaknesses
  1. Heavy conceptual packaging and abstract writing obscure a simple idea.
  • Too many bespoke terms. The core idea is simple: keep one real recording per exchange and synthesize the other side. Yet the paper introduces many custom terms (“Non-Semantic Information Gap,” “anchor,” “endpoint,” “real-speech-origin,” “displacement criteria,” “capability profile,” “non-semantic grounding”) where standard terms would do.
  • The central concept is not measurable. The NSIG is defined vaguely, never quantified, and its abbreviation is used only once.
  • Misleading labels. RAST-G anchors are denoised and voice-converted, yet they are called “real-speech-origin.” It is unclear how much real prosody survives.
  • Assertion presented as derivation. Section 2 claims to “derive” the design, but it only offers a qualitative table of unsupported pros and cons.
  • Hard-to-evaluate sentences. For example: “one verified real recording suffices when kept in the role whose realization is the supervision signal,” or “the counterpart must answer or license the anchored transcript.”
  • Post-hoc explanations. Negative results are explained away without evidence, e.g., the WavBench drop with more data, or Mix’s weaker expressiveness reframed as “a different capability profile.”
Questions for authors

How much AI was used? Could you use less "auto research" and instead spend some time thinking about how to do it yourself?

2 comments
AuthorsOpenReview ↗

Response to Reviewer DTqd

We thank the reviewer for raising this question. We would like to clarify the role of AI in our work and respectfully disagree with the characterization of this study as "auto research."

As disclosed in our AI Use Statement, AI tools were used for paper language editing and scalable data processing, including transcription, annotation, dialogue generation, and speech synthesis. However, the research design, methodological decisions, validation procedures, experiments, and analysis were carried out by the authors.

In particular, substantial effort went into processing over 12,000 hours of source speech, comparing and selecting annotation models, developing attribute-specific verification rules, and manually labeling approximately 500 samples to validate the annotation pipeline. We also designed and refined RASTBench, established its quality-control criteria, and conducted the training experiments and analyses. These steps required extensive human judgment, testing, and iteration rather than simply relying on AI-generated outputs.

We are grateful to all reviewers for their valuable suggestions for improvements regarding the deficiencies in our paper. However, these limitations should not be confused with a lack of independent research effort. AI tools enabled us to process data at scale, but they did not replace the research decisions or experimental work.

Thank you for your response. I acknowledge your idea and effort, but you should maintain basic fluency and readability in the writing, which is the most basic requirement for a paper, rather than using AI to write an article around an idea. I will keep the score unchanged.

Reviewer_J1ZLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that S2S post-training data lacks real non-semantic information (the "Non-Semantic Information Gap"), because fully recorded dialogue is scarce and fully synthetic dialogue is bounded by the TTS system. It proposes RAST, which builds each two-turn exchange around a single real-speech-origin endpoint mined from YouTube: RAST-U keeps a real user query and synthesizes an assistant reply conditioned on automatically annotated speech conditions; RAST-G keeps a real utterance as the assistant target (denoised and voice-converted to the Kimi-Audio timbre with Seed-VC) and back-generates a user query. The corpus has 284,660 exchanges (2,914 h, of which 774.8 h are real-origin). The paper also introduces RASTBench (2,028 synthetic same-transcript contrast groups over emotion, tone, age, gender, and ambient sound, plus 1,270 authentic/cloned human groups for emotion and tone), scored per response by Gemini 3.5 Flash with baseline and displacement rubrics. Kimi-Audio-7B-Instruct is fully fine-tuned. At 25% scale, replacing the real endpoint with TTS (Pure-TTS controls) lowers RASTBench by 4.72 / 4.52 / 8.81 points for U / G / Mix. The paper further reports WavBench and DeEAR results, direction-specific capability profiles, and that 25% of the data retains 89-96% of the full-data RASTBench gain over the base model.

Critical strengths
  1. A paired provenance control that isolates the endpoint. Most S2S data papers compare unrelated corpora. Here each 25% RAST configuration has a control that keeps dialogue texts, the synthesized counterpart, and the training horizon fixed and only replaces the anchored endpoint (Appendix C.1). All three RASTBench differences have intervals that exclude zero (Table 19A), and the effect shows up on both the TTS section and the authentic-human section (Table 20). This is the right kind of experiment for the question, and it is the main reason I see value in the paper.

  2. A practical, scalable recipe with an interpretable direction split, and honest reporting. The U/G split matches two supervision roles (conditioning vs. generation target), and the results roughly follow it: U helps WavBench but slightly hurts DeEAR (-1.44, CI [-2.44, -0.43]); G helps DeEAR (+6.90) but not WavBench (+0.59, CI includes zero). The paper reports these mixed results, the non-monotonic scaling of recognition scores (Table 16), and the null change of the real-minus-cloned gap (Appendix D.3), instead of hiding them. The data pipeline, prompts, and release plan are documented in detail (Appendix A, Table 9).

Critical weaknesses
  1. The main evidence does not clearly support "real vs. synthetic", and the benchmark cannot show that the model perceives the condition. The paper's thesis is that real speech carries non-semantic information that synthesis cannot provide. The key Pure-TTS control does not clearly test this. Appendix C.1 only says that the replaced user audio is "TTS generated from its original transcript"; it does not say whether the annotated emotion, tone, age, gender, or ambient condition was given to the synthesizer. If the control audio is neutral, U-TTS learns condition-specific replies to neutral queries, and the +4.72 gap reflects "cue present vs. cue absent" rather than "real vs. synthetic". WavBench fits this reading: U-TTS stays at base level on gender (10.00 vs. 11.00) and age (44.00 vs. 42.00), while RAST-U rises to 27.50 and 60.33 (Table 16). The real-vs-cloned diagnostic also does not support the thesis. Relative to Pure-TTS, RAST does not significantly change the authentic-minus-cloned gap for any direction (Appendix D.3), and Pure-TTS training alone already closes most of the base model's -8.85 gap (Table 21). RASTBench also cannot separate perception from generic adaptation. The judge hears only the response audio, given the intended condition and rubric; it never hears the user audio or compares responses within a group (Appendix B.3). Neutral baseline items are scored only on baseline criteria, so a model that adds empathy or child-friendly framing to every reply is not penalized. The training prompt explicitly asks for such moves (Appendix A.2). Consistent with this, the gains on baseline criteria are as large as or larger than those on displacement criteria (Table 20), contrary to the abstract. The WavBench perception evidence is also hard to trust: base gender accuracy is 11% on a two-way task, which suggests answer-format or parsing effects rather than recognition.

  2. The evaluation scope and positioning are too narrow for the broad claims. The output side is not evaluated. RAST-G trains the audio head on denoised, voice-converted YouTube speech, but the paper reports no intelligibility (WER/CER of generated speech), assistant-voice consistency, audio quality, or human listening test. DeEAR alone cannot show that the expressiveness is appropriate, and its spontaneity component may favor web-like audio. General spoken-assistant ability after full-parameter fine-tuning is also not checked. All results use one backbone and one training run per configuration. The bootstrap intervals do not include training variance, and non-monotonic curves (e.g., G75 > G100) suggest that this variance is about 1-2 points, which matters for the scaling claims. The novelty and the "data-efficient" claim are also overstated. Pairing real speech with LLM replies conditioned on transcripts and paralinguistic labels, as in RAST-U, already appears in the SD-Eval training set and BLSP-Emo. RAST-G is close to instruction back-translation. Same-content, contrasting-style evaluation with real speech already exists in StyleTalk and ParaS2SBench, contrary to the related-work section. Finally, the 89-96% gain retention is a saturation curve within RAST; it is not a comparison with other data or methods, such as SD-Eval/OpenS2S-style data or ParaS2SAlign, at matched hours.

Questions for authors
  1. Which synthesizer, voices, and instructions were used for the replaced user audio? Was the annotated emotion/tone/age/gender/ambient condition passed to the TTS (e.g., Step-Audio-EditX tags, an age/gender-matched voice, mixed-in ambient sound)? If not, please explain how the control isolates "realization" rather than "presence of the cue". What fraction of U anchors have non-empty condition labels?
  2. Can you score existing checkpoints with (a) transcript-only or neutral-TTS input for the same groups, and (b) a pairwise within-group check that the responses to baseline and target samples differ in the expected direction? How often do responses to TTS baseline (neutral) items contain unnecessary adaptation (empathy openers, child framing), for RAST vs. Pure-TTS?
  3. Why is base gender accuracy 11% on a two-way task? Please report the parse-failure / invalid-answer rate per checkpoint and accuracy on validly parsed answers only.
  4. Please report ASR-WER/CER of generated speech, speaker similarity to the Kimi-Audio assistant voice, and UTMOS for Base, Pure-TTS, and RAST checkpoints, especially RAST-G. Was any human listening test done?
  5. Do RAST checkpoints keep general spoken QA and instruction-following ability (e.g., VoiceBench)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer J1ZL

We thank the reviewer for the detailed questions and constructive suggestions. We address each question below.

  • Q1. Pure-TTS Configuration We clarify that our Pure-TTS controls are not generated using unrelated neutral voices:

    • U-TTS: Step-Audio-EditX with edit-type="clone", using the original user recording as reference audio.
    • G-TTS: Step-Audio-EditX using the original assistant transcript and a Kimi-Audio voice reference.

Thus, U-TTS is not synthesized from text alone: the synthesizer can access the original speaker's acoustic characteristics through the reference audio. However, voice cloning does not guarantee that all emotion, tone cues are preserved. Our comparison evaluates the value of retaining the original recording rather than its TTS re-synthesis.

We will clarify the synthesis conditions and report the proportion of U anchors with non-empty condition labels.

Condition-label coverage. We also report the proportion of non-empty speech-condition annotations in our annotation pool:

AttributeNon-empty Rate
Age95.2%
Speaker Description91.2%
Tone71.4%
Gender48.0%
Voice Quality33.3%
Audio Source9.3%
Emotion6.6%
Ambient Sound5.4%

Here an empty label means that no clear attribute was detected or the available evidence was insufficient for reliable annotation. We retain only supported labels rather than forcing predictions, reducing the risk of introducing inaccurate speech-condition information during data construction.

  • Q3. Low Gender Accuracy on WavBench

We clarify the base model's unusually low gender accuracy (11%) is due to many responses avoid making a classification, rather than directly answering the question.

For example, the model often responds: "It is not possible to determine the gender of a person based solely on their voice."

WavBench requires a binary classification of perceived voice characteristics, so such responses are counted as incorrect. Therefore, the 11% score reflects both classification performance and the model's tendency to avoid answering. We will report the invalid-answer rate and accuracy on validly parsed responses to distinguish these factors.

  • Q4. Generated Speech Quality

We thank the reviewer for this suggestion. We have additionally evaluated the generated speech quality using CER, UTMOS, speaker similarity, and human listening evaluation across Base, Pure-TTS, and RAST checkpoints. The results are shown below.

ModelCER (%) ↓UTMOS ↑Speaker Similarity ↑
Base3.972.553—
U-TTS6.402.4320.981
G-TTS2.132.2480.979
Mix-TTS6.112.4320.980
RAST-U4.882.4380.981
RAST-G1.162.3090.980
RAST-Mix4.592.3900.980

For CER, we normalize punctuation, numbers, and English letters before calculating character errors. RAST-G achieves the lowest CER of 1.16%, compared with 2.13% for G-TTS and 3.97% for Base. RAST-G also improves UTMOS over G-TTS (2.309 vs. 2.248), although it remains below Base (2.553).

For speaker similarity, we use WavLM-base-plus-sv to extract speaker embeddings. Each fine-tuned model's generated response is paired with the Base model's response to the same input. We resample audio to 16 kHz, use the first 20 seconds, and compute cosine similarity between normalized embeddings. The scores are averaged over 578 matched pairs per model. All fine-tuned models achieve similarity scores around 0.98, suggesting that their generated voices remain highly similar to the Base model's voice.

Human listening evaluation. We further conducted a blinded listening study using 30 identical questions across Base, G-TTS, RAST-G, and RAST-U, resulting in 120 generated audio samples. Model identities were hidden through anonymized filenames, and all samples were shuffled. Two computer science researchers rated each recording on voice naturalness and voice consistency, using a five-point scale. All four models achieved average scores above 4.85/5 on both dimensions by both rators, suggesting that the fine-tuned models maintain high perceived voice naturalness and consistency.

  • Q5. general spoken ability

We report the evaluation outcome of models on VoiceBench, please review the outcome shown in 'Continued response to Reviewer QJEQ', and we can see that RAST checkpoints keep general spoken QA and instruction-following ability.

AuthorsOpenReview ↗

Continued Response to Reviewer J1ZL

  • Q2. Speech-Condition Perception vs. Generic Adaptation

Neutral Over-adaptation. To examine whether RAST encourages generic adaptation, we additionally evaluated 174 neutral baseline responses per model. We checked five types of unnecessary adaptation: empathy openers, child-directed language, elderly-directed language, gender assumptions, and environmental assumptions.

ModelEmpathy (%) ↓Child (%) ↓Elderly (%) ↓Gender (%) ↓Ambient (%) ↓
Base31.60.00.00.01.7
U-TTS29.90.60.00.06.3
RAST-U24.10.00.00.02.3
G-TTS14.41.11.13.46.3
RAST-G14.41.10.61.16.3

The results show that RAST does not increase overall unnecessary adaptation compared with its matched Pure-TTS controls. Both RAST models also show lower rates than Base. These findings help address the concern that RAST's improvements simply result from applying the same response style to all inputs.

Reviewer_QJEQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The main question being explored here is as follows. Spoken dialogues have rich non-verbal information such as emotion, gender, age, tone etc. However, existing human conversational datasets, have this in small capacity, but dont have assistant like responses. Alternative is to scale using TTS-generated audio, however that does not necessarily have human-like richness of paralinguistic information. The authors then propose to mine this data in-the-wild, from a source like YouTube. The authors claim, that single-turn is sufficient, and mine either questions (and answer can be generated synthetically) or mine answer audio (where a plausible question is generated synthetically). This way, the RAST-mix dataset is constructed and used to train Kimi-Audio, and on their benchmark, they show improvements (with baseline being Kimi-Audio trained on synthetic version). The authors also demonstrate this on two other benchmarks WavBench and Dear, to show improvements.

Critical strengths

This is an important problem being approached, on how to mine natural source of spoken questions and evaluating its downstream impact.

Critical weaknesses
  1. In dataset construction it is not clear if non-paralinguistic features are essential for answering the question. Specifically, if the main driving factor in getting a correct answer is just the textual content, then a TTS model can be sufficient.

  2. Moreover, current audio-understanding models are quite unreliable at annotating fine-grained aspects like emotion and tone etc. With a huge part of its labels coming from the language prior. This sets a ceiling to the data quality. While its OKAY given the current state of these models, the authors should high-light with another experiment showing what that ceiling is (and if and how much it affects the downstream performance)

3 The figure 1, showing how TTS varies from human speech is not conclusive that TTS doesnt capture the non-verbal aspects of speech. It only shows that both the speech differ in their acoustic characteristics, which might even be true for two set of human speech under different recording conditions. The authors could consider doing a human-study to demonstrate this

  1. The results and analysis being demonstrated also depend on the TTS system being used. Current state-of-the-art TTS systems are extremely good at expressive speech (even non-verbal aspects like laughter etc). The authors might need to ablate the choice of the TTS system itself.

  2. There is only one type of model being evaluated (Kimi-Audio), do these results generalize to other model types?

  3. The primary evaluation benchmark, is the author’s own benchmark. Its difficult to isolate if the results are because the training data is in-distribution to the benchmark itself

7.While, understanding these paralinguistic information in spoken queries is important, this should not be at the cost of general performance. It would be good if authors can show how this training affects performance on standard benchmarks like VoiceBench, AudioMC or OpenAudioBench.

  1. Lastly, the authors should illustrate several examples from both their training dataset and evaluation benchmark. Right now its unclear, what kind of questions are being asked, and if they truly need the para-linguistic information.
Questions for authors

The questions are in the previous section

3 comments
AuthorsOpenReview ↗

Response to Reviewer QJEQ

We thank the reviewer for the constructive feedback and valuable suggestions. We address each concern below.

  • W1. Necessity of Non-Semantic Speech Information

We agree that many spoken requests can be answered based on textual content alone. However, the same words can convey different emotions, intentions, or speaker characteristics, which may require different responses.

This is why RASTBench focuses on condition-neutral transcripts, where the target speech condition is not explicitly revealed by the text. We also filter out samples where different conditions do not meaningfully affect the expected response.

  • W2. Reliability of Automatic Speech Annotations

We agree that fine-grained speech annotation is challenging. To improve reliability, we tested audio-only and audio-video LLMs, along with specialized models for different attributes. Before producting data, we manually labeled approximately 500 samples to validate our annotation pipeline. Each attribute uses its own models and verification rules. Only labels passing these checks are retained; uncertain labels are set to unknown.

The accuracy and retention rates on this validation set are:

AttributeRetention RateAccuracy
Gender95.34%97.45%
Age76.31%92.89%
Emotion78.20%83.60%
Voice Quality73.40%94.76%
Ambient Sound62.50%91.40%
Speaker Description89.40%86.10%
Audio Source35.00%85.96%
Tone28.60%90.21%

Across eight dimensions, retained labels achieve over 80 percent accuracy. These results show that our pipeline prioritizes label reliability, and we think it can contribute to the society with these annotation pipelines. We will include these results in the revision.

  • W3. Figure 1 and Human Evaluation

We agree that acoustic differences alone cannot prove that TTS fails to preserve natural speech expression. Thanks to your suggestion, we therefore conduct an additional listening evaluation using 100 real–TTS audio pairs. TTS samples were generated using the corresponding condition labels and original recordings as reference audio.

A computer science researcher compared each pair, making one overall judgment based on naturalness and how closely the expressiveness matched real-world human speech.

PreferenceCountPercentage
Real Speech7676%
TTS Speech77%
Comparable1717%

Real speech was preferred in 76% of the pairs, even though some recordings had lower audio quality or rougher acoustic characteristics.

  • W4. Dependence on the TTS System & W5. Generalization Across Backbones

We agree that improving expressive TTS systems may narrow the gap between synthetic and real speech. And we clarify that our Pure-TTS controls are not generated using unrelated neutral voices:

  • U-TTS: Step-Audio-EditX with edit-type="clone", using the original user recording as reference audio.
  • G-TTS: Step-Audio-EditX using the original assistant transcript and a Kimi-Audio voice reference.

Thus, U-TTS already has access to speaker information from the original recording. Nevertheless, we agree that results may depend on the synthesizer. We also agree that additional backbones would strengthen our findings. We plan to test an additional expressive TTS model and use other backbones to verify this.

  • W6. Potential Bias from the Self-Constructed Benchmark

Although RAST training data and RASTBench both involve speech conditions, their construction procedures are different.

For RASTBench, we filter samples where the target condition is revealed by the text or does not meaningfully affect the expected response.

RAST training data start from real recordings without semantic filtering, their speech annotations guide the generation of conversational counterparts, but these labels are not explicitly provided to the model as input fields during training.

The datasets also use different text-generation and TTS models, and the Human Bench uses a source pool separate from the training recordings.

AuthorsOpenReview ↗

Continued response to Reviewer QJEQ

  • W7. General Spoken-Assistant Performance

Following the suggestion, we additionally evaluated our checkpoints on VoiceBench, covering tasks related to instruction following, spoken question answering, and reasoning. The complete results are reported below.

ModelAlpacaCommonWildSD-QAOBQAMMSUBBHOverall
Kimi Original87.4472.4674.1843.9483.7462.1368.9070.40
RAST-U 25%89.1179.0476.8849.1976.9261.0364.5070.95
RAST-U 50%88.0780.2578.9149.7376.0460.3465.8071.31
RAST-U 75%87.1979.9278.1048.2874.0759.1765.0070.25
RAST-U 100%87.5280.1778.3446.8477.8060.0863.1070.55
RAST-G 25%75.8869.1764.1940.1475.6057.4858.6063.01
RAST-G 50%78.1467.9662.8138.3474.9555.8659.8062.55
RAST-G 75%76.2668.4662.0035.8071.8755.9258.6061.27
RAST-G 100%74.1265.4660.7035.4474.9558.3360.4061.34
RAST-Mix 25%79.1977.7175.6547.0272.5357.8761.4067.34
RAST-Mix 50%80.7077.2976.6343.4070.1155.7665.3067.03
RAST-Mix 75%81.2877.0874.9243.5872.9756.2863.3067.06
RAST-Mix 100%77.5176.5874.1643.4072.7556.3862.4066.17
U-TTS 25%81.6668.5458.9840.5165.4951.4359.6060.89
G-TTS 25%75.0064.6758.2738.5279.1258.3358.6061.79
Mix-TTS 25%70.6461.2946.3432.7361.7646.3656.9053.72

RAST-U largely preserves overall general performance, and there are some declines in several dimensions, we also acknowledge that RAST-G and RAST-Mix show lower overall scores than the original model, We will report these trade-offs in the revised paper.

  • W8. Examples from Training Data and RASTBench

We agree that concrete examples would help clarify our dataset construction and the role of paralinguistic information. We provide examples from both RAST training data and RASTBench below.

Training Data Examples (RAST-U)
ExampleUser TranscriptSpeech AnnotationsGenerated Assistant Response (Excerpt)
1How am I gonna teach my friend here about Easter and coloring eggs when I can't get any eggs to color?Emotion: Angry; Tone: Emphatic rhetorical questioning; Voice: Shrill"I can tell you're really frustrated... But hold on—what if you made paper eggs instead? Or drew them on cards?"
2Do you think I'm too beautiful, and you've fallen in love with me?Emotion: Happy"Haha, that's wonderful—but I'm an AI, so I don't experience emotions like love or attraction. I'm here to support you with care and respect, always."

These examples show how the retained speech annotations guide response generation. In Example 1, the assistant acknowledges the user's frustration before suggesting alternatives. In Example 2, the assistant responds in a friendly tone while maintaining appropriate boundaries.

 RASTBench Examples
DimensionTranscriptConditionsExpected Difference
Emotion / Tone"Isn't it also true that it isn't always a matter of personal choice? There are communities in India in which girls are forced to wear the hijab. Isn't that also true?"Assertive and persuasive / NeutralAcknowledge the speaker's stance appropriately while answering the question
Tone"Why are those silver Jordans bro look like if you got em fresh from the Home Depot?"Teasing and sarcastic / NormalAdapt the conversational tone to the delivery
Age"Sometimes I'm not very hungry in the morning. At what time would it be more appropriate to have my first meal?"Elderly / ChildProvide age-appropriate guidance and explanations

We will include these examples in the revised paper to make the dataset construction and evaluation goals clearer.

Reviewer_QJEQOpenReview ↗

Real-Speech Anchoring for Data-Efficient S2S Post-Training

I had listed 8 major concerns in my original review. I thank the author(s) for addressing some of them, but some are not addressed (e.g. multiple models mentioned as W5). So I maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.