- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents ScenA, a reference-driven multi-speaker audio scene generation framework that enables natural language control over speaker identities, dialogue, and environmental sounds. Built upon a pretrained audio diffusion transformer, the method introduces a lightweight reference-conditioning mechanism and identifies a Reference Shortcut phenomenon that can prevent the model from learning correct text-reference associations. The authors address this issue through high-noise-biased timestep sampling. Experimental results demonstrate strong performance on multi-speaker dialogue benchmarks and promising capabilities for generating complex audio scenes
Critical strengths
-
Interesting and novel task formulation. Extending multi-speaker TTS toward reference-driven audio scene generation is a compelling direction with practical significance. The proposed natural language interface is particularly appealing.
-
Simple and insightful methodology. The reference-conditioning design is elegant, and the analysis of the Reference Shortcut provides useful insights into reference-conditioned flow-matching models.
-
Promising experimental results. The method achieves strong performance on speaker-binding metrics, while the accompanying audio demonstrations are impressive and illustrate the potential of the proposed framework.
Critical weaknesses
- Insufficient methodological and implementation details.
Several important aspects of the proposed framework require further clarification. In particular, the paper does not sufficiently explain how long and complex natural language instructions are processed, how textual descriptions are temporally aligned with audio events, or whether the generation process relies on particular instruction patterns. Although Appendix A.6 provides a limited prompt-format analysis, the extent of free-form instruction flexibility remains unclear.
Additionally, the motivation for adopting the audio stream of LTX-2.3, rather than a dedicated audio generation backbone, could be better articulated.
- Limited reproducibility and transparency of the training-data preparation pipeline.
The proposed framework relies heavily on automatically generated multi-reference scene descriptions. However, Section 3.4 provides only a high-level overview of the data preparation process. Important details, including the captioning models, annotation prompts, speaker-reference matching procedures, and data filtering criteria, are insufficiently documented.
Furthermore, the quality and reliability of the generated annotations are not quantitatively assessed. Since accurate speaker attribution and event descriptions are essential for learning text-reference binding, additional details and validation of the annotation pipeline are necessary to support the reproducibility and reliability of the proposed method.
- Insufficient analysis of reference-audio contamination and fine-grained scene control.
The paper demonstrates that ScenA can operate with in-the-wild reference recordings containing background sounds. However, it remains unclear how speaker identity is distinguished from environmental information, and whether background sounds in reference recordings can be selectively preserved or suppressed. This is especially relevant given that the method directly incorporates reference audio latents without explicit disentanglement [1].
Moreover, while the paper evaluates sound-event generation, emotional expression, and overlapping speech, the current metrics provide limited evidence of fine-grained temporal control and speaker-specific event attribution. For example, detecting laughter does not necessarily establish whether it is produced by the intended reference speaker [2].
These aspects would benefit from clearer methodological explanations and more targeted evaluations.
[1] Zhang et al. Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Prediction. ICASSP 2025. Paper. Relevant to background preservation and removal in reference-conditioned speech generation.
[2] Wu et al. Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech. SLT 2024. Paper. Relevant to temporal emotional control and nonverbal vocalization generation.
Questions for authors
-
Instruction conditioning: How are long or complex natural language instructions handled? Does the model support flexible temporal descriptions, and how is the correspondence between textual events and generated audio established without explicit temporal alignment?
-
Data preparation and reproducibility: Could the authors provide the specific models, prompts, filtering strategies, and validation procedures used in the multi-reference captioning pipeline? How is the accuracy of speaker-reference assignments and automatically generated scene descriptions ensured?
-
Reference-audio handling: How does the model handle background music, environmental sounds, or nonverbal vocalizations already present in reference recordings? Can these components be controlled independently of speaker identity?
-
Backbone selection and evaluation: What motivates the choice of LTX-2.3 over an audio-only foundation model? Could the authors also clarify the limitations of the current evaluation regarding temporal event alignment and speaker-specific vocalizations?
3 comments
Response to Authors
Thank you for the response and for the additional experiments. However, I do not feel that my concerns have been fully resolved. In particular:
- Instruction conditioning: My question was how long and complex natural language instructions are processed, and whether the model relies on particular instruction patterns. The response describes how individual instruction elements are expressed, but does not address the flexibility and robustness of free-form instructions.
- Data preparation: A 50-sample self-audit (47/50) is too small to establish the reliability of automatically generated annotations across the training set. It remains unclear whether the captioning prompts and full filtering pipeline are fully disclosed.
- Reference-audio handling: The case of nonverbal vocalizations already present in reference audio is not addressed.
- Evaluation: The human study is appreciated, but the low success rates for emotion presence (66.7%) and background timing (62.7%) are not discussed or explained.
In addition, I would kindly suggest that the authors structure their response following the order of the questions raised in my review, and clearly indicate what has been changed (e.g., section/appendix numbers and the nature of each revision). As I do not have the capacity to re-read the entire revised manuscript to locate the changes myself, I hope this can be made explicit.
Given the above, I will maintain my original scores.
Follow-up: Instruction Flexibility, Data Reliability, and Reference-Audio Control
Dear Reviewer Pmk9,
Thank you for the follow-up and for considering the additional experiments.
1. Instruction conditioning
The model accepts free-form instructions without a fixed template and does not require any task-specific preprocessing of the prompt. This is supported by the controlled wording tests in Appendix A.6 and the varied composed instructions on the supplementary demo page. The full prompt conditions generation through the pretrained text encoder and cross-attention, and reference transcripts are not required. Examples with corresponding generated audio on the supplementary demo page include:
Both speakers desperately try to suppress laughter during something serious. The speaker from reference 1 whispers, holding back giggles: "We cannot laugh right now, I'm serious." The speaker from reference 2 snickers under his breath: "I'm trying, I'm trying!" Both burst out laughing simultaneously.
A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.
The speaker from reference 1 announces dramatically: "Ladies and gentlemen, prepare to be amazed... behold!" A loud, sharp WHOOSH followed by a hiss as a thick puff of stage smoke bursts out. The speaker from reference 2 gasps and says: "Wait, where did the rabbit come from?!" The speaker from reference 1 chuckles smugly: "A magician never tells."
2. Data preparation
We performed an additional random audit of 200 training examples and their references, finding correct reference-to-speaker and caption bindings in 94.5% (189/200). Together with the earlier caption-binding check (47/50), 236/250 cases received positive judgments across the two checks (Appendix A.13).
3. Reference-audio handling
WildRef references contain background sounds, vocalizations, and other unwanted content. Upon manual inspection of speech-only dialogue generations conditioned on WildRef references, we found no unintended reproduction of these reference sounds.
We also conducted a new experiment in which we added recorded coughs or laughter to clean references and generated dialogue using them. The results showed no increase in unwanted vocalizations compared with controls, while speaker binding remained strong. New Appendix A.15, Table 15 gives the full procedure and results.
4. Evaluation
Emotion recognition succeeds in 81.8% (54/66) of definite Yes/No judgments, compared with the reported 66.7% (54/81) when “Cannot tell” responses are also counted as unsuccessful. The breakdown by emotion shows that anger is the hardest case, with recognition at 54.8%, compared with 75.0% for joy and 73.3% for surprise. The reference voices used in this evaluation come from LibriSpeech audiobook recordings, and we suspect their relatively calm delivery may make stronger emotions, particularly anger, harder for the model to produce. Our training captions may also be a contributing factor, as they may describe whether an emotion is present more accurately than how strongly it should be expressed.
Backgrounds are heard in 79.7% of judgments, while 62.7% measures whether they continue throughout the dialogue when heard. Manual inspection identified examples where the background starts slightly late, which may explain why listeners marked it as not continuous throughout. Appendix A.11, Table 11 provides the analysis and full breakdown. The results support useful emotion and background control, alongside particularly strong vocalization and discrete-event results.
We hope these additional checks and clarifications address your remaining concerns. We remain available throughout the discussion period and will respond promptly to any further questions.
Response to Reviewer Pmk9
Thank you for the thoughtful review and for recognizing the task formulation, simple methodology, and reference-shortcut insight. We have submitted a revised manuscript incorporating the additional studies, results, and clarifications. We address your concerns below.
Instruction handling and temporal correspondence
Speaker assignments, sentence order, relative event timing, and overlap are all expressed in free-form text. The model learns these relationships from caption-audio pairs through cross-attention and generates the scene jointly, without an explicit event timeline. Appendix A.13 gives training captions with instructions such as an opening tongue-click and a final finger snap. Appendix A.6 examines alternative prompt formats. The supplementary audio examples pair prompts and reference audio with generated scenes spanning dialogue, vocalizations, emotion, ambience, and overlap.
Backbone choice
The intended interface calls for complete scenes combining speech, music, environmental sounds, and expressive vocalizations under natural-language conditioning. A backbone pretrained on in-the-wild soundtracks provides these scene priors, allowing fine-tuning to focus on multi-reference binding. These considerations motivated the choice of the audio stream of LTX-2.3.
Dataset construction and reliability
Revised Appendix A.13 adds the embedding model, matching and filtering procedures, caption grounding, and four training-caption examples. To assess annotation reliability, we additionally audited training examples and their references, finding 94% (47/50) correct caption-speaker bindings among inspected examples. Together with the checkpoint, inference code, baseline configurations, benchmark, evaluation code, and reproduction scripts, these details support independent peer review, benchmark verification, and future work.
Reference contamination and background control
Captions specify what to reuse, such as “The speaker from reference_1 says…” for voice and “The soft, romantic piano and string music from reference_2 persists” for background continuation. Appendix A.14 reports an additional controlled test with one music-containing voice reference and one clean reference. BEATs detects music in 0/100 speech-only generations and 61/100 generations requesting continued music. The appendix reports the accompanying automatic speech and speaker scores. WildRef (Table 2) separately tests natural noisy references, with examples in the supplementary material.
Scene-control evaluation and attribution
Table 6 reports speech preservation alongside event detection. Vocalizations reduce cpSIM by 0.010, SIM-O by 0.008, and ACC by 2.1 percentage points, with unchanged WER. We also conducted an additional human study on 85 CoVoMix-derived scenes to assess target presence, temporal placement, vocalization identity, and intelligibility directly. Across 390 completed judgments, we measure how often each requested feature is heard. When heard, we also assess whether vocalizations, discrete events, emotions, and interruptions occur at the requested point, and whether background sounds continue throughout. Presence and timing success rates are, respectively, 97.6%/98.8% for vocalizations, 89.7%/91.4% for discrete events, 66.7%/87.0% for emotion, 73.6%/88.7% for overlap, and 79.7%/62.7% for backgrounds. Raters identified the intended speaker from coughs or sighs with an 87.3% success rate (55/63), among judgments selecting either reference voice. Mean dialogue intelligibility is 4.49/5. Appendix A.11, Tables 11–12, gives full task-wise results and the questionnaire. These additional human results further support our claims of speaker-aware scene control and temporal instruction following.
We hope the revision and additional checks address your concerns, and we welcome further feedback during the discussion.