acceptodds
Under review as a conference paper at ICLR 2027

ScenA: Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

Abstract

Existing reference-conditioned multi-speaker systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations. Our method, , conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on multiple reference voices and a free-form natural language prompt that describes an entire multi-speaker audio scene. Leveraging such a foundational model allows us to inherit its capacity for natural, non-studio audio: background noise, room acoustics, overlapping dialogue, and spontaneous paralinguistic events, while adding multi-speaker control without any per-turn structure. Concretely, reference latents are concatenated into the model's token sequence and distinguished by lightweight identity-aware positional encodings. However, we identify a critical obstacle to this approach: the . During training under standard noise schedules, the model can identify the matching reference by acoustic similarity to the noisy target, bypassing the text prompt entirely. We address this with a high-noise-biased timestep distribution that forces the model to rely on the text prompt for speaker assignment. We first evaluate broad scene-generation capabilities, demonstrating control over vocalizations, discrete events, emotional delivery, ambience, and simultaneous speech. Additionally, we evaluate on the CoVoMix2-Dialogue benchmark, where it outperforms specialized speech-only multi-speaker systems. Our results demonstrate the advantage of using a general-purpose audio model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline.

Primary area:
generative models
Keywords:
Computer Vision, References, Generative AI, Audio, Diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Pmk9OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents ScenA, a reference-driven multi-speaker audio scene generation framework that enables natural language control over speaker identities, dialogue, and environmental sounds. Built upon a pretrained audio diffusion transformer, the method introduces a lightweight reference-conditioning mechanism and identifies a Reference Shortcut phenomenon that can prevent the model from learning correct text-reference associations. The authors address this issue through high-noise-biased timestep sampling. Experimental results demonstrate strong performance on multi-speaker dialogue benchmarks and promising capabilities for generating complex audio scenes

Critical strengths
  1. Interesting and novel task formulation. Extending multi-speaker TTS toward reference-driven audio scene generation is a compelling direction with practical significance. The proposed natural language interface is particularly appealing.

  2. Simple and insightful methodology. The reference-conditioning design is elegant, and the analysis of the Reference Shortcut provides useful insights into reference-conditioned flow-matching models.

  3. Promising experimental results. The method achieves strong performance on speaker-binding metrics, while the accompanying audio demonstrations are impressive and illustrate the potential of the proposed framework.

Critical weaknesses
  1. Insufficient methodological and implementation details.

Several important aspects of the proposed framework require further clarification. In particular, the paper does not sufficiently explain how long and complex natural language instructions are processed, how textual descriptions are temporally aligned with audio events, or whether the generation process relies on particular instruction patterns. Although Appendix A.6 provides a limited prompt-format analysis, the extent of free-form instruction flexibility remains unclear.

Additionally, the motivation for adopting the audio stream of LTX-2.3, rather than a dedicated audio generation backbone, could be better articulated.

  1. Limited reproducibility and transparency of the training-data preparation pipeline.

The proposed framework relies heavily on automatically generated multi-reference scene descriptions. However, Section 3.4 provides only a high-level overview of the data preparation process. Important details, including the captioning models, annotation prompts, speaker-reference matching procedures, and data filtering criteria, are insufficiently documented.

Furthermore, the quality and reliability of the generated annotations are not quantitatively assessed. Since accurate speaker attribution and event descriptions are essential for learning text-reference binding, additional details and validation of the annotation pipeline are necessary to support the reproducibility and reliability of the proposed method.

  1. Insufficient analysis of reference-audio contamination and fine-grained scene control.

The paper demonstrates that ScenA can operate with in-the-wild reference recordings containing background sounds. However, it remains unclear how speaker identity is distinguished from environmental information, and whether background sounds in reference recordings can be selectively preserved or suppressed. This is especially relevant given that the method directly incorporates reference audio latents without explicit disentanglement [1].

Moreover, while the paper evaluates sound-event generation, emotional expression, and overlapping speech, the current metrics provide limited evidence of fine-grained temporal control and speaker-specific event attribution. For example, detecting laughter does not necessarily establish whether it is produced by the intended reference speaker [2].

These aspects would benefit from clearer methodological explanations and more targeted evaluations.

[1] Zhang et al. Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Prediction. ICASSP 2025. Paper. Relevant to background preservation and removal in reference-conditioned speech generation.

[2] Wu et al. Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech. SLT 2024. Paper. Relevant to temporal emotional control and nonverbal vocalization generation.

Questions for authors
  1. Instruction conditioning: How are long or complex natural language instructions handled? Does the model support flexible temporal descriptions, and how is the correspondence between textual events and generated audio established without explicit temporal alignment?

  2. Data preparation and reproducibility: Could the authors provide the specific models, prompts, filtering strategies, and validation procedures used in the multi-reference captioning pipeline? How is the accuracy of speaker-reference assignments and automatically generated scene descriptions ensured?

  3. Reference-audio handling: How does the model handle background music, environmental sounds, or nonverbal vocalizations already present in reference recordings? Can these components be controlled independently of speaker identity?

  4. Backbone selection and evaluation: What motivates the choice of LTX-2.3 over an audio-only foundation model? Could the authors also clarify the limitations of the current evaluation regarding temporal event alignment and speaker-specific vocalizations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Pmk9

Thank you for the thoughtful review and for recognizing the task formulation, simple methodology, and reference-shortcut insight. We have submitted a revised manuscript incorporating the additional studies, results, and clarifications. We address your concerns below.

Instruction handling and temporal correspondence

Speaker assignments, sentence order, relative event timing, and overlap are all expressed in free-form text. The model learns these relationships from caption-audio pairs through cross-attention and generates the scene jointly, without an explicit event timeline. Appendix A.13 gives training captions with instructions such as an opening tongue-click and a final finger snap. Appendix A.6 examines alternative prompt formats. The supplementary audio examples pair prompts and reference audio with generated scenes spanning dialogue, vocalizations, emotion, ambience, and overlap.

Backbone choice

The intended interface calls for complete scenes combining speech, music, environmental sounds, and expressive vocalizations under natural-language conditioning. A backbone pretrained on in-the-wild soundtracks provides these scene priors, allowing fine-tuning to focus on multi-reference binding. These considerations motivated the choice of the audio stream of LTX-2.3.

Dataset construction and reliability

Revised Appendix A.13 adds the embedding model, matching and filtering procedures, caption grounding, and four training-caption examples. To assess annotation reliability, we additionally audited training examples and their references, finding 94% (47/50) correct caption-speaker bindings among inspected examples. Together with the checkpoint, inference code, baseline configurations, benchmark, evaluation code, and reproduction scripts, these details support independent peer review, benchmark verification, and future work.

Reference contamination and background control

Captions specify what to reuse, such as “The speaker from reference_1 says…” for voice and “The soft, romantic piano and string music from reference_2 persists” for background continuation. Appendix A.14 reports an additional controlled test with one music-containing voice reference and one clean reference. BEATs detects music in 0/100 speech-only generations and 61/100 generations requesting continued music. The appendix reports the accompanying automatic speech and speaker scores. WildRef (Table 2) separately tests natural noisy references, with examples in the supplementary material.

Scene-control evaluation and attribution

Table 6 reports speech preservation alongside event detection. Vocalizations reduce cpSIM by 0.010, SIM-O by 0.008, and ACC by 2.1 percentage points, with unchanged WER. We also conducted an additional human study on 85 CoVoMix-derived scenes to assess target presence, temporal placement, vocalization identity, and intelligibility directly. Across 390 completed judgments, we measure how often each requested feature is heard. When heard, we also assess whether vocalizations, discrete events, emotions, and interruptions occur at the requested point, and whether background sounds continue throughout. Presence and timing success rates are, respectively, 97.6%/98.8% for vocalizations, 89.7%/91.4% for discrete events, 66.7%/87.0% for emotion, 73.6%/88.7% for overlap, and 79.7%/62.7% for backgrounds. Raters identified the intended speaker from coughs or sighs with an 87.3% success rate (55/63), among judgments selecting either reference voice. Mean dialogue intelligibility is 4.49/5. Appendix A.11, Tables 11–12, gives full task-wise results and the questionnaire. These additional human results further support our claims of speaker-aware scene control and temporal instruction following.

We hope the revision and additional checks address your concerns, and we welcome further feedback during the discussion.

Reviewer_Pmk9OpenReview ↗

Response to Authors

Thank you for the response and for the additional experiments. However, I do not feel that my concerns have been fully resolved. In particular:

  1. Instruction conditioning: My question was how long and complex natural language instructions are processed, and whether the model relies on particular instruction patterns. The response describes how individual instruction elements are expressed, but does not address the flexibility and robustness of free-form instructions.
  2. Data preparation: A 50-sample self-audit (47/50) is too small to establish the reliability of automatically generated annotations across the training set. It remains unclear whether the captioning prompts and full filtering pipeline are fully disclosed.
  3. Reference-audio handling: The case of nonverbal vocalizations already present in reference audio is not addressed.
  4. Evaluation: The human study is appreciated, but the low success rates for emotion presence (66.7%) and background timing (62.7%) are not discussed or explained.

In addition, I would kindly suggest that the authors structure their response following the order of the questions raised in my review, and clearly indicate what has been changed (e.g., section/appendix numbers and the nature of each revision). As I do not have the capacity to re-read the entire revised manuscript to locate the changes myself, I hope this can be made explicit.

Given the above, I will maintain my original scores.

AuthorsOpenReview ↗

Follow-up: Instruction Flexibility, Data Reliability, and Reference-Audio Control

Dear Reviewer Pmk9,

Thank you for the follow-up and for considering the additional experiments.

1. Instruction conditioning

The model accepts free-form instructions without a fixed template and does not require any task-specific preprocessing of the prompt. This is supported by the controlled wording tests in Appendix A.6 and the varied composed instructions on the supplementary demo page. The full prompt conditions generation through the pretrained text encoder and cross-attention, and reference transcripts are not required. Examples with corresponding generated audio on the supplementary demo page include:

Both speakers desperately try to suppress laughter during something serious. The speaker from reference 1 whispers, holding back giggles: "We cannot laugh right now, I'm serious." The speaker from reference 2 snickers under his breath: "I'm trying, I'm trying!" Both burst out laughing simultaneously.

A farm at sunrise: a rooster crows. Chickens cluck softly throughout. The speaker from reference 1 says with a yawn: "Way too early for this." The speaker from reference 2 chuckles: "Welcome to country life." The rooster crows again.

The speaker from reference 1 announces dramatically: "Ladies and gentlemen, prepare to be amazed... behold!" A loud, sharp WHOOSH followed by a hiss as a thick puff of stage smoke bursts out. The speaker from reference 2 gasps and says: "Wait, where did the rabbit come from?!" The speaker from reference 1 chuckles smugly: "A magician never tells."

2. Data preparation

We performed an additional random audit of 200 training examples and their references, finding correct reference-to-speaker and caption bindings in 94.5% (189/200). Together with the earlier caption-binding check (47/50), 236/250 cases received positive judgments across the two checks (Appendix A.13).

3. Reference-audio handling

WildRef references contain background sounds, vocalizations, and other unwanted content. Upon manual inspection of speech-only dialogue generations conditioned on WildRef references, we found no unintended reproduction of these reference sounds.

We also conducted a new experiment in which we added recorded coughs or laughter to clean references and generated dialogue using them. The results showed no increase in unwanted vocalizations compared with controls, while speaker binding remained strong. New Appendix A.15, Table 15 gives the full procedure and results.

4. Evaluation

Emotion recognition succeeds in 81.8% (54/66) of definite Yes/No judgments, compared with the reported 66.7% (54/81) when “Cannot tell” responses are also counted as unsuccessful. The breakdown by emotion shows that anger is the hardest case, with recognition at 54.8%, compared with 75.0% for joy and 73.3% for surprise. The reference voices used in this evaluation come from LibriSpeech audiobook recordings, and we suspect their relatively calm delivery may make stronger emotions, particularly anger, harder for the model to produce. Our training captions may also be a contributing factor, as they may describe whether an emotion is present more accurately than how strongly it should be expressed.

Backgrounds are heard in 79.7% of judgments, while 62.7% measures whether they continue throughout the dialogue when heard. Manual inspection identified examples where the background starts slightly late, which may explain why listeners marked it as not continuous throughout. Appendix A.11, Table 11 provides the analysis and full breakdown. The results support useful emotion and background control, alongside particularly strong vocalization and discrete-event results.

We hope these additional checks and clarifications address your remaining concerns. We remain available throughout the discussion period and will respond promptly to any further questions.

Reviewer_B875OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCENA adapts an audio flow-matching transformer pretrained on in-the-wild soundtracks to generate a complete multi-speaker audio scene from a free-form prompt and up to three reference voices. Clean reference latents are concatenated with the noised target tokens, and learned slot embeddings identify the references. The prompt describes speaker turns, dialogue and scene sounds without per-turn speaker tags at inference. The paper identifies a “reference shortcut”: under ordinary timestep sampling, moderately noised target audio remains similar enough to its matching reference for the model to learn speaker assignment from acoustics while ignoring the text. A probe reports at least 90% reference-match accuracy through roughly timestep 0.86. The proposed Beta-plus-Uniform noise sampler puts more training weight near pure noise, where text is needed to establish the binding.

Critical strengths

The reference shortcut is the strongest contribution. The model can obtain a low denoising loss by matching residual target acoustics to a reference, yet generation begins at pure noise, where that path is unavailable. The probe and controlled schedule sweep make this more than an intuition: binding-aware metrics improve as training shifts toward high noise, and the proposed schedule achieves the best cpWER and speaker attribution accuracy among the tested schedules on the ablation subset. The paper also checks a very-high-noise logit-normal control, which shows that the exact Beta-plus-Uniform formula is not the only way to improve binding and helps delimit the claim.

The main dialogue evaluation is substantial. SCENA is compared on all 1,000 CoVoMix2 items, with each speech baseline receiving the same dialogue in its native format. Its cpSIM of 0.628 exceeds the closest reported baseline at 0.582, and its speaker attribution accuracy is 0.916 versus 0.895 for the strongest baseline on that metric. On noisier references, SCENA retains the highest cpSIM and WER is still low, though MOSS-TTSD is better on cpWER and attribution accuracy. The listening test, covering 385 judgments from 53 raters over 34 dialogue items, supplies some perceptual support beyond automatic metrics.

Critical weaknesses
  1. The evidence for broad scene control is thinner than the dialogue evidence. Requested coughs, laughter, doorbells and other events are judged mainly through BEATs maximum-frame confidence, emotions through a RAVDESS-trained classifier, and overlap through an activity detector. These measurements show a response to the prompt but do not establish that the sound occurs at the requested moment, belongs to the requested speaker or integrates naturally with the dialogue. The subjective test asks raters to choose the more natural and reference-similar dialogue; it does not test the paper's strongest claim about controllable ambience, emotion and overlapping speech in complex scenes. Sustained ambience also reduces speaker similarity and attribution, according to the paper's own analysis.

  2. The benchmark comparison and reproducibility both have scope limits. In the public test, 709 of the 1,000 dialogues are truncated to complete turn boundaries to fit the 20-second generation cap. This is described transparently, but it changes the task seen by every system and makes long-scene claims difficult to assess. Exact retraining is also hard to verify because the in-house audio cannot be redistributed. The released checkpoint and evaluation code permit inference replication, but the paper uses diarization and generated captions to build reference-to-target supervision offline; errors in that pipeline could affect which speaker the model learns to associate with a slot. A small audit of those labels would strengthen the shortcut and attribution story.

Questions for authors

Could the authors run a blinded human evaluation of the scene-control tasks in Section 4.4, asking separately whether the requested event happened, whether it occurred at the correct time and for the correct speaker, and whether the dialogue remained intelligible? This would materially affect the claim that the free-form interface provides reliable holistic scene control.

How often are the automatically constructed training references and captions wrong about speaker identity, and how sensitive is speaker binding to that noise? Please also report the main dialogue results separately on the 291 unchanged and 709 truncated items, since the current pooled score does not show whether the 20-second cap drives a large part of the measured advantage.

4 comments
AuthorsOpenReview ↗

Response to Reviewer B875

Thank you for the careful review and for recognizing the reference shortcut and dialogue evaluation as strengths. We have submitted a revised manuscript incorporating the additional studies, results, and clarifications. We address your concerns below.

Evidence for broad scene control

We conducted an additional human study on 85 CoVoMix-derived scenes with vocalizations, discrete events, emotion, overlap, and continuous backgrounds. Raters assessed whether requested features are audible and correctly placed, alongside speaker attribution for vocalizations and dialogue intelligibility. Across 390 completed judgments, mean intelligibility is 4.49/5. We measure how often each requested feature is heard. When heard, we also assess whether vocalizations, discrete events, emotions, and interruptions occur at the requested point, and whether background sounds continue throughout. Presence and timing success rates are, respectively, 97.6%/98.8% for vocalizations, 89.7%/91.4% for discrete events, 66.7%/87.0% for emotion, 73.6%/88.7% for overlap, and 79.7%/62.7% for backgrounds. Appendix A.11, Tables 11–12, provides full task-wise results, voice-attribution outcomes, and the questionnaire. These additional human results further support our claims of controllable scene generation while preserving intelligible dialogue.

Benchmark truncation and split results

All systems receive the same retained dialogue in their native input formats. Truncation retains complete speaker turns. We checked the 291 unchanged and 709 truncated items separately: ScenA leads on cpWER, cpSIM, ACC, and WER on both splits, and is best or tied-best on SIM-O. MOSS-TTSD leads on UTMOS. The advantage therefore also holds without truncation. Revised Appendix A.12, Tables 13–14, contains the full results for these six metrics.

Dataset construction, reproducibility, and annotation reliability

Revised Appendix A.13 now documents the SpeechBrain ECAPA-TDNN embedding model, reference matching and filtering, caption grounding, and four illustrative training captions. To assess annotation reliability, we additionally audited training examples and their references, finding 94% (47/50) correct caption-speaker bindings among inspected examples.

Together with these additions, the full reproducibility package provides the checkpoint, inference code, baseline configurations, complete 1,000-item benchmark, evaluation code, and reproduction scripts for independent peer review, benchmark verification, and future work.

We hope the split analysis and added documentation address your concerns, and we would appreciate your reconsideration as the discussion progresses.

Before the discussion closes, we would like to highlight the additional evidence now included in our revised manuscript. Alongside the benchmark-split results and dataset-construction details provided in response to your concerns, we added a human study of broader scene control, an expanded training-data audit, and controlled tests of unwanted vocalization transfer from references. We believe these additions strengthen the evidence for generating varied audio scenes through free-form natural-language instructions.

I thank the authors for the response and the additional analyses. The truncation split, the dataset-construction details, and the new scene-control human study address part of my concerns.

However, the new human study also shows that scene control is uneven: emotion and overlap are heard in only 66.7% and 73.6% of cases, and backgrounds are correctly sustained in only 62.7% of cases where they are heard. This supports controllability for vocalizations and discrete events, but not yet reliable holistic control over complex scenes. In addition, the annotation audit covers only 50 examples and does not examine how sensitive speaker binding is to the observed label errors. I therefore maintain my score of 2 (weak reject).

AuthorsOpenReview ↗

Additional Evidence on Scene Control and Annotation Reliability

Dear Reviewer B875,

Thank you for considering the additional analyses. We would like to highlight that we further addressed these two concerns in the latest revision, following our discussion with Reviewer Pmk9. For convenience, we summarize the additional evidence below.

1. Scene-control evaluation

Emotion recognition succeeds in 81.8% (54/66) of definite Yes/No judgments, compared with the reported 66.7% (54/81) when “Cannot tell” responses are also counted as unsuccessful. The breakdown by emotion shows that anger is the hardest case, with recognition at 54.8%, compared with 75.0% for joy and 73.3% for surprise. The reference voices used in this evaluation come from LibriSpeech audiobook recordings, and we suspect their relatively calm delivery may make stronger emotions, particularly anger, harder for the model to produce. Our training captions may also be a contributing factor, as they may describe whether an emotion is present more accurately than how strongly it should be expressed.

Backgrounds are heard in 79.7% of judgments, while 62.7% measures whether they continue throughout the dialogue when heard. Manual inspection identified examples where the background starts slightly late, which may explain why listeners marked it as not continuous throughout. Appendix A.11, Table 11 provides the analysis and full breakdown.

The results support useful emotion and background control, alongside particularly strong vocalization and discrete-event results.

2. Annotation reliability

We performed an additional random audit of 200 training examples and their references, finding correct reference-to-speaker and caption bindings in 94.5% (189/200). Together with the earlier caption-binding check (47/50), 236/250 cases received positive judgments across the two checks (Appendix A.13).

We hope these additional results and clarifications address your remaining concerns and would appreciate your reconsideration in light of the latest revision.

Reviewer_XsCcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ScenA, which jointly generates audio scenes containing multi-speaker dialogue and ambient sounds from voice references and natural-language descriptions. Built on the audio stream of LTX-2.3, the method concatenates reference audio latents with the noised target sequence and introduces learned reference-slot embeddings. The authors identify a “reference shortcut,” whereby the model can match references to acoustic information retained in the noised target, bypassing the speaker assignment specified in the text. They propose a high-noise-biased timestep distribution to mitigate this issue. Experiments cover dialogue generation, robustness to noisy references, and scene controls including vocalizations, emotional delivery, ambience, and overlapping speech.

Critical strengths

The task formulation has practical value. Jointly controlling reference-speaker identities, dialogue, and ambient sounds through a unified natural-language interface extends the capabilities of conventional multi-speaker speech synthesis. The ablation studies are reasonably comprehensive: the synthetic ordering experiment demonstrates that low reconstruction loss can coexist with poor instruction following, while the noise-schedule, reference-slot encoding, and data-augmentation experiments provide support for the training recipe. The dialogue-generation results are also strong.

Critical weaknesses

The method primarily achieves its capabilities by constructing paired training data and fine-tuning a pretrained DiT. The authors identify the “reference shortcut” during fine-tuning and propose a remedy. In the synthetic ordering experiment, standard training yields near-chance ordering accuracy, whereas high-noise training reaches approximately 95%, making the phenomenon clear in this controlled setting. In the real-dialogue noise-schedule ablation, however, ACC improves from 83.0% to 86.6%, which is a useful but relatively modest gain. I therefore find the characterization of the inference failure as “catastrophic” overstated for the real-dialogue setting. The evaluation also requires clarification on the points raised below.

Questions for authors

When computing ACC and cpSIM, how are diarization clusters mapped to reference identities? If a generated dialogue consistently swaps the two reference voices, would these metrics penalize the error? Could you provide a simple sanity check that swaps either the reference order or the reference identities specified in the prompt?

Does the reference-shortcut probe retain its reported accuracy when the text caption is removed? Which Transformer parameters are frozen during probe training? Additionally, how do you reconcile the description of catastrophic inference failure with the relatively strong performance of the standard timestep distribution in Figure 4?

Could a small-scale human evaluation verify whether the requested vocalizations and overlapping utterances are produced by the intended reference speakers? For ambience control, please report results both with and without the additional ambience reference, and discuss the impact on speaker-identity preservation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XsCc

Thank you for the careful review and concrete questions. We have submitted a revised manuscript incorporating the additional studies, results, and clarifications. We address your concerns below.

Shortcut severity and the standard-sampler baseline

We removed “catastrophic” from the real-dialogue discussion. The wording was motivated by qualitative speaker-binding failures, whose perceptual severity is not fully captured by automatic metrics. The severe controlled failure is isolated in Appendix A.4, Figures 7–8: standard training reaches low reconstruction loss but near-chance ordering accuracy.

Addressing the shortcut moves ScenA from second-worst to best in reference-voice similarity, while achieving the highest ACC among the compared systems. This is a meaningful practical gain beyond the synthetic setting. Figure 4 also shows progressive ACC improvements and cpWER reductions as standard log-normal training shifts toward higher noise.

Metric mapping and swap sanity check

ACC assigns each diarization cluster to the reference with the nearest speaker embedding and checks aligned words against their prompted speakers. Both complete and partial assignment swaps count as errors. cpSIM matches the voices by similarity without checking sentence assignments. Reversing reference identities only at scoring time on the same 64 outputs changes ACC from 91.1% to 8.9%, while cpSIM stays at 0.613.

Caption-free probe and frozen parameters

All parameters are frozen except the two-layer MLP classifier head. Revised Appendix A.3 now states this explicitly and includes the caption-removal results. In a matched check with empty-text conditioning, removing captions reduces accuracy by an average of 0.1, 0.9, and 12.3 percentage points for t∈[0,0.6]t\in[0,0.6], (0.6,0.8](0.6,0.8], and (0.8,1](0.8,1], respectively. This supports caption-free matching where standard training places most mass.

Human evaluation and ambience

We conducted an additional human study on 85 CoVoMix-derived scenes, testing vocalization identity, requested event timing, overlap adherence, and dialogue intelligibility. Across 390 completed judgments, we measure how often each requested feature is heard. When heard, we also assess whether vocalizations, discrete events, emotions, and interruptions occur at the requested point, and whether background sounds continue throughout. Presence and timing success rates are, respectively, 97.6%/98.8% for vocalizations, 89.7%/91.4% for discrete events, 66.7%/87.0% for emotion, 73.6%/88.7% for overlap, and 79.7%/62.7% for backgrounds. Raters identified the intended speaker from coughs or sighs with an 87.3% success rate (55/63), among judgments selecting either reference voice. Mean dialogue intelligibility is 4.49/5. Full results and the questionnaire appear in Appendix A.11, Tables 11–12. These additional human results further support our claims of speaker-aware scene control and temporal instruction following.

Table 6 quantifies the speaker-preservation cost of ambience, and Appendix A.14 shows that reference music can be continued when requested.

We hope the revision and additional checks address your concerns and would appreciate your reconsideration during the discussion.

Before the discussion closes, we would like to highlight the additional evidence now included in our revised manuscript. Alongside the metric and probe clarifications provided in response to your questions, we added a human study of broader scene control, an expanded training-data audit, and controlled tests of unwanted vocalization transfer from references. We believe these additions strengthen the evidence for generating varied audio scenes through free-form natural-language instructions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Dear Reviewers and Area Chair,

We thank all reviewers for their valuable feedback. Our work introduces ScenA, which generates complete multi-speaker audio scenes from reference voices and a free-form prompt. It also identifies the reference shortcut, a training failure mode in which the model matches references to residual target acoustics instead of learning speaker assignments from text, and addresses it through high-noise-biased timestep sampling. We are glad that reviewers recognized the task’s practical value (XsCc, Pmk9) and novelty (Pmk9), highlighted the reference shortcut as the strongest contribution (B875), and described the methodology as simple and insightful (Pmk9). Reviewers also recognized the strong dialogue results (XsCc, Pmk9), substantial evaluation (B875), and impressive audio demonstrations (Pmk9).

We have uploaded a revised manuscript incorporating additional experiments, analyses, and implementation details. Additions appear in red and deletions in blue strikethrough. The main additions are:

  • Human evaluation of scene control. We conducted an additional listening study assessing vocalizations, sound events, emotions, overlapping speech, and background sounds. The results provide direct perceptual evidence for these capabilities, with particularly strong vocalization and event control while maintaining intelligible dialogue.
  • Validation of the dialogue results. We evaluated unchanged and truncated benchmark dialogues separately. ScenA’s advantages hold on both groups, showing that the conclusions also hold without truncation. Additional sanity checks clarify how the speaker metrics respond to identity errors and further support the reference-shortcut analysis.
  • Training-data transparency and reliability. We added details about the training dataset’s reference construction and filtering, provided caption examples, and manually audited its speaker binding accuracy. The audit found a high rate of correct assignments. Together with the checkpoint, inference code, evaluation set, and evaluation code, these additions support independent verification and future work.

The individual replies address the remaining questions, including instruction handling, backbone selection, reference-background contamination, and ambience control. We believe these additions strengthen the evidence for ScenA’s contributions and hope they help inform the discussion and final assessment.

AuthorsOpenReview ↗

Follow-up on the author responses

Dear Reviewers,

Thank you again for your time and feedback. We posted our responses and revised manuscript on November 11, including the additional human evaluation, benchmark split analysis, and validation checks summarized in our Global Response.

With the discussion period ending on November 18, we would appreciate any follow-up questions or feedback on whether these additions address your concerns. We remain available to clarify any remaining concerns.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.