- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ITAD constructs structured timelines over approximately two million licensed audio clips by fusing complementary weak teachers for sound events, speaker turns, transcripts, and scene descriptions. IIO (Instruction-Input-Output) unifies comprehensive description and attribute prediction as conditional-completion tasks. Speech specialists are then trained using small labeled datasets, model-generated silver labels, and DSP-derived attributes to form the ITAD-E teacher cascade. Its enriched descriptions are distilled into the unified ITAD-S student.
Critical strengths
-
Overall and perception accuracy on MMSU exceed TAC by 7.9 and 14.0 percentage points, respectively.
-
Single-pass decoding retains most QA gains; timestamps reduce generation PE localization error from 2.93 to 1.54 seconds.
Critical weaknesses
-
No-fusion and conventional full-description training controls with matched backbone, data, and training budget are missing; attribute and decoding ablations do not isolate fusion and IIO.
-
Buckeye primarily tests speech onsets; non-speech boundaries, speaker attribution, and fused labels lack human verification. Generation peak offsets also omit complete temporal boundaries.
-
ITAD-S supplies conditioning labels and re-describes generated audio, exposing attribute consistency to correlated errors; external overall metrics do not independently validate individual attributes.
-
Specialist training uses MELD and Common Voice, also sources of MMSU; recording deduplication and speaker splits are unspecified, requiring clarification.
Questions for authors
-
With backbone, data, and training budget matched, how does performance change when teacher fusion is removed or IIO is replaced with conventional full-description training?
-
What are ITAD-S’s non-speech boundary, speaker-attribution, and attribute accuracies against human references, and is the reference transcript supplied to the model during Buckeye evaluation?
-
Do Table 4’s attribute results and model rankings persist when ITAD-S re-description is replaced with independent attribute recognizers or blinded human listening?
-
Which unseen control tags or combinations define zero-shot evaluation, and what are the quantitative success rates for timing, speaker-attribute, and emphasis control on a held-out test set?
2 comments
Thank you to the authors for their detailed response and additional experiments. I recommend further distinguishing between the fidelity of generative reconstruction and the ability to modify attributes according to instructions in the revised manuscript. In addition, while the value of complementary information provided by multiple teachers has been demonstrated, the independent contributions of the specific alignment and fusion strategies remain to be validated. The additional experiments have addressed my main concerns, and I will raise my score from 2 to 3.
We thank the reviewer for the detailed review.
W1/Q1. full-description training baseline.
A full-description-only or comprehensive (Comp.)-only baseline with same backbone, clips, LoRA, schedule and steps reaches 79.0 on MMSU vs. 81.3 single-pass and 81.9 multi-pass for IIO (same reasoner). The gain is in perception (73.9 vs. 77.8), especially MMSU's speaking-style tasks (63.0 vs. 70.7).
A no-fusion control is not well defined since no single teacher produces the full schema (the captioner produces no speaker turns or reliable timestamps; the diarizer produces no non-speech events or scene descriptions; the event detector produces no transcripts). The closest single-teacher systems score below ITAD-S on MMSU with the same reasoner (Qwen3-Omni-Captioner, the captioning teacher: 73.1; TAC: 74.0; ITAD alone: 74.4).
W2/Q2. Human references, held-out data.
Non-speech boundaries (TACOS test, 2,000 clips, human labels):
Speaker attribution (ICSI + Earnings-21, 500 windows of 10–30 s, not in training):
Attributes (held-out speakers, accuracy %):
Buckeye: The transcript is first generated using empty INPUT caption IIO inference pass (Appendix D) and then passed as INPUT with segment boundaries. Now clarified in Appendix E.
W3/Q3. Independent attribute recognizers
We re-scored Table 4 with off-the-shelf recognizers: Whisper-large-v3 (transcript), wav2vec2-large-robust age/gender (audeering), CommonAccent ECAPA (accent), and emotion2vec+ large (emotion)
W4. MELD / Common Voice and MMSU
We audited overlap with MMSU at the clip level matching duration and transcripts, MMSU's emotion tasks use different recordings (CREMA-D-style sentences) and have no matches with our emotion specialist data with MELD train and IEMOCAP session 2-4. Majority of the training data for voice-profile specialist uses silver labels from Qwen3-Omni probe supplemented with a Common Voice accent subset with a speaker-disjoint evaluation split (also see Appendix B) - not part of the 114 MMSU accent clips.
Q4. Zero-shot control
Understand, Ln 83 ".. explicit zero-shot control over event timing, speaker characteristics, and word-level emphasis." and Ln 471 ".. demonstrating zero-shot emphasis control without a dedicated stress-control model." Controls are exercised only through the ITAD-S' description schema the generator was trained on (timestamps, speaker attributes, emphasis tags), on held-out sentences and with attribute combinations not present together in training, without any control-specific objective or data. For quantitative analysis, please see W2 and W3 above.
To further analyse the emphasis control, we score the generated audio with WhiStress (github.com/slp-rl/WhiStress) as an independent word-stress recognizer. A generation counts as a hit when the word WhiStress finds most stressed is the word we marked. In order to test the intensity of the generated stress and whether it follows the marker, we take 50 real recordings from A560E (held-out test), pick one target word in each, and generate each sentence from its ITAD-S caption with three variations: with no emphasis tag, with
<e>on the target, and with<E>on the target.<e>on the target<E>on the targetWithout any tags in the input to the generator, the target is the most stressed word about as often as in the original recording and marking the target word it raises this to 0.75 (<e>) and 0.82 (<E>).