acceptodds
Under review as a conference paper at ICLR 2027

Iterative Timestamped Audio Describer

Abstract

Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce the Iterative Timestamped Audio Describer (ITAD), a foundation model for audio perception that constructs granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an InstructionInput-Output (IIO) completion format, enabling both autonomous decoding and the injection of external priors through a dynamic control surface. We further extend ITAD with expressive speech attributes to obtain ITAD-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Audio Language Models, Audio Understanding, Post-training, Paralinguistics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dsfcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ITAD constructs structured timelines over approximately two million licensed audio clips by fusing complementary weak teachers for sound events, speaker turns, transcripts, and scene descriptions. IIO (Instruction-Input-Output) unifies comprehensive description and attribute prediction as conditional-completion tasks. Speech specialists are then trained using small labeled datasets, model-generated silver labels, and DSP-derived attributes to form the ITAD-E teacher cascade. Its enriched descriptions are distilled into the unified ITAD-S student.

Critical strengths
  1. Overall and perception accuracy on MMSU exceed TAC by 7.9 and 14.0 percentage points, respectively.

  2. Single-pass decoding retains most QA gains; timestamps reduce generation PE localization error from 2.93 to 1.54 seconds.

Critical weaknesses
  1. No-fusion and conventional full-description training controls with matched backbone, data, and training budget are missing; attribute and decoding ablations do not isolate fusion and IIO.

  2. Buckeye primarily tests speech onsets; non-speech boundaries, speaker attribution, and fused labels lack human verification. Generation peak offsets also omit complete temporal boundaries.

  3. ITAD-S supplies conditioning labels and re-describes generated audio, exposing attribute consistency to correlated errors; external overall metrics do not independently validate individual attributes.

  4. Specialist training uses MELD and Common Voice, also sources of MMSU; recording deduplication and speaker splits are unspecified, requiring clarification.

Questions for authors
  1. With backbone, data, and training budget matched, how does performance change when teacher fusion is removed or IIO is replaced with conventional full-description training?

  2. What are ITAD-S’s non-speech boundary, speaker-attribution, and attribute accuracies against human references, and is the reference transcript supplied to the model during Buckeye evaluation?

  3. Do Table 4’s attribute results and model rankings persist when ITAD-S re-description is replaced with independent attribute recognizers or blinded human listening?

  4. Which unseen control tags or combinations define zero-shot evaluation, and what are the quantitative success rates for timing, speaker-attribute, and emphasis control on a held-out test set?

2 comments

We thank the reviewer for the detailed review.

W1/Q1. full-description training baseline.

A full-description-only or comprehensive (Comp.)-only baseline with same backbone, clips, LoRA, schedule and steps reaches 79.0 on MMSU vs. 81.3 single-pass and 81.9 multi-pass for IIO (same reasoner). The gain is in perception (73.9 vs. 77.8), especially MMSU's speaking-style tasks (63.0 vs. 70.7).

TrainingInferenceOverallPerceptionSpeaking style (5 tasks)PitchVocal rangeEmotion
Comp.-onlysingle-pass79.073.963.051.963.938.5
IIOsingle-pass81.376.165.265.762.041.0
IIOmulti-pass81.977.870.770.475.949.6

A no-fusion control is not well defined since no single teacher produces the full schema (the captioner produces no speaker turns or reliable timestamps; the diarizer produces no non-speech events or scene descriptions; the event detector produces no transcripts). The closest single-teacher systems score below ITAD-S on MMSU with the same reasoner (Qwen3-Omni-Captioner, the captioning teacher: 73.1; TAC: 74.0; ITAD alone: 74.4).

W2/Q2. Human references, held-out data.

Non-speech boundaries (TACOS test, 2,000 clips, human labels):

F1ITADITAD-SGemini-3-ProQwen3-Omni-Capt.
Event.420.356.415.367
Segment.647.659.643.664

Speaker attribution (ICSI + Earnings-21, 500 windows of 10–30 s, not in training):

SystemDERMiss / FA / Conf.JER
ITAD-S.125.115 / .002 / .008.32
ITAD.121.113 / .002 / .005.30

Attributes (held-out speakers, accuracy %):

AttributeITAD-SMajority
Emotion, RAVDESS 8-class72.913.3
Emotion, IEMOCAP S1 4-class70.932.9
Emotion, MELD test58.247.7
Gender, Common Voice99.880.0

Buckeye: The transcript is first generated using empty INPUT caption IIO inference pass (Appendix D) and then passed as INPUT with segment boundaries. Now clarified in Appendix E.

W3/Q3. Independent attribute recognizers

We re-scored Table 4 with off-the-shelf recognizers: Whisper-large-v3 (transcript), wav2vec2-large-robust age/gender (audeering), CommonAccent ECAPA (accent), and emotion2vec+ large (emotion)

SystemGen / Age / Acc / EmoWER
ITAD-S.99 / .71 / .62 / .67.17
Seed Audio.99 / .74 / .63 / .68.18
PicoAudio2.57 / .52 / .21 / .23.98
MiDashengLM-Gen.71 / .48 / .19 / .32.99

W4. MELD / Common Voice and MMSU

We audited overlap with MMSU at the clip level matching duration and transcripts, MMSU's emotion tasks use different recordings (CREMA-D-style sentences) and have no matches with our emotion specialist data with MELD train and IEMOCAP session 2-4. Majority of the training data for voice-profile specialist uses silver labels from Qwen3-Omni probe supplemented with a Common Voice accent subset with a speaker-disjoint evaluation split (also see Appendix B) - not part of the 114 MMSU accent clips.

Q4. Zero-shot control

Understand, Ln 83 ".. explicit zero-shot control over event timing, speaker characteristics, and word-level emphasis." and Ln 471 ".. demonstrating zero-shot emphasis control without a dedicated stress-control model." Controls are exercised only through the ITAD-S' description schema the generator was trained on (timestamps, speaker attributes, emphasis tags), on held-out sentences and with attribute combinations not present together in training, without any control-specific objective or data. For quantitative analysis, please see W2 and W3 above.

To further analyse the emphasis control, we score the generated audio with WhiStress (github.com/slp-rl/WhiStress) as an independent word-stress recognizer. A generation counts as a hit when the word WhiStress finds most stressed is the word we marked. In order to test the intensity of the generated stress and whether it follows the marker, we take 50 real recordings from A560E (held-out test), pick one target word in each, and generate each sentence from its ITAD-S caption with three variations: with no emphasis tag, with <e> on the target, and with <E> on the target.

AudioHit (Target word is the most stressed word)
Ours, no tag (neutral)0.31
Ours, <e> on the target0.75
Ours, <E> on the target0.82
Original recording (natural emphasis, no generation)0.38

Without any tags in the input to the generator, the target is the most stressed word about as often as in the original recording and marking the target word it raises this to 0.75 (<e>) and 0.82 (<E>).

Thank you to the authors for their detailed response and additional experiments. I recommend further distinguishing between the fidelity of generative reconstruction and the ability to modify attributes according to instructions in the revised manuscript. In addition, while the value of complementary information provided by multiple teachers has been demonstrated, the independent contributions of the specific alignment and fusion strategies remain to be validated. The additional experiments have addressed my main concerns, and I will raise my score from 2 to 3.

Reviewer_5uqsOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Iterative Timestamped Audio Describer (ITAD) and its speech-centric extension, ITAD-S, for producing structured audio descriptions with timestamps, speaker attribution, and fine-grained speech attributes. Stage 1 constructs training targets by aligning and combining predictions from complementary teachers. Stage 2 trains lightweight specialists to expose paralinguistic attributes and distills their predictions into a unified model. An Instruction-Input-Output (IIO) format supports both single-pass description and attribute-specific completion conditioned on intermediate outputs or external inputs. ITAD-S performs well on fine-grained speech and speaker-centric understanding benchmarks. Its descriptions also improve downstream audio-visual reasoning and provide effective supervision for temporally and controllable audio generation.

Critical strengths
  1. A substantial contribution to a meaningful problem: Explicitly representing acoustic events, speaker turns, and paralinguistic attributes addresses an important limitation of free-form audio descriptions. The approach is coherent and technically well motivated, with strong results particularly on fine-grained speech perception and speaker-centric understanding.

  2. Broad experimental validation: The paper compares against relevant baselines across multiple audio understanding benchmarks and further evaluates the descriptions in audio-visual reasoning and audio generation. These downstream experiments provide useful evidence that the proposed representation preserves information beyond what is needed for a single benchmark.

  3. Clear and informative attribute ablations: Figure 4 provides an intuitive analysis of the information added in Stage 2. This helps explain where the gains come from and why richer descriptions benefit downstream reasoning.

Critical weaknesses
  1. Insufficient specification of the Stage 1 teachers: Although the Qwen3-Omni captioner is identified, the sound event detector and diarization model are described only as “TAC-style” and “TAC-inspired.”. This omission substantially limits reproducibility, particularly given that the underlying audio corpus is proprietary.

  2. Missing evaluation of the Stage 1 ITAD model on the main understanding benchmarks: Table 1 reports ITAD-E and ITAD-S but omits ITAD. Figure 4 partially addresses the value of additional attributes, but its description-level ablation does not replace a direct evaluation of the independently trained ITAD checkpoint.

Questions for authors
  1. Could you specify the exact Stage 1 teacher models, particularly the sound event detector and diarizer? Are these publicly available checkpoints, internally trained models, or adaptations of existing TAC components? If internally trained, could you provide their training details and clarify their availability?

  2. Could you report ITAD’s results on MMSU, MMAR, and MMAU under the same inference and reasoning protocol used in Table 1? This would help establish the contribution of each training stage.

  3. Do you plan to release the ITAD and ITAD-S weights or LoRA adapters, together with inference and data-construction code? This is a substantial piece of work, and releasing these artifacts would make it especially valuable to the community.

1 comment

We thank the reviewer for the detailed review.

W1 / Q1. Stage 1 teachers

TAC-SED is the TAC model from Kumar et al., 2026 (Qwen2-Audio-7B-Instruct with TAC continued pre-training, LoRA r=128, α=256), further LoRA fine-tuned (r=128, α=256, 5k iterations, AdamW 5e-5) on synthetic soundscapes mixed from licensed sound-effect, speech and music libraries. TAC-diarize is similar recipe as above, but trained on ~942 hours of recordings from AMI (close-talk and far-field), VoxConverse, AISHELL-4, CALLHOME, CALLFRIEND, Simsamu, AVA-AVD. During training we randomly choose 5-30s windows, and use turn dropout, random time shifts, and background sound and music from licensed libraries mixed in at 2-18 dB SNR.

W2 / Q2. ITAD on MMSU, MMAR and MMAU

We evaluate ITAD with the same reasoner (Gemini-3.5-Flash, T = 0) with the same prompts.

ModelMMSU Perc.MMSU Reas.MMSU OverallMMAR SpeechMMAR SoundMMAR MusicMMAR OverallMMAU SpeechMMAU SoundMMAU MusicMMAU Overall
ITAD (Stage 1)66.183.174.482.058.853.469.479.072.469.573.6
ITAD-E (Stage 2 teacher cascade)79.087.182.984.056.452.970.380.571.868.373.5
ITAD-S (Stage 2 student)77.886.481.979.653.351.068.082.373.970.775.6

On MMSU, Stage 2 adds +8 points over ITAD (74.4 → 82.9 for the teacher cascade, 81.9 for the distilled student), and the gain comes almost entirely from perception (+12; the student +11). The Stage 2 gains concentrate on speech-centric understanding, which is where the elicited paralinguistic attributes are needed.

Q3 Weights

We will release a pseudocode of the Stage 1 alignment and fusion. In addition to these, our appendix details implementation of Stage 1 and Stage 2, along with task weights and hyperparameters, and the training data for specialists. The current ITAD/ITAD-S weights cannot be released because the licensed audio clips used for training do not permit redistributing derived models.

Reviewer_QnSnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

ITAD (Iterative Timestamped Audio Describer) is proposed in this paper for describing audio files with structured event timelines, speaker attribution, and speech attributes. The proposed work integrates weak-teacher annotations to train a Qwen3-Omni based describer using the Instruction-Input-Output (IIO) completion format, and distills the outputs of speech specialists into the unified ITAD-S model. When combined with frozen text LLMs, the proposed ITAD-S enhances fine-grained analysis of speech along with speaker-centric understanding on MMSU and MSU-Bench datasets, while offering inconsistent improvements on broader audio benchmark datasets. Further experiments are conducted using generated audio descriptions for training audio generators.

Critical strengths
  1. ITAD-S substantially improves fine-grained spoken and speaker-centric understanding on MMSU and MSU-Bench, supporting the value of its timestamped, speaker-attributed representation.
  2. The specialist-to-student pipeline scales detailed speech supervision through automatic annotation while retaining performance close to the expert cascade in a unified model.
  3. The same schema supports audio synthesis, with timestamp ablations demonstrating temporal control and qualitative examples illustrating word-level emphasis control.
Critical weaknesses
  1. The downstream reasoner is not consistently identified for the main results and radar plots, making it difficult to verify that comparisons across describers use the same reasoning configuration.
  2. A matched comprehensive-only training baseline is missing, so the accuracy benefit attributable specifically to IIO training remains unclear.
  3. Without end-to-end inference-cost measurements, comparisons with direct audio models leave the trade-off between accuracy and computation unresolved.
  4. Generation quality depends on a separately trained generator, and the experiments do not isolate the contribution of ITAD-S’s annotations, limiting conclusions about the describer itself.
Questions for authors
  1. Which downstream reasoner and model version were used for each main audio-understanding benchmark and radar plot? Were the reasoner and decoding settings held fixed across describers within each benchmark?
  2. In Figure 4, are the per-task gains measured by adding each attribute separately to the transcript-only base, or incrementally following the cumulative order in the left panel? Please clarify the reference condition for each group.
3 comments

We thank the reviewer for the careful assessment.

W1/Q1. Reasoner.

All tables and radar plots in the paper use the same text-only reasoner, Gemini-3.5-Flash, T=0, top-p 1 (only exception is Table 2's reasoner-ablation). For all the models evaluated following DtR, we use the same prompts from Kumar et. al 2026 (also mentioned in Appendix H). For MSU-Bench we use the benchmark's official prompt. In the radar plots, we compare ITAD-S with Gemini-3.7-Flash and Qwen3Omni-captioner, (1) when they attempt direct audio inference (left radar) or (2) when they attempt the same DtR inference as ITAD-S, replacing their generated descriptions against those generated by ITAD-S'. Now clarified in Section 4.1

W2. Comprehensive-only baseline.

We trained a comprehensive (Comp.)-only baseline where both ITAD and ITAD-S are trained with only the single-decode full-description task. LoRA config, training clips, LR schedule etc are matched across all the rows.

TrainingInferenceOverallPerceptionSpeaking style (5 tasks)PitchVocal rangeEmotion
Comp.-onlysingle-pass79.073.963.051.963.938.5
IIOsingle-pass81.376.165.265.762.041.0
IIOmulti-pass81.977.870.770.475.949.6

Please note, IIO during training is what enables our multi-pass inference (by providing task-conditioned decodes that can be composed), whereas the comprehensive-only baseline is restricted to single-pass inference. Highest gain is observed in paralinguistic perception (speaking style +7). When using IIO during training, both single-pass (matched inference cost) and multi-pass inference setups yield better understanding performance compared to the comprehensive-only training baseline, thus reinforcing that IIO training itself improves audio perception, while also enabling additional gains through multi-pass inference.

W3. Inference cost.

For understanding tasks, ITAD's DtR costs more GPU (A100 80GB, vLLM) time than direct audio inference on a similar-sized Qwen3-Omni-Instruct, but gains +11 % on MMSU (+24 on perception tasks). It should also be noted that description generation is question-independent, so it can be computed once per clip and amortised across questions, for instance MSU-Bench has ~8 questions per clip.

SystemMMSUGPU-s / question
Qwen3-Omni-30B-A3B-Instruct, direct from audio69.30.08
Qwen3-Omni-30B-A3B-Captioner → Qwen3.8-27B72.42.95
ITAD-S single-pass → Qwen3.8-27B80.52.94
ITAD-S multi-pass → Qwen3.8-27B80.93.39

W4. ITAD-S contribution on generation.

Good spot. To study this, we trained a generator with the same config and training clips as mentioned in Appendix G, but this time using the captions generated by Qwen3-Omni-30B-A3B-Captioner instead of ITAD or ITAD-S. Following the same evaluation metrics as Table 4,

Source of captionsGen / Age / Acc / EmoSp.R.WER↓PE↓FLAM↓PE-AV↓Gemini vs. real
Qwen3-Omni-Captioner.94 / .65 / .57 / .64.79.482.482.88.363.14
ITAD-S.99 / .84 / .69 / .77.93.141.542.14.303.95

We can conclude that using the structured schema generated by ITAD as input to the generation model itself contributes to the improvement of every metric, including the text-free ones measured against the original recording.

Q2. Fig. 4.

We show the incremental gains following the left panel's cumulative order, so {{{transcript-only base} + stress} + emotion} ...}, while keeping the same reasoner throughout. Now clarified in the Figure 4 caption.

As we approach the end of discussion period, we wanted to briefly highlight the key additional experiments in our response above:

  • a comprehensive-only baseline to isolate the benefit of IIO in training (removing IIO drops MMSU perf. from 81.9 --> 79.0 as well as eliminates the ability to do task-conditioned multi-pass inference).
  • end-to-end inference-cost measurements (ITAD's describe-then-reason (DtR) costs more GPU time than conventional direct-audio inference, but it also yields atleast 11+ performance. Not to overlook that the "describe" in DtR is question agnostic so, unlike direct-audio inference, can be bootstrapped in several downstream applications.)
  • a Qwen3Omni-captioner supervised generator baseline to isolate the contribution of ITAD-S's structured schema to generation quality.

We have also clarified the reasoner configuration and Figure 4 in the revised draft.

We would appreciate it if the reviewer could revisit and let us know if any concerns remain.

Thank you for the detailed response. The comprehensive-only baseline and controlled generation comparison address my main experimental concerns, while the reasoner configuration and inference costs are now clearer. Although I retain some reservations about presentation and computational cost, the additional evidence strengthens the case for the method, and I raise my rating to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.