- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LongCap, a suite of metrics, a benchmark, and models for timestamped audio captioning (TAC) that extends to hour-long audio recordings. The paper makes three main contributions. First, the authors propose three new metrics, TER, GER, and tcpCapSim, to evaluate the performance of audio language models. Second, they introduce a new audio benchmark, LongCapBench, which consists of diarization subsets of up to 78 minutes and more than a thousand TAC clips, plus 4 long recordings. Third, they present a family of TAC models built on the Qwen3-Omni-Captioner encoder and Qwen3.5 backbones ranging from 0.8B to 122B parameters. The models process audio as a sequence of 8–32 s chunks along with their captions and timestamps, and keep all prior chunks’ outputs in context. Training consists of multi-task pretraining, SFT, and optional self-distillation on longer audio. Validation covers diarization with standard metrics on AliMeeting, AIShell4, AMI, CHiME6, and the authors’ subsets, and includes ablations of holistic vs. chunkwise processing and a comparison of absolute vs. relative timestamps. TAC is evaluated with the authors’ proposed metrics on LongCapBench against MOSS, VibeVoice, Qwen3.8, and Doubao.
Critical strengths
- The long-form diarization performance is strong. On CHiME6, LongCap-Diar reaches 29% DER, while other models are at 70–100% (App. D.1).
- The ablation cleanly explains why holistic speech LLMs fail on long audio. On 50–78 min recordings, holistic processing yields 109.1% DER and 80.6% WER, compared to 20.3% DER and 12.7% WER for chunkwise processing (Table 6).
- LongCapBench consists of timestamped, captioned audio with durations approaching two hours.
Critical weaknesses
- Chunkwise processing with interleaved audio/text and timestamps is already established in the literature; see Whisper, SpeechLLM-XL (Jia et al., 2025), and JEDIC-LLM (Shi et al., 2026). The paper cites the latter two works but does not state how LongCap differs from them. Compared with SpeechLLM-XL’s limited attention window and JEDIC-LLM’s speaker prompt cache, LongCap keeps all prior audio and captions in context. This is possible partly due to a stronger base model that can handle longer contexts.
- The novelty of the three metrics should be better contextualized relative to prior evaluation work. TER is DER computed over LLM-proposed groups, and tcpCapSim follows the tcpWER structure with LLM-based per-key scoring.
- The long-form benchmark is small and potentially biased. TAC-Long has only 4 recordings (7 hours in total), and Diar-Long has only 15 recordings. Meanwhile, the references are corrected drafts from Qwen3.5-Omni-Plus, which is also the source of the training labels. The two may therefore share biases; for example, the 62-key schema matches LongCap’s own output format, which disadvantages the baselines. The authors acknowledge this by moving the AF-Next/LAT comparison to an LLM-preference protocol (App. D.2).
- The positioning of the benchmark omits related work. BLAB [1] covers the same 15-minute to 2-hour range, so “approaching two hours” (L082) is not distinctive by itself. Moreover, BLAB [1], AudioMarathon [2], and CASTELLA [3] are not cited.
- There is no external TAC validation. This relates to the third point about benchmark bias. To demonstrate LongCap’s advantage and to rule out the possibility of a biased benchmark, an external TAC validation (e.g., on LAT-Bench) would be important. MMAU/MMAR are used only for scaling analysis, not for comparison against baselines.
- The aggregated diarization claim masks per-benchmark results. In Table 4, MOSS outperforms LongCap-Diar on all four metrics. LongCap’s gains come mainly from AMI-SDM, the authors’ own subsets, and CHiME6. Fig. 5a also appears to average five conditions, excluding Diar-Long; this should be stated explicitly.
[1] BLAB (Ahia et al., 2025): https://arxiv.org/abs/2505.03054
[2] AudioMarathon (He et al., 2025): https://arxiv.org/abs/2510.07293
[3] CASTELLA (Munakata et al., 2025): https://arxiv.org/abs/2511.15131
Questions for authors
- The Fig. 5 caption mentions “benchmark-specific checkpoints” (L338), but I could not find a description of this in App. D.1. How were checkpoints selected for each benchmark?
- What does “AMI overlap” (L339) in the Fig. 5 caption refer to?
- In Fig. 1, were the other models given full recordings or the same chunking as LongCap? If they received full recordings, Fig. 1 only demonstrates the baselines’ processing failure rather than LongCap’s advantage.
4 comments
Author Response 2
W3. Benchmark size and annotation bias.
Thank you for raising these concerns.
-
The evaluation volume is substantial. LongCapBench-Diar-Long’s 15 recordings provide more evaluation audio than the following established diarization test sets:
Benchmark split Audio duration AliMeeting Test 10.00 h AISHELL-4 Evaluation 12.72 h LongCapBench-Diar-Long 15.02 h TAC-Long’s four recordings contain 7.05 hours, 6,520 events, and 118,986 descriptive attribute key–value pairs in the final references following specialist review. This represents substantial annotation volume. We agree that the number of recordings (15 and 4) limits source diversity and makes recording-level variance hard to estimate, and we will state this as a limitation.
-
Human review substantially revises the model drafts. We acknowledge that using the same draft model can introduce shared annotation bias. Sec. 2.3 describes our annotation workflow and revision statistics. Our experts preferred correcting Qwen3.5-Omni-Plus drafts to writing annotations from scratch, because the drafts help identify less salient events. Specialists then review timestamps and attributes. Approximately 91% of attribute entries on TAC-Long and 58% on TAC-Short are added, removed, or modified between drafts and returned annotations. These figures document annotation-stage revisions, rather than measuring the amount of bias removed.
-
Additional evaluation beyond benchmark-specific formats. LongCap, LAT, and AF-Next use different output formats, which can affect reference-based evaluation. We therefore evaluate on independently annotated LAT-Bench and additionally compare complete caption sets directly against the audio using Gemini 3.1 Pro. Model names and reference captions are withheld, and both candidate orders are judged.
Across the same 300 windows from 100 additional recordings:
Comparator LongCap preferred in both orders Comparator preferred in both orders Ties Order-dependent Incomplete LAT 288 (96.0%) 1 0 9 2 AF-Next 188 (62.7%) 2 1 75 34 AF-Next’s incomplete comparisons include 28 windows exceeding the frozen audio-context limit and six incomplete judgments; all remain in the denominator. These results support Gemini’s preference for LongCap’s caption content without relying on LongCapBench’s reference format. The judge compares coverage and supported details, so a denser caption set has an advantage, and the test does not measure timing. Independent human verification remains outstanding.
W4. Related benchmarks and the scope of long-form captioning.
Thank you for highlighting these relevant works.
-
Acknowledging related benchmarks. We recognize the contributions of BLAB, AudioMarathon, and CASTELLA to long-audio understanding, including temporal localization, reasoning, and audio moment retrieval. We will cite all three and acknowledge their contributions in the related-work discussion.
-
The context is long-form audio captioning. Our reference to recordings “approaching two hours” concerns dense timestamped audio captioning. We agree that duration alone is not distinctive, since BLAB covers the same range. The abstract and introduction state this scope for the model: “For TAC, LongCap … captions full recordings approaching two hours” (L093–095). LongCapBench evaluates structured event descriptions throughout these recordings, which complements the understanding and retrieval tasks evaluated by the cited benchmarks. We will revise the benchmark description near L082 so that it claims dense timestamped caption annotations at this duration, not the duration itself.
W5. External TAC validation.
Thank you for this suggestion. Following your recommendation, we conducted an external evaluation on LAT-Bench, with the setup and results reported in W3, point 3 above. The Gemini A/B results show that LongCap remains strong compared with prior audio captioners LAT and AF-Next on this external benchmark. The native LAT-Bench metric credits at most one prediction per reference event, and LongCap predicts about 8.6× as many events as the references contain (12,460 vs. 1,450 on the 104 English recordings). Most LongCap events therefore cannot contribute to that score, which is why we use the audio-grounded comparison instead.
Author Response 3
W6. Per-benchmark performance and averaging scope.
Thank you for this comment.
-
Scope of Fig. 5(a). The figure equally averages five conditions: AliMeeting, AISHELL-4, AMI-IHM, AMI-SDM, and LongCapBench-Diar-Short. The first four are established public meeting benchmark conditions for speech recognition and diarization. LongCapBench-Diar-Long and CHiME-6 are excluded from this average, so our larger gains on these challenging long recordings do not contribute to Fig. 5(a). Their results are reported separately in Fig. 5(b–c)/Table 4 and Table 5, respectively.
-
The comparison with MOSS varies across benchmarks. MOSS achieves lower errors on all four metrics on AliMeeting and AISHELL-4. AMI-IHM has mixed results: LongCap achieves lower DER/WER, while MOSS achieves lower cpWER/tcpWER. LongCap achieves lower errors on all four metrics on AMI-SDM and LongCapBench-Diar-Short, and lower macro-averaged errors across the five conditions. The systems are competitive overall, with a particularly large advantage for LongCap on AMI-SDM’s distant-microphone recordings. This highlights a difference in performance under far-field conditions, although the comparison does not isolate its cause.
-
Clearer presentation. We will explicitly list the five conditions in the figure caption and discuss the per-benchmark differences in the main text, making the scope of our aggregate claim clearer. We will also note that the "Macro mean" row of Table 4 averages six conditions, including LongCapBench-Diar-Long, and therefore differs from Fig. 5(a).
Q1 / Q2. Checkpoint selection and the “AMI overlap” note.
Thank you for pointing out these outdated statements in the Fig. 5 caption.
-
Origin of the wording. During an earlier data inspection, we found that some AMI evaluation recordings had been inadvertently included in the diarization training set. We removed these recordings and retrained the model. “Benchmark-specific checkpoints” referred to the replacement checkpoint trained after removing the overlap, while “AMI overlap” referred to the earlier training–evaluation leakage. Both phrases were left over from that stage and should have been removed. We will correct the caption.
-
Checkpoint used for the reported results. All reported LongCap-Diar results in the main benchmark comparison use the same final checkpoint from the corrected training run. We have released LongCap-Diar-35B-A3B, together with the vLLM implementation and Docker setup, and welcome reviewers to test it and independently verify its performance.
Q3. Input protocol and interpretation of Figure 1.
Thank you for asking us to clarify the comparison.
-
Input protocol. Neither. Each baseline receives the first ten minutes of every LongCapBench-TAC-Long recording as a single input. We did not give the baselines the full recordings, because they do not produce reasonable output at that length. LongCap processes the full recordings with chunk-wise generation, up to 117 minutes. We will state both setups more clearly in the figure caption.
-
What the comparison establishes. We believe Fig. 1 shows both points you raise. It shows a failure of the baselines: their captions degrade within ten minutes, a much easier input than the full recordings. It also shows LongCap's advantage: over the same first ten minutes, LongCap has lower timeline error and higher caption similarity, and it keeps producing captions for the rest of the recording, where the baselines do not produce reasonable output. Fig. 1 compares complete systems and does not isolate chunk-wise processing; the same-size chunk-wise versus holistic ablation in Table 6 does that.
I thank the authors for their detailed response and additional experiments. My concerns are partly addressed. I think the limited number of long-form recording and external evaluation are still major limitations, and I have reservations about generalizability of the results. Therefore, I am maintaining my score of 2.
Author Response 1
Thank you for your thoughtful review.
Checkpoint release. Following internal review, we have released the LongCap-35B-A3B and LongCap-Diar-35B-A3B checkpoints, together with our vLLM implementation and Docker setup. We welcome reviewers to test the models. We hope the comprehensive audio captions generated by LongCap will support further research in the community. Release page.
W1. Novelty relative to prior chunk-wise speech models.
Thank you for highlighting these relevant precedents. LongCap builds on chunk-wise speech processing established by Whisper, SpeechLLM-XL, and JEDIS-LLM. Our contributions extend this foundation to dense, structured captioning of general audio, with substantial length extrapolation and explicit event-dependency evaluation. We will state the differences below explicitly in the related-work section.
Length extrapolation for general audio captioning. SpeechLLM-XL studies speech recognition, while JEDIS-LLM studies joint recognition and speaker diarization and already shows training on short audio and inferring on long audio for that task. We do not claim length extrapolation itself as new. Our contribution is to show that it holds for dense captioning of overlapping speech, music, and environmental events, with structured descriptions and event boundaries that can span processing chunks. LongCap-SFT-2 is trained on audio of at most 20 minutes, yet captions recordings up to 117 minutes—5.9× its training span (Sec. 3.3; Table 1; Table 8, "Before" rows). Our 2B diarization ablation additionally demonstrates the advantage of chunk-wise over holistic generation at the same model size (Table 6).
Compatibility with a standard LLM pipeline. SpeechLLM-XL studies limited attention to reduce computation and also evaluates full context; JEDIS-LLM introduces a Speaker Prompt Cache to maintain speaker consistency. LongCap retains standard Qwen3.5 attention with YaRN and uses ordinary pretraining and SFT, without adding a task-specific attention or memory module. This compatibility is a practical strength: the captioning formulation fits an established training pipeline through audio/output interleaving, event assignment, and relative timestamp targets. The backbone provides context capacity, while these choices define how the model learns to generate dense event descriptions.
Explicit dependencies among general audio events. Beyond speaker consistency, LongCap supervises relationships among captioned events, including shared sources, continuation, and recurrence. To our knowledge, LongCap is one of the earliest works to supervise and directly evaluate event-link prediction in hour-scale audio captioning (Sec. 3.3; Fig. 9; App. D.4). This makes relationships across time an explicit modeling and evaluation target alongside individual event descriptions. Long-form connectivity F1 is still low (4.2% for 35B-A3B and 14.8% for 122B-A10B against silver references), so we present this as a first measurement, not a solved capability.
Complementary benchmark and evaluation contributions. LongCapBench provides human-corrected event timelines and structured attributes covering speech, music, and environmental sound. TER, GER, and tcpCapSim separately assess timeline errors, event granularity, and descriptive agreement, enabling diagnosis of different failures throughout long recordings (Sec. 2; Table 1).
W2. Relationship to existing evaluation metrics.
Thank you for this comment. We agree with your characterization: TER applies DER accounting to LLM-proposed semantic groups, and tcpCapSim follows the tcpWER structure with LLM-based per-key scoring. Sec. 2.2 cites DER and tcpWER as their origin, and we will state this relationship more directly there. What we add is the LLM-based event grouping, which replaces speaker labels for general audio events, and the complementary GER metric, which measures splitting, merging, and duplication beyond timeline errors.