acceptodds
Under review as a conference paper at ICLR 2027

LongCap: Hour-Scale Timestamped Audio Captioning via Chunk-wise Processing

Abstract

Describing hours of audio requires accurate event timing and sustained descriptive detail throughout the recording. We present LongCap, a framework for timestamped audio captioning (TAC), with the human-annotated LongCapBench and three diarization-inspired metrics: Timeline Error Rate (TER), Granularity Error Rate (GER), and Caption Similarity (tcpCapSim). LongCap interleaves audio chunks with structured event captions, using relative timestamps and retained context to support length extrapolation within a 1M-token inference context window. Event-level reannotation restores caption detail, while systematic studies examine model scaling, dependency supervision, and self-distillation. LongCap-Diar achieves strong benchmark results and extrapolates to 16 hours of concatenated audio despite training on recordings no longer than 40 minutes. Our TAC models are competitive with Qwen3.5-Omni-Plus on short clips, outperform evaluated baselines in caption similarity and timeline accuracy in segment-level comparisons on LongCapBench-TAC-Long, and caption full recordings approaching two hours. On LongCapBench-TAC, 9B and 35B-A3B offer favorable size–performance tradeoffs, while 122B-A10B substantially improves event-dependency recovery. These findings support long-horizon general audio understanding and provide a foundation for always-on voice agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Processing, Audio Caption, Long-form, Diarization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_erbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LongCap, a suite of metrics, a benchmark, and models for timestamped audio captioning (TAC) that extends to hour-long audio recordings. The paper makes three main contributions. First, the authors propose three new metrics, TER, GER, and tcpCapSim, to evaluate the performance of audio language models. Second, they introduce a new audio benchmark, LongCapBench, which consists of diarization subsets of up to 78 minutes and more than a thousand TAC clips, plus 4 long recordings. Third, they present a family of TAC models built on the Qwen3-Omni-Captioner encoder and Qwen3.5 backbones ranging from 0.8B to 122B parameters. The models process audio as a sequence of 8–32 s chunks along with their captions and timestamps, and keep all prior chunks’ outputs in context. Training consists of multi-task pretraining, SFT, and optional self-distillation on longer audio. Validation covers diarization with standard metrics on AliMeeting, AIShell4, AMI, CHiME6, and the authors’ subsets, and includes ablations of holistic vs. chunkwise processing and a comparison of absolute vs. relative timestamps. TAC is evaluated with the authors’ proposed metrics on LongCapBench against MOSS, VibeVoice, Qwen3.8, and Doubao.

Critical strengths
  • The long-form diarization performance is strong. On CHiME6, LongCap-Diar reaches 29% DER, while other models are at 70–100% (App. D.1).
  • The ablation cleanly explains why holistic speech LLMs fail on long audio. On 50–78 min recordings, holistic processing yields 109.1% DER and 80.6% WER, compared to 20.3% DER and 12.7% WER for chunkwise processing (Table 6).
  • LongCapBench consists of timestamped, captioned audio with durations approaching two hours.
Critical weaknesses
  • Chunkwise processing with interleaved audio/text and timestamps is already established in the literature; see Whisper, SpeechLLM-XL (Jia et al., 2025), and JEDIC-LLM (Shi et al., 2026). The paper cites the latter two works but does not state how LongCap differs from them. Compared with SpeechLLM-XL’s limited attention window and JEDIC-LLM’s speaker prompt cache, LongCap keeps all prior audio and captions in context. This is possible partly due to a stronger base model that can handle longer contexts.
  • The novelty of the three metrics should be better contextualized relative to prior evaluation work. TER is DER computed over LLM-proposed groups, and tcpCapSim follows the tcpWER structure with LLM-based per-key scoring.
  • The long-form benchmark is small and potentially biased. TAC-Long has only 4 recordings (7 hours in total), and Diar-Long has only 15 recordings. Meanwhile, the references are corrected drafts from Qwen3.5-Omni-Plus, which is also the source of the training labels. The two may therefore share biases; for example, the 62-key schema matches LongCap’s own output format, which disadvantages the baselines. The authors acknowledge this by moving the AF-Next/LAT comparison to an LLM-preference protocol (App. D.2).
  • The positioning of the benchmark omits related work. BLAB [1] covers the same 15-minute to 2-hour range, so “approaching two hours” (L082) is not distinctive by itself. Moreover, BLAB [1], AudioMarathon [2], and CASTELLA [3] are not cited.
  • There is no external TAC validation. This relates to the third point about benchmark bias. To demonstrate LongCap’s advantage and to rule out the possibility of a biased benchmark, an external TAC validation (e.g., on LAT-Bench) would be important. MMAU/MMAR are used only for scaling analysis, not for comparison against baselines.
  • The aggregated diarization claim masks per-benchmark results. In Table 4, MOSS outperforms LongCap-Diar on all four metrics. LongCap’s gains come mainly from AMI-SDM, the authors’ own subsets, and CHiME6. Fig. 5a also appears to average five conditions, excluding Diar-Long; this should be stated explicitly.

[1] BLAB (Ahia et al., 2025): https://arxiv.org/abs/2505.03054

[2] AudioMarathon (He et al., 2025): https://arxiv.org/abs/2510.07293

[3] CASTELLA (Munakata et al., 2025): https://arxiv.org/abs/2511.15131

Questions for authors
  • The Fig. 5 caption mentions “benchmark-specific checkpoints” (L338), but I could not find a description of this in App. D.1. How were checkpoints selected for each benchmark?
  • What does “AMI overlap” (L339) in the Fig. 5 caption refer to?
  • In Fig. 1, were the other models given full recordings or the same chunking as LongCap? If they received full recordings, Fig. 1 only demonstrates the baselines’ processing failure rather than LongCap’s advantage.
4 comments
AuthorsOpenReview ↗

Author Response 1

Thank you for your thoughtful review.

Checkpoint release. Following internal review, we have released the LongCap-35B-A3B and LongCap-Diar-35B-A3B checkpoints, together with our vLLM implementation and Docker setup. We welcome reviewers to test the models. We hope the comprehensive audio captions generated by LongCap will support further research in the community. Release page.

W1. Novelty relative to prior chunk-wise speech models.

Thank you for highlighting these relevant precedents. LongCap builds on chunk-wise speech processing established by Whisper, SpeechLLM-XL, and JEDIS-LLM. Our contributions extend this foundation to dense, structured captioning of general audio, with substantial length extrapolation and explicit event-dependency evaluation. We will state the differences below explicitly in the related-work section.

  1. Length extrapolation for general audio captioning. SpeechLLM-XL studies speech recognition, while JEDIS-LLM studies joint recognition and speaker diarization and already shows training on short audio and inferring on long audio for that task. We do not claim length extrapolation itself as new. Our contribution is to show that it holds for dense captioning of overlapping speech, music, and environmental events, with structured descriptions and event boundaries that can span processing chunks. LongCap-SFT-2 is trained on audio of at most 20 minutes, yet captions recordings up to 117 minutes—5.9× its training span (Sec. 3.3; Table 1; Table 8, "Before" rows). Our 2B diarization ablation additionally demonstrates the advantage of chunk-wise over holistic generation at the same model size (Table 6).

  2. Compatibility with a standard LLM pipeline. SpeechLLM-XL studies limited attention to reduce computation and also evaluates full context; JEDIS-LLM introduces a Speaker Prompt Cache to maintain speaker consistency. LongCap retains standard Qwen3.5 attention with YaRN and uses ordinary pretraining and SFT, without adding a task-specific attention or memory module. This compatibility is a practical strength: the captioning formulation fits an established training pipeline through audio/output interleaving, event assignment, and relative timestamp targets. The backbone provides context capacity, while these choices define how the model learns to generate dense event descriptions.

  3. Explicit dependencies among general audio events. Beyond speaker consistency, LongCap supervises relationships among captioned events, including shared sources, continuation, and recurrence. To our knowledge, LongCap is one of the earliest works to supervise and directly evaluate event-link prediction in hour-scale audio captioning (Sec. 3.3; Fig. 9; App. D.4). This makes relationships across time an explicit modeling and evaluation target alongside individual event descriptions. Long-form connectivity F1 is still low (4.2% for 35B-A3B and 14.8% for 122B-A10B against silver references), so we present this as a first measurement, not a solved capability.

  4. Complementary benchmark and evaluation contributions. LongCapBench provides human-corrected event timelines and structured attributes covering speech, music, and environmental sound. TER, GER, and tcpCapSim separately assess timeline errors, event granularity, and descriptive agreement, enabling diagnosis of different failures throughout long recordings (Sec. 2; Table 1).

W2. Relationship to existing evaluation metrics.

Thank you for this comment. We agree with your characterization: TER applies DER accounting to LLM-proposed semantic groups, and tcpCapSim follows the tcpWER structure with LLM-based per-key scoring. Sec. 2.2 cites DER and tcpWER as their origin, and we will state this relationship more directly there. What we add is the LLM-based event grouping, which replaces speaker labels for general audio events, and the complementary GER metric, which measures splitting, merging, and duplication beyond timeline errors.

AuthorsOpenReview ↗

Author Response 2

W3. Benchmark size and annotation bias.

Thank you for raising these concerns.

  1. The evaluation volume is substantial. LongCapBench-Diar-Long’s 15 recordings provide more evaluation audio than the following established diarization test sets:

    Benchmark splitAudio duration
    AliMeeting Test10.00 h
    AISHELL-4 Evaluation12.72 h
    LongCapBench-Diar-Long15.02 h

    TAC-Long’s four recordings contain 7.05 hours, 6,520 events, and 118,986 descriptive attribute key–value pairs in the final references following specialist review. This represents substantial annotation volume. We agree that the number of recordings (15 and 4) limits source diversity and makes recording-level variance hard to estimate, and we will state this as a limitation.

  2. Human review substantially revises the model drafts. We acknowledge that using the same draft model can introduce shared annotation bias. Sec. 2.3 describes our annotation workflow and revision statistics. Our experts preferred correcting Qwen3.5-Omni-Plus drafts to writing annotations from scratch, because the drafts help identify less salient events. Specialists then review timestamps and attributes. Approximately 91% of attribute entries on TAC-Long and 58% on TAC-Short are added, removed, or modified between drafts and returned annotations. These figures document annotation-stage revisions, rather than measuring the amount of bias removed.

  3. Additional evaluation beyond benchmark-specific formats. LongCap, LAT, and AF-Next use different output formats, which can affect reference-based evaluation. We therefore evaluate on independently annotated LAT-Bench and additionally compare complete caption sets directly against the audio using Gemini 3.1 Pro. Model names and reference captions are withheld, and both candidate orders are judged.

    Across the same 300 windows from 100 additional recordings:

    ComparatorLongCap preferred in both ordersComparator preferred in both ordersTiesOrder-dependentIncomplete
    LAT288 (96.0%)1092
    AF-Next188 (62.7%)217534

    AF-Next’s incomplete comparisons include 28 windows exceeding the frozen audio-context limit and six incomplete judgments; all remain in the denominator. These results support Gemini’s preference for LongCap’s caption content without relying on LongCapBench’s reference format. The judge compares coverage and supported details, so a denser caption set has an advantage, and the test does not measure timing. Independent human verification remains outstanding.

W4. Related benchmarks and the scope of long-form captioning.

Thank you for highlighting these relevant works.

  1. Acknowledging related benchmarks. We recognize the contributions of BLAB, AudioMarathon, and CASTELLA to long-audio understanding, including temporal localization, reasoning, and audio moment retrieval. We will cite all three and acknowledge their contributions in the related-work discussion.

  2. The context is long-form audio captioning. Our reference to recordings “approaching two hours” concerns dense timestamped audio captioning. We agree that duration alone is not distinctive, since BLAB covers the same range. The abstract and introduction state this scope for the model: “For TAC, LongCap … captions full recordings approaching two hours” (L093–095). LongCapBench evaluates structured event descriptions throughout these recordings, which complements the understanding and retrieval tasks evaluated by the cited benchmarks. We will revise the benchmark description near L082 so that it claims dense timestamped caption annotations at this duration, not the duration itself.

W5. External TAC validation.

Thank you for this suggestion. Following your recommendation, we conducted an external evaluation on LAT-Bench, with the setup and results reported in W3, point 3 above. The Gemini A/B results show that LongCap remains strong compared with prior audio captioners LAT and AF-Next on this external benchmark. The native LAT-Bench metric credits at most one prediction per reference event, and LongCap predicts about 8.6× as many events as the references contain (12,460 vs. 1,450 on the 104 English recordings). Most LongCap events therefore cannot contribute to that score, which is why we use the audio-grounded comparison instead.

AuthorsOpenReview ↗

Author Response 3

W6. Per-benchmark performance and averaging scope.

Thank you for this comment.

  1. Scope of Fig. 5(a). The figure equally averages five conditions: AliMeeting, AISHELL-4, AMI-IHM, AMI-SDM, and LongCapBench-Diar-Short. The first four are established public meeting benchmark conditions for speech recognition and diarization. LongCapBench-Diar-Long and CHiME-6 are excluded from this average, so our larger gains on these challenging long recordings do not contribute to Fig. 5(a). Their results are reported separately in Fig. 5(b–c)/Table 4 and Table 5, respectively.

  2. The comparison with MOSS varies across benchmarks. MOSS achieves lower errors on all four metrics on AliMeeting and AISHELL-4. AMI-IHM has mixed results: LongCap achieves lower DER/WER, while MOSS achieves lower cpWER/tcpWER. LongCap achieves lower errors on all four metrics on AMI-SDM and LongCapBench-Diar-Short, and lower macro-averaged errors across the five conditions. The systems are competitive overall, with a particularly large advantage for LongCap on AMI-SDM’s distant-microphone recordings. This highlights a difference in performance under far-field conditions, although the comparison does not isolate its cause.

  3. Clearer presentation. We will explicitly list the five conditions in the figure caption and discuss the per-benchmark differences in the main text, making the scope of our aggregate claim clearer. We will also note that the "Macro mean" row of Table 4 averages six conditions, including LongCapBench-Diar-Long, and therefore differs from Fig. 5(a).

Q1 / Q2. Checkpoint selection and the “AMI overlap” note.

Thank you for pointing out these outdated statements in the Fig. 5 caption.

  1. Origin of the wording. During an earlier data inspection, we found that some AMI evaluation recordings had been inadvertently included in the diarization training set. We removed these recordings and retrained the model. “Benchmark-specific checkpoints” referred to the replacement checkpoint trained after removing the overlap, while “AMI overlap” referred to the earlier training–evaluation leakage. Both phrases were left over from that stage and should have been removed. We will correct the caption.

  2. Checkpoint used for the reported results. All reported LongCap-Diar results in the main benchmark comparison use the same final checkpoint from the corrected training run. We have released LongCap-Diar-35B-A3B, together with the vLLM implementation and Docker setup, and welcome reviewers to test it and independently verify its performance.

Q3. Input protocol and interpretation of Figure 1.

Thank you for asking us to clarify the comparison.

  1. Input protocol. Neither. Each baseline receives the first ten minutes of every LongCapBench-TAC-Long recording as a single input. We did not give the baselines the full recordings, because they do not produce reasonable output at that length. LongCap processes the full recordings with chunk-wise generation, up to 117 minutes. We will state both setups more clearly in the figure caption.

  2. What the comparison establishes. We believe Fig. 1 shows both points you raise. It shows a failure of the baselines: their captions degrade within ten minutes, a much easier input than the full recordings. It also shows LongCap's advantage: over the same first ten minutes, LongCap has lower timeline error and higher caption similarity, and it keeps producing captions for the rest of the recording, where the baselines do not produce reasonable output. Fig. 1 compares complete systems and does not isolate chunk-wise processing; the same-size chunk-wise versus holistic ablation in Table 6 does that.

I thank the authors for their detailed response and additional experiments. My concerns are partly addressed. I think the limited number of long-form recording and external evaluation are still major limitations, and I have reservations about generalizability of the results. Therefore, I am maintaining my score of 2.

Reviewer_kdo9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes “LongCap,” a model that generates high-density, structured descriptions along the time axis for entire long-duration audio signal, along with its evaluation methods and a benchmark data set. The target task is Timestamped Audio Captioning for events (speech, music, and other sounds) present in hours’ worth of audio signals. Captions are generated based on 62 predefined attributes (such as speaker information, language, emotion, timbre, musical instruments, and spatial characteristics) developed in collaboration with experts in speech, music, and acoustics. The evaluation methods include TER (Timeline Error Rate, which measures discrepancies in event timing, GER (Granularity Error Rate, which measures granularity discrepancies such as excessive segmentation, merging, or duplication of events), and tcpCapSim (Time-Constrained Caption Similarity, which evaluates the semantic consistency of attribute descriptions between events that satisfy temporal constraints). An evaluation benchmark (LongCapBench) was constructed to conduct evaluation experiments. Based on the experimental results, the paper claims that superior results were achieved compared to the reference methods of ASR, dialogization, and captioning.

Critical strengths

This study can be regarded as one of the pioneering works in the field, as it addresses the relatively new challenge of long audio captioning, and presents evaluation data, metrics, and experimental results in comparison with other methods. Furthermore, the proposal of a captioning framework, including spoken language, music, and ambient sound, is particularly interesting and is expected to serve as a valuable reference for future research.

Critical weaknesses

From the perspective of the captioning or recognition model, the weakness lies in the lack of clarity regarding technical novelty. In language model research, approaches for handling long contexts have already been proposed or implemented, and the combination of Qwen3.5 and YaRN adopted in this paper is one such example. While this paper proposes “Chunk-wise Processing” in addition to these, it cannot be considered a major technological advancement. If this paper aims to highlight its significance in addressing long audio signals, it must clearly identify the technical challenges that existing language models cannot handle and present a technical solution designed to mitigate them.

From the perspective of the evaluation methodology, a weakness is that it is unclear whether the evaluation directly assesses the difficulties associated with handling long audio data. The primary challenge in handling long inputs lies in how to retain and utilize information even when its necessity or importance is unknown, while keeping computational and memory costs low. To evaluate the extent to which this problem has been solved, it is necessary to conduct evaluations not only based primarily on local accuracy but also on tasks that cannot be solved without utilizing long contexts. It is not entirely clear from the text of this paper alone to what extent the evaluation experiments presented here address this perspective.

Questions for authors
  1. In Fig. 1(b), it appears that some of the comparison methods show better performance than the proposed method. Is this understanding correct?

  2. A natural approach to handling long acoustic signals involves sequentially applying short-duration models over time, storing the results, and then integrating them during processing of the entire signal. What are the advantages of the proposed method over such an approach?

3 comments
AuthorsOpenReview ↗

Author Response 1

Thank you for your thoughtful review.

Checkpoint release. Following internal review, we have released the LongCap-35B-A3B and LongCap-Diar-35B-A3B checkpoints, together with our vLLM implementation and Docker setup. We welcome reviewers to test the models. We hope the comprehensive audio captions generated by LongCap will support further research in the community. Release page.

W1. Technical challenges beyond context-window extension.

Thank you for raising this question and for recognizing the value of our captioning framework. Qwen3.5 and YaRN provide long-context capacity. The additional challenge is generating dense, accurately timed event descriptions on recordings substantially longer than those seen during training (Context Extrapolation). A long context window alone does not solve this. With the same 2B backbone, holistic processing reaches 109.13% DER on 50–78 minute recordings, against 20.30% for chunk-wise processing (Table 6). Absolute timestamps give DER above 100% beyond the 20-minute training span (Fig. 6a). We address this through the following contributions, with comparisons to prior long-form captioners LAT and AF-Next.

  1. Length extrapolation through chunk-wise processing. Chunk-wise generation keeps caption emission near the event's endpoint, while chunk-relative timestamps avoid timestamp values growing with recording position. LongCap-SFT-2 is trained on audio of at most 20 minutes, yet captions recordings of up to 117 minutes, 5.9× its training span (Sec. 3.3; Table 1; Table 8, "Before" rows). LAT and AF-Next-Captioner report training spans of up to 30 minutes, without comparable reported evidence of extrapolation beyond those spans.

  2. Compatibility with a standard LLM pipeline. Our formulation changes audio/output interleaving and event targets while retaining a standard Qwen3.5 backbone with YaRN and ordinary pretraining and SFT. It adds no task-specific attention or memory module. This compatibility is a practical strength. LAT adds a global-timeline stage with iterative cropping and temporal downsampling, while AF-Next replaces RoPE with timestamp-based Rotary Time Embeddings.

  3. Long-range dependencies between events. LAT and AF-Next output timestamped captions without explicit links between events. To our knowledge, LongCap is one of the earliest works to supervise and evaluate event-link prediction in hour-scale audio captioning (Sec. 3.3; Fig. 9; App. D.4). LongCap predicts whether an event shares a source with, continues, or is a recurrence of an earlier event. This makes relationships between distant events an explicit prediction target. We present it as a first measurement, not a solved capability; W2 gives the numbers.

  4. New metrics and benchmark. TER, GER, and tcpCapSim separately measure timeline errors, event-granularity errors, and descriptive agreement (Sec. 2.2). LongCapBench provides human-corrected timelines and a 62-key attribute schema for recordings up to 117 minutes, extending evaluation beyond LAT-Bench's 30-minute span (Table 1).

AuthorsOpenReview ↗

Author Response 2

W2. Evaluation of local events and long-range dependencies.

Thank you for raising this point. We agree that long-form audio captioning should capture both local event information and dependencies across time. Recovering these dependencies requires information beyond an isolated local event. Our evaluation covers both aspects, and we will make the long-context part more prominent in the main text.

  1. Local event quality. We introduce TER, GER, and tcpCapSim to assess event timing, granularity, and descriptive agreement. LongCapBench provides human-corrected timestamps and attributes, reviewed by specialist annotation groups. Together, these support detailed evaluation of individual event captions throughout a recording.

  2. A task that requires long context: speaker consistency. DER and cpWER require one consistent label per speaker across the whole recording, which local chunks alone cannot provide. LongCapBench-Diar-Long has 15 recordings of 50–78 minutes with 7–19 speakers each. LongCap-Diar reaches 6.02% DER and 13.13% cpWER there, against 15.45% and 18.54% for the best evaluated baseline on each metric (Table 4). At the same 2B model size, chunk-wise processing lowers cpWER from 115.87% to 29.01% relative to holistic processing (Table 6).

  3. Long-range event dependencies. The “Dependency Modeling” study in Sec. 4.3 (Fig. 9; App. D.4) evaluates relationships among events across hour-scale recordings, requiring information beyond isolated local captions. Prior long-form captioners such as LAT and AF-Next do not report an equivalent evaluation of cross-event dependencies. Our results show that this remains difficult: against silver references, long-form connectivity F1 is 4.2% for 35B-A3B and 14.8% for 122B-A10B, compared with 63.4% and 68.1% on short clips. We therefore present it as a first measurement of a hard problem.

  4. Computation and memory. LongCap retains all prior audio and captions, so cost grows with recording length. On LongCapBench-TAC-Long, LongCap generates about 3k caption tokens per minute of audio on average, and the audio itself adds about 0.8k tokens per minute (13 per second). The longest recording (117 minutes) therefore takes roughly 450k tokens in total, within the 1M-token window. LongCap adds no extra module, so its memory use is that of the standard Qwen3.5 backbone at the same context length. We will state this cost in the limitations.

AuthorsOpenReview ↗

Author Response 3

Q1. Are some baselines better in Figure 1(b)?

Thank you for this question. Yes, some baselines have lower GER in Fig. 1(b). However, GER’s split/merge component measures inconsistencies within activity shared by the reference and predictions, while missing activity is penalized by TER. Because GER is normalized by total reference event duration, substantial omissions can produce a low GER. Its comparison is therefore most informative when temporal coverage is comparable. As footnote 4 of the paper notes, several low-GER baselines in Fig. 1 have substantially higher TER, so their lower GER alone does not establish better event segmentation or overall captioning quality.

Q2. Advantages over short-duration captioning followed by integration.

Thank you for suggesting this comparison. Sequential short-duration captioning followed by integration is a reasonable alternative. LongCap differs in the following ways:

  1. Consistency across the recording without a separate integration stage. LongCap conditions every chunk on the retained audio and captions, so identities stay consistent as it generates. The speaker-consistency results in W2 show this over 50–78 minutes. The proposed alternative must recover the same associations afterwards from text alone. For event links specifically, we do not claim an advantage yet: in App. D.4, relabeling links with a text LLM gives higher long-form F1 than the models' native links (17.4/22.3% vs. 4.2/14.8%).

  2. Processing boundaries need not coincide with event boundaries. Both approaches can use fixed windows without semantic pre-segmentation. The challenge is handling events that span those windows, such as continuing music or background sound. LongCap assigns an event to the chunk containing its endpoint and represents its start relative to that chunk, allowing a negative start for an event beginning earlier. Retained audio supplies the preceding context (Secs. 3.1–3.2). Independently generated captions can fragment the same event across windows, leaving the integration stage to reconcile partial or duplicated descriptions. LongCap’s formulation directly accommodates events spanning multiple processing chunks.

Reviewer_SfsQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents LongCap for timestamped audio captioning of long recordings. It interleaves audio chunks with structured event captions, uses relative timestamps, and retains earlier audio and captions in context. Training combines diarization, captioning, and text data, followed by two stages of SFT with event-wise reannotation. The paper introduces LongCapBench and three evaluation metrics and evaluates five model sizes. General captioning reaches recordings approaching two hours; the 16-hour experiment concerns speech recognition and diarization.

Critical strengths
  1. Useful task and benchmark. The structured representation and human-corrected LongCapBench cover speech, music, and sound effects. This is a practical contribution to evaluating detailed descriptions of long audio.
  2. A workable long-audio training recipe. Chunking, relative timestamps, and event-wise reannotation enable caption generation beyond the initial training duration. The model-size studies provide useful observations about descriptive detail and long-output reliability.
Critical weaknesses

W1: Limited methodological novelty The paper represents a substantial effort toward long-form audio captioning, but I find its methodological novelty limited. The architecture uses established components, and training follows standard pretraining and SFT with specific data mixtures. Much of the captioning supervision comes from Qwen3.5-Omni-Plus, so the approach largely transfers a proprietary model’s descriptions into a system for longer audio. The paper should clarify what new capability or methodological insight is established beyond this integration and synthetic-label distillation.

W2: Training and benchmark annotation pipelines are closely aligned Qwen3.5-Omni-Plus generates captions for SFT and supplies the initial LongCapBench annotations, both using the same structured caption schema. This shared process may favor the teacher’s event selection and annotation preferences. I acknowledge that benchmark recordings are excluded from training and that humans substantially revise the labels. However, these safeguards do not establish that the reported gains generalize beyond the shared annotation setup. Since the main captioning evidence relies on this benchmark, I am not yet convinced that it demonstrates broader usefulness to the community. Evaluation on independently annotated audio would strengthen this claim.

W3: Unclear event completion at chunk boundaries Assigning events to the chunk containing their endpoint requires the model to infer completion without access to subsequent audio. Although earlier audio is retained and events can span chunks, pauses near boundaries may cause premature emission or incorrect deferral. The paper should clarify how these cases are handled and report performance for events crossing chunk boundaries. This matters because reliable timing is central to the proposed task.

Questions for authors
  1. Boundary handling: How are pauses and unfinished events near chunk boundaries handled? Can you report accuracy for events crossing boundaries versus events contained within a chunk?
  2. Checkpoint selection: Why was test WER used to select the absolute-timestamp checkpoint? Please clarify whether other comparisons use test-based selection and provide validation-selected results for this ablation.
  3. Text mixing: Figure 6c changes both the mixture and total token budget. Do you have a comparison using the same training budget to isolate the effect of text data?
3 comments
AuthorsOpenReview ↗

Author Response 1

Thank you for your thoughtful review.

Checkpoint release. Following internal review, we have released the LongCap-35B-A3B and LongCap-Diar-35B-A3B checkpoints, together with our vLLM implementation and Docker setup. We welcome reviewers to test the models. We hope the comprehensive audio captions generated by LongCap will support further research in the community. Release page.

W1. Methodological novelty beyond integration and distillation.

Thank you for recognizing the effort behind LongCap and for raising the question of novelty. We first answer the point about distillation, then compare LongCap with two prior long-form audio captioners, LAT and AF-Next.

What is established beyond distillation. Qwen3.5-Omni-Plus supplies clip-level labels only: it annotates clips of at most one minute (Sec. 3.3.2), and its own long-form output degrades within the first 10 minutes in Fig. 1. The hour-scale capability therefore cannot be inherited from the teacher; it comes from the formulation described below. Two further results go beyond the teacher's labels. First, we find that the teacher's captions lose attribute density for later events, and event-wise reannotation removes this effect (Fig. 8c). Second, on LongCapBench-TAC-Short, LongCap exceeds the teacher in caption similarity (Fig. 7a).

  1. Length extrapolation through chunk-wise processing. The key contribution of our chunk-wise processing is its demonstrated extrapolation. LongCap-SFT-2 is trained on audio of at most 20 minutes, yet captions recordings of up to 117 minutes, 5.9× its training span (Sec. 3.3; Table 1; Table 8, "Before" rows). Table 6's 2B diarization ablation shows the advantage of chunk-wise over holistic processing on long recordings at the same model size. LAT and AF-Next report training spans of up to 30 minutes, and we found no reported evaluation beyond those spans.

  2. Compatibility with a standard LLM pipeline. Chunk-wise processing works on a standard Qwen3.5 backbone with YaRN context extension, using ordinary pretraining and SFT, so it fits directly into mainstream practice. We see this as a simplicity strength rather than a weakness. In contrast, LAT and AF-Next adapt their models for long audio with extra designs: LAT adds a global-timeline stage with iterative cropping and temporal downsampling, and AF-Next replaces RoPE with timestamp-based Rotary Time Embeddings.

  3. Long-range dependencies between events. LAT and AF-Next output timestamped captions without explicit links between events. To our knowledge, LongCap is one of the earliest works to supervise and evaluate event-link prediction in hour-scale audio captioning (Sec. 3.3; Fig. 9; App. D.4). LongCap predicts whether an event shares a source with, continues, or recurs from an earlier event. We present this as a first measurement, not a solved capability: against silver references, long-form connectivity F1 is 4.2% for 35B-A3B and 14.8% for 122B-A10B, far below the short-form scores of 63.4% and 68.1%. Linking events across an hour is therefore much harder than describing them, and the benchmark makes that gap measurable.

  4. New metrics and benchmark. We introduce three metrics, TER, GER, and tcpCapSim, which separately measure timeline errors, event-granularity errors, and descriptive agreement (Sec. 2.2). We also introduce LongCapBench with extensive professional human labeling (Table 1). Compared with LAT-Bench, it is longer (up to 117 vs. 30 minutes), finer-grained (8.5–15.4 vs. 1.4 events per minute), and more structured (a 62-key attribute schema vs. natural-language descriptions).

AuthorsOpenReview ↗

Author Response 2

W2. Training and benchmark annotation pipelines are closely aligned.

Thank you for raising this concern. To test whether our gains extend beyond LongCapBench, we evaluate LongCap on LAT-Bench, which is annotated independently of our pipeline.

  1. Granularity limits the native metric. LAT-Bench is a good fit for this test: it contains long recordings (up to 30 minutes) and its own human-verified annotations. However, LongCap is much finer-grained than its references. On the 104 English recordings, LongCap predicts 12,460 events (median 3 s), whereas the references contain only 1,450 (median 34 s). The native metric credits at most one prediction per reference event, so most LongCap events cannot contribute to its score.

  2. Audio-grounded A/B test. We therefore compare complete caption sets directly against the audio. We sample three 1-minute windows (early, middle, and late) from the full-recording outputs of 100 recordings, 300 windows in total. Gemini-3.1-Pro hears the audio and compares LongCap's captions with a competitor's for coverage and supported details. Model names and LAT-Bench references are withheld, and both candidate orders are judged.

  3. Results. Gemini prefers LongCap in both orders on 96.0% and 62.7% of the 300 windows; the prior captioners are preferred in both orders on only 1 and 2 windows:

CompetitorLongCap preferred in both ordersCompetitor preferred in both ordersTiesOrder-dependentIncomplete
LAT288 (96.0%)1092
AF-Next188 (62.7%)217534

AF-Next's incomplete comparisons include 28 windows exceeding the frozen audio-context limit and six incomplete judgments; all remain in the denominator.

W3 / Q1. Event completion at chunk boundaries.

Thank you for raising this concern and for suggesting the analysis.

How boundaries are handled. LongCap emits each event in the chunk that contains its endpoint. If a sound is still ongoing at the end of a chunk, the model learns from the training targets to defer it. The event is then emitted in a later chunk, with a negative relative start, using the retained audio. There is no hand-crafted rule for pauses; the model itself predicts whether an event has ended.

Endpoint accuracy. We evaluated LongCap-35B on all 1,500 LongCapBench-TAC-Short clips and measured the median absolute endpoint error of automatically matched events. Two factors may contribute to these errors:

  1. A chunk boundary limits the future audio available for judging whether an event has ended.
  2. Longer events are harder to localize precisely.

The table separates events by duration and by whether they cross a chunk boundary:

Event durationContained: median error (s)Crossing: median error (s)
≤2 s0.1700.162
2–5 s0.1880.186
5–10 s0.1970.207
10–30 s0.2000.240
>30 s0.0960.265

Reading the table. For events up to 10 s, crossing and contained events differ by at most 0.01 s. The gap grows with event duration and reaches 0.17 s for events longer than 30 s.

Limitation and possible fix. We will acknowledge the limited right context in our limitations section. One possible fix is to defer events whose predicted endpoints fall within a few seconds (e.g., 2 s) of a chunk's end, so that their endpoints and captions are reconsidered after reading the next chunk. We leave this to future work.

AuthorsOpenReview ↗

Author Response 3

Q2. Checkpoint selection for the timestamp ablation.

Thank you for pointing this out.

  1. Validation-based selection. We agree that checkpoints should be selected on a validation set.

  2. Our setup. Our experiments did not include a separate validation set. For the relative-timestamp model, we therefore used the final checkpoint (100k steps) without selection. For the absolute-timestamp model, DER exceeds 100% at every checkpoint, so it cannot meaningfully rank them. We instead used test WER and chose the checkpoint closest to the relative model's WER.

  3. Selection does not change the conclusion. The table compares the relative model with the best and worst values across all absolute-timestamp checkpoints (30k–100k steps):

ModelWER (%)DER (%)
Absolute timestamps (best–worst)9.73–12.20144.28–639.96
Relative timestamps, final checkpoint12.7220.24

The absolute-timestamp model has lower WER at every checkpoint, by 0.5 to 3.0 points, and DER above 144% at every checkpoint. Thus, the choice of absolute-timestamp checkpoint does not change our conclusion: relative timestamps cost slight WER degradation but reduce DER from over 144% to 20.24%.

Q3. Isolating the effect of text mixing.

Thank you for this suggestion. We agree that Fig. 6(c) changes both the data mixture and the total training budget.

  1. Original setup: fixed diarization compute. In Fig. 6(c), both models see the same amounts of diarization data. The text-mixed model trains on additional text, so its total budget is larger.

  2. New setup: fixed total compute. We now compare the diarization-only model with an early checkpoint of a model trained on diarization and text, within the same overall training budget. This means less compute is spent on diarization.

  3. The conclusion does not change. Even with less diarization training, the text-mixed model still has lower WER on the long-form recordings of Fig. 6(c):

ModelWER (%)DER (%)
w/o text7.999.87
w/ text, match diarization compute7.089.52
w/ text, match overall compute7.239.66

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.