acceptodds
Under review as a conference paper at ICLR 2027

ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

Abstract

Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation–training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-llms, multimodal, audio, reinforcement-learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WXS9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first introduces EgoSCAPE, a benchmark containing 700 egocentric audio clips with human-verified dense captions, sound sources, timelines, and temporal grounding queries. It also proposes ATTEST, a GRPO-based training framework combining counterfactual audio corruption, attention-density rewards, and segment-level credit assignment. Results across three backbones show improvements in captioning and temporal grounding while preserving performance on the evaluated general audio benchmarks.

Critical strengths
  1. The benchmark provides detailed annotations for real audio. EgoSCAPE combines dense captions, sound sources, timelines, and grounding queries. Audio-only verification by three annotators and the reported agreement support annotation quality.

  2. The training design is well motivated and supported by ablations. In particular, segment-level advantages address credit assignment in long outputs. They outperform sequence-level advantages with the same rewards, showing a benefit beyond reward design alone.

  3. The evaluation is broad and shows consistent improvements. Results cover three backbones, multiple grounding datasets, general audio benchmarks, and audio corruption tests.

  4. The authors promise to release the benchmark annotations and training code upon acceptance.

Critical weaknesses
  1. The counterfactual reward can provide misleading feedback. If the model hallucinates an event after silencing, it can penalize a correct prediction on the original audio. The ablations demonstrate usefulness, but the frequency of such incorrect penalties is not examined.

  2. The refinement prompt prohibits adding events missed by the detector, and human verification focuses on existing annotations. It is unclear whether missing audible events were systematically recovered.

Questions for authors
  1. Can you estimate how often the counterfactual reward penalizes correct original captions because of hallucinations on the silenced audio?
  2. Did annotators actively search for missing events? How many annotations were added, removed, or corrected?
  3. Does a human evaluation of model outputs confirm the hallucination reductions measured by FLAM?
4 comments
AuthorsOpenReview ↗

Response to Reviewer WXS9 (1/2)

We thank the reviewer for their positive and constructive feedback. Our responses to each concern are below, and we have updated the manuscript accordingly.

R1. Correct predictions being penalized by RCAGR_\text{CAG} (Weakness 1 and Question 1)

R1.1 How the penalty works

The counterfactual reference D^w′\hat{D}'_w is decoded once and shared by all rollouts in a group, and Eq. 8 normalizes the window score within each window. As a result, a penalty that every rollout receives equally has almost no effect on training. If all rollouts correctly predict an event in a silenced window and the reference also predicts it, each rollout gets about the same penalty, so its CAG advantage is close to zero. The penalty changes the update mainly when rollouts disagree, and in that case the GT-anchored DAC advantage (Eq. 7, η3=1\eta_3 = 1) still rewards the rollouts that predicted the correct event. A correct prediction is discouraged only when the model would produce it from its prior alone, which is the behavior CAG is designed to reduce.

R1.2 Frequency of correct predictions included in RCAGR_\text{CAG}

To address this question, we measure how often RCAGR_\text{CAG} penalizes a correct prediction. Among all predicted segments that fall inside a silenced window, we compute the percentage that match a ground-truth segment on the original audio (tIoU ≥\ge 0.5 and FENSE ≥\ge 0.4) and also receive a nonzero CAG penalty. The table below reports this percentage over the course of RL training. Such cases are rare even early in training (6.7%) and decrease steadily to 1.9% as the model learns to stop predicting events on silenced audio.

Training iterations% GT Penalized in RCAGR_\text{CAG}
0-1256.7
125-2504.5
250-3753.3
375-5002.5
500-6251.9

R1.3 Gating the counterfactual penalty

Included in Appendix D.8 in the updated manuscript.

We also train another version of RCAGR_\text{CAG}. For this version, we only penalize a segment d^j(g)\hat{d}^{(g)}_j in Eq. (1) if it does not match to a ground-truth segment with temporal IoU ≥\ge 0.5 and FENSE ≥\ge 0.4. We report this gated version of RCAGR_\text{CAG} in the table below, which achieves similar performance to ATTEST showing that gating is not necessary. We have included this result in Appendix D.8 Table 19 of the updated manuscript.

ModelESDAC: Overall [email protected]ESDAC: Overall Hal%ESDAC: Long (>120s) [email protected]ESDAC: Long (>120s) Hal%CMTAG: [email protected]CTAG: [email protected]
Qwen2.5Omni + SFT + ATTEST0.42730.50.41434.20.9090.297
Qwen2.5 Omni + SFT + ATTEST GT-gated RCAGR_\text{CAG}0.42031.10.40933.90.9020.286

ESDAC: EgoSCAPE Dense Audio Captioning, CMTAG: Clotho-Moment Temporal Audio Grounding, CTAG: CASTELLA Temporal Audio Grounding

R2. More details about human annotation and refinement prompt (Weakness 2 and Question 2)

R2.1 Why the refinement prompt disallows new events.

The omni-LLM sees the video, so letting it add events would risk annotations driven by what is visible rather than what is audible. This would undermine an audio-only benchmark. Restricting it to keep, drop, or relabel prevents such visual leakage.

R2.2 Verification and correction.

Annotators listened to the audio only (no video) and accepted or rejected each entity, optionally suggesting a corrected boundary or caption. An expert adjudication pass then resolved every flagged entity using these suggestions. Additions came from missed events the annotators noted while correcting a neighboring segment. Most corrections were drops, which shows that verification mainly removed false positives, the dominant failure mode of automatic annotation.

Task% Add.% Drop.% Mod.
Dense Audio Captioning3.567.329.2
Audio Timeline Generation1.774.324.0
Temporal Audio Grounding2.357.340.4

Additional Note: Hallucination rate counts a prediction as hallucinated only if it is unmatched and also has low FLAM audio-text confidence, so predictions of genuinely audible sounds are unlikely to be counted as hallucinations by our metrics.

AuthorsOpenReview ↗

Response to Reviewer WXS9 (2/2)

R3. Human evaluation of model outputs for hallucination reduction (Question 3)

Included in Appendix D.9 in the updated manuscript

We perform a small human evaluation over 50 dense captioning samples randomly sampled from the EgoSCAPE benchmark and ask 3 annotators to go through each audio sample and its predictions to mark whether a predicted sound event is hallucinated or not. We perform this study on Qwen2.5 Omni model with SFT, TimeProRL and ATTEST. The annotators do not know which prediction is obtained from which model. For each segment, human hallucination % reports the majority decision between the three annotators. Additionally, we report the Fleiss κ\kappa to show the inter-annotator agreement for each model. This analysis has been added to Appendix D.9 Table 20 of the updated manuscript.

ModelHal % (50 clip subset)Human Hall. % (50 clip subset)Fleiss κ\kappa
Qwen 2.5 Omni + SFT47.543.80.742
+ TimeProRL32.133.00.712
+ ATTEST30.229.40.723

We thank the reviewer for their feedback of our paper and for their effort reviewing our manuscript. We would appreciate if the reviewer can read our responses to their reviews and acknowledge if their concerns are resolved. We understand that reviewing requires significant effort and we are very grateful to the reviewer's participation in this process.

Thank you to the authors for the detailed responses and clarifications. I will maintain my original score.

Reviewer_TwpiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ATTEST, a GRPO-based framework for improving audio temporal grounding. It introduces a counterfactual grounding reward, an attention-density reward, and token-wise advantages. The paper also presents EgoSCAPE, a benchmark with dense annotations for real-world audio. Experiments show improvements in temporal grounding and reduced hallucinations across multiple models and datasets.

Critical strengths
  • The counterfactual grounding reward is an interesting idea for encouraging models to rely on acoustic evidence. The overall framework is simple and does not require architectural changes.

  • The paper provides a useful benchmark and fairly extensive experiments. The improvements in grounding performance and hallucination reduction are encouraging.

Critical weaknesses
  • The paper does not convincingly address long-audio-specific challenges. The benchmark is not particularly long, with an average duration of only 67.6 seconds and a median of 43.3 seconds. More importantly, the proposed methods address general problems such as hallucination, temporal grounding, and credit assignment, which also exist in short audio. I do not see what makes these methods particularly suited for long audio. The experiments also do not clearly demonstrate why long audio requires these techniques or whether they are especially effective on longer recordings.

  • The counterfactual reward does not fully guarantee evidence grounding. CAG penalizes predictions that persist after audio is silenced, but does not directly penalize new hallucinations in the silenced region. For example, if a predicted dog barking becomes a bird singing after intervention, the penalty may be small. It is therefore unclear whether CAG encourages genuine acoustic grounding or simply sensitivity to intervention.

Questions for authors
  • Can the authors better demonstrate the long-audio-specific benefits, for example by controlling event density and output length across different audio durations?

  • How often does the model generate new hallucinations after audio intervention? Can the authors analyze this behavior before and after ATTEST?

  • Since EgoSCAPE annotations are initially generated using audio, video, and narrations, is there any independent audio-only validation of annotation completeness and temporal accuracy?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Twpi (1/2)

We thank the reviewer for their feedback. Our responses to each concern are below, and we have updated the manuscript accordingly.

R1. Long-audio specific gains (Weakness 1 and Question 1)

R1.1 What is specific to long audio?

  • We agree with the reviewer that problems such as hallucination, temporal grounding and credit assignment also apply to shorter audios, however, long audio captioning makes these problems worse.
    • Figure 3 shows that even within EgoSCAPE benchmark, hallucination is more severe for long audio samples compared to short audio samples.
    • As shown in Table 1, the hallucination % for most of the baselines, especially the ones that we use as base models, is higher for EgoSCAPE which consists of long audios compared to AudioSet-SL which consists of short audios. Please refer to R1.2 for a more controlled experiment between short and long audios.
    • To highlight the importance of token-wise fine-grained advantage for credit assignment over long (>120s) and short (<30s) audio samples, we provide the following table as an ablation similar to Table 3. This shows that dense credit assignment helps more in the case of long audio samples to improve their [email protected] score showing that the problem of credit assignment is more severe for long audio samples because of long response lengths.
ModelShort (<30s) [email protected]Short (<30s) Hal%Long (>120s) [email protected]Long (>120s) Hal%
Qwen2.5 Omni + ATTEST0.43828.20.41434.2
Qwen2.5 Omni + ATTEST w/o token-adv.0.43428.60.39834.9

R1.2 Controlled experiment for short vs. long audios

  • Controlled event density experiment. To remove the effect of sound event density, and show the effect of the proposed approach on short audios (<30s) versus long audios (>120s), we show the improvement of the proposed approach on EgoSCAPE benchmark on dense audio captioning using all samples that have an event density between 0.7 and 0.9 events per second. We can see from the below table that (i) the SFT-tuned models show more hallucination rate on long audios compared to short audios, and (ii) ATTEST reduces the gap between hallucination rate of short and long audios across all the three models.
Model[email protected] (Short)Hal% (Short)[email protected] (Long)Hal% (Long)Hal% Gap (Short−Long)
Qwen2.5 Omni + SFT0.38444.80.35952.2-7.4
+ ATTEST0.43429.10.41032.1-3.0
Qwen3 Omni + SFT0.38043.20.35152.2-9.0
+ ATTEST0.43528.60.41931.8-3.2
AudioFlamingo 3 + SFT0.35941.30.32846.9-5.6
+ ATTEST0.41029.70.39434.1-4.4
  • Controlled output length experiment. Dense audio caption outputs for long audio (>120s) samples are inherently long compared to short audio (<30s) samples. Hence, to show the effect of using ATTEST on similar output lengths, we show the performance of ATTEST on the temporal audio grounding (coarse) queries of EgoSCAPE similar to Appendix D.2 Table 13. Temporal grounding outputs typically consist of a time interval that does not depend on the duration of the audio. With controlled output length on TAG task, we see that the [email protected] gap for short and long audio samples is reduced after ATTEST.
Model[email protected] (Short)[email protected] (Long)[email protected] Gap (Short−Long)
Qwen2.5 Omni + SFT0.5680.3190.249
+ ATTEST0.5970.4580.139
Qwen3 Omni + SFT0.5490.3260.223
+ ATTEST0.6050.4790.126
AudioFlamingo 3 + SFT0.5010.3080.193
+ ATTEST0.6290.4900.139

R1.3. Regarding benchmark data (Weakness 1)

Included as a limitation in Appendix A in the updated manuscript

Compared to the only prior benchmark that includes dense audio captions (FTAR [R1]), our audios are about 6.8 times longer on an average and the max duration is 5 minutes compared to 10 seconds of FTAR (refer to Appendix B.1 Table 5). Dense audio captioning requires significant amount of manual annotation effort and validating the dense captions for audios longer than 5 minutes remains a challenge. We agree with the reviewer that the community needs dense captioning benchmarks longer than 5 minutes, and we believe our benchmark is a step towards the same. We have added this as a limitation in Appendix A.

[R1] Wang, Hualei, et al. "Listening between the frames: Bridging temporal gaps in large audio-language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 31. 2026.

AuthorsOpenReview ↗

Response to Reviewer Twpi (2/2)

R2. New hallucinations on audio intervention (Weakness 2 and Question 2)

R2.1 Penalty during training time

Included in Appendix D.8 in the updated manuscript.

  • We agree with the reviewer that we do not penalize new hallucinations in the silenced region after corruption. This is because we only run a single greedy decoding of the silenced audio as mentioned in L217-219. Hence, there is no penalty or advantage computation on the single rollout of the silenced audio during training.
  • To penalize new hallucinations introduced by silence, we train a new version of ATTEST called ATTEST-Sil in which we generate rollouts on the silenced audios (reward formulation present in Appendix D.8). This reward penalizes any predicted segment including new ones in silenced window W\mathcal{W}. We report the results of this variant below. Note that we apply this variant as a replacement of the existing RCAGR_\text{CAG} reward. Compared to original ATTEST, the performance of this variant is similar or slightly worse on different metrics and tasks. We have added this new variant of CAG in Appendix D.8 Table 19 of the updated manuscript.
ModelESDAC: Overall [email protected]ESDAC: Overall Hal%ESDAC: Long (>120s) [email protected]ESDAC: Long (>120s) Hal%CMTAG: [email protected]CTAG: [email protected]
Qwen2.5 Omni + ATTEST0.42730.50.41434.20.9090.297
Qwen2.5 Omni + ATTEST-Sil0.42031.60.40833.80.8990.278

ESDAC: EgoSCAPE Dense Audio Captioning, CMTAG: Clotho-Moment Temporal Audio Grounding, CTAG: CASTELLA Temporal Audio Grounding

R2.2 Whether ATTEST encourages genuine grounding

Section 5.5 Figure 4 shows that when we intervene the samples from EgoSCAPE with audios from AudioSet-SL, ATTEST-tuned models can still correctly predict the acoustic events in the intervened segment. This shows that the models learn genuine audio grounding and not just sensitivity to intervention.

R2.3 New hallucinations before and after ATTEST

  • Appendix D.5 Figure 7 shows the robustness behavior of different models before and after applying ATTEST. Across all the models, we observe that the frequency of model predicting hallucinated segments in a silenced region reduces significantly after applying ATTEST. Note that this figure counts any hallucinated event in the silenced region.
  • New hallucinations. To count only the new hallucinations in the silenced region, we report the following table for different models. A new hallucination is a predicted segment in the silenced region of the silenced audio, that does not match to any segment in the original audio in the same region. Here we report the percent of total audio samples in EgoSCAPE in which there is at least one new hallucination that appears in the silenced region while performing dense audio captioning. A good model should not predict any segment within the silenced region. The following table confirms that even if we only count new hallucinations (as opposed to counting any type of hallucinations in Appendix D.5 Figure 7), ATTEST is significantly better in reducing them compared to the SFT model (before ATTEST).
ModelShort (<30s)Medium (30–120s)Long (>120s)
Qwen2.5 Omni + SFT5.42%8.42%35.04%
+ ATTEST0.00%2.89%6.84%
Qwen3 Omni + SFT11.33%28.16%41.88%
+ ATTEST2.46%6.84%12.82%
AudioFlamingo 3 + SFT8.37%55.00%80.34%
+ ATTEST2.96%9.47%17.09%

R3. Audio-only validation of benchmark (Question 3)

As mentioned in Appendix B.4, the participants only had access to the audio during the human verification stage of the benchmark. We report the annotation quality of the benchmark and inter-annotator agreement in Table 9 of the Appendix. The participants were specifically instructed to only check the annotations based on the audio that is provided to them and reject temporal segments and captions that contain any description or information about a visual entity or any entity that the audio does not support.

We thank the reviewer again for their feedback of our paper and for their time reviewing our manuscript. We would appreciate if the reviewer can read our responses to their review and adjust their rating if their concerns are resolved. We understand that reviewing requires significant effort and we are very grateful to the reviewer's participation in this process.

Reviewer_FM2AOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of temporal grounding in long audio streams, where the aim is to identify episodes or events in the audio, determine the start and end times of those episodes, and associate audio captions with those episodes. The paper makes two main types of contributions to this problem. First, they have developed a benchmark dataset that provides longer forms audios with associated audio captions, audio timelines and annotations. This dataset is built on an existing dataset, but is complemented with manually verified captions, timelines and annotations. Second, the have devised a reinforcement learning technique to avoid hallucinations in temporal grounding.

Critical strengths

The densely annotated audio benchmark is a useful contribution to research in this field. The main merit of this benchmark is that it provides reasonably dense labelling of events in longer form audios.

The proposed reinforcement learning framework provides a novel method for improving the accuracy of predictions by using a form of counterfactual testing that reduces the likelihood of hallucinations in temporal grounding.

Critical weaknesses

In audio benchmarks, it is important to have a reasonable diversity of speakers so that the evaluated methods do not overfit to one class of speakers. It is not clear from the paper how diverse the benchmark dataset is.

Questions for authors
  1. Can you provide information about the diversity of speakers in your benchmark, i.e., number of languages, number of speakers, number of native/non-native speakers of those languages, audio source (news media, studio conversations, public conversations, etc)?

  2. What types of corruption operators are used in the counterfactual scheme to derive new audios from the original audios? (e.g., background noise, variations in amplitude, temporal reordering of audio segments, etc)

  3. Can you provide some assessment of the statistical significance of the differences between ATTEST and the other methods tested in your main results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer FM2A (1/2)

We thank the reviewer for their positive and constructive feedback. Our responses are below, and we have updated the manuscript accordingly.

R1. Diversity of the dataset (Weakness 1 & Question 1)

Included as Appendix B.3 in the updated manuscript.

We agree that speaker diversity is important for any benchmark involving speech. We note, however, that EgoSCAPE is primarily a sound-event benchmark: speech accounts for about 32% of dense caption segments (self speech 10.1%, other speech 21.6%; Appendix B.2 Fig. 5f), and 222 of the 700 audio samples contain no speech at all. The remaining segments cover the wearer’s interactions with objects, body sounds, ambient sounds, and electronic media, spanning more than 40 everyday scenarios such as cooking, cleaning, shopping, and carpentry. All samples are unscripted, in-the-wild egocentric recordings rather than broadcast or studio audio. With this context, we provide the diversity-specific statistics below and also added it to Appendix B.3 Tables 6-8 of the updated manuscript.

R1.1 Number of languages

We show the distribution of languages in the speech samples (478 out of 700 total). Although the majority language is English, the benchmark consists of samples from a diverse set of languages. Some of the samples consisted of speech from more than one language.

LanguageSpeech samples% of 478 speech samples% of 700 samples
English39482.4%56.3%
Spanish336.9%4.7%
Hindi285.9%4.0%
Japanese71.5%1.0%
Mandarin/Chinese61.3%0.9%
Unidentified non-English61.3%0.9%
Russian51.0%0.7%
Malayalam40.8%0.6%
Italian30.6%0.4%
Arabic30.6%0.4%
Telugu30.6%0.4%
Bengali30.6%0.4%
Korean20.4%0.3%
Tamil20.4%0.3%
Nepali10.2%0.1%
Portuguese10.2%0.1%
Sinhala10.2%0.1%
Romanian10.2%0.1%
(no speech, so no language)--31.7% (222)

This distribution is based on the language detected by the Whisper ASR model while performing ASR.

R1.2 Regions where the clips are sourced from

The Ego4D data was collected by a group of institutions distributed world-wide. Each institution recorded its participants locally, so the collection sites reflect broad geographic diversity. We show the distribution of the source institutes for the 700 samples that we have included in the benchmark. This shows that the benchmark consists of samples sourced from diverse parts of the world.

InstituteCountrySamples (all)% of 700
Facebook Reality Labs (vendor collection, Seattle area)USA17925.6%
King Abdullah Univ. of Science & Technology (KAUST)Saudi Arabia10515.0%
IIIT HyderabadIndia10114.4%
Georgia TechUSA649.1%
University of BristolUK568.0%
University of CataniaItaly557.9%
University of MinnesotaUSA405.7%
University of TokyoJapan405.7%
Indiana UniversityUSA395.6%
National University of SingaporeSingapore213.0%

R1.3 Number of speakers

The below table shows the number of speakers in each of the audio samples among the 700. As mentioned before, 222 audio samples do not contain any speech. We compute the number of speakers using the narrations provided in Ego4D when they correspond to the speech segments.

SpeakersClips (all)% of 700
022231.7%
117024.3%
217124.4%
37610.9%
4334.7%
5+284.0%

R1.4 Scenarios covered

Ego4D labels the scenarios associated with each video. Following is the distribution of samples across different scenarios in the benchmark showcasing the top 15 scenarios. Our benchmark's subset consists of a total of 40 different scenarios from Ego4D. This table shows the diversity of audio sources in our benchmark.

ScenarioSamples (all)% of 700
Cleaning / laundry18726.7%
Cooking18226.0%
Making coffee8712.4%
Playing cards507.1%
Clothes, other shopping476.7%
Grocery shopping indoors415.9%
Playing board games385.4%
Working at desk355.0%
Talking with family members334.7%
Carpenter263.7%
Indoor Navigation (walking)243.4%
Making Bricks243.4%
Attending a party213.0%
Watching tv213.0%
Fixing something in the home202.9%

R1.5 Native/Non-native Speakers

Ego4D videos do not contain any information about native/non-native speakers in a video. Since our benchmark is derived from Ego4D, it lacks the same information as well.

AuthorsOpenReview ↗

Response to Reviewer FM2A (2/2)

R2. Corruption operator in Counterfactual Reward RCAGR_\text{CAG} (Question 2)

We evaluated five corruption operators (Appendix D.4, Table 16): silence, RMS-matched pink noise (background noise), time-shuffle (temporal reordering), swapping in a random audio clip, and low-pass filtering. Silence works best, giving [email protected] = 0.427 and Hal% = 30.5 on EgoSCAPE, compared with 0.409–0.421 and 31.0–32.1% for the alternatives. It also gives the best CASTELLA [email protected] (0.297 vs. 0.255–0.268). The reason is that RCAGR_\text{CAG} needs an operator that removes the acoustic evidence in a window, so that any prediction surviving the intervention is a hallucination, and silence is the only operator that guarantees this. For the same reason, we did not use amplitude scaling as it preserves the events rather than removing them.

R3. Statistical significance of the results (Question 3)

We assess significance with a paired bootstrap test (10,000 iterations) over the test samples (audio samples for EgoSCAPE captioning tasks and queries for Clotho-Moment). We report the mean difference (Δ\Delta), its 95% confidence interval, and the two-sided p-value, with Bonferroni correction across the 12 comparisons (significant if p<0.05/12≈0.0042p < 0.05/12 \approx 0.0042). For brevity, we report [email protected] on EgoSCAPE for dense audio captioning (DAC) and audio timeline generation (AT), and [email protected] on Clotho-Moment for temporal audio grounding (TAG).

Base modelTask (metric)vs SFT: Δvs SFT: 95% CIvs SFT: p-valuevs TimePro-RL: Δvs TimePro-RL: 95% CIvs TimePro-RL: p-value
Qwen2.5 OmniDAC ([email protected])+0.060[+0.053, +0.067]< 0.0001+0.017[+0.011, +0.022]< 0.0001
Qwen2.5 OmniAT ([email protected])+0.035[+0.029, +0.040]< 0.0001+0.009[+0.004, +0.015]0.0013
Qwen2.5 OmniTAG ([email protected])+0.034[+0.029, +0.040]< 0.0001+0.014[+0.008, +0.019]< 0.0001
AudioFlamingo 3DAC ([email protected])+0.056[+0.048, +0.063]< 0.0001+0.012[+0.008, +0.017]< 0.0001
AudioFlamingo 3AT ([email protected])+0.042[+0.035, +0.048]< 0.0001+0.013[+0.008, +0.019]< 0.0001
AudioFlamingo 3TAG ([email protected])+0.053[+0.045, +0.061]< 0.0001+0.011[+0.007, +0.016]< 0.0001

The improvements of ATTEST over both SFT and TimePro-RL are statistically significant (p < 0.0042, Bonferroni-corrected) for all three tasks on both base models.

Reviewer_FM2AOpenReview ↗

Response to authors' comments

Thank you for providing the additional information. It would be useful to add this information to the supplementary material if accepted. Overall, I will maintain my original rating for the manuscript.

We thank the reviewer for their feedback to our paper and for their acknowledgement to our response. We will incorporate more of the above changes to the appendix of our paper (Appendix B.3 already incorporated).

Comments on the paper

1 comment

Summary of the Discussion Period

We thank the AC and reviewers for their time and feedback.

Scores. WXS9: 3 (weak accept), FM2A: 3 (weak accept), Twpi: 2 (weak reject). Both WXS9 and FM2A acknowledged that our responses addressed their concerns and kept their positive ratings. Reviewer Twpi said that their concerns are partially addressed and we have responded to their remaining concerns.

Strengths highlighted by reviewers

  • EgoSCAPE is a useful benchmark. It provides dense, human-verified annotations (dense captions, sound sources, timelines, grounding queries) on real long-form audio (FM2A, WXS9, Twpi).
  • The counterfactual grounding reward is novel and interesting, and ATTEST needs no architectural changes (FM2A, Twpi).
  • The training design is well motivated. Token-wise advantages give gains beyond reward design alone (WXS9).
  • The evaluation is broad, with consistent improvements across three backbones, multiple grounding datasets, and general audio benchmarks (WXS9, Twpi).

Key concerns and our responses

  • Long-audio-specific benefits (Twpi). Controlled experiments (fixed event density and fixed output length) show ATTEST narrows the short-vs-long performance gap on all backbones. Token-wise advantages help long clips most. Benchmark length was added as a limitation (App. A). Provided results for comparison with LAT-Audio on LAT-Bench to show generalizability beyond 5 minutes of audio.
  • New hallucinations after silencing (Twpi). ATTEST sharply reduces new hallucinations (e.g., 35.0% → 6.8% on long clips). A variant that explicitly penalizes them performs on par with ATTEST (App. D.8), justifying our design choice.
  • Correct predictions penalized by the counterfactual reward (WXS9). This is rare (6.7% → 1.9% during training), and a ground-truth-gated variant performs on par with ATTEST (App. D.8), confirming our initial design without gating.
  • Annotation quality (WXS9, Twpi). Verification was audio-only with high agreement (κ\kappa 0.76–0.80). Most corrections were removals of false positives.
  • Validity of the hallucination metric (WXS9). A blinded human study confirms the FLAM-based trends (App. D.9).
  • Diversity, corruption operators, significance (FM2A). We added diversity statistics (17+ languages, 7 countries, 40 scenarios; App. B.3). Five corruption operators were already compared (App. D.4). Gains over SFT and TimePro-RL are statistically significant (paired bootstrap, Bonferroni-corrected provided in response).

Manuscript updates: App. A (limitation), App. B.3 (diversity, Tabs. 6–8), App. D.8 (counterfactual-reward variants, Tab. 19), App. D.9 (human hallucination evaluation, Tab. 20).

We are grateful for the reviewers' participation throughout this process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.