- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper first introduces EgoSCAPE, a benchmark containing 700 egocentric audio clips with human-verified dense captions, sound sources, timelines, and temporal grounding queries. It also proposes ATTEST, a GRPO-based training framework combining counterfactual audio corruption, attention-density rewards, and segment-level credit assignment. Results across three backbones show improvements in captioning and temporal grounding while preserving performance on the evaluated general audio benchmarks.
Critical strengths
-
The benchmark provides detailed annotations for real audio. EgoSCAPE combines dense captions, sound sources, timelines, and grounding queries. Audio-only verification by three annotators and the reported agreement support annotation quality.
-
The training design is well motivated and supported by ablations. In particular, segment-level advantages address credit assignment in long outputs. They outperform sequence-level advantages with the same rewards, showing a benefit beyond reward design alone.
-
The evaluation is broad and shows consistent improvements. Results cover three backbones, multiple grounding datasets, general audio benchmarks, and audio corruption tests.
-
The authors promise to release the benchmark annotations and training code upon acceptance.
Critical weaknesses
-
The counterfactual reward can provide misleading feedback. If the model hallucinates an event after silencing, it can penalize a correct prediction on the original audio. The ablations demonstrate usefulness, but the frequency of such incorrect penalties is not examined.
-
The refinement prompt prohibits adding events missed by the detector, and human verification focuses on existing annotations. It is unclear whether missing audible events were systematically recovered.
Questions for authors
- Can you estimate how often the counterfactual reward penalizes correct original captions because of hallucinations on the silenced audio?
- Did annotators actively search for missing events? How many annotations were added, removed, or corrected?
- Does a human evaluation of model outputs confirm the hallucination reductions measured by FLAM?
4 comments
Response to Reviewer WXS9 (2/2)
R3. Human evaluation of model outputs for hallucination reduction (Question 3)
Included in Appendix D.9 in the updated manuscript
We perform a small human evaluation over 50 dense captioning samples randomly sampled from the EgoSCAPE benchmark and ask 3 annotators to go through each audio sample and its predictions to mark whether a predicted sound event is hallucinated or not. We perform this study on Qwen2.5 Omni model with SFT, TimeProRL and ATTEST. The annotators do not know which prediction is obtained from which model. For each segment, human hallucination % reports the majority decision between the three annotators. Additionally, we report the Fleiss to show the inter-annotator agreement for each model. This analysis has been added to Appendix D.9 Table 20 of the updated manuscript.
| Model | Hal % (50 clip subset) | Human Hall. % (50 clip subset) | Fleiss |
|---|---|---|---|
| Qwen 2.5 Omni + SFT | 47.5 | 43.8 | 0.742 |
| + TimeProRL | 32.1 | 33.0 | 0.712 |
| + ATTEST | 30.2 | 29.4 | 0.723 |
We thank the reviewer for their feedback of our paper and for their effort reviewing our manuscript. We would appreciate if the reviewer can read our responses to their reviews and acknowledge if their concerns are resolved. We understand that reviewing requires significant effort and we are very grateful to the reviewer's participation in this process.
Thank you to the authors for the detailed responses and clarifications. I will maintain my original score.
Response to Reviewer WXS9 (1/2)
We thank the reviewer for their positive and constructive feedback. Our responses to each concern are below, and we have updated the manuscript accordingly.
R1. Correct predictions being penalized by RCAG (Weakness 1 and Question 1)
R1.1 How the penalty works
The counterfactual reference D^w′ is decoded once and shared by all rollouts in a group, and Eq. 8 normalizes the window score within each window. As a result, a penalty that every rollout receives equally has almost no effect on training. If all rollouts correctly predict an event in a silenced window and the reference also predicts it, each rollout gets about the same penalty, so its CAG advantage is close to zero. The penalty changes the update mainly when rollouts disagree, and in that case the GT-anchored DAC advantage (Eq. 7, η3=1) still rewards the rollouts that predicted the correct event. A correct prediction is discouraged only when the model would produce it from its prior alone, which is the behavior CAG is designed to reduce.
R1.2 Frequency of correct predictions included in RCAG
To address this question, we measure how often RCAG penalizes a correct prediction. Among all predicted segments that fall inside a silenced window, we compute the percentage that match a ground-truth segment on the original audio (tIoU ≥ 0.5 and FENSE ≥ 0.4) and also receive a nonzero CAG penalty. The table below reports this percentage over the course of RL training. Such cases are rare even early in training (6.7%) and decrease steadily to 1.9% as the model learns to stop predicting events on silenced audio.
R1.3 Gating the counterfactual penalty
We also train another version of RCAG. For this version, we only penalize a segment d^j(g) in Eq. (1) if it does not match to a ground-truth segment with temporal IoU ≥ 0.5 and FENSE ≥ 0.4. We report this gated version of RCAG in the table below, which achieves similar performance to ATTEST showing that gating is not necessary. We have included this result in Appendix D.8 Table 19 of the updated manuscript.
ESDAC: EgoSCAPE Dense Audio Captioning, CMTAG: Clotho-Moment Temporal Audio Grounding, CTAG: CASTELLA Temporal Audio Grounding
R2. More details about human annotation and refinement prompt (Weakness 2 and Question 2)
R2.1 Why the refinement prompt disallows new events.
The omni-LLM sees the video, so letting it add events would risk annotations driven by what is visible rather than what is audible. This would undermine an audio-only benchmark. Restricting it to keep, drop, or relabel prevents such visual leakage.
R2.2 Verification and correction.
Annotators listened to the audio only (no video) and accepted or rejected each entity, optionally suggesting a corrected boundary or caption. An expert adjudication pass then resolved every flagged entity using these suggestions. Additions came from missed events the annotators noted while correcting a neighboring segment. Most corrections were drops, which shows that verification mainly removed false positives, the dominant failure mode of automatic annotation.
Additional Note: Hallucination rate counts a prediction as hallucinated only if it is unmatched and also has low FLAM audio-text confidence, so predictions of genuinely audible sounds are unlikely to be counted as hallucinations by our metrics.