- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Authors incorporate active evidence acquisition into the audio captioning process to reduce the passiveness of current methods which describe audio in a single pass. The proposed pipeline includes three models (1) a Captioner that writes an initial caption, (2) a text-only Composer that reads only the caption and asks follow-up questions, and (3) an audio-conditioned Instruct model that answers them. The Composer decides when to stop and writes the final caption from the total evidence. Training uses a reward (gold-to-prediction) in which a text model judge answers gold-derived MCQs using only the generated caption, plus a new credit assignment scheme (LOOP-GRPO) that scores each question by how much the reward drops when that QA pair is removed. Authors report state-of-the-art captioning accuracy among open-weight-backbone models.
Critical strengths
(1) Clear problem motivation: The passive caption limitation is a clear problem with today’s models. The provided three role split is clean. Keeping Composer text-only is an interesting design choice since it forced the model to reason about what is missing from the description instead of re-listening.
(2) Credit assignment is well thought-out: Rather than one reward for the whole interaction, authors score each question by removing it from the completed evidence and measuring the drop, and score stopping and synthesis separately. Credit goes only to the tokens of the relevant action. This is a reasonable fit to a multi-turn setting where a question can be useful or redundant.
Critical weaknesses
(1) All metrics are LLM-as-a-judge and the same judge is used across the GRPO reward and the evaluation metrics: Training optimises a frozen Qwen3.6-27B text-only judge answering MCQs from the caption alone and evaluation also uses Qwen3.6-27B answering benchmark MCQs from the caption alone. The only difference between the two is the question bank. There is training → evaluation leakage making it unclear if the model overfit to the LLM-as-a-judge process. Authors should provide metrics using different judge model and provide additional evaluations that do not depend on LLM-as-a-judge (e.g., human evaluation)
(2) Active acquisition seems to contribute very little: Authors' Composer achieves 64.17% at K = 1 and 64.43% at K = 4. Further, it declines to 64.30% at K = 8. Further, in Table 2a, the full system beats a GRPO-trained single-pass captioner by only ~2 points. The authors’ central claim is that active audio querying assists captioning, but the evidence is unclear as the gains seem small and appear to come from GRPO rather than active question answering.
(3) Hallucination is stated as motivation but never measured: The abstract states the task requires avoiding "unsupported details”. However, the reward does not penalise fabrication and no metrics measure it. A model could use a false positives strategy by adding hallucinated detail at no cost and still gain reward for covering gold facts.
(4) No seeds or confidence intervals.
Questions for authors
(1) Kimi-Audio-7B scores 2.9% and Audio-Flamingo 3-7B 6.3% on Omni-Cloze. Why are the scores so low? What is expected score with random guessing?
(2) Was an audio-conditioned Composer tried?
6 comments
W2&Q2: Active querying and Composer design
(1) Contribution of active querying. A one-query budget already includes audio evidence acquisition; increasing it measures the marginal benefit of further rounds. We directly compare interaction against the GRPO-trained C0 Captioner and no-QA rewriting with the same Final model.
Qwen3.6-27B accuracy (%)
| Setting | MMAR | MMAU | Omni-Cloze |
|---|---|---|---|
| C0 Captioner | 58.6 | 70.8 | 62.3 |
| C0 + no-QA rewrite | 58.7 | 70.8 | 62.5 |
| ACE-Cap Final | 64.1 | 73.0 | 64.4 |
No-QA rewriting changes accuracy by +0.1/0.0/+0.2 points; full interaction exceeds it by 5.4/2.2/1.9 points. To examine adaptivity, preplanned queries use only C0 and match each trajectory's actual query count:
| Query strategy | MMAR Acc. (%) | MMAU Acc. (%) |
|---|---|---|
| Queries preplanned from C0 | 61.2 | 71.2 |
| Adaptive ACE-Cap Final | 64.1 | 73.0 |
Adaptive querying gains 2.9/1.8 points, supporting the value of using intermediate answers to guide subsequent questions.
(2) Audio-conditioned Composer. We evaluated an audio-conditioned self-questioning and answering variant by directly reusing our trained C0 Captioner, Qwen2.5-Omni-7B, without additional training. C0 is trained independently of Composer and Instruct. Keeping its weights fixed preserves its learned captioning distribution; applying their role-specific training objectives would alter that distribution. This direct-reuse design also accommodates the rebuttal time constraints. ACE-Cap uses a text-only Composer to plan from C0 and accumulated QA evidence, with Instruct answering from audio.
| Setting | MMAR Acc. (%) | MMAU Acc. (%) |
|---|---|---|
| C0 Captioner | 58.6 | 70.8 |
| Audio + text (C0 Captioner) | 59.3 | 71.1 |
| Text only (Ours) | 64.1 | 73.0 |
Self-questioning improves over C0 by 0.7/0.3 points. ACE-Cap exceeds this variant by 4.8/1.9 points, supporting text-based planning coupled with a dedicated audio-answering role.
W4: Confidence intervals and training seeds
We report DeepSeek-v4.1-Flash accuracy and bootstrap 95% confidence intervals on 1,000 questions per benchmark, using 2,000 bootstrap replicates and resampling seed 20261112.
| Dataset | Caption | Accuracy (%) | 95% CI (%) |
|---|---|---|---|
| MMAR | Qwen2.5-Omni-7B | 51.3 | [48.0, 54.5] |
| MMAR | Qwen3-Omni-30B-A3B-Captioner | 59.5 | [56.5, 62.6] |
| MMAR | C0 Captioner | 59.8 | [56.7, 62.8] |
| MMAR | ACE-Cap Final | 63.9 | [61.0, 66.9] |
| MMAU | Qwen2.5-Omni-7B | 66.2 | [63.3, 69.1] |
| MMAU | Qwen3-Omni-30B-A3B-Captioner | 70.5 | [67.7, 73.2] |
| MMAU | C0 Captioner | 71.3 | [68.6, 74.2] |
| MMAU | ACE-Cap Final | 74.6 | [71.9, 77.2] |
These intervals quantify evaluation-sample uncertainty. We appreciate the suggestion and will conduct multi-seed training and evaluation in future work.
W1&W3&Q1: Evaluation reliability, factual grounding, and baseline behavior
Thank you for raising evaluator dependence and factual reliability. We provide cross-judge validation, direct acoustic assessment, and an analysis of baseline abstention behavior.
(1) Cross-judge performance. The same captions are evaluated by DeepSeek-v4.1-Flash alongside the training judge, Qwen3.6-27B.
ACE-Cap leads both external baselines under both judges. Its margins over Qwen3-Omni-30B-A3B-Captioner are 8.8/4.4 points on MMAR and 6.6/4.1 points on MMAU under Qwen3.6-27B/DeepSeek-v4.1-Flash.
(2) Added-claim grounding. Qwen3.6-27B extracts newly added Final claims, and Qwen3-Omni-30B-A3B-Instruct assesses anonymized claims against the original audio, without benchmark questions or synthetic references.
This automated audit directly quantifies acoustic support, contradiction, and uncertainty in the added content. Within the MCQ bank, incorrect answers receive −1, versus −0.5 for Not Given, favoring abstention over incorrect answers. The acoustic audit extends assessment to statements beyond that bank. We will include human evaluation in a future version.
(3) Low baseline scores and random guessing. The low scores are associated with frequent Not Given decisions. On Omni-Cloze, Qwen3.6-27B gives:
The judge abstains on 90.9%/87.2% of Kimi/AF3 inputs. Not Given earns zero correctness credit, explaining much of their low accuracy. Uniform guessing over four content choices yields 25.0%, or 20.0% over all five choices including Not Given. Always selecting Not Given yields 0.0%. Empty captions retain the question, options, and cloze context, measuring text-only answerability.