acceptodds
Under review as a conference paper at ICLR 2027

ACE-Cap: Active Evidence Acquisition via Agentic Co-Evolution for Long-Paragraph Fine-Grained Audio Captioning

Abstract

Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners remain passive one-shot generators: once a relevant detail is overlooked, they cannot identify the evidence gap, query the audio for targeted information, or adaptively determine when sufficient evidence has been collected. We formulate this task as active evidence acquisition and introduce Agentic Co-Evolution for Captioning (ACE-Cap), a framework that couples a Composer and an Instruct model through multi-turn interaction. A Captioner first produces an initial description. Conditioned only on this description and the interaction history, a text-only Composer asks targeted questions about unresolved acoustic attributes, while an audio-conditioned Instruct model provides grounded answers. The Composer then decides when to terminate the interaction and synthesizes the accumulated evidence into a final caption. ACE-Cap trains these roles through a unified gold-to-prediction reward derived from fixed, gold-grounded multiple-choice questions and a frozen caption-only judge. To assign credit within variable-length interactions, we introduce Leave-One-Out Per-turn Group Relative Policy Optimization (LOOP-GRPO). It replaces the trajectory-wide scalar advantage with span-aligned signals for individual questions, stopping decisions, and final synthesis. These signals measure leave-one-out evidence contributions, evidence sufficiency, and evidence preservation, respectively. Role-wise warm-up followed by alternating Composer and Instruct optimization keeps each update a well-defined single-policy problem while allowing the two roles to co-evolve. Extensive experiments across diverse audio benchmarks demonstrate state-of-the-art performance among evaluated models with open-weight backbones under a common caption-as-evidence protocol.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio caption

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_N5s6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Authors incorporate active evidence acquisition into the audio captioning process to reduce the passiveness of current methods which describe audio in a single pass. The proposed pipeline includes three models (1) a Captioner that writes an initial caption, (2) a text-only Composer that reads only the caption and asks follow-up questions, and (3) an audio-conditioned Instruct model that answers them. The Composer decides when to stop and writes the final caption from the total evidence. Training uses a reward (gold-to-prediction) in which a text model judge answers gold-derived MCQs using only the generated caption, plus a new credit assignment scheme (LOOP-GRPO) that scores each question by how much the reward drops when that QA pair is removed. Authors report state-of-the-art captioning accuracy among open-weight-backbone models.

Critical strengths

(1) Clear problem motivation: The passive caption limitation is a clear problem with today’s models. The provided three role split is clean. Keeping Composer text-only is an interesting design choice since it forced the model to reason about what is missing from the description instead of re-listening.

(2) Credit assignment is well thought-out: Rather than one reward for the whole interaction, authors score each question by removing it from the completed evidence and measuring the drop, and score stopping and synthesis separately. Credit goes only to the tokens of the relevant action. This is a reasonable fit to a multi-turn setting where a question can be useful or redundant.

Critical weaknesses

(1) All metrics are LLM-as-a-judge and the same judge is used across the GRPO reward and the evaluation metrics: Training optimises a frozen Qwen3.6-27B text-only judge answering MCQs from the caption alone and evaluation also uses Qwen3.6-27B answering benchmark MCQs from the caption alone. The only difference between the two is the question bank. There is training → evaluation leakage making it unclear if the model overfit to the LLM-as-a-judge process. Authors should provide metrics using different judge model and provide additional evaluations that do not depend on LLM-as-a-judge (e.g., human evaluation)

(2) Active acquisition seems to contribute very little: Authors' Composer achieves 64.17% at K = 1 and 64.43% at K = 4. Further, it declines to 64.30% at K = 8. Further, in Table 2a, the full system beats a GRPO-trained single-pass captioner by only ~2 points. The authors’ central claim is that active audio querying assists captioning, but the evidence is unclear as the gains seem small and appear to come from GRPO rather than active question answering.

(3) Hallucination is stated as motivation but never measured: The abstract states the task requires avoiding "unsupported details”. However, the reward does not penalise fabrication and no metrics measure it. A model could use a false positives strategy by adding hallucinated detail at no cost and still gain reward for covering gold facts.

(4) No seeds or confidence intervals.

Questions for authors

(1) Kimi-Audio-7B scores 2.9% and Audio-Flamingo 3-7B 6.3% on Omni-Cloze. Why are the scores so low? What is expected score with random guessing?

(2) Was an audio-conditioned Composer tried?

6 comments

W1&W3&Q1: Evaluation reliability, factual grounding, and baseline behavior

Thank you for raising evaluator dependence and factual reliability. We provide cross-judge validation, direct acoustic assessment, and an analysis of baseline abstention behavior.

(1) Cross-judge performance. The same captions are evaluated by DeepSeek-v4.1-Flash alongside the training judge, Qwen3.6-27B.

DatasetCaptionQwen3.6-27B Acc. (%)DeepSeek-v4.1-Flash Acc. (%)
MMARQwen2.5-Omni-7B51.851.3
MMARQwen3-Omni-30B-A3B-Captioner55.359.5
MMARC0 Captioner58.659.8
MMARC0 + no-QA rewrite58.759.7
MMARACE-Cap Final64.163.9
MMAUQwen2.5-Omni-7B65.266.2
MMAUQwen3-Omni-30B-A3B-Captioner66.470.5
MMAUC0 Captioner70.871.3
MMAUC0 + no-QA rewrite70.870.8
MMAUACE-Cap Final73.074.6

ACE-Cap leads both external baselines under both judges. Its margins over Qwen3-Omni-30B-A3B-Captioner are 8.8/4.4 points on MMAR and 6.6/4.1 points on MMAU under Qwen3.6-27B/DeepSeek-v4.1-Flash.

(2) Added-claim grounding. Qwen3.6-27B extracts newly added Final claims, and Qwen3-Omni-30B-A3B-Instruct assesses anonymized claims against the original audio, without benchmark questions or synthetic references.

DatasetAdded claimsSupported, n (%)Contradicted, n (%)Uncertain, n (%)
Omni-Cloze3,2542,634 (80.9)382 (11.7)238 (7.3)
MMAR1,6631,193 (71.7)375 (22.5)95 (5.7)
MMAU2,0401,498 (73.4)392 (19.2)150 (7.4)

This automated audit directly quantifies acoustic support, contradiction, and uncertainty in the added content. Within the MCQ bank, incorrect answers receive −1, versus −0.5 for Not Given, favoring abstention over incorrect answers. The acoustic audit extends assessment to statements beyond that bank. We will include human evaluation in a future version.

(3) Low baseline scores and random guessing. The low scores are associated with frequent Not Given decisions. On Omni-Cloze, Qwen3.6-27B gives:

Caption evidenceAccuracy (%)Not Given (%)
Empty24.670.8
Kimi-Audio-7B-Instruct2.990.9
Audio-Flamingo 3-7B6.387.2

The judge abstains on 90.9%/87.2% of Kimi/AF3 inputs. Not Given earns zero correctness credit, explaining much of their low accuracy. Uniform guessing over four content choices yields 25.0%, or 20.0% over all five choices including Not Given. Always selecting Not Given yields 0.0%. Empty captions retain the question, options, and cloze context, measuring text-only answerability.

W2&Q2: Active querying and Composer design

(1) Contribution of active querying. A one-query budget already includes audio evidence acquisition; increasing it measures the marginal benefit of further rounds. We directly compare interaction against the GRPO-trained C0 Captioner and no-QA rewriting with the same Final model.

Qwen3.6-27B accuracy (%)

SettingMMARMMAUOmni-Cloze
C0 Captioner58.670.862.3
C0 + no-QA rewrite58.770.862.5
ACE-Cap Final64.173.064.4

No-QA rewriting changes accuracy by +0.1/0.0/+0.2 points; full interaction exceeds it by 5.4/2.2/1.9 points. To examine adaptivity, preplanned queries use only C0 and match each trajectory's actual query count:

Query strategyMMAR Acc. (%)MMAU Acc. (%)
Queries preplanned from C061.271.2
Adaptive ACE-Cap Final64.173.0

Adaptive querying gains 2.9/1.8 points, supporting the value of using intermediate answers to guide subsequent questions.

(2) Audio-conditioned Composer. We evaluated an audio-conditioned self-questioning and answering variant by directly reusing our trained C0 Captioner, Qwen2.5-Omni-7B, without additional training. C0 is trained independently of Composer and Instruct. Keeping its weights fixed preserves its learned captioning distribution; applying their role-specific training objectives would alter that distribution. This direct-reuse design also accommodates the rebuttal time constraints. ACE-Cap uses a text-only Composer to plan from C0 and accumulated QA evidence, with Instruct answering from audio.

SettingMMAR Acc. (%)MMAU Acc. (%)
C0 Captioner58.670.8
Audio + text (C0 Captioner)59.371.1
Text only (Ours)64.173.0

Self-questioning improves over C0 by 0.7/0.3 points. ACE-Cap exceeds this variant by 4.8/1.9 points, supporting text-based planning coupled with a dedicated audio-answering role.

W4: Confidence intervals and training seeds

We report DeepSeek-v4.1-Flash accuracy and bootstrap 95% confidence intervals on 1,000 questions per benchmark, using 2,000 bootstrap replicates and resampling seed 20261112.

DatasetCaptionAccuracy (%)95% CI (%)
MMARQwen2.5-Omni-7B51.3[48.0, 54.5]
MMARQwen3-Omni-30B-A3B-Captioner59.5[56.5, 62.6]
MMARC0 Captioner59.8[56.7, 62.8]
MMARACE-Cap Final63.9[61.0, 66.9]
MMAUQwen2.5-Omni-7B66.2[63.3, 69.1]
MMAUQwen3-Omni-30B-A3B-Captioner70.5[67.7, 73.2]
MMAUC0 Captioner71.3[68.6, 74.2]
MMAUACE-Cap Final74.6[71.9, 77.2]

These intervals quantify evaluation-sample uncertainty. We appreciate the suggestion and will conduct multi-seed training and evaluation in future work.

Reviewer_1cULOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ACE-Cap, an audio captioner model with its Loop-GRPO training method that iteratively collects cues and composes the results.

Critical strengths

(1) For the ambiguous rewards, the work smartly designs the credits to make the whole GRPO to be more learnable. (2) This is a sensible multi-agent training pipeline that uses different stages to achieve stable training (3) The proposed work is demonstrated to be effective experimentally

Critical weaknesses

(1) As a very related baseline, the Omni-Captioner also uses iterative info acquisition. but it's missing in table 1 (2) The author claims that one-pass captioning is not ideal but still uses Gemini to generate the reference in one pass as the reference. (3) It may not clear if the progressive/iterative info acquisition is really beneficial - a potential ablation study is to still use GRPO and the questions from Gemini, but without the composor or iterative querying the Qwen-Omni model.

Questions for authors

According to Figure 4, a question is: for different categories of queries, is there a "precision" metric or measure? e.g., how accurate are the "TM" queries answered?

3 comments

Q1: Query-category reliability and temporal reasoning

We audit actual Instruct answers using Qwen3-Omni-Instruct with the original audio, query, and answer. It assigns supported (S), contradicted (C), or uncertain (U); S requires support for all factual propositions. The automated audio-grounded support rate is S/(S+C+U).

Query categoryMMAU support (%)Omni-Cloze support (%)
TM73.390.1
CT85.887.1
EV73.989.9
MU77.287.4
SG77.883.9
SP85.387.1
VO84.294.2

TM support reaches 73.3%/90.1% on MMAU/Omni-Cloze; contradiction rates are 25.9%/9.4%. This directly characterizes the reliability of temporal answers in actual trajectories.

As complementary evidence, we evaluate TACOS AudioSet Strong Set temporal QA. Both models freely answer from audio; Qwen3.6-27B maps each answer to an option for comparison with gold.

ModelCorrect / TotalAccuracy (%)
Base Qwen2.5-Omni526 / 1,12846.6
ACE-Cap Instruct599 / 1,12853.1

ACE-Cap Instruct improves temporal QA accuracy by 6.5 points.

W1&W2&W3: Baselines, supervision, and adaptive querying

(1) Omni-Captioner baseline. The audio-only model from the Omni-Captioner paper is already included in Table 1 as Audio-Captioner-7B. We will clarify this naming correspondence.

(2) Gemini gold-caption and MCQ construction. Appendix B describes a multi-stage pipeline. Gemini performs complementary analyses of acoustic properties, events, speech, music, and other applicable aspects; a separate synthesis prompt combines them into a gold caption. Gold-MCQs are then linked to supporting caption spans and validated. Figure 5 includes audio-grounding and MCQ-only checks that filter questions lacking acoustic evidence or answerable from question/option cues alone.

This provides evidence-checked supervision. ACE-Cap learns to acquire useful information omitted from its initial caption during inference.

(3) Benefits of iterative and adaptive querying. We compare the C0 Captioner, a no-QA rewrite using the same Final model, preplanned queries, and adaptive ACE-Cap. Preplanned queries are determined from C0 before observing new answers and match each trajectory's actual query count.

Accuracy (%) — DeepSeek-v4.1-Flash / Qwen3.6-27B

SettingMMARMMAU
C0 Captioner59.8 / 58.671.3 / 70.8
C0 + no-QA rewrite59.7 / 58.770.8 / 70.8
Queries preplanned from C061.4 / 61.271.0 / 71.2
Adaptive ACE-Cap Final63.9 / 64.174.6 / 73.0

Under DeepSeek-v4.1-Flash, ACE-Cap exceeds no-QA rewriting by 4.2/3.8 points and preplanned querying by 2.5/3.6 points on MMAR/MMAU. Qwen3.6-27B confirms the advantage over preplanned querying at 2.9/1.8 points. These controls support the benefits of additional QA evidence and answer-conditioned question selection.

Thank you for your positive feedback and for recognizing our work. We will add these controls and the per-category support rates to the main paper, discuss the contradiction rates, and include human verification in future work.

Reviewer_zZWqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper presents a multi-agent framework for fine-grained audio captioning, combining iterative evidence acquisition with a new credit-assignment scheme and alternating optimization. The overall benchmark results are promising.

Critical strengths
  1. Noticeable improvements across multiple benchmarks

  2. Engineering-oriented and experiment-dense demonstration

Critical weaknesses

Main Argument:

i.Aggregate quality utility may actually degenerate into a regular final-caption reward. From Eq. 3, the authors propose that bg=(rgF−r0)+λ[min⁡(rgE,rgF)−r0]b_g=(r_g^F-r_0)+\lambda[\min(r_g^E, r_g^F)-r_0], and claim that the second term encourages the model to retain the evidence it has already obtained. However, when rgE≥rgFr_g^E \ge r_g^F, that is, when the cumulative evidence score is higher than the final caption score, we will have bg=(1+λ)(rgF−r0)b_g=(1+\lambda)(r_g^F-r_0). At this point, it is clear that the reward function is no longer dependent on the specific score of the cumulative evidence and degenerates into a scaled form of the final caption reward. This is exactly what might happen when the final caption loses some of the accumulated evidence, and we do not have lossless evidence in practice.

More importantly, even if rgE=rgFr_g^E = r_g^F could happen, it does not mean the final caption retains facts. For example, the accumulated evidence correctly includes fact A, and the final caption correctly includes fact B; it seems both have the same accuracy, but fact A is still missing. It is therefore unclear what additional evidence-preservation behavior this term encourages beyond optimizing final-caption quality. The paper does not verify whether this limitation affects the core conclusions.

ii. Stopping reward 和 Final reward are actually the same. The author defines: ugS=bg−cSTg,ugF=bgu_g^S=b_g-c_ST_g, u_g^F=b_g, however, the paper sets cS=0c_S=0, therefore ugS=ugF=bg\boxed{u_g^S=u_g^F=b_g}, this means that the original rewards obtained from stopping and final synthesis are exactly the same. It is unclear how the stopping signal specifically captures evidence sufficiency or the value of terminating an interaction, rather than merely rewarding the overall quality of a completed trajectory.

iii. Figure 4(b) is too small hard to read and difficult to interpret as evidence for the proposed optimization method. The two Composers differ in both model size and training, and the untrained larger Composer actually achieves higher final-caption accuracy at K=4K=4. A smaller evidence-to-caption gap alone does not establish superior evidence acquisition or preservation.

Overall, this is a work with a high score on the benchmarks but not trying to solve the academic research problem; more like first finding out this solution can work, then trying to make up some theory to make the solution make sense. I personally do not recommend this work for this venue; however, it is good for a journal or a domain-specific conference like ICASSP or Interspeech. The community will still appreciate the engineering experience.

Questions for authors
  1. The leave-one-out utility is computed by deleting a QA pair while keeping all subsequent interactions unchanged. However, later questions and answers depend on earlier evidence, making the resulting sequence potentially inconsistent with the actual interaction policy. Why should this static deletion score be considered an appropriate estimate of a question's contribution to sequential evidence acquisition?

  2. What evidence demonstrates that alternating co-evolution is preferable to continued single-role training under a matched optimization budget?

  3. The proposed framework involves three models and multiple rounds of audio processing, which may introduce substantially higher inference costs. Why are inference latency, average output token count, and total number of model calls not reported?

  4. Why does the paper only report the maximum query budget, without providing the distribution of the actual number of queries, it is difficult to determine whether the model actually learns to stop adaptively rather than simply exhausting the query budget.

8 comments

W1&W3: Evidence-preservation utility and fact-level retention

Thank you for distinguishing aggregate reward from preservation of specific facts. When rgE≥rgFr_g^E \geq r_g^F, Eq. 3 reduces to (1+λ)(rgF−r0)(1+\lambda)(r_g^F-r_0). The utility couples evidence and caption quality through their minimum; we additionally evaluate synthesis using fixed-evidence ablation and direct proposition tracking.

(1) Fixed-evidence synthesis ablation. Table 3(b) holds Decision trajectories and evidence fixed:

Final settingEvidence accuracy (%)Caption accuracy (%)Gap (pp)
Without Final credit64.863.41.4
Full Final64.864.40.4

Final credit increases caption accuracy by 1.0 point and reduces the evidence-to-caption gap from 1.4 to 0.4 points.

(2) Tracking the same QA facts into Final. On Omni-Cloze, Qwen3.6-27B extracts atomic QA propositions adding information beyond C0 and evaluates their preservation in Final, accounting for temporal scope and epistemic qualifiers.

DatasetEvaluated factsRetained (%)Omitted (%)Contradicted (%)Uncertain (%)
Omni-Cloze11,06371.911.414.91.9

Final retains 71.9% of these newly acquired propositions. This textual audit directly addresses whether the same information survives synthesis. We will enlarge Figure 4(b) and clarify its role as a cross-backbone query-budget comparison.

W2&Q4: Stopping credit and actual stopping behavior

With cS=0c_S=0, ugS=ugF=bgu_g^S=u_g^F=b_g. Their distinction is action-specific assignment: stopping credit supervises the STOP decision, while final credit supervises final-caption tokens.

(1) Stopping-credit ablation.

SettingMMAR Acc. (%)Omni-Cloze Acc. (%)
Full ACE-Cap64.164.4
Without stopping credit63.664.0

Removing stopping credit reduces accuracy by 0.5/0.4 points.

(2) Actual QA counts and stopping outcomes. We analyzed all 1,000 trajectories per benchmark:

Dataset1 query (%)2 queries (%)3 queries (%)4 queries (%)Mean queriesNatural FINISH (%)Budget-forced (%)
MMAR36.433.023.17.52.092.57.5
MMAU26.929.132.012.02.388.012.0

The Composer voluntarily terminates before the budget on 92.5%/88.0% of MMAR/MMAU trajectories, averaging 2.0/2.3 queries. Other/recovery outcomes are 0.0% on both benchmarks. These results show that the policy adjusts interaction length to the evolving evidence.

Q1: Static LOO and sequential counterfactuals

Static LOO provides a tractable estimate of a QA pair's contribution to the realized evidence. We test this approximation with delete-and-reroll: fix the preceding history, remove the target QA pair, and regenerate subsequent Composer questions and Instruct answers under the same remaining budget. Both utilities use a shared retained-trajectory reference and benchmark binary correctness.

DatasetDeleted turnAgreement (%)MAE
MMARt189.60.104
MMARt291.80.082
MMARt395.30.047
MMARt1–t391.20.088
MMAUt187.80.122
MMAUt290.50.095
MMAUt394.30.057
MMAUt1–t390.10.099

We evaluate turns with subsequent interactions to regenerate and aggregate by question–turn micro-average. Static LOO achieves 91.2%/90.1% agreement and 0.088/0.099 MAE.

As a supplementary control, we retain the target QA, regenerate later interactions, and then apply static deletion, comparing against delete-and-reroll with the regenerated retained trajectory as reference:

DatasetTurnsAgreement (%)MAE
MMARt1–t391.60.084
MMAUt1–t391.50.085

The similar results support static LOO as a practical approximation for question-level credit assignment.

Reviewer_kCbwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ACE-Cap, a framework that formulates fine-grained audio captioning as an active evidence acquisition problem rather than passive one-shot generation. It consists of three roles: a Captioner that generates an initial caption, a text-only Composer that identifies missing information and generates targeted questions, and an audio-conditioned Instruct model that answers these questions based on the audio. The Composer iteratively collects evidence, decides when to stop, and synthesizes the final caption.

The main technical contribution is LOOP-GRPO, a role-specific credit assignment method that separately optimizes question generation, stopping decisions, and final caption synthesis. In particular, it uses leave-one-out evidence contribution to assess the utility of individual question-answer turns, while separate utilities encourage evidence sufficiency and preservation during final synthesis. The authors further introduce role-wise warm-up and alternating optimization to mitigate non-stationarity between the Composer and Instruct models.

The framework is trained on approximately 50K audio clips, with synthetic gold captions and gold-grounded multiple-choice questions used to construct a unified Gold-to-Prediction reward. Experiments on MMAU, MMAR, MMSU, and Omni-Cloze show competitive performance, with reported accuracies of 73.0%, 64.1%, 70.4%, and 64.4%, respectively. The paper also provides ablations on credit assignment, training stages, query budgets, and evaluator sensitivity.

Critical strengths
  1. A novel formulation of audio captioning as active evidence acquisition. Rather than treating audio captioning as a one-shot generation task, the paper introduces an interactive framework that explicitly identifies missing information, acquires additional audio-grounded evidence, and adaptively determines when to stop. The separation between the text-only Composer and the audio-conditioned Instruct model is particularly interesting, as it provides a structured way to improve the completeness of fine-grained audio descriptions.

  2. Action-specific credit assignment for multi-turn interaction. LOOP-GRPO distinguishes the contributions of question generation, stopping decisions, and final evidence-preserving synthesis. The leave-one-out formulation provides a meaningful alternative to trajectory-level rewards or adjacent reward differences, and the ablation results offer empirical evidence that these different credit signals contribute to downstream performance.

Critical weaknesses
  1. Limited validation of genuine audio-grounded caption quality. The proposed G2P reward is constructed from synthetic gold captions and fixed gold-grounded MCQs, and the main evaluation also relies on text-only LLM judges to assess the information contained in generated captions. Although this protocol measures whether captions preserve benchmark-relevant information, it does not directly establish that the acquired details are acoustically correct or that unsupported claims are reduced. In particular, the reward assigns scores based on a fixed bank of questions, leaving unsupported statements outside the bank largely unaccounted for. This is an important limitation for a framework whose central motivation is to acquire reliable acoustic evidence rather than simply produce more detailed descriptions. Additional human evaluation or independent audio-grounded evaluation would make the main claims substantially more convincing.

  2. Insufficient isolation of the proposed method's contributions and practical costs. Some comparisons, particularly the query-budget analysis, involve Composers with different backbone architectures and training procedures. Therefore, the observed differences cannot be attributed solely to LOOP-GRPO or alternating co-evolution. Moreover, the proposed framework introduces multiple models and multi-turn audio inference, but the paper does not provide a systematic analysis of inference latency, computational cost, or the trade-off between interaction overhead and caption quality. It is therefore difficult to determine whether the improvements justify the additional complexity compared with stronger one-shot captioning systems.

Questions for authors
  1. Evaluation validity and audio grounding: Since the G2P training reward and the primary evaluation protocol both rely on text-only LLM judges, how do the authors ensure that the observed improvements reflect better audio-grounded understanding rather than better alignment with the judge? Have the authors conducted human evaluations or independent evaluations in which generated claims are explicitly verified against the original audio, including an assessment of unsupported claims and hallucinations?

  2. Reward coverage and factual reliability: The Gold-MCQ bank is constructed from synthetic gold captions, and the Not Given option is never a correct answer during reward construction. How does the proposed reward handle details that are not covered by the fixed MCQ bank, particularly unsupported or potentially hallucinated statements? Could the authors provide evidence that optimizing this reward does not encourage overly assertive descriptions?

  3. Method attribution and efficiency: In the query-budget analysis, the base Composer and the proposed Composer use different backbones. Could the authors provide a controlled comparison using the same backbone and training data to isolate the contribution of LOOP-GRPO and alternating co-evolution? Additionally, what are the inference latency, number of audio-model calls, and computational overhead compared with one-shot captioning baselines?

7 comments

W1&Q1&Q2: Audio grounding, reward coverage, and evaluator robustness

Thank you for highlighting factual reliability. We provide direct audio-grounded validation and cross-judge evaluation.

(1) Acoustic support of added details. Qwen3.6-27B extracts claims newly added from the C0 Captioner output to Final. Qwen3-Omni-30B-A3B-Instruct then assesses anonymized claims against the original audio, without benchmark questions or synthetic references.

DatasetAdded claimsSupported, n (%)Contradicted, n (%)Uncertain, n (%)
Omni-Cloze3,2542,634 (80.9)382 (11.7)238 (7.3)
MMAR1,6631,193 (71.7)375 (22.5)95 (5.7)
MMAU2,0401,498 (73.4)392 (19.2)150 (7.4)

This automated audit supports most added details and directly measures contradictions and uncertainty beyond the fixed MCQ bank.

(2) Reward coverage. G2P assigns +1 to correct answers, −1 to incorrect content answers, and −0.5 to Not Given. This rewards evidence coverage while favoring abstention over incorrect answers. The audio-grounded audit above evaluates additional statements beyond the bank's coverage.

(3) Cross-judge validation. We evaluate the same captions with DeepSeek-v4.1-Flash alongside Qwen3.6-27B.

DatasetCaptionQwen3.6-27B Acc. (%)DeepSeek-v4.1-Flash Acc. (%)
MMARQwen2.5-Omni-7B51.851.3
MMARQwen3-Omni-30B-A3B-Captioner55.359.5
MMARC0 Captioner58.659.8
MMARC0 + no-QA rewrite58.759.7
MMARACE-Cap Final64.163.9
MMAUQwen2.5-Omni-7B65.266.2
MMAUQwen3-Omni-30B-A3B-Captioner66.470.5
MMAUC0 Captioner70.871.3
MMAUC0 + no-QA rewrite70.870.8
MMAUACE-Cap Final73.074.6

ACE-Cap leads both external baselines under both judges, supporting the consistency of its performance advantage. We will add human evaluation in a future version to complement these acoustic and cross-judge assessments.

W2&Q3: Method attribution and the quality–cost trade-off

We address attribution through component ablations and continued-training controls, and quantify inference cost on all 1,000 questions in each benchmark.

(1) Continued-training comparison. The paper's Table 2(b) compares credit-assignment choices, and Table 3(a) examines component contributions. Our additional comparison starts from ACE-Cap warm-up before Alternating and keeps the Captioner frozen:

Training scheduleMMAR Acc. (%)MMAU Acc. (%)
C0 Captioner (single-pass reference)58.670.8
ACE-Cap warm-up (before Alternating)62.772.0
Composer-only continuation62.972.1
Instruct-only continuation63.372.5
Alternating64.173.0

Alternating updates exceed Composer-only continuation by 1.2/0.9 points and Instruct-only continuation by 0.8/0.5 points on MMAR/MMAU. This supports adapting query planning and synthesis alongside the audio-answering model.

(2) Inference cost. Accuracy entries below are DeepSeek-v4.1-Flash / Qwen3.6-27B.

DatasetSettingAccuracy (%)Component time (s/question)Output tokens/questionInstruct calls/question
MMARC0 Captioner59.8 / 58.612.0502.40.0
MMARACE-Cap63.9 / 64.140.11,434.02.0
MMAUC0 Captioner71.3 / 70.811.3463.90.0
MMAUACE-Cap74.6 / 73.039.41,366.12.3

We replay saved inputs on one GPU per component at batch size 1. Time and tokens sum Captioner, Decision, Instruct, and Final generation; timing includes synchronized audio encoding, prefill, and decoding after warmup. Each QA query invokes Instruct once.

ACE-Cap gains 5.5/2.2 accuracy points under Qwen3.6-27B at 3.3×/3.5× the C0 component time and approximately 2.9× the output tokens, making the quality–cost trade-off explicit.

Thank you again for your helpful comments. Our responses now include direct audio checks, cross-judge results, continued-training comparisons, and inference costs. We would appreciate your feedback on whether these additions address your concerns and would be happy to clarify any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair and reviewers

We thank the Area Chair and all four reviewers for their feedback. Two questions shaped the discussion: does adapting each question to the preceding answer improve the caption, and how often does the recording support the added details?

Contribution and main result. ACE-Cap recovers details omitted from an initial audio caption. A text-only Composer identifies gaps, asks an audio-conditioned Instruct model targeted questions, and decides when to stop and synthesize the evidence. LOOP-GRPO assigns credit separately to question selection, stopping, and synthesis. When Qwen3.6-27B answers benchmark questions using the caption as evidence, ACE-Cap improves accuracy over the trained single-pass captioner from 58.6% to 64.1% on MMAR, 70.8% to 73.0% on MMAU, and 62.3% to 64.4% on Omni-Cloze.

Strengths noted by reviewers. kCbw and N5s6 highlighted the motivation for active acquisition, the division between text-based planning and audio answering, and the action-specific credit design. zZWq noted the gains across benchmarks and the breadth of the experiments. 1cUL highlighted the credit design and staged training.

Evidence added or clarified during the discussion

Question raisedEvidence
Does adaptive questioning help? (1cUL, N5s6)We matched the query count for each trajectory and compared adaptive questions with questions planned from the initial caption. Adaptive querying gained 2.9 points on MMAR and 1.8 on MMAU under Qwen3.6-27B, and 2.5 and 3.6 points under DeepSeek-v4.1-Flash. Rewriting without QA changed Qwen-judged accuracy by at most 0.2 points across the three benchmarks.
Are added details grounded in the audio? (kCbw, N5s6, 1cUL)An automated judge checked new claims against the original audio without benchmark questions or synthetic references. It classified 80.9% of claims on Omni-Cloze, 71.7% on MMAR, and 73.4% on MMAU as supported. It classified 11.7%, 22.5%, and 19.2%, respectively, as contradicted, with the remainder uncertain. Contradictions remain a limitation even where QA accuracy improves.
Does turn-level credit identify useful evidence, and does synthesis preserve it? (zZWq)Static leave-one-out credit agreed with deleting a QA turn and regenerating later turns in 91.2% of comparisons on MMAR and 90.1% on MMAU. With evidence fixed, final-synthesis credit narrowed the evidence-to-caption accuracy gap from 1.4 to 0.4 points. A textual audit found that final captions retained 71.9% of 11,063 newly acquired QA facts on Omni-Cloze.
Does alternating training help? (kCbw, zZWq)From the same warm-up checkpoint with matched optimization budgets, alternating updates reached 64.1% on MMAR and 73.0% on MMAU, compared with 63.3% and 72.5% for Instruct-only continuation and 62.9% and 72.1% for Composer-only continuation.
When does the model stop, and at what cost? (kCbw, zZWq)The Composer stopped before the query limit on 92.5% of MMAR and 88.0% of MMAU trajectories, averaging 2.0 and 2.3 queries. Across 1,000 questions per benchmark, summed component time per question rose from 12.0 to 40.1 seconds on MMAR and from 11.3 to 39.4 seconds on MMAU.

Status of the discussion

Reviewer kCbw raised their rating from 2 to 3 (Weak acceptance) after the rebuttal. They noted that the audio-grounded checks, cross-judge evaluation, and continued-training controls strengthened the evidence, and that the timing measurements clarified the inference overhead. They considered their main concerns sufficiently addressed while retaining reservations about the factual accuracy of added details and the gain in caption quality relative to inference cost.

Reviewer 1cUL’s raised their rating from 2 to 3 (Weak acceptance) after the rebuttal. In our public response, we clarified the Omni-Captioner baseline and the construction of the gold captions and questions, and reported the no-QA and matched-query controls.

Our replies to Reviewers zZWq and N5s6 provide the credit-assignment analyses, grounding checks, querying controls, and uncertainty estimates summarized above. As of this update, the public discussion does not indicate which concerns these two reviewers consider resolved. We ask the Area Chair to consider the additional evidence alongside the original reviews.

In the revision, we will include the no-QA and matched-query controls, report support and contradiction rates by query category, and present accuracy gains alongside query counts and inference costs. Human verification and evaluation across training seeds remain open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.