- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ASID-Verify, an attribute-centered pipeline for verifying and refining audiovisual captions. Initial captions and aligned speech transcripts are combined, audited against the video and ASR evidence, and revised using attribute-organized reports. The process repeats with fixed annotation models. The authors construct ASID-1M, containing single-attribute and all-attribute captions for approximately 121K videos, and fine-tune Qwen2.5-Omni models through single-attribute, all-attribute, and longer-context training stages. Evaluation covers captioning, caption-based question answering, temporal grounding, and attribute-conditioned instruction following.
Critical strengths
(1) The paper addresses an important data-quality problem. Detailed audiovisual captions can improve coverage while introducing unsupported statements, so identifying specific errors and omissions before instruction tuning is practically useful. The explicit separation between verification and targeted refinement provides an inspectable curation process.
(2) The evaluation spans several complementary uses of captions, including downstream question answering and temporal grounding. The additional judge evaluation in Appendix F.3 and the alternative training schedules in Appendix F.4 are useful checks. The manuscript also appropriately acknowledges that progressive training order is an implementation choice rather than a demonstrated necessity.
Critical weaknesses
(1) The experiments do not adequately isolate attribute-level verification as the source of improvement.
The main methodological claim is that semantic attributes resolve a granularity mismatch in caption verification. However, the proposed verifier changes several properties of feedback together: it uses predefined attribute-specific checklists, identifies affected snippets, explains evidence, and distinguishes errors from omissions. The refiner then follows these localized reports while minimizing unrelated edits. Figure 2 contrasts this against coarse caption-level feedback, but the manuscript does not establish whether a caption-level verifier was given equally explicit instructions for issue localization, evidence reporting, and targeted correction.
A decisive control would retain the same checklist coverage, localized snippets, evidence requirements, and refinement instructions while removing only attribute grouping. Without this control, the observed advantage may reflect more actionable feedback or more explicit verification instructions rather than semantic attributes as the critical verification unit.
Section 3.4 repeatedly describes verification budgets as matched, but the operational definition of this budget is unclear. Equal round counts or model-call counts do not establish equal generated-token budgets, checklist coverage, or inference expenditure. Appendix I provides the attribute-audit and refinement templates in Figures 17 and 18, but does not provide a comparably explicit template for the caption-level baseline used in the central experiment.
The downstream supervision ablation does not resolve this attribution problem. Section 5.3 explicitly acknowledges that Table 6a matches source videos and instance counts without matching target-token counts. Richer captions therefore change both supervision content and the amount of supervised text. Figure 3 provides useful stage-wise evidence, but does not report a controlled downstream comparison between caption-level and attribute-level refinement under matched supervision budgets.
Consequently, the current evidence supports the usefulness of the complete curation pipeline more strongly than the claimed distinctive role of attribute-level verification. Since this distinction is the principal methodological contribution, the missing controls materially affect my recommendation.
(2) The evidence for fine-grained reliability is insufficiently independently validated, and part of the downstream evaluation is internally inconsistent.
The headline improvements in Section 3.4 and Figure 1 rely on automatic No-Error, No-Missing, and Clean diagnostics. The manuscript does not sufficiently specify their evaluation sample, aggregation procedure, judging model, or independence from the models used for annotation and verification. These details are essential because repeated repair can improve agreement with an automatic auditor without establishing the same improvement in factual correctness against the video.
The human study in Appendix F.1 assesses overall usability using Pass, Borderline, and Fail labels. The authors explicitly distinguish this measure from the automatic diagnostics. This is a reasonable usability check, but it does not independently validate the reported difference between caption-level and attribute-level verification. A paired, blinded human assessment of those conditions is needed to establish that the central improvement reflects fewer factual errors and omissions rather than evaluator alignment. The additional judge in Table 10 evaluates generated model captions, so it does not directly validate this central annotation-repair comparison.
There is also a concrete consistency problem in Table 5. For both Gemini-3-Pro and Gemini-2.5-Flash, the reported mean IoU is too low to be compatible with the reported recalls under standard same-query temporal-grounding definitions. The successful predictions counted by the recall columns alone already require a larger mean IoU than the table reports. Appendix E describes standard mean-IoU and recall evaluation without explaining an alternative aggregation convention that would reconcile these results. This affects the comparison used to support temporally precise captioning.
Finally, Table 6b reports zero or near-zero instruction-following performance for several strong baselines, but the corresponding judging criteria and evaluation procedure are not documented sufficiently to explain whether the metric measures requested-attribute inclusion, exclusion of unrequested content, factual correctness, or a combination. Because these choices can substantially change the interpretation, the table requires human-calibrated validation before supporting strong comparative claims.
Together, these issues weaken the evidence for the paper’s central reliability and instruction-following conclusions.
Questions for authors
(1) What are the complete prompts for the caption-level verification baseline and its refiner? Were they instructed to identify individual erroneous snippets and omissions, provide supporting evidence, and make minimal targeted edits? Does the attribute-level advantage persist against an equally detailed issue-level verifier that omits only attribute grouping?
(2) What exactly is matched in the verification budgets reported in Section 3.4 and Figure 1: calls, input tokens, output tokens, monetary cost, or another quantity? Please report the actual budgets and clarify whether both conditions inspect the same semantic checklist and audiovisual evidence.
(3) Do the downstream gains persist when caption-level and attribute-level refinement use the same videos, training-token budget, optimization steps, and evaluation protocol? How much of the improvement in Table 6a remains after controlling for target-caption length and supervised-token count?
(4) How are No-Error, No-Missing, and Clean defined and aggregated, and which model evaluates them? Is this evaluator independent of the annotation and verification models? Can the central comparison be confirmed by paired, blinded human judgments of factual errors and omissions, rather than overall usability alone?
(5) How can the mean IoU and recall entries for Gemini-3-Pro and Gemini-2.5-Flash in Table 5 be reconciled? Were all columns computed from the same queries, predictions, and failure-handling rules? Please provide the evaluation code and corrected results if necessary.
(6) What exact instructions, scoring criteria, sample counts, and attribute combinations are used in Table 6b? What explains the zero scores for strong baseline models on the one- and two-attribute conditions, and do blinded human judgments reproduce these findings?