acceptodds
Withdrawn from ICLR 2027

Attribute-Level Data Self-Evolution for Audiovisual Instruction Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained video understanding relies on detailed and reliable audiovisual captions, yet caption enrichment often introduces factual errors without equally fine-grained verification and correction. Even under caption-level verification, which checks each detailed caption as a whole, we observe a critical granularity mismatch, i.e., coarse feedback fails to pinpoint fine-grained errors and omissions, limiting targeted refinement. To address this mismatch, we introduce ASID-Verify, an attribute-centered framework for verifying and refining audiovisual captions. A verifier audits each caption against the video and available speech transcripts, returning an attribute-organized report that identifies detected errors and omissions and explains each issue. A refiner uses this feedback for targeted correction and completion, and the revised caption is re-audited in the next round, enabling data self-evolution without updating the annotation models. A single round of attribute-level verification already reaches an error-free rate of 52.5%, matching two rounds of caption-level verification at 52.3%, and a second round raises it to 75.4%. With this framework, we construct ASID-1M, a million-scale collection of single- and all-attribute captions for 121K videos, and train ASID-Captioner on it via supervised fine-tuning. Across seven benchmarks spanning captioning, caption-based QA, and temporal grounding, ASID-Captioner improves fine-grained caption quality over open-source baselines and rivals Gemini-3-Pro on several captioning evaluations, while also following attribute-conditioned instructions more accurately. We will release ASID-1M, ASID-Verify, the evaluation suite, and model checkpoints.

Primary area:
datasets and benchmarks
Keywords:
audiovisual captioning, fine-grained video understanding, instruction data curation, attribute-level verification, iterative refinement, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kqoUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ASID-Verify, an attribute-centered pipeline for verifying and refining audiovisual captions. Initial captions and aligned speech transcripts are combined, audited against the video and ASR evidence, and revised using attribute-organized reports. The process repeats with fixed annotation models. The authors construct ASID-1M, containing single-attribute and all-attribute captions for approximately 121K videos, and fine-tune Qwen2.5-Omni models through single-attribute, all-attribute, and longer-context training stages. Evaluation covers captioning, caption-based question answering, temporal grounding, and attribute-conditioned instruction following.

Critical strengths

(1) The paper addresses an important data-quality problem. Detailed audiovisual captions can improve coverage while introducing unsupported statements, so identifying specific errors and omissions before instruction tuning is practically useful. The explicit separation between verification and targeted refinement provides an inspectable curation process.

(2) The evaluation spans several complementary uses of captions, including downstream question answering and temporal grounding. The additional judge evaluation in Appendix F.3 and the alternative training schedules in Appendix F.4 are useful checks. The manuscript also appropriately acknowledges that progressive training order is an implementation choice rather than a demonstrated necessity.

Critical weaknesses

(1) The experiments do not adequately isolate attribute-level verification as the source of improvement.

The main methodological claim is that semantic attributes resolve a granularity mismatch in caption verification. However, the proposed verifier changes several properties of feedback together: it uses predefined attribute-specific checklists, identifies affected snippets, explains evidence, and distinguishes errors from omissions. The refiner then follows these localized reports while minimizing unrelated edits. Figure 2 contrasts this against coarse caption-level feedback, but the manuscript does not establish whether a caption-level verifier was given equally explicit instructions for issue localization, evidence reporting, and targeted correction.

A decisive control would retain the same checklist coverage, localized snippets, evidence requirements, and refinement instructions while removing only attribute grouping. Without this control, the observed advantage may reflect more actionable feedback or more explicit verification instructions rather than semantic attributes as the critical verification unit.

Section 3.4 repeatedly describes verification budgets as matched, but the operational definition of this budget is unclear. Equal round counts or model-call counts do not establish equal generated-token budgets, checklist coverage, or inference expenditure. Appendix I provides the attribute-audit and refinement templates in Figures 17 and 18, but does not provide a comparably explicit template for the caption-level baseline used in the central experiment.

The downstream supervision ablation does not resolve this attribution problem. Section 5.3 explicitly acknowledges that Table 6a matches source videos and instance counts without matching target-token counts. Richer captions therefore change both supervision content and the amount of supervised text. Figure 3 provides useful stage-wise evidence, but does not report a controlled downstream comparison between caption-level and attribute-level refinement under matched supervision budgets.

Consequently, the current evidence supports the usefulness of the complete curation pipeline more strongly than the claimed distinctive role of attribute-level verification. Since this distinction is the principal methodological contribution, the missing controls materially affect my recommendation.

(2) The evidence for fine-grained reliability is insufficiently independently validated, and part of the downstream evaluation is internally inconsistent.

The headline improvements in Section 3.4 and Figure 1 rely on automatic No-Error, No-Missing, and Clean diagnostics. The manuscript does not sufficiently specify their evaluation sample, aggregation procedure, judging model, or independence from the models used for annotation and verification. These details are essential because repeated repair can improve agreement with an automatic auditor without establishing the same improvement in factual correctness against the video.

The human study in Appendix F.1 assesses overall usability using Pass, Borderline, and Fail labels. The authors explicitly distinguish this measure from the automatic diagnostics. This is a reasonable usability check, but it does not independently validate the reported difference between caption-level and attribute-level verification. A paired, blinded human assessment of those conditions is needed to establish that the central improvement reflects fewer factual errors and omissions rather than evaluator alignment. The additional judge in Table 10 evaluates generated model captions, so it does not directly validate this central annotation-repair comparison.

There is also a concrete consistency problem in Table 5. For both Gemini-3-Pro and Gemini-2.5-Flash, the reported mean IoU is too low to be compatible with the reported recalls under standard same-query temporal-grounding definitions. The successful predictions counted by the recall columns alone already require a larger mean IoU than the table reports. Appendix E describes standard mean-IoU and recall evaluation without explaining an alternative aggregation convention that would reconcile these results. This affects the comparison used to support temporally precise captioning.

Finally, Table 6b reports zero or near-zero instruction-following performance for several strong baselines, but the corresponding judging criteria and evaluation procedure are not documented sufficiently to explain whether the metric measures requested-attribute inclusion, exclusion of unrequested content, factual correctness, or a combination. Because these choices can substantially change the interpretation, the table requires human-calibrated validation before supporting strong comparative claims.

Together, these issues weaken the evidence for the paper’s central reliability and instruction-following conclusions.

Questions for authors

(1) What are the complete prompts for the caption-level verification baseline and its refiner? Were they instructed to identify individual erroneous snippets and omissions, provide supporting evidence, and make minimal targeted edits? Does the attribute-level advantage persist against an equally detailed issue-level verifier that omits only attribute grouping?

(2) What exactly is matched in the verification budgets reported in Section 3.4 and Figure 1: calls, input tokens, output tokens, monetary cost, or another quantity? Please report the actual budgets and clarify whether both conditions inspect the same semantic checklist and audiovisual evidence.

(3) Do the downstream gains persist when caption-level and attribute-level refinement use the same videos, training-token budget, optimization steps, and evaluation protocol? How much of the improvement in Table 6a remains after controlling for target-caption length and supervised-token count?

(4) How are No-Error, No-Missing, and Clean defined and aggregated, and which model evaluates them? Is this evaluator independent of the annotation and verification models? Can the central comparison be confirmed by paired, blinded human judgments of factual errors and omissions, rather than overall usability alone?

(5) How can the mean IoU and recall entries for Gemini-3-Pro and Gemini-2.5-Flash in Table 5 be reconciled? Were all columns computed from the same queries, predictions, and failure-handling rules? Please provide the evaluation code and corrected results if necessary.

(6) What exact instructions, scoring criteria, sample counts, and attribute combinations are used in Table 6b? What explains the zero scores for strong baseline models on the one- and two-attribute conditions, and do blinded human judgments reproduce these findings?

Reviewer_1PyhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is about the quality of fine-grained audiovisual caption data. The more detailed a caption is, the more likely it is to contain factual errors. Checking the whole caption as one unit (the authors call this caption-level verification) gives feedback that is too coarse to tell which sentence is wrong or what is missing. The authors propose ASID-Verify, which has three stages. S1 produces initial annotations from several sources (AVoCaDO, Seed-1.6, and Whisper/WhisperX speech transcripts). S2 uses Seed-1.6 to ensemble them into one complete caption and cross-checks them at the text level. In S3, a verifier checks the caption against the video and the speech transcript and reports errors and omissions for each of eight semantic attributes (scene, character, object, action, narrative, speech, camera, emotion); a refiner fixes only the reported issues, and the result is audited again. Below I call S3 attribute-level verification; the authors present it as "data self-evolution". The experiments test one round and two rounds. With this pipeline the authors build ASID-1M (about 121K videos, each with 8 single-attribute captions plus 1 all-attribute caption) and train ASID-Captioner (3B / 7B) on Qwen2.5-Omni with three-stage SFT. The core evidence has two parts. (1) Under matched verification budgets, one round of attribute-level verification reaches a No-Error rate of 52.5, on par with 52.3 after two rounds of caption-level verification, and two rounds reach 75.4. (2) On 7 benchmarks (captioning, caption-based QA, and caption-based temporal grounding), ASID-Captioner outperforms the open-source models it is compared with. The 7B model has a total error of 35.9 on Video-SALMONN-2 (Gemini-3-Pro: 35.7) and an average score of 81.2 on UGC-VideoCap (Gemini-3-Pro: 81.9).

Critical strengths
  1. The problem is real, and the deliverables are directly useful to the community. The reliability of fine-grained audiovisual caption data is a practical bottleneck in training video LLMs today. The paper promises to release ASID-1M, the verification pipeline, the evaluation suite, and the model weights, and the appendix gives the prompts for annotation, ensembling, verification, and refinement (Figure 14–18), so the pipeline can be reproduced by following them.

  2. Large effort and wide evaluation. The paper covers an initial pool of about 135K videos, final data of about 121K videos, two model sizes, 7 benchmarks, and a human spot check of 1,500 videos. The lead over open-source models is clear: the 7B model has a total error of 35.9 on Video-SALMONN-2 (AVoCaDO: 37.3), an average of 81.2 on UGC-VideoCap (AVoCaDO: 73.2), a VDC average accuracy of 40.9 (AVoCaDO: 36.3), and 61.2 / 34.0 on caption-based QA with Daily-Omni / WorldSense (AVoCaDO: 50.1 / 25.7).

  3. The paper is clear about the trade-off between coverage and reliability, and it reports unfavorable results honestly. The paragraph "Granularity alone is insufficient" in §3.4 shows that multi-attribute annotation raises the No-Missing rate from 42.5 to 72.9 but lowers the No-Error rate from 59.1 to 35.2. Table 12 concedes that mixed training beats SA→AA, and the authors accordingly call the staged training order "an implementation choice rather than a requirement". In Appendix F.3, after switching the judge to Qwen3.5-27B, they also state plainly that Gemini-3-Pro remains better.

  4. The data ablation has basic controls. Table 6a fixes the source videos and the number of training samples (20K) and changes only the annotations, and the authors say up front that target token counts are not matched.

Critical weaknesses

W1 (Major) The core claim that attribute-level verification beats caption-level verification rests on a single set of automatic diagnostic numbers whose definitions, judging procedure, and sample size are not reported

The paper's title and its first contribution both rest on attribute-level verification. The only support is one comparison in §3.4 and Figure 1(a), listed as No-Error / No-Missing / Clean: attribute-level, 52.5 / 87.5 / 47.4 after one round and 75.4 / 93.2 / 67.4 after two; caption-level, 42.8 / 81.3 / 39.8 after one round and 52.3 / 81.8 / 42.6 after two.

These numbers have five problems. (1) The three metrics are never defined: Appendix F.1 calls them "automatic diagnostics" but gives no judge model, prompt, or sample size, and if the judge is Seed-1.6 or reuses the checklist from the attribute-level audit, attribute-level refinement is optimizing against the scoring rubric and the comparison is circular. (2) The paper does not say whether "matched verification budgets" are matched by number of calls, tokens, or cost. (3) If Clean means "no errors and no omissions", Clean after two attribute-level rounds cannot be below 75.4 + 93.2 − 100 = 68.6, yet the reported value is 67.4. (4) The human spot check of 1,500 videos covers only the final data and does not compare the two verification methods. (5) The paper already has a well-controlled 20K training setup (Table 6a) but trains no control model on caption-level-verified data, so there is no direct evidence yet of how much attribute-level verification contributes to the final model.

Requested additions: definitions of the three metrics, the judge model and prompt, the sample size, and how the budget is measured; one more row in the 20K setting of Table 6a with training results for caption-level verification (same budget, same number of rounds); and, if possible, a manual check of about 100 samples for each verification method.

W2 (Major) The sources of the main gains are not separated and look more like distillation of the teacher model Seed-1.6; the final model's hallucination does not decrease

Seed-1.6 acts as the S1 annotator, the S2 ensembler, and the S3 verifier, but Seed-1.6 itself does not appear in any comparison table. The paper's own figures suggest that a large share of the gains comes from learning Seed-1.6's annotations and verbatim speech transcripts rather than from attribute-level verification:

  • Seed-1.6's raw annotations alone are already fairly clean, and most of the downstream gain is already obtained at S1. Figure 3 has no value labels, so the numbers below are read from bar heights (for 3(a), calibrated with the two values given in the text, 52.5 and 75.4). In Figure 3(a), single-source Seed-1.6 annotation at S1 has a No-Error rate of about 67, higher than the 52.5 after one round of attribute-level verification. S2 multi-source ensembling lowers it to about 35, and it only returns to 75.4 after two rounds of S3, so a good part of the S3 improvement goes into repairing errors introduced by S2. In Figure 3(b) (3B model), the missing rate is about 44 when training on the original GT, about 30 with S1 annotations, and about 25 with S3 annotations.
  • Hallucination does not decrease. The 7B model's hallucination rate on Video-SALMONN-2 is 15.4, the same as the base model's 15.4. The 3B model is at 18.3 against 16.4 for its base (and Table 6a bolds the even higher 18.9 as best). The authors explain that finer captions mention real details the reference does not annotate (§5.3), but they do not test this explanation.
  • Caption length is not reported. Caption-based QA and temporal grounding both have Gemini-2.5-Pro answer from the caption alone, which favors longer captions.
  • The VDC judge is the teacher model Seed-1.6. The student's output style matches the judge's, so there is a risk of self-preference, and the recheck in Appendix F.3 does not cover VDC.

Requested additions: add the caption results of Seed-1.6 itself to Tables 1, 2, and 4; report the exact values in Figure 3 and the average caption length of each model; recheck VDC with a judge outside the Seed family; and, for ASID-Captioner and the base model, manually check about 100 items each that were judged as hallucinations.

W3 (Important) The evidence behind the attribute instruction-following and temporal grounding results is flawed

  • The attribute instruction-following evaluation (Table 6b) has no protocol. The number of samples, how attribute combinations are sampled, and the judge prompt and criteria are all unspecified. Gemini-3-Pro and Gemini-2.5-Pro score 0 on single-attribute requests, and AVoCaDO scores 0 in all four settings. That is hard to believe; more likely the criteria measure conformity to the ASID training format.
  • In the temporal grounding results (Table 5), the mIoU of the two Gemini models contradicts their own recall. Gemini-3-Pro has [email protected] of 46.4, so its mIoU is at least 0.3 × 46.4 ≈ 13.9 (about 21.2 once the other two recall levels are taken into account), but the table reports 8.8. Gemini-2.5-Flash has [email protected] of 40.0, so its mIoU is at least 12.0, but the table reports 6.1. By recall, ASID-Captioner 7B (44.1 / 26.3 / 11.0) and Gemini-3-Pro (46.4 / 26.0 / 10.2) are roughly even.

Requested additions: give the full protocol for Table 6b, or evaluate directly on IF-VidCap (ICLR 2026), the public benchmark the authors already cite; check the mIoU values in Table 5 and revise the statement that ASID-Captioner "performs strongly across all metrics" accordingly.

Questions for authors
  1. How are No-Error, No-Missing, and Clean each defined? Which model judges them, and with what prompt? On how many videos are they computed? Is the judge independent of Seed-1.6 and of the attribute-level audit prompt? What are "matched verification budgets" matched on?
  2. In the 20K setting of Table 6a, what are the results on Video-SALMONN-2 and VDC when training on data processed with caption-level verification (same budget, same number of rounds)?
  3. How does Seed-1.6's own captioning perform on Tables 1, 2, and 4? How large is the remaining gap between ASID-Captioner 7B and the teacher model?
  4. In Table 5, the mIoU of Gemini-3-Pro (8.8) and Gemini-2.5-Flash (6.1) contradicts their recall. Is this a calculation error, or are the denominators different? What are the sample size and the judging criteria for Table 6b, and do the Gemini models really score 0 on single-attribute requests?
  5. Table 7 lists 150k + 136k + 53k samples across the three training stages. How does this relate to the roughly 121K × 9 ≈ 1.09M captions, and is all of ASID-1M used for training? Also, §3.4 says training uses 200K captions while §5.3 says 20K training samples, yet Figure 3(b) and the Original GT row of Table 6a (43.9 / 22.3) appear to come from the same experiment. Please clarify.

If the answers to questions 1–3 show that the metrics are independent of the method itself and that attribute-level verification has a clear advantage in the downstream control, I will raise my score.

Reviewer_rhDROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ASID-Verify, an attribute-level verification and refinement work for curating high-quality audio-visual instruction data. The proposed pipeline combines multi-score catpion generation, caption ensemble, and iterative attribute-level verification to identify and correct fine-grained errors and omissions. The author constructs ASID-1M using proposed pipeline which contains nearly one million single-attribute and all-attribute captions from 121K videos. The author finetunes Qwen2.5-omini using single-attribute, all-attribute, and long-context training stages in progress. Experiments demonstrate consistent improvements over seven benchmarks.

Critical strengths
  • The motivation is well-established. The paper identifies the limitation of caption-level verification: feedback may fail to precisely localize fine-grained errors and missing information. Reformulating feedback by semantic attributes is intuitive and practically useful for improving annotation quality.

  • The results are promising. The proposed framework achieves a No-Error rate if 75.4%, compared with 52.3% for caption-level verification after two rounds. The finetuned 7B captioner achieves 40.9 average accuracy on VDC and 81.2 on UGC-VideoCap, outperforming some larger models.

  • The dataset contribution is substantial. ASID-1M provides a large collection of detailed audio-visual captions with both single-attribute and all-attribute supervision.

Critical weaknesses
  • The technical novelty is limited beyond existing verification and refinement pipelines. The primary contribution is to organize LLM-generated verification feedback according to predefined semantic attributes and iteratively refine the captions. The components like multi-source annotation, structured feedback generation, and iterative refinement, are already well established. The proposed ''data self-evolution'' is more like iterative refinement or test-time sampling, which is close to existing LLM-base iterative correction frameworks.

  • The performance gains from attribute-level verification and richer supervision is insufficiently disentangled. The experiment does not fully isolate the contribution of attribute-level verification to the final performance. Moreover, the author acknowledges that target-token counts are not matched in Table. 6. Therefore, the improvement may come from longer and more information, rather than the proposed attribute structure. To further validate the superiority in annotation quality, the downstream task performance comparison between models trained on attribute-level and caption-level verification is necessary. The alternative is to validate finetuned captioner on reasonable downstream tasks.

  • The designed metric is not clearly claimed. The author calculates the total value of categorized metrics in video-SALMONN-2-testset, but averages the categorized metrics in UGC-VideoCap. The discrepancy of the choice between these two datasets should be claimed.

  • The reported number is not reliable and consistent. In Table.5, the Gemini-3 Pro is reported to achieve an mIoU of 8.8% while obtaining [email protected] of 46.4%. If both metrics are evaluated on the same predictions and groundtruth, an [email protected] of 46.4% implies an mIoU of ate least 13.9% $\mathrm{mIoU}=\frac{1}{N}\sum_{i=1}^{N}u_i$, [email protected] means ui≥0.3u_i\geq 0.3, therefore mIoU≥0.3×R1\@0.3=0.464×0.3=0.1392=13.92%\mathrm{mIoU}\geq 0.3\times R1\@0.3 = 0.464\times0.3=0.1392=13.92\%. The same problem also appears on Gemini-2.4-Flash results (mIoU=6.1%, [email protected]=40.0%) so the mIOU should at least be 0.400×0.3=12.0%0.400\times 0.3=12.0\%. The author should clarify the reasons here.

Questions for authors

Refer to the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.