acceptodds
Under review as a conference paper at ICLR 2027

TAG: Task-Conditioned Adaptive Diagnosis for Multimodal Data

Abstract

Diagnosing incorrect answer annotations is important for reliable training and evaluation of multimodal models. Existing approaches use direct VLM judgments or representation-based detectors, but differences in datasets, representations, and noise conditions complicate understanding when each strategy is effective. We compare these approaches under a common protocol across three multimodal benchmarks with controlled answer errors. The comparison reveals potential gains from selecting strategies for individual tasks over the best shared strategy. On AVA-Bench, encoder preferences vary across abilities but remain relatively stable across the tested noise levels, whereas detector preferences vary with both ability and noise. To demonstrate the value of these findings, we introduce Task-conditioned Adaptive Diagnosis (TAG), which identifies tasks and estimates noise conditions to select and combine diagnostic strategies, with confidence-based fallback to a shared strategy. Cross-validation shows that TAG improves AUROC over the best shared strategy by 1.24 percentage points on AVA-Bench, with smaller gains on ScienceQA and MMMU, demonstrating that task-conditioned selection can translate diagnostic preference differences into improved diagnosis.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Data Quality Diagnosis, Multimodal Learning, Adaptive Diagnosis, Data-Centric AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fT14OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper investigates the detection of incorrect answer annotations in multimodal datasets by systematically comparing different encoders, label-error detectors, and direct VLM judgments across multiple tasks and noise conditions. Based on the observation that encoder and detector preferences vary across tasks and noise levels, the authors propose Task-conditioned Adaptive Diagnosis (TAG), which selects task-specific encoders and noise-conditioned detectors with a confidence-based fallback mechanism. Experiments on AVA-Bench, ScienceQA, and MMMU demonstrate improvements over a shared diagnostic strategy, particularly on AVA-Bench.

Critical strengths
  1. The paper systematically evaluates eight encoders and eleven detectors across three multimodal benchmarks and multiple noise levels.
  2. The analysis reveals that encoder preferences are relatively stable across noise levels but differ across tasks, while detector preferences depend on both task characteristics and noise conditions.
  3. The ablation studies demonstrate the contributions of task-specific encoder selection and detector adaptation.
Critical weaknesses
  1. The proposed TAG framework mainly combines existing encoders, label-error detectors, prototype-based task identification, and confidence-based fallback through a heuristic selection mechanism.
  2. The methodological contribution appears relatively incremental, with limited novelty beyond task-conditioned algorithm selection.
  3. Although TAG achieves noticeable improvements on AVA-Bench, the gains on ScienceQA and MMMU are only 0.16 and 0.17 AUROC percentage points. Moreover, direct VLM judgment outperforms TAG on MMMU, raising concerns about the general applicability of the proposed approach.
  4. The experiments exclusively consider artificially injected errors through correctness-label flipping in multiple-choice datasets, which may not adequately reflect the complexity of real-world multimodal annotation errors.
  5. The confidence-based fallback mechanism routes approximately 69% of MMMU samples to the shared strategy and reduces the improvement compared with disabling the gate.
  6. The paper lacks a detailed computational cost analysis, making it difficult to assess whether the relatively modest performance gains justify the additional complexity.
Questions for authors

Please see the weaknesses.

2 comments

Thank you for recognizing our systematic comparison and the task- and noise-dependent diagnostic preferences revealed by the analysis. We clarify how these findings motivate TAG and provide additional evidence on its applicability, fallback mechanism, and computational cost.

W1. Motivation and design

Should questions testing different abilities be checked with the same representation and diagnostic signal? TAG grew out of comparing eight encoders and eleven detectors across tasks and noise rates, using identical images, questions, and candidates and evaluating all rankings against the same errors.

On all three benchmarks, post-hoc selection of the best encoder–detector pair per task outperformed the best shared pair under matched noise conditions. Encoder preferences differed across tasks but changed less often across noise rates. Detector preferences varied with both. Between adjacent rates, the preferred detector and encoder changed 13 vs. 4 times on AVA-Bench, 15 vs. 1 on ScienceQA, and 10 vs. 3 on MMMU.

These findings motivate TAG (Fig. 1). Training-batch performance determines one encoder per task across states and three detectors per task and noise state. Prototype matching estimates which task rule to apply, and an AUM probe estimates the batch noise state. TAG averages the selected detectors’ normalized ranks and falls back to a shared strategy when task assignment is uncertain.

In each fold, we learned rules on four batches and froze them before diagnosing the fifth. Detectors could use its observed answer labels. Its true task labels and flip records were withheld from fitting and routing, and the flip records served only to evaluate rankings. TAG’s held-out AUROC gains over Global were 1.24, 0.16, and 0.17 percentage points on AVA-Bench, ScienceQA, and MMMU, respectively.

W2. Contribution beyond existing adaptive selection

LSCP [1, §3.4] selects outlier detectors by agreement with a local pseudo-target. Ali et al. [2, §3.5] predict detector weights from tabular dataset characteristics. SNLC [3] studies robust VQA training under semantically related answer replacements. Our contribution is to characterize representation and detector preferences across multimodal tasks and noise conditions and test whether learning them improves answer-annotation diagnosis in an unseen mixed-task batch.

On all three full datasets, task-specific gains over the best shared pair exceeded the 95th percentile of gains from 500 association-group task-label permutations (Table 1). On held-out AVA batches, with encoder and fusion fixed, task- and noise-conditioned detector selection gained 0.54 AUROC percentage points over Global-top3 (bootstrap mean). Task-specific encoders with reselected detectors added 0.69 points (Appendix C, Table 14).

W3. Small gains on ScienceQA and MMMU

On ScienceQA, Global already reaches 0.9749 AUROC, above direct Qwen2.5-VL-7B judgment at 0.9213. TAG reaches 0.9765, a gain of 0.16 percentage points (paired-bootstrap 95% CI [0.03, 0.35]). On MMMU, direct judgment leads TAG (0.7093 vs. 0.6788), and TAG’s 0.17-point gain over Global remains uncertain (95% CI [−0.08, 0.45]; Appendix C, Tables 13–15).

Beyond AVA, we tested each parent dataset’s frozen held-out rules on 1,152 original MMMU answer pairs aligned to MMMU-Pro [4] and 292 ScienceQA questions aligned to LIME [5]. Detectors were refitted within each subset at 10%–40% nominal injected noise.

TAG − Global, task-weighted AUROC / AP gains (pp)10%20%30%40%
MMMU-Pro-aligned+0.36 / +0.53+0.25 / +0.43+0.34 / +0.30+0.65 / +0.70
LIME-aligned+1.66 / +2.05+0.85 / +0.80+1.08 / +0.99+0.73 / +0.47

Methods use the same tasks containing both flipped and unflipped questions within each condition. Across rates, mean question coverage over folds and seeds is 97.2%–100% for MMMU-Pro and 57.7%–90.9% for LIME.

TAG also exceeds Global-top3 in mean task-weighted AUROC on both subsets at all four rates. This supports gains over shared diagnosis beyond AVA with same-dataset calibration. Direct judgment leads in our main MMMU evaluation and exceeds TAG in both AUROC and AP on LIME at 30% and 40% noise. We plan to study task-conditioned selection with direct judgment in the candidate pool, considering diagnostic performance and computational budget.

References

[1] Zhao et al. LSCP: Locally Selective Combination in Parallel Outlier Ensembles. arXiv:1812.01528, 2018.

[2] Ali et al. Adaptive Ensemble Selection for Noisy Labels on Tabular Data. arXiv:2609.32976, 2026.

[3] Zhang et al. Learning Visual Question Answering on Controlled Semantic Noisy Labels. Pattern Recognition, 138, 109339, 2023.

[4] Yue et al. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL, 2025.

[5] Zhu et al. LIME: Less Is More for MLLM Evaluation. Findings of ACL, 2025.

W4. Beyond artificial label flips

Drawing on the label-noise models of Natarajan et al. [6] and Liu and Tao [7], we compare strategies on the same errors at each noise rate. For a specified fraction of questions, we swap the correctness labels of the reference answer and one original distractor. Images, questions, and candidate texts stay fixed, with one answer marked correct per question.

Using external human annotations, we matched AVA birds to CUB++ species corrections (fgvcdata 0.1.0) [8], MMMU questions to reviewed OCRBench v2.1 answers [9, §11.1], and AVA textures to DTD attributes identifying a second valid answer [10]. LIME [5] retains the original keys for all 292 matched ScienceQA questions and provides no corrected keys for this evaluation.

All methods ranked the same original questions with the original pairs and labels, without injected errors. We froze TAG’s rules, refitted detectors within each held-out subset, and used external references only for evaluation. Entries count known cases in approximately the top 10% of each ranking. Global is the training-selected shared strategy. Judge is direct Qwen2.5-VL-7B judgment.

Subset (questions)Known casesReview budgetTAGGlobalJudge
AVA birds, initial (2,330)27 incorrect keys2331158
AVA birds, expanded (2,351)48 incorrect keys23615611
MMMU (109)6 corrected keys11000
AVA textures (4,487)10 with a second valid answer449546

The expanded bird subset adds 21 questions to the initial cohort. Their corrected species are absent from all original options. All 48 corrections lie outside the frozen pairs, so hits locate incorrect keys. On the initial subset, Judge had higher overall AUROC (0.7410 vs. 0.7371), but TAG found more errors within budget.

W5. Why fallback reduces the MMMU gain

The gate thresholds the margin between the two highest task-prototype similarities, targeting at least 95% empirical task-assignment accuracy on retained training-batch predictions. On evaluable fallback questions, forced task-conditioned scoring underperforms Global on ScienceQA but outperforms it on MMMU. On ScienceQA, routing about 16% to Global raises the observed AUROC gain from 0.11 to 0.16 percentage points. On MMMU, routing about 69% to Global reduces the gain from 0.41 points (95% CI [−0.03, 0.86]) to 0.17 (95% CI [−0.08, 0.45]). Both intervals include zero. No AVA fold triggered fallback (Table 3 and Appendix C, Tables 14, 17, 18).

Task-assignment confidence need not track diagnostic benefit. Future work could calibrate the gate on training-batch diagnostic performance.

W6. Computational cost and diagnostic gains

Timings use AVA’s held-out batch B5 (fold five, 20% noise, seed 0, 23,291 questions, 46,582 candidates), with rules frozen from the other four batches. Encoding and cached-feature diagnosis used one NVIDIA A800 80 GB PCIe GPU. We timed prototype matching separately on CPU.

StageWorkloadTime
Calibration from cached scores93,162 training questionsStrategy fitting 17.48 s, TAG router preparation 9.58 s, router calibration 10.54 s
Model loading, encoding, and savingSame 160 B5 questions, 16 per taskGlobal 470.140 s, TAG 673.278 s (+43.2%)
CPU prototype matchingNormalized features, all 23,291 questions0.0472 s (median of five runs)
Fixed-encoder diagnosisFull B5, cached InternVL3.5 features, 11 candidate detectors, three scores per questionShared Top-3 16.91 s, task-specific Top-3 26.67 s
Complete cached-feature diagnosisFull B5, detector fitting, scoring, routing, and aggregationGlobal 10.762 s, Global-top3 16.473 s, TAG 81.918 s

The last two rows give medians of three runs. Differences use unrounded medians.

With the encoder fixed, task-specific selection used seven detectors across B5 and took 9.75 s longer. AUROC rose from 0.9391 to 0.9473 and AP from 0.8593 to 0.8707 on the same questions. Full TAG required two additional encoders on B5. From cached features, TAG took 65.44 s longer than Global-top3 (4.97× runtime). Across five AVA folds, TAG gained 1.23 AUROC and 2.48 AP percentage points over Global-top3 with the same three-score fusion rule. Full-batch cold-start time from raw inputs was not measured.

References

[6] Natarajan et al. Learning with Noisy Labels. NeurIPS, 2013.

[7] Liu and Tao. Classification with Noisy Labels by Importance Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3), 447–461, 2016.

[8] Anderson et al. Facing the Hard Problems in FGVC. arXiv:2006.13190, 2020.

[9] GuangJian Team. PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence. arXiv:2609.37712, 2026.

[10] Cimpoi et al. Describing Textures in the Wild. CVPR, 2014.

Thank you again for your careful review and constructive suggestions.

Reviewer_Q86yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies incorrect answer annotation detection by comparing direct VLM judgments with 88 encoder–detector combinations across three multimodal benchmarks under controlled label corruption. It proposes TAG, which selects encoders and combines detectors according to predicted tasks and estimated noise levels, with confidence-based fallback. Five-fold evaluations and ablations show a 1.24-point AUROC improvement over the best shared strategy on AVA-Bench, with substantially smaller gains on ScienceQA and MMMU.

Critical strengths
  1. The paper systematically compares representation choices and diagnostic signals under a common evaluation protocol. Analyses across tasks and noise levels, together with matched-checkpoint comparisons, provide useful evidence about when different diagnostic strategies perform well.

  2. The AVA-Bench improvement is supported by grouped cross-validation, paired confidence intervals, and ablations separating task-dependent selection from detector averaging. Improvements at fixed review budgets also show that the gains can help prioritize injected annotation errors for inspection.

Critical weaknesses
  1. The evaluation is limited to synthetic label reversals between correct and incorrect options. There is no validation on naturally occurring annotation errors or evidence that the resulting data cleaning improves downstream training or evaluation. It remains unclear whether the learned diagnostic preferences extend beyond the specific corruption process used in the experiments.

  2. The benefits of the adaptive framework are concentrated on AVA-Bench. Gains on ScienceQA and MMMU are only 0.16 and 0.17 AUROC points, and the MMMU improvement is not statistically established. TAG also underperforms the direct VLM judge on MMMU, while its confidence gate reduces the gain. These results provide limited support for a consistent advantage over simpler diagnostic strategies.

Questions for authors

Please see the Critical Weaknesses section for my main concerns.

4 comments

Thank you for your careful evaluation of our study. We appreciate your recognition of the common comparison protocol and the held-out results on AVA-Bench. We address your concerns about natural annotation errors and downstream benefits in W1, and the consistency of gains over simpler diagnostic strategies in W2.

W1. Natural annotation errors and downstream benefits

For the controlled comparison, we draw on the label-noise formulations of Natarajan et al. [1] and Liu and Tao [2], keeping inputs fixed while changing observed labels. In multiple-choice QA, this lets us vary error rates on the same questions: we retain the reference answer and one original distractor, then swap their correctness labels for a specified fraction of questions. Exactly one answer remains marked correct, and all methods are evaluated on identical content and recorded errors. Wei et al. [3, §4.2] showed on CIFAR-N that matching aggregate transition rates with synthetic noise did not reproduce how human errors depend on individual examples.

To address your concern about naturally occurring annotation errors, we conducted additional evaluations using existing human annotations and correction records. We matched AVA bird questions to CUB++ labels from the fixed fgvcdata 0.1.0 release [4], MMMU questions to reviewed OCRBench v2.1 answers [5, §11.1], and AVA texture questions to DTD attributes [6]. Diagnosis retained the original questions and observed labels, without injecting errors. TAG’s selection rules remained frozen; detectors were refitted within each held-out subset, and the external references were used only for evaluation.

We counted known cases in approximately the top 10% of each method’s risk ranking. Global is the training-selected shared strategy; Judge is direct Qwen2.5-VL-7B judgment. Entries count known cases recovered.

SubsetKnown casesReview budget (questions)TAGGlobalDirect judge
AVA birds, initial 2,330 questions27 incorrect keys2331158
AVA birds, expanded to 2,351 questions48 incorrect keys23615611
MMMU, 109 questions6 corrected keys11000
AVA textures, 4,487 questions10 with a second valid answer449546

On the bird subset, rules learned from synthetic corruption also improve the prioritization of naturally incorrect keys. This evaluates error localization: all 48 corrected answers lie outside the frozen candidate pairs, including 21 absent from every original option. The expanded cohort retains all initial questions and adds these 21 cases. On the initial cohort, the judge has slightly higher overall AUROC than TAG (0.7410 versus 0.7371), while TAG recovers more known errors within the review budget.

Using the original candidate pairs, none of the three methods recovered a known error within an 11-question review budget on MMMU. All six corrected answers lie outside those pairs.

These evaluations use existing annotation errors while retaining multiple-choice questions. Extending the comparison to open-ended QA would first require verified response-level references that allow multiple valid answers. We could then compare task-conditioned and shared strategies on the same verified errors. Cross-dataset rule transfer remains future work.

We appreciate your suggestion to examine downstream effects. Evaluating how data cleaning affects model training and how corrected answer keys affect benchmark rankings would be a valuable direction for future work.

References

[1] Natarajan et al. Learning with Noisy Labels. NeurIPS, 2013.

[2] Liu and Tao. Classification with Noisy Labels by Importance Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3), 447–461, 2016.

[3] Wei et al. Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ICLR, 2022.

[4] Anderson et al. Facing the Hard Problems in FGVC. arXiv:2006.13190, 2020.

[5] GuangJian Team. PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence. arXiv:2609.37712, 2026.

[6] Cimpoi et al. Describing Textures in the Wild. CVPR, 2014.

Our responses to the remaining points continue in the next comment.

W2. Consistency of gains over simpler diagnostic strategies

Thank you for highlighting the variation in gains across datasets.

On AVA, TAG gains 1.23 AUROC percentage points over Global-top3 with the same detector count and fusion rule. The bootstrap mean gains are 0.54 points from task- and noise-conditioned detector selection under a shared encoder, and 0.69 points from task-specific encoders with reselected detectors (Table 2).

On ScienceQA, Global already reaches 0.9749 AUROC, and TAG adds 0.16 percentage points (paired-bootstrap 95% CI [0.03, 0.35]; Appendix C, Table 14). On MMMU, TAG’s mean AUROC is 0.6788 versus 0.6771 for Global. The confidence gate routes approximately 69% of questions to Global, reducing the observed gain from 0.41 points (95% CI [−0.03, 0.86]) to 0.17 points (95% CI [−0.08, 0.45]). It is calibrated for task-assignment accuracy, which need not maximize diagnostic performance. Direct judgment reaches 0.7093 AUROC, outperforming both TAG and Global. These intervals condition on the fitted fold rules.

To further examine whether the benefits extend beyond AVA, we tested each parent dataset’s frozen held-out rules on curated question subsets at nominal injected-noise rates of 10–40%: 1,152 original MMMU pairs aligned to MMMU-Pro [7] and 292 ScienceQA questions aligned to LIME [8]. Detectors were independently refitted within each subset.

TAG − Global, task-weighted AUROC / AP (percentage points)10%20%30%40%
MMMU-Pro-aligned+0.36 / +0.53+0.25 / +0.43+0.34 / +0.30+0.65 / +0.70
LIME-aligned+1.66 / +2.05+0.85 / +0.80+1.08 / +0.99+0.73 / +0.47

Across the four noise rates, the mean fraction of questions included in metric calculation is 97.2–100% for MMMU-Pro and 57.7–90.9% for LIME. Methods use the same evaluable tasks within each condition.

On the evaluable tasks of both curated subsets, TAG’s mean task-weighted AUROC exceeds Global and Global-top3 at all four noise rates. LIME’s AP differences against Global-top3 are mixed (−0.14 to +0.45 points). On LIME, direct judgment leads in both AUROC and AP at 30% and 40% noise. The matched ensemble size and fusion rule in the Global-top3 comparison support an AUROC benefit from adaptive selection beyond detector averaging.

We also appreciate your observation about direct judgment on MMMU. A useful direction for future work is to include it in the candidate strategy pool and study selection based on task and computational budget.

References

[7] Yue et al. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL, 2025.

[8] Zhu et al. LIME: Less Is More for MLLM Evaluation. Findings of ACL, 2025.

Thank you again for your thoughtful comments and constructive suggestions.

Thank you for the response. It addresses most of my concerns and I will raise my score.

Thank you for taking the time to consider our response and revisit your assessment. We are glad that our response addressed most of your concerns. Your questions prompted us to examine how well TAG identifies naturally occurring annotation errors and to explain more clearly why its gains vary across datasets. We have revised the manuscript in light of this discussion, and we appreciate the care you put into reviewing our work.

Reviewer_Wb3vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to detect incorrect answer annotations in multimodal data. The comparison uses three multiple-choice benchmarks (AVA-Bench, ScienceQA, MMMU). The authors observe an “oracle gap”: choosing the best strategy for each task post hoc beats the best shared strategy. Based on these observations, the authors propose Task-conditioned Adaptive Diagnosis (TAG). TAG assigns each question to a task by prototype similarity, estimates the batch noise level with an AUM probe, selects a per-task encoder and the top-3 detectors for that task and noise level, and falls back to the shared strategy when the task assignment is uncertain. In five-fold cross-validation, TAG improves AUROC over the best shared strategy by 1.24 points on AVA-Bench, and by 0.16 and 0.17 points on ScienceQA and MMMU.

Critical strengths
  • The paper studies an important problem. Incorrect answer annotations in multimodal training data and benchmarks hurt training and can distort model comparisons. A systematic study of which diagnostic signals work, and when, is useful for the community.

  • The common comparison protocol is useful in itself. It compares many encoders, detectors, and VLM judges under the same splits, noise levels, and metrics.

  • The paper proposed TAG, which seems to be reasonable in evaluating diagnose strategy. The paper reports where TAG does not help too, e.g., the confidence gate lowers the MMMU gain, and the MMMU VLM judge beats every strategy.

Critical weaknesses
  • The presentation is hard to follow and can be improved. Problem setup somehow is written vague, and I have to infer what the pipeline and strategies are before reading in-depth of the paper.

  • Limited Scope: Three benchmarks (multi-choice VQA) are limited. All three are image–text multiple-choice QA, and the noise construction depends on that format. AUROC is already 0.93–0.98 on AVA-Bench and ScienceQA. This is only a specific setup of much larger applications of multi-modal data.

  • The gains are marginal and the technical contribution is limited. The method is mainly a lookup table over a fixed candidate pool, close to classic algorithm selection, with little explanation of why certain signals suit certain tasks.

  • TAG might be hard to apply in practice. It needs injected-error training batches with task labels from the same dataset, and it assumes the dataset identity is known. Transfer to new datasets is not tested.

  • The VLM judge baseline is weak. Only 7–8B judges are tested.

Questions for authors
  • How well would the method extend beyond image–text multiple-choice data, for example to coding, agent trajectories, or open-ended generation?

  • How does the method perform with more, and more diverse, benchmarks? For example, open-ended VQA, instruction-tuning data, or other modalities. Do the encoder and detector preferences found on AVA-Bench transfer to them?

  • How do the preferences and TAG gains look at realistic noise rates (1–5%), and on naturally occurring annotation errors (e.g., a manually verified subset of ScienceQA or MMMU)?

  • Can rules learned on one dataset transfer to another with a similar task taxonomy, so that a new dataset does not need injected-error training batches?

  • Does better diagnosis lead to downstream benefits, such as better models after cleaning, or changed benchmark rankings?

5 comments

Thank you for recognizing the importance of detecting incorrect answer annotations and the value of our common comparison protocol. We provide clarifications and additional results to address your concerns.

W1. What motivates TAG, and how does it work?

We study when choosing encoder–detector pairs by task improves on a shared strategy for detecting answer annotation errors in multimodal question answering, and how preferences across tasks and noise levels can guide detection on new batches. On AVA-Bench, ScienceQA, and MMMU, our post-hoc analysis showed that selecting encoder–detector pairs by task yielded additional AUROC gains over the best shared pair under matched noise conditions. Within a task, the preferred detector changed more often than the preferred encoder between adjacent noise rates: 13 vs. 4 times on AVA, 15 vs. 1 on ScienceQA, and 10 vs. 3 on MMMU.

These findings motivate the two choices in Fig. 1. TAG selects an encoder by task and three detectors by task and batch noise state. On a held-out batch, TAG matches each question to a task prototype, estimates the batch noise state with an AUM probe, and averages the selected detectors’ normalized ranks to score each question. When task assignment is uncertain, TAG uses the shared strategy for the estimated noise state. Across held-out folds, TAG improved AUROC over the shared strategy (Global) by 1.24, 0.16, and 0.17 percentage points on AVA, ScienceQA, and MMMU, respectively.

W2. Limited scope

Thank you for raising this concern. AVA-Bench, ScienceQA, and MMMU cover different tasks but share a multiple-choice format. This lets us compare diagnostic strategies across tasks using the same definition of an answer-key error and vary the noise rate on the same questions. Drawing on the label-transition models of Natarajan et al. [1] and Liu and Tao [2], we retain the reference answer and one original distractor per question, then swap their correctness labels for a specified fraction of questions. Exactly one answer remains marked correct, and every method is evaluated on the same images, questions, candidates, and recorded errors. Under this protocol, task-specific selection yielded additional AUROC gains over the best shared pair on all three benchmarks in the post-hoc comparison (W1). Wei et al. [3, §4.2] showed on CIFAR-N that matching aggregate human label-transition rates did not reproduce which examples people tended to mislabel. We therefore evaluate naturally occurring errors separately in Q3.

Global already achieves high AUROC on ScienceQA and AVA-Bench. On ScienceQA, TAG yields a small gain, from 0.9749 to 0.9765. On AVA-Bench, Global reaches 0.9339, and using a shared top-three detector ensemble adds only 0.01 AUROC percentage points. With three detectors in both strategies, TAG gains a further 1.23 points over the shared ensemble on held-out batches. The gain also translates into higher precision at a fixed review budget. When reviewing the highest-risk 5% of questions within each AVA task, 94.77% of the questions flagged by TAG contained an injected error, compared with 93.39% for Global (Appendix C.2, Table 21).

W3. Technical contribution and magnitude of the gains

Thank you for raising this question. A common formulation of outlier model selection is to choose a detector or ensemble for a target dataset. TAG addresses a finer-grained question: when do different tasks within the same multimodal QA batch benefit from different diagnostic signals? Fig. 2b illustrates the distinction. With Qwen3-VL fixed, SimiFeat [4], which uses agreement with neighboring labels, ranked first for both absolute and relative depth at 10% noise. At 20%, 30%, and 40%, it remained first for absolute depth, while AUM [5], which uses the observed label’s average training margin, ranked first for relative depth. Thus, the two tasks favored different signals at the same noise rate, and the preferred signal for relative depth also changed with noise. TAG learns selection rules from calibration batches. On a held-out batch, it selects an encoder by inferred task, then selects and combines three detectors based on that task and the estimated batch noise state.

On held-out AVA-Bench batches, TAG improved average precision (AP) by 2.48 percentage points over Global-top3 with the same ensemble size and fusion rule. AP rose from 0.8597 for Global-top3 to 0.8700 when detectors were selected by task under the shared encoder, then to 0.8845 when TAG also selected the encoder by task and reselected its detectors. The corresponding bootstrap mean AUROC gains were 0.54 and 0.69 percentage points, with paired-bootstrap 95% confidence intervals of [0.48, 0.61] and [0.64, 0.74], respectively. W2 gives the fixed-budget comparison with Global: when only the highest-risk 5% of questions within each AVA task were reviewed, TAG identified more injected errors (Appendix C.2, Table 21).

*Our responses to the remaining points continue in the next comment. *

W4. Practical applicability and transfer

We calibrate TAG within a known dataset and evaluate it on a separate batch. In each fold, we use four training batches with task labels and recorded answer-key flips to build task prototypes and select encoders and detectors. We freeze these rules before diagnosing the fifth batch. Base detectors can use the observed answer labels in that batch, while its true task labels and flip records are withheld from fitting and routing. Full-data preference profiles originally motivated the choice of task-specific encoders. When we also restricted design selection to the training folds, the held-out AUROC gain over Global on AVA-Bench remained 1.24 percentage points.

In exploratory tests, we matched question text to task names and descriptions. Errors in assigning questions to tasks reduced the benefit of task-specific detector selection. TAG uses task prototypes built by averaging the normalized representations of training examples for each task. Even when task names correspond across datasets, feature neighborhoods and detector-score distributions may differ. We will further explore reusing learned diagnostic rules across datasets with corresponding tasks to reduce calibration requirements.

W5. Why use 7–8B VLM judges?

We chose Qwen2.5-VL-7B and InternVL2.5-8B because both models also serve as encoders in our study. For each model, this lets us compare direct judgments of answer correctness with detection using features from the same frozen checkpoint on the same candidate answers.

Q1. How would TAG extend to open-ended generation, code, or agent trajectories?

W2 explains why we use multiple-choice QA. Extending TAG to other formats requires defining what counts as an annotation error and which unit to score. For open-ended generation, the scoring criteria must allow for multiple valid responses. For code, a test that checks a stated requirement can reveal that a program was incorrectly labeled as correct. For agent trajectories, an action label must be checked against the state and constraints at that step, since failure of the overall trajectory does not pinpoint a mislabeled action.

With verified annotations at the response, program, or action level, we could test whether selecting encoders and detectors by task improves error detection over a shared strategy. We leave this evaluation to future work.

Q2. How would TAG extend to more diverse benchmarks, and would AVA-Bench preferences transfer?

As discussed in Q1, open-ended VQA and instruction-tuning data can admit several valid responses. To evaluate annotation-error detection, we would first need a reliable way to determine when the provided reference answer is itself incorrect. A change in input modality poses a different question. For example, an audio question with a multiple-choice answer key could use the error definition in W2. The evaluation therefore depends on which annotation is being checked and how its correctness can be verified.

For transfer, we would first need to establish that the source and target tasks test the same ability. For example, if both benchmarks ask which of two objects is closer to the camera, we could apply the encoder and detectors selected for relative depth on AVA-Bench directly to the target questions. This is the cross-dataset comparison we plan to explore in W4.

Q3. Do the gains hold at 1–5% noise and on naturally occurring errors?

Low injected noise. We evaluated the selection rules learned at 10–40% noise on the original AVA-Bench, ScienceQA, and MMMU questions, using the answer-key swaps described in W2 to inject errors at 1%, 2%, and 5%. On AVA-Bench and ScienceQA, TAG outperformed Global in both AUROC and average precision (AP) at all three rates. On MMMU, AP gains were small but positive throughout, while AUROC differences were mixed.

TAG − Global, AUROC / AP (percentage points)1% noise2% noise5% noise
AVA-Bench+1.78 / +7.25+1.77 / +7.04+1.79 / +5.74
ScienceQA+0.26 / +3.48+0.72 / +4.22+0.37 / +2.40
MMMU+0.32 / +0.05−0.04 / +0.30+0.27 / +0.21

Same as the main experiments (3 corruption seeds, 5 held-out folds).

Naturally occurring errors. We identified two subsets where human-reviewed references document corrections to the original labels. CUB++ [6] provides corrected species labels for images used in AVA-Bench bird-identification questions. OCRBench v2.1 [7] provides revised answers for six of the 109 questions we matched to MMMU. These corrections let us test whether TAG flags questions whose original answer keys were wrong. For AVA-Bench texture questions, human annotations from the Describable Textures Dataset (DTD) [8] identify cases where a second answer option is also valid. LIME [9] selects challenging ScienceQA questions and retains the original keys for all 292 matched items, so it provides no corrected keys for this evaluation.

Continued.

Each method ranked the same questions using their original candidate pairs. We counted how many known cases appeared in approximately the top 10% of each ranking. To evaluate a broader set of known bird-identification errors, we retained all 2,330 questions in the initial subset and added 21 questions whose original keys were wrong. In all 21, the corrected species was absent from every original answer option.

Evaluation subsetKnown cases in subsetQuestions reviewed per methodTAGGlobalDirect judge
AVA-Bench birds, initial 2,330 questions27 incorrect keys2331158
AVA-Bench birds, expanded to 2,351 questions48 incorrect keys23615611
MMMU, 109 questions6 corrected keys11000
AVA-Bench textures, 4,487 questions10 with a second valid answer449546

On the expanded bird subset, reviewing 236 questions recovered 15 of the 48 known errors with TAG (31.3%), compared with 6 for Global (12.5%) and 11 for the direct judge (22.9%). On texture questions, TAG recovered one more known case than Global and one fewer than the direct judge. On MMMU, none of the methods flagged a known error within budget. All six revised answers fell outside the original candidate pairs.

TAG’s gains over Global thus persisted at lower noise rates on AVA-Bench and ScienceQA and extended to naturally occurring answer-key errors in both bird subsets. The gain was smaller on texture questions, and we observed no gain on the evaluated MMMU corrections.

Q4. Can a new dataset be diagnosed without target calibration?

Thank you for raising the question of transfer. Please see W4, where we clarify the calibration used in our evaluation and discuss extending TAG to a new dataset without target calibration.

Q5. Can more accurate error diagnosis improve downstream model performance or change benchmark rankings?

Thank you for suggesting these downstream evaluations. TAG supports data cleaning and reliable benchmark evaluation by prioritizing questions at high risk of annotation error for review. On AVA-Bench, reviewing the highest-risk 5% of questions within each task yielded 94.77% precision for TAG versus 93.39% for Global, averaged over the four injected noise rates of 10%, 20%, 30%, and 40% (Appendix C.2, Table 21). On the expanded bird subset in Q3, reviewing 236 questions recovered 15 known answer-key errors with TAG, compared with 6 for Global and 11 for the direct judge.

In future work, we plan to evaluate models trained on cleaned data and to rescore the same benchmark predictions using corrected answer keys to assess changes in model rankings. Thank you again for your thoughtful feedback. We would be happy to address any further questions.

References

[1] Natarajan et al. Learning with Noisy Labels. NeurIPS, 2013.

[2] Liu and Tao. Classification with Noisy Labels by Importance Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3), 447–461, 2016.

[3] Wei et al. Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ICLR, 2022.

[4] Zhu et al. Detecting Corrupted Labels Without Training a Model to Predict. ICML, 2022.

[5] Pleiss et al. Identifying Mislabeled Data using the Area Under the Margin Ranking. NeurIPS, 2020.

[6] Anderson et al. Facing the Hard Problems in FGVC. arXiv:2006.13190, 2020.

[7] GuangJian Team. PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence. arXiv:2609.37712, 2026.

[8] Cimpoi et al. Describing Textures in the Wild. CVPR, 2014.

[9] Zhu et al. LIME: Less Is More for MLLM Evaluation. Findings of ACL, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.