- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper investigates the detection of incorrect answer annotations in multimodal datasets by systematically comparing different encoders, label-error detectors, and direct VLM judgments across multiple tasks and noise conditions. Based on the observation that encoder and detector preferences vary across tasks and noise levels, the authors propose Task-conditioned Adaptive Diagnosis (TAG), which selects task-specific encoders and noise-conditioned detectors with a confidence-based fallback mechanism. Experiments on AVA-Bench, ScienceQA, and MMMU demonstrate improvements over a shared diagnostic strategy, particularly on AVA-Bench.
Critical strengths
- The paper systematically evaluates eight encoders and eleven detectors across three multimodal benchmarks and multiple noise levels.
- The analysis reveals that encoder preferences are relatively stable across noise levels but differ across tasks, while detector preferences depend on both task characteristics and noise conditions.
- The ablation studies demonstrate the contributions of task-specific encoder selection and detector adaptation.
Critical weaknesses
- The proposed TAG framework mainly combines existing encoders, label-error detectors, prototype-based task identification, and confidence-based fallback through a heuristic selection mechanism.
- The methodological contribution appears relatively incremental, with limited novelty beyond task-conditioned algorithm selection.
- Although TAG achieves noticeable improvements on AVA-Bench, the gains on ScienceQA and MMMU are only 0.16 and 0.17 AUROC percentage points. Moreover, direct VLM judgment outperforms TAG on MMMU, raising concerns about the general applicability of the proposed approach.
- The experiments exclusively consider artificially injected errors through correctness-label flipping in multiple-choice datasets, which may not adequately reflect the complexity of real-world multimodal annotation errors.
- The confidence-based fallback mechanism routes approximately 69% of MMMU samples to the shared strategy and reduces the improvement compared with disabling the gate.
- The paper lacks a detailed computational cost analysis, making it difficult to assess whether the relatively modest performance gains justify the additional complexity.
Questions for authors
Please see the weaknesses.
2 comments
W4. Beyond artificial label flips
Drawing on the label-noise models of Natarajan et al. [6] and Liu and Tao [7], we compare strategies on the same errors at each noise rate. For a specified fraction of questions, we swap the correctness labels of the reference answer and one original distractor. Images, questions, and candidate texts stay fixed, with one answer marked correct per question.
Using external human annotations, we matched AVA birds to CUB++ species corrections (fgvcdata 0.1.0) [8], MMMU questions to reviewed OCRBench v2.1 answers [9, §11.1], and AVA textures to DTD attributes identifying a second valid answer [10]. LIME [5] retains the original keys for all 292 matched ScienceQA questions and provides no corrected keys for this evaluation.
All methods ranked the same original questions with the original pairs and labels, without injected errors. We froze TAG’s rules, refitted detectors within each held-out subset, and used external references only for evaluation. Entries count known cases in approximately the top 10% of each ranking. Global is the training-selected shared strategy. Judge is direct Qwen2.5-VL-7B judgment.
| Subset (questions) | Known cases | Review budget | TAG | Global | Judge |
|---|---|---|---|---|---|
| AVA birds, initial (2,330) | 27 incorrect keys | 233 | 11 | 5 | 8 |
| AVA birds, expanded (2,351) | 48 incorrect keys | 236 | 15 | 6 | 11 |
| MMMU (109) | 6 corrected keys | 11 | 0 | 0 | 0 |
| AVA textures (4,487) | 10 with a second valid answer | 449 | 5 | 4 | 6 |
The expanded bird subset adds 21 questions to the initial cohort. Their corrected species are absent from all original options. All 48 corrections lie outside the frozen pairs, so hits locate incorrect keys. On the initial subset, Judge had higher overall AUROC (0.7410 vs. 0.7371), but TAG found more errors within budget.
W5. Why fallback reduces the MMMU gain
The gate thresholds the margin between the two highest task-prototype similarities, targeting at least 95% empirical task-assignment accuracy on retained training-batch predictions. On evaluable fallback questions, forced task-conditioned scoring underperforms Global on ScienceQA but outperforms it on MMMU. On ScienceQA, routing about 16% to Global raises the observed AUROC gain from 0.11 to 0.16 percentage points. On MMMU, routing about 69% to Global reduces the gain from 0.41 points (95% CI [−0.03, 0.86]) to 0.17 (95% CI [−0.08, 0.45]). Both intervals include zero. No AVA fold triggered fallback (Table 3 and Appendix C, Tables 14, 17, 18).
Task-assignment confidence need not track diagnostic benefit. Future work could calibrate the gate on training-batch diagnostic performance.
W6. Computational cost and diagnostic gains
Timings use AVA’s held-out batch B5 (fold five, 20% noise, seed 0, 23,291 questions, 46,582 candidates), with rules frozen from the other four batches. Encoding and cached-feature diagnosis used one NVIDIA A800 80 GB PCIe GPU. We timed prototype matching separately on CPU.
| Stage | Workload | Time |
|---|---|---|
| Calibration from cached scores | 93,162 training questions | Strategy fitting 17.48 s, TAG router preparation 9.58 s, router calibration 10.54 s |
| Model loading, encoding, and saving | Same 160 B5 questions, 16 per task | Global 470.140 s, TAG 673.278 s (+43.2%) |
| CPU prototype matching | Normalized features, all 23,291 questions | 0.0472 s (median of five runs) |
| Fixed-encoder diagnosis | Full B5, cached InternVL3.5 features, 11 candidate detectors, three scores per question | Shared Top-3 16.91 s, task-specific Top-3 26.67 s |
| Complete cached-feature diagnosis | Full B5, detector fitting, scoring, routing, and aggregation | Global 10.762 s, Global-top3 16.473 s, TAG 81.918 s |
The last two rows give medians of three runs. Differences use unrounded medians.
With the encoder fixed, task-specific selection used seven detectors across B5 and took 9.75 s longer. AUROC rose from 0.9391 to 0.9473 and AP from 0.8593 to 0.8707 on the same questions. Full TAG required two additional encoders on B5. From cached features, TAG took 65.44 s longer than Global-top3 (4.97× runtime). Across five AVA folds, TAG gained 1.23 AUROC and 2.48 AP percentage points over Global-top3 with the same three-score fusion rule. Full-batch cold-start time from raw inputs was not measured.
References
[6] Natarajan et al. Learning with Noisy Labels. NeurIPS, 2013.
[7] Liu and Tao. Classification with Noisy Labels by Importance Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(3), 447–461, 2016.
[8] Anderson et al. Facing the Hard Problems in FGVC. arXiv:2006.13190, 2020.
[9] GuangJian Team. PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence. arXiv:2609.37712, 2026.
[10] Cimpoi et al. Describing Textures in the Wild. CVPR, 2014.
Thank you again for your careful review and constructive suggestions.
Thank you for recognizing our systematic comparison and the task- and noise-dependent diagnostic preferences revealed by the analysis. We clarify how these findings motivate TAG and provide additional evidence on its applicability, fallback mechanism, and computational cost.
W1. Motivation and design
Should questions testing different abilities be checked with the same representation and diagnostic signal? TAG grew out of comparing eight encoders and eleven detectors across tasks and noise rates, using identical images, questions, and candidates and evaluating all rankings against the same errors.
On all three benchmarks, post-hoc selection of the best encoder–detector pair per task outperformed the best shared pair under matched noise conditions. Encoder preferences differed across tasks but changed less often across noise rates. Detector preferences varied with both. Between adjacent rates, the preferred detector and encoder changed 13 vs. 4 times on AVA-Bench, 15 vs. 1 on ScienceQA, and 10 vs. 3 on MMMU.
These findings motivate TAG (Fig. 1). Training-batch performance determines one encoder per task across states and three detectors per task and noise state. Prototype matching estimates which task rule to apply, and an AUM probe estimates the batch noise state. TAG averages the selected detectors’ normalized ranks and falls back to a shared strategy when task assignment is uncertain.
In each fold, we learned rules on four batches and froze them before diagnosing the fifth. Detectors could use its observed answer labels. Its true task labels and flip records were withheld from fitting and routing, and the flip records served only to evaluate rankings. TAG’s held-out AUROC gains over Global were 1.24, 0.16, and 0.17 percentage points on AVA-Bench, ScienceQA, and MMMU, respectively.
W2. Contribution beyond existing adaptive selection
LSCP [1, §3.4] selects outlier detectors by agreement with a local pseudo-target. Ali et al. [2, §3.5] predict detector weights from tabular dataset characteristics. SNLC [3] studies robust VQA training under semantically related answer replacements. Our contribution is to characterize representation and detector preferences across multimodal tasks and noise conditions and test whether learning them improves answer-annotation diagnosis in an unseen mixed-task batch.
On all three full datasets, task-specific gains over the best shared pair exceeded the 95th percentile of gains from 500 association-group task-label permutations (Table 1). On held-out AVA batches, with encoder and fusion fixed, task- and noise-conditioned detector selection gained 0.54 AUROC percentage points over Global-top3 (bootstrap mean). Task-specific encoders with reselected detectors added 0.69 points (Appendix C, Table 14).
W3. Small gains on ScienceQA and MMMU
On ScienceQA, Global already reaches 0.9749 AUROC, above direct Qwen2.5-VL-7B judgment at 0.9213. TAG reaches 0.9765, a gain of 0.16 percentage points (paired-bootstrap 95% CI [0.03, 0.35]). On MMMU, direct judgment leads TAG (0.7093 vs. 0.6788), and TAG’s 0.17-point gain over Global remains uncertain (95% CI [−0.08, 0.45]; Appendix C, Tables 13–15).
Beyond AVA, we tested each parent dataset’s frozen held-out rules on 1,152 original MMMU answer pairs aligned to MMMU-Pro [4] and 292 ScienceQA questions aligned to LIME [5]. Detectors were refitted within each subset at 10%–40% nominal injected noise.
Methods use the same tasks containing both flipped and unflipped questions within each condition. Across rates, mean question coverage over folds and seeds is 97.2%–100% for MMMU-Pro and 57.7%–90.9% for LIME.
TAG also exceeds Global-top3 in mean task-weighted AUROC on both subsets at all four rates. This supports gains over shared diagnosis beyond AVA with same-dataset calibration. Direct judgment leads in our main MMMU evaluation and exceeds TAG in both AUROC and AP on LIME at 30% and 40% noise. We plan to study task-conditioned selection with direct judgment in the candidate pool, considering diagnostic performance and computational budget.
References
[1] Zhao et al. LSCP: Locally Selective Combination in Parallel Outlier Ensembles. arXiv:1812.01528, 2018.
[2] Ali et al. Adaptive Ensemble Selection for Noisy Labels on Tabular Data. arXiv:2609.32976, 2026.
[3] Zhang et al. Learning Visual Question Answering on Controlled Semantic Noisy Labels. Pattern Recognition, 138, 109339, 2023.
[4] Yue et al. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL, 2025.
[5] Zhu et al. LIME: Less Is More for MLLM Evaluation. Findings of ACL, 2025.