TAG: Task-Conditioned Adaptive Diagnosis for Multimodal Data

Diagnosing incorrect answer annotations is important for reliable training and evaluation of multimodal models. Existing approaches use direct VLM judgments or representation-based detectors, but differences in datasets, representations, and noise conditions complicate understanding when each strategy is effective. We compare these approaches under a common protocol across three multimodal benchmarks with controlled answer errors. The comparison reveals potential gains from selecting strategies for individual tasks over the best shared strategy. On AVA-Bench, encoder preferences vary across abilities but remain relatively stable across the tested noise levels, whereas detector preferences vary with both ability and noise. To demonstrate the value of these findings, we introduce Task-conditioned Adaptive Diagnosis (TAG), which identifies tasks and estimates noise conditions to select and combine diagnostic strategies, with confidence-based fallback to a shared strategy. Cross-validation shows that TAG improves AUROC over the best shared strategy by 1.24 percentage points on AVA-Bench, with smaller gains on ScienceQA and MMMU, demonstrating that task-conditioned selection can translate diagnostic preference differences into improved diagnosis.