acceptodds
Under review as a conference paper at ICLR 2027

When Aggregate Accuracy Does Not Identify Functional Change: Sharp Bounds on AV-LLM Adaptation Effects

Abstract

Fine-tuning Audio-Visual Large Language Models (AV-LLMs) is commonly evaluated by in-domain and cross-domain joint accuracy, measured with both audio and video available. Yet the same reported accuracies can conceal different functional changes. Even audio-only, video-only, and joint accuracies omit whether the two isolated inputs succeed on the same questions. We define isolated-path coverage as the fraction of questions answered correctly by at least one single-modality input. Conversion is joint accuracy minus that coverage. From before-and-after aggregate accuracies, we derive a sharp identified set for the conversion change and establish exactly when its sign is identifiable. The joint-minus-video gap combines conversion with the fraction of questions answered correctly with audio alone but not video alone. Its change can therefore have the opposite sign to the conversion change. Our audit spans 56 checkpoint–dataset evaluations and 40 before-and-after fine-tuning comparisons across Qwen2.5-Omni, Qwen3-Omni, MiniCPM-o, and VITA. Aggregate reports identify none of the 40 conversion-change signs, while the conventional gap points in the opposite direction in 17 comparisons. Paired per-question evaluations recover the missing overlap. They reveal a cross-domain conversion reversal under the same adaptation and a coverage gain offset by a conversion loss behind a nearly unchanged joint score. The balance between coverage and conversion varies across adaptation datasets and model families. AV-LLM adaptation calls for paired functional profiles alongside joint accuracy to establish what changed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
audio-visual large language models, multimodal adaptation, partial identification, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_c426OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what aggregate accuracies reveal about AV-model adaptation. Coverage measures correctness in at least one unimodal run. Conversion is joint accuracy minus coverage. Overlap bounds identify possible conversion changes.

Critical strengths
  1. Classical probability bounds become a practical reporting rule. One additional overlap-change statistic resolves the ambiguity.
Critical weaknesses
  1. Coverage and the bounds build on established ideas, limiting theoretical novelty.
  2. How this analysis can guide practical training or inference to improve performance remains unclear.
Questions for authors
  1. When some questions in the test set require only one modality to answer, is conversion still useful for understanding the model ability?

  2. How can this analysis guide training or inference to improve performance? Can you demonstrate gains under matched compute budgets on an independent test set?

Reviewer_TQ9NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The article addresses an issue concerning the use of aggregated accuracy metrics for evaluating audiovisual language models: such aggregated statistics fail to fully reveal the extent to which the fusion of the two modalities contributes to the final result. The authors propose augmenting evaluation reports with isolated-path coverage, conversion, and paired evaluation, demonstrating that this provides a much better assessment of that contribution. The evaluation was performed on various models like Qwen2.5-Omni, Qwen3-Omni, MiniCPM-o, and VITA.

Critical strengths
  • The article identifies a significant research issue present in many publications concerning the evaluation of audio-video models: the fact that aggregated metric results - such as audio-only accuracy, video-only accuracy, and joint accuracy - often fail to provide a complete picture of which specific components of a model’s architecture contribute to its performance.
  • The analysis of paired predictions proposed by the authors decomposes adaptation effects into "coverage" and "conversion." This allows system engineers to distinguish between a genuine improvement in the model’s knowledge and shortcomings within the signal fusion module.
Critical weaknesses
  • From a purely theoretical standpoint, the novelty of the proposed solution is somewhat limited. The theoretical basis for the solutions presented is, to a large extent, a formalized analysis of the overlap of prediction errors, framed within statistical bounds, rather than an entirely new mathematical technique. Nevertheless, I consider the practical and diagnostic value of the publication sufficient for acceptance.
Questions for authors

The authors did not provide a separate discussion of limitations. In the authors' view, what are the limitations of the method they presented? What are the limitations concerning the evaluation conducted and, consequently, the conclusions drawn from its results?

1 comment

Thank you for your responses to my questions.

Reviewer_tJLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how we evaluate fine-tuned Audio-Visual LLMs. Currently, papers report separate average accuracies for audio-only (AA), video-only (VV), and joint inputs (AVAV). The authors show that these aggregate averages conceal critical behavior because they omit whether audio and video succeed on the same questions. By decomposing accuracy into Coverage (questions solvable by at least one modality alone) and Conversion (net gain from true joint synergy), they prove mathematically and empirically across 40 fine-tuning comparisons (Qwen2.5/3-Omni, MiniCPM-o, VITA) that conventional metrics like AV−VAV - V frequently give the wrong conclusion. They recommend saving per-question correctness logs to point-identify real functional change.

Critical strengths
  1. Demonstrates clearly why aggregate accuracy hides whether gains come from single-modality shortcuts or true cross-modal fusion.

  2. Thorough Empirical Audit: Evaluates 40 fine-tuning runs across multiple architectures and benchmarks, showing that standard metrics reverse sign in 17 out of 40 cases.

Critical weaknesses
  1. The paper presentation is poor to deliver the most concise information, which makes it hard for me to understand the intuition for this study.

  2. I understand you think fine-tuning does not really improve audio-video alignment, but what does it mean if you evaluate the model on "video-only" and "audio-only" in this AV understanding QA task? Avoiding shortcuts is the main claim this paper should make, which means with only one modality, the model should not deliver meaningful understanding (and the most intelligent model would refuse to answer, which yields low accuracy but is better).

  3. Section 3 is full of formulas; there should be a better way, such as adding some demos, to show how you design the evaluation.

Questions for authors
  1. May the authors better explain the intuition of the work? When the model is evaluated on a single modality, does the evaluation make sense?

  2. Consider rewriting Section 3; currently, it is full of discrete concepts and notations, which is not direct for an evaluation process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.