- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Functional Interaction Profile (FIP), a diagnostic framework for characterizing how multimodal large language models (MLLMs) use and combine modality-specific evidence. The paper argues that aggregate accuracy alone is insufficient to distinguish whether a model can access task-relevant modality information, whether the effect of a modality depends on the question being asked, and whether the model responds to semantic changes rather than nuisance variations. To address this, the authors develop a functional view of multimodal interaction based on controlled four-cell contrasts and introduce three central properties—accessibility, conditionality, and selectivity—together with four additional diagnostic coordinates, forming a seven-coordinate FIP.
The theoretical analysis formalizes conditionality as a response-surface interaction term and relates it to local response geometry and decision failures. The authors further establish that accessibility, conditionality, and selectivity capture distinct aspects of multimodal behavior and therefore should not be collapsed into a single scalar metric. The resulting FIP is intended to complement conventional accuracy by providing a more fine-grained characterization of modality use and failure modes.
For validation, the paper constructs controlled four-cell evaluation families spanning 22 semantic dimensions and four modalities, including images, video, environmental audio, and speech. The experiments evaluate LLaVA-OneVision-7B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, covering 5,012 evaluations over 4,000 distinct families. The results show that conditionality generally becomes measurable relatively late in the model, while modality has a stronger effect on accessibility and conditionality than model identity, with models differing more substantially in selectivity. The analysis also distinguishes decision failures from missing-evidence failures and identifies substantial weaknesses in temporal reasoning for video and audio despite successful performance on corresponding text descriptions. Additional experiments with Qwen3-Omni-30B-A3B suggest that conditionality can improve substantially even when four-cell task accuracy remains very low, illustrating the diagnostic value of reporting interaction properties separately from accuracy.
Critical strengths
-
The validity and generality of FIP are strongly dependent on the controlled evaluation construction.
The proposed coordinates are defined through declared observables, semantic transformations, nuisance transformations, and a particular four-cell scoring interface. This controlled design is useful for isolating specific behavioral properties, but it also means that the resulting FIP values characterize model behavior relative to the chosen operationalization rather than providing a generally benchmark-independent measure of multimodal interaction. The paper provides theoretical arguments for why the coordinates are distinct, but there is comparatively less evidence that the same conclusions and rankings remain stable under substantially different task constructions, semantic transformations, or evaluation interfaces. This limits the strength of the paper's broader claims about multimodal model behavior.
-
The empirical evidence is not yet sufficient to support the broadest conclusions about multimodal understanding.
The main analysis is based on three models and a controlled suite of 22 semantic dimensions, with most conclusions drawn from the resulting model–modality slices. While the experiments are extensive within this setting, the limited model panel and highly controlled benchmark make it difficult to determine whether the observed patterns reflect general properties of MLLMs or properties of the selected models, modalities, and task construction. In particular, the distinction between missing evidence, decision failures, and genuine modality-specific reasoning limitations is compelling as a diagnostic interpretation, but stronger claims about the underlying capabilities of multimodal models would benefit from validation on more diverse architectures, scales, datasets, and naturally occurring multimodal tasks.
Critical weaknesses
-
The validity and generality of FIP are strongly dependent on the controlled evaluation construction.
The proposed coordinates are defined through declared observables, semantic transformations, nuisance transformations, and a particular four-cell scoring interface. This controlled design is useful for isolating specific behavioral properties, but it also means that the resulting FIP values characterize model behavior relative to the chosen operationalization rather than providing a generally benchmark-independent measure of multimodal interaction. The paper provides theoretical arguments for why the coordinates are distinct, but there is comparatively less evidence that the same conclusions and rankings remain stable under substantially different task constructions, semantic transformations, or evaluation interfaces. This limits the strength of the paper's broader claims about multimodal model behavior.
-
The empirical evidence is not yet sufficient to support the broadest conclusions about multimodal understanding.
The main analysis is based on three models and a controlled suite of 22 semantic dimensions, with most conclusions drawn from the resulting model–modality slices. While the experiments are extensive within this setting, the limited model panel and highly controlled benchmark make it difficult to determine whether the observed patterns reflect general properties of MLLMs or properties of the selected models, modalities, and task construction. In particular, the distinction between missing evidence, decision failures, and genuine modality-specific reasoning limitations is compelling as a diagnostic interpretation, but stronger claims about the underlying capabilities of multimodal models would benefit from validation on more diverse architectures, scales, datasets, and naturally occurring multimodal tasks.
Questions for authors
-
Clarification on robustness to the evaluation construction.
How sensitive are the reported FIP coordinates and the resulting cross-model/modality conclusions to the specific choice of semantic transformations, nuisance transformations, declared observables, and four-cell task construction? In particular, do the main qualitative conclusions remain consistent under alternative valid transformations or task families? Evidence addressing this would materially affect my assessment of whether FIP captures a robust property of multimodal interaction rather than behavior specific to the proposed benchmark.
-
Clarification on the scope and generality of the empirical claims.
To what extent do the authors expect the reported relationships among accessibility, conditionality, selectivity, and failure modes to generalize beyond the three evaluated models and the controlled suite of 22 semantic dimensions? Are there results on additional model families, model scales, or naturally occurring multimodal tasks that support this generalization? This would materially affect how broadly I interpret the empirical conclusions of the paper.
4 comments
Rebuttal by Authors (2/2)
Q2. Do the relationships among the coordinates and the failure modes generalize beyond the three models and the 22 controlled dimensions?
Yes, in two different ways. Some relationships follow from the theory and hold for any model. Others are empirical, and the new models and the natural benchmarks test whether they persist.
Relationships from the theory. Theorem 1 states that success with margin requires . Proposition 3 states that a constant offset repairs a family exactly when its four scores are threshold-feasible. Proposition 4 states that none of the three properties follows from the others. These hold for every model and every four-cell family. The data follow them as well. Across the panel, only one of 1,132 evaluations with is solved (§6.3). The held-out Qwen3-Omni, never used to design the suite, follows the same floor, with none of its 53 evaluations with solved and all 179 of its successes at .
New model families and scales. We evaluated four models outside the panel on the temporal properties.
| Model | Language model | Properties tested | Result |
|---|---|---|---|
| Qwen3-Omni-30B-A3B | Qwen3, 4.3× larger | All five, in video and audio | 4/192 solved, while the live controls stay solved |
| Gemma 4 12B | Gemma, non-Qwen | Speed, duration, repetition and rotation in video | 1/128 solved, while the live control solves 32/32 and the text description 128/128 |
| Phi-4-multimodal | Phi-4 | Audio properties | Fails the audio live controls, so its zeros are uninformative (App. I.13) |
| LFM3-Audio-2B | LFM | Repetition and simultaneity in audio | Fails the audio live controls, so it is reported like Phi-4 |
As seen in the table, every model that passes its live controls shows the same temporal failure, across the Qwen and Gemma families and sizes from 7B to 30B. A zero counts as evidence only when the model solves the live controls on the same media, a rule we fixed before running the new models. Phi-4 and LFM3-Audio fail these controls, so their zeros are reported as uninformative.
Naturally occurring tasks. In the paper, we apply the same response-level comparison to six natural benchmarks, with 36,000 model–example evaluations (App. I.11). There, too, accuracy hides changes in evidence use. On Clotho-AQA, replacing the audio with mean tokens leaves Qwen's accuracy at 31%, yet restoring the audio changes 926 of 2,000 answers.
Empirical relationships of the panel. Correlations among coordinates, such as between accessibility and conditionality, are properties of this panel and are reported as such. The release recommendation rests on the coordinates themselves, not on these correlations. We propose that releases report accessibility, signed conditionality and selectivity, together with four-cell success and the decision-failure share.
The theoretical relationships hold for every four-cell family, and the held-out model follows them. The temporal failure persists in every new model that passes its live controls, including a non-Qwen family and a larger scale. The natural benchmarks show that accuracy can hide changes in how the modality is used. We will add Gemma 4 and LFM3-Audio to App. I.13, and state in §6 which relationships are theoretical and which are empirical.
We will keep all additions to the main text short and place the detailed results in the appendix, so the paper stays within the page limit and retains all its current content.
We hope we have addressed the reviewer's concerns and would be happy to provide any additional details as needed.
Follow-Up on Our Rebuttal Responses
Dear Reviewer kctQ,
We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.
Best regards,
The Authors
Follow-Up on Our Rebuttal Responses
Dear Reviewer kctQ,
The discussion period closes soon. You wrote that evidence on robustness to the evaluation construction and on generality would materially affect your assessment. We provided both:
- Robustness (Q1). We tested eight alternative constructions: paraphrased questions, reversed answer order, a probability margin, other answer candidates, an A/B response format, the U1 nuisance reference, unseen recording conditions, and 16 and 32 frames. Each leaves the corresponding finding in place.
- Generality (Q2). Theorem 1 and Propositions 3–4 hold for any four-cell family, and the held-out Qwen3-Omni follows the floor (none of its 53 evaluations with is solved). The temporal failure persists in Gemma 4 12B, a non-Qwen model (1/128, while text solves 128/128), and in Qwen3-Omni-30B (4/192). On six natural benchmarks with 36,000 evaluations, accuracy hides changes in how the modality is used.
We would be glad to hear your thoughts on these results before the discussion closes.
Best regards,
The Authors
Rebuttal by Authors (1/2)
Thank you for stating that the paper is clear and informative. The two weaknesses correspond to the two questions, so we answer them together.
Q1. How sensitive are the FIP coordinates and the conclusions to the semantic and nuisance transformations, the declared observables and the four-cell construction?
The coordinates are measured with declared transformations, as in any controlled measurement, and a release must state them (App. G.5). Their values can therefore change when these transformations change. We tested whether the main findings persist when these choices change. We varied the wording, the answer interface, the observable, the nuisance edits and the media sampling, one at a time.
"no" or "yes"instead of"yes" or "no"in the instruction"Yes"/"No"candidates and greedy decoding, on 500 LLaVA image and 96 Qwen2.5 temporal familiesAs seen in the table, each change moves some values but leaves the conclusion in place. The candidate change is the clearest case. With
"Yes"/"No", LLaVA's four-cell success rises from 18 to 368 of 500 families, while Γ stays positive and threshold feasibility barely changes. The evidence was therefore already in the scores, and the new candidates mainly moved the decision, as the decision-failure diagnosis predicts.Each row tests one finding of the paper, and each finding persists under the alternatives tested for it. These findings are the sign and ranking of Γ, the split between decision failures and missing evidence, the selectivity ordering and the temporal zeros. We will add this table to App. I.5.