acceptodds
Under review as a conference paper at ICLR 2027

Quantifying Cross-Modal Interaction in Multimodal Large Language Models

Abstract

Multimodal large language models map images, video and audio into a joint representation processed alongside text. Yet a modality affecting the output does not establish that the model uses it as the instruction requires. We therefore investigate when modality information is \em functionally usable by a language model. To this end, we introduce a functional geometry that compares inputs by the model's response rather than their representation, making it invariant to how the representation is parameterized. Within this geometry, we define the Functional Interaction Profile (FIP), which measures (i) whether task-relevant evidence influence the response, (ii) whether its effect depends on the question, and (iii) whether meaning impacts it more than nuisance. Experiments on three open models across four modalities confirm the profile's predicted invariances. Applied to these models, the profile reveals which evidence does not reach the response: no evaluated model perceives how events unfold in time, although the same questions are solved from a text description. A larger model still fails these questions, but its profile shows this evidence beginning to interact with the question before its accuracy changes. Accuracy signals whether a model fails, whereas the profile shows \em why it fails and \em whether it is improving. We thus propose that MLLM releases report the profile alongside accuracy, and release the controlled suite and evaluation harness to compute it. By shifting the evaluation from static outcomes to dynamic behavior, FIP can support the development of more transparent and robust multimodal models.

Primary area:
interpretability and explainable AI
Keywords:
Multimodal large language models, Cross-modal interaction, Functional geometry.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kctQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Functional Interaction Profile (FIP), a diagnostic framework for characterizing how multimodal large language models (MLLMs) use and combine modality-specific evidence. The paper argues that aggregate accuracy alone is insufficient to distinguish whether a model can access task-relevant modality information, whether the effect of a modality depends on the question being asked, and whether the model responds to semantic changes rather than nuisance variations. To address this, the authors develop a functional view of multimodal interaction based on controlled four-cell contrasts and introduce three central properties—accessibility, conditionality, and selectivity—together with four additional diagnostic coordinates, forming a seven-coordinate FIP.

The theoretical analysis formalizes conditionality as a response-surface interaction term and relates it to local response geometry and decision failures. The authors further establish that accessibility, conditionality, and selectivity capture distinct aspects of multimodal behavior and therefore should not be collapsed into a single scalar metric. The resulting FIP is intended to complement conventional accuracy by providing a more fine-grained characterization of modality use and failure modes.

For validation, the paper constructs controlled four-cell evaluation families spanning 22 semantic dimensions and four modalities, including images, video, environmental audio, and speech. The experiments evaluate LLaVA-OneVision-7B, Qwen2.5-Omni-7B, and MiniCPM-o 4.5, covering 5,012 evaluations over 4,000 distinct families. The results show that conditionality generally becomes measurable relatively late in the model, while modality has a stronger effect on accessibility and conditionality than model identity, with models differing more substantially in selectivity. The analysis also distinguishes decision failures from missing-evidence failures and identifies substantial weaknesses in temporal reasoning for video and audio despite successful performance on corresponding text descriptions. Additional experiments with Qwen3-Omni-30B-A3B suggest that conditionality can improve substantially even when four-cell task accuracy remains very low, illustrating the diagnostic value of reporting interaction properties separately from accuracy.

Critical strengths
  1. The validity and generality of FIP are strongly dependent on the controlled evaluation construction.

    The proposed coordinates are defined through declared observables, semantic transformations, nuisance transformations, and a particular four-cell scoring interface. This controlled design is useful for isolating specific behavioral properties, but it also means that the resulting FIP values characterize model behavior relative to the chosen operationalization rather than providing a generally benchmark-independent measure of multimodal interaction. The paper provides theoretical arguments for why the coordinates are distinct, but there is comparatively less evidence that the same conclusions and rankings remain stable under substantially different task constructions, semantic transformations, or evaluation interfaces. This limits the strength of the paper's broader claims about multimodal model behavior.

  2. The empirical evidence is not yet sufficient to support the broadest conclusions about multimodal understanding.

    The main analysis is based on three models and a controlled suite of 22 semantic dimensions, with most conclusions drawn from the resulting model–modality slices. While the experiments are extensive within this setting, the limited model panel and highly controlled benchmark make it difficult to determine whether the observed patterns reflect general properties of MLLMs or properties of the selected models, modalities, and task construction. In particular, the distinction between missing evidence, decision failures, and genuine modality-specific reasoning limitations is compelling as a diagnostic interpretation, but stronger claims about the underlying capabilities of multimodal models would benefit from validation on more diverse architectures, scales, datasets, and naturally occurring multimodal tasks.

Critical weaknesses
  1. The validity and generality of FIP are strongly dependent on the controlled evaluation construction.

    The proposed coordinates are defined through declared observables, semantic transformations, nuisance transformations, and a particular four-cell scoring interface. This controlled design is useful for isolating specific behavioral properties, but it also means that the resulting FIP values characterize model behavior relative to the chosen operationalization rather than providing a generally benchmark-independent measure of multimodal interaction. The paper provides theoretical arguments for why the coordinates are distinct, but there is comparatively less evidence that the same conclusions and rankings remain stable under substantially different task constructions, semantic transformations, or evaluation interfaces. This limits the strength of the paper's broader claims about multimodal model behavior.

  2. The empirical evidence is not yet sufficient to support the broadest conclusions about multimodal understanding.

    The main analysis is based on three models and a controlled suite of 22 semantic dimensions, with most conclusions drawn from the resulting model–modality slices. While the experiments are extensive within this setting, the limited model panel and highly controlled benchmark make it difficult to determine whether the observed patterns reflect general properties of MLLMs or properties of the selected models, modalities, and task construction. In particular, the distinction between missing evidence, decision failures, and genuine modality-specific reasoning limitations is compelling as a diagnostic interpretation, but stronger claims about the underlying capabilities of multimodal models would benefit from validation on more diverse architectures, scales, datasets, and naturally occurring multimodal tasks.

Questions for authors
  1. Clarification on robustness to the evaluation construction.

    How sensitive are the reported FIP coordinates and the resulting cross-model/modality conclusions to the specific choice of semantic transformations, nuisance transformations, declared observables, and four-cell task construction? In particular, do the main qualitative conclusions remain consistent under alternative valid transformations or task families? Evidence addressing this would materially affect my assessment of whether FIP captures a robust property of multimodal interaction rather than behavior specific to the proposed benchmark.

  2. Clarification on the scope and generality of the empirical claims.

    To what extent do the authors expect the reported relationships among accessibility, conditionality, selectivity, and failure modes to generalize beyond the three evaluated models and the controlled suite of 22 semantic dimensions? Are there results on additional model families, model scales, or naturally occurring multimodal tasks that support this generalization? This would materially affect how broadly I interpret the empirical conclusions of the paper.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Thank you for stating that the paper is clear and informative. The two weaknesses correspond to the two questions, so we answer them together.

Q1. How sensitive are the FIP coordinates and the conclusions to the semantic and nuisance transformations, the declared observables and the four-cell construction?

The coordinates are measured with declared transformations, as in any controlled measurement, and a release must state them (App. G.5). Their values can therefore change when these transformations change. We tested whether the main findings persist when these choices change. We varied the wording, the answer interface, the observable, the nuisance edits and the media sampling, one at a time.

ElementChange testedWhat stays the same
Question wordingIndependent paraphrases, 784 source–paraphrase pairsThe ranking of families by signed Γ\Gamma (rank correlation 0.99, lowest in video at 0.81)
Answer order"no" or "yes" instead of "yes" or "no" in the instructionThe sign of Γ\Gamma in every family
Declared observableProbability margin p(yes) − p(no) instead of the log-score margin, on all 5,012 evaluationsFour-cell success and threshold feasibility, identical by construction, and the sign of Γ\Gamma in 97.7% of evaluations (rank correlation 0.87)
Answer candidates"Yes"/"No" candidates and greedy decoding, on 500 LLaVA image and 96 Qwen2.5 temporal familiesΓ>0\Gamma > 0 and threshold feasibility (LLaVA 495 and 425 vs. 494 and 422 of 500), and the temporal zeros (0/72)
Four-cell constructionPairwise A/B choice between the two questions instead of yes/no, on 200 blurred image families per modelDecision failures are recovered more often than missing evidence, strongly for LLaVA (87% vs. 24%) and less for Qwen2.5 (19% vs. 8%)
Nuisance referenceU1 instead of U0, at floors from 10⁻⁸ to 10⁻⁴The more selective model in every modality
Unseen nuisanceFour new recording conditions for speech (telephone, pitch, babble)Selectivity predicts which model keeps its answers (92% vs. 46%)
Temporal sampling8, 16 and 32 frames on 160 video familiesZero success on the temporal properties, with the live control solved

As seen in the table, each change moves some values but leaves the conclusion in place. The candidate change is the clearest case. With "Yes"/"No", LLaVA's four-cell success rises from 18 to 368 of 500 families, while Γ\Gamma stays positive and threshold feasibility barely changes. The evidence was therefore already in the scores, and the new candidates mainly moved the decision, as the decision-failure diagnosis predicts.

Each row tests one finding of the paper, and each finding persists under the alternatives tested for it. These findings are the sign and ranking of Γ\Gamma, the split between decision failures and missing evidence, the selectivity ordering and the temporal zeros. We will add this table to App. I.5.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q2. Do the relationships among the coordinates and the failure modes generalize beyond the three models and the 22 controlled dimensions?

Yes, in two different ways. Some relationships follow from the theory and hold for any model. Others are empirical, and the new models and the natural benchmarks test whether they persist.

Relationships from the theory. Theorem 1 states that success with margin mm requires Γ≥4m\Gamma \ge 4m. Proposition 3 states that a constant offset repairs a family exactly when its four scores are threshold-feasible. Proposition 4 states that none of the three properties follows from the others. These hold for every model and every four-cell family. The data follow them as well. Across the panel, only one of 1,132 evaluations with ∣Γ∣<1|\Gamma| < 1 is solved (§6.3). The held-out Qwen3-Omni, never used to design the suite, follows the same floor, with none of its 53 evaluations with ∣Γ∣<1|\Gamma| < 1 solved and all 179 of its successes at ∣Γ∣≥11.9|\Gamma| \ge 11.9.

New model families and scales. We evaluated four models outside the panel on the temporal properties.

ModelLanguage modelProperties testedResult
Qwen3-Omni-30B-A3BQwen3, 4.3× largerAll five, in video and audio4/192 solved, while the live controls stay solved
Gemma 4 12BGemma, non-QwenSpeed, duration, repetition and rotation in video1/128 solved, while the live control solves 32/32 and the text description 128/128
Phi-4-multimodalPhi-4Audio propertiesFails the audio live controls, so its zeros are uninformative (App. I.13)
LFM3-Audio-2BLFMRepetition and simultaneity in audioFails the audio live controls, so it is reported like Phi-4

As seen in the table, every model that passes its live controls shows the same temporal failure, across the Qwen and Gemma families and sizes from 7B to 30B. A zero counts as evidence only when the model solves the live controls on the same media, a rule we fixed before running the new models. Phi-4 and LFM3-Audio fail these controls, so their zeros are reported as uninformative.

Naturally occurring tasks. In the paper, we apply the same response-level comparison to six natural benchmarks, with 36,000 model–example evaluations (App. I.11). There, too, accuracy hides changes in evidence use. On Clotho-AQA, replacing the audio with mean tokens leaves Qwen's accuracy at 31%, yet restoring the audio changes 926 of 2,000 answers.

Empirical relationships of the panel. Correlations among coordinates, such as ρ=0.95\rho = 0.95 between accessibility and conditionality, are properties of this panel and are reported as such. The release recommendation rests on the coordinates themselves, not on these correlations. We propose that releases report accessibility, signed conditionality and selectivity, together with four-cell success and the decision-failure share.

The theoretical relationships hold for every four-cell family, and the held-out model follows them. The temporal failure persists in every new model that passes its live controls, including a non-Qwen family and a larger scale. The natural benchmarks show that accuracy can hide changes in how the modality is used. We will add Gemma 4 and LFM3-Audio to App. I.13, and state in §6 which relationships are theoretical and which are empirical.

We will keep all additions to the main text short and place the detailed results in the appendix, so the paper stays within the page limit and retains all its current content.

We hope we have addressed the reviewer's concerns and would be happy to provide any additional details as needed.

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer kctQ,

We appreciate your feedback on our submission. In response to your comments, we have presented new experimental results and provided additional clarifications. As the discussion period is coming to an end, we would appreciate knowing whether our responses have addressed your concerns.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer kctQ,

The discussion period closes soon. You wrote that evidence on robustness to the evaluation construction and on generality would materially affect your assessment. We provided both:

  • Robustness (Q1). We tested eight alternative constructions: paraphrased questions, reversed answer order, a probability margin, other answer candidates, an A/B response format, the U1 nuisance reference, unseen recording conditions, and 16 and 32 frames. Each leaves the corresponding finding in place.
  • Generality (Q2). Theorem 1 and Propositions 3–4 hold for any four-cell family, and the held-out Qwen3-Omni follows the Γ\Gamma floor (none of its 53 evaluations with ∣Γ∣<1|\Gamma| < 1 is solved). The temporal failure persists in Gemma 4 12B, a non-Qwen model (1/128, while text solves 128/128), and in Qwen3-Omni-30B (4/192). On six natural benchmarks with 36,000 evaluations, accuracy hides changes in how the modality is used.

We would be glad to hear your thoughts on these results before the discussion closes.

Best regards,

The Authors

Reviewer_9aWYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an MLLM uses the modality input in the way the question requires. Each "family" crosses two modality states with two yes/no questions, so only the combination determines the answer. The core quantity is the 2×2 interaction contrast Γ = s11 − s10 − s01 + s00, computed on teacher-forced yes-minus-no scores.

The authors place Γ inside a "functional geometry", a pullback pseudodistance that is invariant to reparameterizing the representation, and prove several properties:

  • Γ is zero for additive responses and unaffected by additive biases.
  • Solving all four cells with margin m requires Γ ≥ 4m.
  • A single threshold solves a family iff min(s00, s11) > max(s01, s10). This splits failures into decision failures, which a constant offset fixes, and response failures, which it cannot.

Γ is combined with six other measures (accessibility, selectivity, text-surrogate gap, mediation, onset, execution variability) into a seven-coordinate Functional Interaction Profile (FIP).

Validation. LLaVA-OneVision-7B is tested on image, and Qwen2.5-Omni-7B and MiniCPM-o 4.5 on image, video, The paper asks whether an MLLM uses the modality input in the way the question requires. Each "family" crosses two modality states with two yes/no questions, so only the combination determines the answer. The core quantity is the 2×2 interaction contrast Γ = s11 − s10 − s01 + s00, computed on teacher-forced yes-minus-no scores.

The authors place Γ inside a "functional geometry", a pullback pseudodistance that is invariant to reparameterizing the representation, and prove several properties:

  • Γ is zero for additive responses and unaffected by additive biases.
  • Solving all four cells with margin m requires Γ ≥ 4m.
  • A single threshold solves a family iff min(s00, s11) > max(s01, s10). This splits failures into decision failures, which a constant offset fixes, and response failures, which it cannot.

Γ is combined with six other measures (accessibility, selectivity, text-surrogate gap, mediation, onset, execution variability) into a seven-coordinate Functional Interaction Profile (FIP).

Validation. LLaVA-OneVision-7B is tested on image, and Qwen2.5-Omni-7B and MiniCPM-o 4.5 on image, video, audio and speech. The suite is synthetic: 9 model–modality slices, 22 dimensions, 5,012 evaluations. Qwen3-Omni-30B-A3B and Phi-4-multimodal are held out. Main findings:

  • Under a logit-lens readout, interaction becomes readable at 74–81% of depth.
  • Most image failures are decision failures. A shared offset takes LLaVA from 14/400 to 266/400 with Γ unchanged. For Qwen video, the same offset rescues 0 of 334 failures.
  • Five temporal dimensions are solved in 0 of 1,268 evaluations. Text descriptions of the same scenes are solved 192/192.
  • Qwen3 stays at 0–4.7% success on these five, but its |Γ| is higher on three of them.

The paper proposes that MLLM releases report FIP alongside accuracy.

Critical strengths
  1. The decision-failure / response-failure split is useful and cheap. It needs only the four scores, and the paper shows it matters. 40% of the 3,094 failures can be fixed by a per-family threshold, ranging from 84% (LLaVA image) to 17% (Qwen video). Accuracy is zero for both kinds of failure, so accuracy alone cannot tell them apart. This is the part of the paper I expect others to use.
  2. The experiments are controlled and reported honestly. Resampling is done per family. Text surrogates and live controls rule out unanswerable questions. The authors also report results that cut against them: the failed pre-registered |Γ| < 2 prediction on Qwen3, the PID estimator's instability across seeds, and the lack of any AUROC gain. I trust the numbers even where I disagree with the conclusions drawn from them.
Critical weaknesses

Main weakness: the framework is much heavier than what the results use, and the central claim is not shown. That claim is that FIP reveals what accuracy does not, so releases should report it.

  • What actually does the work is simple. It is Γ, which the authors say is the classical factorial interaction contrast and not new (Sec. 2), plus a threshold-feasibility check. The supporting results are short algebra:
    • Prop. 2 is linearity.
    • Theorem 1 sums four margin inequalities.
    • Prop. 3 compares a min with a max.
  • The functional geometry does no work in the experiments.
    • The local pullback metric ranks the semantic edit above the nuisance edit in only 53/136 families (39%), and the paper uses this to argue against gradient probes.
    • Theorem 2 links Γ to the local mixed derivative. But path quadrature reproduces Γ within 10% in only 3 of 112 rows, and base-point derivatives can have the opposite sign to Γ (Fig. 5).
    • The invariance "validation" (Γ changes by under 1e-15 after an offset) is linearity checked in floating point.
    • The 3,118× representation comparison uses transforms that are not compensated downstream, so it does not test Prop. 1 (App. I.3 says as much).
  • The other coordinates are redundant or hard to interpret.
    • Accessibility and conditionality have a rank correlation of 0.95 across the nine slices.
    • C_med is unbounded (≈2,720 for Qwen video) and depends on which corruption is used.
    • Onset is 74–81% under the peak/4 rule. Under the floor-only rule it is 22–59%, and the modalities fall into an order (Tab. 15).
    • V_exec is zero everywhere.
  • The two "beyond accuracy" claims in the abstract have thin support.
    • "Why it fails" is the threshold check above, which does not need FIP.
    • "Whether it is improving" rests on one comparison: Qwen2.5 vs. Qwen3 on three dimensions, with n = 32/64/32. It has four problems:
      • It compares raw |Γ| across models without S_f, although the paper's own normalization flips 8 of 36 interaction orderings.
      • It does not report the sign of Γ.
      • It does not report threshold feasibility, which is the direct test of "structure the decision rule does not yet use."
      • The result was a failed pre-registered prediction, reinterpreted afterwards.
    • On prediction: once source success is controlled for, the high- vs. low-|Γ| gap falls to 0.064 [0.000, 0.160], and AUROC changes by 0.003 (App. I.4).

This matters because the paper's recommendation is that releases report a seven-coordinate profile. No case in the paper shows FIP reaching a different, checkable conclusion than four-cell accuracy plus the threshold check.

Secondary weaknesses

  • External validity of the headline temporal claim.
    • All stimuli are synthetic: 8-frame 256×256 2D shapes, CLEVR renders, ESC-50 mixtures.
    • Every model that passes the live controls has a Qwen-family LM; Phi-4 fails them.
    • The text surrogates are written by the stimulus generator and state the queried property directly. So 192/192 shows the question format works, not that the media contain the information in a form a model can perceive.
    • Qwen merges frame pairs, so it sees only 4 temporal positions.
    • "No evaluated model perceives how events unfold in time" goes beyond the data: MiniCPM solves video direction 22/22, and event order is solved in 75% of evaluations.
  • Main-text claims stronger than the appendix.
    • Sec. 5 says FIP "predicts success where synergy does not"; App. I.4 shows it adds little once source success is known.
    • Sec. 5 says the local metric "separates both variants from random displacements". This holds only for Qwen: for MiniCPM video the random energy is 2.7e-5 and the semantic energy 2.6e-5 (Tab. 8).
    • Sec. 1 says "layer-wise sensitivity measures the same quantity locally", which Fig. 5 contradicts.
  • Selectivity. Half of the nuisance pool changes the reference answer. Selectivity was therefore restricted after the fact to the U0 side, with no matching U1 analysis. The 10^4–10^5 ratios come from denominators near the 1e-8 floor.
  • "The modality, not the model" is confounded. It comes from a variance split with 4 modality levels and only 2 models, and each modality has its own tasks.

Questions For Authors

Clarifications (answers could change my assessment):

  1. For Qwen3 on speed, repetition and simultaneity: what are the signed Γ, the S_f-normalized C_ctx, and the threshold feasibility, compared with Qwen2.5? If Γ is positive and feasibility rises, the "improving" claim has support; otherwise it does not.
  2. Is there any result where a coordinate other than Γ and threshold feasibility changes a conclusion? Concretely: two models or slices that tie on four-cell accuracy and feasibility, where FIP leads to a different conclusion that can be checked.
  3. Is the pullback metric or Theorem 2 used anywhere in the empirical analysis, other than to show that local measures fail? If not, what does the functional-geometry framing add over defining Γ as a factorial contrast on the answer score?
  4. Which layers were measured for onset? Some bootstrap intervals are [74%, 74%]. Does "same depth across modalities" hold under thresholds other than peak/4, or under a tuned-lens readout?

Suggestions (future work):

This matters because the paper's recommendation is that releases report a seven-coordinate profile. No case in the paper shows FIP reaching a different, checkable conclusion than four-cell accuracy plus the threshold check.

Secondary weaknesses

  • External validity of the headline temporal claim.
    • All stimuli are synthetic: 8-frame 256×256 2D shapes, CLEVR renders, ESC-50 mixtures.
    • Every model that passes the live controls has a Qwen-family LM; Phi-4 fails them.
    • The text surrogates are written by the stimulus generator and state the queried property directly. So 192/192 shows the question format works, not that the media contain the information in a form a model can perceive.
    • Qwen merges frame pairs, so it sees only 4 temporal positions.
    • "No evaluated model perceives how events unfold in time" goes beyond the data: MiniCPM solves video direction 22/22, and event order is solved in 75% of evaluations.
  • Main-text claims stronger than the appendix.
    • Sec. 5 says FIP "predicts success where synergy does not"; App. I.4 shows it adds little once source success is known.
    • Sec. 5 says the local metric "separates both variants from random displacements". This holds only for Qwen: for MiniCPM video the random energy is 2.7e-5 and the semantic energy 2.6e-5 (Tab. 8).
    • Sec. 1 says "layer-wise sensitivity measures the same quantity locally", which Fig. 5 contradicts.
  • Selectivity. Half of the nuisance pool changes the reference answer. Selectivity was therefore restricted after the fact to the U0 side, with no matching U1 analysis. The 10^4–10^5 ratios come from denominators near the 1e-8 floor.
  • "The modality, not the model" is confounded. It comes from a variance split with 4 modality levels and only 2 models, and each modality has its own tasks.
Questions for authors
  1. For Qwen3 on speed, repetition and simultaneity: what are the signed Γ, the S_f-normalized C_ctx, and the threshold feasibility, compared with Qwen2.5? If Γ is positive and feasibility rises, the "improving" claim has support; otherwise it does not.
  2. Is there any result where a coordinate other than Γ and threshold feasibility changes a conclusion? Concretely: two models or slices that tie on four-cell accuracy and feasibility, where FIP leads to a different conclusion that can be checked.
  3. Is the pullback metric or Theorem 2 used anywhere in the empirical analysis, other than to show that local measures fail? If not, what does the functional-geometry framing add over defining Γ as a factorial contrast on the answer score?
  4. Which layers were measured for onset? Some bootstrap intervals are [74%, 74%]. Does "same depth across modalities" hold under thresholds other than peak/4, or under a tuned-lens readout?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

Thank you for the insightful review. We answer the questions first, since our responses to the weaknesses build on their results.

Q1. For Qwen3-Omni on speed, repetition and simultaneity, what are the signed Γ, the normalized C_ctx and the threshold feasibility, compared with Qwen2.5-Omni?

We report the three quantities for both models on the same 32 families per property. The normalized conditionality is Cctx=median ∣Γ∣/SfC_{\text{ctx}} = \text{median}\,|\Gamma| / S_f, where SfS_f is the median absolute clean answer margin of each model on the live controls of the same run. Live controls are solvable questions on the same media, such as event order.

Each entry gives Qwen2.5-Omni → Qwen3-Omni, with counts out of 32 families.

PropertyMedian signed Γ\GammaFamilies with Γ>0\Gamma > 0CctxC_{\text{ctx}}Threshold-feasibleFour-cell success
Speed (video)0.22 → 2.9123 → 300.05 → 0.370 → 140 → 0
Repetition (video)−0.48 → 2.165 → 300.08 → 0.280 → 00 → 0
Repetition (audio)0.89 → 10.0326 → 310.36 → 1.9517 → 290 → 3
Simultaneity (audio)0.41 → 7.9124 → 320.16 → 1.530 → 150 → 1

Normalization works against Qwen3-Omni, because its SfS_f is larger (video 7.84 vs. 6.27, audio 5.16 vs. 2.75). Its CctxC_{\text{ctx}} still rises 5–10× on speed, audio repetition and simultaneity. The live controls stay solved (video event order 32/32 → 32/32, audio event order 25/32 → 29/32, audio event presence 23/32 → 23/32).

As seen in the table, Qwen3-Omni improves clearly on speed, audio repetition and simultaneity. Its signed Γ\Gamma is positive in almost every family and much larger than Qwen2.5-Omni's. Its CctxC_{\text{ctx}} is clearly higher, and many more of its families become threshold-feasible. On video repetition, Γ\Gamma also turns positive but does not yet separate the cells. Four-cell success, however, stays below 10% on every property. In the paper, our pre-registered hypothesis was that Qwen3-Omni's median ∣Γ∣|\Gamma| would stay below 2 on speed, repetition and simultaneity, and §6.3 reports that it exceeded this bound on all three. The signed Γ\Gamma and threshold feasibility now show that this rise is real progress, not a change of scale. This validates the claim of §6.3 that conditionality records progress that is missed by accuracy. We will report signed Γ\Gamma, CctxC_{\text{ctx}} and threshold feasibility in §6.3.

Q2. Is there a result where a coordinate other than Γ and threshold feasibility changes a checkable conclusion?

Yes. When two models tie on four-cell success and threshold feasibility, selectivity predicts which one stays correct under new recording conditions. The table compares Qwen2.5-Omni and MiniCPM-o on the 57 speech families solved by both models.

57 shared speech familiesQwen2.5-OmniMiniCPM-o 4.5
Four-cell success57/5757/57
Threshold-feasible57/5757/57
Median ∥Γ∥\|\Gamma\|13.2518.05
Median separation margin MM2.732.98
Median nuisance JS (lower is better)0.0000490.0055

As seen in the table, MiniCPM-o even has the larger Γ\Gamma and margin, so every measure computed from the four scores ties or favors it. Only the nuisance JS, the denominator of selectivity, points the other way. MiniCPM-o reacts more than 100 times more strongly than Qwen2.5-Omni to changes that should not affect the answer, such as a different channel or speaker. Its answer distributions therefore respond more strongly to how the speech sounds.

We tested whether this changes MiniCPM-o's answers, with a prediction registered in a hashed manifest before scoring. Under new recording conditions, Qwen2.5-Omni should keep more families correct and lose less of its normalized margin, with both 97.5% intervals excluding zero. We applied four recording conditions, none of them used to estimate selectivity, to all four cells and scored both models in the same run. One family was no longer solved on clean audio in this run, which leaves 56.

Families still solved (of 56)Qwen2.5-OmniMiniCPM-o 4.5
Telephone channel5616
Pitch shift, +2 semitones5433
Pitch shift, +4 semitones5225
Background babble4528
Mean retention92%46%

As seen in the table, Qwen2.5-Omni keeps 92% of the families correct and MiniCPM-o only 46%. The difference is +46.9 points (97.5% CI [38.0, 56.3]) and holds in every condition. Qwen2.5-Omni also loses less of its normalized margin (difference −0.267, 97.5% CI [−0.361, −0.162]). Among the 15 families where the two models have matched clean margins, the gap remains (90% vs. 33%), so the margin difference does not explain it. Every measure computed from the four scores ties or favors MiniCPM-o, whereas selectivity correctly predicts Qwen2.5-Omni's higher retention under new recording conditions. We will include this test in the paper.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

Q3. Is the pullback metric or Theorem 2 used in the empirical analysis beyond showing that local measures fail, and what does the functional geometry add over a factorial contrast?

Both shape the empirical analysis. Theorem 2 gives Γ\Gamma its interpretation, and the local study with the pullback metric is the reason every coordinate of the profile is a finite contrast. Γ\Gamma itself is the classical factorial contrast, as §2 states, and we do not claim its algebra as new. Functional geometry contributes three distinct things.

The functional viewpoint defines every measurement. Inputs are compared by the model's response, not by their position in representation space. Accessibility is the Jensen–Shannon divergence between the model's answer distributions for two inputs. Its square root is a metric on answer distributions, and pulling this metric back through the model gives the functional pseudodistance of Definition 1. Selectivity is the ratio of the median divergence for answer-changing edits to the median divergence for answer-preserving edits. Every coordinate is therefore invariant to reparameterizing the representation, provided the inputs and edits are mapped along with it (Proposition 1). Representation summaries lack this property, and the 3,118× example in §5 illustrates Proposition 1.

Theorem 2 gives Γ\Gamma its meaning as conditionality.

Γ=∫01∫01DUE2f(U0+σΔU,E0+τΔE)[ΔU,ΔE]dτdσ.\Gamma = \int_0^1\int_0^1 D^2_{UE} f\big(U_0+\sigma\Delta U,E_0+\tau\Delta E\big)[\Delta U,\Delta E] d\tau d\sigma .

Γ\Gamma is the mixed curvature of the response surface accumulated over the edit, so it measures how much the effect of the modality changes with the question. This identity, Proposition 2 (Γ\Gamma vanishes for every pair of inputs exactly when the response is additive) and the uniqueness result (App. C.2) together let Γ\Gamma be read as conditionality rather than as a generic statistic.

The local study confirms what the theory predicts. The pullback metric is first order, while Γ\Gamma integrates second-order terms over the whole edit. Theorem 2 therefore predicts that a gradient, or a single-point mixed derivative, can disagree with Γ\Gamma, even in sign. In the Qwen slices, the metric separates structured edits from random displacements, yet across all 136 local families it ranks the answer-changing edit above the answer-preserving one in only 39% of them, and single-point derivatives can have the opposite sign to Γ\Gamma (Fig. 5a, App. I.3). This is why every coordinate of the profile is a finite contrast. The coarse path quadrature is a numerical diagnostic, and no reported measurement relies on it.

We will state these three roles at the start of §3. The offset check is an implementation check of Proposition 2, and we will describe it as such in the paper.

Q4. Which layers were measured for onset, and does the agreement across modalities hold under other thresholds or under a tuned-lens readout?

Every decoder layer is measured, namely layers 0–27 for LLaVA-OneVision and Qwen2.5-Omni and layers 0–35 for MiniCPM-o. At each layer, the pre-answer state passes through the model's final normalization and unembedding, and Γ\Gamma is computed from the resulting scores. Onset is the first layer, as a fraction of depth, at which the median Γ\Gamma reaches the threshold for three consecutive layers. Intervals such as [74%, 74%] arise because layers are discrete. Every bootstrap resample selects the same layer, L20 of 27 in Qwen2.5-Omni.

SliceFloor only10% of peak25% of peak (paper)50% of peak
LLaVA image44%67%81%85%
Qwen image56%67%74%81%
Qwen video59%67%74%74%
Qwen audio41%67%74%81%
Qwen speech22%67%74%81%
MiniCPM image49%74%77%89%
MiniCPM video49%63%77%89%
MiniCPM audio34%57%74%77%
MiniCPM speech23%57%77%86%

As seen in the table, interaction becomes readable in the second half of the network in all nine slices, under the three tested peak-relative thresholds. Within Qwen2.5-Omni, the four modalities reach the 10% and 25% thresholds at exactly the same layer. In the first half of the network, no slice exceeds 4% of its peak.

The floor-only rule measures something different. It marks the first layer where any effect rises above numerical rounding, and in speech this effect is below 0.2% of the peak. The paper already reports this rule, together with the ordering of modalities it produces (App. I.6).

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

The readout uses the model's own answer head. This head needs no training, which matches the label-free design of the profile. As §6.1 states, it under-reads early layers. Onset therefore marks where interaction becomes readable as an answer, not necessarily where it is first computed, and this is the claim of §6.1. A tuned lens answers a different question, namely when a trained decoder can first read the interaction, and could give an earlier layer. All modalities of a model share the same readout, so the agreement across modalities does not come from the readout.

This confirms that interaction becomes readable late in every model and modality, in the second half of the network under all three peak-relative thresholds. We will add the threshold sweep to App. I.6.

W1. The framework is heavier than what the results use, and the claim that the profile reveals what accuracy does not.

The results show this claim directly. Each row below is a question that accuracy cannot answer.

Question accuracy cannot answerAnswer from the profile
Does one shared calibration offset repair the failures?It repairs 252 of 386 LLaVA image failures, mostly decision failures, and none of 334 Qwen video failures, mostly missing evidence (§6.2)
Is a model that still fails improving?Yes for Qwen3-Omni, whose Γ\Gamma increases and whose feasibility rises at below 10% accuracy (Q1)
Which of two equally accurate models stays correct when the input changes?The more selective one, with 92% vs. 46% retention in a pre-registered test (Q2)

The threshold check is part of the profile, since it comes from the same four scores as conditionality (Proposition 3). Q3 explains the role of functional geometry. We will add this table to §6.4, with the Qwen3-Omni result in §6.3 and the selectivity test in §6.4.

The mandatory core is small. We propose to include accessibility, signed conditionality and selectivity, together with four-cell success and the decision-failure share. All five come from candidate scores, and selectivity additionally needs the scores of answer-preserving variants. Accessibility correlates with conditionality on this panel (ρ=0.95\rho = 0.95) but remains logically independent of it (Proposition 4). It separates a model that ignores the modality from one that responds to it but not to the question, two cases with Γ\Gamma near zero that need different fixes. Onset, mediation, the text-surrogate gap and execution variability are optional (§6.4). We will state the mandatory core and the optional coordinates explicitly in §6.4 and App. G.5, and add the accessibility example after Proposition 4.

The profile is not designed to predict whether a family fails, which accuracy already does. It is designed to answer the three questions above. We will state this distinction in §5.

W2. Selectivity: the U0 restriction, the U1 side and the floor.

The U0 restriction follows the definition of selectivity, since a nuisance variant must preserve the reference answer. Variants of U1 scored against the U0 reference do not preserve the answer. We have now added the matching U1 analysis, in which the nuisance edit is applied to both scenes and each is scored against its own clean cell. All terms come from one fresh run per model.

ModalityMore selective model, U0 referenceMore selective model, U1 referenceSame at floors 10⁻⁸, 10⁻⁶ and 10⁻⁴
ImageMiniCPM-oMiniCPM-oYes
VideoMiniCPM-oMiniCPM-oYes
AudioQwen2.5-OmniQwen2.5-OmniYes
SpeechQwen2.5-OmniQwen2.5-OmniYes

As seen in the table above, the more selective model is the same under both references in every modality and at every floor. This fresh check covers 20 families per non-speech modality, each with selected nuisance types. The large image ratios come from the original panel, where the nuisance responses are genuinely small, with 30–52% of image families below 10⁻⁶. In that panel, even at a floor of 10⁻⁴ (10,000 times higher) image selectivity stays above 600 for every model. The model differences that §6.2 reports in video, audio and speech remain unchanged at every floor. We will add the U1 analysis and the floor sweep to App. I.8.

Reviewer_PC7kOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper starts from a simple point: a modality can affect the output of a multimodal large language model (MLLM) without the model using that modality the way the instruction requires. The authors carry the interaction contrast of a classical 2×2 factorial design over to model outputs and call it conditionality: Γ = s₁₁ − s₁₀ − s₀₁ + s₀₀, where s is the difference between the length-normalized log-probabilities of "yes" and "no". The experimental unit is a checkerboard family: two modality states × two question predicates, with "yes" on the two diagonal cells and "no" on the two off-diagonal cells, so that neither the modality nor the question alone determines the answer. On this basis the authors define the Functional Interaction Profile (FIP). Its core is three coordinates: accessibility (whether answer-changing edits move the output), conditionality Γ, and selectivity (whether answer-changing edits move the output more than answer-preserving perturbations do). The text-surrogate gap, causal mediation, interaction onset depth and execution variability bring the total to seven coordinates. The paper also uses threshold feasibility to split failures into two types: decision failure (the four scores are ordered correctly but the threshold is in the wrong place, so a single constant offset fixes it) and missing evidence (no threshold separates the cells). The experiments cover three models, LLaVA-OneVision-7B (image only), Qwen2.5-Omni-7B and MiniCPM-o 4.5, on four modalities (CLEVR rendered images, 8-frame synthetic videos, mixed ESC-50 environmental sounds, LibriSpeech speech) and 22 semantic dimensions, for a total of 5,012 evaluations over 4,000 families. On top of that there are 4,445 activation interventions, 36,000 evaluations on six natural benchmarks, and two held-out models (Qwen3-Omni-30B-A3B, Phi-4-multimodal). The paper reports four main findings: (1) interaction becomes readable from the layer readout only at 74–81% of network depth; (2) image failures are mostly decision failures (of the 386 LLaVA image failures, one shared offset repairs 252), whereas video failures are mostly missing evidence (of the 334 Qwen video failures, it repairs 0); (3) five dimensions (speed, duration, repetition, rotation, simultaneity/succession) have no successes across 1,268 video/audio evaluations, yet all 192/192 are solved once the media are replaced by text descriptions; (4) on three of these dimensions, Qwen3-Omni's accuracy stays near zero while its |Γ| rises by about an order of magnitude. On this evidence, the authors recommend that model releases report FIP alongside accuracy.

Critical strengths
  1. The question is well posed, the core design is simple and works, and "decision failure vs. missing evidence" is a diagnosis with practical use. The checkerboard design turns "does the model use the modality according to the question" into a testable statement: Γ is unaffected by a constant offset applied to all four cells (Prop. 2), and answering all four cells correctly with margin m requires Γ ≥ 4m (Thm. 1). On this design, threshold feasibility separates two kinds of failure that both sit at zero accuracy and points each to a different fix (calibration or perception). The evidence is concrete. For LLaVA images, one offset fitted on 100 calibration families and applied to the remaining 400 raises four-cell success from 14 to 266 with Γ unchanged; the same offset, without refitting, raises success on the paraphrased questions from 8/158 to 113/158 (Tab. 20, App. I.9). Of the 334 Qwen video failures, only 56 are threshold-feasible, and the shared offset repairs 0.

  2. The temporal finding has controls behind it and goes further than "models are bad at temporal questions". The five dimensions score 0/1,268, while the event-presence dimension, also in video/audio, is solved 64% of the time and the event-order dimension 75% (Tab. 26). The result is a cliff, with no gradual slope between these five dimensions and the rest. The text surrogates are solved 192/192 on both Qwen models (Tab. 29), which rules out the question wording, the label structure and the scoring method itself as the cause of the zeros. Across the full panel, only 1 of the 1,132 evaluations with |Γ| < 1 succeeds, and the smallest |Γ| among the 1,918 successes is 0.938 (Tab. 27), which gives a solvability screen that needs no labels. Compared with work such as TempCompass, the paper goes one step further: it locates the problem in evidence that never reaches the answer, as opposed to a decision that fails to use the evidence.

  3. The statistical practice and the honesty of the reporting are uncommon among analysis papers of this kind. Families are the sampling unit, and every median, ratio and correlation cited in the main text comes with a family-clustered bootstrap interval (Tab. 7). The criteria for the held-out models and the layer-14 intervention were fixed before the analysis. Unfavorable results are reported as they came out: the pre-specified prediction that "|Γ| stays below 2" fails on three dimensions (App. I.13); the off-target effect of the layer-26 intervention is about as large as the on-target effect (ratio 0.86–1.32), and the authors drop the causal localization claim because of it (App. I.6); FIP does not improve per-sample failure prediction (AUROC change 0.00282 [−0.01204, 0.01958], App. I.4); Phi-4 fails the control dimensions, and the authors do not count its zeros as supporting evidence (Tab. 30).

  4. The study is large and thoroughly executed. Beyond the 5,012 controlled evaluations, there are 4,445 activation interventions with main-effect and negative controls, 784 paraphrase-stability evaluations (rank correlation of signed Γ 0.9885, App. I.5), three null-input controls, and 12–18 answer-preserving perturbation variants per family. All checkpoint revision hashes are listed (Tab. 3), and the authors commit to releasing the task suite and evaluation code. The natural-benchmark section is useful in its own right. On Clotho-AQA, Qwen scores 31.30% with the original audio and 31.45% when the audio is replaced by mean tokens, yet restoring the audio changes 926 of 2,000 predictions (App. I.11). This reminds readers that unchanged accuracy after removing a modality does not mean the model ignores that modality.

Critical weaknesses

W1 (Major) The conclusion that "current MLLMs cannot perceive how events unfold in time" extrapolates beyond what the evidence covers

This is the central finding of the abstract, the §6.3 title ("What Current MLLMs Cannot Perceive") and the conclusion, yet it rests on a narrow operationalization and a narrow model panel.

  • The stimuli take a specific form. The videos are 8-frame 256×256 synthetic renders, and once Qwen2.5-Omni's encoder merges every two adjacent frames, only 4 temporal positions remain (App. H.2); the environmental sounds are 2-second ESC-50 clips placed in a 4.3-second mix. The paper does not check whether humans can discriminate these media, and the text surrogates are written directly from the generation parameters (App. I.13), so they show that the questions are answerable in language, not that the media are perceptually discriminable.
  • Model coverage is narrow. Working from Tab. 28, 1,133 of the 1,268 evaluations come from Qwen2.5-Omni. All three panel models and Qwen3-Omni use Qwen-family language models, and Phi-4, the only non-Qwen model, fails the control dimensions (App. I.13).
  • The §6.2 claim "The modality, not the model, determines how evidence is used" confounds modality with task. Each modality has its own set of semantic dimensions (App. I.7), and the model factor has only two levels. At the level of individual dimensions the models differ a lot: on video direction, MiniCPM scores 22/22 and Qwen 0/84 (Tab. 28).
  • The evidence for "A larger model begins to perceive these properties before its accuracy shows it" is thin. This conclusion is in the abstract, yet each dimension has only 32–64 families. The paper looks only at the absolute value |Γ| for Qwen3-Omni and does not report the signed Γ or the threshold-feasible share, although by Prop. 3 Γ must have the right sign and outweigh the main effects before a threshold can use it. The reading also comes after a pre-specified prediction failed, which makes it post hoc.

I suggest the authors add two checks of limited scope: (a) for each of the five zero-score dimensions, sample a small number of families and have humans answer from the media alone (watching or listening only); (b) on at least one zero-score dimension (e.g. speed or repetition), raise the frame count to 16 or 32 to test whether the zeros depend on the "8 frames, 4 temporal positions" input. The wording of the abstract, §6.2 and §6.3 should also be pulled back to the panel and suite that were actually measured.

W2 (Major) The claim that "MLLM releases should report FIP" lacks support

  • FIP adds nothing to failure prediction. By the authors' own analysis, adding FIP changes AUROC by only 0.00282, with an interval that crosses zero (App. I.4), and §5 states that "accuracy on the original questions remains the strongest single predictor".
  • Most coordinates either do not discriminate on this panel or lead to conclusions that depend on the definition. Execution variability is 0 in all nine slices (model × modality). Interaction onset depth falls at 74–81% in all nine slices, but spreads over 22%–59% if it is defined by the numerical floor alone. Causal mediation has no upper bound and reaches 2,720 for Qwen video (Tab. 15).

The part of the paper that actually yields new understanding is three quantities: "four-cell success + signed Γ + threshold feasibility (share of decision failures)". I suggest the authors narrow the release-reporting recommendation to this core and state the conditions under which the other coordinates carry information.

Questions for authors
  1. Were the media for the five zero-score dimensions checked by human judges? If 20 families are sampled per dimension and people answer from the media alone (watching or listening only), what is their accuracy?
  2. On the speed or repetition dimension, do Qwen2.5-Omni's Γ and four-cell success change when the frame count is raised from 8 to 16 or 32 (or when natural video clips are used)?
  3. What are the full prompts and candidate strings (capitalization, punctuation, whether the prompt lists yes/no)? On a subset, how often does the verdict from candidate scoring agree with the model's greedy-decoded answer? I ask because on the CLEVR color dimension, Qwen2.5-Omni reaches four-cell success of 71/72 while LLaVA-OneVision-7B gets only 9/72 (Tab. 28). In the LLaVA example in Fig. 8, all four cell scores are positive, and 5.44 nats must be subtracted before the two cells that should answer no turn negative. Judging the color of a CLEVR object is not hard for a 7B-scale vision-language model, so an affirmative bias this large may come from the scoring interface. If that is the case, "image failures are mainly a calibration problem" describes the interface more than the model.
  4. For Qwen3-Omni on the speed, repetition and simultaneity/succession dimensions, what are the median signed Γ, the threshold-feasible share, and the conditionality normalized by its own S_f?
  5. Which FIP coordinates do the authors consider mandatory for release reporting? Can they give an example in which the full FIP changed a practical decision that "four-cell success + signed Γ + share of decision failures" could not?
8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/6)

Thank you for the concrete suggestions. We answer the questions first, since our responses to W1 and W2 build on their results.

Q1. Can humans discriminate the media of the five zero-score dimensions?

The text surrogates establish that the questions are answerable, but not that the queried properties are perceptible in the media. We therefore added a human audit of the five temporal properties.

For each property, we sampled 10 families at random from the held-out cohort, with repetition tested in both video and audio: 60 families, 120 clips and 240 yes/no answers. Each clip was presented alone and in the same form as the model input: the same 8 frames played at 2 frames per second, or the original audio mixture. The annotator answered both questions of each clip. Clip identities were hidden and the presentation order was randomized. Paired clips, correct answers and model outputs were not shown. The table below compares the annotator with both Qwen models on the same 60 families.

DimensionFamiliesHuman cell accuracyHuman four-cell successQwen2.5-Omni four-cell successQwen3-Omni four-cell success
Speed (video)10100%10/100/100/10
Duration (video)10100%10/100/100/10
Rotation (video)10100%10/100/100/10
Repetition (video)10100%10/100/100/10
Repetition (audio)10100%10/100/100/10
Simultaneity (audio)1095%9/100/101/10

From the media alone, the annotator solves 59 of the 60 families (99% of answers), most after a single play (89% of video clips, 72% of audio clips). On the same families, Qwen2.5-Omni solves none and Qwen3-Omni solves one. The queried properties are therefore perceptible in the media, and the text surrogates show that the questions are answerable (192/192). This validates our claim that zero four-cell success on the five temporal properties is a failure of the models. The evidence is present in the media but does not reach the answer. We will include the human audit in the paper.

Q2. Do Qwen2.5-Omni's Γ and four-cell success on speed and repetition change at 16 or 32 frames?

We used 8 frames so that each temporal state is controlled. The zero scores do not depend on frames alone, since the two audio properties have no frames and also score zero. To test the video properties directly, we re-rendered all 160 held-out video families at 16 and 32 frames. These cover speed and repetition, and also duration and rotation, with event order as the live control, a question on the same video that the model solves, confirming it processes the media. Motion paths, clip duration and event timing are unchanged, and only the sampling is denser. Qwen2.5-Omni therefore receives 8, 16 and 32 frames, which become 4, 8 and 16 temporal positions after its encoder merges adjacent frames. All three frame counts were scored in the same run.

DimensionFrames (temporal positions)Four-cell successMedian signed ΓFamilies with Γ > 0Threshold-feasible
Speed (video)8 (4)0/320.1323/320/32
16 (8)0/320.8832/320/32
32 (16)0/320.5631/320/32
Repetition (video)8 (4)0/32−0.565/320/32
16 (8)0/321.1930/320/32
32 (16)0/321.3832/320/32
Duration (video)8 (4)0/320.7532/322/32
16 (8)0/321.6932/320/32
32 (16)0/321.2832/320/32
Rotation (video)8 (4)0/321.1332/3225/32
16 (8)0/320.3132/320/32
32 (16)0/320.6932/3218/32
Event order (video, live control)8 (4)32/3226.0032/3232/32
16 (8)32/3224.8132/3232/32
32 (16)32/3225.0932/3232/32

At 16 and 32 frames, Qwen2.5-Omni still solves none of the 128 target families, while the live control stays at 32/32. Denser sampling strengthens the interaction, and on repetition Γ > 0 rises from 5/32 to 32/32 families. However, a single threshold still cannot separate the yes and no cells in any speed, repetition or duration family. The result is the same with "Yes"/"No" candidates. This confirms that zero four-cell success on the five temporal properties is a failure of the models, not an artifact of the 8-frame input. We will include this test in the paper.

AuthorsOpenReview ↗

Rebuttal by Authors (2/6)

Q3. Prompts, candidate strings, greedy agreement and LLaVA's affirmative bias.

Prompt and candidates. All models receive the same prompt, wrapped in each model's own chat template. The prompt lists both answers. For LLaVA-OneVision, the full input is

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<image>
Answer the question using only "yes" or "no".
Question: {question}
Answer:<|im_end|>
<|im_start|>assistant

Qwen2.5-Omni uses the same system turn and places <|vision_bos|><|VIDEO|><|vision_eos|> (or the audio or image equivalent) before the instruction. The candidates are " yes" and " no", in lowercase with a leading space and no punctuation. Each is a single token (LLaVA ids 9834 and 902), so length normalization has no effect. Reversing the order in the instruction to "no" or "yes" preserves the sign of Γ\Gamma in every family. Both candidates are teacher-forced after the assistant header, and the cell score is

s=log⁡p(" yes"∣x,q)−log⁡p(" no"∣x,q).s = \log p(\text{" yes"} \mid x, q) - \log p(\text{" no"} \mid x, q).

Agreement with greedy decoding. On a subset of 96 image families, we decoded every cell greedily with the prompt unchanged. Every greedy answer was a valid yes or no. The table gives the share of cells where the greedy answer matches the sign of ss.

SubsetFamiliesLLaVAQwen2.5
Color7267%99.7%
Other image dimensions2456%92%

Qwen2.5-Omni's scores agree closely with its greedy answers. LLaVA's disagreement may come from the affirmative bias, which we examine next.

Is LLaVA's bias in the interface or in the model? Following the reviewer's suggestion, we rescored the same cells with three other candidate pairs and compared four-cell success with greedy decoding.

Color, 72 families, four-cell successLLaVAQwen2.5
" yes" / " no"871
" Yes" / " No"169
"yes" / "no"6572
"Yes" / "No"6572
Greedy decoding6572

As seen in the table, the two pairs with a leading space fail for LLaVA, and the two without it solve 65 of 72 color families, the same as greedy decoding, with scores that agree with greedy decoding in every color cell. The bias therefore lies in the leading-space token of the interface. The image evidence in LLaVA's scores, measured by Γ\Gamma and threshold feasibility, lies in the model. On all 500 LLaVA image families, the change from " yes"/" no" to "Yes"/"No" raises four-cell success from 18 to 368, while the median Γ\Gamma stays nearly the same (13.4 vs. 11.3), and so does the number of threshold-feasible families (425 vs. 422). The new candidates therefore change LLaVA's answers but leave this evidence nearly unchanged, which is what a decision failure means.

The shared offset performs the same repair and recovers the 5.44-nat gap of Fig. 8. Fitted under " yes", it lifts held-out success from 14 to 266 of 400 families. Fitted under "Yes", it is only 0.80 nats and leaves almost nothing to repair (290 → 288). We will add this to the experimental setup of the paper. Both candidate pairs will be scored, and the pair that agrees best with greedy decoding will be used. The evaluation harness, which we will release, will perform this check by default.

AuthorsOpenReview ↗

Rebuttal by Authors (3/6)

A test without yes/no tokens. The results above still measure the evidence through yes/no tokens. If the decision-failure families contain the evidence, the model should also solve them when it answers in a different form. We tested this by asking the model to choose between the two questions of a family. For each image, it picks which question, A or B, has the answer "yes". We re-rendered 200 image families with Gaussian blur, so that both models fail on some of them. Each image was shown with the prompt

Exactly one of the following embedded yes/no questions has the answer "yes". A: "{q₀}" B: "{q₁}". Which option has answer "yes"? Answer only "A" or "B".

The response is scored by the margin log⁡p(A)−log⁡p(B)\log p(\text{A}) - \log p(\text{B}), and a family counts as solved when both images are answered correctly. Each entry counts the families solved with the A/B interface, out of the families with that diagnosis from the yes/no scores.

Yes/no diagnosisLLaVAQwen2.5
Solved11/11 (100%)148/162 (91%)
Decision failure135/156 (87%)5/26 (19%)
Missing evidence8/33 (24%)1/12 (8%)

LLaVA solves 87% of its decision-failure families with the A/B interface, compared with 24% of its missing-evidence families. This supports the practical value of the diagnosis: families whose yes/no scores are threshold-separable are substantially more recoverable under an alternative response format. The separation is smaller for Qwen2.5-Omni (19% versus 8%), so the recovery rate depends on the model and interface. Because the A/B prompt presents both questions together, this experiment tests recovery under an alternative task formulation as well as alternative answer tokens. We will include the greedy-decoding check and the A/B test in the paper.

In conclusion, the candidate scores agree with greedy decoding once the candidates match the preferred answer form of the model. LLaVA's affirmative bias comes mainly from the leading-space token, and changing the candidates mainly moves the decision, not the evidence. The four-cell diagnosis identified correctly which failures such a shift repairs. Qwen2.5-Omni's temporal zeros persist under every candidate pair and under greedy decoding (0/72), so they remain missing evidence.

Q4. For Qwen3-Omni on speed, repetition and simultaneity, what are the median signed Γ, the threshold-feasible share and the normalized conditionality?

A large ∣Γ∣|\Gamma| alone does not show progress. By Proposition 3, Γ\Gamma must also have the right sign and outweigh the main effects before a threshold can separate the cells. We therefore report the three requested quantities for Qwen2.5-Omni and Qwen3-Omni on the same 32 families per property. The normalized conditionality is Cctx=median ∣Γ∣/SfC_{\text{ctx}} = \text{median}\,|\Gamma| / S_f, where SfS_f is the median absolute clean answer margin of each model on the live controls of the same run.

Each entry gives Qwen2.5-Omni → Qwen3-Omni, with counts out of 32 families.

DimensionMedian signed Γ\GammaFamilies with Γ>0\Gamma > 0CctxC_{\text{ctx}}Threshold-feasibleFour-cell success
Speed (video)0.22 → 2.9123 → 300.05 → 0.370 → 140 → 0
Repetition (video)−0.48 → 2.165 → 300.08 → 0.280 → 00 → 0
Repetition (audio)0.89 → 10.0326 → 310.36 → 1.9517 → 290 → 3
Simultaneity (audio)0.41 → 7.9124 → 320.16 → 1.530 → 150 → 1

The rise is not a change of scale. Qwen3-Omni has a larger SfS_f (video 7.84 vs. 6.27, audio 5.16 vs. 2.75), yet its normalized conditionality still rises 5–10× on speed, audio repetition and simultaneity. The live controls stay solved (video event order 32/32 → 32/32, audio event order 25/32 → 29/32, audio event presence 23/32 → 23/32).

On speed, audio repetition and simultaneity, Qwen3-Omni's Γ\Gamma is positive in 94–100% of families, and its scores become threshold-separable in 44–91% of families, while four-cell success stays below 10%. On video repetition, Γ\Gamma turns positive in 30 of 32 families but does not yet separate the cells. This validates the claim of §6.3 that conditionality records progress that is missed by accuracy. We revise §6.3 to report signed Γ\Gamma, CctxC_{\text{ctx}} and threshold feasibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.