acceptodds
Under review as a conference paper at ICLR 2027

When Do Cross-Modal Gains Warrant a Semantic Role? A Role-Conditioned Audit of Derived Pairing Rules

Abstract

Cross-modal gains can change learning, but they do not by themselves show that the assigned relation explains the gain: a nuisance scaffold can reproduce the same endpoint pattern. The cross-modal alignment spuriousness (CMAS) audit makes this premise-failure gap explicit through three layers: phenomenon (whether a gain is observed), mechanism (which information and computational routes are active), and boundary (whether nuisance-relative utility clears declared margins). In a five-seed DOLOS case restricted to one corpus, carrier/backbone, split, and validation-only known-offset endpoint, the primary provenance–time contrasts are inconclusive at the declared margins, while phase is recoverable but phase-selected directions do not outperform rank-matched random subspaces. A second same-carrier role is likewise not authorized at this scale. The operational conclusion is bounded: cross-modal gains can change learning, but they do not by themselves authorize the assigned semantic role beyond declared alternatives.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
cross-modal representation learning, multimodal self-supervision, semantic authorization, cross-modal alignment spuriousness, interpretability auditing, interventional probing, audiovisual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eahEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Research Question


The paper asks whether performance gains from a derived cross-modal pairing rule support the intended relation. Gains alone cannot establish this, because temporal proximity, identity cues, or leakage may also explain them.

Their Method


The authors propose CMAS, an audit protocol that specifies the claimed relation, nuisance control, an endpoint with independently defined answers, and decision margins. It organizes four types of evidence: whether the rule affects training, whether relevant information is recoverable with probes, whether the rule improves performance over the control, and whether performance depends on a specified information pathway.

CMAS is evaluated in a single DOLOS case study with five seeds. Real-P selects audio–video token pairs that are close in time and have similar phases in a POS-derived rPPG signal extracted from the face video. It also weights pairs by envelope similarity. The Time-only control retains temporal proximity and uses uniform weights. Performance is measured by Recall@1 for recovering injected audio–visual offsets.

Results

The primary contrasts are inconclusive. Real-P also shows an inconclusive improvement of approximately 2.4 percentage points over Time-only. Phase information can be decoded from the visual features, but removing phase-predictive directions performs similarly to removing random directions, while retaining them performs worse than retaining random directions.

The synthetic positive control detects a large constructed effect. Natural route deletion is not tested; A4 is exercised only in the synthetic setting. Overall, the evidence does not establish a semantic role for Real-P at the declared scale, but it also does not establish that such a role is absent.

Critical strengths
  1. Careful controls and honest reporting. The controls target relevant alternative explanations. Using another clip’s rPPG signal, disrupting its temporal alignment, and using a donor clip from the same person examine provenance, alignment, and identity. The authors also specify the endpoint and margins in advance, report individual seed effects and simultaneous confidence intervals, and avoid interpreting inconclusive results as evidence of no effect.

  2. A useful distinction between decodability and task use. Phase information can be decoded from the visual features, but removing phase-predictive directions performs similarly to removing matched random directions, while retaining them yields lower performance than retaining random directions. These results support the paper’s caution that decodability alone does not establish useful information use on the evaluated task.

Critical weaknesses

1. The case study provides limited evidence that CMAS can distinguish competing explanations.

The paper asks when a performance gain supports a semantic interpretation, but the case study does not establish a clear gain for Real-P over Time-only. The estimated improvement is 2.4 percentage points, with a simultaneous 95% confidence interval of −3.7 to +8.5 points. It is also unclear whether Real-P represents a previously established claim or a rule constructed for this audit. The study therefore provides limited evidence about how CMAS would evaluate an established improvement whose explanation is uncertain.

Only five training seeds are used, and uncertainty remains large relative to the chosen margins of 2.5 and 5 percentage points. Returning an inconclusive verdict is appropriate, but it provides limited validation of the audit’s ability to distinguish explanations.

The donor and temporal controls are useful. However, the study lacks a calibration case where a gain is known to come entirely from a nuisance factor and CMAS should reject its semantic interpretation. The synthetic positive control detects a constructed improvement of approximately 70 percentage points. Detecting such a large effect does not establish reliable discrimination between semantic and nuisance explanations at the much smaller declared margins.

2. Limited demonstrated novelty and an unclear presentation of the framework.

Prior work already establishes the importance of probing controls and the distinction between decodable information and information used by a model [1, 2, 3]. Interpreting effects against meaningful margins also follows established statistical practice [4]. CMAS combines these ideas, but the case study does not clearly demonstrate the additional diagnostic value of that combination.

Although phase-probing results are reported, the final natural-data verdict relies mainly on A3. A1 lacks quantitative results, and A4 is tested only in the synthetic setting. The broader framework therefore remains only partially validated.

The presentation also makes the framework difficult to follow. Real-L is insufficiently defined, and the PCS abbreviation is not explained, although its loss is specified mathematically. The mapping between the three conceptual layers and A1–A4 changes across sections. Consistent definitions and fewer repeated qualifications would make the contribution clearer.

References

[1] Hewitt, John, and Percy Liang. "Designing and interpreting probes with control tasks." Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp). 2019.

[2] Ravichander, Abhilasha, Yonatan Belinkov, and Eduard Hovy. "Probing the probing paradigm: Does probing accuracy entail task relevance?." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

[3] Elazar, Yanai, et al. "Amnesic probing: Behavioral explanation with amnesic counterfactuals." Transactions of the Association for Computational Linguistics 9 (2021): 160-175.

[4] Anvari, Farid, and Daniël Lakens. "Using anchor-based methods to determine the smallest effect size of interest." Journal of Experimental Social Psychology 96 (2021): 104159.

Questions for authors

Questions


  1. Origin of Real-P and the missing gain. Is Real-P taken from prior work, or was it designed for this study? What is the deception detection accuracy for each arm? If none of the arms improves the task the model is trained for, it is unclear which gain the audit is trying to explain.

  2. Independence from the pulse signal P. What is the chance-level Recall@1 for the endpoint? Does the endpoint depend on the masked cross-attention? If so, which mask is used for the shifted inputs? Also, does Time-only use the temporal radius estimated from P’s period, as suggested by Figure 1?

  3. Data and leakage. The paper uses 955 clips from “five shared subjects”, while the original DOLOS paper reports about 1,675 clips from 213 subjects. How was this subset selected? Do the training and validation sets contain clips from the same subjects?

  4. Numerical consistency. Figure 4 reports five-seed means for each case, which should be multiples of 1/30, but most values appear to be multiples of 1/6. Also, the natural arm in Table 3 (0.3412) does not match any of the primary arms. How were these numbers computed, and which checkpoints were used for the probing analysis?

Suggestions

  1. Apply CMAS to an established derived-pairing method with a documented performance gain, such as the cluster-based cross-modal targets of [1]. This would give the audit a concrete gain to explain.

  2. Add a negative control where the gain is deliberately driven by a nuisance factor, such as temporal locality or identity, and test whether CMAS correctly rejects the semantic interpretation.

  3. Run a prospective power analysis and use more seeds or a larger evaluation set so that the declared margins can be resolved. The paper's own post-hoc analysis implies that five seeds can only reliably detect effects of about 0.06, which is larger than both margins (0.025 and 0.05).

  4. Clearly specify how the three layers map to A1–A4, and define all key terms such as Real-L and PCS.


[1] Alwassel, Humam, et al. "Self-supervised learning by cross-modal audio-video clustering." Advances in neural information processing systems 33 (2020): 9758-9770.

Reviewer_Wac8OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CMAS, an audit framework for assessing whether gains from cross-modal pairing rules support their claimed semantic interpretation. In the DOLOS case study, the main comparisons are inconclusive, while phase information can be decoded without clear evidence that the model preferentially uses phase-related directions for the evaluated task.

Critical strengths
  • Useful distinctions between types of evidence. The framework separates effects on training, information decodability, and task utility, avoiding the assumption that decodable information necessarily drives task performance.

  • Cautious interpretation of results. The paper reports uncertainty, uses matched random-subspace controls, and explicitly treats the main comparisons as inconclusive rather than claiming that the effects are absent.

Critical weaknesses
  • I found terms like “semantic authorization” and “role” hard to follow. I would prefer a more direct explanation of what might account for the performance gains.
  • I sometimes lost track of how the experiments fit the framework, since what counts as a “phenomenon” or a “mechanism” seems to change across sections.
  • I can't tell what a specific relation contributes when pair selection, density, and weighting all differ between the compared configurations.
  • I think phrases like “a new paradigm” and “phase-detectability paradox” oversell what the results actually show, especially when the main comparisons are inconclusive.
Questions for authors

See Weaknesses!

Reviewer_AredOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CMAS, an audit framework for determining when gains from derived cross-modal supervision justify assigning a semantic interpretation to the relation used for pairing. The central argument is that an endpoint gain alone is non-identifying: the same improvement may arise from the intended semantic relation or from nuisance factors such as temporal locality, identity, leakage, or routing. CMAS separates evidence into activity, recoverability, nuisance-relative utility, and route dependence, with predeclared controls, endpoints, and effect-size margins.

The framework is instantiated in a five-seed DOLOS audio-visual case study. The primary phase/envelope pairing rule is compared against time-only, donor, derangement, and subject-preserving controls using a validation-only known-offset Recall@1 endpoint. The primary utility contrasts are inconclusive at the declared margins. Phase information is recoverable, but interventions targeting phase-predictive directions do not outperform rank-matched random subspaces. The paper therefore concludes that the observed cross-modal effects do not currently authorize the proposed semantic role in this specific carrier and experimental setting.

Critical strengths

The paper identifies an important methodological problem: downstream gains from cross-modal supervision do not by themselves identify why the supervision helped. CMAS provides a principled structure for separating optimization activity, information recoverability, nuisance-relative utility, and route-specific use. This is a useful corrective to interpretations that move too quickly from improved performance to claims about semantic representation.

Critical weaknesses

The empirical validation is too narrow to establish CMAS as a broadly useful audit framework. The main natural experiment uses only five seeds, one corpus, one backbone/carrier, one validation split, and one specialized correspondence endpoint. Transfer to other datasets and architectures is explicitly untested. This is consequential because the paper makes a fairly general methodological argument, while the empirical demonstration remains highly case-specific.

Questions for authors

Why should the current DOLOS experiment be considered sufficient validation of CMAS as a general audit methodology rather than primarily an illustrative case study? Evidence on even one additional dataset, backbone, or cross-modal relation would materially strengthen my confidence in the framework’s portability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.