acceptodds
Withdrawn from ICLR 2027

When Same Sign Is Not Enough: Auditing Modality-Dominance Claims

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A modality score is not yet a modality-dominance claim. Prior work established that removal semantics can change explanations and that importance estimates can be uncertain, but these issues are usually treated separately. We ask whether a directional modality claim remains statistically resolved across a declared family of plausible interventions. We audit a fixed text–numeric predictor under three removal operators, separate the observed reliance gap from an operator-specific empirical no-signal reference, and use paired inference that propagates dependent-sample and finite-null-bank uncertainty. In a leakage-audited financial forecasting stress test, all 21 operator–measurement null-centred point estimates favour numeric reliance. Yet only 5/7 measurements are resolved across the full operator family; 2/7 remain unresolved under same-date regime-matched removal. A prespecified marginal-versus-regime-matched magnitude contrast excludes zero on all seven measurements. Thus complete point-sign agreement can coexist with incomplete family-level inferential resolution. The contribution is a claim-level audit that determines when intervention semantics and uncertainty support a modality-dominance statement; the underlying attribution score is unchanged.

Primary area:
interpretability and explainable AI
Keywords:
modality attribution, removal semantics, feature removal, uncertainty quantification, explainable AI, financial forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QrQwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper is situated in the literature on explainability for multimodel models, in particular models that take text and time series data as input. The outputs of explanation methods in this context are, according to the authors, often used to make claims about one modality dominating the other, e.g. text being more important than the numerical data. The authors propose methodology to directly make such a claim, including statistical inference. Therefore they combine ideas from feature importance and uncertainty quantification. The main finding is that even if all individual scores suggest that one domain is more important than another, it could be that not off scores are significantly different from zero, which would affect the claim.

Critical strengths

The question of whether feature importance scores allow a particular conclusion that is commonly made a particular literature is interesting.

Critical weaknesses
  • The paper is difficult to read, even as someone who knows the XAI literature pretty well.
  • The central finding (scores might be negative but not be statistically significant) is nice, but at the same time also kind of common knowledge, so I don't see yet that the finding justifiees acceptance at ICLR.
  • I am not convinced yet by the methodological construction of the explainability method. It is well known that permutation approaches are not really meaningful when the goal is to learn something about the data / the model's behavior on realistic data, and mixing the observation with another real observation might similarly suffer from extrapolation. Why not directly use something like conditional/observational imputation, where we have theory that links the feature importance scores to something meaningful, for example, conditional independence?
Questions for authors
  • Why do you choose the three members of your declared family? Why not something like observational/conditional value functions?
Reviewer_Q8aqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers a setting of a predictor which takes text and numeric inputs. It argues that same-sign point estimates of an attribution score (called modality reliance in the paper) need not have confidence intervals that all exclude zero. This is illustrated experimentally with a single logistic probe applied to a financial dataset.

Critical strengths

Could not identify any strengths of the submission.

Critical weaknesses

(W1) The basic setup and notation are almost impossible to parse.

The beginning of Section 3.1, where the basic setting is introduced, surfaces a number of concerns, such as:

  • in the very first sentence, it is unclear whether the implication is that x=(xt,xn)x = (x_t, x_n)? If so, good to spell out to help the reader.
  • the notation ω1\omega_{1} to ω3\omega_{3} is not even explained: marginal permutation, backward-within-ticker donation, same-date regime-matched donation -> these concepts are not even connected to a citation, let alone explained.
  • donor di,j(ω,s)d_{i,j}^{(\omega, s)} is not motivated, explained or introduced.
  • the symbol NmN_m is not even defined.
  • the notion of trapezoidal integration is not introduced or explained properly.

These are just some of the issues, which make the paper almost impossible to understand.

(W2) The writing is extremely difficult to follow in other places as well.

The introduction seems unusual, compared to a typical paper. Problem motivation and related literature are not really introduced in much depth, and the fourth paragraph talks about empirical results without enough context. This does not seem to belong to the introduction at all. Similarly, the dataset used is described in related work, which also seems out of place. Both of these likely belong in experiments.

(W3) The paper makes no theoretical statements.

The paper makes no theoretical claims, which shifts the burden of evidence onto the empirical side, which is not convincing. Beyond this, the key point raised seems elementary: the same-sign point estimates need not have confidence intervals that exclude zero, which seems like a feature of any type of statistical uncertainty considered in the ML literature.

(W4) The paper’s empirical evaluation is limited.

The paper’s experimental focus is on a single dataset and a single logistic predictor. This makes the evaluation very limited, and in addition to the basic setting being so poorly described, it makes the contribution hard to judge.

Questions for authors

Please see weaknesses.

Reviewer_oadwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper takes one fixed L2 logistic probe on FNSPID, text embeddings plus 71 numeric features predicting the next-10-day volatility regime, and asks whether “numeric dominates text” survives three removal operators: marginal permutation, backward-within-ticker donation, and same-date regime-matched donation. For each operator it computes a confidence-loss gap D (Equation 3), subtracts an empirical null centre from a bank of 50 label-shuffled models to get D* (Equation 5), and builds sup-t simultaneous intervals across the three operators from a paired circular block bootstrap (Section 3.2). On seven overlapping measurements, O1 to O3 on the original universe, U1 to U3 on a disjoint 19-ticker universe and T1 on 2023-H2, all 21 D* point estimates are negative, but the regime-matched intervals for U1 and U2 cross zero, so 5/7 measurements are called family-resolved and 2/7 unresolved (Table 2). A secondary marginal-versus-regime-matched contrast excludes zero on all seven. There is no new attribution method. The contribution is the audit logic, this one instance, and the reporting checklist in Section 6.

Critical strengths

The machinery is the right machinery and the numbers show why it is needed. Under marginal permutation the no-signal models already produce a gap between −0.00166 and −0.02508 on the seven measurements (Section 5.1), the same order as the centred effects in Table 2. So a plain permutation importance on this probe would actually have called numeric dominant even with the labels shuffled. The paired bootstrap keeps all three operators on the same resampled rows, which is what makes the ω1 to ω3 contrast meaningful. Also, Section 4 and Section 7 say plainly that the seven measurements overlap, that O1 to O3 were used to pick the target, and that the null is not a valid equal-reliance null. This is good.

Critical weaknesses

The 5/7 versus 2/7 number is basically a thresholding effect, and the paper reads the wrong pattern out of it. In the ω3 column of Table 2 the three disjoint-universe measurements sit at −0.00422, −0.00451 and −0.00466, within 0.0005 of each other, while O1 to O3 and T1 sit between −0.0164 and −0.0193. That is the real finding: on the new universe, regime-matched removal shrinks the numeric gap about four times, uniformly. What separates U3 (resolved) from U1 and U2 (unresolved) is only the interval half-width, 0.0039 against 0.0051 and 0.0065, and U2 misses zero by 0.00055. So the sentence in Section 5.1 that U3 is resolved and therefore the unresolved cases do not support a universe-level pattern is backwards. The point estimates show exactly a universe-level pattern, and at this effect size the resolved label is sitting right on the boundary. Since the whole paper hangs on “21/21 same sign but 5/7 resolved”, I would like to see how that count moves with the block length (Appendix A.1), the confidence level and the null subsample size, and I would like the text to report the ω3 magnitude pattern instead of the label count.

Questions for authors

Please report, for each of the seven measurements, test AUC for the joint probe, a text-only probe and a numeric-only probe. This is the question which matters most for my rating. If text-only is at chance on the U universe, the framing changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.