- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper distinguishes direct model dependence from predictive dependence involving later time-series inputs. It defines direct and total reference effects using conditional replacements, introduces OMIC and AUOMIC to assess support and ranking consistency, and interprets the total-minus-direct residual as a mediation diagnostic. Validation includes a synthetic VAR and audits of TIMING, TimeSHAP, and KARMA on classification, regression, and forecasting tasks derived from Beijing PM2.5.
Critical strengths
-
Relevant problem. Temporal dependence can obscure whether an attribution reflects direct model use or predictive information about other inputs. Making the intended explanation target explicit is valuable.
-
Transparent evaluation framework. The paper states OMIC's relationship to established ranking statistics and acknowledges several limitations. Synthetic examples and public code make its claims inspectable.
Critical weaknesses
1. The effect definitions do not support the claimed mediation interpretation
The assertion that every direct edge is also a total edge (Definition 1, line 117) is false under Eqs. (2)-(6). Let S and epsilon be independent standard Gaussians, M = S + epsilon, and f(S,M) = S-M. Then Delta_dir > 0, but g_tot(v) = v-E[M|S=v] = 0, giving Delta_tot = 0. Moreover, for f(S,M) = S, the model ignores M, yet Delta_dir = sqrt(2/pi) and Delta_tot = 2/sqrt(pi), producing a positive mediation residual. This arises because the two references use different source-replacement distributions. With smaller noise, the residual ratio approaches one despite no model-mediated pathway. These exact Gaussian counterexamples undermine both the edge-inclusion claim and the interpretation of Med as a mediation diagnostic; additional assumptions or a revised interpretation are necessary.
2. The ranking metric and experimental implementation do not establish the intended evaluation claims
AUOMIC can conceal incorrect ordering within a sparse support. With 1,000 candidate edges, ten nonzero effects 1,...,10, and the correct ten-edge support ranked entirely backwards, Eqs. (10)-(12) give AUOMIC approximately 0.99398 at pi_max = 0.75, versus 1 for perfect ordering. Thus, a near-perfect score need not establish accurate within-support ranking. Separately, Eq. (6) measures distance between conditional mean responses, whereas the public delta_tot implementation averages residual-preserving paired output differences, a generally different estimand. The reported mf ratio is defined as a mean, but mediation computes the fraction exceeding 0.5. The authors should identify the revision used for Table 1, reconcile these definitions, and establish what the resulting scores actually validate.
Table 1's separate task audits and similar direct/total scores also leave the practical gain over existing faithfulness metrics unclear. The repository's baselines and planted-mediation results should substantiate that gain under the same definitions.
3. Additional submission concern regarding repository anonymity
The repository linked in Section 4.1 retains identifying contributor metadata in its public history. Specifically, commit 1aba634adb574bf927c0cec13cff3d7a092de4ad, dated September 5, 2026, 17:37:59 UTC, modifies data/raw/beijing_pm2/PRSA_data_2010.1.1-2014.12.31.csv and ts-mule/docs/make.bat. Its metadata exposes a personal-name-style commit.author.name, an email in commit.author.email, and an associated author.login distinct from the anonymous repository account. This creates a potential de-anonymization route if the contributor is a paper author.
Questions for authors
-
What assumptions exclude the counterexamples above, and how should
Medand its ratio be interpreted when effects cancel orDelta_tot = 0? -
Which total-effect estimand and mf statistic generated Table 1 and Figures 2-4? Please identify the corresponding code revision and reconcile it with the equations.
-
How should AUOMIC be interpreted when the correct sparse support is internally misranked? Can targeted checks with known absent, present, and cancelling mediation establish the diagnostic claims under a consistent definition?
Dear Authors,
Thank you for your response. I appreciate your acknowledgment of several concerns, but some fundamental issues remain unresolved.
In particular, Q1 and Q2 concern mathematically incorrect claims that require formal corrections rather than terminological clarifications. Q3 remains insufficiently addressed, as no concrete evidence demonstrates when direct and total relevance lead to different evaluations. For Q4, please clarify which supplementary experiments address estimation sensitivity and what their findings are.
I would appreciate further clarification, particularly on Q1 and Q2, before reconsidering my score.
Best,
Reviewer 45xk