- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that the treatment of 0th-order (bias or reference-point) residuals in standard LRP causes signal collapse and sign flips in deep networks. A constant 'minimal probe' b^m (Assumption 3.1) is inserted into an αβ-style rule (Eq. 5), the surviving term (Eq. 7) is interpreted as an angular deviation θ (Def. 3.2), and the proposed π-rule (Eq. 15 & 16) is claimed to help CNNs and ViTs.
Critical strengths
Whether the handling of bias terms contributes to relevance distortion in deep networks is a fair question, and isolating it with Uniform LRP (§5.1, Appendix B) is a good idea.
Critical weaknesses
I do not think the claim that 0th-order terms cause signal collapse, and that the π-rule removes the cause, is established, for four reasons.
-
The 'noise' of Eqs. 6 & 7 does not seem to me to follow from the αβ-rule or DTD. Eq. 5 adds b^m to every (i,j) term, although Σ_i Z_ji R_j = R_j - c·b̃_j (Eqs. 2 & 3) already covers the bias contribution. The αβ-rule also conserves relevance only for α - β = 1, yet the sweep (§5.2), the 1:1 upper-layer rule of LRP_com and Figures 3 & 4 lie outside it (α - β = 0 at 1:1), so sign flips are expected without any b.
-
§4.3 itself ends with R_bias ∝ (α − β)b^m, which is non-zero unless α = β. With π = 2PM/(P+M), the coefficient becomes π(α − β)/(α + β), i.e., -10 for the αβ-rule against +14 for the π-rule at α:β = 2:1, P = 30, M = 70. Moreover, α_π/β_π = (α/β)(M/P), so the rule does not 'propagate the originally intended α:β ratio' (§4.2).
-
For ViTs the abstract reports 'marginal improvements', yet LRP_com^π is worse than LRP_com^αβ on all six metrics for ViT-B and ViT-L (12 of 12 comparisons, Tables 2-7). Plain LRP_(ε) beats LRP_com^π in 17 of 17 CNN rows of Table 2, so the 'consistently superior' claim (§5.3) is not particularly strong (only a weak baseline is compared). SGLRP is omitted from Table 1, and in Table 2 the π-rule is worse in 7 of 17 SGLRP rows.
-
In Fig. 2, θ depends only on P^m and M^m and is zero for the π-rule by construction, so its suppressed variance is close to tautological. In Fig. 4, rescaling (α, β) jointly multiplies both terms of Eq. 6 equally, so Eq. 7 predicts a scale-invariant noise-to-signal ratio. Given that the magnitudes could very well overflow after ~8 layers, this should also be ruled out.
I also found there to be some significant overstatements and information I could not find:
- The "unequivocally proves" claim in the abstract very much overstates the unchecked assumption, and I could not find the derivations cited in the Reproducibility Statement in Appendices A or B.
Questions for authors
-
How is θ computed for the π-rule in Figure 2, given θ = 0 by construction?
-
Do collapse and sign flips persist at α - β = 1 (e.g., 2:1) once biases are zeroed, and how do Figures 3(c) and 4 look in float64?
-
How does an αβ-rule with the π-rule's effective ratio (αM:βP) perform on its own?
-
Please reconcile 'marginal improvements' for ViTs with the 12 of 12 worse comparisons, and add SGLRP to Table 1 with paired confidence intervals.
3 comments
Do your experiments implement R = α·(positive share) − β·(negative share) with β ≥ 0, as in Eqs. 5 and 16, or a signed β? In the first case α + β = 1 gives total relevance α − β (zero at 0.5:0.5), and in the second Eqs. 5 and 16 need rewriting.
Dear Reviewer mrLk,
Thank you for your prompt follow-up.
To answer your question directly: our experiments implement , exactly as written in Equations 5 and 16, and the equations do not need to be rewritten. The use of the negative sign before is not a peculiar notation introduced by us, but rather strictly follows the explicit normalization steps of the standard LRP formulation (e.g., as referenced in "Layer-wise relevance propagation: an overview" by G. Montavon et al.).
To clear up any confusion, here is the step-by-step derivation from the most basic LRP formula to our Equation 5, explicitly showing how the negative sign emerges during normalization.
1. The Basic LRP Formula The fundamental propagation of relevance can be expressed as: where is the contribution score and is the relevance score.
2. Conceptual -rule When separating the positive and negative contributions, the rule is commonly conceptualized in literature as: (This is the shorthand form that most authors and reviewers are familiar with.)
3. Explicit Normalization of Contributions (-rule) However, to rigorously analyze the structural interaction of our "minimal probe," we cannot rely on the abbreviated form. We must explicitly unroll how the raw contributions () are normalized.
Let be the raw contribution. The normalized positive and negative contribution scores ( and ) are defined by explicitly factoring out the negative sign of the inhibitor to represent the "share" as a positive fraction:
4. Final Expanded Formulation By substituting these explicitly signed normalizations back into the conceptual equation, we get the fully expanded formulation:
This fully expanded form precisely matches the "" formulation presented in our Equation 5. Because the negative sign is mathematically extracted during the normalization step, applying the condition (with ) does not violate total relevance conservation.
We fully concede that due to space constraints, our brief definition of () right before Equation 5 was too condensed and omitted this crucial step-by-step sign separation, making it difficult to bridge the gap from the basic formula. We will explicitly add this full derivation in the revised manuscript to ensure the mathematical rigor is clear to all readers.
Thank you again for your careful reading and for helping us identify this missing theoretical bridge.
Dear Reviewer mrLk,
We sincerely thank you for your deep and highly critical examination of both our mathematical premises and experimental details. Your sharp feedback has been invaluable in helping us identify and correct overstated claims and omitted prerequisite conditions. We address your critical concerns below.
Q1 & Q2 We apologize for not explicitly detailing the normalization prerequisites of the Z-rule in the main text, which understandably led to confusion regarding Equation (5) and the noise propagation dynamics. However, we must respectfully point out that the standard LRP αβ-rule operates under the condition of α + β = 1, not α - β = 1 as you suggested. Furthermore, our implementation strictly adheres to the standard prerequisite where the sign of the inhibitor is explicitly normalized in the Z-rule denominator (i.e., z−→−Z=−z−/∑z−). Therefore, the expectation of sign flips based on the α - β = 1 assumption is rooted in a misaligned mathematical premise. Regarding Figure 2, θ is not tautologically forced to zero by construction. The θ values are empirical observations derived via Equation (9), based on the actual activation counts (P and M) during the forward pass. The core structural finding of our paper is not that we mathematically force θ to converge to zero. Rather, it is that by aligning the trajectory via the π-rule, we successfully suppress the variance of the accumulated noise, thereby empirically demonstrating that the previously ignored 0th-order bias term exerts a fatal, destructive impact on the actual propagation process.
Q3 We completely agree with your observation that the initial ratio of the ϵ-rule demonstrates strong performance in specific configurations (Table 2). However, the primary contribution of the π-rule is not to marginally outscore every baseline on every metric, but to provide fundamental robustness against heuristic parameter tuning. While the standard αβ-rule requires exhaustive searches to find optimal ratios, the π-rule fundamentally neutralizes structural noise, significantly reducing performance fluctuations across different parameter settings. We gladly accept your constructive suggestion and will add a comparative experiment applying the π-rule's effective ratio (αM:βP) to the standard αβ-rule in our revision, which will further highlight this robustness.
Q4 We fully concede your point here. Characterizing the ViT results as "marginal improvements" was an inappropriate overstatement. As you correctly noted, the performance declined across all 12 comparisons, representing a clear limitation of our current methodology. As briefly discussed in our "Boundary of Validity" section, the minimal probe assumption is insufficient to capture the highly non-linear residual dynamics (such as GELU) present in Transformer architectures. We humbly accept your critique: we will immediately remove the overstated claims from the Abstract and explicitly document the performance degradation in ViTs as a core limitation. Furthermore, we will add the SGLRP results to Table 1 with paired confidence intervals, exactly as requested.
We are deeply grateful for your rigorous review, which will unequivocally improve the honesty, precision, and soundness of our final manuscript.