acceptodds
Under review as a conference paper at ICLR 2027

A Framework for Analyzing DNN Representation Quality: Dynamics of Interaction Generalizability

Abstract

This paper addresses the core challenge in the field of symbolic interaction, i.e., how to define, quantify, and track the dynamics of generalizable and non-generalizable interactions encoded by a DNN throughout the training process. Specifically, this work builds upon the recent theoretical achievement in explainable AI, which proves that the detailed inference patterns of DNNs can be strictly rewritten as a small number of AND-OR interaction patterns. Based on this, we propose an efficient method to quantify the generalizability of each interaction and discover distinct three-phase dynamics of the generalizability of interactions during training. In particular, the early phase of training typically removes noisy and non-generalizable interactions and learns simple and generalizable interactions. The second and the third phases tend to re-learn increasingly complex interactions that are harder to generalize. Experimental results show that the dynamics of eliminating and re-learning non-generalizable interactions are temporally aligned with the dynamics of the training-testing loss gap.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Generalization Analysis, Learning Dynamics, Deep Learning Theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_SVapOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to quantify the generalizability of interactions, where by interactions they mean the XAI technique to represent a model's behaviour via combinations of AND and OR operations. This approach is shown to yield insights into the training-time dynamics of CNNs and small Transformer models.

Critical strengths

The weakness described below makes it very hard for me to formulate a list of critical strengths.

Critical weaknesses
  • W1: The paper is extremely hard to follow, and it is very poorly structured. This is suggestive of heavy reliance of AI tools for writing purposes. For instance:

    • The background is inserted in the middle of the Introduction, which makes everything hard to follow
    • Some pieces of text are highlighted with a strange grey vertical bar, which is unusual and apparently useless
    • The paper relies on a video demo to explain one key component of the proposed method, which, other than being unusual, is also detrimental to the flow of the paper, given also the poor quality of the video
    • The main question highlighted in the Introduction feels detached from the abstract, and it is only later contextualized with it.
  • W2: Authors did not release the source code.

  • W3: The precise meaning of AND and OR interactions is never discussed, and their precise meaning remains ambiguous. Also, the procedure for extracting them is confined to the Appendix, despite being key to the paper.

  • W4: The authors say that an interaction is generalizable if the baseline DNN concurrently encodes the same AND interaction (line 242-243). I do not particularly agree as different DNNs may encode alternative interactions, and authors do not discuss how their approach handles them. Also, the two DNNs are trained on different data, hence the same interactions may not emerge due to different training regimes.

Questions for authors

N/A

Reviewer_N8XFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to address the question of how to quantify generalizability of symbolic interaction in an effort to address the interpretability or DNN model explainability, based on some previous studies especially (Ren, et al, 2024). The assumption here is that the representation learnt by DNN can be rewritten as a small number of AND-OR interaction patterns, which helps with explainability. The authors argued that model quality can be interpreted through changes in the inference patterns and its generalizability.

Critical strengths

I find the paper tends to rely too heavily on a few previous studies, and assumptions, hence its contributions are barely minimum. Furthermore, I find that the assumption of a series AND-OR interaction patterns (as shown in Fig. 1) would provide significant benefits for model transparency is a weak one. Since for DNN model it is likely that you would have a massive if AND-OR logic model, which would not be helpful in terms of enhancing explainability. In some way, this is similar to Genetic Programming where we could have produced a very large rule-based tree model, as an equivalent to a machine learning model, but in this case, GP is still not easily interpretable due to its large size. I find this line of thinking of using a classic rule-based system to explain modern deep learning model is quite naive one.

I find the paper is not easy to follow, and it is not very well motivated as to why the proposed model is going to make any difference in the area of explainability.

Critical weaknesses

Research is not well motivated.

Reading not easy to follow.

At most, the contributions are probably very minor.

Two metrics are introduced in Equation (3), but the authors did not explain why these are really new contributions to help us interpret better a trained DNN model, and its predicted result??

page 7: "Phase 2: As training progresses, the average generalizability of interactions H gradually decreases, and the number of interactions N usually increases". Here, how would you measure "generalizability"? and during the training phase? It is not easy to understand what is going on, plus you mentioned "average generalizability", which could be a poor indicator, as you used "averaged" value.

Questions for authors

Q1: Why a massive AND-OR logic model would help with overall DNN model explainability?

Q2: Two metrics are introduced in Equation (3), but the authors did not explain why these are really new contributions to help us interpret better a trained DNN model, and its predicted result??

Q3: page 7: "Phase 2: As training progresses, the average generalizability of interactions H gradually decreases, and the number of interactions N usually increases". Here, how would you measure "generalizability"? and during the training phase? It is not easy to understand what is going on, plus you mentioned "average generalizability", which could be a poor indicator, as you used "averaged" value.

Reviewer_3vxAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the generalizability of the interactions in a deep network evolves during training. The authors follow the AND-OR interaction framework: a network's output can be approximated by a few interactions over combinations of input variables. Within this framework, the authors turn it into a testable transfer criterion: a second network is trained separately on the other half of the training set, and if it also encodes the same interaction with the same direction, the interaction is regarded as generalizable.

This paper gives a measure and a finding. The measure implements the criterion as a single interaction-extraction pass, and characterizes the whole training run with two quantities: H is the share of generalizable strength, and N is the number of interactions needed to cover 80% of the strength. This paper finds that the training process follows a three-phase trajectory: noise is removed first and simple generalizable interactions are settled on, after which increasingly complex and increasingly hard-to-generalize interactions are added. The authors claim that this trajectory coincides in time with the widening of the training-testing loss gap. Two interventions also support this account: removing non-generalizable interactions reduces the loss gap, and penalizing them improves accuracy.

This paper validates the above results on four datasets and six models, and separately compares two large language models against each other on SQuAD. The subtraction intervention removes non-generalizable interactions and then recomputes the loss; the penalization experiment observes a rise in the generalizable share and in accuracy. The authors state that the results are only empirical observations, that the mathematical explanation is left as future work, and that their compute cannot train a language model from scratch.

Critical strengths
  1. This work turns a definition that was hard to compute into a quantity that can be computed. The original definition requires an interaction to recur in unseen samples with consistent effects, which amounts to checking every one of an exponential number of input combinations; the authors replace it with whether the interaction transfers to another network trained only on held-out data, with the same effect direction (Eq. 3), so that the generalizability of each interaction can be read off from a single interaction extraction. This change takes representation quality from an abstract statement at the model level down to specific combinations of input variables, and gives all later analysis a unified computational footing.

  2. It does not stop at surface descriptions such as "the network learns more complex patterns late in training", but characterizes the whole training run with two quantities and gives the phase boundaries in quantitative form. H and N measure the share of generalizable strength and the number of interactions needed to cover 80% of that strength, and across the four datasets and six supervised models examined in the paper, the two move consistently in opposite directions: H rises and then falls, while N falls and then rises. Appendix H further states the phase boundaries as explicit conditions on the maximum of H and on its subsequent change, rather than judging the turning points of a curve by eye. For a paper that is mainly empirical, turning an observation into a checkable definition has value in itself.

  3. The paper's validation is interventional rather than correlational. It does not stop at "the loss gap is large when there are many non-generalizable interactions", but removes the contribution of non-generalizable interactions from the output and checks whether the loss gap then narrows; Appendix J further turns this idea into a penalty during training and reports an accuracy gain over not adding it. Even if each of these two steps still leaves room for deeper analysis, this idea of removing the term and seeing whether the result moves as expected supports a mechanistic claim better than a correlation coefficient alone, and is easier for later work to reuse.

Critical weaknesses
  1. This paper does not compare itself with the closest existing work. Three claims, namely the three-phase dynamics, the correlation between metrics and generalization, and the transferability measure, have all been made before in Arpit et al. (ICML 2017), Kalimeris et al. (NeurIPS 2019), Jiang et al. (ICLR 2020) and methods such as CKA, and the paper cites none of them. More directly, the Zhang et al. (2021) that the paper itself cites already gives an interaction-level account of overfitting, yet Appendix I.2 still attributes that mechanism to this paper. The paper claims a new framework, but with these three claims already having precedents, the only increment that can be settled is per-interaction quantification, and the paper does not say what it adds over existing work.

  2. The three-phase dynamics is the paper's main conclusion, yet the evidence behind it is insufficient. For each model only four time points are taken, with no indication of which epochs they correspond to, and each is a single training run; no seed, variance or error bar is reported, so it is impossible to tell whether the shape of the curve is a general property or the fluctuation of a single run. The training hyperparameters are not given at all, so the experiments cannot be reproduced. The paper treats it as a general conclusion across models and datasets, but the experiments cover only six supervised models, and on the LLM side there is only one snapshot, still in the first phase. Supporting it would require repeated training and a reported spread.

  3. The paper presents "penalizing non-generalizable interactions improves accuracy" as a practical value, but the experiment behind it has no baseline. Appendix J compares only adding the penalty against not adding it, and places no existing regularization method under the same setting; the value of the penalty coefficient λ and the way it was tuned are not reported, nor are seeds or variance. With this design the paper can show only that the term helps, not that it is better than existing methods, and the latter is what the practical value of this work rests on.

  4. The paper claims that non-generalizable interactions cause overfitting and that suppressing them improves performance, but this chain is never tested in full. The subtraction experiment in §3.2 reports only the loss gap, the penalization experiment in Appendix J reports accuracy, and the two use different models and data. They are also two different interventions, and the loss gap and accuracy are never measured together. The paper therefore does not rule out a direct objection: the accuracy gain may be only the regularization effect of the penalty term.

Questions for authors
  1. If the per-interaction localization of generalizability is taken out on its own, which conclusions can it support that existing methods cannot? Please give one concrete judgment, and say whether χ can predict the loss gap better than CKA or H-score on the same checkpoints. If it cannot, please say which layer the increment of this work falls on. If this comparison is to be added, what extra computation does it require, and is the cost acceptable? The paper itself cites Zhang et al. (2021), which already gives an interaction-level account through the effect of dropout, yet Appendix I.2 still presents the demonstration that non-generalizable interactions are a key driver of overfitting as its own conclusion; please say what the increment over that work is. If these answers all point to a limited increment, please consider how the contribution of this work should then be increased.

  2. Please say under what conditions the three-phase dynamics holds and under what conditions it fails. If training is repeated with three to five random seeds on the same model, do the number of phases and the two boundaries t1 and t2 change? If the time points are refined from four to more than ten, is the three-phase shape still clear, or may two adjacent phases merge? Under what architecture, scale or initialization does the regularity no longer appear? Also, are there models or datasets in the experiments that do not show three phases? If so, reporting these negative cases would say more about the boundary of the regularity than reporting only positive ones. The paper says training follows standard settings commonly used in practice; please give the full hyperparameters, learning rate and random seeds so that the experiments can be reproduced. Should the three-phase conclusion also apply to LLMs? If not, please state the scope of the conclusion explicitly in the main text.

  3. Please give the setting in which "penalizing non-generalizable interactions improves accuracy" really holds, for example at what data scale, degree of overfitting and compute budget it is better than established methods such as weight decay, dropout and early stopping. Also, under what conditions is it not worth it or does not work at all? Does every step require an extra interaction extraction, and how large is that cost relative to a single forward pass? How exactly should the penalty coefficient λ be chosen, is the result sensitive to its value, and is there a selection rule that does not tune on the test set? If it is not better than existing regularization in most settings, please make clear what the standing of this conclusion should be.

  4. Please give an experiment that argues the chain "non-generalizable interactions cause overfitting and suppressing them improves performance" through in one go, for example by recording the generalizable share, the training-testing loss gap and test accuracy in the same training run, and by performing both the subtraction and the penalization interventions on the same models and data, to see whether the three move together. If a full experiment is too costly, you could at least say what you would expect to see if accuracy were also recorded in the subtraction experiment of §3.2. On what grounds? Also, the mechanism evidence in Appendix J, a generalizable share rising from 28% to 39%, comes from CIFAR-10 with VGG-11, while the accuracy gains come from three further settings; can you explain why the two batches of settings differ, and whether the same holds under one setting? Also, how would you rule out the counter-explanation that the accuracy gain comes only from the intrinsic regularization effect of the penalty term? For example, by applying the same penalty to a random subset of interactions with a matched effect size, or by using a penalty unrelated to generalizability with equal strength, would the gain still hold? If the accuracy gain can be reproduced without any change in generalizability, you may need to reconsider how much of the mechanistic conclusion of this work remains.

Reviewer_smnpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper build a AND-OR framework that aim to explain the mechanisms of LLM over AND-OR interaction model and try to establish a systematic view of the learning process. Specifically, the paper proposes a three-phased dynamics of the training process, which includes 1) removal of non-generalizable interaction + gaining of some generalizable ones; 2) learning more complex yet not generalizable interaction; and 3) learning complex and generalizable interactions.

Critical strengths
  • Experiments cover several architectures (CNNs, BERT) and both vision and NLP tasks.
Critical weaknesses
  • The paper is very hard to read. I was often lost following the main argument. Key terms (“interaction,” “generalizable,” “order,” the phases) are explained through long chains of references to earlier work in the same line, so the paper is hard to follow without reading those first. Figures 1–3 are dense, with tiny unlabeled axes, and it’s not clear what each bar or color represents without careful decoding. The writing also relies on strong claims (“it has been proven,” “the healthiest stage of training”) instead of clear explanations.

  • The three-phase structure feels tautological and lacks real implications. An interaction is called non-generalizable when a model trained on held-out data does not share it. So it is expected that these interactions grow as the model overfits and the train–test gap widens; the alignment largely follows from the definition. The phase boundaries are also defined after the fact from the same quantities (Appendix H), and the “alignment” is shown on four hand-picked timepoints. The paper does not show that the phases predict anything we can’t already see from the loss curves, for example detecting overfitting earlier than the validation loss.

  • Core findings were already established but are not cited. Spindle-shaped distribution at initialization (Sec. 3.1, “Before Training”; Fig. 3, timepoint A): Zhang et al., arXiv:2405.10262, 2024; Ren et al., arXiv:2407.19198, 2024. Phase 1 removes mid/high-order noise and learns low-order interactions (Intro, Phase 1 bullet; Sec. 3.1, Phase 1; Fig. 3, A→B): Zhang et al., arXiv:2405.10262; Ren et al., arXiv:2407.19198. Later phases learn increasingly high-order interactions, leading to overfitting (Intro, Phase 2–3 bullets; Sec. 3.1, Phases 2–3; Fig. 3, B→D): Zhang et al., arXiv:2405.10262; Ren et al., arXiv:2407.19198. The dynamics align with the train–test loss gap (Abstract; Sec. 3.1, Finding 3; Fig. 3; Conclusion): Zhang et al., arXiv:2405.10262. Low-order interactions generalize better than high-order ones (Sec. 2.2.2; Fig. 2): Zhou et al., AAAI 2024 (cited, but the result is presented as a new verification). Generalizability as interactions shared with another independently trained model (Sec. 2.2.1, Eq. 3): close to Chen et al., ICLR 2024 (cited only for universal matching, not as prior work for the metric).

  • The findings are not standardized and can’t be compared or acted upon. The generalizability H depends on many arbitrary choices: the 10 input variables chosen per sample (random words, or patches from the first ReLU layer); the masking baseline; the top-k threshold; the number of baseline models. Interaction strengths are not normalized in the main metric (only in Appendix M), there are no error bars across seeds or samples, and the timepoints are picked by hand. So it’s unclear whether the numbers can be compared across models or tasks, or used in practice, e.g. as an early-stopping signal.

  • The experimental details are confusing. It’s not stated which checkpoint of the baseline DNN is used, or how many samples the dynamics in Fig. 3 are computed over. Eq. (3) uses one baseline DNN, while Appendix D says k = 2 is used in experiments. The model–dataset pairs are inconsistent: Sec. 3.2 lists BERT-large on AG News, Fig. 3 shows BERT-medium on AG News, and Appendix E says both were trained on AG News. Appendix E.2 refers to a “Section 3.3” that doesn’t exist. In the LLM experiment, Qwen is used as the “baseline” for DeepSeek, which isn’t trained on held-out data, so it doesn’t follow the method’s own definition.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.