- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Tensor Network Datamodels (TNDMs), using CP and TT representations to predict model outcomes from training-data subsets. Its central motivation is that complementary or redundant training groups can induce high-degree interactions while retaining low tensor rank. Experiments evaluate subset prediction, attribution, interaction recovery, and data selection on controlled tasks and pretrained-model fine-tuning.
Critical strengths
- The paper addresses a meaningful limitation of additive training-data attribution: the utility of a training group can depend on which other groups are present. The distinction between interaction degree and tensor rank provides a useful motivation for modeling these dependencies compactly.
- The controlled experiments provide encouraging evidence, including improved subset prediction and interaction recovery. Appendix J.15 is particularly interesting: interactions change from synergy to redundancy during training, while non-additive effects remain compressible. These observations suggest useful directions for studying training-data dependencies.
Critical weaknesses
The evidence connecting the proposed structural explanation to practical benefits on natural data remains incomplete. The paper acknowledges its relationship to TN-SHAP and established tensor representations, making the training-data-specific insights particularly important. Sport–Music shows improved prediction, but does not clearly establish a distinctive higher-order interaction benefit over quadratic models. On AG News, cross-budget comparisons favor degree-truncated and RBF baselines. Moreover, the dynamics in Appendix J.15 come from exhaustive enumeration on a controlled task, rather than recovery by a sparsely fitted TNDM. Thus, the paper does not yet establish when the proposed structure reliably occurs and translates into better data interventions.
Questions for authors
What evidence supports the connection between higher-order, low-rank structure and the observed gains in the fine-tuning experiments? In particular, do existing results demonstrate improved joint-removal prediction or data selection against strong nonlinear baselines on natural data? Please distinguish demonstrated results from proposed future experiments.
2 comments
Follow-up
Thank you for the helpful review. Changes are in blue in the revised PDF.
- When the structure appears: Appendix A.6 shows that complementary and interchangeable groups give TT ranks at most two but exponentially many coefficients in sparse bases.
- Effect on interventions: the four-part relation learner has this setup. TT lowers prediction/removal error from 0.251/0.131 (ProxySPEX tree) to 0.0046/0.0061 and recovers Shapley values and interactions more accurately in all three learner seeds (Table 24).
- Natural data: on AG News classifier refitting, TT has the lowest two-group removal error at 256 and 512 observations (0.144 and 0.094 vs. 0.575 and 0.524 for ProxySPEX; Table 25), and the lowest selection regret at sizes three and five on RoBERTa–LoRA (Table 26).
- Higher-order terms: removing CP terms above degree two lowers Sport–Music from 0.668 to 0.634 (Table 23).
We hope to have addressed your concerns and answered your questions. We remain available if you have any more questions during the end of the discussion period.
If we properly addressed your concerns and questions, we kindly ask you to consider raising your score.
We thank the reviewer. The revision addresses expressivity, spurious interactions, and training order.
1. Definition 3 and prior work. We agree that fitting Definition 3 is a standard low-rank tensor completion problem. Our contribution is a principled, unified formulation of datamodeling that makes degree bounds, sparsity, and low rank explicit (Section 2), and an analysis of why low rank suits training-data dependencies (Section 3): complementary and interchangeable groups can generate high-order interactions with compact tensor representations. We show that settings where the data consists of complementary and redundant groups is where linear datamodels fail, where TNDM represent the training data model compactly. Appendix A.6 makes the distinction from sparsity precise. An expansion with s nonzero coefficients in any product basis has CP rank at most s. Our coverage construction with m required relations has TT ranks at most two under a paired ordering, but CP rank 2m, requiring at least 2m coefficients in every exact product-basis expansion (Theorems 5–6). Section 4.4 and Table 23 connect this structural motivation to the eight-hop transformer experiment.
TN-SHAP and TN-SHAP-G [1,2] are specifically focused on Shapley values and interactions and explain predictions from a fixed model. TNDMs predict outcomes after training on unseen data subsets. Park et al. [3] develop tensor-product models for functional ANOVA. We connect training-group dependencies to tensor rank and test prediction, joint removals, and attribution, building on [1,2] for attribution computations (Sections 2, 3, and 5).
2. Additive expressivity and spurious interactions. We agree that an additive function with d nonzero effects can require CP rank d, whereas TT represents it with ranks at most two. Appendix K.3 tests whether TNDMs assign interactions to noninteracting groups on the Figure 4 retraining game, where 22 of 28 pairwise and 52 of 56 third-order interactions are zero. At 64 queries, TNDM (CP) has the smallest spurious interactions (Section 4.4; Appendix K.3, Table 27) among the compared methods: normalized RMS 0.092 for pairs and 0.161 for triples, versus 0.451 and 0.461 for KernelSHAP-IQ. Its false-positive rates are 2% and 6%, versus 29% and 27%. TNDM (TT) gives 0.189 and 0.239 (Table 27).
3. Model scale and training regimes. Retraining cost, not the datamodel, limits the number of groups: each observed subset requires one training run. The fitted TNDM grows linearly with the number of groups, and the 24-group reasoning experiment (Section 4.1; Table 1) predicts outcomes over 224 subsets from at most 1,024 observed runs. On Qwen2-7B fine-tuning (Section 4.3; Tables 2 and 23) with ten optional groups and fixed background data, CP improves held-out R2 from 0.565 for quadratic regression to 0.668, and removing terms above degree two lowers it to 0.634 (Appendix K.2, Table 23).
4. Training order. A datamodel describes outcomes under a fixed training procedure, including example order, or averages over that randomness. We add a controlled experiment to training order ablations and a dedicated paragraph at the of the Appendix K.3. In a batch-order control, reversing or permuting the order changes negative held-out cross-entropy by 0.0033 and 0.0021 on average (Appendix K.3). AG News uses seeded shuffling; the Figure 4 learner uses full-batch training, which has no example order.
References
[1] Farzaneh Heidari, Chao Li, and Guillaume Rabusseau. Tractable Shapley Values and Interactions via Tensor Networks. AISTATS, PMLR 300:3043–3051, 2026.
[2] Farzaneh Heidari and Guillaume Rabusseau. TN-SHAP-G: Graph-Structured Tensor Network Surrogates for Shapley Values and Interactions. ICML, PMLR 306, 2026.
[3] Seokhun Park, Insung Kong, Yongchan Choi, Chanmoo Park, and Yongdai Kim. Tensor Product Neural Networks for Functional ANOVA Model. ICML, PMLR 267:48041–48085, 2025.
The comparisons show that no single method is best across all evaluated settings. Our contribution identifies a complementary regime where training-data dependencies admit compact low-rank representations despite requiring many interaction terms in a product-basis expansion (Section 3; Appendix A.6). The eight-hop transformer experiment illustrates the value of this structure (Section 4.4; Table 23), while AG News classifier refitting demonstrates improved joint-removal prediction (Section 4.3; Table 25). Figure 4 additionally shows that CP provides accurate individual and higher-order attributions from one fit, with errors competitive with ProxySPEX at 64 observations.