acceptodds
Under review as a conference paper at ICLR 2027

Learning Structurally Distinct Factors in Many-Body Approximation for Tensors

Abstract

Decomposing tensors in an interpretable manner is essential in data analysis and knowledge discovery. Many-body approximation (MBA) for non-negative tensors is a tensor decomposition method that allows each factor to be interpreted as an interaction among a subset of modes of a given tensor. In this paper, we first uncover a structural dependency in MBA: factors containing common modes necessarily share identical slices, regardless of the model parameters. To remove this structural dependency while retaining the convex optimization procedure of MBA, we propose extended many-body approximation (EMBA), which extends each mode of the input tensor by one dimension before applying MBA. We prove that entries selected from different EMBA factors are not identical as functions of the parameters, thereby eliminating the structural dependency present in MBA. Experiments on reconstruction using reduced set of factors and recursive approximation show the improved explanatory capacity and robustness of EMBA over MBA. In factor quantization experiments, EMBA almost perfectly reconstructs synthetic data generated from Boolean factors even under 1-bit quantization. These results demonstrate that EMBA enables structurally distinct factors whose utility extends beyond full-tensor reconstruction, highlighting the importance of factor structure in tensor decomposition.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
tensor decomposition, interpretable representation learning, structured representation learning, nonnegative tensor factorization, information geometry

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_LsdhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper concerns many-body approximation (MBA), the recent method for factorization of non-negative tensors, and tackles its structural challenge of dependent factors, induced by the design of the method. This dependence is induced by structure of the MBA method and paper proposes an extension of the method (EMBA), that adds a zero-filled boundary to every tensor mode, applies MBA, and extracts the factors corresponding to the original domain. The authors prove that entries from different extracted factors are not identical functions of the parameters and thus proposed EMBA does not share the 'factor dependency' of the original MBA. Experiments evaluate the resulting representations under factor deletion, recursive approximation, and quantization.

Critical strengths

Paper is well and clearly written, timely and potentially influential in removing the structural gap in recently proposed MBA method for non-negative tensor factorization. Evaluating the usefulness of individual factors beyond complete reconstruction is thus worthwhile and enlightening. The construction is simple to implement, and the theoretical results appear correct.

Empirically, the reported quantization improvements are substantial: at equal stated storage, EMBA reduces COIL100 KL divergence relative to MBA by approximately more that 20%20\% at both, two bits as well as in one bit, quantization regimes (Table 2). Factor-deletion improvements also extend to traffic and EEG data, although the EEG advantage is not uniform across deletion patterns.

Critical weaknesses
  1. Missing bounds\textbf{Missing bounds}: Proposition 4.3 (and the paper overall) proves a non-inrease of the KL divergence when using EMBA DKL(T(extend)∣∣P(MBA))≥DKL(T(original)∣∣P(EMBA))D_{KL}(\mathbf{T}^{(extend)} || \mathbf{P}^{(MBA)}) \geq D_{KL}(\mathbf{T}^{(original)} || \mathbf{P}^{(EMBA)}) compared to MBA on extendedextended tensor yet leaves the relation to original MBA, i.e., DKL(T(original)∣∣P(MBAorig))D_{KL}(\mathbf{T}^{(original)} || \mathbf{P}^{(MBA_{orig})}), where P(MBAorig)\mathbf{P}^{(MBA_{orig})} denotes MBA solution to original tensor approximation, unspecified.

This is also stated on lines 316−317316-317: "... The proposition does not compare EMBA with MBA fitted directly to the original tensor; this comparison is evaluated empirically in Section 5. ...". Consequently, Proposition 4.3 does not provide a direct guarantee relative to MBA fitted to the original tensor.

It seems that a comparison of the optimal infima might follow by considering zero-extended original MBA distributions as limits of distributions in the extended family. I.e. It directly follows from infimum over the larger set (the zero-extended original MBA distributions can be approached within the extended family). Could the authors clarify this relationship? Formalizing it would clarify whether EMBA changes the factor representation alone or also improves the optimal reconstruction KL.

Anyway, such a comparison of infima would not automatically establish an ordering between the finite numerical solutions returned by the algorithms. So if the above turns useful, the authors should distinguish these two levels of comparison and clarify what guarantees apply to the numerical solutions.

  1. Insufficient characterization of practical benefit relative to computational cost.\textbf{Insufficient characterization of practical benefit relative to computational cost.} With all factors retained and un-quantized, MBA and EMBA achieve identical or practically identical reconstruction errors reported in Table 2 and the “no deletion” rows of Tables 6 and 7. On the other hand EMBA demonstrates benefits under factor deletion, recursive approximation, and quantization, including improved compressed reconstruction. Nevertheless, the practical trade-off remains insufficiently characterized: the scalability experiments report up to 6.3x longer fitting time, while real-data quantization is evaluated on a single tensor constructed from ten COIL100 images. Reporting fitting costs alongside compression results on broader datasets would clarify when the additional computation is justified.

The empirical improvements are promising, but the theoretical criterion seems weaker than the interpretation attached to it. I lean toward weak reject pending a clearer treatment of boundary optimization, and stronger evidence connecting factor robustness to the claimed benefits.

Questions for authors

Please address weaknesses 1 and 2 described above.

Additionally, please clarify how exact zero padding is handled in the natural-parameter (information geometric) formulation and its implementation. In particular, how is the logarithm-based initialization described in Appendix D.3 applied to the zero-padded tensor, given that (log⁡0\log 0) is undefined? Is any smoothing or special treatment of zero entries used?

4 comments

We thank the reviewer for the careful reading and constructive feedback.

W1. Relation between EMBA and original-domain MBA

We agree with the reviewer's observation. We have added two propositions establishing that (1) MBA and EMBA represent exactly the same family of normalized distributions on the original domain, and (2) the infima of the reconstruction KL divergences for MBA on the original and zero-extended tensors coincide.

Specifically, let P(θ)P(\theta) denote an MBA reconstruction on the original domain, PMBA(θ~)P^{\mathrm{MBA}}(\widetilde{\theta}) an MBA reconstruction on the zero-extended domain, and PEMBA(θ~)P^{\mathrm{EMBA}}(\widetilde{\theta}) its normalized restriction to the original domain. The reconstruction families coincide because extended MBA parameters contributing to the same original-domain interaction can be combined into original MBA parameters, while any original MBA parameterization can also be embedded into the extended domain.

Let m(θ~)m(\widetilde{\theta}) denote the total probability mass of the extended MBA reconstruction on the original domain. Then

DKL(Textend∥PMBA(θ~))=DKL(Toriginal∥PEMBA(θ~))−log⁡m(θ~).D_{\mathrm{KL}}(T^{\mathrm{extend}}\|P^{\mathrm{MBA}}(\widetilde{\theta})) = D_{\mathrm{KL}}(T^{\mathrm{original}}\|P^{\mathrm{EMBA}}(\widetilde{\theta}))-\log m(\widetilde{\theta}).

Since m(θ~)≤1m(\widetilde{\theta})\leq 1, the infimum of the extended-domain KL divergence is at least that of original-domain MBA. Conversely, as the reviewer suggests, any original-domain MBA reconstruction can be extended so that its normalized restriction remains unchanged while the probability mass on the padded entries approaches zero. This establishes the reverse inequality and hence equality of the infima.

Thus, EMBA changes the factor representation without enlarging the reconstruction family or improving the optimal reconstruction KL divergence. As the reviewer notes, equality of the infima does not guarantee an ordering between finite numerical solutions. In our experiments, MBA and EMBA reconstructions agree within the stopping tolerance. We have included the general proofs in the appendix. We have also revised the Introduction, Related Work, and Conclusion to explicitly position EMBA as a method for obtaining structurally distinct factor representations within the same reconstruction model family as MBA.

W2. Computational cost and broader experimental evidence

We agree that the computational cost of EMBA should be evaluated alongside its benefits. To strengthen the empirical evidence, we have added experiments on ten additional COIL-100 tensors constructed using different image selection patterns, evaluating factor deletion and quantization under the same settings as in the paper. We have also extended the quantization experiments to EEG and METR-LA.

The 6.3-fold increase in fitting time is the maximum observed in our scalability experiments. These experiments show that MBA and EMBA exhibit similar scaling trends, with their fitting times differing by an approximately constant factor over the tested tensor sizes. Meanwhile, Figure 6 in Appendix F.3 demonstrates substantial improvements in quantized reconstruction quality relative to storage. Although this figure evaluates the storage–accuracy trade-off rather than the computation–accuracy trade-off, the results suggest that EMBA's additional fitting cost can be worthwhile when quantization robustness is important.

We will include a more detailed comparison of fitting times alongside compression results in the final version to further clarify this trade-off.

Q. Zero padding and initialization, App. D.3

We agree that the treatment of zero entries and boundary solutions requires clarification. Both MBA and EMBA allow zero entries in the input tensor, including Boolean tensors, and evaluate the KL divergence using the convention 0log⁡(0/p)=00\log(0/p)=0 (numerically, ϵ=10−12\epsilon=10^{-12} is added to both arguments). Since reconstructed distributions are strictly positive for finite natural parameters, a KL infimum may lie on the boundary and need not be attained at finite parameter values. In our experiments, we terminate the optimization when the residual falls below 10−510^{-5}. Thus, the reported results are obtained from finite numerical solutions rather than assuming that a boundary optimum is attained.

Regarding initialization, the description in Appendix D.3 was incorrect, and we have corrected it. Our implementation does not use the logarithm-based initialization: Newton's method starts from θ=0\theta=0 (the uniform distribution) for both MBA and EMBA. The input tensor enters the optimization only through its marginals η^\hat\eta, so the logarithm of a zero entry never appears, and no smoothing or other special treatment of zero entries is used.

Thanks to authors for updating their paper and adding the "reconstruction" and "KL infima" equivalence Propositions in response to the raised weaknesses.

I still have some concerns left, however. Firstly, I am not sure the equivalency claim also covers correctly the normalization of the tensor producing EMBA in (7). In particular, in the equivalence proof (Proposition 4.4) reparameterization argument " ... each extended natural parameter contributes to the log reconstruction through cumulative sums ..." in the appendix B both proven inclusions hold, but do not consider Normalization that is a final step when producing EMBA in (7) and thus it is not entirely clear to me that every extended parameterization is admissible ... Similar question arises in the case of proof of Proposition 4.5: how is normalization treated?

As for W2, I appreciate added experiments. They are leveraging COIL 100 dataset again however. Hence an experimental evidence of generalization of the method to other real world datasets stays thin.

I'll keep my score unchanged for now.

Thank you for reading our rebuttal and the new propositions. We agree that explaining how normalization is handled in Eq. (7) more explicitly would make the proofs clearer.

Regarding Proposition 4.4, we have revised the proof to make the role of normalization in Eq. (7) explicit.

For any learned extended-domain parameters θ~\widetilde{\theta}, we can construct original-domain MBA parameters θˉ\bar{\theta} using the cumulative-sum structure of MBA. On the original domain, extended parameters whose indices are all at most 22 contribute only a constant. Other parameters contribute to interactions involving only the modes whose extended indices exceed 22. These contributions can be combined into original-domain MBA parameters, including lower-order interactions. As a result,

e^c P_i(\bar{\theta}),$$ where $c$ is independent of $i$. Since $P(\bar{\theta})$ is normalized, the sum of $P'^{(\mathrm{EMBA})}$ is $e^c$. Normalization in Eq. (7) therefore gives $P^{(\mathrm{EMBA})}=P(\bar{\theta})$. Conversely, any original-domain MBA parameters can be embedded into the extended domain, and normalization of the restriction recovers the original MBA reconstruction. Thus, MBA and EMBA represent the same family of normalized distributions on the original domain. **Regarding Proposition 4.5**, the existing proof accounts for normalization through the identity $$D_{\mathrm{KL}}(T^{(\mathrm{extend})}\Vert P^{(\mathrm{MBA})})= D_{\mathrm{KL}}(T^{(\mathrm{original})}\Vert P^{(\mathrm{EMBA})}) -\log m,$$ where $m$ is the probability mass of the extended MBA reconstruction on the original domain. Since $0<m\leq1$, this identity, together with Proposition 4.4, establishes one direction of the infimum inequality. For the reverse inequality, we show that any original-domain MBA reconstruction can be preserved while making $m$ arbitrarily close to $1$. First, we embed arbitrary original-domain MBA parameters into the extended domain. We then set the one-body parameter at index $2$ to $A>0$ in every mode. By the cumulative-sum structure, this gives every entry in the original domain the same multiplier $e^{AN}$, while entries in the padded region receive smaller multipliers. As a result, the normalized restriction remains the original MBA reconstruction, and the probability mass $m$ on the original domain approaches $1$ as $A\to\infty$. Hence, $-\log m\to0$, giving the reverse inequality and therefore equality of the KL infima. **Regarding the experimental evidence**, as noted in our response to W2, our evaluation also includes real-world datasets other than COIL-100. Specifically, as mentioned in Section 5.1, Appendix F.2 evaluates factor deletion on METR-LA (traffic data) and EEG (brain signals). On both datasets, EMBA achieves lower mean KL divergence and higher mean cosine similarity than MBA for every number of deleted factors (Table 6). In addition, Table 10 in Appendix F.6 includes comparisons between MBA and EMBA on these datasets, including quantization experiments. Thus, our experiments cover multiple real-world datasets from different domains and evaluate EMBA under both factor deletion and quantization, rather than relying solely on COIL-100.

Thank you for the clarification and for adding the propositions to paper. This alleviates my W1concerns. And my apologies I missed extended the quantization experiments to EEG and METR-LA. It still keeps the experimental evidence rather narrow, but together with improved theory I will adjust my review score one notch up to 3. Thank you.

Reviewer_MPQkOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Many-body Approximation (MBA), a non-negative tensor decomposition in which, e.g., a fourth-order tensor is approximated by a product of lower-order factors, each of which is interpretable (X_{123} captures the interaction among the first three modes). The authors observe that in MBA the parameterization forces distinct factors to share identical slices (Prop. 3.1). They propose EMBA: pad every mode of the tensor with one extra (zero) slice, run MBA on the padded tensor, then restrict the factors to the original indices. The main theoretical claim, Theorem 4.1, is that two entries from distinct EMBA factors are no longer forced by the parameterization to be the same function of θ. The construction keeps the convex optimization structure of MBA. Experiments compare MBA and EMBA on COIL-100 images, METR-LA traffic and EEG data, under factor deletion, recursive re-approximation and low-bit quantization, plus a synthetic Boolean example.

Critical strengths
  1. The identified problem is real and clearly stated: the MBA parameterization forces identical slices across factors that share modes, which undermines the interpretability that motivates the method. The fix is simple, keeps the convexity of the original optimization problem, and the empirical pattern (same overall fit, factors more robust under deletion and quantization) is consistent across three real datasets.
Critical weaknesses
  1. The main theorem does not deliver what the paper needs. Theorem 4.1 proves that two entries from distinct EMBA factors are not identical as functions of θ, i.e. Y_i(θ) ≢ Y_j(θ). This says nothing about the learned factors: it does not imply Y_i(θ̂) ≠ Y_j(θ̂) at the optimum, nor any form of factor identifiability. Removing a forced equality is not the same as guaranteeing distinct or identifiable factors, yet the abstract and introduction present the result as "eliminating the structural dependency". As stated, the theorem is nearly vacuous, since it holds as soon as the two entries differ in a single parameter, and it does not explain the empirical gains. I consider this the most serious problem in the paper.

  2. What EMBA actually changes is not made precise. In every reported setting the fitted tensor is identical for MBA and EMBA; only the split among factors changes, which is a reparameterization (a change of reference level in a log-linear model). The paper should state and prove this rather than "evaluating it empirically" (Sec. 4.2.3), and should compare against the zero-cost alternative of re-gauging the original MBA factors post hoc. The zero-mass padding slice also sits outside the positive-tensor setting in which MBA is defined, which likely explains the 12–14 vs 3 Newton iterations and is not addressed.

Taken together, the paper's theoretical claim does not support its stated contribution, and its method is a reparameterization of an existing model presented as a new one. Addressing these would require a different paper rather than a revision.

Questions for authors
  1. Can you give any statement about the learned factors at the optimum, for example conditions under which Y_i(θ̂) ≠ Y_j(θ̂), or an identifiability result? Without this, what does Theorem 4.1 guarantee in practice?
  2. Please confirm whether the fitted tensor P is always identical for MBA and EMBA. If so, state EMBA as a reparameterization and compare it with post-hoc re-gauging of MBA factors.
  3. How are zero entries (the padding slice, Boolean zeros, missing values) handled in the KL objective, given that MBA is defined for strictly positive tensors? Is the optimum attained?
  4. Please report seeds and significance for Tables 1 and 3, and the numbers behind Fig. 4.
9 comments

We thank the reviewer for the careful reading and constructive feedback.

Q1. Learned factors at the optimum

Theorem 4.1 addresses equalities imposed by the parameterization. In Proposition 3.1, corresponding MBA slices are identical for every parameter value. EMBA removes this unconditional requirement.

We can further characterize exactly when the corresponding EMBA entries are equal. Equality holds if and only if the fitted natural parameters satisfy a specific linear equation, whose coefficients are not all zero by Theorem 4.1. If this equation is not satisfied, the entries necessarily differ.

For example, consider the slices in Eq. 3. Here, Y uses original-domain indices, while the extended natural parameters use indices in which level 1 is the padding. Thus,

Y_{i,1}^{(1,3)}=\widetilde X_{i+1,2}^{(1,3)}.$$ Defining $$\Delta_i = \log Y_{i,1}^{(1,2)}-\log Y_{i,1}^{(1,3)},$$ the factor definition gives $$\Delta_i = \frac12(\widetilde\theta_{1,2,1}-\widetilde\theta_{1,1,2})+\sum_{a=2}^{i+1}(\widetilde\theta_{a,2,1}-\widetilde\theta_{a,1,2}).$$ The corresponding EMBA slices coincide if and only if their entries are equal for every index. For the first entry, substituting $i=1$ gives $$\frac12(\widetilde\theta_{1,2,1}-\widetilde\theta_{1,1,2})+\widetilde\theta_{2,2,1}-\widetilde\theta_{2,1,2}=0.$$ For each subsequent entry, $i=2,\ldots,I_1$, equality holds if and only if $$\frac12(\widetilde\theta_{1,2,1}-\widetilde\theta_{1,1,2})+\sum_{a=2}^{i+1}(\widetilde\theta_{a,2,1}-\widetilde\theta_{a,1,2})=0.$$ The entire slices coincide precisely when all these conditions hold simultaneously. These equations are independent: their coefficients on the parameters $$\widetilde\theta_{2,2,1},\ldots,\widetilde\theta_{I_1+1,2,1}$$ form an invertible lower-triangular matrix. More generally, consider the corresponding EMBA slices obtained by fixing all non-shared indices to 1 in the original-domain coordinates, as in Proposition 3.1. Equality of these slices is equivalent to a homogeneous linear system in the extended natural parameters. Its coefficient matrix has rank equal to the number of entries in either slice. We have added this result as a new proposition, with its proof in the appendix. Thus, MBA enforces these slice equalities for every parameter value, whereas EMBA permits them only when the stated parameter relations hold. The conditions apply directly to finite fitted parameters. We do not claim that optimization necessarily avoids them.

Q2. Identical reconstruction, reparameterization, and post-hoc re-gauging

We agree that EMBA can be viewed as a reparameterization of MBA. Both methods represent the same family of normalized distributions on the original domain and yield the same KL projection in the ideal optimization limit. In our experiments, the reconstructed tensors agree within the stopping tolerance.

In EMBA, the factors are computed from the extended natural parameters θ~\widetilde{\boldsymbol{\theta}}, followed by extraction of their original-domain entries. Alternatively, we can construct θˉ\bar{\boldsymbol{\theta}} with the same shape as those of the original-domain MBA from θ~\widetilde{\boldsymbol{\theta}}, such that the factors computed from θˉ\bar{\boldsymbol{\theta}} yield the same reconstructed tensor as EMBA. For example, in a third-order 2-body approximation,

\bar\theta_{r,s}^{(1,2)}=\widetilde\theta_{r+1,s+1,1}.$$ The three terms in the first expression contribute under the same condition on the original domain and can therefore be combined into a single parameter. The reverse inclusion also holds, and the padding mass can approach zero without changing the restricted distribution. We will add the general proof to the appendix. EMBA enables the interaction factors of MBA to be learned directly through KL projection, without the structural slice equalities imposed by MBA or the need for post-hoc re-gauging. Inversely, the standard re-gauging methods we tested do not, in general, recover the EMBA factors. MBA uses the first level of each mode as its reference. Choosing a different reference level only changes which slices are forced to coincide. Alternatively, using the mean over all levels of each mode as the reference rather than a particular level (sum-to-zero) makes the averages of the log factors over their non-shared modes equal across factors. Equivalently, the geometric means of the factors over those modes are equal. EMBA imposes neither equality. In our numerical check, neither the slices at any choice of reference levels nor the averages of the log factors coincided. Although EMBA and post-hoc re-gauging are different procedures, applying re-gauging to MBA factors may offer a practical alternative to EMBA. To evaluate this, we compared EMBA with post-hoc re-gauging of MBA factors using corner-point constraints (first, last, most frequent, and 40 randomly selected reference levels) and sum-to-zero constraints, while retaining MBA's rule for sharing lower-order terms. All these factorizations preserve the same reconstructed tensor. The detailed results are provided in Appendix F.6, Table 10, of the revised paper. On the COIL-100 tensor used in the paper, EMBA outperforms all evaluated corner-point choices across all metrics, whereas sum-to-zero outperforms EMBA. Sum-to-zero also performs better on the COIL-100 tensor. Results on EEG, METR-LA, and Boolean tensors are mixed, with EMBA outperforming sum-to-zero on some metrics and vice versa. These results show that factor representation affects performance under factor deletion and quantization, even when the reconstructed tensor is unchanged. EMBA provides a useful choice of factor representation, while sum-to-zero re-gauging is an alternative. In summary, EMBA removes structural equalities between factors within the log-linear model that MBA formulates for tensor decomposition using information geometry. The standard post-hoc re-gauging methods we tested do not generally recover the EMBA factors, and the resulting representations behave differently under factor deletion and quantization. EMBA thus provides a distinct way to obtain factors within MBA's model class without forcing either the slices or the geometric means described above to coincide. We have revised the Introduction, Related Work, and Conclusion to make this distinction explicit throughout the paper.

Q3. Handling zero entries and existence of the optimum

We use the convention 0log⁡(0/p)=00\log(0/p)=0 for p>0p>0 when evaluating the KL divergence. Thus, zero entries in the input tensor, including Boolean zeros and the zero-padded entries introduced by EMBA, are handled directly in the objective. Missing values, however, are distinct from observed zeros and are not explicitly modeled as missing entries in our formulation.

Although the input tensor may contain zeros, MBA and EMBA reconstructions remain strictly positive for finite natural parameters. Consequently, the KL infimum may lie on the boundary of the model family and need not be attained at finite parameter values. The theoretical results therefore concern the infimum rather than assuming the existence of a finite minimizer.

In our experiments, optimization starts from zero natural parameters and terminates when the residual falls below 10−510^{-5}. The reported reconstructions are thus finite numerical solutions. For EMBA, the remaining probability mass on the padded entries is approximately 10−710^{-7} at termination, illustrating how the optimization approaches the boundary without attaining it.

Q4 Additional datasets and detailed experimental results

MBA and EMBA are deterministic in our implementation, with natural parameters initialized to zero, so random initialization seeds are not applicable. In Table 1, the reported mean ± standard deviation reflects differences across factors, while in Table 3, it reflects differences across the three tensor densities. Similarly, the variation reported in factor deletion and recursive approximation experiments arises from different factor selection patterns rather than randomness in optimization.

To assess variability across input data, we additionally evaluated ten COIL-100 tensors constructed using different image selections, using the same factor deletion and quantization procedures. We have included these additional results and the numerical values underlying Figure 4 in the revised paper.

Reviewer_LwN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Many-Body Approximation (MBA), a non-negative tensor decomposition that writes a tensor as a product of tensor factors, each capturing an interaction among a subset of modes. The authors show that MBA’s parameterization forces factors sharing modes to contain identical slices, and propose Extended MBA (EMBA): pad every mode of the tensor with one extra zero slice, run MBA on the padded tensor, then restrict the factors back to the original indices and renormalize. Experiments on COIL-100 images, METR-LA traffic, and EEG data show that, with the same overall fit, EMBA factors are more robust than MBA’s under factor deletion, recursive re-approximation, and low-bit quantization, at the cost of more Newton iterations.

Critical strengths
  1. The problem is real and clearly stated. MBA forces distinct factors to share identical slices.

  2. The fix is simple and keeps the convex optimization structure.

Critical weaknesses
  1. Theorem 4.1 only shows that entries from distinct factors are not identical as functions of θ\theta. It says nothing about the learned factors at the optimum or about identifiability. Please either tone down the claim or provide something stronger, e.g., local identifiability, or at least numerical evidence. Please also explain how this property accounts for the empirical gains.

  2. MBA is defined for strictly positive tensors, but the padding slice is exactly zero, so the optimum lies on the boundary. EMBA needs 12–14 Newton iterations versus 3 for MBA, which may be related. Please explain how zeros enter the KL objective and the log-based initialization, whether the optimum is actually attained, and whether the 150-iteration cap or step halving affects results.

  3. The quantization study uses a single tensor built from 10 COIL-100 images.

  4. Reconstruction after deletion is an operational proxy for interpretability. On the synthetic data, a direct measure of factor recovery against the known truth would be more convincing.

Questions for authors
  1. Please report seeds, confidence intervals, and the numbers behind Figure 4, and extend quantization to more datasets.

  2. Proposition 4.3 essentially shows that normalizing a sub-mass-one tensor cannot increase KL, and it compares against MBA on the extended tensor, not the original. A relation to MBA on the original tensor would be more meaningful.

  3. How is the 0.5 binarization threshold applied, given that P is normalized to sum to one?

2 comments

We thank the reviewer for the careful reading and constructive feedback.

Q1 Additional datasets and detailed experimental results

MBA and EMBA are deterministic in our implementation, with natural parameters initialized to zero. Therefore, random initialization seeds and confidence intervals across repeated runs on the same input are not applicable. The low-rank baselines in Table 2 were evaluated over five random seeds.

The reported mean ± standard deviation in factor deletion and recursive approximation experiments reflects differences across the selected factor subsets or factor selection patterns, rather than randomness in optimization.

To assess variability across input data, we additionally evaluated ten COIL-100 tensors constructed using different image selections, using the same factor deletion and quantization procedures. We have included these additional results in the revised paper.

Q2. Relation between EMBA and original-domain MBA

We agree that the original KL-divergence inequality compares EMBA with MBA on the extended tensor, rather than MBA applied directly to the original tensor. To address this, we have added a proposition establishing that MBA and EMBA represent exactly the same family of normalized distributions on the original domain.

Thus, EMBA changes the factor representation without enlarging the reconstruction family, and the infima of their reconstruction KL divergences on the original domain coincide. We have included the proof in the appendix. We have also clarified in the Introduction, Related Work, and Conclusion that EMBA changes the factor representation rather than the reconstruction model family.

Q3. Binarization threshold in Table 3

Thank you for pointing out this ambiguity. Only in the Boolean tensor experiment reported in Table 3, we apply an additional binarization step after reconstructing the tensor from the quantized factors. Specifically, MBA and EMBA are applied to the normalized input tensor. After quantizing the learned factors and reconstructing their product, we multiply the reconstruction by the sum of the original Boolean tensor entries to restore its scale. We then apply the threshold of 0.5 and compute the accuracy against the original Boolean tensor, without renormalizing the quantized reconstruction to sum to one.

In all other experiments, we evaluate the reconstructions obtained from the quantized factors directly, without applying any additional binarization step.

We will clarify this distinction in the revised manuscript.

AuthorsOpenReview ↗

Follow-up Clarifications on the Critical Weaknesses

Thank you again for your constructive feedback. We would like to supplement our earlier rebuttal by addressing the points raised in the Critical Weaknesses section more explicitly.

W1. Clarification of Theorem 4.1. As discussed in our response to MPQk, we have added a new proposition (Proposition 4.2) that characterizes the conditions under which corresponding EMBA factor slices coincide. Specifically, their equality is equivalent to a homogeneous linear system in the extended natural parameters, whose coefficient matrix has full row rank. Thus, unlike MBA, where the corresponding slices coincide for every parameter value, EMBA permits equality only when the learned parameters satisfy these conditions. For clarity, we do not claim that optimization necessarily avoids these conditions.

W2. Zero entries and boundary optimization. We have corrected Appendix D.3: both MBA and EMBA start Newton's method from zero natural parameters, and the input tensor enters the optimization only through its marginals. Thus, no logarithm of zero-valued input entries is taken, and no smoothing is required. The KL infimum may lie on the boundary and need not be attained at finite natural parameters; our reported results are finite numerical solutions obtained using a residual tolerance of 10−510^{-5}.

Newton's method is limited to 150 iterations, with step halving enabled when a proposed step does not decrease the residual. EMBA requires 12–14 iterations in the scalability experiments (Appendix F.1), compared with 3 and 5 for MBA, well below the iteration limit. The greater iteration count comes from approaching the boundary: on METR-LA and the Boolean tensors, where the optimum of MBA itself lies on the boundary because of zero entries, MBA also requires 11–12 iterations, as EMBA does.

W3. Broader quantization experiments. We have extended the quantization experiments to EEG and METR-LA, in addition to evaluating ten further COIL-100 tensors. The results are reported in Appendix F.6.

W4. Factor recovery on synthetic Boolean tensors. We agree that a direct comparison with known factors is more convincing. We therefore evaluated factor recovery on the synthetic tensors T=A∧BT=A\wedge B, where X(1,2)X^{(1,2)} corresponds to AA, X(2,3)X^{(2,3)} to BB, and X(1,3)X^{(1,3)} in Cases 4–6 has no corresponding interaction in the ground truth.

We measured the AUC for distinguishing true ones from zeros, the minimum number of misclassified entries over classification thresholds, and the standard deviation of the log entries of X(1,3)X^{(1,3)}, which should be zero for a constant factor.

MetricMBAEMBA
AUC for X(1,2)X^{(1,2)} and X(2,3)X^{(2,3)} (Cases 1–6)0.908–0.9841.000
Misclassified entries at the best threshold (out of 10,000)519–1,3690
Std. of log X(1,3)X^{(1,3)} (Cases 4–6)3.73–4.670.07–0.13

These results show that EMBA perfectly separates the true ones and zeros in AA and BB in these evaluated settings, while keeping the factor corresponding to the absent interaction nearly constant. In contrast, MBA introduces substantial variation into that factor. This provides direct empirical evidence of improved factor recovery on the synthetic examples, complementing our factor-deletion experiments. We have added these factor recovery results to Appendix F.3 of the revised manuscript.

Reviewer_5PpkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This contribution considers the problem of learning distinct factors in many-body approximations (MBA) of tensors. Specifically, this paper shows that a shortcoming of the MBA of a nonnegative tensor T\mathbf{T} is that is has factors that are identical. To address this shortcoming, the authors propose to zero-pad T\mathbf{T}, and run MBA on the zero-padded tensor. The effectiveness of this change is shown empirically, on real datasets, and through some theoretical results. Overall, the paper is well-written, however, there are some significant concerns regarding the novelty and usefulness of the proposed method.

Critical strengths

S1. The paper is well-written and organized. The authors identify an important problem in MBA, and propose a simple solution to it.

S2. Empirical results are promising.

Critical weaknesses

W1. It is not clear why zero-padding is preferable to changing the parametrization of MBA. At this stage, the proposed algorithm seems more like a reparametrization/optimization trick, rather than a new method. Furthermore, since MBA is reminiscent of fitting graphical log-linear models to data (Eq. (2) is basically a Gibbs distribution), it is not clear what this zero-padding accomplishes that cannot be accomplished using existing graphical model approaches; see also [R1] below.

W2. The computational cost of zero-padding the input tensor is unclear, especially in cases where the tensor cannot be stored in memory and has to be streamed from data storage.

References:

[R1] James E. Johndrow, Anirban Bhattacharya, David B. Dunson "Tensor decompositions and sparse log-linear models," The Annals of Statistics, Ann. Statist. 45(1), 1-38, (February 2017)

Questions for authors

Q1. This is related to W1. A theoretical comparison or experiment that addresses the structural dependencies of MBA without zero-padding would help clarify the novelty of EMBA.

Q2. While a scalability analysis of EMBA is provided in Sec. 4.2.2, please address point W2 above.

Q3. While theorem 4.1 states that "two entries drawn from distinct EMBA factors are not identical as functions of θ\boldsymbol{\theta}", it does not seem to show the main claim of the paper, i.e., that structural dependencies are eliminated. Please clarify this point.

4 comments

We thank the reviewer for the careful reading and constructive feedback.

W1 and Q1. Motivation for EMBA and relation to existing parameterizations

We agree that MBA is closely related to hierarchical log-linear models. Johndrow et al. [R1] study the relationship between sparse log-linear models and low-rank tensor decompositions. In contrast, MBA constructs interaction factors by distributing log-linear parameters among factors associated with subsets of modes. Our work focuses on the structural equalities imposed by this factor construction.

As discussed in our response to MPQk (Q2), MBA and EMBA represent the same log-linear model family on the original domain. EMBA obtains factors through MBA optimization on an extended domain, removing the forced cross-factor slice equalities without enlarging the reconstruction family. We have revised the Introduction, Related Work, and Conclusion to clarify this distinction.

Changing the reference level in MBA only changes which factor slices are forced to coincide. Sum-to-zero re-gauging removes these slice equalities but instead imposes equalities between the geometric means of factors over their non-shared modes. EMBA imposes neither type of equality, and the tested post-hoc re-gauging methods do not generally recover its factors.

To evaluate re-gauging as a practical alternative, we compared EMBA with corner-point constraints (first, last, most frequent, and 40 randomly selected reference levels) and sum-to-zero constraints, retaining MBA's rule for sharing lower-order terms. These representations preserve the fitted reconstruction. Results are provided in Appendix F.6, Table 10, of the revised paper.

On the COIL-100 tensor used in the paper, EMBA outperforms all evaluated corner-point choices, whereas sum-to-zero outperforms EMBA. Results on EEG, METR-LA, and Boolean tensors are mixed. Thus, although sum-to-zero is a practical alternative, EMBA provides a distinct way to obtain factors directly through KL projection on the extended domain, without post-hoc re-gauging.

Q2. Computational cost of zero-padding

Our current implementation explicitly constructs the zero-padded tensor in memory. However, the input tensor is needed only to compute the target expectation parameters η^B\hat{\boldsymbol{\eta}}_{\mathcal B} before optimization. Since the padded entries are all zero, these parameters can be computed by streaming the original tensor once, without explicitly constructing the padded tensor.

The subsequent optimization uses the model tensor on the extended domain and the Fisher information matrix, as in MBA applied to a tensor of the extended shape. Thus, while streaming can eliminate the additional storage required for the padded input, the model-side memory requirements remain. We will clarify this distinction and the current implementation's limitations in the revision.

Q3. Interpretation of Theorem 4.1

In this paper, structural dependency refers to the fact that certain slices or groups of entries from different MBA factors are necessarily identical, regardless of the natural parameter values (Proposition 3.1). Theorem 4.1 shows that, in EMBA, no two entries from different factors are necessarily identical as functions of the parameters. Thus, the equalities found in MBA are not merely shifted to different locations; no such forced equality exists anywhere between EMBA factors.

The theorem does not rule out the possibility that entries from different factors coincide at particular learned parameter values. However, such a coincidence requires the learned parameters to satisfy a linear equation equating two distinct functions of the natural parameters.

Thank you for the rebuttals; I appreciate the effort the authors put into the revised manuscript and additional experiments.

My issue with W1 and Q1 still remains, and it is reinforced by the sum-to-zero results of Table 10 in the revised manuscript. At this stage it is not clear whether EMBA optimize any property of the individual factors that cannot be obtained through a suitable gauge choice on the original domain. Furthermore, it is not clear if constraints on the MBA optimization problem can preserve the EMBA results, instead of relying on zero-padding.

I retain my score of 2.

Thank you for your further comments. We understand your concern that alternative gauge choices or constraints on the original-domain MBA optimization might yield similar factor representations. We would like to clarify why we introduced EMBA rather than adopting such alternatives.

MBA provides an information-geometric framework in which factors are constructed from interaction energies through KL projection. While choosing a gauge for a particular downstream task is a valid alternative, our goal was to remove MBA's forced slice equalities while preserving this energy-based interpretation and factor construction. EMBA achieves this by applying the same KL projection on the extended domain, rather than redistributing interaction terms after fitting.

Although we do not rule out a re-gauging that reproduces the EMBA factors, how to recover the specific factors learned by EMBA from the MBA factors is not evident. Indeed, the post-hoc re-gauging methods evaluated in our additional experiments did not reproduce the EMBA factors. Table 10 also shows that sum-to-zero can perform better in some settings. Thus, we do not claim optimality among gauge choices; our contribution is a concrete method for obtaining structurally distinct factors directly through the MBA framework.

We also do not rule out an equivalent constrained formulation on the original domain. Rather, zero-padding provides a principled construction that preserves MBA's factor interpretation while removing its forced slice equalities. We hope this distinction clarifies the novelty of EMBA.

AuthorsOpenReview ↗

Follow-up: Additional Evidence for EMBA's Factor Representation

Thank you again for your feedback. As a follow-up to our previous response, we would like to share additional experiments conducted during our discussion with Reviewer LwN4 and MPQk, addressing the practical usefulness of EMBA's factor representation.

We evaluated synthetic Boolean tensors with known interactions between modes (1,2) and (2,3), but no interaction between modes (1,3). EMBA accurately recovered the true interactions while producing a nearly constant factor for the absent interaction (Appendix F.3).

We also evaluated the effect of deleting the absent-interaction factor on 15 additional tensors (five random seeds per setting). The results are summarized below (mean ± standard deviation).

ponep_{\mathrm{one}}MethodKL after deletionRelative KL ratio
0.3MBA296,700 ± 29,9900.3315 ± 0.0444
Sum-to-zero7,997 ± 1,3070.0687 ± 0.0100
EMBA720 ± 1680.0066 ± 0.0014
0.5MBA1,258,611 ± 99,0410.6273 ± 0.0832
Sum-to-zero24,888 ± 3,2590.1230 ± 0.0161
EMBA1,210 ± 1140.0070 ± 0.0007
0.7MBA2,911,396 ± 45,8101.0151 ± 0.0260
Sum-to-zero34,372 ± 2,5820.1626 ± 0.0113
EMBA1,166 ± 880.0067 ± 0.0005

The relative KL ratio compares deletion of the absent-interaction factor with deletion of a true-interaction factor. EMBA achieved lower reconstruction KL divergence than MBA and sum-to-zero on every tested tensor, with relative ratios below 1% across all settings (Appendix F.7).

We acknowledge that results on real datasets are mixed and do not claim that EMBA is universally preferable to alternative representations obtained through post-hoc re-gauging of MBA factors. However, these experiments demonstrate a distinct benefit of EMBA's factor representation: when the underlying interaction structure is known, its factors more faithfully reflect which interactions are present or absent. We believe this provides concrete evidence of EMBA's value, and hope it helps address your concern about its practical usefulness.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Key Contributions

Dear Reviewers, ACs, and SACs,

We thank all reviewers for their constructive feedback, which has helped us strengthen the theoretical analysis, experimental evidence, and presentation of our work. We summarize the key contributions and revisions below.

  • Novelty & Motivation – Our original goal was to remove MBA's forced cross-factor slice equalities while preserving its energy-based interpretation of interaction factors. Through discussions with Reviewers MPQk and 5Ppk, we further clarified this contribution by establishing that MBA and EMBA represent the same family of normalized reconstructions. We have revised the Introduction and Conclusion to make clear that EMBA changes how interaction factors are constructed, rather than expanding the reconstruction model family.

  • Theoretical Results – Building on Theorem 4.1 in the original submission, we added Proposition 4.2 to characterize when EMBA factor slices can coincide, addressing concerns raised by Reviewers MPQk and LwN4. We also established the equivalence of normalized reconstruction families and KL infima (Propositions 4.4–4.5), addressing concerns raised by Reviewers MPQk and Lsdh.

  • Empirical Results – Reviewers recognized the promising empirical advantages of EMBA over MBA demonstrated in the original submission, including across multiple domains (Sections 5.1–5.3 and Appendix F.2). In response to reviewers' feedback, we expanded the COIL-100 evaluation (Appendix F.4), broadened the quantization experiments (Appendix F.6), and added synthetic experiments evaluating ground-truth interaction recovery specifically requested by Reviewer LwN4 (Appendix F.3).

  • Practical Benefits over Alternative Factor Representations – Discussions with Reviewers MPQk and 5Ppk motivated us to consider post-hoc re-gauging of MBA factors as an alternative approach to factor representation, leading to new comparisons with corner-point and sum-to-zero re-gauging (Appendix F.6). Further synthetic experiments demonstrate EMBA's ability to distinguish present from absent interactions, with lower factor-deletion KL divergence than MBA and sum-to-zero (Appendix F.7). Reviewer MPQk explicitly acknowledged EMBA's superior recovery of absent interactions over sum-to-zero as concrete evidence for the EMBA representation.

We believe these results establish EMBA as a principled and practically useful approach to constructing interaction factors without the forced slice equalities between MBA factors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.