- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study discusses whether it is possible to obtain upper bound control for the joint Fisher divergence of the data when only some (rather than all) variables are used in each prediction task. The author understands the problem from the perspective of a Gaussian model and extends it to functional ANOVA. At the end of the article, the theoretical results were verified through experiments.
Critical strengths
-
Established upper and lower bounds for rate matching in the Gaussian case, and characterized the influence of condition number on the control ratio (Theorem 2).
-
Established the marginal control accuracy under density-ratio ANOVA (Theorem 3), and the matching upper and lower bounds of score learning (Theorem 4).
Critical weaknesses
-
The results of Section 4 only discuss the properties of the population distribution, and do not answer what the optimal Fisher risk for estimating $ \mu, \Sigma $ (under finite samples) is, or whether the optimal statistical estimation rate has a similar formulation. Please distinguish between population comparison minimax optimality and statistical estimation minimax optimality, and explain how much support the former's conclusion can provide for the latter's rate analysis.
-
The number of multivariate interaction terms is extremely large, which brings a significant amount of computational cost. In many cases, not all interactions are significant, so there should be many zeros in in equation (21). However, existing theories have not yet taken into account the impact of sparse selection of interactions on the problem, or have directly assumed which interaction terms are known to be non-zero, which is quite unrealistic.
Questions for authors
-
The model setting and processing results of the Gaussian model and ANOVA are separated: the two constitute the two main lines of the article, so the writing is not coherent, and it is difficult to understand that, to what extent Section 5 and Section 6 inherit, and expand the results of Section 4.
-
Why is there no influence of $ b_j $ in the minimax rate of Theorem 4, even though those in equation (21), to some extent, describe the smoothness of the function $ p_{\theta} $?
-
There are problems with the formula expression and assumptions in some parts of this study, such as the definition in $ \mathbb P(B \subseteq A)$ in equation (14), and the assumption in equation (13): when $ d $ is even, for a density ratio of it will result in , which shows exponential growth in dimension . The existing theoretical analysis has not discussed the impact of dimension .
-
Suggest that the paper should provide more specific steps for implementing the algorithm, such as pseudocode.
2 comments
Update on Revisions Addressing Your Main Concerns
Thank you again for your patience and insightful comments. We would like to briefly highlight two additions in the latest revision addressing your main concerns:
- Matching upper and lower bounds on finite-sample risk when both the mean and the full covariance matrix are unknown, under the stated conditions (Weakness 1; Appendix A.5, Proposition 8).
- A method for selecting interactions without knowing in advance which coefficients are nonzero, together with an error guarantee. We also provide a rule for reducing computation by ruling out some candidates early (Weakness 2; Section 6 and Appendix C.2, Proposition 12).
We have also substantially revised the writing. Further details are provided in our revision summary and our latest reply to WyPd (which also discusses research decisions related to your question about sparse interactions). We would appreciate any further feedback you may have.
Response to Reviewer J3Xa
Thank you for the detailed comments. We respond in the same order below.
Main weaknesses
Section 4 now includes an initial explanation of the two notions of optimality. Under the theorem's assumptions, our comparison upper bound turns a bound on marginal estimation error into one on joint error. Using the comparison lower bound to obtain a statistical lower bound also requires studying how hard it is to distinguish the models from finite data. We will try to explain more clearly.
Appendix A.5 adds a Gaussian example with only one unknown variance parameter. At a fixed observation fraction, the comparison factor grows cubically with the condition number, while the coefficient in the estimator's asymptotic risk grows quadratically. This shows that the worst-case error amplification in a comparison bound need not be an unavoidable cost of estimation.
We have not yet derived the optimal finite-sample risk when both the mean and covariance are unknown. We will continue working on this question.
We specify the candidate basis functions, but do not assume that we know which coefficients are nonzero. The model allows zero coefficients. However, our analysis does not yet address how to use sparsity to select interactions. We have noted this limitation in Section 6 and will explore ways to address it.
Questions
Sections 4 and 5 study different models, but both ask how marginal error controls joint error. Observing every pair can suffice for Gaussian models, but may not suffice beyond them. Section 5 therefore studies which interactions need to be observed and how this affects error. Section 6 studies the sample requirements of a finite-basis version of these models.
We have revised the introduction and section openings to explain these connections, and moved the general local operator formulation from Section 5 to Appendix B.2.
bj is the maximum amplitude of a basis function. It affects the constants in the error bounds and how many samples are needed to reach the stated rate (Appendix C, Eqs. (95) and (100) of the original submission). Theorem 4 gives the rate for a fixed basis, so it does not spell out each of these dependencies. We have added this explanation after the theorem. Appendix C.3, Eq. (110), now gives an explicit sufficient condition on the sample size that includes bj.
Your example is correct: R grows exponentially with dimension d, making the constants in our general bound deteriorate rapidly. We have added discussion in Section 5 and Appendix B.3 and further analyzed the example's independent blocks. For comparison with the reference distribution, error amplification can be controlled by the probability of observing both variables in each block together. It need not grow exponentially with dimension.
We have also clarified the definition of Pr(B⊆A) in Eq. (14) of the original submission. Here B is a fixed set of variables, A is the random observed set, and Pr(B⊆A) is the probability that all variables in B are observed in the same record.
We will continue improving the writing and making the implementation steps more explicit. The revision already includes the steps for computing the moment estimator in Appendix C.2 and for fitting and selecting the mixture models in Appendix D.2.