- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how reference choice affects linear regression on Gaussian distributions in Bures–Wasserstein geometry. It characterizes the prediction error that remains after refitting the regression coefficients and proposes learning the reference covariance by minimizing prediction residuals. Synthetic experiments verify the theoretical scaling, and 5 real-data applications evaluate improvements over fixed-reference regression and other linear baselines.
Critical strengths
A precise explanation of reference-induced error. The theory distinguishes changes that regression coefficients can absorb from genuine approximation error. It quantifies the latter and identifies cases where it vanishes, providing insight beyond the general observation that the geometry is curved.
Critical weaknesses
-
The practical relevance of the main assumption remains unclear. The theory assumes that the response is linear at some suitable reference, and the synthetic experiments construct this scenario. The paper provides limited guidance for recognizing when this assumption is reasonable or assessing robustness when no such reference exists.
-
The real-data evidence leaves uncertainty about the practical gains. Results use single outer splits without uncertainty intervals. Regularization is evaluated for raw covariance features but not fixed-reference tangent features, leaving open whether reference learning improves over a suitably regularized version of the same representation. All 600 runs in the additional simulations stopped at the iteration or backtracking limit. Testing sensitivity to initialization and optimization budgets would help establish whether the reported improvements are reproducible across a range of practical settings.
Questions for authors
-
How can practitioners assess whether reference mismatch is a substantial source of prediction error? What evidence suggests that reference learning remains useful when no reference makes the response exactly linear?
-
Given the reported stopping conditions, what evidence shows that the prediction gains are stable across different initial references and larger optimization budgets?
3 comments
Thank you for your response, the additional experiments, and for addressing my questions. However, the response does not address the critical weakness concerning the practical relevance of the main assumptions and the scenarios in which they hold. Therefore, I will keep my score unchanged.
Thank you for asking when reference learning helps and how to assess its gains. Our previous reply reported the nonlinear-response experiment (Appendix H.5), the fixed-reference ridge comparison (G.4), molecular uncertainty and KOA refits (Table 4 and G.7), and initialization and budget sensitivity (H.6). This revision adds Propositions 2 and 3, referenced in Sections 3.3 and 3.5.
Weakness 1. Practical relevance of tangent linearity
1(a). When is the approximation reasonable? Proposition 2 (Appendix C.5) uses a first-order Taylor predictor in the tangent coordinates of a candidate reference. If the conditional mean has Hessian operator norm at most along the coordinate segments from the reference to the inputs, and the tangent coordinate has a finite fourth moment, the squared approximation error is at most . The bound is small for a smooth response and inputs concentrated near the reference.
1(b). What happens when no reference is exactly linear? Write the conditional mean as , with tangent-linear at a candidate reference and . Proposition 2 shows that the square root of the profiled excess risk differs from by at most , where projects onto the affine tangent feature space at . A correction that reduces this residual norm by more than strictly lowers the population risk after refitting.
In the nonlinear-response experiment (Appendix H.5), the centered quadratic term in the Gaussian mean is orthogonal to every affine tangent feature space for the latent inputs. It creates a positive approximation error at every reference and adds the same amount to each reference's profiled risk. In two sets of 20 paired replicates, learning the reference reduced mean test MSE by 16.7% and 18.5%. The 95% Monte Carlo intervals for fixed minus learned MSE were [0.132, 0.276] and [0.077, 0.212].
1(c). How can we decide whether to apply the correction? Our earlier reply proposed comparing fixed and learned references on the same held-out observations. Proposition 3 (Appendix H.6) considers independent i.i.d. validation of training-fitted checkpoints, including the zero-update fit. Under a conditional variance bound on the paired loss differences, with probability at least given training, the selected predictor's risk is within of the lowest checkpoint risk, where is the validation error radius. On the same event, an observed paired validation MSE reduction greater than implies lower risk than the zero-update fit.
Weakness 2. Evidence for practical gains
2(a). Split uncertainty and refitting. The 95% paired structure-group bootstrap intervals (Table 4 and Appendix G.7) compare fixed-reference OLS with learned-reference OLS, conditional on the fitted models and conformers. The MSE reductions were 0.0643 [0.0100, 0.1237] on BioHL and 0.0889 [−0.0541, 0.2507] on KOA. The KOA interval includes zero. To assess sensitivity to refitting, we also ran ten repetitions of grouped five-fold outer cross-validation on the same KOA molecules, with four-fold grouped selection within each outer training set. Mean out-of-fold MSE fell from 0.9510 to 0.8787 (7.61%), with a reduction in every repetition.
2(b). Ridge on fixed tangent features. The fixed-reference baseline fits ridge to the BW tangent features at the training barycenter with the same ordinary covariates and an unpenalized intercept. We scaled the features on training data and chose the penalty from a 15-value grid by leave-one-out error (Appendix G.4 and Section 4.4). Learned-reference OLS lowered test MSE relative to this baseline by 9.88%, 20.47%, 42.01%, 10.67%, and 18.32% on the NHANES, ESOL, BioHL, KOA, and Gas splits, respectively.
2(c). Stopping, initialization, and update budgets. The original runs ended at the iteration cap in 574 of 600 cases and at the backtracking limit in 26 (Appendix H.2). The sensitivity analysis uses matched samples for two initial references across 12 simulation settings with 50 replicates each (Appendix H.6). At caps of 60, 120, and 240 updates, validation-selected pooled test MSE fell by 4.29%, 5.02%, and 5.45% relative to the frozen perturbed reference and by 2.04%, 2.37%, and 2.56% relative to the frozen training barycenter. At 240 updates, six of 12 condition means improved from the barycenter and validation selected zero updates in 314 of 600 runs. Using the final iterate increased pooled test MSE at both starts as the cap rose from 60 to 240. Doubling the backtracking allowance from 40 to 80 for the 27 affected runs in the perturbed-start comparison changed the selected test MSE by at most .
Thank you for asking how a practitioner can use the theory when the generating reference is unknown, and whether simpler choices of regularization or stopping could explain the application gains.
1. Recognizing when reference adaptation helps. The tangent-linearity assumption gives a geometric interpretation to the excess risk. A practitioner can assess the predictive value of adaptation without knowing whether that assumption holds. Section 5 describes a paired comparison: fit both references on the same training data, select the update schedule within training, and compare squared errors on the same held-out observations. With the Gaussian inputs and the OLS readout held fixed, the paired loss difference measures the predictive value of learning the reference on that task. This gives practitioners a way to decide whether adaptation is worthwhile even when the generating reference is unknown (BioHL and KOA, Appendix G.7).
To examine misspecification directly, Appendix H.5 adds a quadratic term in the Gaussian mean with standard deviation one quarter that of the original noiseless signal. At fixed covariance every tangent-linear predictor is affine in the mean, so no Gaussian reference can represent this term exactly. Learning the reference nevertheless reduced mean test MSE in two independent sets of simulations. The quadratic dependence remains outside the model throughout this experiment. The gain shows that a better choice of coordinates can still improve the overall linear approximation without requiring an exactly tangent-linear response.
2. Practical gains, regularization and stopping. Your suggested ridge comparison isolates an important alternative explanation. We fitted ridge at the training BW barycenter, leaving the tangent features and ordinary covariates unchanged and selecting its penalty within training. Learned-reference OLS gave lower test MSE on all five application splits (Appendix G.4). Table 4 gives paired uncertainty intervals for the two small molecular test sets; on KOA the single-split interval crosses zero, while repeated grouped partitions favor the learned reference on average (Appendix G.7). Ridge changes the weight assigned to each tangent coordinate at the barycenter. Learning the reference changes the coordinates themselves, so a linear combination can express a different function of the same Gaussian input. The five matched comparisons show that this flexibility had predictive value even when the fixed-reference model could tune its regularization.
The stopping flags you identified give no certificate of stationarity. We therefore tested the prediction consequence of the stopping rule directly. Across the exact-covariance conditions in Appendix H.6, validation-selected updates reduced pooled test MSE from both the perturbed start and the training barycenter at every budget examined. Extending the backtracking allowance for the perturbed-start runs that hit the limit had negligible effect on their selected test MSE. Validation could retain the starting point, and taking the last iterate at the largest budget gave higher pooled error than taking the last iterate at the smallest budget. Including zero updates makes the fixed BW barycenter an available choice whenever fitting starts there. A practitioner can therefore use validation to choose both whether to adapt and when to stop, while retaining an ordinary linear predictor for new distributions.