acceptodds
Under review as a conference paper at ICLR 2027

D4R: Communication-Efficient Robust Reduced-Rank Regression for Distributed Data

Abstract

Large-scale multivariate data are often distributed across multiple machines and contaminated by outliers, which complicates efficient and interpretable modeling of shared low-rank structure. To address both challenges, we propose a **D**istributed **R**obust **R**educed-**R**ank **R**egression (D4R) framework based on a refined surrogate loss. D4R separates local outlier correction from global low-rank estimation while reducing both computation and communication costs. Theoretically, we establish non-asymptotic error bounds showing that D4R reaches the target statistical rate after finite communication rounds. Simulations and real-data studies demonstrate the robustness and efficiency of our method.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
robust statistics, distributed learning, reduced-rank regression, communication efficiency, non-asymptotic analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wFCrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes distributed robust reduced-rank regression under a row-sparse additive response-contamination model. A block-diagonal surrogate separates local contamination adjustment from global coefficient estimation. Workers hard-threshold their residual rows and send adjusted gradients; the central machine performs a closed-form rank-constrained regression update using its local covariance. The analysis gives coefficient-error contraction and a target rate combining low-rank estimation error with a contamination term, using an oracle threshold schedule. Experiments compare D4R with distributed alternatives and a full-sample oracle under several noise distributions and contaminated public datasets.

Critical strengths
  1. The estimator has a useful decomposition.

  2. The coefficient-error analysis has substantive value within its assumptions.

Critical weaknesses
  1. The practical tuning and experimental implementation are not sufficiently reliable as supplied. Equation (21) permits a saturated contamination estimate that absorbs every residual row, giving zero residual sum of squares and an undefined or negative-infinite log term.

  2. The central efficiency claims are stronger than the demonstrated evidence. The advertised centralized O(N2q)O(N^2 q) term is avoidable: with a thin QR factorization X=QRX=QR, the reduced-rank step can use F=QT(Y−sqrt(N)C)F=Q^T(Y-sqrt(N)C) and the singular vectors of F, without materializing an N-by-N projection.

Questions for authors
  1. What explicit constraint or criterion prevents the saturated hBIC solution, and how is that rule related to the oracle threshold schedule? Does the corrected practical procedure recover the clean rank-one example above and remain stable to the loss floor and small noise perturbations?

  2. Which exact code and parameter settings generated the reported experiments? Please explain the DC-D4R sqrt(n)-versus-sqrt(N) scoring mismatch, distinguish selected candidate rank from the rank of an averaged coefficient matrix, and provide corrected comparisons using the stated tuning protocol. Do the results hold for moderate, difficult-to-detect contamination as well as the large injected shifts?

  3. What are the computation and communication costs against an optimized centralized robust RRR implementation using factored projections, including all candidate-rank fits and scoring exchanges? Which contributions to wall time arise from parallel execution rather than reduced total arithmetic work?

  4. Please correct or restrict Lemma C.1's claim for arbitrary epsilon: the displayed Bernstein bound has a linear-tail regime, while Assumption 2 does not involve log(1/epsilon). Also, which experimental regimes satisfy the conditions for the constant-round result, and which demonstrate robustness empirically outside those conditions?

7 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the positive assessment of our work and the detailed comments on practical tuning, experimental implementation, and computational efficiency. We have carefully addressed the comments and revised the manuscript accordingly, with all revisions highlighted in red. Our point-by-point responses are provided below.

For clarity, our detailed responses to each weakness and question are provided in separate comments below.

AuthorsOpenReview ↗

Response to W1 and W2

W1

The practical tuning and experimental implementation are not sufficiently reliable as supplied. Equation (21) permits a saturated contamination estimate that absorbs every residual row, giving zero residual sum of squares and an undefined or negative-infinite log term.

Response:

Thank you for this observation. We score only candidates with ∥CIk⋅∥2,0≤smax⁡<n\lVert C_{\mathcal I_k\cdot}\rVert_{2,0}\le s_{\max}<n and a positive residual sum of squares. When the current residual matrix is nonzero, we include the zero contamination estimate. When the residual matrix is zero, we set the contamination estimate to zero without evaluating hBIC.

Restricting candidate model complexity is consistent with established model selection procedures. Wang et al. [1] impose a model-size restriction in hBIC, while She and Chen [2] impose a model-complexity constraint in robust reduced-rank regression. We state this rule after Eq. (21) in the revised manuscript.

References

[1] Wang, L., Kim, Y., and Li, R. (2013). Calibrating nonconvex penalized regression in ultra-high dimension. The Annals of Statistics, 41(5), 2505–2536.

[2] She, Y., and Chen, K. (2017). Robust reduced-rank regression. Biometrika, 104(3), 633–647.

W2

The central efficiency claims are stronger than the demonstrated evidence. The advertised centralized O(N2q)O(N^2q) term is avoidable: with a thin QR factorization X=QRX=QR, the reduced-rank step can use F=QT(Y−NC)F=Q^T(Y-\sqrt{N}C) and the singular vectors of FF, without materializing an NN-by-NN projection.

Response:

Thank you for this observation. With a thin QR factorization X=QRX=QR, the centralized reduced-rank update can be computed through a rank-rr approximation of Q⊤(Y−NC)∈Rp×qQ^\top(Y-\sqrt{N}C)\in\mathbb{R}^{p\times q}, followed by a triangular solve, without explicitly constructing the N×NN\times N projection matrix.

Under balanced partitions with N=mnN=mn, the computational advantages of D4R arise primarily from its distributed architecture. The centralized method requires a QR factorization of the full N×pN\times p design matrix, with complexity O(Np2)O(Np^2), whereas D4R only requires factorizing the central machine's n×pn\times p design matrix, with complexity O(np2)O(np^2). Both methods involve rank-rr approximations of p×qp\times q matrices. Moreover, D4R distributes the computation of adjusted gradients across machines, requiring O(npq)O(npq) operations per machine per round, compared with O(Npq)O(Npq) for the corresponding centralized computation. The total gradient computation remains O(Npq)O(Npq), while the communication cost is O(mpq)O(mpq) scalars per round.

These comparisons highlight the computational benefits of D4R in reducing the central factorization cost and enabling parallel computation without pooling the full dataset. Section 3 gives the one-time QR factorization costs, the gradient computation costs per machine and in total, and the communication volume per round.

AuthorsOpenReview ↗

Response to Q1 and Q2

Q1

What explicit constraint or criterion prevents the saturated hBIC solution, and how is that rule related to the oracle threshold schedule? Does the corrected practical procedure recover the clean rank-one example above and remain stable to the loss floor and small noise perturbations?

Response:

Thank you for the question. As described in W1, we score only candidates with ∥CIk⋅∥2,0≤smax⁡<n\lVert C_{\mathcal I_k\cdot}\rVert_{2,0}\le s_{\max}<n and a positive residual sum of squares. This restriction is consistent with the model-size constraints used in established model selection procedures, including Wang et al. (2013) and She and Chen (2017).

Within the admissible candidate set, hBIC selects the threshold by balancing goodness of fit against model complexity. For a fixed coefficient estimate, the hBIC minimizer in Eq. (21) also minimizes a penalized least-squares criterion with a row-sparsity penalty over the same candidate set. The associated implicit squared threshold is

\lVert Y_{I_k\cdot} -X_{I_k\cdot}B^{(t)} -\sqrt{N}C_{I_k\cdot}^{(t)} \rVert_F^2.$$ This provides a connection between hBIC selection and the theoretical threshold schedule in Theorem 5. Since hard-thresholding depends on the row norms of the residuals, different thresholds may yield the same contamination support. Therefore, if the support selected by hBIC can also be obtained using a threshold satisfying the conditions of Theorem 5, the corresponding concentration bounds and contraction analysis remain applicable, yielding the same order of estimation error. #### Q2 > Which exact code and parameter settings generated the reported experiments? Please explain the DC-D4R $\sqrt{n}$-versus-$\sqrt{N}$ scoring mismatch, distinguish selected candidate rank from the rank of an averaged coefficient matrix, and provide corrected comparisons using the stated tuning protocol. Do the results hold for moderate, difficult-to-detect contamination as well as the large injected shifts? **Response:** Thank you for these questions. We clarify the implementation and tuning procedures and provide additional results under lower-magnitude contamination. **Code and experimental settings** We have provided the implementation code for D4R. The simulation setup, competing methods, evaluation metrics, and tuning procedures are described in Section 5.1, with additional experimental details in Appendix E. **DC-D4R rank selection** The $\sqrt{n}$ and $\sqrt{N}$ scaling conventions are equivalent under a corresponding reparameterization of the contamination matrix. Specifically, $\sqrt{n}C_j=\sqrt{N}\widetilde C_j$, where $\widetilde C_j=\sqrt{n/N}C_j$. With this consistent reparameterization, the fitted residuals and RSC scores are identical, so the scaling convention does not affect rank selection. DC-D4R selects its candidate rank by minimizing the RSC over $\bar r\in\lbrace1,\ldots,8\rbrace$, with complexity penalty $\mu\bar r$, where $\mu=N+q$. For DC-D4R, $\hat r$ in Table R1 denotes the selected candidate rank rather than the numerical rank of the averaged coefficient matrix. **Results under lower-magnitude contamination** We further evaluate the methods under Gaussian response contamination with standard deviation 0.10, following the simulation settings in Section 5.1. Under this setting, D4R correctly selects the true rank $r=3$ and achieves a Frobenius estimation error of 0.570, comparable to Dist-CSL (0.564), though higher than the full-sample oracle (0.241). These results show that D4R maintains competitive estimation accuracy and accurate rank selection under lower-magnitude contamination. **Table R1.** Coefficient estimation errors and selected ranks under Gaussian response contamination with standard deviation 0.10, following the simulation settings of Section 5.1. Entries are mean (SE) over 100 replications. | Method | $\lVert \Delta\rVert_F$ | $\lVert \Delta\rVert_\infty$ | $\lVert \Delta\rVert_\Sigma$ | $\hat r$ | | ----------- | ----------------------: | ---------------------------: | ---------------------------: | ---------: | | Full-sample | 0.241 (0.005) | 0.040 (0.002) | 0.205 (0.003) | 3.0 (0.0) | | D4R | 0.570 (0.057) | 0.104 (0.015) | 0.485 (0.043) | 3.0 (0.0) | | DC-D4R | 1.044 (0.014) | 0.145 (0.003) | 0.822 (0.009) | 8.0 (0.0) | | Dist-CSL | 0.564 (0.022) | 0.096 (0.006) | 0.488 (0.019) | 3.0 (0.0) | | Dist-QRR | 1.865 (0.189) | 0.383 (0.039) | 1.449 (0.127) | 3.8 (0.8) | | Dist-RRR | 1.154 (0.024) | 0.159 (0.007) | 0.902 (0.022) | 14.4 (0.3) |
Reviewer_9rgEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a Distributed Robust Reduced-Rank Regression (D4R) framework for multivariate regression with distributed observations contaminated by outliers. It claims that the standard communication-efficient surrogate likelihood (CSL) doesn’t work for the stacked parameter (B, C), because the local and global Hessians differ in structure and scale. It proposes to replace the surrogate Hessian with a block-diagonal one, which allows the different update mechanism of B and C. The paper establishes non-asymptotic estimation guarantees the desired statistical error rate can be attained after a constant number of communication rounds and presents simulation and real-data results supporting the effectiveness of D4R.

Critical strengths
  1. The paper includes experiments examining different noise distributions, contamination proportions, signal strengths, sample allocations, and communication rounds. The use of four real datasets and repeated experimental runs is also a positive feature.

  2. The separable surrogate is a clean and efficient design. Each round sends one p×q matrix per machine. The C-step is local and exact, and the B-step has a closed form.

Critical weaknesses
  1. The real contribution of this work needs more clarification by comparing with existing works Liu et al. (2026) and Chen et al. (2022a). The CSL mismatch under contamination problem and distributed reduced-rank regression methodology are already proposed in previous works so this paper need to further present the new insight beyond above contents.

  2. The communication-efficiency claim requires more quantification support. The computational complexity and the number of communication rounds reported in ths paper are not enough to support the efficiency claim. The comparison with the competing distributed methods under the same experimental settings can clearly show its efficiency. Also, reporting wall-clock training time and communication volume would make the efficiency claim more convincing.

Questions for authors

Why assumption 2 is described as mild? Its implications for n, p, m and contamination levels should be discussed more concretely.

4 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the positive assessment of our work and the constructive comments on its core contributions, communication efficiency, and theoretical assumptions. We have carefully addressed the comments and revised the manuscript accordingly, with all revisions highlighted in red. Our point-by-point responses are provided below.

For clarity, our detailed responses to each weakness and question are provided in separate comments below.

AuthorsOpenReview ↗

Response to W1

W1

The real contribution of this work needs more clarification by comparing with existing works like Liu et al. (2026) and Chen et al. (2022a). The CSL mismatch under contamination problem and distributed reduced-rank regression methodology are already proposed in previous works so this paper need to further present the new insight beyond above contents.

Response:

Thank you for the insightful comment. We clarify the core contributions of D4R and highlight their distinctions from prior studies [1, 2].

Novelty and significance of the problem setting.

Large-scale multivariate response data are often distributed across multiple machines and contaminated by outliers, making robust low-rank estimation an important problem. However, existing studies have addressed only parts of this problem. Specifically, [1] studies distributed estimation under adversarial contamination for scalar responses, whereas [2] develops distributed reduced-rank regression without robustness. To the best of our knowledge, D4R is the first framework for distributed low-rank multivariate regression under sample-wise adversarial contamination, with corresponding algorithmic and statistical guarantees.

Surrogate-based distributed optimization for robust low-rank estimation.

Existing methods develop surrogate-based estimation under adversarial contamination for scalar responses [1] and distributed reduced-rank regression without contamination [2]. D4R constructs a refined block-diagonal surrogate that separates local row-sparse contamination estimation from global rank-constrained estimation. Each machine performs row-wise hard thresholding and communicates an adjusted gradient, allowing the central machine to construct a pseudo-response and obtain a closed-form rank-constrained update. This approach accommodates adversarial contamination without iterative gradient descent at the central machine or communication of the high-dimensional contamination matrix.

Theoretical insights into statistical accuracy.

Existing theoretical analyses of robust reduced-rank regression have primarily focused on joint prediction error for the coefficient and contamination components. To the best of our knowledge, direct non-asymptotic estimation guarantees for the low-rank coefficient matrix under sample-wise adversarial contamination remain unexplored, even in centralized settings. A key challenge lies in the coupling between the low-rank coefficient matrix and the row-sparse contamination component, which complicates the analysis of coefficient estimation errors.

We address this challenge by establishing bounds on local contamination estimation errors and analyzing their effects on subsequent global rank-constrained updates (cf. Appendix B). Under Assumptions 1 and 2, Theorem 5 establishes

\lesssim \sigma\left( \sqrt{\frac{r(p+q+\log(2/\epsilon))}{N}} + {\frac{O(q+\log N)}{N}} \right).$$ **This bound characterizes the estimation accuracy of $B^\ast$, separating the low-rank estimation error from the additional error induced by adversarial contamination**. Moreover, Corollary 6 shows that this rate can be attained within a constant number of communication rounds. Overall, D4R provides a unified framework for distributed robust reduced-rank regression under sample-wise adversarial contamination, combining a communication-efficient surrogate-based algorithm with non-asymptotic estimation guarantees directly for the low-rank coefficient matrix. We discuss these contributions in Section 1 and explain how D4R differs from [1, 2] in Section 2. **References** [1] Liu, S., Zhao, D., and Yang, H. (2026). Distributed learning with adversarial contamination: an efficient and optimal approach based on dynamic regularization. *Statistics and Computing*, 36(4), Article 150. [2] Chen, C., Xu, W., and Zhu, L. (2022). Distributed estimation in heterogeneous reduced rank regression: With application to order determination in sufficient dimension reduction. *Journal of Multivariate Analysis*, 190, 104991.
AuthorsOpenReview ↗

Response to W2

W2

The communication-efficiency claim requires more quantification support. The computational complexity and the number of communication rounds reported in this paper are not enough to support the efficiency claim. The comparison with the competing distributed methods under the same experimental settings clearly show its efficiency. Also, reporting wall-clock training time and communication volume would make the efficiency claim more convincing.

Response:

Thank you for this suggestion. We provide additional quantitative evidence on communication efficiency and clarify the wall-clock training-time comparisons already reported in the paper.

Communication costs and estimation accuracy

Tables R1 and R2 report the per-round and cumulative communication costs of D4R and Dist-CSL, respectively, together with their estimation errors at selected communication rounds. Under the same p×qp\times q message-exchange schedule, both methods transmit 2(m−1)pq2(m-1)pq floating-point values per round. For example, when (n,m)=(500,8)(n,m)=(500,8), each method transmits 8,400 values per round and 67,200 over eight rounds. At round 8, D4R achieves Frobenius estimation errors of approximately 0.239 across all three configurations, compared with 2.221–2.277 for Dist-CSL, demonstrating substantially lower estimation errors at the same communication volume.

Wall-clock training time

The wall-clock training-time comparison is already reported in Table E.6 of Appendix E.6 of the paper. It provides measured computation times under the paper's experimental settings, complementing the communication-volume comparison in Tables R1 and R2.

Table R1. D4R estimation errors and communication costs at selected rounds under the settings of Figure 2. Errors are mean (SE) over 100 replications. Communication counts cover one eight-round fit and exclude rank-selection overhead.

(n,m)(n,m)Floats per round (uplink + downlink)Cumulative floats (u=8u=8)∥Δ∥F\lVert \Delta\rVert_F, u=1u=1u=5u=5u=8u=8∥Δ∥Σ\lVert\Delta\rVert_\Sigma, u=1u=1u=5u=5u=8u=8
(500,8)(500,8)8,400 (4,200 + 4,200)67,2008.553 (0.547)0.246 (0.005)0.239 (0.005)7.106 (0.421)0.215 (0.004)0.210 (0.004)
(800,5)(800,5)4,800 (2,400 + 2,400)38,4006.973 (0.195)0.240 (0.004)0.239 (0.005)6.044 (0.271)0.211 (0.004)0.210 (0.004)
(1000,4)(1000,4)3,600 (1,800 + 1,800)28,8006.301 (0.255)0.241 (0.004)0.239 (0.005)5.575 (0.361)0.211 (0.004)0.210 (0.004)

Table R2. Dist-CSL estimation errors and communication costs at selected rounds under the same settings and message-exchange schedule as Table R1. Errors are mean (SE) over 100 replications.

(n,m)(n,m)Floats per round (uplink + downlink)Cumulative floats (u=8u=8)∥Δ∥F\lVert \Delta\rVert_F, u=1u=1u=5u=5u=8u=8∥Δ∥Σ\lVert\Delta\rVert_\Sigma, u=1u=1u=5u=5u=8u=8
(500,8)(500,8)8,400 (4,200 + 4,200)67,2009.003 (0.665)2.497 (0.078)2.277 (0.057)7.270 (0.426)2.240 (0.064)1.993 (0.032)
(800,5)(800,5)4,800 (2,400 + 2,400)38,4007.099 (0.335)2.284 (0.051)2.265 (0.051)5.636 (0.252)2.000 (0.028)1.981 (0.028)
(1000,4)(1000,4)3,600 (1,800 + 1,800)28,8006.133 (0.294)2.238 (0.055)2.221 (0.056)4.916 (0.242)1.954 (0.032)1.935 (0.034)
AuthorsOpenReview ↗

Response to Q1

Q1

Why assumption 2 is described as mild? Its implications for nn, pp, mm and contamination levels should be discussed more concretely.

Response:

Thank you for this question. Using N=mnN=mn, Assumption 2 can be equivalently written as

C{p+log⁡nn+r(p+q)N+Olog⁡NN}≤1.C\lbrace \frac{p+\log n}{n}+\frac{r(p+q)}{N}+\frac{O\log N}{N}\rbrace \leq 1.

Notably, the sample-size requirement for low-rank estimation, characterized by r(p+q)r(p+q), is governed by the total sample size NN rather than the local sample size nn, allowing q>nq>n when the overall sample-size requirement is satisfied. A similar sample-size requirement, N≳mp+Olog⁡NN\gtrsim mp+O\log N, appears in the scalar-response distributed robust estimation framework studied in [1].

More concretely, Assumption 2 implies

m\leq\frac{N}{C(p+\log n)},\qquad O\leq\frac{N}{C\log N}.$$ Thus, for a fixed total sample size $N$, the number of machines is limited by the local sample-size requirement for covariance estimation. Moreover, the number of contaminated observations is bounded above by order $N/\log N$, requiring the contamination proportion to vanish as $N$ increases. We have added a remark following Assumption 2 to clarify its implications for the problem dimensions and contamination levels in the revised manuscript. **Reference** [1] Liu, S., Zhao, D., and Yang, H. (2026). Distributed learning with adversarial contamination: an efficient and optimal approach based on dynamic regularization. *Statistics and Computing*, 36(4), Article 150. --- ##
Reviewer_Sts7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at reduced-rank regression when the data are spread across m machines and some rows of Y are corrupted. The model is the usual mean-shift one, Y = XB + sqrt(N)C + E, with B low rank and C row-sparse. The authors adapt the CSL idea by replacing the surrogate Hessian with a block-diagonal one. This decouples B and C: each machine hard-thresholds its own rows of C locally, and the central machine updates B with a closed-form reduced-rank step. The main theorem gives a Frobenius error for B of order sigmasqrt(r(p+q)/N) + sigmaO*(q+log N)/N after a constant number of communication rounds, where O is the number of corrupted rows. Experiments are simulations (four noise types) and four real datasets with synthetic contamination, compared against Dist-CSL, Dist-QRR, Dist-RRR and a one-shot averaging variant of their own method.

Critical strengths
  1. The estimation guarantee is for B itself, not the joint (B, C), and it holds in a distributed setting with a constant number of rounds. The paper argues this is a gap in earlier robust RRR work, and the bound has a clear reading: an oracle term plus a contamination term that grows linearly in O/N. The one-round contraction result and its multi-round extension are easy to follow.

  2. The block-diagonal surrogate is a simple idea and it works. It makes the C-update local (and an exact proximal step), and the simulations back it up. Under normal noise D4R essentially matches the full-sample oracle (0.240 vs 0.238 in Frobenius error), while Dist-CSL and Dist-QRR are off by an order of magnitude. The DC-D4R ablation and the rounds experiment (Fig. 2) support the claim that iterating matters.

Critical weaknesses
  1. The theory and the practice don't line up well. Theorem 4 chooses the thresholds lambda_k^(t) using quantities that are not known in practice (sigma, the initial error, the constant A, the local contamination o_k), while the experiments tune lambda with an hBIC and pick the rank with RSC. There is no guarantee for the data-driven version, and the initialization B^(0) is not really discussed in the main text even though the round count depends on it. Also, the analysis assumes sub-Gaussian noise, yet one of the two main settings in Table 1 is Cauchy noise, where D4R is about 2x worse than the oracle (1.55 vs 0.77). So the claim of robustness is partly outside what is proved. Contamination is also only in Y, with no leverage points in X, which is a narrower notion than the "adversarial" wording suggests. The main text has no lower bound or optimality discussion for the O*q/N term, and no discussion of a breakdown point.

  2. The empirical evidence is weaker than it looks. In the real-data experiments the contamination is synthetic and very easy to spot (a constant shift of sqrt(N) added to every response of the selected rows), which suits a thresholding method. D4R is flat in O/N and DC-D4R is nearly identical on Airline SIR, so these experiments don't say much about the iterative part. Some baselines look under-tuned: Dist-RRR reports rank 15 in Table 1 under both noise types, i.e. no rank reduction at all, which makes it a weak comparison. In the main text there is also no comparison with the centralized She & Chen estimator on real data and no robust-aggregation baseline, and the simulations are small (p=15, q=40). The "communication-efficient" claim is argued through the number of rounds, but I couldn't find in the main text a count of what is actually transmitted per round compared to the baselines.

Questions for authors

How is B^(0) obtained, and how sensitive are the results to a poor start? Condition (20) only requires e_B^(0)/sigma <= N^alpha1, which is quite loose.

Is there any result for the hBIC/RSC-tuned estimator, or at least an argument that it picks thresholds in the range required by Theorem 4? 1. What happens if the lambda's are mis-specified by a constant factor?

  1. Does the proof need the corruption C* to be independent of X and E, or can it be chosen after seeing the data? That matters for the word "adversarial".

  2. Why does Dist-RRR return rank 15 in Table 1, and were Dist-CSL and Dist-QRR tuned as carefully as D4R?

  3. Could you report the per-round communication cost (in floats) against the baselines, and try contamination that varies in size and sign, plus a few corrupted rows in X? This is a suggestion, not a requirement for my assessment.

  4. Could you discuss minimax optimality of the contamination term, and what the method does when the machines are unbalanced? Again, a suggestion only.

8 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the positive assessment of our work and the insightful comments on the theoretical guarantees, practical tuning, and empirical evaluation. We have carefully addressed the comments and revised the manuscript accordingly, with all revisions highlighted in red. Our point-by-point responses are provided below.

For clarity, our detailed responses to each weakness and question are provided in separate comments below.

AuthorsOpenReview ↗

Response to W1

W1

The theory and the practice don't line up well. Theorem 4 chooses thresholds λk(t)\lambda_k^{(t)} using quantities that are not known in practice (σ\sigma, the initial error, the constant AA, the local contamination oko_k), while the experiments tune lambda with an hBIC and pick the rank with RSC. There is no guarantee for the data-driven version, and the initialization B(0)B^{(0)} is not really discussed in the main text even though the round count depends on it. Also, the analysis assumes sub-Gaussian noise, yet one of the two main settings in Table 1 is Cauchy noise, where D4R is about 2x worse than the oracle (1.55 vs 0.77). So the claim of robustness is partly outside what is proved. Contamination is also only in YY, with no leverage points in XX, which is a narrower notion than the "adversarial" wording suggests. The main text has no lower bound or optimality discussion for the Oq/NOq/N term, and no discussion of a breakdown point.

Response:

We thank the reviewer for the detailed and constructive comments. We address the initialization and convergence analysis in Q1, the relationship between theoretical guarantees and practical hBIC/RSC selection in Q2, the scope of adversarial contamination in Q3, and the minimax comparison in Q6. We provide further clarification and additional experimental evidence below.

Robustness under heavy-tailed noise

Theorem 5 establishes non-asymptotic estimation guarantees under sub-Gaussian noise, while our experiments also evaluate heavy-tailed distributions. D4R retains the squared loss across all noise settings, whereas the full-sample oracle employs Huber loss under heavy-tailed noise and has access to the true contamination support and coefficient rank. Under Cauchy noise, Table 1 reports Frobenius estimation errors of 1.552 for D4R and 0.768 for the full-sample oracle. D4R has the lowest estimation error among all non-oracle methods in this setting.

Robustness to shifted covariates

We further evaluate D4R under simultaneous covariate shifts and response contamination. Specifically, 10% of the design rows are shifted in a common direction by a vector whose norm equals the median norm of the clean design rows. A separately selected 10% of the response rows are contaminated, with the nonzero entries of C∗C^\ast independently drawn from N(0,0.52)\mathcal N(0,0.5^2), consistent with Section 5.1. Responses are generated using the shifted design. As shown in Table R1, D4R achieves a Frobenius error of 0.233, comparable to the full-sample oracle (0.232), and outperforms the other distributed methods. This experiment provides additional empirical evidence of robustness to covariate shifts combined with response contamination. We have also included these experimental results in Appendix E.8 of the revised manuscript.

Table R1. Coefficient estimation errors and selected ranks under shifted covariates and response contamination. Entries are mean (SE) over 100 replications.

Method∥Δ∥F\lVert\Delta\lVert_F∥Δ∥∞\lVert\Delta\lVert_\infty∥Δ∥Σ\lVert\Delta\lVert_\Sigmar^\hat r
Full-sample0.232 (0.004)0.039 (0.002)0.204 (0.004)3.0 (0.0)
D4R0.233 (0.005)0.040 (0.002)0.205 (0.004)3.0 (0.0)
DC-D4R0.256 (0.009)0.044 (0.003)0.226 (0.008)3.0 (0.0)
Dist-CSL2.842 (0.271)0.477 (0.046)2.411 (0.203)3.9 (0.4)
Dist-QRR2.554 (0.471)0.527 (0.087)2.083 (0.349)3.2 (0.2)
Dist-RRR4.876 (0.064)0.676 (0.024)3.868 (0.043)15.0 (0.0)

Breakdown-point analysis

She and Chen (2017, Theorem 1) established that the finite-sample breakdown point of conventional reduced-rank regression is exactly 1/n1/n. For robust reduced-rank regression with the row-sparsity constraint ∥C∥2,0≤ϱ\lVert C\rVert_{2,0}\leq\varrho, their Theorem 4 further implies a risk-based finite-sample breakdown point of at least (ϱ+1)/n(\varrho+1)/n. In the distributed setting, D4R employs row-sparsity penalization and local hard-thresholding to identify contaminated observations. Under the conditions of Theorem 5, the estimation error bound depends on the number of contaminated observations OO, providing a finite-sample robustness guarantee for distributed reduced-rank regression.

Reference

She, Y., and Chen, K. (2017). Robust reduced-rank regression. Biometrika, 104(3), 633–647.

AuthorsOpenReview ↗

Response to W2 (Part 1/3)

W2

The empirical evidence is weaker than it looks. In the real-data experiments the contamination is synthetic and very easy to spot (a constant shift of N\sqrt{N} added to every response of the selected rows), which suits a thresholding method. D4R is flat in O/NO/N and DC-D4R is nearly identical on Airline SIR, so these experiments don't say much about the iterative part. Some baselines look under-tuned: Dist-RRR reports rank 15 in Table 1 under both noise types, i.e. no rank reduction at all, which makes it a weak comparison. In the main text there is also no comparison with the centralized She & Chen estimator on real data and no robust-aggregation baseline, and the simulations are small (p=15p=15, q=40q=40). The "communication-efficient" claim is argued through the number of rounds, but I couldn't find in the main text a count of what is actually transmitted per round compared to the baselines.

Response:

We thank the reviewer for these comments. We provide additional empirical comparisons and clarify the roles of iterative refinement, baseline tuning, and communication.

Response contamination and the benefits of iterative updates

The real-data experiments use constant response shifts, while the simulations consider row-sparse Gaussian response contamination. Table R1 further examines contamination in both XX and YY. D4R and DC-D4R achieve similar results on Airline SIR. In the simulations, D4R achieves lower Frobenius estimation errors than DC-D4R: 0.240 versus 0.394 under Gaussian noise and 1.552 versus 2.482 under Cauchy noise. Table R4 further reports the estimation errors across communication rounds.

Centralized robust RRR and robust aggregation

We added the centralized robust RRR estimator of She and Chen (2017), denoted by Centralized 4R, to the real-data comparisons, following the experimental settings in Section 5.2. Table R2 compares its test MSE with D4R across four datasets and different contamination proportions. D4R achieves lower test MSE on Beijing PM2.5 and Facebook Live, whereas Centralized 4R performs better on Airline SIR and Facebook Metrics. These results have also been incorporated into Section 5.2 and Appendix E.7 of the revised manuscript. Robust aggregation methods typically address Byzantine workers or corrupted model updates, whereas D4R considers sample-wise response contamination within individual machines.

Table R2. Test MSE (SE) of D4R and the centralized robust RRR estimator of She and Chen (2017) on four real datasets.

DatasetO/NO/ND4RCentralized 4R
Airline SIR0.050.851 (0.001)0.831 (0.005)
0.100.851 (0.001)0.830 (0.004)
0.150.851 (0.001)0.832 (0.005)
0.200.851 (0.001)0.833 (0.004)
Beijing PM2.50.050.595 (0.003)0.632 (0.009)
0.100.595 (0.003)0.633 (0.009)
0.150.595 (0.003)0.635 (0.009)
0.200.596 (0.003)0.640 (0.009)
Facebook Live0.050.836 (0.003)0.844 (0.013)
0.100.836 (0.003)0.846 (0.013)
0.150.835 (0.003)0.848 (0.014)
0.200.836 (0.003)0.851 (0.014)
Facebook Metrics0.050.839 (0.011)0.806 (0.028)
0.100.837 (0.010)0.808 (0.027)
0.150.834 (0.009)0.811 (0.026)
0.200.839 (0.009)0.799 (0.020)

Dist-RRR rank selection

Dist-RRR selects its rank through adaptive weighted singular-value thresholding, with the threshold tuned by validation prediction error. For p=15p=15 and q=40q=40, this procedure retains all 15 feasible components under both noise distributions in Table 1, rather than fixing the rank at 15 in advance. Table R3 further reports mean selected ranks of 15.0, 30.0, 37.6, and 40.0 for p=30,40,50p=30,40,50, respectively.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses to Reviewers

We thank all three reviewers for their careful evaluations and constructive feedback. We have provided detailed point-by-point responses and revised the manuscript accordingly, with all revisions highlighted in red. The key responses and corresponding revisions are summarized below.

  • Core contributions and novelty (Reviewer 9rgE, W1). We clarified the novelty of D4R in distributed low-rank multivariate regression under sample-wise adversarial contamination. We emphasized the refined block-diagonal surrogate for local contamination estimation and closed-form global rank-constrained updates, together with direct non-asymptotic coefficient estimation guarantees. We strengthened the contribution summary in Section 1 and the comparisons with existing methods in Section 2.
  • Theoretical analysis and assumptions (Reviewer Sts7, W1 and Q1, Q3, Q6; Reviewer 9rgE, Q1; Reviewer wFCr, Q4). We clarified the conditions for constant-round convergence, the implications of Assumption 2, and the scope of data-adaptive adversarial contamination. We refined the proof of Lemma C.1 and further discussed initialization sensitivity, minimax rates, breakdown points, and unequal data partitions. The manuscript revisions include a remark following Assumption 2 and the revised concentration proof in Appendix C. Additional initialization experiments are reported in Table R6 of our response to Reviewer Sts7.
  • Practical tuning and implementation (Reviewer Sts7, Q2 and Q4; Reviewer wFCr, W1 and Q1–Q2). We clarified the relationship between hBIC/RSC selection and theoretical thresholds, introduced restrictions preventing saturated hBIC solutions, and established additional support- and rank-recovery guarantees. We also provided the D4R implementation code and clarified DC-D4R's rank selection and baseline tuning. The hBIC restriction is specified after Eq. (21), with additional theoretical results in Appendix B.3 and D4R's tuning procedures described in Section 5.1.
  • Theoretical scope and empirical robustness (Reviewer Sts7, W1 and W2; Reviewer wFCr, Q2). We distinguished the theoretical guarantees under sub-Gaussian noise from empirical robustness under heavy-tailed noise. We added experiments involving simultaneous covariate shifts and response contamination, in which D4R achieves estimation accuracy close to the full-sample oracle. We also evaluated its performance under lower-magnitude Gaussian contamination. The covariate-shift experiments are included in Appendix E.8, while the lower-magnitude results are reported in Table R1 of our response to Reviewer wFCr.
  • Expanded empirical comparisons (Reviewer Sts7, W2 and Q4). We added the centralized robust reduced-rank regression estimator of She and Chen (2017) to the real-data comparisons and extended simulations to larger feature dimensions. We also clarified Dist-RRR's rank-selection behavior and the benefits of iterative refinement over one-shot averaging. The additional empirical results are incorporated into Section 5.2 and Appendices E.7 and E.9.
  • Communication costs and computational complexity (Reviewer Sts7, W2 and Q5; Reviewer 9rgE, W2; Reviewer wFCr, W2 and Q3). We quantified per-round communication costs, compared D4R with Dist-CSL at matched communication volumes, and accounted for communication during rank selection. We also refined the complexity comparison with an optimized centralized implementation using thin QR factorization, distinguishing parallelization benefits from reductions in total computation. The computational and communication analysis is clarified in Section 3, with existing wall-clock results in Appendix E.6 and detailed communication comparisons in the rebuttal tables.

We believe these additional analyses, experiments, and clarifications have strengthened the manuscript and more clearly established the contributions, theoretical scope, and practical performance of D4R.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.