acceptodds
Withdrawn from ICLR 2027

Privacy-Preserving Logistic Regression Training with A Faster Gradient Variant

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training logistic regression over encrypted data has emerged as a prominent approach to addressing security concerns in recent years. In this paper, we introduce an efficient gradient variant, termed the quadratic gradient, which is specifically designed for privacy-preserving logistic regression while remaining equally effective in plaintext optimization. By incorporating this quadratic gradient, we enhance Nesterov's Accelerated Gradient (NAG), Adaptive Gradient (AdaGrad), and Adam algorithms. We evaluate these enhanced algorithms across various datasets, with experimental results demonstrating state-of-the-art convergence rates that significantly outperform traditional first-order gradient methods. Furthermore, we apply the enhanced NAG method to implement homomorphic logistic regression training, achieving comparable performance within only four iterations. The proposed quadratic-gradient approach offers a unified framework that synergizes the advantages of first-order gradient-based methods and second-order Newton-type methods, suggesting broad applicability to diverse numerical optimization tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Homomorphic Encryption, Logistic Regression, Quadratic Gradient, Simplified Fixed Hessian, Nesterov’s Accelerated Gradient

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BuVqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a gradient modification called the quadratic gradient for logistic regression training. It rescales each gradient coordinate using a diagonal matrix computed from a fixed Hessian approximation, then applies this modification to NAG, AdaGrad, and Adam. The authors evaluate these optimizers on eight datasets in plaintext and implement the NAG version over encrypted data using HEAAN. The encrypted experiments cover six datasets and compare four iterations of the proposed method with seven iterations of the baseline. The paper reports shorter training times and claims that the modification provides a general way to combine the efficiency of first-order methods with curvature information.

Critical strengths
  1. The design is simple and fits the constraints of encrypted computation.

  2. The paper includes an encrypted implementation with concrete cost measurements. The authors implement the method in HEAAN and report training time, storage, accuracy, and AUC.

Critical weaknesses
  1. The paper does not make a strong case for the importance of this contribution on an already mature problem. Encrypted logistic regression has been studied for many years. For an ICLR paper, I would need a clearer explanation of why this particular modification matters now. The encrypted experiments revisit six relatively small tabular datasets and compare only with Kim et al. (2018a). The larger financial and MNIST experiments are conducted in plaintext, so they do not demonstrate improved scalability of encrypted training.

  2. The experiments do not show that the proposed gradient modification is responsible for the AdaGrad and Adam gains. The paper uses a fixed positive scaling for each gradient coordinate throughout training (pp. 5-7). In AdaGrad, that same scaling appears in both the gradient and the accumulated normalization, so it cancels apart from its effect on the numerical stabilizer. The same issue applies to Adam's first and second moments. This makes the claim that these methods benefit from added curvature information difficult to justify. Meanwhile, the enhanced AdaGrad uses a learning rate of 0.07 instead of 0.01, and the enhanced Adam uses 0.011 instead of 0.001 (p. 7). These are sevenfold and elevenfold increases. The paper does not show what happens when the original optimizers receive the same learning rates or an equal tuning budget. The faster curves could therefore be explained by the changed learning rates. This directly weakens the claim of a general optimization improvement.

Questions for authors
  1. What advantage remains for enhanced AdaGrad and Adam when the original and enhanced versions use the same learning rates and an equal tuning budget? How do you explain the claimed benefit given that fixed coordinate scaling largely cancels through adaptive normalization?

  2. Could you add comparisons with more recent methods for logistic regression training under homomorphic encryption? The current comparison is limited to Kim et al. (2018a), so it is hard to tell whether the proposed method still offers a meaningful advantage over newer approaches.

Reviewer_cT5aOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses privacy-preserving logistic regression training over homomorphically encrypted data. The author propose quadratic gradient, which bridges first-order gradient-based methods and the second-order Newton-type algorithms. Quadratic gradient utilizes a diagonal approximation of the fixed Hessian that provides a valid lower bound on the Hessian of the log-likelihood regardless of the sign of its entries, which ensures monotone ascent of the fixed-Hessian update and so evit from repeated Hessian inversion the main computational burden of the Newtom-type methods. The authors incorporate the quadratic gradient into NAG, AdaGrad, and Adma and evaluate them in plaintext, where the enhanced variants converge faster than the original optimizers. Then they implement the enhanced NAG on encrypted data with HEAAN and compare it with a prior homomorphic logistic regression method, reporting fewer iterations and lower training time with accuracy and AUC claimed to be comparable.

Critical strengths

1. Sound construction of the diagonal Hessian bound

The proposed diagonal approximation is a valid lower bound on the Hessian regardless of the sign of its entries, ensuring monotone ascent of the fixed-Hessian update without the sign restriction of SFH.

2. Reduced iterations and training time

The enhanced NAG converges faster than the original NAG in plaintext, and reduces the number of iterations and the training time in the encrypted setting compared with a prior method.

Critical weaknesses

1. Novelty

The quadratic gradient seems to simply multiply each gradient coordinate by a constant and the construction of diagoanl approxmiation appears to follow directly from the standard diagonal dominance argument. It would help to clarify what is new beyond applying these known techniques to existing optimizers.

2. Effect on AdaGrad and Adam

AdaGrad and Adam seems to be nearly invariant to the quadratic gradient, since a constant per-coordinate scaling cancels out in the normalized updates. It is therefore unclear why the quadratic gradient would enhance these optimizers at all. Moreover, Figures 2 and 3 use much larger learning rates than the baselines, so it is also unclear whether the experiments support the claimed improvement.

3. Theoretical guarantees

No convergence analysis is provided for NAG, AdaGrad, or Adam combined with the quadratic gradient.

4. Baselines

The paper presents the quadratic gradient as a generalization of SFH, yet it is compared only with vanilla first-order optimizers at their default learning rates.

5. Performance in the encrypted setting

The paper claims performance comparable to the baseline in the encrypted setting, but in Tables 1 and 2 the AUC is lower on five of six datasets and the storage cost is doubled.

Questions for authors
  1. What is the methodological novelty of the quadratic gradient?
  2. Why would the quadratic gradient improve AdaGrad and Adam theoretically?
  3. How do the enhanced variants compare with the baselines under the same learning rate?
  4. Can the authors provide convergence guarantees for NAG, AdaGrad, and Adam combined with the quadratic gradient?
  5. How are the features normalized in the experiments? If they are nonnegative, how does B~\tilde{B} differ from the SFH approximation?
Reviewer_nAVTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the problem of running first-order optimizers for logistic regression on encrypted data via homomorphic encryption (HE). The authors build on the "simplified fixed Hessian" (SFH) approach from past work; the idea is that Newton's method but with a fixed Hessian that always lower bounds the Hessian (for maximization, upper bounds it for minimization) leads to more conservative gradient steps and thus still converges, but does not require recomputing the full Hessian in every iteration. SFH takes this further by having the fixed Hessian be diagonal. However, SFH is prone to singularity issues and the fixed Hessian might not always be non-negative which is assumed for convergence. The authors propose a "quadratic gradient" method to alleviate these issues. It adds a numerical stability constant that avoids the singularity issue, and while SFH summed the rows of the fixed Hessian to get its diagonal, the authors sum the absolute values and take the negation. They show the quadratic gradient can be used as a drop-in replacement in algorithms like Nesterov acceleration, AdaGrad, and Adam. Evaluations show the quadratic gradient outperforms the standard gradient in terms of utility, and that under HE the method is more efficient than standard Nesterov acceleration.

Critical strengths

None.

Critical weaknesses
  • The method seems to be a minor modification of past work. Adding the stability constant ϵ\epsilon is a pretty standard way to deal with singularity issues, e.g. the AdaGrad / Adam optimizers use a similar trick. Using absolute values to force the fixed Hessian's eigenvalues to lower-bound the true eigenvalues makes the SFH approach more widely applicable, but is a pretty heavy-handed way to force the condition that B~≼Hˉ\tilde{B} \preccurlyeq \bar{H} (in particular, if Hˉ\bar{H} is a constant times the all-ones matrix, B~\tilde{B} can over-estimate the eigenvalues by n\sqrt{n} where nn is the number of rows/columns) and also not a particularly interesting or insightful way to force it.
  • The evaluations seem to miss key comparisons:
    • They use a different learning rate schedule for the standard and enhanced algorithms. It's not clear how much the learning rate schedule is responsible for improvements in the MLE metric.
    • The authors are building upon the work of Bonte and Vercauteren, 2018, yet the experiments don't seem to be either comparing to their method or demonstrating that their method fails in some of the described settings, instead comparing to a method with no preconditioning on the gradients.
    • The dimension is pretty small in most of the experiments, and as mentioned above the looseness of the fixed Hessian approximation the authors use could increase substantially with the dimension. It's not clear from the evaluations if this issue will manifest in practice or not. Even experiments on synthetic logistic regression datasets (run without HE, just to measure utility rather than performance) could be informative and should be easy to run.
Questions for authors
  • Why do the standard and enhanced variants use different learning rate schedules? Did you try using the standard variant with your learning rate schedule to demonstrate that the learning rate schedule couldn't explain the difference in performance by itself?
  • In which of the evaluation settings does Bonte and Vercauteren, 2018 fail? You mention MNIST early in the paper, but what about the rest?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.