- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a gradient modification called the quadratic gradient for logistic regression training. It rescales each gradient coordinate using a diagonal matrix computed from a fixed Hessian approximation, then applies this modification to NAG, AdaGrad, and Adam. The authors evaluate these optimizers on eight datasets in plaintext and implement the NAG version over encrypted data using HEAAN. The encrypted experiments cover six datasets and compare four iterations of the proposed method with seven iterations of the baseline. The paper reports shorter training times and claims that the modification provides a general way to combine the efficiency of first-order methods with curvature information.
Critical strengths
-
The design is simple and fits the constraints of encrypted computation.
-
The paper includes an encrypted implementation with concrete cost measurements. The authors implement the method in HEAAN and report training time, storage, accuracy, and AUC.
Critical weaknesses
-
The paper does not make a strong case for the importance of this contribution on an already mature problem. Encrypted logistic regression has been studied for many years. For an ICLR paper, I would need a clearer explanation of why this particular modification matters now. The encrypted experiments revisit six relatively small tabular datasets and compare only with Kim et al. (2018a). The larger financial and MNIST experiments are conducted in plaintext, so they do not demonstrate improved scalability of encrypted training.
-
The experiments do not show that the proposed gradient modification is responsible for the AdaGrad and Adam gains. The paper uses a fixed positive scaling for each gradient coordinate throughout training (pp. 5-7). In AdaGrad, that same scaling appears in both the gradient and the accumulated normalization, so it cancels apart from its effect on the numerical stabilizer. The same issue applies to Adam's first and second moments. This makes the claim that these methods benefit from added curvature information difficult to justify. Meanwhile, the enhanced AdaGrad uses a learning rate of 0.07 instead of 0.01, and the enhanced Adam uses 0.011 instead of 0.001 (p. 7). These are sevenfold and elevenfold increases. The paper does not show what happens when the original optimizers receive the same learning rates or an equal tuning budget. The faster curves could therefore be explained by the changed learning rates. This directly weakens the claim of a general optimization improvement.
Questions for authors
-
What advantage remains for enhanced AdaGrad and Adam when the original and enhanced versions use the same learning rates and an equal tuning budget? How do you explain the claimed benefit given that fixed coordinate scaling largely cancels through adaptive normalization?
-
Could you add comparisons with more recent methods for logistic regression training under homomorphic encryption? The current comparison is limited to Kim et al. (2018a), so it is hard to tell whether the proposed method still offers a meaningful advantage over newer approaches.