acceptodds
Under review as a conference paper at ICLR 2027

Damped Gauss–Newton Search for Multi-Metric Hyperparameter Optimization

Abstract

We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a black-box, target-seeking damped Gauss–Newton method for multiple validation responses. In contrast to uninformed candidate search such as grid or random search, the method uses numerical sensitivity information inferred from previously observed changes in hyperparameters and validation performance. Successive full-vector observations are used to construct an iterative secant approximation of the local Jacobian, which relates changes in the performance vector to changes in all optimized hyperparameters; a damped Gauss–Newton system then produces a joint update of the complete hyperparameter vector rather than optimizing coordinates independently. The method does not differentiate through model training or require dedicated coordinate-wise perturbation runs: following the initial two evaluations used to establish the first secant, each subsequent iteration requires one new full-vector evaluation. Tikhonov regularization stabilizes the resulting local inverse problem when the number of optimized parameters exceeds the number of performance responses. Unlike conventional multi-objective HPO aimed at approximating a Pareto front, our formulation seeks a specified target operating point in the multi-response performance space. We evaluate the same optimization principle in three complementary settings: four-dimensional XGBoost training-time HPO on three public classification datasets, six-dimensional LoRA/SFT post-training of Qwen3-VL-8B on a binary VQAv2 subset, and eight-dimensional post-processing threshold optimization on fixed classifier outputs. On the XGBoost benchmarks, the proposed method achieves validation performance comparable to grid search, random search, and TPE under a smaller number of model evaluations, although it does not consistently outperform these black-box baselines. The VLM study demonstrates joint post-training hyperparameter updates using TPR/FPR feedback and reveals strongly non-monotonic response and sensitivity trajectories, including a high-TPR/high-FPR degenerate regime. The threshold study further demonstrates target seeking in an underdetermined two-response, eight-parameter system and shows sensitivity to initialization and the importance of retaining the best observed iterate. Across these settings, the results support trajectory-based secant sensitivity as an evaluation-efficient local optimization mechanism complementary to global black-box HPO.

Primary area:
optimization
Keywords:
Hyperparameter Optimization, Newton Method, Multi-Objective Optimization, Numerical Optimization, Jacobian-Based Optimization, Machine Learning, Black-Box Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_P54UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a black box method for hyperparameter optimization that aims to match multiple validation metrics to target values. It formulates the problem as least squares optimization and jointly updates hyperparameters using a sensitivity matrix estimated from successive evaluations and a regularized Gauss Newton step. After two initial evaluations, each iteration requires only one additional evaluation. The method is tested on XGBoost tuning, LoRA training of a vision language model, and class specific threshold optimization. The XGBoost results show performance comparable to standard search methods with fewer evaluations, while the other experiments reveal nonmonotonic behavior and sensitivity to initialization.

Critical strengths
  1. The motivation is practical. Reducing the number of training runs matters when each evaluation is expensive. In many deployments the user also wants a specific operating point, such as a given TPR and FPR pair, rather than a full Pareto front. Eq.(1) states this problem clearly. Keeping both TPR and FPR visible also helps reveal cases where higher recall comes with more false positives.

  2. The paper is open about its limits. The authors state that the method does not consistently beat the baselines, that it depends on initialization, and that keeping the best iterate is necessary.

Critical weaknesses
  1. The authors report comparable XGBoost validation accuracy with eight evaluations, compared to twenty for random search and TPE. However, the paper lacks comparisons under equal evaluation budgets and performance curves across different budgets. As a result, the claimed advantage in evaluation efficiency over standard search methods is not well supported. The remaining experiments mainly demonstrate nonmonotonic behavior and sensitivity to initialization, without showing a clear practical advantage over standard HPO methods.

  2. The VLM study has no baseline, and the tuned model performs close to chance. There is one run of nine iterations, with no comparison against fixed hyperparameters, random search, or TPE at the same training budget.

  3. The images and the text descriptions do not match. I think the authors need to carefully check the details of the paper. Figure 10 includes a line stating 94.7% TP and 16.3% FP, which conflicts with its caption of 94.0/15.3. The caption of Figure~8 refers to iteration 200, while the plot runs to about 250.

Questions for authors
  1. Could you clarify how Eq. (4) distinguishes individual hyperparameter effects when all parameters change simultaneously? What evidence supports interpreting the plotted curves as parameter-specific sensitivities?

  2. Why are TPR and FPR almost equal on the held out split in Case~II? What is the zero shot performance of Qwen3-VL-8B on this subset? Could you add a fixed hyperparameter baseline and random search or TPE under the same training budget?

  3. Search time is a key practical concern in HPO. Could you report the total search time, including training, validation, and optimizer overhead? Does the proposed method reach similar performance faster than the baselines under comparable hardware and parallelization settings?

2 comments

I agree with the other two reviewers that this submission violates the double-blind review policy. Given this anonymity violation, I do not think it is necessary to proceed further with the rebuttal process.

Thank you for your review, constructive comments, and time. The anonymized GitHub repository inadvertently contained some author-identifying information, which has since been corrected. We acknowledge this oversight and appreciate your understanding.

Reviewer_JC69OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper should be desk rejected because it violates the policy regarding anonimity

Critical strengths

This paper should be desk rejected because it violates the policy regarding anonimity

Critical weaknesses

This paper should be desk rejected because it violates the policy regarding anonimity

Questions for authors

This paper should be desk rejected because it violates the policy regarding anonimity

Reviewer_nqC9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Desk Reject

When following this link to the GitHub, the README.md references a preprint on arxiv and how to cite the authors. According to https://iclr.cc/Conferences/2027/AuthorGuidelines this violates

ICLR 2027 is double blind, which means that all submitted papers should be anonymous. Any paper where author identity is revealed in either the main text or the supplementary material will be desk rejected. Note that related arxiv papers by the same authors do not break anonymity; if cited, these should be cited in third person.

Rating and verdicts below are only a mock review - paper should be desk rejected.

Critical strengths
Critical weaknesses
Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.