Damped Gauss–Newton Search for Multi-Metric Hyperparameter Optimization

We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a black-box, target-seeking damped Gauss–Newton method for multiple validation responses. In contrast to uninformed candidate search such as grid or random search, the method uses numerical sensitivity information inferred from previously observed changes in hyperparameters and validation performance. Successive full-vector observations are used to construct an iterative secant approximation of the local Jacobian, which relates changes in the performance vector to changes in all optimized hyperparameters; a damped Gauss–Newton system then produces a joint update of the complete hyperparameter vector rather than optimizing coordinates independently. The method does not differentiate through model training or require dedicated coordinate-wise perturbation runs: following the initial two evaluations used to establish the first secant, each subsequent iteration requires one new full-vector evaluation. Tikhonov regularization stabilizes the resulting local inverse problem when the number of optimized parameters exceeds the number of performance responses. Unlike conventional multi-objective HPO aimed at approximating a Pareto front, our formulation seeks a specified target operating point in the multi-response performance space. We evaluate the same optimization principle in three complementary settings: four-dimensional XGBoost training-time HPO on three public classification datasets, six-dimensional LoRA/SFT post-training of Qwen3-VL-8B on a binary VQAv2 subset, and eight-dimensional post-processing threshold optimization on fixed classifier outputs. On the XGBoost benchmarks, the proposed method achieves validation performance comparable to grid search, random search, and TPE under a smaller number of model evaluations, although it does not consistently outperform these black-box baselines. The VLM study demonstrates joint post-training hyperparameter updates using TPR/FPR feedback and reveals strongly non-monotonic response and sensitivity trajectories, including a high-TPR/high-FPR degenerate regime. The threshold study further demonstrates target seeking in an underdetermined two-response, eight-parameter system and shows sensitivity to initialization and the importance of retaining the best observed iterate. Across these settings, the results support trajectory-based secant sensitivity as an evaluation-efficient local optimization mechanism complementary to global black-box HPO.