- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a black box method for hyperparameter optimization that aims to match multiple validation metrics to target values. It formulates the problem as least squares optimization and jointly updates hyperparameters using a sensitivity matrix estimated from successive evaluations and a regularized Gauss Newton step. After two initial evaluations, each iteration requires only one additional evaluation. The method is tested on XGBoost tuning, LoRA training of a vision language model, and class specific threshold optimization. The XGBoost results show performance comparable to standard search methods with fewer evaluations, while the other experiments reveal nonmonotonic behavior and sensitivity to initialization.
Critical strengths
-
The motivation is practical. Reducing the number of training runs matters when each evaluation is expensive. In many deployments the user also wants a specific operating point, such as a given TPR and FPR pair, rather than a full Pareto front. Eq.(1) states this problem clearly. Keeping both TPR and FPR visible also helps reveal cases where higher recall comes with more false positives.
-
The paper is open about its limits. The authors state that the method does not consistently beat the baselines, that it depends on initialization, and that keeping the best iterate is necessary.
Critical weaknesses
-
The authors report comparable XGBoost validation accuracy with eight evaluations, compared to twenty for random search and TPE. However, the paper lacks comparisons under equal evaluation budgets and performance curves across different budgets. As a result, the claimed advantage in evaluation efficiency over standard search methods is not well supported. The remaining experiments mainly demonstrate nonmonotonic behavior and sensitivity to initialization, without showing a clear practical advantage over standard HPO methods.
-
The VLM study has no baseline, and the tuned model performs close to chance. There is one run of nine iterations, with no comparison against fixed hyperparameters, random search, or TPE at the same training budget.
-
The images and the text descriptions do not match. I think the authors need to carefully check the details of the paper. Figure 10 includes a line stating 94.7% TP and 16.3% FP, which conflicts with its caption of 94.0/15.3. The caption of Figure~8 refers to iteration 200, while the plot runs to about 250.
Questions for authors
-
Could you clarify how Eq. (4) distinguishes individual hyperparameter effects when all parameters change simultaneously? What evidence supports interpreting the plotted curves as parameter-specific sensitivities?
-
Why are TPR and FPR almost equal on the held out split in Case~II? What is the zero shot performance of Qwen3-VL-8B on this subset? Could you add a fixed hyperparameter baseline and random search or TPE under the same training budget?
-
Search time is a key practical concern in HPO. Could you report the total search time, including training, validation, and optimizer overhead? Does the proposed method reach similar performance faster than the baselines under comparable hardware and parallelization settings?
2 comments
Thank you for your review, constructive comments, and time. The anonymized GitHub repository inadvertently contained some author-identifying information, which has since been corrected. We acknowledge this oversight and appreciate your understanding.
I agree with the other two reviewers that this submission violates the double-blind review policy. Given this anonymity violation, I do not think it is necessary to proceed further with the rebuttal process.