- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to select a high-quality candidate under a limited budget for exact evaluation. The proposed QualityHit loss trains a scoring model using sets of candidates that meet different utility thresholds. At deployment, the model ranks the candidates, the highest-scoring candidates are evaluated exactly, and the best evaluated candidate is returned. Experiments on scheduling and routing tasks report improvements over regression and ranking baselines.
Critical strengths
The problem is natural, and the loss offers a simple way to align training with the quality of the final selection. The experiments compare several baseline families using the same model architecture and training budget. The consistently positive mean gains across tasks and budgets provide initial evidence that the loss can improve candidate selection.
Critical weaknesses
-
The experimental data and task settings are insufficiently specified. Key details about instance sources, problem sizes, candidate generation, and input features are missing. The supplementary material also lacks the generators and exact evaluation procedures used in the reported experiments. These omissions prevent readers from reconstructing the test settings and assessing their difficulty and representativeness.
-
The practical value of the reported improvements remains unclear. The experiments use controlled candidate pools without establishing how these pools reflect the data, constraints, and evaluation costs of real applications. For a contribution centered on a simple loss design, evidence from practically meaningful tasks is essential. Improvements on pools whose sources and construction are unclear do not adequately establish the method’s value for practical candidate screening.
The presentation also needs substantial improvement. The appendix repeatedly precedes short proofs of elementary facts with overviews that repeat the same reasoning. The normalization described for Figure 2 does not match the plotted quantities, and Figure 3 does not clearly define the comparison shown in panel (a). These issues seriously hinder interpretation of the experimental results. Using AI tools to assist with writing does not lessen the authors’ responsibility for the clarity and accuracy of the manuscript. The authors should thoroughly check the text, figures, and captions so that readers can understand the results without guessing or filling in missing explanations.
I see value in this simple loss design. Even modest gains on an important task using real data could constitute a worthwhile contribution. At present, the data description and experimental evidence are insufficient to establish that value. I would consider raising my recommendation to weak acceptance if the authors clarify the experimental settings and provide compelling evidence of practical benefits.
Questions for authors
Which datasets or generation procedures supply the instances for each task family? Please specify the problem sizes, candidate construction, input features, and exact evaluation procedures, and provide the corresponding materials.
What evidence demonstrates that the method improves performance on datasets from meaningful real applications? Please clarify the practical gains over appropriate baselines and the cost of exact evaluation in these settings.