- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
OSTA selects among utility models and LLM based agents using estimated reliability and a cost penalty. Experiments on six decision datasets show improved prediction accuracy. The reported efficiency results count the cost of selected outputs in an evaluation with cached candidate answers.
Critical strengths
The results show useful differences between the utility models and LLM agents. Selecting among them improves average prediction accuracy. The appendix explains the calibration procedure in detail, and the ablations help identify which reliability signals contribute to performance.
Critical weaknesses
- The cost savings are not demonstrated for live inference. Appendix A.5 says that peer LLM answers are already available, while the reported cost counts only the selected answer.
- The routing comparison gives OSTA more information. It uses current candidate answers, while the baselines make selections with less information.
- The statistical reporting is confusing. Figure 4 refers to 60K test samples, while Section 4.2 describes bootstrap samples. The actual test sizes and resampling procedure need clarification.
Questions for authors
What are the accuracy and cost results when every LLM call needed for selection is charged? Does the advantage remain when all methods receive the same information at selection time?