- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes SparseBank, which uses G-optimal design and OLS completion to enable sparse measurement for workflow-bank selection and routing. Theoretically, it establishes prediction error bounds and their implications for bank-selection and routing regret. Experiments on APPS, CodeContests, and HumanEval+ evaluate the effectiveness of the proposed method.
Critical strengths
- The work addresses a practically relevant problem of reducing workflow evaluation costs through sparse measurements. The empirical results on APPS and CodeContests demonstrate the effectiveness of the proposed approach.
- Theoretical guarantees are established linking utility estimation errors to downstream bank-selection and routing regret, providing a principled basis for analyzing the quality of the resulting decisions.
Critical weaknesses
- The theoretical guarantees rely on strong assumptions that are not verified in real-world experiments. On APPS, the sufficient sample size for Theorem 1 exceeds the 800 distinct query–workflow pairs. Although Corollary 4 does not require this sample size condition, its model and noise assumptions remain unverified.
- HumanEval+ does not seem to demonstrate a statistically conclusive acquisition advantage, raising concerns about the consistency of the empirical improvements across benchmarks.
The introduction could better motivate the key challenges and clarify how the proposed approach differs from existing experimental design methods. As I am not an expert in optimal experimental design, I find it difficult to fully assess its methodological novelty relative to prior work.
Questions for authors
Given that the sufficient sample-size condition in Theorem 1 exceeds the number of distinct query–workflow pairs on APPS, to what extent do the theoretical guarantees inform the observed empirical performance?
Dear Reviewer q1wi,
We would like to briefly highlight that the revised manuscript now incorporates the additional analyses addressing your concerns, particularly the matched-support comparisons, alternative-support controls, and independent CodeContests execution results (Sections 6.4–6.5 and Appendices D–E).
We appreciate your review and would welcome any further questions or concerns during the remaining discussion period.