- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BRACE, a method for selecting validation tasks when deciding whether to accept an agent update. It learns how updates affect different tasks from historical results, then chooses tests expected to reduce the risk of making a wrong decision. Experiments with SkillOpt and EvoSkill on SpreadsheetBench and DocVQA show fewer validation executions with similar or slightly higher final success rates. The authors also provide ablations of task selection and the use of historical information.
Critical strengths
- The topic is important, and selecting tests according to their effect on the acceptance decision appears reasonable.
- The experiments compare several validation methods and include controlled comparisons using the same candidate updates.
Critical weaknesses
-
The reliability of the stopping rule needs further evaluation. Each activation check uses only four candidate updates, and the uncertainty intervals cover the true values less often than intended. Additional Monte Carlo samples cannot correct an inaccurate prediction model. I would like to see how often harmful updates are accepted at different stages of the 100-run experiment, especially between model refreshes.
-
The quality and cost results in Table 1 cover different periods. Cost is averaged over 100 runs, but success is measured only on run 100, after BRACE has accumulated substantial history. This leaves its performance in earlier runs unclear. Reporting success and cost together at several stages would make the comparison more informative. In addition, the gains over DPP are only 0.25–0.50 percentage points, so statistical uncertainty should be considered when discussing higher success rates.
-
The experiments are limited to spreadsheet manipulation and document visual question answering, with validation drawn from a fixed 40-task catalog. It is unclear whether the historical response patterns remain useful in more open-ended, long-horizon interactions, where the effects of an update may unfold over many steps and depend on changing environment states.
Questions for authors
-
Do the validation savings outweigh the total cost of collecting history, training the response model, and running BRACE?
-
BRACE uses historical information to guide validation while leaving candidate generation unchanged. Could historical information also guide skill updates, so that skill evolution and validation improve together? I would like to see some discussion of this possibility beyond the current validation-only use of the response model.
3 comments
Following up on our response to your review
Thank you again for the thought and care you put into your review. We appreciated how clearly you explained what you wanted to understand better about the evaluation. Your questions prompted us to examine our results more closely, and we are grateful that your feedback helped us explain what we found more clearly.
If you have a chance to look through our full response, we would be glad to hear whether it addresses your concerns or if there is anything you would like us to explain further.
Our sincere thanks for helping us improve the evaluation
As the rebuttal discussion draws to a close, we wanted to thank you once more for the thought and care you put into reviewing our work.
We especially appreciated how clearly you explained what you wanted to understand about BRACE’s behavior over time. Your questions prompted us to examine the evaluation across different stages and carry out additional analyses. That work helped us give a clearer account of reliability and cost.
We are grateful that your comments led us to look more closely at the evidence. We learned from doing so, and we wanted to express our appreciation before the discussion ends. Thank you again for contributing your time and expertise to improving the paper.
We wish you good health, happiness, and continued success in your research and work.
Follow-up to Reviewer vqHD
Thank you again for asking how BRACE’s reliability, quality, and cost change over a campaign. In response, we audited harmful commits between checkpoint refreshes and checked predictive interval coverage at the actual stopping points (W1). We also compared held-out success and cumulative validation cost at the same run indices (W2), and ran separate experiments with changing validation tasks and with skill evolution without resets (W3). We report timing that accounts for the cost of collecting history and fitting the response model (Q1), and discuss how paired evaluation results could inform future skill proposals and their validation (Q2).
If your schedule allows, we would be grateful if you could take another look at our response. If you find that the additional results and revisions address your concerns, we would be grateful if you would consider raising your rating.