acceptodds
Under review as a conference paper at ICLR 2027

Test What Matters: Learning to Verify Updates in Self-Evolving Agents

Abstract

Self-evolving language agents improve by updating instructions, memories, and skills, but identifying beneficial updates incurs substantial verification cost. When candidates are evaluated using fixed task sets or random sampling, the selected tasks may not match those actually affected by the update. This mismatch can waste evaluations on unaffected tasks and leave critical improvements or regressions undetected, leading to incorrect update decisions. We introduce BRACE, Bayesian Risk-Aware Candidate Evaluation, which selects validation tasks to inform candidate-specific deployment decisions. BRACE learns shared response structure offline from historical paired outcomes to guide validation task selection. During online verification, BRACE updates predictions from new paired observations and uses Decision Value of Information (DVI) to prioritize tests by their expected reduction in deployment risk. A calibrated posterior-risk threshold governs stopping. We evaluate BRACE in SkillOpt and EvoSkill on SpreadsheetBench-Verified and DocVQA across four language-model configurations. With Qwen3.5-4B, BRACE reduces fresh online validation episodes by 54.0–82.3% relative to validation with a determinantal point process (DPP) coreset while achieving higher mean terminal-run success rates. Cross-model comparisons also show lower total validation cost and higher mean terminal-run success rates than host-native validation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving Agents, Update Verification, Adaptive Task Selection, Bayesian Decision Theory, Value of Information

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_vqHDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BRACE, a method for selecting validation tasks when deciding whether to accept an agent update. It learns how updates affect different tasks from historical results, then chooses tests expected to reduce the risk of making a wrong decision. Experiments with SkillOpt and EvoSkill on SpreadsheetBench and DocVQA show fewer validation executions with similar or slightly higher final success rates. The authors also provide ablations of task selection and the use of historical information.

Critical strengths
  1. The topic is important, and selecting tests according to their effect on the acceptance decision appears reasonable.
  2. The experiments compare several validation methods and include controlled comparisons using the same candidate updates.
Critical weaknesses
  1. The reliability of the stopping rule needs further evaluation. Each activation check uses only four candidate updates, and the uncertainty intervals cover the true values less often than intended. Additional Monte Carlo samples cannot correct an inaccurate prediction model. I would like to see how often harmful updates are accepted at different stages of the 100-run experiment, especially between model refreshes.

  2. The quality and cost results in Table 1 cover different periods. Cost is averaged over 100 runs, but success is measured only on run 100, after BRACE has accumulated substantial history. This leaves its performance in earlier runs unclear. Reporting success and cost together at several stages would make the comparison more informative. In addition, the gains over DPP are only 0.25–0.50 percentage points, so statistical uncertainty should be considered when discussing higher success rates.

  3. The experiments are limited to spreadsheet manipulation and document visual question answering, with validation drawn from a fixed 40-task catalog. It is unclear whether the historical response patterns remain useful in more open-ended, long-horizon interactions, where the effects of an update may unfold over many steps and depend on changing environment states.

Questions for authors
  1. Do the validation savings outweigh the total cost of collecting history, training the response model, and running BRACE?

  2. BRACE uses historical information to guide validation while leaving candidate generation unchanged. Could historical information also guide skill updates, so that skill evolution and validation improve together? I would like to see some discussion of this possibility beyond the current validation-only use of the response model.

3 comments
AuthorsOpenReview ↗

Follow-up to Reviewer vqHD

Thank you again for asking how BRACE’s reliability, quality, and cost change over a campaign. In response, we audited harmful commits between checkpoint refreshes and checked predictive interval coverage at the actual stopping points (W1). We also compared held-out success and cumulative validation cost at the same run indices (W2), and ran separate experiments with changing validation tasks and with skill evolution without resets (W3). We report timing that accounts for the cost of collecting history and fitting the response model (Q1), and discuss how paired evaluation results could inform future skill proposals and their validation (Q2).

If your schedule allows, we would be grateful if you could take another look at our response. If you find that the additional results and revisions address your concerns, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

Following up on our response to your review

Thank you again for the thought and care you put into your review. We appreciated how clearly you explained what you wanted to understand better about the evaluation. Your questions prompted us to examine our results more closely, and we are grateful that your feedback helped us explain what we found more clearly.

If you have a chance to look through our full response, we would be glad to hear whether it addresses your concerns or if there is anything you would like us to explain further.

AuthorsOpenReview ↗

Our sincere thanks for helping us improve the evaluation

As the rebuttal discussion draws to a close, we wanted to thank you once more for the thought and care you put into reviewing our work.

We especially appreciated how clearly you explained what you wanted to understand about BRACE’s behavior over time. Your questions prompted us to examine the evaluation across different stages and carry out additional analyses. That work helped us give a clearer account of reliability and cost.

We are grateful that your comments led us to look more closely at the evidence. We learned from doing so, and we wanted to express our appreciation before the discussion ends. Thank you again for contributing your time and expertise to improving the paper.

We wish you good health, happiness, and continued success in your research and work.

Reviewer_dAnjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Self-evolving agents that keep persistent skills or instructions have to decide, for every proposed edit, whether to commit it or keep the incumbent. Current systems mostly validate edits on a fixed task panel or a random minibatch. The motivating analysis shows why this is a poor fit: of 40 SkillOpt edits, 38 fix some tasks and break others, and the median edit changes the outcome on only 8 of the 40 validation tasks. A random panel therefore spends most of its budget on tasks the edit does not touch, and the improvements and regressions it does find can cancel out.

BRACE frames validation as a commit-or-preserve decision. Offline, it fits a low-rank ordered-probit model to historical candidate-incumbent paired outcomes, so that a new edit's description gives a prior over its effect on every task. Online, it chooses the next task by Decision Value of Information (DVI), the expected reduction in the Bayes risk of committing a harmful edit or rejecting a beneficial one, and stops when the posterior risk falls below a calibrated threshold. BRACE is plugged into SkillOpt and EvoSkill on SpreadsheetBench-Verified and DocVQA. With Qwen3.5-4B it needs 54.0-82.3% fewer fresh validation episodes than a DPP coreset; once startup, the host's retained evaluations and the checkpoint checks are charged, the saving is 24.9-40.1%, and mean terminal success is slightly higher. In replay on frozen candidates, DVI reaches the same mean regret as Random with half as many pairs, and as Value-variance selection with 5.0 instead of 6.5 pairs.

Critical strengths

The objective matches the problem. BRACE puts an explicit loss on committing a harmful edit and on rejecting a beneficial one, and chooses tests by how much they reduce that loss, which is the decision the host actually has to make. The evaluation is careful in two places. Section 5.3 and the ledger separate the fresh-episode saving from the amortized one and charge history acquisition and checkpoint checks to BRACE. And frozen-candidate replay removes the difference in edit histories between end-to-end runs; the reduction in pairs survives it (Table 9) and holds for all four executor models in Table 7.

Critical weaknesses

The abstract treats higher terminal success as a finding on the same footing as the cost reduction, and it quotes only the 54.0-82.3% fresh-episode saving, not the 24.9-40.1% amortized saving that reflects system cost; the conclusion does the same. With four campaigns the success differences are not resolved, so what the experiments support is cheaper validation at comparable decision regret, with success no worse than the controls. The cross-model results in Table 6, where BRACE is above Native in all 16 settings, point the same way but report no dispersion, so they do not settle the question. This is the main reason I score the paper as a weak rather than a clear accept. The abstract should lead with the amortized cost, the pair counts and the matched regret.

Questions for authors

Appendix B.2 defines paired campaign differences against Native, and in Table 3 BRACE beats DPP by 0 or 1 audit task per campaign. Could you report the paired difference against DPP as well, and if it covers zero, describe the result in the abstract as lower cost with success no worse than the controls?

How does the saving against DPP evolve over a campaign? Using the per-run averages in Table 5, BRACE's extra startup cost (1,520 vs 160 episodes) would be recovered after roughly 8 to 15 host runs, but early runs start from a small bank and may need more pairs than the average suggests. A plot of cumulative episodes against run index for BRACE and DPP would show where the break-even actually falls. I don't think another benchmark is needed.

3 comments
AuthorsOpenReview ↗

Follow-up to Reviewer dAnj

Thank you again for reading the results closely and helping us clarify the main finding. We revised the abstract and conclusion to emphasize the savings in amortized validation cost. In our response, we also summarize the paired terminal success differences versus DPP and point to the controlled replay, where DVI used fewer paired tests at comparable decision regret (W1 and Q1). To address your question about the extra startup cost, we report cumulative validation cost at several run indices in an additional matched EvoSkill–DocVQA experiment (Q2).

If your schedule allows, we would be grateful if you could take another look at our response. If you find that the additional results and revisions address your concerns, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

A brief follow-up on our response

Thank you again for reading our results so closely and for taking the time to share how you interpreted them. It was helpful to see the evidence from your perspective. Your comments made us think more carefully about what readers would take away from the experiments, and they helped us present the main findings with greater care.

If you have a moment to look at our full response, we would value your thoughts on whether it addresses your concerns and whether anything would benefit from further explanation.

AuthorsOpenReview ↗

Our thanks for your thoughtful assessment

As the rebuttal discussion comes to a close, we would like to express our sincere thanks once again for your close reading of our results and your thoughtful assessment of the evidence.

We appreciated both your recognition of the decision-oriented formulation and your suggestions about how to present the main findings. Your perspective helped us place greater emphasis on the cost results and examine how the savings developed over a campaign.

Your review encouraged us to think more carefully about what readers should take away from the experiments. We wanted to acknowledge how helpful that perspective was during the revision. Thank you for sharing your judgment and for taking the time to explain it so clearly.

With our warm thanks, we wish you good health, happiness, and continued success in your research.

Reviewer_pDSHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BRACE, a Bayesian framework for verifying candidate updates in self-evolving agents. BRACE learns shared candidate–task response structure from historical paired evaluations, updates its predictions as new evidence arrives, and selects validation tasks according to their expected reduction in deployment-decision loss. A calibrated posterior-risk threshold determines when verification stops. Experiments integrate BRACE into SkillOpt and EvoSkill on SpreadsheetBench-Verified and DocVQA, with additional evaluations across four model configurations. Relative to DPP coreset validation, BRACE reports 24.9–40.1% fewer amortized validation episodes, including startup and checkpoint checks but excluding configuration search, while achieving slightly higher mean terminal success.

Critical strengths
  1. The decision-oriented formulation is well motivated. Identifying tasks affected by an update is insufficient when improvements and regressions can offset each other. Evaluating tests by their consequences for the commit-or-preserve decision provides a coherent objective that directly reflects the purpose of update verification.

  2. The controlled experiments help separate the proposed mechanisms. Acquisition comparisons share the response model, posterior-update procedure, and terminal decision rule. The response-model ablations also use identical task prefixes. These controls provide useful evidence that both cross-task response transfer and decision-directed acquisition contribute to the observed improvements.

  3. The experimental protocol and cost accounting are comparatively transparent. The paper distinguishes logical pairs from fresh executions, accounts for cached outcomes, and includes history acquisition and checkpoint checks in its rollout ledger. It also separates historical fitting, calibration, and audit data, and explicitly distinguishes Monte Carlo confirmation from model-calibration guarantees.

Critical weaknesses
  1. The evidence for improved deployment quality is limited. In Table 1, BRACE exceeds DPP by only 0.25–0.5 percentage points. Table 3 shows that these differences correspond to one or two additional successes across the four campaign endpoints. The paper appropriately describes the four-campaign statistics as descriptive, but more independent campaigns and paired uncertainty estimates are needed to establish a reliable quality advantage. Furthermore, success is measured only at run 100, whereas cost is averaged over the entire campaign, leaving performance during earlier stages of history accumulation unclear.

  2. Risk calibration needs stronger validation beyond the reported setting. Appendix A.3 correctly states that numerical confirmation controls sampling error relative to the fitted posterior, rather than model misspecification. However, each checkpoint-check block contains only four candidate updates, and nominal 90% and 95% intervals achieve 87% and 93% coverage. These checks provide limited evidence about rare harmful updates or changes in the candidate distribution. The finite-pool target also uses a 40-task catalog with one matched execution per task, so low replay regret does not automatically establish reliable decisions under fresh execution draws or changing workloads.

  3. Rollout savings do not yet establish overall computational savings. The episode ledger includes substantial startup and checking costs, which is a strength. Nevertheless, verifier computation is excluded: online and hypothetical posterior refits use 200 optimization steps, and stopping confirmation can require batches of up to 2202^{20} predictive samples. Without wall-clock measurements, hardware usage, and the frequency of expensive refits and confirmation rounds, it is difficult to assess the practical efficiency benefit, particularly for relatively inexpensive executor models.

  4. Conditioning on single-sided cached outcomes is statistically underspecified. Appendix A.1 describes how a known incumbent or candidate outcome restricts the support of the paired effect. However, a model of Y=Rcandidate−RincumbentY=R_{\mathrm{candidate}}-R_{\mathrm{incumbent}} alone does not generally determine the conditional probabilities given one branch outcome: Y=0Y=0 combines both-success and both-failure events. Restricting the allowed outcomes therefore does not fully specify the required conditional distribution. The additional assumptions or joint outcome model should be stated explicitly, since this conditioning affects prediction, acquisition, and stopping.

  5. The end-to-end comparisons do not fully isolate the benefit of DVI. Panel baselines use observed mean differences, whereas BRACE combines historical prediction, posterior updating, adaptive acquisition, and risk-based stopping. The shared-posterior replay experiments address part of this concern, but they do not establish the contribution of each mechanism to the complete evolution trajectory. End-to-end Random and Value-variance variants using the same historical model and stopping rule, together with a sequential paired-testing baseline, would make the system-level comparison more informative.

Questions for authors
  1. Could you report paired confidence intervals against DPP and additional independent campaigns? How do deployment quality and cumulative validation cost evolve across runs, rather than only at the run-100 endpoint?

  2. What are the wall-clock and compute costs of response-model fitting, DVI scoring, branch refitting, and stopping confirmation? At what reuse horizon does BRACE become cheaper after including verifier computation?

  3. What exact conditional likelihood is used when only one branch outcome is cached? Does the implementation model the four joint success/failure outcomes, or introduce an additional assumption to divide the probability mass associated with Y=0Y=0?

  4. How do Random and Value-variance perform end to end when given the same historical predictor, caching rules, and stopping criterion as BRACE? How does BRACE compare with a sequential paired-testing validator?

  5. How stable are the full-catalog reference decisions across fresh execution draws? Could you evaluate rotating or previously unseen validation tasks, longer evolution without resetting host state, and asymmetric penalties for harmful commits?

5 comments
AuthorsOpenReview ↗

Follow-up to Reviewer pDSH

Thank you again for your careful review and detailed questions. In the first part of our response, we compare held-out success and cumulative validation cost at the same run indices (W1 and Q1), report a full-catalog audit of harmful commits (W2), and provide timing results that include verifier computation (W3 and Q2). In the second part, we clarify how BRACE uses a single cached outcome and point to the rule in Appendix A.1 (W4 and Q3). We also report end-to-end comparisons under a shared verifier and with a sequential paired validator (W5 and Q4), and examine fresh execution draws, skill evolution without resets, changes to the validation catalog, and asymmetric penalties (Q5).

If your schedule allows, we would be grateful if you could take another look at our responses. If you find that the additional results and revisions address your concerns, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

Follow-up on our author response

We are grateful for the care you took in working through the technical details of our paper. It was especially helpful that you set out your questions so clearly. They showed us where our reasoning needed a fuller explanation and helped guide the additional analyses. Thank you for giving us the chance to work through those parts more carefully.

If you have time to look over both parts of our response, we would be grateful to hear whether they answer your questions. We would be happy to explain any points that are still unclear.

Thank you for the additional analyses. Your responses addressed many of my concerns. I hope the authors to incorporate every relevant comment into their revision.

Reviewer_ifuvOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This work presents BRACE, a decision framework for incremental updates to self-improving language model agents. The framework learns a representation of historical evaluation results for incremental agent skill changes (offline) and leverages it to evaluate proposed changes by selecting evaluation samples more efficiently and weighing the update's risk/reward to the agent (online). The goal BRACE targets is to reach online update decisions faster and more reliably than current methods. The authors evaluate BRACE on two agent-skill-evolution harnesses across two datasets (one spreadsheet navigation, one multimodal document QA) against 4 baselines and find that BRACE slightly improves performance while significantly reducing the number of evaluations required to decide whether to adopt an agent update.

Critical strengths

The work tackles an important and timely problem: quantifying incremental improvements in self-iterating agents. The figures are visually appealing, and the tables are formatted well. While the performance of BRACE doesn't significantly outperform the baseline in terms of success rate, it does decrease the required number of evaluation items to reach an update/no update decision.

Critical weaknesses

I'm very concerned with the presentation of this work. The authors use many LLM-isms in the abstract and across the entire paper, making it hard to read. Instead of using simple language, the manuscript continues to use unnecessarily complex sentence structures. Similar concerns persist for the formalization of the problem and the experimental setup. The paper contains a substantial amount of theoretical derivations, but their presentation makes it difficult to understand why these components were chosen and how they contribute to the overall method. These issues seriously undermine the contributions this paper may have. I would suggest a full rewrite of the paper.

The paper introduces the topic of self-improving agents by mentioning agents that iterate on their instructions, memories, and skills. However, the key experimental results only evaluate the evolution of skills using the SkillOpt and EvoSkill harnesses. The contribution claim and statements across the paper should be adjusted accordingly.

Also, no code or artifacts are provided, making it difficult to evaluate whether the experimental setup/results align with the claims presented in the paper. The same applies to validating the correctness of the implementation of the proposed method.

Questions for authors
  1. Why were SpreadsheetBench-Verified and DocVQA chosen as the evaluation datasets? The former is a spreadsheet manipulation dataset, and the latter checks for multimodal document question answering, which seem quite mutually exclusive.
  2. Why was such a small model (Qwen3.5-4B) chosen, when you report partial results on Qwen3.5-27B in the appendix (a stronger model, performing better than the 4B version)?
  3. Why are experimental results on task selection presented in Section 3 before even introducing the BRACE method in the subsequent method?
4 comments
AuthorsOpenReview ↗

Follow-up to Reviewer ifuv

Thank you again for pointing out where the original manuscript was hard to follow. Your feedback helped us simplify the writing and clarify the scope of the skill-update experiments (W1 and W2). We added detailed pseudocode in Appendix A.4 of the revised manuscript and confirmed that we will release the complete BRACE source code upon acceptance (W3). In our response, we also explain why we chose the two benchmarks (Q1), why we used the smaller model for the main study and the larger model for targeted evaluations (Q2), and how the observations in Section 3 motivated the response model and test selection rule (Q3).

If your schedule allows, we would be grateful if you could take another look at the revised manuscript. If you find that these revisions and clarifications address your concerns, we would be grateful if you would consider raising your rating.

Thank you for your efforts in improving the paper. Please see my comments below in response to your rebuttal. I am willing to raise my scores, but still have concerns.

W1: I believe the Introduction has improved a lot and some necessary changes to the abstract have been made.

I understand that Section 3 has a purpose, however its presentation is still very confusing. Also, if this is supposed to serve as motivation for the BRACE method, why does 1(c) already use your method?

My concern around the presentation of the theoretical foundations of the method in Section 4 remains. I believe it would benefit from a much simpler presentation, with more detailed technical derivations in the appendix. My current understanding of the method is the following: learn from past skill update results -> predict its new change's effects -> select the best tests to decide on accept/reject -> update estimates on tests as results come in -> stop testing when risk of making an incorrect decision is sufficiently low -> commit/reject update. Is this correct? If so, this could be explained a lot clearer in the manuscript.

W2: I believe this clarification should also be present in the abstract. While I understand an approach like this is relevant for "instructions, memories, and skills" as denoted in the abstract, your work focusses on skills. This should be clear to a reader unfamiliar with the details of SkillOpt and EvoSkill.

Q1: Thank you for clarifying. While it is a limited set of applications, I understand.

Q2: I understand the computational constraints. However, I have concerns that the cost saving against DPP scales well from such a compact model to a bigger one.

Q3: Please see my comments regarding Section 3 above.

AuthorsOpenReview ↗

Follow-up on your suggestions for the revised manuscript

Thank you for taking the time to reread our manuscript and write back. We appreciate that you pointed out what had improved and where the explanation was still hard to follow.

W1 & Q3. Motivation and method presentation. Figure 1(c) examines how test selection affects the decision to adopt an update. We first observed that a skill update often helped some tasks and hurt others, and that results from similar past updates helped us identify the tasks that changed (Figures 1(a–b)). We then asked which tests would best inform the adoption decision. We compared multiple test-selection strategies at matched budgets, holding the historical predictor and final decision rule fixed and disabling early stopping. Decision-directed selection consistently yielded lower decision regret across the budgets tested (Figure 1(c)). This finding motivated us to incorporate the rule into the full BRACE validation procedure. Following your feedback, we revised the explanation of Figure 1(c) to make it easier to see why we ran the comparison and how its results informed the design of BRACE. Please see Section 3 of the latest revision.

Your summary accurately captures the BRACE validation procedure. Thank you for writing out the procedure step by step and suggesting how to present it more simply. Your feedback helped us decide what to explain first in Section 4. Following your suggestion, we now open Section 4 of the latest revision with a plain-language overview of the full validation procedure. Appendix A provides the technical details.

W2. Scope in the abstract. We especially appreciate that you considered how the abstract would read to someone unfamiliar with SkillOpt and EvoSkill. In response to your suggestion, we revised the abstract to make clear from the outset that the paper focuses on skill updates. Please see the abstract in the latest revision.

Q2. Test-selection efficiency on larger models. Thank you for asking how the savings carry over to larger models. In the controlled replay on both Qwen3.5-9B and Qwen3.5-27B, decision-directed selection used substantially fewer paired tests and achieved lower observed decision regret than the other strategies in the comparison (Table 7). These results show that BRACE consistently improves test-selection efficiency across the larger models evaluated.

Thank you again for continuing the discussion after reading our revision. Your explanation of how you understood the method and where you still had questions helped us focus this round of revisions. If you find that the latest revision and our clarifications address your remaining concerns, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

Thank you for rereading the revision and continuing the discussion

As the rebuttal discussion draws to a close, we would like to thank you once more for rereading our revision and continuing the conversation with further suggestions.

We especially appreciated that you told us which changes had helped and where the explanation still needed work. Your account of how you understood the procedure gave us a concrete way to improve its presentation. We took those suggestions seriously as we prepared the latest revision.

We know that returning to a revised paper and writing further feedback takes additional time and thought. We are grateful that you made that effort. The exchange helped us understand how the paper reads to someone encountering the method for the first time, and we learned a great deal from it.

Thank you again for your patience and help. We wish you good health, happiness, and continued success in your research and work.

Reviewer_st9XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how self-evolving agents decide to keep or reject a proposed update to their skills or instructions. Validating the updates often tends to be expensive and the authors propose BRACE. It learns from past updates how changes tend to affect different tasks and uses this to predict the effect of a new update. During validation, BRACE tests the tasks most likely to change the keep or reject decision, scored by a measure called Decision Value of Information and stops once the risk of a wrong decision is low enough. Across two agent systems and two benchmarks, BRACE reduces validation costs by 24.9–40.1% compared with DPP while achieving similar or slightly better performance. There are also additional experiments showing that learning from past updates and targeted task selection help reduce evaluation costs.

Critical strengths
  1. The analysis in Section 3 shows that most updates affect only a few tasks and that using information from previous updates helps select more useful validation tasks.

  2. The ablation experiments show that DVI performs better than alternative task-selection methods and that learning relationships between tasks improves prediction accuracy.

  3. The paper provides a detailed breakdown of evaluation costs, reports results across multiple runs, and discusses limitations in calibration.

Critical weaknesses

It is unclear to me how well BRACE would perform without agent resets. In the experiments, each run starts from the same initial skill, while BRACE retains its history. I feel this may make new updates similar to those already seen and in a continuously evolving agent, previous changes could affect how new updates behave. Would it be possible to report results without resets?

I have major concerns about the paper's presentation, as the writing is unnecessarily complex and repetitive, at times resembling AI-generated text.

  1. I feel the paper hard to follow. Even after several careful readings, it took considerable effort to understand the method and the evaluation protocol. The text introduces many experimental units and several cost units without a single place that explains how they relate.
  2. Also the paper often uses complicated technical language where simpler explanations might be sufficient. Some sections repeat the same ideas without adding much new information (for example, the method descriptions in the intro and Section 4, cost reduction discussions in Sec. 5.2 and 5.3). Several terms are also used before being properly explained (“activation”, “retained host evaluation” in Section 5.1, “terminal-run success,”). I also feel that the paper focuses more on describing the mathematical formulation than explaining why certain design choices were made.

Overall,

I am somewhat uncertain about my assessment, and I want to make this clear to the AC. I remain unsure whether the contribution is sufficiently novel or is mainly an engineering combination of existing methods.

The method proposed in the paper lacks comparisons with related work, for example, on active-testing and sequential methods. I believe these comparisons are important to understand paper's novelty and show what it contributes beyond existing approaches.

I do not see an obvious correctness issue, and the reported cost reductions are meaningful within the evaluated setting. However, I am not sure what broader research insight follows from the results beyond the useful motivating findings in Section 3.

Questions for authors

There should be some clarification of why SpreadsheetBench-Verified and DocVQA were chosen instead of EvoSkill’s original benchmarks, OfficeQA and SealQA, otherwise it reads abit of a benchmark selection bias.

3 comments
AuthorsOpenReview ↗

Follow-up to Reviewer st9X

Thank you again for reading our paper carefully and sharing your concerns. In response to your comments, we ran an experiment without skill resets (W1) and clarified the method and evaluation protocol in the revised manuscript (W2). We also explained how BRACE uses past updates to choose tests, added comparisons with active evaluation and sequential paired testing (W3), and explained why we chose the two benchmarks (Q1).

If your schedule allows, we would be grateful if you could take another look at our response. If you find that the additional results and revisions address your concerns, we would be grateful if you would consider raising your rating.

AuthorsOpenReview ↗

A brief follow-up on our response

Thank you again for reading our paper with such care. We especially appreciated that you took the time to point out where a reader might get stuck. Your comments were a real help as we worked through the revision. They helped us make the method easier to follow and strengthen the evaluation.

If you have a chance to look through our full response, we would be grateful to hear whether it addresses your concerns and whether there is anything we could explain more clearly.

AuthorsOpenReview ↗

Our sincere thanks as the discussion draws to a close

As the rebuttal discussion draws to a close, we wanted to write once more to express our sincere thanks for the time and care you put into reading our paper.

We especially appreciated that you explained where the presentation was difficult to follow and asked how the evaluation would change when skills were carried forward. We returned to your comments throughout the revision. They helped us see the paper from a reader’s perspective and gave us concrete directions for improving the writing and strengthening the evaluation.

We have learned from your feedback, and we wanted you to know how much we appreciated the thought behind it. Thank you again for contributing your time and expertise to this work.

We wish you good health, happiness, and continued success in your research.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional experiments and revised manuscript

Thank you for carefully reviewing our paper and sharing your feedback. We have posted our point-by-point responses and uploaded the revised manuscript.

Our responses report additional experiments on skill evolution without resets and on changes to the validation task pool, along with analyses of stopping reliability, held-out success, and validation cost. We also compared test-selection rules under a shared verifier and ran a separate comparison with a sequential paired validator. The revised manuscript clarifies the method and evaluation protocol and includes detailed pseudocode.

If your schedule allows, we would appreciate your taking a look at the new results and the revised manuscript. If you find that these updates address your concerns, we would be grateful if you would consider raising your rating. We would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.