- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HERA studies when a tool-using agent should complete a task and when it should abstain. It constructs matched feasible and infeasible environments for the same request, then uses failures of the current agent to guide new environments and edits to the agent harness. The language model itself remains fixed. On HERA-Bench, which contains 60 held-out task pairs, the paper reports an increase in abstention success from 61.7% to 83.3% and paired success from 48.3% to 70.0% over the base harness. The final harness is also evaluated on 19 additional models without further optimization.
Critical strengths
-
Keeping the task request fixed while changing the environment's feasibility is a useful design. The reference solver, alternative-path checks, and manual review described in Sections 3.1, B.3, and D also avoid equating a failed agent rollout with an infeasible task.
-
The baseline choices address the main alternatives to adaptive environment generation: optimizing only the harness, adding tasks without adaptation, and generating tasks from the initial harness's failures. This makes the comparison more informative than a comparison with H0 alone.
-
The same final harness improves both Abstain and Pair on all 19 additional models in Table 3. The average gains are 15.3 and 12.5 percentage points, respectively. This provides useful evidence that the learned changes transfer beyond the model used for development.
Critical weaknesses
-
The 19-model experiment tests the transfer of one frozen harness, not the co-evolution procedure across models. The full optimization loop is run with GPT-5.6-Luna as the task-execution model (Section 4). Since the execution model's failures guide both task generation and harness updates, the results do not yet show whether the advantage over the optimization baselines persists with a different execution model.
-
The domain-disjoint split is useful, but training and test tasks still come from the same construction pipeline. It is unclear which infeasibility mechanisms, rather than just domains, are unseen at test time. Appendix E.3 describes how the training tasks change across rounds; it does not answer this train-test overlap question. The evidence is therefore stronger for transfer to new domains than for handling new failure mechanisms.
-
The contribution of diagnosis-guided generation is not separated from that of online hard-example selection. HERA uses current failures both to guide proposals and to decide which batches to retain (Section 3.2; Appendix B.4). The reported baselines do not include a control that keeps current-harness screening but removes diagnostic guidance from generation. Likewise, the low initial scores on new batches in Figure 2 are partly expected from the admission rule. The comparisons support the full system, but leave open how much diagnostic guidance adds over simpler online filtering.
Questions for authors
-
Does the advantage over task-augmented evolution and task generation from H0 failures persist when another model family is used to run the full co-evolution loop? A comparison using one additional execution model, with comparable optimization budgets, would be useful here.
-
Which infeasibility mechanisms are shared between training and HERA-Bench, and which, if any, are held out? Could you report results by mechanism, particularly for cases whose failure mechanism was absent from training?
-
What happens if the environment and task generators do not receive the current failure diagnosis, but candidate pairs still undergo certification and current-harness screening at tau = 0.4? Keeping the harness optimization unchanged and comparing on HERA-Bench at matched task counts and generation budgets would distinguish diagnostic guidance from online filtering.
4 comments
Thank you again for your helpful feedback! We wanted to follow up and see whether our rebuttal has addressed the questions and concerns you raised. Please let us know if anything remains unclear or if there are any new questions we can help clarify. We would be glad to elaborate!
Thank you for the detailed response. The clarifications have addressed my main concerns, and I will maintain my already positive rating. I appreciate the authors’ efforts and encourage them to incorporate these clarifications into the revised manuscript.
Thank you for your positive feedback and continued support! We will incorporate these clarifications into the revised manuscript.
Thank you for your positive assessment and insightful feedback! We are glad you found our paired-task design useful, our baseline comparisons informative, and our cross-model transfer results encouraging. We address your comments and questions below.
1. Harness Transfer and Cross-Model Co-evolution (W1 & Q1)
Thank you for this valuable suggestion! We agree that running the full co-evolution process with another model family would provide additional insights into its generalizability.
Our primary goal is to demonstrate the effectiveness of harness–environment co-evolution in addressing the reliability challenge of agentic abstention, while also investigating the transferability and cost efficiency of the resulting harness, which we demonstrate across the transferability experiments in Table 3 and API cost analysis in Figure 3.
We acknowledge that transferring a frozen harness is different from transferring the entire co-evolution process. However, repeating the full harness evolution and generalization experiments across additional model families would result in substantial additional computational and API costs, beyond the scope of the present study. We therefore prioritize demonstrating generalization through extensive cross-model transfer experiments and additional out-of-distribution evaluations (which we report on below), which provide positive evidence of transferability. We appreciate the reviewer’s suggestion and view further robustness testing and cross-model replication of the complete co-evolution process as a valuable direction for future work.
2. Infeasibility Mechanisms and Generalization (W2 & Q2)
Thank you for raising this question! In our pipeline, for each task and environment, we encourage the generation of different infeasibility types (referred to as environment mutations in our paper) through N independent proposers (L192–L194), with each modification tailored to the task’s requirements and environment. The training set and HERA-Bench (test set) use different task instances and environment implementations, providing different contexts for these infeasibility types. During our experiments, we observed that a single environment mutation can fall into multiple infeasibility categories (e.g., revoking access to a required record introduces both an access restriction and missing evidence), so it is difficult to assign each mutation to a single infeasibility type. We agree that a more detailed exploration of mutation types and their overlap would be valuable and plan to pursue this in future work.
Additionally, to better assess whether HERA’s improvements extend beyond new failure mechanisms, we evaluate the frozen H₀ and Final harnesses on two external agentic abstention benchmarks, AgentAbstain [1] and the WebShop tasks from Agentic Abstention [2], using GPT-5.6 Luna without further harness optimization. We will add these additional results to our revised manuscript.
AgentAbstain: results on four categories (%). Pair requires success on both the feasible and infeasible tasks.
WebShop from Agentic Abstention: We evaluate all 251 environment-based tasks (Missing Target) in the WebShop setting of Agentic Abstention, along with their corresponding original tasks. We exclude request-based tasks, which modify the user instruction, to focus on environment-based abstention.
3. Contribution of Diagnosis-Guided Generation (W3 & Q3)
Thank you for this insightful suggestion! Diagnosis-guided generation (as part of environment evolution in our paper) and online hard-example selection play complementary roles in our pipeline: diagnosis guides generation toward the current agent’s weaknesses, while selection retains sufficiently challenging candidates. When we removed diagnostic guidance from generation while retaining certification and current-harness screening under the same generation budget, the current harness successfully completed most candidate tasks. As a result, too few task pairs met the admission criterion to proceed to the next co-evolution round. This suggests that diagnostic guidance helps sustain co-evolution by making the generation of challenging tasks more efficient.
[1] AgentAbstain: Do LLM Agents Know When Not to Act?
[2] Agentic Abstention: Do Agents Know When to Stop Instead of Act?