acceptodds
Under review as a conference paper at ICLR 2027

HERA: Harness–Environment Co-Evolution for Reliable Agentic Abstention

Abstract

Large language model (LLM) agents are increasingly capable of acting in com- plex tool-use environments, yet they often fail to recognize when tasks are in- feasible and no valid solution exists. Recent work has formalized this reliabil- ity gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to lim- ited generalization to unseen failure modes. We introduce HERA, a framework for harness–environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which perfor- mance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while maintaining feasible task performance, out- performing all compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agentic Abstention, Harness–Environment Co-evolution, LLM Agents, Harness Optimization, Environment Generation, Abstention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4VCAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HERA studies when a tool-using agent should complete a task and when it should abstain. It constructs matched feasible and infeasible environments for the same request, then uses failures of the current agent to guide new environments and edits to the agent harness. The language model itself remains fixed. On HERA-Bench, which contains 60 held-out task pairs, the paper reports an increase in abstention success from 61.7% to 83.3% and paired success from 48.3% to 70.0% over the base harness. The final harness is also evaluated on 19 additional models without further optimization.

Critical strengths
  1. Keeping the task request fixed while changing the environment's feasibility is a useful design. The reference solver, alternative-path checks, and manual review described in Sections 3.1, B.3, and D also avoid equating a failed agent rollout with an infeasible task.

  2. The baseline choices address the main alternatives to adaptive environment generation: optimizing only the harness, adding tasks without adaptation, and generating tasks from the initial harness's failures. This makes the comparison more informative than a comparison with H0 alone.

  3. The same final harness improves both Abstain and Pair on all 19 additional models in Table 3. The average gains are 15.3 and 12.5 percentage points, respectively. This provides useful evidence that the learned changes transfer beyond the model used for development.

Critical weaknesses
  1. The 19-model experiment tests the transfer of one frozen harness, not the co-evolution procedure across models. The full optimization loop is run with GPT-5.6-Luna as the task-execution model (Section 4). Since the execution model's failures guide both task generation and harness updates, the results do not yet show whether the advantage over the optimization baselines persists with a different execution model.

  2. The domain-disjoint split is useful, but training and test tasks still come from the same construction pipeline. It is unclear which infeasibility mechanisms, rather than just domains, are unseen at test time. Appendix E.3 describes how the training tasks change across rounds; it does not answer this train-test overlap question. The evidence is therefore stronger for transfer to new domains than for handling new failure mechanisms.

  3. The contribution of diagnosis-guided generation is not separated from that of online hard-example selection. HERA uses current failures both to guide proposals and to decide which batches to retain (Section 3.2; Appendix B.4). The reported baselines do not include a control that keeps current-harness screening but removes diagnostic guidance from generation. Likewise, the low initial scores on new batches in Figure 2 are partly expected from the admission rule. The comparisons support the full system, but leave open how much diagnostic guidance adds over simpler online filtering.

Questions for authors
  1. Does the advantage over task-augmented evolution and task generation from H0 failures persist when another model family is used to run the full co-evolution loop? A comparison using one additional execution model, with comparable optimization budgets, would be useful here.

  2. Which infeasibility mechanisms are shared between training and HERA-Bench, and which, if any, are held out? Could you report results by mechanism, particularly for cases whose failure mechanism was absent from training?

  3. What happens if the environment and task generators do not receive the current failure diagnosis, but candidate pairs still undergo certification and current-harness screening at tau = 0.4? Keeping the harness optimization unchanged and comparing on HERA-Bench at matched task counts and generation budgets would distinguish diagnostic guidance from online filtering.

4 comments

Thank you for your positive assessment and insightful feedback! We are glad you found our paired-task design useful, our baseline comparisons informative, and our cross-model transfer results encouraging. We address your comments and questions below.

1. Harness Transfer and Cross-Model Co-evolution (W1 & Q1)

Thank you for this valuable suggestion! We agree that running the full co-evolution process with another model family would provide additional insights into its generalizability.

Our primary goal is to demonstrate the effectiveness of harness–environment co-evolution in addressing the reliability challenge of agentic abstention, while also investigating the transferability and cost efficiency of the resulting harness, which we demonstrate across the transferability experiments in Table 3 and API cost analysis in Figure 3.

We acknowledge that transferring a frozen harness is different from transferring the entire co-evolution process. However, repeating the full harness evolution and generalization experiments across additional model families would result in substantial additional computational and API costs, beyond the scope of the present study. We therefore prioritize demonstrating generalization through extensive cross-model transfer experiments and additional out-of-distribution evaluations (which we report on below), which provide positive evidence of transferability. We appreciate the reviewer’s suggestion and view further robustness testing and cross-model replication of the complete co-evolution process as a valuable direction for future work.

2. Infeasibility Mechanisms and Generalization (W2 & Q2)

Thank you for raising this question! In our pipeline, for each task and environment, we encourage the generation of different infeasibility types (referred to as environment mutations in our paper) through N independent proposers (L192–L194), with each modification tailored to the task’s requirements and environment. The training set and HERA-Bench (test set) use different task instances and environment implementations, providing different contexts for these infeasibility types. During our experiments, we observed that a single environment mutation can fall into multiple infeasibility categories (e.g., revoking access to a required record introduces both an access restriction and missing evidence), so it is difficult to assign each mutation to a single infeasibility type. We agree that a more detailed exploration of mutation types and their overlap would be valuable and plan to pursue this in future work.

Additionally, to better assess whether HERA’s improvements extend beyond new failure mechanisms, we evaluate the frozen H₀ and Final harnesses on two external agentic abstention benchmarks, AgentAbstain [1] and the WebShop tasks from Agentic Abstention [2], using GPT-5.6 Luna without further harness optimization. We will add these additional results to our revised manuscript.

AgentAbstain: results on four categories (%). Pair requires success on both the feasible and infeasible tasks.

CategoryHarnessActAbstainPair
Conflicting ConstraintsH₀87.0650.2040.59
Final87.4556.4743.92
Critical Tool FailureH₀68.25100.0068.25
Final74.21100.0074.21
High-Stakes ActionH₀80.6251.0444.58
Final83.7560.8351.04
Insufficient Tool CapabilityH₀88.7790.7079.47
Final88.7793.3382.11

WebShop from Agentic Abstention: We evaluate all 251 environment-based tasks (Missing Target) in the WebShop setting of Agentic Abstention, along with their corresponding original tasks. We exclude request-based tasks, which modify the user instruction, to focus on environment-based abstention.

HarnessAbstainPair
H₀0.800.00
Final10.761.20

3. Contribution of Diagnosis-Guided Generation (W3 & Q3)

Thank you for this insightful suggestion! Diagnosis-guided generation (as part of environment evolution in our paper) and online hard-example selection play complementary roles in our pipeline: diagnosis guides generation toward the current agent’s weaknesses, while selection retains sufficiently challenging candidates. When we removed diagnostic guidance from generation while retaining certification and current-harness screening under the same generation budget, the current harness successfully completed most candidate tasks. As a result, too few task pairs met the admission criterion to proceed to the next co-evolution round. This suggests that diagnostic guidance helps sustain co-evolution by making the generation of challenging tasks more efficient.

[1] AgentAbstain: Do LLM Agents Know When Not to Act?

[2] Agentic Abstention: Do Agents Know When to Stop Instead of Act?

Thank you again for your helpful feedback! We wanted to follow up and see whether our rebuttal has addressed the questions and concerns you raised. Please let us know if anything remains unclear or if there are any new questions we can help clarify. We would be glad to elaborate!

Thank you for the detailed response. The clarifications have addressed my main concerns, and I will maintain my already positive rating. I appreciate the authors’ efforts and encourage them to incorporate these clarifications into the revised manuscript.

Thank you for your positive feedback and continued support! We will incorporate these clarifications into the revised manuscript.

Reviewer_N6Z6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HERA, a harness–environment co-evolution framework for agentic abstention. HERA constructs matched feasible and infeasible tasks through controlled environment mutations, then iteratively uses agent failures to generate new challenges and optimize the agent harness. On the 60-pair HERA-BENCH, the evolved harness improves GPT-5.6-Luna’s abstention accuracy from 61.7% to 83.3%. The same harness is evaluated on 19 additional models, with broadly positive transfer results.

Critical strengths
  • Addresses an important reliability problem for tool-using agents.

  • The matched feasible/infeasible design helps measure correct abstention without ignoring over-abstention.

  • Harness–environment co-evolution is an interesting and potentially useful idea.

Critical weaknesses
  • HERA does not outperform all baselines on the primary Pair metric: its 70.0% is below the 73.3% achieved by task generation from H0 failures. This contradicts several textual claims.

  • HERA-BENCH contains only 60 pairs, apparently with one rollout per configuration, and no confidence intervals or significance tests are reported.

  • Training and evaluation tasks are produced by the same generation and mutation pipeline. The harness may exploit pipeline-specific cues such as explicit unavailable-field or recoverability indicators.

  • The paper claims domain-disjoint evaluation, but Appendix C.1 notes that task-level disjointness does not guarantee disjoint environment families or source domains.

Questions for authors
  • How do the authors reconcile the “outperforming all methods” claim with the stronger 73.3% Pair score of the H0-failure baseline?

  • Was each task executed only once? Can confidence intervals, repeated runs, or paired significance tests be provided?

  • What overlap exists between training and HERA-BENCH in environment families, source domains, tool schemas, mutation types, and generation models?

  • Was HERA-BENCH locked before harness development? Were benchmark revisions made with knowledge of the final harness’s behavior?

  • Is performance available on an independently authored abstention benchmark?

  • How are complete and partial abstentions scored? Is an LLM judge involved?

7 comments
AuthorsOpenReview ↗

Response Part 1 (W1, Q1)

Thank you for your thoughtful suggestions to better assess robustness and generalization! We appreciate your suggestions and respond to each below.

1. Clarifying the baseline comparison (W1 & Q1)

Thank you for your suggestions on improving the clarity and precision of our claims. HERA outperforms task generation from H₀ failures on all three metrics on the final cumulative training pool: Abstain improves from 76.0% to 86.0%, Act from 82.0% to 88.0%, and Pair from 68.0% to 78.0% (Table 2; Appendix E.2). This advantage extends to both Abstain (83.3% vs. 78.3%) and Act (76.7% vs. 75.0%) on HERA-Bench, our held-out evaluation set, although the baseline achieves a higher Pair score (73.3% vs. 70.0%) (Table 1). In the revised manuscript, we will replace “outperforming all compared methods” with a more precise statement: HERA achieves the highest Abstain and Act scores on HERA-Bench and leads on all three metrics on the final cumulative training pool.

AuthorsOpenReview ↗

Response Part 2 (W2, Q2)

2. Statistical robustness (W2 & Q2)

Thank you for your suggestions on statistical robustness. We additionally evaluated the fixed Base and Final harnesses over three rounds across all 20 models on the 60 task pairs (14,400 episode evaluations). The tables below report mean ± sample SD.

Abstain:

ModelBase (%)Final (%)Δ Final − Base (pp)
GPT-5.6-Luna59.44 ± 2.5585.56 ± 3.47+26.11 ± 5.85
GPT-5.6-Terra80.56 ± 0.9686.67 ± 2.89+6.11 ± 3.85
GPT-5.6-Sol78.89 ± 0.9691.11 ± 0.96+12.22 ± 0.96
GPT-6-Astra83.33 ± 0.0091.67 ± 1.67+8.33 ± 1.67
Claude Opus 571.11 ± 0.9680.56 ± 0.96+9.44 ± 1.92
Claude Sonnet 552.78 ± 2.5561.67 ± 1.67+8.89 ± 1.92
Gemini 3.1 Pro Preview61.67 ± 2.8988.33 ± 1.67+26.67 ± 4.41
Gemini 3.8 Flash62.78 ± 5.0982.78 ± 1.92+20.00 ± 6.67
Gemini 3.5 Flash-Lite72.22 ± 4.1980.00 ± 5.00+7.78 ± 9.18
Grok 4.670.00 ± 2.8983.89 ± 2.55+13.89 ± 3.47
DeepSeek-V4.1-Flash63.89 ± 3.8581.67 ± 4.41+17.78 ± 4.81
GLM-5.237.78 ± 3.4755.00 ± 3.33+17.22 ± 6.31
GLM-5.353.33 ± 1.6765.00 ± 4.41+11.67 ± 4.41
GLM-5.3-Flash50.56 ± 5.8562.78 ± 3.85+12.22 ± 9.48
Kimi K363.33 ± 4.4180.56 ± 6.94+17.22 ± 10.05
Qwen3.8-2.4T-A95B50.56 ± 1.9274.44 ± 4.19+23.89 ± 5.85
Qwen3.8 Flash63.33 ± 3.3372.78 ± 2.55+9.44 ± 5.85
MiniMax-M347.78 ± 2.5558.33 ± 6.01+10.56 ± 7.70
Mistral Medium 3.517.78 ± 3.4746.67 ± 3.33+28.89 ± 4.19
Mistral Large 38.89 ± 5.3620.00 ± 2.89+11.11 ± 4.19

Act:

ModelBase (%)Final (%)Δ Final − Base (pp)
GPT-5.6-Luna66.67 ± 2.8973.33 ± 1.67+6.67 ± 1.67
GPT-5.6-Terra82.78 ± 1.9277.22 ± 1.92-5.56 ± 3.85
GPT-5.6-Sol78.89 ± 2.5580.56 ± 0.96+1.67 ± 1.67
GPT-6-Astra81.11 ± 0.9681.67 ± 0.00+0.56 ± 0.96
Claude Opus 570.00 ± 1.6777.78 ± 3.47+7.78 ± 2.55
Claude Sonnet 570.00 ± 6.0171.67 ± 4.41+1.67 ± 9.28
Gemini 3.1 Pro Preview71.11 ± 3.8575.00 ± 5.77+3.89 ± 5.09
Gemini 3.8 Flash64.44 ± 3.8578.33 ± 0.00+13.89 ± 3.85
Gemini 3.5 Flash-Lite66.11 ± 0.9669.44 ± 1.92+3.33 ± 2.89
Grok 4.682.22 ± 0.9681.11 ± 0.96-1.11 ± 0.96
DeepSeek-V4.1-Flash82.78 ± 2.5582.22 ± 2.55-0.56 ± 0.96
GLM-5.267.22 ± 9.1875.00 ± 3.33+7.78 ± 10.05
GLM-5.375.56 ± 5.3676.11 ± 4.19+0.56 ± 6.94
GLM-5.3-Flash62.22 ± 2.5570.56 ± 6.31+8.33 ± 8.66
Kimi K378.33 ± 1.6776.11 ± 1.92-2.22 ± 0.96
Qwen3.8-2.4T-A95B74.44 ± 1.9272.22 ± 0.96-2.22 ± 0.96
Qwen3.8 Flash75.00 ± 1.6780.56 ± 0.96+5.56 ± 1.92
MiniMax-M373.33 ± 2.8976.67 ± 2.89+3.33 ± 5.00
Mistral Medium 3.557.22 ± 2.5562.22 ± 0.96+5.00 ± 1.67
Mistral Large 320.56 ± 0.9628.33 ± 1.67+7.78 ± 2.55

Pair:

ModelBase (%)Final (%)Δ Final − Base (pp)
GPT-5.6-Luna42.78 ± 5.3665.56 ± 0.96+22.78 ± 4.81
GPT-5.6-Terra69.44 ± 0.9670.56 ± 2.55+1.11 ± 3.47
GPT-5.6-Sol66.11 ± 0.9678.33 ± 0.00+12.22 ± 0.96
GPT-6-Astra72.78 ± 0.9679.44 ± 0.96+6.67 ± 1.67
Claude Opus 555.00 ± 3.3366.11 ± 3.47+11.11 ± 2.55
Claude Sonnet 538.33 ± 4.4143.89 ± 2.55+5.56 ± 5.09
Gemini 3.1 Pro Preview40.56 ± 0.9668.89 ± 5.36+28.33 ± 5.77
Gemini 3.8 Flash43.33 ± 1.6765.56 ± 0.96+22.22 ± 2.55
Gemini 3.5 Flash-Lite48.33 ± 1.6755.00 ± 2.89+6.67 ± 3.33
Grok 4.659.44 ± 2.5571.11 ± 1.92+11.67 ± 2.89
DeepSeek-V4.1-Flash57.78 ± 5.0971.11 ± 5.36+13.33 ± 2.89
GLM-5.226.11 ± 1.9243.33 ± 4.41+17.22 ± 5.36
GLM-5.343.89 ± 0.9650.56 ± 5.36+6.67 ± 4.41
GLM-5.3-Flash30.56 ± 1.9246.67 ± 5.77+16.11 ± 7.70
Kimi K352.22 ± 2.5564.44 ± 3.47+12.22 ± 4.19
Qwen3.8-2.4T-A95B41.67 ± 1.6758.33 ± 4.41+16.67 ± 6.01
Qwen3.8 Flash49.44 ± 4.8161.67 ± 2.89+12.22 ± 6.74
MiniMax-M335.56 ± 1.9247.78 ± 3.85+12.22 ± 1.92
Mistral Medium 3.512.22 ± 3.8532.78 ± 2.55+20.56 ± 6.31
Mistral Large 33.89 ± 0.967.22 ± 2.55+3.33 ± 3.33

Confidence intervals and paired significance tests. Across the 19 transfer models, Final's macro-average gains are 14.39 pp in Abstain (95% CI: [11.02, 17.89]), 3.13 pp in Act ([1.11, 5.12]), and 12.43 pp in Pair ([8.98, 15.96]). CIs use 100,000 paired task-cluster bootstrap resamples, retaining all models and all three rounds within each task pair. Two-sided paired task-cluster permutation tests yield Holm-adjusted p < 0.01 for all three macro-average gains, correcting across the three metrics.

AuthorsOpenReview ↗

Response Part 3 (W3, Q3–Q6)

3. Overlap Between Training and HERA-Bench (Q3)

Thank you for this insightful question! We clarify the overlap below and will include these details in the revised manuscript.

  • Environment families and source domain: We explicitly use different source domains for the training set and HERA-Bench, ensuring that the resulting environments are also distinct.

  • Tool schema: Only 1 of HERA-Bench’s 240 distinct tool interfaces shares both its tool name and input schema with training, after removing environment prefixes and ignoring descriptions, field ordering, and default-value annotations.

  • Mutation type: In our pipeline, we encourage mutation diversity in two ways: (1) N independent proposers explore diverse candidate modifications for each task–environment pair (L192-L194); and (2) each modification is tailored to the task’s requirements and environment. Training tasks and HERA-Bench (test set) use different task instances and environment implementations, providing different contexts for these modifications. We agree that a more detailed exploration of mutation types and their overlap would be valuable and plan to pursue this in future work.

  • Generation models: We use different models to construct the training set and HERA-Bench. Specifically, we use GPT-5.6 Luna for training set construction, GPT-5.6 Sol for HERA-Bench construction, and GPT-6 Astra for independent solvability checks.

4. Benchmark Finalization and Revisions (Q4)

HERA-Bench was finalized and locked before harness development began. No revisions to HERA-Bench were made based on knowledge of the final harness's behavior. Additionally, to assess generalization beyond HERA-Bench, we evaluated the final harness on two external benchmarks: Agentic Abstention and AgentAbstain, as detailed in our responses to W3 and Q5 below.

5. Evaluation on External Benchmarks (W3 & Q5)

We are happy to share additional evaluation results on external benchmarks! To assess whether HERA’s improvements extend beyond tasks produced by our generation pipeline, we evaluated the frozen H₀ and final harnesses on two external benchmarks: AgentAbstain [1] and the WebShop tasks from Agentic Abstention [2]. All evaluations used GPT-5.6 Luna without further harness optimization.

AgentAbstain: results on four categories (%). Pair requires success on both the feasible and infeasible tasks.

CategoryHarnessActAbstainPair
Conflicting ConstraintsH₀87.0650.2040.59
Final87.4556.4743.92
Critical Tool FailureH₀68.25100.0068.25
Final74.21100.0074.21
High-Stakes ActionH₀80.6251.0444.58
Final83.7560.8351.04
Insufficient Tool CapabilityH₀88.7790.7079.47
Final88.7793.3382.11

WebShop from Agentic Abstention: We evaluate all 251 environment-based tasks (Missing Target) in the WebShop setting of Agentic Abstention, along with their corresponding original tasks. We exclude request-based tasks, which modify the user instruction, to focus on environment-based abstention.

HarnessAbstainPair
H₀0.800.00
Final10.761.20

These results suggest some transfer beyond HERA’s generation pipeline, with gains on WebShop and in four AgentAbstain categories.

6. Abstention Scoring and Human Validation (Q6)

As in AgentAbstain, partial abstention can count as success in HERA-Bench’s binary pass/fail evaluation. An agent may report supported findings while explaining which part of the request it cannot complete and why. To pass, it must ground this explanation in observed evidence, avoid unsupported claims or disallowed actions, and meet the task’s reporting requirements. For example, in the weather task (Figures 7–8, Appendix), it may report the available forecast information while withholding a postponement decision because the required precipitation probability is unavailable; this would count as successful in our evaluation. Each task’s evaluator uses deterministic checks of the final answer, tool observations, and changes to the environment. No LLM judge is used for these scores.

To assess the reliability of these deterministic checks, we also conducted a human evaluation using 100 trajectories. These covered all 60 task pairs and 20 LLMs, with five trajectories per model. An additional 10 trajectories were used for pilot annotation and excluded from the agreement estimate. Human judgments agreed with the automatic evaluator on 92 of the 100 evaluation trajectories (92.0%). The annotation records (https://anonymous.4open.science/r/hera-rebuttal-supplement-EC13), including the tasks, human and automatic decisions, rationales, and supporting evidence, are available in Excel and JSON formats.

Reviewer_GkBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HERA, a framework that co-evolves agent harnesses and environments to enable reliable agentic abstention, including an automated pipeline for generating paired tasks, and a co-evolution method that improves the harness while generating more challenging environments and tasks. Experiments shows that the evolved harnesses are effective and transferable across models.

Critical strengths

The paper addresses the timely problem of reliable agentic abstention through a training-free harness-environment co-evolution pipeline. Experiments across 19 LLMs show its effectiveness, transferability, and cost efficiency.

Critical weaknesses
  1. HERA-Bench is limited to 60 unique task pairs, and the quality of the generated environments and tasks remains unclear.
  2. The main results are based on a single run, with no assessment of statistical robustness. Given the small size of HERA-Bench, this limits the strength of the results and conclusions.
Questions for authors
  1. The quality of the environment and task generation pipeline remains unclear. Could the authors provide human validation results for the generated environments and tasks to indicate their quality?
  2. What's the principle to define and generate infeasible tasks in the paired-task design?
  3. Does"infeasible candidatse" in L189-L190 refer to invalid or low-quality tasks? Since "infeasible" is also used in the paired-task design, using the same term for both concepts might be ambiguous.
  4. What's the principle to define and evaluate a successful abstention behavior? And what's the human agreement rate for that evaluation?
  5. As the training and held-out evaluation sets are generated by the same pipeline, though from different domains, they may share similar patterns that the evolution process could exploit. Have the authors considered evaluating HERA on out-of-distribution agentic abstention benchmarks, such as the ones in Agentic Abstention or AgentAbstain, to further assess its robustness?

If the authors address the weaknesses and questions above, I would be happy to consider raising my score.

5 comments
AuthorsOpenReview ↗

Response Part 1 (W1, Q1–Q3)

Thank you so much for reviewing our work in detail, and for sharing your helpful and insightful feedback! We’re glad you found HERA effective, transferable, and cost-efficient. We respond in turn to each of the points you raise.

1. HERA-Bench Curation and Quality Validation (W1 & Q1)

We agree that validating the quality of HERA-Bench is important. We provide more details on Hera-Bench curation below and will add this to the camera-ready version of the paper.

The Hera-Bench curation process involved three rounds of review, discussion, and revision. As part of this process, three of the authors independently reviewed a sample of 21 feasible-infeasible task pairs from HERA-Bench and their corresponding environments, then discussed disagreements and refined the validation criteria. For each pair, we checked that (1) the feasible task was actually achievable, (2) the infeasible task had a valid and reasonable reason for the agent to abstain, and (3) both the tasks and their environments were realistic and well-designed.

During the first round, we identified several issues: some environments included unnecessary tools, some task descriptions prescribed specific steps rather than stating the user's goal, and some scenarios were overly contrived. These findings led to targeted improvements in the task generation pipeline across subsequent rounds. We will describe these quality checks more clearly in the main text and would be happy to share the supporting review materials, such as slides. We will formally release HERA-Bench to the public after the rebuttal period.

2. Definition and Construction of Infeasible Tasks (Q2)

We consider a task infeasible when the user’s request cannot be fully satisfied using the information and tools available in the environment while respecting the task’s constraints (Appendix B.3). We construct each pair from a verified solvable task by changing a critical environmental condition while keeping the request and public tool interface unchanged (Sec. 3.1; Appendix B.3). We validate the pair through reference execution and check for alternative completion paths; if a valid alternative is found, we throw out the infeasible task and start over (Appendices B.2–B.3). Figures 7 and 8 in Appendix F.1 provide examples of an infeasible task modification: making the precipitation probability unavailable prevents a definitive decision on whether to postpone outdoor work.

3. Clarifying the Term “Infeasible Candidates” (Q3)

Thank you for pointing this out to us! In L189–L190, we meant candidates that did not pass validation during initial task generation. Failing these checks does not, by itself, establish infeasibility. We will revise the sentence in the updated manuscript to clarify that we retain only candidates with verified executable solutions, and reserve “infeasible” for paired instances where the request cannot be fully satisfied given the available information, tools, and task constraints.

AuthorsOpenReview ↗

Response Part 2 (W2, Q4–Q5)

4. Abstention Criteria and Human Agreement (Q4)

How we evaluate successful abstention behavior. We count abstention as successful when the agent identifies a blocker based on observed evidence, avoids unsupported claims or disallowed actions, and meets the task’s reporting requirements. This goes beyond a simple refusal.

For example, in the weather task (Figures 7–8), the forecast shows 93°F and winds of 0–10 mph, but the precipitation probability is unavailable. Successful abstention means reporting the closest station and available forecast information, identifying the missing probability, and withholding the postponement decision because the 40% precipitation threshold cannot be checked. Simply saying “I cannot complete this task” does not pass: it neither identifies the observed blocker nor reports the supported findings required by the task.

Each task has a deterministic evaluator that checks the final response, recorded tool observations, and resulting environment state. The Abstain score is the percentage of infeasible instances that pass these checks (Appendix C.2).

Human agreement rate. Thank you for raising this question! We agree that human annotation is important and will include the following details in the revised version of the paper.

To validate the automatic abstention evaluator, we randomly sampled 110 trajectories from HERA-Bench, using 10 for pilot annotation and the remaining 100 for the main human evaluation. The 100 evaluation trajectories covered all 60 task pairs and 20 LLMs, with five trajectories per model. For each trajectory, we examined the task requirements, tool observations, and final response to assess whether the agent identified an evidence-backed blocker, avoided unsupported claims or disallowed actions, and satisfied the reporting requirements. Each human annotation included a binary decision, a written rationale, and references to supporting evidence. Human judgments agreed with the automatic evaluator on 92 of the 100 evaluation trajectories (92.0%); the 10 pilot trajectories were excluded from this estimate. In all eight disagreements, human review judged the abstention successful, whereas the automatic evaluator did not. This indicates that the automatic evaluator was more conservative in this sample.

We have uploaded the annotation records to the anonymous repository (https://anonymous.4open.science/r/hera-rebuttal-supplement-EC13). Each record includes the corresponding task, human review decision, automatic decision, rationale, and evidence references. We provide both Excel and JSON versions, with the Excel version formatted for easier reading.

5. Evaluation on AgentAbstain and Agentic Abstention (W2 & Q5)

We are happy to share additional evaluation results on Agentic Abstention [1] and AgentAbstain [2] ! We additionally evaluated the frozen H₀ and Final harnesses on AgentAbstain and the WebShop tasks from Agentic Abstention, using GPT-5.6 Luna without further harness optimization. We will add these additional results in our revised manuscript.

AgentAbstain: results on four categories (%). Pair requires success on both the feasible and infeasible tasks.

CategoryHarnessActAbstainPair
Conflicting ConstraintsH₀87.0650.2040.59
Final87.4556.4743.92
Critical Tool FailureH₀68.25100.0068.25
Final74.21100.0074.21
High-Stakes ActionH₀80.6251.0444.58
Final83.7560.8351.04
Insufficient Tool CapabilityH₀88.7790.7079.47
Final88.7793.3382.11

WebShop from Agentic Abstention: We evaluate all 251 environment-based tasks (Missing Target) in the WebShop setting of Agentic Abstention, along with their corresponding original tasks. We exclude request-based tasks, which modify the user instruction, to focus on environment-based abstention.

HarnessAbstainPair
H₀0.800.00
Final10.761.20

Once again, thank you for all your detailed feedback and we look forward to answering any further questions you may have during the discussion period!

[1] Agentic Abstention: Do Agents Know When to Stop Instead of Act?

[2] AgentAbstain: Do LLM Agents Know When Not to Act?

Thank you again for taking the time to consider our responses! We hope that our rebuttal has addressed your main concerns. If you have any remaining questions or would like further clarification on any aspect of the manuscript, please feel free to let us know. We would be glad to elaborate!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.