acceptodds
Under review as a conference paper at ICLR 2027

From Failures to Harness: Admission-Gated Runtime Patch Selection for Hard Harness Defects in LLM Agents

Abstract

Large language model (LLM) agents rely on a runtime harness for tool calls, execution budgets, and recovery. Evaluating repairs to this layer requires distinguishing task success from recovery under an exercised defect. We introduce an injected-defect setting and an evaluation protocol that measures repair under recorded trigger exposure, same-family transfer, and clean-task regression. TARPS (Trigger-Aware Replay for Patch Selection) implements candidate search and admission using replay and reference-repair screening. Across seven frozen domain–defect cohorts on τ²-bench, reward-only selection raises mean repair from 10% for immediate first-candidate selection to 57%; adding clean-task checks reaches 76%, matching full TARPS. TARPS attains 76% transfer versus 71% for reward plus clean checks; the incremental effect of its remaining checks is unresolved at this sample size. Separate admission analyses show that reference-mechanism inclusion, refusal, and transfer measure distinct properties, and coverage changes which failures enter evaluation. Together, these findings identify execution checking as the main source of the observed gain and clarify what admission and trigger-aware evaluation measure.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
tool-augmented LLM agents, runtime harnesses, program repair, patch synthesis, admission control, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2euuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies repair of defects in the runtime harness of tool-using LLM agents. It argues that reward recovery alone cannot establish that a harness defect was repaired, because an episode may never exercise the defect, and proposes an evaluation protocol based on injected defects with recorded trigger events: repair and transfer are scored only over episodes in which the defect is “covered”. TARPS is a patch-selection procedure that enumerates candidate operators from a small DSL, screens replay instances using the known reference repair, and admits a candidate only if it passes covered replay on diagnosis instances and a majority vote on a clean probe. On seven τ²-bench domain–defect cells, TARPS reaches 71%/76% repair/transfer versus 10%/14% for committing the first candidate.

Critical strengths
  • The paper targets a real gap, i.e., task success does not mean the defect was repaired, so evaluation needs to check whether the defect was actually exercised.
Critical weaknesses
  • The scale of the experiments is too small to support the claims. The main results come from only one agent on one benchmark with 42 tasks. I do not consider the results reliable or generalizable at this sample size. Even based on current results, the paper’s own ablation shows that the proposed components do not work well. In Table 1, full TARPS gets 76.2/76.2 (repair/transfer), while “reward + clean check” without the two TARPS-specific checks gets 76.2/71.4. The 4.8-point transfer difference is only one task difference. So the main ideas of the method, the coverage-aware admission and the reference-repair screening, do not show any measurable benefits. The gain over NoGate comes from simply running the candidates and checking reward.
  • The admission problem is patch overfitting, which the Automated Program Repair community has studied for a decade. The clean probe, the repeated replay, and the coverage check all have direct counterparts there. The paper cites and compares none of this. Moreover, the coverage condition and the reference-repair screen of the gate require the injected defect spec and the reference repair, which do not exist for a real harness bug.
Questions for authors
  • What do you claim the coverage-aware admission and the reference-repair screening contribute?
  • Can you report results on a larger and fresh task pool?
Reviewer_LTgROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies repair of defects in an LLM agent’s harness. The authors inject four defect families, propose an evaluation protocol that credits a repair only on episodes where the defect’s trigger events actually fired, and present TARPS, which enumerates patch operators from a task’s required actions and admits a single operator or pair only if it passes covered replay on reference-screened diagnosis instances plus a clean-task regression check.

Critical strengths
  1. Important problem and useful evaluation distinction. The paper correctly identifies that successful task execution does not necessarily imply that an underlying harness defect has been repaired. Explicitly tracking defect exposure is particularly useful for silent and compound failures.
  2. Useful ablations are present. These experiments clarify how much of the observed improvement comes from ordinary execution feedback rather than the complete TARPS procedure.
Critical weaknesses

W1. My primary concern is that TARPS generates patch candidates from the benchmark's structured required_actions, rather than identifying potentially faulty harness components from the observed failure trajectory. For example, suppose an agent has access to 100 tools, but successfully solving a particular task requires get_reservation_details. If the benchmark explicitly provides this required action, TARPS can generate targeted dispatch-restoration and result-validation patches for that tool without having to infer its relevance from the failed execution.

Although this does not directly reveal the correct patch, it substantially simplifies the localization problem by supplying information about the intended solution structure. In realistic agent deployments, the correct sequence of required tool actions is generally not known in advance. Determining which tools should have been invoked, and whether an observed failure originates from the LLM, the harness, or an external tool, is itself an important part of diagnosis.

W2. The full TARPS procedure also assumes access to the injected defect specification and a reference repair during admission. These are used to establish defect exposure and select rescuable replay instances. This is reasonable for constructing a controlled evaluation protocol, but it limits the applicability of the proposed method to real-world harness failures, where neither the defect mechanism nor a valid reference repair is generally known. The authors investigate oracle-free admission variants, which is valuable. However, the initial variant performs substantially worse, while the revised screening approach remains exploratory and requires considerably more executions.

W3. The paper would be stronger if it could establish a clearer benefit of the additional checks under matched experimental conditions. Adding ordinary regression checks to reward-based selection matches the repair performance of full TARPS. Its additional transfer improvement is small relative to the reported uncertainty. This raises questions about the contribution of the proposed admission mechanism: Which specific components are necessary for effective repair selection, beyond standard execution testing and regression validation?

Questions for authors

See weaknesses above

Reviewer_YDneOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to repair defects in an LLM agent's runtime harness (tool dispatch, result handling, step budgets), with a focus on silent or compound defects where task reward may credit an incomplete fix. The authors inject four defect families into the τ²-bench airline and retail domains, and score repair only on episodes where the defect was actually triggered. The proposed method, TARPS, enumerates candidate patches from a small DSL and admits a candidate only if it passes coverage-aware replay and a clean-task check; otherwise it refuses. In the main experiments TARPS achieves 71% repair and 76% transfer, compared with 10% and 14% for committing the first candidate. In the frozen-cohort ablation, however, reward plus clean-task checks also reaches 76% repair, the same as TARPS.

Critical strengths
  • Scoring repair only on episodes where the defect was triggered is a sensible evaluation choice, since it prevents partial fixes from being counted as full repairs. Fig. 5 illustrates this well: under the 5-step cap the swap defect is never reached. I think this protocol is the most reusable part of the paper.
  • The reporting is transparent. The paper includes an ablation ladder, paired bootstrap intervals, exact sign-flip p-values, and clearly labels post hoc analyses. The authors also state directly that reward plus clean checks matches the full method.
Critical weaknesses
  • The ablation suggests the gate adds little. On the frozen cohorts, full TARPS ties reward plus clean checks on repair (76.2 vs. 76.2) and is 4.8 points higher on transfer with an interval of [-38, 43] (Tables 1, 18). The remaining gain over NoGate amounts to testing several candidates before committing, which is the standard generate-and-validate approach in program repair.
  • All selection rules, including the simple ones, rely on privileged inputs: candidates are enumerated from the task's required actions, using a DSL that contains an inverse for each injected defect (Table 3), and failure confirmation and scoring use the known triggers. The checks specific to TARPS (trigger coverage and the reference-repair screen) add nothing measurable over RC, and their oracle-free replacement does much worse than simply omitting them (23.8 vs. 76.2 repair, Table 1).
  • The evaluation is small. Each cell has 3 diagnosis and 3 transfer instances, the regression check uses a single clean probe, and the task pools were reused during protocol development (Table 9). The headline sign-flip p-value is 0.0625.
  • The results appear noisy. The fixed reference repair scores 71/71 in the primary matrix but 90.5/85.7 in v7.
  • All defects are synthetic, and the baselines are weak: HarnessFix is an internal heuristic, and LLM-Diag only appears in an unpaired two-instance pilot and in the τ-bench v1 runs with the local model, where it never commits a patch.
  • The paper is hard to follow in places. Results are spread across several cohorts (v6, v7, pilots, reruns) that are referred to by internal run names, and Fig. 7 does not match Tables 5/6.
Questions for authors
  1. Are the "required task actions" used for candidate enumeration (L185, L281) the τ²-bench annotated actions checked by the evaluator? If so, how does TARPS perform when candidates are derived only from the failed trace?
  2. Why does full TARPS refuse the airline budget-cut cell, while RC/RCC admit restore_budget and reach 100/100? Is this caused by the single clean probe?
  3. Could you rerun the ablation on task pools that were not used during development, with larger nD/nT and more clean probes? A Full minus RC difference with an interval excluding zero would change my assessment.
  4. Fig. 7 shows 12 replay votes and no admitted candidate, but Tables 5/6 report that HarnessFix admitted one operator in that cell. Which run does Fig. 7 correspond to?
  • Evaluating on naturally occurring harness bugs, e.g., those reported in open-source agent frameworks, would make the results more convincing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.