- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies repair of defects in the runtime harness of tool-using LLM agents. It argues that reward recovery alone cannot establish that a harness defect was repaired, because an episode may never exercise the defect, and proposes an evaluation protocol based on injected defects with recorded trigger events: repair and transfer are scored only over episodes in which the defect is “covered”. TARPS is a patch-selection procedure that enumerates candidate operators from a small DSL, screens replay instances using the known reference repair, and admits a candidate only if it passes covered replay on diagnosis instances and a majority vote on a clean probe. On seven τ²-bench domain–defect cells, TARPS reaches 71%/76% repair/transfer versus 10%/14% for committing the first candidate.
Critical strengths
- The paper targets a real gap, i.e., task success does not mean the defect was repaired, so evaluation needs to check whether the defect was actually exercised.
Critical weaknesses
- The scale of the experiments is too small to support the claims. The main results come from only one agent on one benchmark with 42 tasks. I do not consider the results reliable or generalizable at this sample size. Even based on current results, the paper’s own ablation shows that the proposed components do not work well. In Table 1, full TARPS gets 76.2/76.2 (repair/transfer), while “reward + clean check” without the two TARPS-specific checks gets 76.2/71.4. The 4.8-point transfer difference is only one task difference. So the main ideas of the method, the coverage-aware admission and the reference-repair screening, do not show any measurable benefits. The gain over NoGate comes from simply running the candidates and checking reward.
- The admission problem is patch overfitting, which the Automated Program Repair community has studied for a decade. The clean probe, the repeated replay, and the coverage check all have direct counterparts there. The paper cites and compares none of this. Moreover, the coverage condition and the reference-repair screen of the gate require the injected defect spec and the reference repair, which do not exist for a real harness bug.
Questions for authors
- What do you claim the coverage-aware admission and the reference-repair screening contribute?
- Can you report results on a larger and fresh task pool?