- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Living-Harness, a self-evolving harness for interactive LLM agents. After each scored episode, a domain-level Evolution-SOP guides three steps. A posterior generator abstracts the trajectory and evaluator signals. A memory extractor and a workflow extractor then produce candidate updates. These updates pass schema, scope, evidence, constraint and merge gates before they are written into two persistent stores: episodic memory (trigger conditions, failure patterns, recovery actions) and a state graph (state nodes, repair edges, transition rules). Tools and base context stay frozen. Top-3 retrieval from both stores is given to the actor in later episodes.
Critical strengths
The motivation is clear and practical. An agent can see from feedback that it should transfer a user to a human, and still fail to call the tool the next time. Separating task-local reflection from persistent, scoped procedural repair is a useful distinction, and Figure 4 shows it concretely.
The design is reasonably careful. Updates happen only after scoring, so episode evidence never affects its own score. Tools and base context are frozen. Candidate updates pass explicit commit gates. The appendix gives all Evolution-SOPs and extractor prompts, which helps reproducibility.
Absolute numbers on τ²-Bench are high, and there are ablations of the SOP, memory and graph. The harness-state growth tables (Tables 4–5) are informative.
Critical weaknesses
1.Much of the gain may come from hand-written, benchmark-specific domain knowledge rather than self-evolution.
The Evolution-SOPs in Appendix E are manually written and contain concrete solution rules for the test domains. The Telecom SOP lists the exact repair tools (refuel_data, grant_app_permission, reset_apn_settings plus reboot for no-service, and others). It specifies the troubleshooting order. It even says transfer_to_human_agents should dominate for terminal service cases, which is exactly the repair shown in Figure 4. It also states that “high-frequency misses” should be promoted to stable branch rules. This reads like knowledge distilled from looking at failures on these same benchmarks.
In Figure 3, replacing the SOP with a generic extractor drops τ²-Bench to 73.38. That is almost the same as Reflexion (73.02). So without the domain SOP, the method shows essentially no gain over the strongest baseline. A fair comparison would give the same SOP text to the baselines, for example in the actor prompt or in the AWM/ReasoningBank extraction prompts. The paper should also explain how the SOPs were written and whether the authors looked at test trajectories when writing them.
2.The evaluation protocol appears to adapt on the test set, and the reported numbers mix cycles.
The paper does not state whether each evolution cycle reruns the same task instances. Table 4 suggests it does: Airline edges grow by exactly 200 per cycle and Retail by exactly 456. That is consistent with 4 edges per task on the 50 Airline and 114 Retail tasks. If so, memory built from failures on a task in cycle n is retrieved for the same task in cycle n+1. The score-before-update protocol only rules out reuse within a cycle. The limitations section concedes that robustness to “reduced recurrence” and held-out task families is not established. Under this protocol, the cross-backbone transfer in Table 3 is also on the same MultiWOZ tasks the state was built from, so it shows reuse rather than generalization.
3.There are also several number inconsistencies:
Table 1 reports Airline 88.00, which is the Cycle 2 peak, while Retail and Telecom use the Cycle 3 values. This amounts to best-cycle selection on the test set. Telecom Cycle 0 in Table 2 is 57.39. That equals GPT-5.2’s τ²-Bench average in Table 1, not its Telecom score (52.23). It also does not match a k/114 success rate.
Questions for authors
1.How were the Evolution-SOPs written? Did the authors look at trajectories from the evaluation tasks? Can you report baselines (Reflexion, AWM, ReasoningBank) with the same SOP content available, and Living-Harness with SOPs written without access to these benchmarks?
2.Does each cycle rerun the same task instances? Please report results with the harness evolved on one split and evaluated on held-out tasks or task families, for both the main results and the cross-backbone transfer. Why does Table 1 use the Cycle 2 Airline score? Why does Telecom Cycle 0 equal 57.39?
5 comments
Response to Reviewer 9kLj (Part 2 of 3)
W2. The evaluation protocol appears to adapt on the test set, and the reported numbers mix cycles. The paper does not state whether each evolution cycle reruns the same task instances. Table 4 suggests it does: Airline edges grow by exactly 200 per cycle and Retail by exactly 456. That is consistent with 4 edges per task on the 50 Airline and 114 Retail tasks. If so, memory built from failures on a task in cycle n is retrieved for the same task in cycle n+1. The score-before-update protocol only rules out reuse within a cycle. The limitations section concedes that robustness to "reduced recurrence" and held-out task families is not established. Under this protocol, the cross-backbone transfer in Table 3 is also on the same MultiWOZ tasks the state was built from, so it shows reuse rather than generalization.
Q2. Does each cycle rerun the same task instances? Please report results with the harness evolved on one split and evaluated on held-out tasks or task families, for both the main results and the cross-backbone transfer.
Response to W2 and Q2: Each evolution cycle reruns the same task instances. τ²-Bench applies updates after each scored episode, while MultiWOZ-2.4 synchronizes updates after all four episodes in a window are scored (App. B.1). The main experiments therefore measure how procedural repairs accumulate during continued interaction, and each episode is scored before its updates are applied, so they cannot affect its own recorded score.
To evaluate unseen tasks, we split each τ²-Bench domain 7:3 into disjoint update and evaluation sets (80/34 tasks for Retail and Telecom, 35/15 for Airline). The harness evolves on the update set for three cycles and is then frozen for evaluation. Living-Harness achieves the best result in every domain, averaging 75.90 versus 68.67 for ReasoningBank and 63.86 for the GPT-5.2 base (Table B; App. C, Table 7). The frozen state also transfers to other backbones on held-out tasks without further evolution (Table C; App. C, Table 8), raising the average from 86.75 to 92.77 for GLM-5 and from 75.90 to 80.72 for Kimi-K3. GLM-5 benefits even though its base already exceeds the GPT-5.2 harness that built the state.
Table B. Held-out evaluation on τ²-Bench with disjoint update and evaluation sets (GPT-5.2, Pass@1 %; average weighted by task count).
| Method | Retail | Airline | Telecom | Avg |
|---|---|---|---|---|
| GPT-5.2 base | 76.47 | 66.67 | 50.00 | 63.86 |
| AWM | 79.41 | 46.67 | 44.12 | 59.04 |
| ReasoningBank | 79.41 | 66.67 | 58.82 | 68.67 |
| Living-Harness | 91.18 | 73.33 | 61.76 | 75.90 |
Table C. Cross-backbone transfer on held-out τ²-Bench tasks. Bold marks the better setting for each backbone.
| Backbone | Harness state | Retail | Airline | Telecom | Avg |
|---|---|---|---|---|---|
| GLM-5 | None | 79.41 | 93.33 | 91.18 | 86.75 |
| GLM-5 | Transferred | 91.18 | 86.67 | 97.06 | 92.77 |
| Kimi-K3 | None | 85.29 | 73.33 | 67.65 | 75.90 |
| Kimi-K3 | Transferred | 91.18 | 73.33 | 73.53 | 80.72 |
Response to Reviewer 9kLj (Part 3 of 3)
W3. There are also several number inconsistencies: Table 1 reports Airline 88.00, which is the Cycle 2 peak, while Retail and Telecom use the Cycle 3 values. This amounts to best-cycle selection on the test set. Telecom Cycle 0 in Table 2 is 57.39. That equals GPT-5.2's τ²-Bench average in Table 1, not its Telecom score (52.23). It also does not match a k/114 success rate.
Q2 (continued). Why does Table 1 use the Cycle 2 Airline score? Why does Telecom Cycle 0 equal 57.39?
Response to W3 and Q2: Table 1 applies the same rule to Living-Harness and all interactive baselines alike: the best performance in each domain over three evolution cycles. Airline therefore uses Cycle 2 (88.00), and Table 2 also lists Cycle 3 (86.00). This follows recent harness-evolution work, which reports the best harness found during search in Meta-Harness [1] and the best-so-far agent in DGM [2]. We now state this rule in the Table 1 caption. The Telecom Cycle 0 entry should read 52.63 (60/114); we have corrected it in Table 2, along with the corresponding Table 1 entry.
Thank you again for your careful review; we welcome further questions.
References:
[1] Meta-Harness: End-to-End Optimization of Model Harnesses, arXiv 2026.
[2] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, ICLR 2026.
Response to Reviewer 9kLj (Part 1 of 3)
Thank you for your positive assessment and for recognizing the practical motivation of persistent procedural repair, the careful update design, and the value of our experiments and state-growth analyses. We address your questions below.
Response to W1 and Q1: Evolution-SOP is generated automatically by GPT-5.2 from each environment's existing specifications and system prompts, not manually written; no evaluation trajectories or ground truth are used to construct or revise it. We have corrected the manuscript's description of manually specified rules (App. D). The troubleshooting branches, repair tools, and escalation conditions in the Telecom SOP come from the domain policy and tool definitions that τ²-Bench provides to every agent. Its rules on recurring failures specify how the extractor should treat failure modes that recur across evolution loops during a run, rather than encoding outcomes observed in advance.
SOP content alone does not explain the gains. Giving the baselines the same Evolution-SOP text changes their τ²-Bench averages by only −12.95 to +3.60 points, and no SOP-equipped baseline exceeds 60.07, well below Living-Harness even without its SOP (73.38) (Table A; App. C, Table 9). The effect of the text is also inconsistent across domains: it raises the two memory-based baselines on Retail but lowers all three baselines on Telecom, by 10.53 to 25.44 points. Within Living-Harness, the same SOP raises every domain and the average by 9.71 points (73.38 → 83.09), because it governs how evaluated evidence becomes gated memory and state-graph repairs rather than acting as additional instructions. The gains therefore come from how Living-Harness uses the SOP, not from the SOP text.
Table A. Equal-information comparison on τ²-Bench (GPT-5.2, Pass@1 %; average weighted by task count).