- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduced CounterGraft, a method to capture richer feedback signals during agent self-improvement.
Given an incumbent agent A0 and a candidate agent A1, both perform a task x, producing trajectories T0 and T1. A handoff, including an action and its associated execution state, is determined in T0, and A1 resumes from it to finish the task (hence “grafting”). Comparing this execution with A1’s own execution may provide richer feedback on whether the A0->A1 modifications are meaningful.
The method is empirically tested on ALFWorld and AgentDojo, with 1.3-1.9 pp gains on AgentDojo and 0.7 pp on ALFWorld over the strongest adapted baseline.
Critical strengths
- Grafting a partial trajectory to a new policy could provide richer signals and enable better credit assignment. The motivation is sound.
Critical weaknesses
- The manuscript is very hard to follow. In particular, (1) many terms are introduced without sufficiently clear explanations, e.g., incumbent, handoff, departure, and release; (2) it does not clearly establish the problem setting for “agent self-improvement,” which could refer to different problems.
- Too many details are missing from the experimental settings. (1) How are the 64 development tasks and 159 confirmation tasks selected for AgentDojo? The original benchmark has 97 user tasks and 629 security test cases, so the numbers do not match directly. For ALFWorld, the source and role of the separate 200-task set are unclear, although the 134-task confirmation workload matches the standard unseen split. (2) The paper mentions “3 targets” in the AgentDojo experiments, but provides little information on what these are. (3) Details on the initial incumbent agents are also lacking. Did you use any open-source agent framework? What tools does this agent have?
Questions for authors
- I find it hard to understand Figure 1. What extra information is obtained from grafting the child agent onto the parent agent’s trajectory? By reflecting on the child agent’s trajectory alone, the meta-agent could also plausibly find the bug in the stopping rule. The problem in this example seems identifiable from ordinary trajectory feedback.
- How did you implement the Self-Refine baseline (Madaan et al., 2023)? It was originally designed for non-agent tasks. Adapting this method from output refinement to agent-program self-improvement would be non-trivial.
2 comments
Thanks for your response. The clarifications were helpful, but I still think they do not fully address my concerns. In particular, the data construction and experimental settings still appear somewhat arbitrary, and needs further justification.
RE 1. I am still confused by the example in Figure 1. You mentioned the “incumbent’s post-refund state” but neither of the two execution branches in panel (a) involves a refund. Did I miss anything?
RE 2. It is still unclear how F1-F4 were selected. By “an upstream development-feedback proposal stage” do you mean that you ran an initial agent, generated revisions, and grouped their failure modes? Was this grouping and selection performed by an agent or by humans? Do F1-F4 correspond, respectively, to “action suppression, excessive verification, state-dependent control flow, and premature termination”?
RE 3. Thanks for explaining the data splits. This explains what they contain, but not why the experiments were designed this way. In particular, for AgentDojo, why combine 21 unearned units from F3 with 43 units selected by SHA-256 ordering?
RE 4. Your implementation appears to be a substantial adaptation of Self-Refine (Madaan et al., 2023). I would expect such adaptations to be clearly documented in the paper.
The paper would also require significant work to improve its readability. Given these considerations, I'll maintain my score.
To Reviewer 1i57: Counterfactual information, experimental construction, and Self-Refine adaptation
Thank you for your questions! We explain the additional execution evidence, test its value against complete trace feedback, and specify the experimental construction and baseline adaptation.
1. What information does Figure 1 add beyond a trace?
The additional observation is the successor's outcome from the incumbent-selected handoff. Comparing it with the successor's own execution holds the continuation policy fixed and measures the value of the changed choice. Paired endpoints alone do not determine this comparison, as formalized by the non-identification result.
In Figure 1, the same successor succeeds from the incumbent's post-refund state but fails from its own terminal handoff. A trace can suggest a stopping-rule repair; grafting executes the alternative and measures the replacement's disadvantage. The controlled experiment below tests whether providing that evidence improves the repairs found.
We test its practical value against a baseline that receives complete traces, tool returns, failures, and costs. Across five paired searches for each of F1/F2/F4, with proposer, executor, 16 opportunities, and selector matched, counterfactual feedback reduces harmful replacements 122 → 78 (36.1%), lowers them in 12/15 pairs, and increases passed units by 1.53/159 per search. Beneficial replacements rise 623 → 708 while departures rise 889 → 912. These results directly address the ordinary-reflection alternative: adding executed counterfactual evidence improves the revisions found with the same proposal budget and selector.
2. Agent setting and starting programs.
GPT-5.4 revises executable
policy.pyandprompts.py; the executor model stays fixed. Programs interact with AgentDojo's Banking, Slack, Travel, and Workspace environments or ALFWorld's embodied actions. The incumbent is the reference program, a successor is its revision, a handoff combines an action and restorable state, a departure is a changed selected behavior, and release admits a frozen candidate under the contract.F1–F4 come from an upstream development-feedback proposal stage and are selected before confirmation for different failure profiles: action suppression, excessive verification, state-dependent control flow, and premature termination. F3 develops the procedure; F1/F2/F4 are the evaluated repair targets. All methods receive the same target-specific starting files. The execution supplement will identify the runner/framework and version, exposed tool schemas, initial program files, and artifact genealogy.
3. Workload construction.
AgentDojo counts executable units:
(suite, user_task_id)for utility and(suite, user_task_id, injection_task_id)for security. Development contains 26 utility and 38 security units: 21 unearned units from F3 evidence plus 43 selected by SHA-256 ordering, frozen and shared across methods. Confirmation contains 49 utility and 110 security units, excluding all exact development and previously opened source-edge units. Utility pass means legitimate-task completion; security pass means attack prevention; execution errors receive zero. Broader template overlap is reported separately.ALFWorld development uses
valid_seen(140 games); complete-method confirmation usesvalid_unseen(134). The separate audit uses all 200 games injson_2.1.1/valid_trainin our data distribution, frozen before candidate evaluation. These are a separate collection, not a sample from the 140 games. The 140-versus-200 comparison has zero path, game-hash, and trial-identity overlap; task families can overlap.4. Self-Refine adaptation.
Each Self-Refine opportunity critiques the current executable program using ordinary development feedback, then produces replacement policy/prompt files. A valid revision becomes the next iteration's current program; an invalid revision consumes its opportunity and retains the latest valid program. Final selection is applied separately to evaluated candidates using the common evaluator and qualification rule. Counterfactual records inform that selector for all methods but are withheld from the Self-Refine proposer. This preserves iterative critique–revision while controlling the editable interface, development workload, and confirmation protocol.
We will put these definitions and construction details alongside the main experimental setup and enlarge Figure 1 so that the three executions and their distinct roles are immediately legible.
Thank you again for your thoughtful feedback and for considering these additional results and clarifications!