- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considerates procedural editing as closed-loop repair of SOP violations in tool-using agents. It identifies a gap between successful teacher-forced rewriting and live action generation. It also points out that it is insufficient of correcting only the first error in a trajectory. The paper proposes ProcedureAware Closed-loop Editing (PACE) which is built on AlphaEdit by conditioning value optimization on recorded pre-action reasoning and escalating updates when execution returns to an identical divergence. After each round of edit, the agent is rerun and subsequent violations provide further supervision. Across four model–scaffold settings, PACE achieves the highest or tied-highest repair rate
Critical strengths
-
The paper formulates procedural editing as closed-loop, trajectory-level repair and identifies two limitations of static factual-editing evaluations, whch is a useful distinction for evaluating agent behavior.
-
The paper introduces PACE, combining decision-state value conditioning and bounded no-progress escalation with a controller. PACE uses execution feedback to select subsequent corrections while retaining the localization and projected-update structure of its editing backend.
-
The evaluation on SOPBench incidents across two Qwen models and two scaffolds, with additional GPT-OSS-20B results. PACE achieves the highest or tied-highest mean repair rate in all four main settings.
Critical weaknesses
-
The contributions of the two proposed mechanisms are not adequately isolated. Section 4.2 states that model-level ablations determine whether decision-state conditioning is enabled, but the paper does not present the corresponding full ablation results or explain whether configuration selection used an independent validation set. PACE uses conditioning and escalation on Qwen3-4B but only escalation on Qwen3-8B. Consequently, the main results do not reveal the benefit of conditioning across models. The pooled installed-but-inert statistics also do not directly show how much conditioning reduces this failure mode.
-
For Qwen3-4B, conditioning uses 40 value-optimization steps rather than 25 reported in AlphaEdit. Escalation additionally increases the norm allowance by 1.5 times and adds 15 optimization steps. Matching the number of optimization steps therefore does not match optimization effort or update strength. These changes may be useful of PACE, but further matched controls are needed to distinguish decision-state alignment and adaptive escalation from simply optimizing longer or applying stronger updates.
-
The installation-time comparison is useful, but end-to-end time per successful repair would better capture the cost of repeated rollouts and checker calls. I believe a small experiment accumulating edits across distinct incidents would further clarify the scope of the persistence claim.
Questions for authors
For the rebuttal, I encourage the authors to further clarify these points mentioned in the weakness. I am open to revisiting my score if these concerns are adequately addressed.
3 comments
Weakness 4
Our claim is that an installed correction stays in the weights and keeps holding as more edits are added in the same repair loop, even after the context is reset. Our existing runs support this.
-
Earlier corrections survive later edits. Under PACE, 89 of 195 successful repairs (46%) needed 2–5 sequential edits. In each case the final rollout starts from a fresh context and passes all five SOPBench checks with every accumulated edit installed. Each earlier correction therefore kept working after later edits were written on top of it. This is what the protection in Section 3.4 is designed for. For AlphaEdit the figure is 66 of 144 (46%).
-
Locality holds as edits accumulate. We grouped the held-out locality probes by the number of edits in the final model:
| Edits in final model | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| PACE, SOP-Bench-amz locality | 0.86 | 0.87 | 0.87 | 0.87 | 0.84 |
| PACE, own-domain locality | 0.90 | 0.82 | 0.86 | 0.87 | 0.76 |
| AlphaEdit, own-domain locality | 0.90 | 0.90 | 0.87 | 0.85 | 0.68 |
Cross-benchmark locality stays flat from one to five edits. Own-domain locality drops at five edits for both editors, and less for PACE (0.76) than for AlphaEdit (0.68). Items that reach five edits are mostly the hardest, unrepaired ones, and half of their own-domain probes share the edited user goal. The drop therefore mixes edit count with item difficulty and should not be read as the effect of accumulating edits alone.
- General capability does not drift with more edits. PACE's change in macro accuracy stays within 0.01 points at and on Qwen3-4B, and stays within 0.1 points of the unedited model at both budgets on both models (Table 2).
These results cover accumulation of up to five edits within one incident's repair loop. We will state this in Section 2, add the breakdown by edit count above to the appendix, and note in Section 5 that editing across many distinct deployed incidents is left to future work.
Dear Reviewer 1MwN,
Thank you for your detailed review. We wanted to follow up on our responses. If you have a chance before the discussion closes, we’d appreciate hearing whether our replies address your main concerns.
Thank you for your time and valuable feedback!
We thank the reviewer and address each point below.
We add the ablation below and clarify the paper's framing. The core contribution is closed-loop procedural editing, which has two diagnosed failure modes:
Escalation is PACE's default controller and helps across settings. Conditioning is a targeted remedy for A1 and is selected per model.
Ablation (ReAct, B=5, 3 replicates).
Configuration selection. We did not use a separate validation set. The only choice made per model is whether conditioning is on or off; there is no hyperparameter sweep. The held-out locality, transfer, and general-capability probes (Sec. 5.3) were not used to make this choice. We will state this in the paper.
Conditioning (Qwen3-4B, ReAct, B=5, 3 replicates). We gave AlphaEdit PACE's full optimization budget, 40 value-optimization steps plus the same escalation rung, and left out only conditioning:
Escalation. On Qwen3-8B, PACE and AlphaEdit use the same steps, norm clamp and solver, and PACE has no conditioning. The only difference is escalation, and it raises function-calling success from 0.44 to 0.66 (13 vs. 2 items, p = 0.007; Table 1, Sec. 5.2).
We will add these matched controls as an ablation table. Following the reviewer's suggestion, we will also add a uniform-strength control that applies the escalated configuration to every edit.
End-to-end time per successful repair. We computed this from our run logs. Repair time covers the whole closed loop: edit installation, live re-rollouts, and divergence mining plus SOP checking. It leaves out held-out probing, which is evaluation only. We divide repair time by the number of repaired items, pooling three replicates on one H100.
Total wall time across the four Qwen settings (3 replicates, 359 items):
We will add both tables and the cost breakdown to Section 4.4.