acceptodds
Under review as a conference paper at ICLR 2027

Procedural Editing: Closed-Loop Repair of SOP Violations in Tool-Use LLM Agents

Abstract

Standard Operating Procedures (SOPs) govern tool-use workflows by specifying which actions are permitted, under what conditions, and in what order. A single skipped check, premature tool call, or early termination can therefore invalidate an otherwise plausible outcome. Such failures are natural targets for model editing: an SOP checker can localize the first divergent decision and provide the context, emitted action, and required replacement. However, we identify two reasons why traditional model editing transfers poorly to agent trajectories. First, reasoning agents act from a decision state that includes their generated rationale; consequently, an edit can achieve perfect teacher-forced rewrite accuracy yet leave the live action unchanged. Second, correcting one decision rarely repairs an entire trajectory, because the agent may diverge again downstream. We formulate procedural editing as closed-loop, trajectory-level repair and introduce PACE (Procedure-Aware Closed-loop Editing). PACE optimizes corrections at the recorded decision state, reruns the agent after every update, converts each new divergence into the next edit, and strengthens a correction when execution returns to the identical failing state. Across 119 SOPBench incidents, two Qwen3 models, and both scaffolds, PACE achieves the highest or tied-highest SOP success in all four settings, reaching 0.50-0.51 under ReAct and 0.52-0.66 under function calling. It retains 0.81-0.86 of previously passing tasks, while preserving general-capability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, model editing, tool use, procedural alignment, standard operating procedures

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1MwNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considerates procedural editing as closed-loop repair of SOP violations in tool-using agents. It identifies a gap between successful teacher-forced rewriting and live action generation. It also points out that it is insufficient of correcting only the first error in a trajectory. The paper proposes ProcedureAware Closed-loop Editing (PACE) which is built on AlphaEdit by conditioning value optimization on recorded pre-action reasoning and escalating updates when execution returns to an identical divergence. After each round of edit, the agent is rerun and subsequent violations provide further supervision. Across four model–scaffold settings, PACE achieves the highest or tied-highest repair rate

Critical strengths
  1. The paper formulates procedural editing as closed-loop, trajectory-level repair and identifies two limitations of static factual-editing evaluations, whch is a useful distinction for evaluating agent behavior.

  2. The paper introduces PACE, combining decision-state value conditioning and bounded no-progress escalation with a controller. PACE uses execution feedback to select subsequent corrections while retaining the localization and projected-update structure of its editing backend.

  3. The evaluation on SOPBench incidents across two Qwen models and two scaffolds, with additional GPT-OSS-20B results. PACE achieves the highest or tied-highest mean repair rate in all four main settings.

Critical weaknesses
  1. The contributions of the two proposed mechanisms are not adequately isolated. Section 4.2 states that model-level ablations determine whether decision-state conditioning is enabled, but the paper does not present the corresponding full ablation results or explain whether configuration selection used an independent validation set. PACE uses conditioning and escalation on Qwen3-4B but only escalation on Qwen3-8B. Consequently, the main results do not reveal the benefit of conditioning across models. The pooled installed-but-inert statistics also do not directly show how much conditioning reduces this failure mode.

  2. For Qwen3-4B, conditioning uses 40 value-optimization steps rather than 25 reported in AlphaEdit. Escalation additionally increases the norm allowance by 1.5 times and adds 15 optimization steps. Matching the number of optimization steps therefore does not match optimization effort or update strength. These changes may be useful of PACE, but further matched controls are needed to distinguish decision-state alignment and adaptive escalation from simply optimizing longer or applying stronger updates.

  3. The installation-time comparison is useful, but end-to-end time per successful repair would better capture the cost of repeated rollouts and checker calls. I believe a small experiment accumulating edits across distinct incidents would further clarify the scope of the persistence claim.

Questions for authors

For the rebuttal, I encourage the authors to further clarify these points mentioned in the weakness. I am open to revisiting my score if these concerns are adequately addressed.

3 comments

We thank the reviewer and address each point below.

Weakness 1

We add the ablation below and clarify the paper's framing. The core contribution is closed-loop procedural editing, which has two diagnosed failure modes:

  • Installed-but-inert edits (A1): an edit passes its teacher-forced probe but does not change the live action.
  • Single-edit insufficiency (A2): one corrected decision rarely makes the whole trajectory compliant.

Escalation is PACE's default controller and helps across settings. Conditioning is a targeted remedy for A1 and is selected per model.

Ablation (ReAct, B=5B=5, 3 replicates).

ConfigurationQwen3-4BQwen3-8B
AlphaEdit0.280.51
AlphaEdit + escalation0.390.51
AlphaEdit + escalation + conditioning0.500.44
  • Escalation never lowers success. It raises Qwen3-4B ReAct success from 0.28 to 0.39 and Qwen3-8B function-calling success from 0.44 to 0.66 (p=0.007p=0.007, Table 1).
  • Conditioning is a targeted remedy for A1, and its effect depends on the model. On Qwen3-4B it adds +0.11 over escalation with the number of optimization steps matched (14 items better / 4 worse, p=0.03p=0.03). In the same runs, it lowers the inert rate, computed over all edit rounds, from 0.70 to 0.49. On Qwen3-8B it lowers success, so PACE turns it off there. Under function calling, τ^\hat\tau is empty, so conditioning has no effect by construction (Sec. 4.3).

Configuration selection. We did not use a separate validation set. The only choice made per model is whether conditioning is on or off; there is no hyperparameter sweep. The held-out locality, transfer, and general-capability probes (Sec. 5.3) were not used to make this choice. We will state this in the paper.

Weakness 2 we ran matched controls. Neither more value-optimization steps nor a stronger write explains PACE's gains.

Conditioning (Qwen3-4B, ReAct, B=5, 3 replicates). We gave AlphaEdit PACE's full optimization budget, 40 value-optimization steps plus the same escalation rung, and left out only conditioning:

VariantStepsEscalationConditioningSOP success
AlphaEdit + escalation25✓–0.37 ± 0.03
AlphaEdit + escalation (matched control)40✓–0.39 ± 0.04
AlphaEdit + conditioning40–✓0.43 ± 0.09
PACE40✓✓0.50 ± 0.09
  • Raising the step count from 25 to 40 adds only 0.02 (0.37 → 0.39).
  • With the step count matched, conditioning alone (0.43) beats the control even though the control also has escalation (0.39).

Escalation. On Qwen3-8B, PACE and AlphaEdit use the same steps, norm clamp and solver, and PACE has no conditioning. The only difference is escalation, and it raises function-calling success from 0.44 to 0.66 (13 vs. 2 items, p = 0.007; Table 1, Sec. 5.2).

We will add these matched controls as an ablation table. Following the reviewer's suggestion, we will also add a uniform-strength control that applies the escalated configuration to every edit.

Weakness 3

End-to-end time per successful repair. We computed this from our run logs. Repair time covers the whole closed loop: edit installation, live re-rollouts, and divergence mining plus SOP checking. It leaves out held-out probing, which is evaluation only. We divide repair time by the number of repaired items, pooling three replicates on one H100.

GPU-min per successful repairReAct 4BReAct 8BFC 4BFC 8B
PACE6.812.65.03.4
AlphaEdit16.111.94.73.6
LoRA18.514.840.019.2
GRPO14.415.669.913.6

Total wall time across the four Qwen settings (3 replicates, 359 items):

Repair loop (GPU-h)Successful repairs
PACE21.7195
AlphaEdit20.6144
GRPO33.3111
LoRA35.0103
  • The cost of the closed loop is small in absolute terms. One PACE repair loop takes 2.2–6.4 GPU-minutes per item. Re-rollouts take 24–41% of that time, and mining plus checking take 5–9%.
  • PACE adds little repair time. Its total repair time is about 5% above AlphaEdit's (21.7 vs. 20.6 GPU-hours), and it repairs 35% more items. That makes it 22% cheaper per success overall.
  • PACE is cheaper than the training-based baselines. Over the four settings it costs 2.7× less per success than GRPO and 3.0× less than LoRA. Per setting, it is 1.2–14× cheaper than either.

We will add both tables and the cost breakdown to Section 4.4.

Weakness 4

Our claim is that an installed correction stays in the weights and keeps holding as more edits are added in the same repair loop, even after the context is reset. Our existing runs support this.

  1. Earlier corrections survive later edits. Under PACE, 89 of 195 successful repairs (46%) needed 2–5 sequential edits. In each case the final rollout starts from a fresh context and passes all five SOPBench checks with every accumulated edit installed. Each earlier correction therefore kept working after later edits were written on top of it. This is what the KpK_p protection in Section 3.4 is designed for. For AlphaEdit the figure is 66 of 144 (46%).

  2. Locality holds as edits accumulate. We grouped the held-out locality probes by the number of edits in the final model:

Edits in final model12345
PACE, SOP-Bench-amz locality0.860.870.870.870.84
PACE, own-domain locality0.900.820.860.870.76
AlphaEdit, own-domain locality0.900.900.870.850.68

Cross-benchmark locality stays flat from one to five edits. Own-domain locality drops at five edits for both editors, and less for PACE (0.76) than for AlphaEdit (0.68). Items that reach five edits are mostly the hardest, unrepaired ones, and half of their own-domain probes share the edited user goal. The drop therefore mixes edit count with item difficulty and should not be read as the effect of accumulating edits alone.

  1. General capability does not drift with more edits. PACE's change in macro accuracy stays within 0.01 points at B=1B=1 and B=5B=5 on Qwen3-4B, and stays within 0.1 points of the unedited model at both budgets on both models (Table 2).

These results cover accumulation of up to five edits within one incident's repair loop. We will state this in Section 2, add the breakdown by edit count above to the appendix, and note in Section 5 that editing across many distinct deployed incidents is left to future work.

Dear Reviewer 1MwN,

Thank you for your detailed review. We wanted to follow up on our responses. If you have a chance before the discussion closes, we’d appreciate hearing whether our replies address your main concerns.

Thank you for your time and valuable feedback!

Reviewer_APWfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACE, a closed-loop model editing method that repairs SOP violations in tool-using LLM agents through checker-guided action corrections and decision-state-conditioned weight updates while seeking to preserve existing capabilities.

Critical strengths

An interesting approach to targeted agent repair. The paper extends model editing to procedural failures in tool-using agents, combining localized parameter updates with execution feedback. The distinction between successful teacher-forced rewriting and actual behavioral correction is particularly insightful. This provides a compelling perspective on repairing specific agent failures using very limited corrective supervision.

Strong results under a carefully controlled evaluation protocol. PACE achieves the highest or tied-highest repair success across all four main settings while largely preserving previously successful behavior and general capabilities. Importantly, the baselines receive the same checker feedback and closed-loop repair opportunities. These results provide meaningful evidence for the practical value of targeted procedural editing.

Critical weaknesses

Limited evaluation on larger failure sets. The main experiments involve 119 initial failure incidents, with only 29–30 per setting. It remains unclear whether PACE would remain effective as the number of failure trajectories increases substantially, and whether it would retain its performance advantage over competing baselines.

Questions for authors
  1. I would appreciate the authors’ perspective on the seemingly counterintuitive choice of conditioning action corrections on a recorded, potentially erroneous thought. Is there a theoretical or mechanistic explanation for why this can be beneficial, particularly when the agent regenerates its thoughts during execution? Could this optimization also induce unintended behavioral patterns, such as systematic inconsistencies between the generated reasoning and the actions taken? Would it be possible to provide relevant ablation studies to better understand this mechanism and its potential effects?

  2. Suggested additional evaluation—scalability: How do repair success and locality change with larger independently sampled failure sets and longer sequences of edits accumulated across different tasks within one model? Such evidence would materially strengthen confidence that the approach scales beyond the current setting of at most five edits per independently repaired incident.

  3. Clarification and budget-controlled validation: Conditioning uses 40 value-optimization steps versus AlphaEdit’s 25, while escalation adds another 15 steps and relaxes the norm bound. How much of PACE’s advantage remains when AlphaEdit receives comparable optimization effort and norm allowances? A matched-budget comparison would help distinguish the benefits of the proposed mechanisms from those of stronger optimization.

3 comments

Limited evaluation on larger failure sets.

1. Each incident is a repeated closed-loop experiment, not a single data point. Each item is a closed repair loop: up to five sequential edits, each followed by a full live rollout and re-mining, then 25 held-out probe rollouts (Appendix B). Across the four Qwen settings, the 119 incidents (30 (Qwen3-4B ReAct) + 29 (Qwen3-8B ReAct) + 30 (Qwen3-4B FC) + 30 (Qwen3-8B FC)) produce 2,499 repair loops, 7,500 installed edits and 62,370 held-out rollouts (Table 4), costing 1,014 H100 GPU-hours. A GPT-OSS-20B ReAct setting adds 30 more incidents (149 in total, 2,181 GPU-hours). Almost all of the cost comes from live execution. 70% goes to held-out probing, and every edit is checked by a full trajectory rollout. That cost, 8.5 H100 GPU-hours per failure, limits the number of incidents per setting.

2. The incidents are a diverse, domain-balanced sample from a larger pool of failures. The 119 incidents cover 6 SOPBench domains, 28 user goals, and all 5 failure categories: 62 action-graph violations, 30 premature exits, 13 tool-call errors, 8 constraint violations and 6 stalls (Appendix B). Each edit set must come from the edited model's own failures. We sample five per domain from the minable failures, meaning those where the checker finds a first divergence and the resulting edit request passes our filters:

ModelScaffoldCountbankdmvhealthcarehotellibraryonline_marketuniversityTotal
——Should-succeed tasks4836456924606288
Qwen3-4BReActDivergent / minable / used23 / 17 / 521 / 14 / 523 / 15 / 554 / 47 / 510 / 8 / 534 / 32 / 53 / 3 / –168 / 136 / 30
Qwen3-8BReActDivergent / minable / used25 / 14 / 514 / 6 / 520 / 10 / 540 / 27 / 57 / 4 / 421 / 19 / 51 / 1 / –128 / 81 / 29
GPT-OSS-20BReActDivergent / minable / used24 / 20 / 516 / 5 / 521 / 6 / 558 / 25 / 512 / 10 / 529 / 19 / 51 / 0 / –161 / 85 / 30
Qwen3-4BFCDivergent / used25 / 517 / 528 / 565 / 510 / 549 / 52 / –196 / 30
Qwen3-8BFCDivergent / used14 / 516 / 532 / 568 / 513 / 548 / 51 / –192 / 30

The scarcest domain sets the sample size, not the total number of failures. Under ReAct there are 81–136 minable failures per model. hotel and online_market supply 52–58% of them, while library supplies only 4–10. The Qwen3-8B library count of 4 is why that setting has 29 items, and university (at most 3) is excluded. We balance domains on purpose: base compliance ranges from 27.7% to 64.2% across domains (Table 4), so an unbalanced pool would let one procedure family drive the aggregate score. In the revision we will add this table to Appendix B and state the sampling rationale in Section 5.1.

3. The advantage is statistically significant at this sample size. We compare methods with paired, per-item sign tests on replicate-averaged success (Section 5.1). PACE beats AlphaEdit by 14/1 items (p = 0.001), 9/1 (p = 0.02) and 13/2 (p = 0.007) in three of four settings. These are item-level wins, so they are unlikely to come from a few unusual incidents. The ranking also holds across the budget curve (Fig. 3, AUC in Appendix D) and on the additional GPT-OSS-20B model (Table 6: 48.9% vs. 46.7%).

whether PACE would remain effective as the number of failure trajectories increases

  1. Scaling to many failures: Our runs test accumulation of up to five sequential edits inside one incident's repair loop.

Earlier corrections survive later edits. Under PACE, 89 of 195 successful repairs (46%) needed 2–5 sequential edits. In each of these, the final rollout starts from a fresh context and passes all five SOPBench checks with every accumulated edit installed. Each earlier correction therefore kept working after later edits were written on top of it, which is the purpose of the K_p protection (Section 4.4). AlphaEdit shows the same pattern: 66 of 144 (46%).

Locality holds as edits accumulate. The table groups held-out locality by the number of edits in the final model, pooled over the four Qwen settings:

Edits in final model12345
PACE, SOP-Bench-amz locality0.860.870.870.870.84
PACE, own-domain locality0.900.820.860.870.76
AlphaEdit, own-domain locality0.900.900.870.850.68

Whether PACE would remain effective as the number of failure trajectories increases [continue]

Cross-benchmark locality stays flat from one to five edits. Own-domain locality falls at five edits for both editors, and less for PACE (0.76) than for AlphaEdit (0.68). This drop should not be read as the effect of accumulating edits alone. Items reach five edits mainly because they are hard: 107 of the 116 PACE items with five edits (92%) were never repaired. Up to half of their own-domain probes also share the edited user goal. General capability also stays flat with budget: PACE stays within 0.1 points of the unedited model at both B = 1 and B = 5 on both models (Table 2).

Scope. Each incident is repaired starting from the unedited model. This matches the paper's problem formulation (Section 3), which defines repair per incident and scores it by trajectory-level success. Within that scope, the results above show that corrections persist and locality holds across up to five accumulated edits. Lifelong editing, where hundreds of incidents are installed into one deployed model, is a different problem: it adds its own questions, such as interference between incidents and capacity of the protected subspace. It is outside the scope of this paper. PACE is well placed for that setting. Its null-space solve was designed for thousands of sequential edits. In the revision we will:

  • state the scope of the persistence claim in Section 6 (Limitations),
  • add the edit-count breakdown to Appendix D,
  • name lifelong editing across many incidents as a direct extension.

Q1. Why condition on a recorded, possibly erroneous thought? Could it cause reasoning–action inconsistency? Ablations?

Why it helps. A ReAct agent does not sample its action from the context c. It samples it after writing a thought τ, so from c⊕τ (§4.2). PACE does not change where the edit is read. The key still sits at the anchor token inside c, which comes before any thought, so the edit fires on every rollout. What changes is the target the value is optimized for. Without conditioning, the value only has to produce a* right after c. That is a state from which the action is never sampled, so the edit can pass teacher-forced probes and still be overridden once the agent writes its rationale. This is the "installed but inert" failure: 48–77% of such edits under ReAct, versus 1–6% under function calling.

The recorded thought τ̂ is a deliberate choice, for two reasons:

  • It is the on-policy state. Rollouts are greedy, and the context up to the divergence is unchanged. So the edited agent usually reproduces this same rationale, and c⊕τ̂ is the decision state it actually reaches.
  • It is the hardest case to overcome. τ̂ argues for the wrong action a⁻. A value that makes a* win after this rationale is robust to the very reasoning that caused the error.

So τ̂ is conditioned on, not imitated. Its being erroneous is what makes it the right state to optimize against. Under function calling τ̂ is empty, and the objective reduces exactly to AlphaEdit's.

Ablation (Qwen3-4B, ReAct, B = 5, 3 replicates). We compared PACE with AlphaEdit given the same 40 steps and the same escalation, so conditioning is the only difference:

  • Inert rate: across all installed edits, it drops from 0.70 to 0.49.
  • SOP success: it rises from 0.39 to 0.50, with PACE better on 6 items and worse on 1.

Reasoning–action inconsistency. The edit targets the action, not the thought. At the edited decision point, the agent can therefore write a thought that leaned toward a⁻ and then emit a*. This mismatch is local by construction: the key fires only at the anchored state, and the update is projected away from preserved keys. It does not spread into general behavior. PACE keeps 0.81 of previously passing held-out tasks, versus 0.83 for AlphaEdit, and changes general-capability accuracy by at most 0.09 points (Fig. 4, Table 2). We will add a direct measurement: thought–action agreement on edited and held-out rollouts, with and without conditioning.

Q2. Scalability: larger failure sets and edits accumulated across tasks

Larger, independently sampled sets. We re-ran the comparison on larger domain-balanced sets drawn from the same pools. These are 58 items on Qwen3-4B and 50 on Qwen3-8B under ReAct, with 3 replicates each:

Model (items)PACEAlphaEdit
Qwen3-4B (58)0.54 ± 0.020.34 ± 0.03

Both results match the 29–30-item sets in Table 1, where Qwen3-4B shows a significant gain (0.28 → 0.50) and Qwen3-8B ReAct a tie. The advantage does not shrink as the sample grows.

Why the sample per setting is limited. Edit sets must come from the edited model's own minable failures, and the scarcest domain caps a balanced draw. Under ReAct, the Qwen models have 81–135 minable failures, but library contributes only 4–8 of them. The full study is already large: 2,499 closed repair loops, 7,500 installed edits and 62,370 held-out rollouts, costing about 8.5 GPU-hours per incident (Table 3).

Edits accumulated across tasks. We agree this is the natural next test. Within one repair loop, edits already accumulate and are protected by K_p (§4.4). Locality and general capability stay at AlphaEdit's level after up to five cumulative edits. PACE reuses AlphaEdit's null-space solve unchanged, and that solve was designed for long sequences of edits. In the revision we will add a cumulative experiment: install the full edit set task by task into one model and report repair success and locality as functions of the number of accumulated edits.

Q3. Matched-budget comparison

We gave AlphaEdit PACE's full optimization budget and norm allowance. This means the same 40 value-optimization steps and the same escalation rung (1.5× norm clamp, +15 steps on repeated divergence). Only conditioning was left out.

Variant (Qwen3-4B, ReAct)StepsEscalation (1.5× clamp, +15 steps)ConditioningSOP success
AlphaEdit + escalation25✓–0.37 ± 0.03
AlphaEdit + escalation (matched)40✓–0.39 ± 0.04
PACE40✓✓0.50 ± 0.03
  • Extra steps add little. Going from 25 to 40 steps adds 0.02 under ReAct. Under function calling, where conditioning has no effect, it changes success by at most 0.03 (0.51 → 0.48 on Qwen3-4B, 0.69 → 0.72 on Qwen3-8B).
  • With effort and norm allowance matched, conditioning still adds 0.11.
  • Escalation is tested by itself on Qwen3-8B. There PACE and AlphaEdit use identical steps, clamp and solver. Escalation applies its stronger write only to divergences that repeat byte-for-byte, and it raises function-calling success from 0.44 to 0.66 (13 vs. 2 items, p = 0.007; Table 1). Its benefit comes from strengthening only the edits that stalled. To separate that targeting from simply writing harder, we will add a uniform-strength control that applies the escalated configuration to every edit.
Reviewer_z8sYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes weight editing as a way to repair SOP violations in tool-use agents. SOPBench's executable checker is used to find the first divergent decision in a failed trajectory and the action the SOP requires there. An AlphaEdit-style edit then writes that correction into the model, and the task is rerun. Each new divergence becomes the next edit, up to five. PACE adds two changes on top of AlphaEdit's unchanged projected solve. It optimizes the edit after the model's recorded Thought, so that ReAct edits are not "installed but inert". It also strengthens an edit once when the agent returns to the identical failure. On 119 incidents with Qwen3-4B and Qwen3-8B, under ReAct and function calling, PACE repairs about 0.50–0.66 of the edited tasks at five edits and beats closed-loop AlphaEdit in three of four settings.

Critical strengths

The installed-but-inert finding is useful for anyone editing agents. Under ReAct, roughly half to three quarters of first edits pass the teacher-forced probe but do not change the live action. This shows that the standard editing metric does not carry over to reasoning agents, and it supports scoring edits by live trajectory success.

Critical weaknesses

[W1] The setting needs an executable SOP oracle, and when one exists, much simpler solutions are available and are not compared. PACE's supervision comes from SOPBench's checker. That checker is the SOP itself implemented as code: per-constraint verifiers run against the live database, plus an action graph of the required prerequisite calls. SOPBench notes that its permissibility check can run in real time during the interaction. The paper never says whether this checker is available when the agent is deployed, and both answers weaken the case for editing weights. For example:

  • If it is available, the same information PACE uses could be given to the agent directly. Three options are each straightforward and use exactly PACE's inputs: (a) feed the checker's message ("at this step the SOP requires login_user") back to the agent and retry, within the same five-attempt budget and on the same task; (b) expose the checker as a tool the agent calls before acting; (c) block the violating call and substitute the required action. Option (a) matches PACE's evaluation almost exactly, since success is measured by rerunning the same task, and the feedback names the correct action. Options (b) and (c) would also cover unseen tasks without changing the model. SOPBench itself shows that giving only the relevant tools, a form of tool filtering, raises pass rates substantially. The paper calls runtime enforcement "complementary" but tests none of these.
  • If it is not available, PACE has no source of corrections at deployment. The only use would then be offline: collect corrections, edit, and deploy without the checker. That setting is judged by how well the model does on unseen tasks, not by re-solving the task the checker already fixed. The paper's headline metric measures the ones on the same task that checker has been applied.

[W2] The edits patch individual instances rather than procedures, so the claimed benefit of putting the correction in the weights is not shown. The paper's argument against in-context approaches is that a weight edit persists across context resets. In the experiments, though, each incident gets its own freshly edited model. Held-out transfer is low: in Figure 3, only around 5–15% of other failing tasks are fixed, often no more than AlphaEdit or LoRA. Meanwhile, a single-task edit breaks about 15–20% of tasks that previously passed. So what persists is mostly a fix for one already-seen trajectory, at a measurable cost elsewhere. A stored correction retrieved in later episodes (ExpeL- or Agent Workflow Memory-style, both cited) would persist too, without touching the weights. Edits accumulated across many incidents in one model are never tested, and that is the setting where persistence would matter. The method contribution over AlphaEdit is also small. It amounts to one change in where the value is optimized and one escalation step, and on Qwen3-8B only the escalation step is used.

Questions for authors
  1. Is the SOP checker assumed to be available when the agent is deployed? If yes, how does PACE compare to feedback-and-retry with the checker's message, checker-as-tool, and a runtime guard, on the same incidents and with the same budget? If no, what is the intended deployment loop, and why is repair of the edited task itself the main metric?
  2. SOPBench's evaluator reports pass or fail on each dimension. How exactly are the first divergence and the required action a∗a^* derived from it, and does this use any task-specific information beyond the SOP code and the database?
  3. Can you show results with edits accumulated across many incidents in a single model, evaluated on unseen tasks that involve the same SOP rules? Without this, what evidence is there that the edits change the procedure rather than the instance?
2 comments

We thank the reviewer and address each point below.

The setting needs an executable SOP oracle, and when one exists, much simpler solutions are available and are not compared. ... SOPBench itself shows that giving only the relevant tools, a form of tool filtering, raises pass rates substantially.

The SOP and tool filtering are already in the prompt. All rollouts use SOPBench's oracle tool list, which exposes only the tools in the task's action graph, and the full structured SOP constraints in the system prompt. The tool-filtering gain the reviewer cites is therefore already part of our baseline. Every repaired failure occurs even though the violated rule is stated in the agent's context.

The paper never says whether this checker is available when the agent is deployed ... The paper calls runtime enforcement "complementary" but tests none of these.

The checker is an offline auditor. In our experiments it never runs during an episode. SOPBench runs in prompt mode, so violating calls are neither blocked nor flagged. The checker scores each completed trajectory, and PACE mines the next correction between rollouts. PACE therefore targets the setting where an executable checker or an LLM judge reviews failures after the fact and the patched agent then runs without it.

When an oracle is available at deployment, runtime enforcement is the right tool, such as blocking or substituting the violating call (options b and c). It complements editing: editing changes the model's own action preference, and the guard remains as a safety net.

If it is available, the same information PACE uses could be given to the agent directly. ... (a) feed the checker's message ("at this step the SOP requires login_user") back to the agent and retry, within the same five-attempt budget and on the same task ...

We add experiments: the same correction provided in the prompt. Following this suggestion, we gave each failing item the correction PACE installs, as text, in two forms:

Direct tuple: exact same tuple used for training including the decision state, the emitted action, and the SOP-required action. M-rule: an general rule that describe the tuple in natural language such as "Before apply_credit_card, you must call login_user." Each note goes either in the system prompt (Sys) or in the first user message (User). Results are final SOP success rate under same budget on Qwen3-8B, with 30 function-calling (FC) items and 29 ReAct items.

MethodFC @5ReAct @5
PACE (paper, 3 reps)0.660.51
Direct tuple (Sys)0.600.45
Direct tuple (User)0.770.41
M-rule (Sys)0.700.52
M-rule (User)0.730.45

The ruslts shows PACE matching prompting with similar performance.

These baselines rerun a short, single-session conversation in which the note is only a few turns from the decision. This is the most favorable condition for in-context correction. Models use instructions less reliably when the instructions are spread across turns or placed in the middle of long contexts, and weight edits retain such information better than prompting over multi-turn interactions [1,2,3]. Weight edits also keep the prompt length constant, and they need no retriever to decide which of many accumulated notes applies to the current state.

Our central claim concerns parametric repair. Given the same single-incident supervision, PACE outperforms other editors (ROME, MEMIT, AlphaEdit) and fine-tuning and RL baselines (LoRA, FT-M, GRPO) in trajectory success. We will add the prompting comparison to the paper and revise the claims accordingly.

[1] LLMs Get Lost in Multi-Turn Conversation. ICLR 2026.

[2] Towards Effective Model Editing for LLM Personalization. 2025.

[3] Lost in the Middle: How Language Models Use Long Contexts. TACL, 2024.

The edits patch individual instances rather than procedures, so the claimed benefit of putting the correction in the weights is not shown. ... In the experiments, though, each incident gets its own freshly edited model. ... Edits accumulated across many incidents in one model are never tested ...

To clarify the scope: the paper tests accumulation within a long-horizon task. Each repair loop installs up to five sequential edits at successive divergent decisions of the same task. The held-out probes score the final model carrying all of that task's edits. Each repair loop starts from the unedited model, and the held-out probes score a model that carries one incident's edits. What the experiments do show: given one failed trajectory, PACE repairs that incident in the weights.

A stored correction retrieved in later episodes would persist too, without touching the weights.

We compared two memory baselines on Qwen3-8B with the same items, budget, and probes: item memory, which replays the stored rule in later episodes of the same task, and a domain memory bank, which pools rules across a domain and is applied in fresh runs. Item memory matches PACE on repair (FC 0.70 vs. 0.66, ReAct 0.52 vs. 0.51; sign test p = 1 in both). Memory also transfers better to held-out tasks (FC: 0.23 item / 0.36 bank vs. 0.08 for PACE; ReAct: 0.10 / 0.12 vs. 0.06). On this benchmark, a stored correction therefore persists at least as well as a weight edit. Editing's remaining advantages are a constant prompt length (the domain banks reach 0.8–3.2k characters per domain) and applicability when the deployer cannot change the prompt. We will add these results and present memory and editing as complementary.

Held-out transfer is low ... only around 5–15% of other failing tasks are fixed, often no more than AlphaEdit or LoRA.

Transfer is near the floor for every projected editor (Fig. 3). This follows from the design: the key is extracted at an incident-specific anchor (Section 3), so the edit repairs an instance, not a procedure family. We will describe PACE as incident repair throughout and state.

Meanwhile, a single-task edit breaks about 15–20% of tasks that previously passed.

Part of this gap is not caused by the edit. Locality is scored against pass labels recorded from one earlier rollout, and long greedy rollouts are not reproducible across runs (Section 5.1, Paired evaluation). In the memory experiment above, rerunning the unedited model on the same ReAct probes gives a locality of 0.75. PACE's 0.81–0.86 is at or above this reference. We will add a no-edit rerun reference to Fig. 3 and revise the text so that the gap is not read as damage caused entirely by the edit.

The method contribution over AlphaEdit is also small. ... on Qwen3-8B only the escalation step is used.

The mechanisms are deliberately minimal, and the paper's main contribution lies elsewhere:

  1. Diagnosis. Edits that pass teacher-forced probes are inert in the live rollout in 48–77% of cases under ReAct, versus 1–6% under function calling (Section 4.2, A1). A single edit repairs only 0.27–0.31 of trajectories (A2).
  2. Protocol. Closed-loop repair scored by trajectory success. We apply it to every baseline, and it raises success for every editor (Fig. 2).
  3. Two targeted fixes. Each addresses one diagnosed failure without changing the backend's solve. That a single escalation rung lifts Qwen3-8B FC success from 0.44 to 0.66 (13 vs. 2 items, p=0.007; Table 1) shows how much the diagnosed failure costs. Where the fix does not trigger (Qwen3-8B ReAct, where regenerated reasoning makes identical re-divergences rare), PACE ties AlphaEdit, and we report this as is.

We will reorder the contributions as diagnosis, protocol, then mechanisms.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to AC/SAC/PC

Dear AC/SAC/PC,

We would like to sincerely appreciate the valuable feedback from all the reviewers and their acknowledgement of our contributions:

  • Problem formulation and diagnosis: The paper treats procedural editing as a repair loop driven by a checker, scored by whether the whole episode follows the procedure. It finds two failures of standard editing. First, edits pass the usual test but don't change the agent's live action: 48–77% of edits under ReAct, versus 1–6% under function calling. Second, one edit rarely fixes a long episode: only 27–31% are repaired.
  • Method (PACE): A diagnose, install, verify, adapt loop with two targeted fixes. The edit is optimized after the agent's own reasoning, which addresses inert edits, and a stronger edit is applied when the same divergence repeats. PACE keeps the localization.
  • Evaluation: Across 119 incidents, 2,499 repair loops and 62,370 held-out rollouts, PACE is best or tied-best in all four settings (0.50–0.66 success). It installs edits about 3× faster than GRPO under function calling.

Strengths raised by reviewers

  • The inert-edit finding is new and useful. All three reviewers note it. z8sY says it shows the standard editing metric doesn't carry over to reasoning agents. APWf calls it "particularly insightful."
  • The closed-loop, episode-level framing is a useful distinction for evaluating agents (1MwN, z8sY). It also extends model editing to procedural agent failures (APWf).
  • Results are strong and controlled. PACE is best or tied-best in all 4 settings: 2 Qwen models × 2 agent setups, plus GPT-OSS-20B. Baselines get the same checker feedback and repair loop. Previously passing behavior and general capability are mostly preserved (APWf, 1MwN).

Concerns and how the rebuttal answered them

Robustness of the results (APWf, 1MwN)

  • Sample size: each incident is a full repair loop of about 8.5 H100 GPU-hours. The 149 incidents are drawn evenly across 6 domains and all 5 failure types.
  • Significance: per-item sign tests are significant in 3 of 4 settings (p = 0.001–0.02). A larger 58-item Qwen3-4B set gives PACE 0.54 vs 0.34 for AlphaEdit, so the advantage holds as the sample grows.

Where the gains come from (APWf Q1/Q3, 1MwN W1/W2)

  • Conditioning: with optimization steps and edit strength matched, AlphaEdit reaches 0.39 and PACE 0.50. Extra steps alone add about 0.02. Conditioning lowers the inert rate from 0.70 to 0.49, which ties the method directly to the diagnosis.
  • Escalation: it never lowers success. On Qwen3-8B with function calling it raises success from 0.44 to 0.66 (p = 0.007).
  • Conditioning on reasoning that may be wrong: that reasoning is the state the agent actually reaches before acting. Overriding it is the hardest case, and any resulting thought/action mismatch stays local.

Cost (1MwN)

  • PACE's repair time is about 5% above AlphaEdit's, and it repairs 35% more items (195 vs 144). That makes it 22% cheaper per success, and 2.7–3× cheaper per success than GRPO or LoRA.

Persistence and side effects (APWf, 1MwN, z8sY)

  • In 46% of successful repairs, 2–5 edits were stacked, and earlier fixes kept working after later ones were added.
  • Cross-benchmark locality stays flat (0.84–0.87) as edits accumulate, and general capability stays within 0.1 points of the unedited model.
  • The apparent 15–20% of broken tasks is partly rerun noise: rerunning the unedited model gives locality 0.75, while PACE scores 0.81–0.86.

Setting and baselines (z8sY)

  • The SOP and the filtered tool list are already in every prompt, so the repaired failures happen even though the rule is in the agent's context.
  • in a new experiment, PACE performs about as well as giving the agent the same correction as text (FC 0.66 vs 0.60–0.77; ReAct 0.51 vs 0.41–0.52). Unlike prompting, weight edits keep prompt length fixed and need no step to pick which stored note applies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.