acceptodds
Under review as a conference paper at ICLR 2027

Rebind: Learning Safer Agent Skills from Evolving Experience

Abstract

Even a successful execution can leave a skill's incorrect rules undetected. For example, a reusable calendar skill might select the correct participants by choosing the first matching meeting, but then invite the wrong people after the meeting is canceled and recreated. Simply fixing an execution error might allow the invitations to be sent, but it leaves the selection rule unchanged. We propose Rebind, which learns safer executable skills by building rehearsals from the current skill. Each rehearsal changes the environment surrounding the objects used by the skill while maintaining the feasibility of the requests. Task results and actual effects provide feedback for repair. In the calendar example, Rebind detects invitations to former participants and repairs the skill to copy participants from the current confirmed meeting. It replays previous requests and rehearsals to check the revision, then uses the revised skill to build the next rehearsal. Across ten tool workflows, Rebind safely completes all 462 controlled conditions, compared with 425 for a baseline that prepares rehearsals in advance, and reduces harmful conditions from 33 to zero. The learned skills can be reused directly without further language-model calls. On 3,018 injected AgentDojo and AgentDyn conditions, Rebind completes 86.1% of tasks with a 0% attack success rate, compared with 78.4% and 14.4%, respectively, for SkillOpt.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent safety, skill learning, prompt injection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LZrCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

Agents often store successful procedures as reusable executable skills. The authors argue that a skill being runnable, or succeeding on its source tasks, does not mean its decision rules are correct. In particular, its rule for selecting which object to act on may match the correct choice in the original environment but diverge once the environment changes. This can cause unauthorized effects; for example, a calendar skill copies participants from a canceled meeting rather than its recreated replacement.

The authors propose Rebind, which learns safer skills through a rehearsal loop: 1. It perturbs the environment around the objects the current skill relies on. 2. A reference program verifies that each perturbed task remains feasible. 3. Task and effect feedback drives LLM-based code repair, with replay of all prior conditions to prevent regression. 4. New rehearsals are built from the revised skill's behavior. 5. The learned skill is then executed directly, without LLM calls at deployment.

On AgentDojo and AgentDyn, Rebind reports 86.1% utility under attack with 0% attack success, versus 78.4% and 14.4% for SkillOpt. In a controlled study over ten tool workflows, it safely completes all 462 conditions with no harmful effects, outperforming variants that precompute rehearsals or build them only from the initial skill. It also improves results on the file-skill benchmarks SkillSafetyBench and SkillLearnBench.

Critical strengths
  1. I think the problem observation is interesting. The skill that agent derives from tasks it has seen might be based on the wrong scenario and might cause additional harms when applied in other tasks, this observation might be beneficial for future skill-learning.

  2. The ablation supports the method's central design choice. The experiment shows that under the same conditions, building rehearsals from the revised skill outperforms building them in advance or only from the initial skill.

Critical weaknesses
  1. The controlled evaluation is circular and in-distribution. Rehearsal operations, reference programs, and effect checkers are hand-written per tool family (Table A4, A.5). The repair model also receives guidance on the relevant relationships (A.6). Evaluation conditions are drawn from the same operation families and judged by the same checkers used during learning. The perfect 462/462 and 180/180 results therefore show that Rebind can satisfy the authors' own oracle within that oracle's distribution, not that it produces safer skills in general.

  2. The headline security result does not show Rebind's learning effect, and I think that is a framing issue. The 0% attack success on AgentDojo/AgentDyn comes from executing skill code directly, with no LLM at deployment. The unlearned Initial Skill already has 2/3018 attack successes (0.1%), and Precomputed Rehearsals also has 0%. Under the same execution interface, Rebind's gain over Precomputed Rehearsals in joint success is +0.99 points with a 95% interval of [-0.96, 3.07], which is not significant. Yet the abstract frames the result as Rebind (0%) versus SkillOpt (14.4%).

  3. The paper was really hard to read and understand, especially the algorithm diagram as well as the sudo code. Moreover, I think the paper's entire writing and expression is mixed up with different concepts and words jumping back and forth

Questions for authors
  1. Can the authors report results on held-out evaluations of environment changes from operation families not in the learning library?
  2. Do the authors agree that the 0% attack success rate stems mainly from direct code execution, given Initial Skill's 0.1% and Precomputed Rehearsals' 0%? If so, will the abstract and Section 4.1 be reframed?
  3. Why are the results identical across all three (depth, distractor) configurations for every method, including Initial Skill and Repair-only?

Clarification:

  1. What is the human cost of adding a new tool family, in terms of the number of operations, the lines of code for reference programs and effect checkers, and the time required? How do we scale it up in future?
5 comments
AuthorsOpenReview ↗

Response to Reviewer LZrC (1/3): the security claim (W2, Q2) and the identical configurations (Q3)

Dear Reviewer LZrC,

Thank you for your review. Each weakness you list pointed to something we could test or change, and we have acted on all three. We start with your Q2, because its answer separates what direct execution does from what learning adds. The held-out evaluation that your Q1 asks for is in our second comment, and the third covers the diagram, the pseudocode and the wording.

Q2 / W2. Does the 0% attack success stem mainly from direct code execution, and will the abstract and Section 4.1 be reframed? Yes to both. A deployed skill is a program, and no model reads tool output while it runs, so an injected instruction has no one to instruct. This holds before any learning: Initial Skill has only 2 attack successes in 3,018 conditions (Table B3). Section 4.1 of the submission said so; the abstract now says it as well, and ends:

The learned skills can be reused directly without further language-model calls. Direct execution means that no model reads injected text as an instruction, with or without learning. On 3,018 injected AgentDojo and AgentDyn conditions, learning raises joint success (task success without attack success) from 52.2% to 86.1%, compared with 70.9% for SkillOpt.

Section 4.1 no longer compares Rebind's zero with SkillOpt's and Progent's attack counts, and its paragraph on direct execution says that the near-zero attack success is present before any learning. Table 1 has three blocks by execution interface, so that the effect of learning can be read as the difference between Initial Skill and Rebind, both in the last block. With clean and injected conditions counted together, Table B4 gives this difference as 34.09 points of joint success [27.33, 41.13].

Rebind's lead over Precomputed Rehearsals on these benchmarks is 0.99 points, as you note, and the reason lies in the attacks: they insert text. They do not change which meeting is current or who is on it, so they rarely reach the selection rules, by which a skill decides which object to act on. It is in these rules that the two differ. When those relations do change, the two diverge and the intervals exclude zero:

Safe completionsRebindPrecomputed RehearsalsDifference in points [95% interval]
New held-out panel (144 conditions; our next comment)1441319.03 [1.11, 16.67]
First panel of Table 2 (462 conditions)4624258.0 [1.2, 16.0]

These gains come from individual selection rules. Over the two original panels of Table 2 (462 and 180 conditions), Rebind is safe on 61 conditions where Precomputed Rehearsals is not. Deleting the responsible rule from Rebind's skill loses the gain in all 61 cases, and copying that rule alone into the Precomputed Rehearsals skill recovers it in 60 (Table E8).

Q3. Why are the results identical across the three (depth, distractor) configurations? Because the configurations vary two quantities that no selection rule depends on, as Section 4.2 now explains. Depth is the number of superseded versions created before the current object, and the distractor count is the number of alternative records created after it (Appendix C.7). A skill that takes the first match picks a superseded version whether there are 2, 4 or 7 of them, so it fails in all three configurations. A skill that takes the current confirmed object picks it however many distractors follow, and is safe in all three. Each skill therefore gets the same verdict in every configuration: all 60 of Rebind's skills are safe, compared with 52 and 55 of 60 for the two controls (Table C10). Appendix E.9 confirms this skill by skill for all five methods, including Initial Skill and Repair only: of 60 skills each, 13 and 23 are safe in all three configurations, and the rest in none.

AuthorsOpenReview ↗

Response to Reviewer LZrC (2/3): circular evaluation and held-out operation families (W1, Q1)

Q1 / W1. Are there results on held-out evaluations of environment changes from operation families not in the learning library? Yes. The frozen skills from the submission complete all 144 conditions of a held-out panel, none of them harmful, while the two rehearsal controls complete 131 and 132 (third panel of Table 2; Appendix E.1). In 23 of the panel's 24 states, the change is one that the operation library (Table A4) cannot produce.

Your W1 names three things that the submitted panels share with the learning process: the operation families, the checkers and the relation guidance. We designed the new tests around this list. In the table below, each test after the first row removes one or more of the three, and the last row removes all three at once. The columns "Changes," "Scored by" and "Guidance" say which. Three kinds of scorer appear: the effect checks used in learning, a separately implemented evaluator of ours that does not call them, and an independent checker written by a different model. Cells give safe completions, with harmful conditions in parentheses.

Test (conditions)ChangesScored byGuidancePrecomputedStarting-SkillRebind
Table 2, first panel (462)from the libraryeffect checks used in learningyes425 (33)432 (21)462 (0)
Held-out panel (144)outside the library (23 of 24 states)separately implemented evaluatoryes131 (13)132 (7)144 (0)
Rule-generated changes (804)outside the libraryevaluator and independent checkeryes800 (4)780 (11)804 (0)
Pairs of rule-generated changes (672)those that alter what the request refers toevaluator and independent checkeryes656 (16)640 (8)672 (0)
Newly learned skills (384)the 24 states of the held-out panelindependent checkerno337 (42)335 (46)358 (23)

The held-out changes use mechanisms outside Table A4: an event edited in place, a file arriving from another owner, a new share, a change of owner. The library, by contrast, alters an event only by canceling it and recreating it under a new identifier, and it never changes a file's sharing or owner. The skills were frozen before the panel was built.

In the next two rows we no longer choose the states ourselves. A fixed rule applies every state-changing tool call to each object the request refers to, and to one object the request does not involve; the call arguments come from a grid fixed in advance. The row of rule-generated changes contains only those that the library's operations cannot produce. The rule's changes are then applied two at a time, which gives 13,932 conditions, and Rebind's skills complete every one of them, none harmful. The row of pairs reports the 672 of these conditions in which the pair changes what the request refers to, such as who must be invited. Rebind is the only method with no harmful condition in these two rows. When the same rule is applied to the Slack, banking and GitHub workflows (the other tool families), Initial Skill has 164 harmful conditions and no learned skill has any (714 conditions, scored by each family's effect checks; a checker that a different model wrote for these families gives the same safe counts).

Our evaluator and the independent checker agree. Rescored with the independent checker of Appendix E.2, the held-out panel gives the same counts, and on the rule-generated changes and their pairs the two give the same verdict on every execution.

The test with newly learned skills also removes the other two things your W1 lists: learning runs without the relation guidance, and scoring uses neither our evaluator nor our effect checks. Rebind and both controls learn again from 80 new starting skills, using core learning only (the first stage of the method). The independent checker was written by a different model, GLM-5.3, from a plain-language description of the requests; that description was drafted by a separate agent with no access to our evaluator, our effect checks or our results. The checker was frozen before any learned skill was scored.

Even so, Rebind completes more conditions safely than either control and has about half as many harmful ones (last row of the table). All four pre-specified comparisons, safe rate and harmful rate against each control, stay below 0.05 after the Holm adjustment (exact sign tests over the 80 starting skills; Table E2). So Rebind's advantage holds on changes that the library cannot produce, under an independent checker, and without the relation guidance.

AuthorsOpenReview ↗

Response to Reviewer LZrC (3/3): readability (W3), and the cost of a new tool family

W3. Hard to read, especially the algorithm diagram and the pseudocode; concepts and words jump back and forth. We agree, and we apologize for the extra effort. We rebuilt all three: the diagram, the pseudocode and the wording. The new diagram and the new pseudocode are on page 4 of the revised PDF.

Figure 2 now shows one round of the loop as four numbered steps on the calendar example: (1) run and trace, (2) build a rehearsal, (3) evaluate, (4) repair and replay. In the example, the skill first picks the canceled meeting and, after the repair, the confirmed one.

Algorithm 1 now has 17 numbered lines in two labeled parts, core learning and refinement. Its four steps are numbered as in Figure 2 and in the opening of Section 3. That opening also defines the terms that the algorithm uses: a candidate is a version of the skill considered during learning; learning proceeds in stages, core learning first and then refinement; and each stage runs one or more rounds of the four steps. It also says what the current skill is and explains why learning starts from two candidates. Appendix A.3 defines each step, and Algorithm A1 gives one refinement stage in ten lines. The figure, the algorithm and the text now follow the same four steps, with the same numbers.

As for the wording, four terms are renamed and used consistently throughout the main text: "environment-setup program" is now "operation," "prepared environment state" is "changed state," "reference execution" is "reference program" and "evolving skill" is "current skill." A rehearsal is built from the skill's "trace," which the submission called "observations." The main text also explains terms such as selection rule, unauthorized effect and tool workflow, and Section 3.4 states in one sentence how a candidate is selected. The passages that walked through the loop again in Sections 1, 3.3, 3.4 and 6 are cut to a sentence or two each. If a passage is still hard to follow, we would be glad to hear which one.

Clarification: human cost of a new tool family. Slack, banking and GitHub each took less than a day, and the calendar and drive family, which serves six workflows, about three days. The table gives the quantities you ask for, family by family, with lines counted per workflow. Table E9 has the remaining components, and Appendix E.8 the times.

Tool family (workflows)OperationsReference program (lines)Effect checks (lines)All components (lines)Time by hand
Slack (1)1656112under a day
Banking (1)116139264under a day
GitHub (2)1 each15173402–404under a day
Calendar and drive (6, shared)19169–185351–4651,163–1,384about three days

None of this is repeated for each skill: the components are written once, with code shared within a tool family. Everything else (3,451 lines) is written once and shared by all families.

How do we scale it up? Our answer is to generate the per-family part automatically, and we have done so for three workflows beyond the paper's ten. A language model wrote every component from the tool documentation, the request and a generic description of the interface. In all three cases the first version passed our automatic checks, and nobody edited it. The generation call for a workflow took four to five minutes and cost USD 0.57 to 1.07. Learning with these generated components raises safe completion from 102 (Initial Skill) to 137 of 174 conditions, on evaluation states and checks written by a different model (Appendix E.8). This also bears on your W1: on 22 further public tasks, a language model, not the authors, wrote the reference programs and effect checks, and the benchmarks' official evaluators scored the result. Learning still raised joint success under attack from 205 to 296 of 392 conditions.

Thank you again for the review. We hope that the new page 4 and the held-out evaluation address your concerns, and we would be happy to discuss any point further.

Best regards, The authors

Reviewer_i21QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies reusable agent skills implemented as executable programs that call tool APIs, and identifies the following failure mode: repairing a skill until it runs can leave an incorrect object-selection rule in place, which can especially backfire if the environment changes later. Rebind addresses this during skill "learning", by perturbing the environment around the object the current skill used, checks with independent reference program that the reequest remains feasible (essentially a unit test?), runs the skill, and feeds task outcome and actual effects to an LLM repair model that rewrites the code. Earlier requests and rehearsals are replayed to check each revision, and the revised skill guides the next rehearsal. The selected program is then frozen and reused with no model calls. Evaluation spans AgentDojo/AgentDyn under injection (87.9% "clean" utility, 0% attack success), a controlled study of 462 environment-change conditions, SkillSafetyBench and SkillLearnBench, plus budget curves and a replay ablation.

Critical strengths
  • The failure mode is real and well characterised. A repair that makes a skill executable can leave an incorrect selection rule intact, so a skill can pass every test it has seen and still act on the wrong object once the environment changes. This is a sharp account of why skill libraries are unsafe, beyond just prompt-injection framing.
  • The study mechanism is sound. Perturbing the environment around the object a skill used, checking feasibility with an independent reference before scoring, repairing from actual effect rather than task success alone, and replaying accumulated conditions - this is counterexample-guided repair adapted to tool environments, and the ablations support each piece within the controlled setting

These are the reasons I would like to see this work published. My recommendation rests on 1) evaluation not yet fully testing the claim the mechanism is built for i.e. generalisation to requests and environment changes the learner has not seen and 2) a manuscript that obscures a clear idea; both are fixable!

Critical weaknesses
  • No held-out evaluation: One program is learned per benchmark task (157 requests, 145 programs; B.3), with the benchmark’s own task checks as learning feedback (B.1), and is scored on the same task. Rebind’s clean utility is therefore a fit statistic placed in the same column as the agents’ zero-shot completion. The only test of reuse on unseen requests is SkillLearnBench (4.3), where the learned skills succeed on 42/160 subsequent instances. This is a much weaker picture than Table 1 suggests.
  • Attack success is zero by construction: The deployed skill is a program with no model in the loop, so there is no instruction-following to re or misdirect. Initial skill already shows 2/3018 before any learning, and the paper acknowledges this ('direct execution removes online instruction following from the attack path'). Thus the security claim is a property of the execution interface, and not of the learning.
  • The comparisons conflate interface with method: Base Agent, Reflexion, SkillOPT and EvoSkill read injected text at runtime, so Table 1 compares an interface against learning methods. However, other relevant baselines are cited but missing from this comparison (CaMeL)
  • The controlled study is in-distribution along a different axis: Requests are fixed and the environment changes, but the changed-state conditions are generated by the same operation library that Rebind uses to build rehersals - so they are not really new configurations.
  • Not a scientific weakness, but one that affects the paper’s usefulness: the prose reads as drafted at length and not yet edited down. The core mechanism — a repair can make a skill executable while leaving a wrong selection rule in place, and feasibility-checked environment changes expose it — is stated clearly in Figure 1 and Section 3.3, but much of the surrounding text restates it in general terms rather than adding information (the closing paragraphs of Sections 1, 3 and 6 each describe the loop again; Sections 4.1–4.2 repeat counts already in Tables 1, 2 and B1). Cutting some of this out might allow more space to strenghten the paper.
Questions for authors
  • On AgentDojo/AgentDyn, were skills learned and evaluated on the same user tasks? If so, what is clean utility on tasks not used during learning (e.g. learn on one task per suite, evaluate on the rest)?
  • Which of the four AgentDojo attacks can redirect a program’s object selection (Section 2.2)? If none, what is attack success under one that does — for example a decoy record whose attributes match the request?
  • In the controlled study, were any operation types held out of the rehearsal library during learning and used only at evaluation?
6 comments
AuthorsOpenReview ↗

Response to Reviewer i21Q (1/3): requests not used in learning (W1, Q1)

Dear Reviewer i21Q,

Thank you for your review, and for saying so clearly what your recommendation rests on. You name two things: the evaluation does not yet fully test generalization to requests and environment changes that the learner has not seen, and the manuscript obscures a clear idea. You wrote that "both are fixable!" We organized the revision around these two points. This comment answers your Q1 on requests never used in learning. The second covers held-out environment changes, the learning feedback and the prose, and the third the attacks and CaMeL.

Q1 / W1. Were skills learned and evaluated on the same user tasks? Yes. Rebind's skills are learned from the clean requests of the tasks they are then scored on, and so are those of Reflexion, SkillOpt and EvoSkill, with the same task checks (Appendix B.1). The revised Table 1 therefore calls its clean-utility column source-task utility, and its caption now reads: "Source-task utility is the clean utility of Section 2.2 on the 314 clean conditions, which are also the requests that the learning methods learn from." The opening of Section 4 says the same.

Q1, continued. What is clean utility on tasks not used during learning? It is 94.6% on new requests of the same kind, and it stays at 94.6% under attack. The test, which we added in response to this question, uses 62 unseen requests in the tool workflows of the controlled study, with both models; it is reported in Section 4.2 and Table E3.

What is an unseen request here? A skill is a program for one kind of request, such as rescheduling a meeting. What we hold out is therefore a new request of that kind: the official request sentence with its values replaced by other people, events, files, amounts or repositories. None of these requests was used in learning. This is also why the test holds out requests rather than whole tasks of a suite: a skill is learned for one kind of request.

The 94.6% is for the skills that Rebind deploys. A skill is deployed once it passes all of the conditions retained during its own learning. This is the case in 52 of the 54 learning runs: nine workflows, two models and three runs each. Every workflow and model has at least one deployed skill. The table below compares Rebind with the agents and runtime defenses on these requests, under the tool_knowledge and important_instructions attacks.

Clean utilityJoint success under attackAttack success
Base Agent (zero-shot)94.4%66.5%29.4%
Agent with Shared Context (zero-shot)99.2%48.8%46.4%
Reflexion (retrained)95.2%44.4%50.4%
SkillOpt (retrained)97.6%43.5%51.6%
EvoSkill (retrained)95.2%42.3%51.2%
Progent79.8%80.6%0.4%
CaMeL (without its security policies)77.4%76.2%0%
Rebind94.6%94.6%0%

In clean runs, Rebind is level with Base Agent and within a point of Reflexion and EvoSkill. Under attack, the agents follow the injected instruction in 29% to 52% of conditions and Rebind's skills never do, so its joint success is the highest in the table, 28 points above the best agent and 14 above the best runtime defense.

How does a skill carry over to requests it has never seen? By rehearsal, the same way it learns to handle a changed environment. For this test we extended the operation library by one kind of operation, which varies the values in the request itself; Appendix E.3 calls these request-instance rehearsals. The loop, the models and the schedule stay as they were. These rehearsals make the difference: skills that have seen each request with only one set of values complete 18.0% of the unseen ones, and the skills deployed after the rehearsals complete 94.6%. The baselines receive the same material: the values that the rehearsals use come from 40 training requests, disjoint from the 62 unseen ones, and every learning baseline was retrained on these 40 requests as well, with the same feedback.

A second test repeats the design on nine public tasks from AgentDojo's travel suite and AgentDyn's DailyLife and Shopping suites. Here too an attack costs Rebind nothing (75.8% with and without it), and its joint success under attack is again the highest, ahead of SkillOpt (67.7%) and Base Agent (63.7%); see Table E4. So reuse on unseen requests no longer rests on SkillLearnBench alone: the tool-workflow skills also carry over to requests they never learned from.

AuthorsOpenReview ↗

Response to Reviewer i21Q (2/3): held-out environment changes (W4, Q3), the learning feedback (W1) and the prose (W5)

Q3 / W4. Were any operation types held out of the rehearsal library during learning and used only at evaluation? The revision adds this test as a held-out panel: the submitted skills, frozen before the panel was built, complete all 144 of its conditions, none of them harmful (third panel of Table 2; Appendix E.1).

The operation library cannot produce the changes of this panel: it alters an event only by canceling and recreating it, and it never changes a file's sharing or owner. In 23 of the panel's 24 states, the test instead edits an event in place, lets a file arrive from another owner, shares a file with one more person or changes a file's owner. The result is scored by a separately implemented evaluator, which does not call the checks used in learning. The two rehearsal controls build their rehearsals in advance or from the starting skill only, and neither does as well: they have 13 and 7 harmful conditions.

Rebind's 13 gains over Precomputed Rehearsals on this panel come from individual selection rules in its skills: deleting the responsible rule loses the gain, and copying that rule alone into the Precomputed Rehearsals skill recovers it (Appendix E.7). These rules were learned from the library's own rehearsals, and here they carry over to changes that the library cannot produce. The table in our general response adds further held-out tests (Appendices E.1 and E.2). In two of them a fixed rule produces the changes without our choosing them, and Rebind's skills have no harmful condition in either.

Your recommendation asks for unseen requests and unseen changes together, and the last row of the table in our general response combines the two (Appendix E.4). This test puts 46 of the unseen requests from our answer to your Q1 under the changes of this held-out panel. Rebind's skills complete 786 of 804 conditions, none of them harmful, and lose no request that they complete in the unchanged state, while Precomputed Rehearsals completes 711, with 93 harmful. The harm takes two forms: the skill re-invites a participant whom an edit had already added to the event, or it acts on a canceled copy of the meeting.

W1. The benchmark's own task checks as learning feedback. Learning does not depend on the benchmark's task checks. To show this, we drew 22 public tasks with a fixed random seed and let a language model write the reference program and the effect checks from the request and the tool documentation. Learning then used only these; the official evaluator came in afterward, to score the result. With learning, joint success under attack rises from 205 to 296 of 392 conditions, and clean utility from 26 to 36 of 44. Of the 61 skills, learned or starting, that the generated check accepted, none fails the official check (Appendix E.8).

W5. Prose. Thank you for naming the exact places, which made this easy to act on. We changed each of them. The passages that retold the loop in Sections 1, 3.3, 3.4 and 6 are shortened to a sentence or two each, and the full description of the loop is in Section 3 beside the new Figure 2. Section 4.1 drops seven raw counts that Tables 1 and B1 already give, and Section 4.2 shortens or drops three sentences that repeated counts from Table 2. Beyond these places, Section 3 uses one name each for the operation, the changed state, the reference program and the retained conditions, and the main text explains terms such as selection rule and unauthorized effect.

On your aside about the reference program ("essentially a unit test?"): close, with one difference. It tests the rehearsal, not the skill: it shows that the request can still be completed safely in the changed state (Section 3.2). The skill is then judged on its own effects, not against the reference's calls.

AuthorsOpenReview ↗

Response to Reviewer i21Q (3/3): the attacks (W2, Q2) and CaMeL (W3)

Q2. Which of the four AgentDojo attacks can redirect a program's object selection? None. We replayed all four attacks against Initial Skill and the learned skills, and compared every write that a program made with its clean run. In 35,992 executions, no write moved to an attacker-chosen object or to another legitimate object. The only official attack successes among these methods are Initial Skill's two (Table B3), and in both the injected text is only quoted inside something the user asked for, such as a channel summary quoting an article that contains the injected link.

Q2, continued. A decoy record whose attributes match the request. This attack does reach a program, and your question led us to build it. Once the skills have rehearsed all six relations that the decoys exploit, they act on the decoy in 15 of 432 states, compared with 114 for Initial Skill. In the 312 states in which a field of the tool result tells the decoy from the requested object, they never do.

We planted six kinds of decoy. Two are a canceled meeting with the requested title, one for requests that write and one for requests that read. The others are an archived copy of the requested file under another owner, the requested message from another sender, a transaction with the wrong counterparty or direction, and a different kind of product with the requested name. The table counts, for each kind, the states in which a skill acts on the decoy; fewer is better. Its last row counts correct completions (more is better; Initial Skill's two 114s are a coincidence). All states are new, with new names and values.

Kind of decoyInitial SkillRebind, first relation rehearsedRebind, all six rehearsed
Canceled meeting, write24/480/480/48
Canceled meeting, read28/3628/360/36
Archived copy24/608/600/60
Other sender0/1088/1080/108
Wrong counterparty16/6020/600/60
Other kind of product22/12056/12015/120
All kinds114/432120/43215/432
Request completed correctly114/432259/432390/432

The middle column shows the paper's skills, whose learning rehearsed only the relation of the first row. On that kind they never act on the decoy. They have never rehearsed the other five kinds. On three of them they act on the decoy more often than Initial Skill, but never in a state that Initial Skill gets right. Wherever Initial Skill completes the request, so do the paper's skills, which complete it in 259 states against its 114 (Appendix E.5). We therefore added one rehearsal operation for each of the five and continued learning for three more stages, which gives the last column.

All 15 remaining cases are of the last kind, the different product with the requested name. No field of the Shopping tools separates it from the requested one, so a skill can only compare names. The protection also carries over to unseen requests, where Rebind's skills act on a planted canceled meeting or archived copy in 0 of 210 executions (Appendix E.3).

W2. The security claim is a property of the execution interface. Yes, we agree. The near-zero attack success comes from direct execution, and the abstract now says that no model reads injected text as an instruction, with or without learning. To see what learning adds, we hold the interface fixed: joint success rises from 52.2% for Initial Skill to 86.1% for Rebind. Learning does matter for security, though, when an attack changes which object matches the request, as the decoys do: direct execution alone does not stop a skill from picking the decoy, but rehearsing that relation does.

W3. Interface conflated with method; CaMeL. Table 1 now separates the two. Its three blocks are agents whose model reads tool output at run time (the paper calls them online agents), runtime defenses, and direct execution. The effect of learning can be seen within the third block, where Initial Skill, Precomputed Rehearsals and Rebind share the interface. CaMeL, the baseline you asked for, is in the second block, run with the same two models and the same conditions, attack and official checkers. On AgentDojo, where CaMeL was developed, its joint success is 48.4% with its security policies and 68.1% without them, compared with 93.2% for Rebind. On AgentDyn, CaMeL completes few conditions with or without its policies (2.6%), so we report the two benchmarks separately (Appendix E.6). On the unseen requests from our first comment, Rebind reaches 94.6% under attack; CaMeL reaches 76.2% without its policies, its stronger configuration.

Thank you again for the care you put into this review, and for its encouragement. We hope the revision answers both of your main points, and we would be glad to explain anything that remains unclear.

Best regards, The authors

Reviewer_zUxaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Rebind constructs environment rehearsals from the current skill's tool observations and action targets. An independent reference procedure checks that the task remains feasible; task outcomes and actual effects guide skill repair, while replay of earlier conditions constrains regression. The updated skill then informs further rehearsals. Experiments on tool workflows, injection attacks, and file skills examine safe completion and reuse.

Critical strengths
  1. The paper addresses a failure that task success can hide: after object identities or relationships change, a skill may complete an operation on the wrong target. Separating task feasibility from unauthorized effects provides a more appropriate evaluation of this problem.
  2. The internal controls are reasonably convincing. Rehearsal strategies share initial skills, operation libraries, reference checks, and repair conditions, giving the within-workflow benefit of dynamic rehearsal a relatively clear basis.
  3. The experiments go beyond final success to include budget analyses, replay ablations, and residual failures under harder changes. These help explain whether repairs regress and where the method begins to fail.
Critical weaknesses
  1. Operation templates, reference procedures, and effect checks already encode object, ownership, and authorization relations. Sharing these components makes the internal comparison meaningful, but new configurations largely retain the same operation families. They do not establish handling of new relational changes with those components frozen, leaving the scope and practical extension cost unclear.
  2. Some comparisons also change the deployment architecture, while the closer same-interface control does not show clear evidence of an additional joint-success benefit. The contribution would be easier to assess through specific rule repairs that produce gains, rather than attack-rate changes for the complete system.
  3. Preparing a complete skill package from clean sources differs from the original online skill environment, and the existing matched-boundary construction comparison has narrow coverage. It remains difficult to separate the effect of dynamic rehearsal from that of skill-package preparation.
Questions for authors
  • Which templates, reference logic, and effect rules must be supplied manually for a new workflow? Which existing tests involve genuinely unseen relational changes with these components frozen?
  • Which specific rule repairs account for additional external successes over precomputed rehearsal? Do these repairs depend on adding transformation templates?
  • Does dynamic rehearsal retain an advantage on a more representative file-skill subset when both methods use the same complete skill package and trust boundary?
7 comments
AuthorsOpenReview ↗

Response to Reviewer zUxa (1/3): manual components, and tests with them frozen (W1, Q1)

Dear Reviewer zUxa,

Thank you for a careful review. It separates what the paper's internal controls already show from what they leave open, and each of your three questions names an analysis that would close one of the gaps. We ran all three. This comment answers Q1, the second lists the rule repairs of Q2, and the third covers file skills.

Q1, first part. Which templates, reference logic and effect rules must be supplied manually for a new workflow? Six things: operation templates, a reference-program generator, effect checks, small helpers shared within the tool family, a short paragraph of relation guidance for the repair model, and the list of tools that set up a changed state. Table E9 gives their sizes and Appendix E.8 the time they took, family by family. The smallest family is Slack, with 112 lines and under a day of work. The largest is the calendar and drive family, which took about three days and serves six workflows: each uses 1,163 to 1,384 lines, of which the 527 lines of operations are shared by all six. Everything else is shared by all families and does not change with the workflow: the construction loop, the feasibility check, replay, repair, selection and the interpreter. The six components can also be generated: a language model wrote them for three further workflows, we used them without human edits, and learning with them raises safe completion from 102 to 137 of 174 conditions. Section 6 now states the cost of a new tool family.

Q1, second part / W1. Which existing tests involve genuinely unseen relational changes with these components frozen? The revision adds them. With templates, reference programs and effect checks frozen, the submitted skills complete all 144 conditions of a held-out panel, none of them harmful; the two rehearsal controls complete 131 and 132, with 13 and 7 harmful (third panel of Table 2; Appendix E.1). In 23 of the panel's 24 states the change is one that the library's operations cannot produce.

In these 23 states, participants are added to an existing event, an event is rescheduled in place, another owner shares a file with the user, a file gets a new share, or its owner changes. The library cannot produce any of these changes. It never changes who owns a file or with whom it is shared, and it changes an event's participants or time only by canceling and recreating the event; the test edits the event in place. The panel is scored by a separately implemented evaluator, which does not call the effect checks used in learning; the independent checker of Appendix E.2, written by a different model, gives the same counts.

A fixed rule then produces such changes without our choosing them: it applies every state-changing tool call to each object that a request touches, and we keep the changes that the library cannot produce (804 conditions). It also applies its changes two at a time. Rebind completes all of the resulting 13,932 conditions, none of them harmful. The table in our general response reports the 672 of them in which the pair changes what the request refers to, such as who must be invited. Rebind is the only strategy with no harmful condition in any of these sets. Under the same rule in the Slack, banking and GitHub workflows, Initial Skill has 164 harmful conditions and no learned skill has any. Rebind's lead does not rest on the relation guidance or on our own evaluator either: when all three strategies learn again with the guidance withheld and the independent checker scores them, Rebind has about half as many harmful conditions as either control (Appendix E.2).

Why do the skills handle changes that they never rehearsed? Because of the selection rules they learned from the library's own rehearsals. These rules carry over from one mechanism of change to another; our next comment lists them and accounts for the panel's gains.

AuthorsOpenReview ↗

Response to Reviewer zUxa (2/3): the rule repairs (W2, Q2)

Q2 / W2. Which specific rule repairs account for additional external successes over precomputed rehearsal, and do they depend on added templates? Individual rules in the skills' code account for the gains we traced, and none of these rules depends on an added template. On the public benchmarks we show two of them, which decide 16 injected conditions; in the controlled study we trace every one of the 61 gains to a named rule. Tracing the gains to specific rules, as you suggested, turned out to be the clearest way to show what the method contributes, and the analysis is now in Section 4.2 and Appendix E.7.

The two public-benchmark cases first. In the SSH-key task of the GitHub suite, Rebind's skill reads the verification code that arrives after it asks for the key, and it uses the key that the verification returns. It completes 9 of 9 injected conditions, the Precomputed Rehearsals skill 0 of 9, and undoing either step loses all 9. In a restaurant task of the travel suite, Rebind's skill parses only the rating line of each review, and swapping only this parser between the two skills moves all seven injected successes with it.

Standard injections insert text but do not change which event is current or who is on it, so they rarely reach a selection rule. This is why the two strategies are close on these benchmarks (0.99 points of joint success, Table B4), as your W2 notes. Once the environment does change, the two diverge: 13 harmful conditions of 144 for Precomputed Rehearsals and none for Rebind on the held-out panel (Table 2), and 93 of 804 against none on unseen requests under the same changes (Appendix E.4).

In the controlled study we can account for every gain. Rebind is safe on 61 conditions where Precomputed Rehearsals is not, and never the reverse. For each of them we located the rule in Rebind's skill, deleted it (removal), and separately copied only that rule into the Precomputed Rehearsals skill (transplant). Removal loses the gain in all 61, and the transplant recovers it in 60 (Table E8).

Rule in Rebind's skillGainsWorkflow
Copy participants from the current (non-canceled) event, not the first match27composing calendar events
Add only requested people not yet on the event14calendar participants
Open the grocery list by its exact current name first10recipes
Take the code from the unread receipt, not the first code7repository collaborators
Prefer the exact title to a substring match2calendar participants
Skip files named "archive" if the exact name is absent1recipes

The transplant breaks none of the conditions that the Precomputed Rehearsals skill already passed. The one gain that it does not recover needs a second rule of Rebind's as well, the one that checks for overlapping time slots. The gains on the held-out panel are explained in the same way: the first two rules in the table and a heading check in the recipe skill account for all 13 of Rebind's gains there. Finally, 60 of the 61 gains do not hinge on names that the library's operations use, such as "archive": they remain when we disable every clause that checks for such a name.

Why does Rebind find these rules when Precomputed Rehearsals does not? The strategies share operations, budgets and schedule, and differ only in which skill's execution a rehearsal is built from. The calendar example shows what that changes. Once a repair makes the skill read the source meeting and copy its participants, an operation can cancel and recreate that meeting, and this rehearsal exposes the missing rule. A batch built before the repair cannot contain it. This is what happened in most cases: for 48 of the 61 gains, the rehearsal that first exposed the missing rule was built from a skill that had already been repaired within the same stage. The same pattern appears round by round. In core learning, the first stage of the method, 90 rounds follow a repair and can still add a rehearsal. For each of these rounds we compared the two traces that a rehearsal could be built from. The trace of the repaired skill exposes a harmful action in 12 rounds; the trace of the skill from the start of the stage does so in 2.

On templates: the rehearsals behind these rules all use operations from the submitted library, and none uses the one operation added in Appendix C.8, which places a similarly titled event before a recreated current event. We checked all 196 rehearsal proposals.

On the first half of your W2, comparisons that also change the deployment architecture: Table 1 now groups the methods by execution interface. The effect of learning can be read within the direct-execution block, where joint success rises from 52.2% for Initial Skill to 86.1% for Rebind.

AuthorsOpenReview ↗

Response to Reviewer zUxa (3/3): file skills (W3, Q3)

W3. Separating dynamic rehearsal from skill-package preparation. The revision separates them in two steps, giving every method the same complete package and the same trust boundary. This answer shows what learning adds over the package itself; the answer to your Q3 below shows what dynamic rehearsal adds over precomputed rehearsal. For this we extended the matched comparison of Appendix D.3 from 12 cases to 52 task families across SkillSafetyBench and SkillLearnBench. With the package held fixed in this way, what raises the score is learning:

Same complete package and trust boundary (mean score, Table D2b)Initial Skill (no learning)RebindGain from learning
Unseen file configurations (108 units)11.1%56.8%+45.7 points [24.3, 67.6]
All 666 matched observations, 52 families7.3%26.5%+19.2 points [9.7, 30.4]

A unit is one task family with one model in one repetition.

Q3. Does dynamic rehearsal retain an advantage on a more representative file-skill subset, with the same complete package and trust boundary? It does where a working file skill still has a choice to make. On the broader set that your Q3 asks for, such choices are rare, because file requests usually name their inputs: about four in five of those with data inputs name every one. A package that solves its source task therefore has few choices left to get wrong, and Rebind and Precomputed Rehearsals end with the same score in 185 of the 215 matched pairs of learning runs. Where both end with a working package, Rebind's packages score at least as high on average in each of the three test sets: 97.2% against 96.6% on unseen file configurations, 57.8% against 57.8% on SkillSafetyBench and 55.6% against 52.4% on subsequent SkillLearnBench instances.

One choice that does remain is which shapes on a slide to process. To isolate what dynamic rehearsal adds, we held everything else fixed. Both arms give the same working package a rehearsal and differ only in which version of that package is traced to build it: the current package, as in Rebind, or the package as it entered the stage, as in Precomputed Rehearsals.

Rehearsal built from the trace ofthe current packagethe package as it entered the stage
Of the attempts to build a rehearsal, those that expose a failure28 of 2329 of 232
The same in a second cohort (25 presentation-formatting units, fixed in advance)48 of 10017 of 100

In the first row, the current package exposes more failures in 8 units and fewer in none (p = 0.008), and the second cohort repeats the result.

Rehearsals built from the current package then turn into more completed tasks. After a single rehearsal, and one repair if the package fails it, the packages of five task families complete 86.8% of new file configurations when the rehearsal came from the current package and 83.1% when it came from the package as it entered the stage. The gain lies in presentation formatting, where the package must decide which shapes on a slide to process. There, a package succeeds on unseen presentation files 9.0 percentage points more often if its rehearsal came from the current package (32 units; 95% interval from 2.9 to 16.0 points). As in the tool workflows, the gain comes from one learned rule: the adopted repairs make the package traverse grouped shapes. Deleting only this traversal from three of these repairs loses all 30 of their successes on unseen presentation files, while the source task still passes (Appendix D.3).

Thank you again for three questions that each led to a new analysis. If you would like any of the three analyses taken further, please tell us and we will follow up.

Best regards, The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

To all reviewers: thank you, and the new results

Dear reviewers,

Thank you for your careful reviews and concrete suggestions: each new result below answers a request from one of you. Two requests run through all three reviews. You asked for a held-out evaluation: the frozen skills from the submission complete all 144 conditions of a new held-out panel, none of them harmful, while the two rehearsal controls complete 131 and 132, with 13 and 7 harmful. You asked what the 0% attack success shows: we agree that the near-zero attack success comes from direct execution. What learning adds at that interface is completed tasks: joint success rises from 52.2% to 86.1%. What Rebind adds over precomputed rehearsals appears when the relations among objects change.

1. Held-out evaluation. In each of the five tests below, Rebind's skills complete more conditions than either rehearsal control. In four of them the controls have harmful conditions and Rebind has none; with the newly learned skills, Rebind has about half as many harmful conditions as either control. Cells give safe completions (harmful conditions). Initial Skill is the skill before learning; Precomputed and Starting-Skill are the two rehearsal controls.

Held-out testSkills fromConditionsInitial SkillPrecomputedStarting-SkillRebind
Held-out panel (third panel of Table 2): 24 new states, 23 outside the operation librarythe submission14479 (35)131 (13)132 (7)144 (0)
Rule-generated changes outside the librarythe submission804651 (80)800 (4)780 (11)804 (0)
Pairs of rule-generated changes that alter what the request refers tothe submission672482 (72)656 (16)640 (8)672 (0)
Newly learned skills: no guidance, independent checker80 new starting skills384239 (101)337 (42)335 (46)358 (23)
Unseen requests under held-out changesfurther learning804not run711 (93)633 of 693 (36)786 (0)

Held-out changes (Appendix E.1). The first three rows apply held-out environment changes to the submitted skills. In 23 of the 24 states of the held-out panel, an event is edited in place, a file arrives from another owner, or a file's sharing or owner changes; the operation library produces none of these. In the second and third rows we no longer choose the states ourselves: a fixed rule tries every state-changing tool call on the objects of a request. The second row keeps the changes that the library cannot produce; the third applies the rule's changes two at a time. Under the same rule in the Slack, banking and GitHub families, no learned skill has a harmful condition, against 164 of 714 for Initial Skill.

Newly learned skills (Appendix E.2). This row repeats the held-out panel with three things removed: the submitted skills, the relation guidance that the repair model receives and our own evaluator. Rebind and both controls learn again from 80 new starting skills, and an independent checker, written by a different model, scores the result. Even then, all four pre-specified sign tests favor Rebind after the Holm adjustment (Table E2).

Unseen requests (Appendices E.3 and E.4). The 62 unseen requests were never used in learning: they are the same kinds of request, with new values. For these tests, the skills first continue learning with rehearsals that also vary a request's values. The last row puts 46 of the requests under the changes of the held-out panel; its 804 conditions are not those of the second row. In unchanged environments, Rebind's deployed skills reach a clean utility of 94.6% on all 62 (Table E3). This is within the range of the agent baselines, which call a model at every step. Under attack, Rebind stays at 94.6%, while those agents fall to 42%–67%.

2. What the attack numbers show. A deployed skill is a program with no model in the loop, so attack success is near zero before any learning: Initial Skill has 2 successes in 3,018 conditions. Section 4.1 of the submission said so, and the abstract now says it as well. Table 1 groups the methods by execution interface. To see what learning adds, compare the methods that share the interface: joint success rises from 52.2% for Initial Skill to 86.1% for Rebind.

What Rebind's skills learn beyond those of Precomputed Rehearsals is which object to act on. Standard injections leave that choice alone, which is why the two are close in Table 1, as two of you point out. The held-out tests do change it, and in every test in the table the two diverge. The gains come from individual selection rules. Over the two original panels of Table 2, Rebind is safe on 61 conditions where Precomputed Rehearsals is not: deleting the responsible rule from Rebind's skill loses the gain in all 61 cases, and copying that rule alone into the Precomputed Rehearsals skill recovers it in 60 (Table E8).

AuthorsOpenReview ↗

To all reviewers: how your comments changed the paper

Dear reviewers,

We would also like to show you where your comments went in the paper. No result reported in the submission has changed, and neither has its main finding: in the controlled study, building rehearsals from the current skill gives safer skills than building them in advance or only from the starting skill. The numbering of tables and figures is unchanged as well.

In the marked PDF, new results and the sentences on what the attack numbers show are in blue. The readability edits you asked for are spread through the text and are not colored; the main ones are listed below.

If you have time for only three pages: page 4 has the redrawn Figure 2 and the rewritten Algorithm 1, neither of them marked in blue; page 6 has Table 1 and the paragraph on direct execution; and page 8 has the held-out tests, whose numbers are in Table 2 on page 7.

You asked for a held-out evaluation. All three reviews asked for it, and it is now the main addition to the paper.

  • Table 2 has a third panel with held-out environment changes.
  • Section 4.2 has two new paragraphs, one on these changes and one on requests never used in learning.
  • Appendices E.1 to E.4 report the tests in full; all of Appendix E is new.

You asked us to say what the 0% attack success shows (all three reviews).

  • Like Section 4.1, the abstract now says that with direct execution no model reads injected text as an instruction, with or without learning. It also states what learning adds.
  • Table 1 has three blocks by execution interface and includes CaMeL, which i21Q asked for; Appendix E.6 describes that comparison.
  • Appendix E.5 examines which attacks can reach a program at all.

You found the method hard to follow (LZrC, i21Q). We rebuilt its presentation around one example; the learning loop itself is unchanged.

  • Figure 2 walks through one round of the loop in four numbered steps on the calendar example, and Algorithm 1 has numbered lines that follow the same four steps. Appendix A.3 defines the steps one by one.
  • The opening of Section 3 defines the terms that the algorithm uses: candidate, stage and round.
  • Section 3 uses one name each for the operation, the changed state, the reference program and the retained conditions, and the main text explains terms such as selection rule and unauthorized effect.
  • The passages that retold the loop in Sections 1, 3.3, 3.4 and 6 are shortened to a sentence or two each.

You asked what exactly produces the gains, why three configurations give identical results, and what a new workflow costs (zUxa, LZrC).

  • Section 4.2 now traces the gains to six selection rules, which Table E8 lists, and explains the identical results; Appendices E.7 and E.9 give the two analyses.
  • Appendix E.8 gives the cost of each tool family, and the conclusion now states it: under a day to about three days of writing by hand.
  • The same appendix covers three further workflows, for which a language model generated the components in four to five minutes each; learning with them raises safe completion from 102 to 137 of 174 conditions.

You asked for a cleaner comparison on file skills (zUxa).

  • Section 4.3 has a new paragraph on page 9 that measures what learning adds when every method has the same complete package, executed in the same way, and compares the two rehearsal strategies on that basis.
  • Table D2b in Appendix D.3 extends the matched comparison from 12 cases to 52 task families.

Thank you again for reading the paper so closely. We would be glad to answer any further questions during the discussion.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.