- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
Agents often store successful procedures as reusable executable skills. The authors argue that a skill being runnable, or succeeding on its source tasks, does not mean its decision rules are correct. In particular, its rule for selecting which object to act on may match the correct choice in the original environment but diverge once the environment changes. This can cause unauthorized effects; for example, a calendar skill copies participants from a canceled meeting rather than its recreated replacement.
The authors propose Rebind, which learns safer skills through a rehearsal loop: 1. It perturbs the environment around the objects the current skill relies on. 2. A reference program verifies that each perturbed task remains feasible. 3. Task and effect feedback drives LLM-based code repair, with replay of all prior conditions to prevent regression. 4. New rehearsals are built from the revised skill's behavior. 5. The learned skill is then executed directly, without LLM calls at deployment.
On AgentDojo and AgentDyn, Rebind reports 86.1% utility under attack with 0% attack success, versus 78.4% and 14.4% for SkillOpt. In a controlled study over ten tool workflows, it safely completes all 462 conditions with no harmful effects, outperforming variants that precompute rehearsals or build them only from the initial skill. It also improves results on the file-skill benchmarks SkillSafetyBench and SkillLearnBench.
Critical strengths
-
I think the problem observation is interesting. The skill that agent derives from tasks it has seen might be based on the wrong scenario and might cause additional harms when applied in other tasks, this observation might be beneficial for future skill-learning.
-
The ablation supports the method's central design choice. The experiment shows that under the same conditions, building rehearsals from the revised skill outperforms building them in advance or only from the initial skill.
Critical weaknesses
-
The controlled evaluation is circular and in-distribution. Rehearsal operations, reference programs, and effect checkers are hand-written per tool family (Table A4, A.5). The repair model also receives guidance on the relevant relationships (A.6). Evaluation conditions are drawn from the same operation families and judged by the same checkers used during learning. The perfect 462/462 and 180/180 results therefore show that Rebind can satisfy the authors' own oracle within that oracle's distribution, not that it produces safer skills in general.
-
The headline security result does not show Rebind's learning effect, and I think that is a framing issue. The 0% attack success on AgentDojo/AgentDyn comes from executing skill code directly, with no LLM at deployment. The unlearned Initial Skill already has 2/3018 attack successes (0.1%), and Precomputed Rehearsals also has 0%. Under the same execution interface, Rebind's gain over Precomputed Rehearsals in joint success is +0.99 points with a 95% interval of [-0.96, 3.07], which is not significant. Yet the abstract frames the result as Rebind (0%) versus SkillOpt (14.4%).
-
The paper was really hard to read and understand, especially the algorithm diagram as well as the sudo code. Moreover, I think the paper's entire writing and expression is mixed up with different concepts and words jumping back and forth
Questions for authors
- Can the authors report results on held-out evaluations of environment changes from operation families not in the learning library?
- Do the authors agree that the 0% attack success rate stems mainly from direct code execution, given Initial Skill's 0.1% and Precomputed Rehearsals' 0%? If so, will the abstract and Section 4.1 be reframed?
- Why are the results identical across all three (depth, distractor) configurations for every method, including Initial Skill and Repair-only?
Clarification:
- What is the human cost of adding a new tool family, in terms of the number of operations, the lines of code for reference programs and effect checkers, and the time required? How do we scale it up in future?
5 comments
Response to Reviewer LZrC (2/3): circular evaluation and held-out operation families (W1, Q1)
Q1 / W1. Are there results on held-out evaluations of environment changes from operation families not in the learning library? Yes. The frozen skills from the submission complete all 144 conditions of a held-out panel, none of them harmful, while the two rehearsal controls complete 131 and 132 (third panel of Table 2; Appendix E.1). In 23 of the panel's 24 states, the change is one that the operation library (Table A4) cannot produce.
Your W1 names three things that the submitted panels share with the learning process: the operation families, the checkers and the relation guidance. We designed the new tests around this list. In the table below, each test after the first row removes one or more of the three, and the last row removes all three at once. The columns "Changes," "Scored by" and "Guidance" say which. Three kinds of scorer appear: the effect checks used in learning, a separately implemented evaluator of ours that does not call them, and an independent checker written by a different model. Cells give safe completions, with harmful conditions in parentheses.
| Test (conditions) | Changes | Scored by | Guidance | Precomputed | Starting-Skill | Rebind |
|---|---|---|---|---|---|---|
| Table 2, first panel (462) | from the library | effect checks used in learning | yes | 425 (33) | 432 (21) | 462 (0) |
| Held-out panel (144) | outside the library (23 of 24 states) | separately implemented evaluator | yes | 131 (13) | 132 (7) | 144 (0) |
| Rule-generated changes (804) | outside the library | evaluator and independent checker | yes | 800 (4) | 780 (11) | 804 (0) |
| Pairs of rule-generated changes (672) | those that alter what the request refers to | evaluator and independent checker | yes | 656 (16) | 640 (8) | 672 (0) |
| Newly learned skills (384) | the 24 states of the held-out panel | independent checker | no | 337 (42) | 335 (46) | 358 (23) |
The held-out changes use mechanisms outside Table A4: an event edited in place, a file arriving from another owner, a new share, a change of owner. The library, by contrast, alters an event only by canceling it and recreating it under a new identifier, and it never changes a file's sharing or owner. The skills were frozen before the panel was built.
In the next two rows we no longer choose the states ourselves. A fixed rule applies every state-changing tool call to each object the request refers to, and to one object the request does not involve; the call arguments come from a grid fixed in advance. The row of rule-generated changes contains only those that the library's operations cannot produce. The rule's changes are then applied two at a time, which gives 13,932 conditions, and Rebind's skills complete every one of them, none harmful. The row of pairs reports the 672 of these conditions in which the pair changes what the request refers to, such as who must be invited. Rebind is the only method with no harmful condition in these two rows. When the same rule is applied to the Slack, banking and GitHub workflows (the other tool families), Initial Skill has 164 harmful conditions and no learned skill has any (714 conditions, scored by each family's effect checks; a checker that a different model wrote for these families gives the same safe counts).
Our evaluator and the independent checker agree. Rescored with the independent checker of Appendix E.2, the held-out panel gives the same counts, and on the rule-generated changes and their pairs the two give the same verdict on every execution.
The test with newly learned skills also removes the other two things your W1 lists: learning runs without the relation guidance, and scoring uses neither our evaluator nor our effect checks. Rebind and both controls learn again from 80 new starting skills, using core learning only (the first stage of the method). The independent checker was written by a different model, GLM-5.3, from a plain-language description of the requests; that description was drafted by a separate agent with no access to our evaluator, our effect checks or our results. The checker was frozen before any learned skill was scored.
Even so, Rebind completes more conditions safely than either control and has about half as many harmful ones (last row of the table). All four pre-specified comparisons, safe rate and harmful rate against each control, stay below 0.05 after the Holm adjustment (exact sign tests over the 80 starting skills; Table E2). So Rebind's advantage holds on changes that the library cannot produce, under an independent checker, and without the relation guidance.
Response to Reviewer LZrC (3/3): readability (W3), and the cost of a new tool family
W3. Hard to read, especially the algorithm diagram and the pseudocode; concepts and words jump back and forth. We agree, and we apologize for the extra effort. We rebuilt all three: the diagram, the pseudocode and the wording. The new diagram and the new pseudocode are on page 4 of the revised PDF.
Figure 2 now shows one round of the loop as four numbered steps on the calendar example: (1) run and trace, (2) build a rehearsal, (3) evaluate, (4) repair and replay. In the example, the skill first picks the canceled meeting and, after the repair, the confirmed one.
Algorithm 1 now has 17 numbered lines in two labeled parts, core learning and refinement. Its four steps are numbered as in Figure 2 and in the opening of Section 3. That opening also defines the terms that the algorithm uses: a candidate is a version of the skill considered during learning; learning proceeds in stages, core learning first and then refinement; and each stage runs one or more rounds of the four steps. It also says what the current skill is and explains why learning starts from two candidates. Appendix A.3 defines each step, and Algorithm A1 gives one refinement stage in ten lines. The figure, the algorithm and the text now follow the same four steps, with the same numbers.
As for the wording, four terms are renamed and used consistently throughout the main text: "environment-setup program" is now "operation," "prepared environment state" is "changed state," "reference execution" is "reference program" and "evolving skill" is "current skill." A rehearsal is built from the skill's "trace," which the submission called "observations." The main text also explains terms such as selection rule, unauthorized effect and tool workflow, and Section 3.4 states in one sentence how a candidate is selected. The passages that walked through the loop again in Sections 1, 3.3, 3.4 and 6 are cut to a sentence or two each. If a passage is still hard to follow, we would be glad to hear which one.
Clarification: human cost of a new tool family. Slack, banking and GitHub each took less than a day, and the calendar and drive family, which serves six workflows, about three days. The table gives the quantities you ask for, family by family, with lines counted per workflow. Table E9 has the remaining components, and Appendix E.8 the times.
| Tool family (workflows) | Operations | Reference program (lines) | Effect checks (lines) | All components (lines) | Time by hand |
|---|---|---|---|---|---|
| Slack (1) | 1 | 6 | 56 | 112 | under a day |
| Banking (1) | 1 | 16 | 139 | 264 | under a day |
| GitHub (2) | 1 each | 15 | 173 | 402–404 | under a day |
| Calendar and drive (6, shared) | 19 | 169–185 | 351–465 | 1,163–1,384 | about three days |
None of this is repeated for each skill: the components are written once, with code shared within a tool family. Everything else (3,451 lines) is written once and shared by all families.
How do we scale it up? Our answer is to generate the per-family part automatically, and we have done so for three workflows beyond the paper's ten. A language model wrote every component from the tool documentation, the request and a generic description of the interface. In all three cases the first version passed our automatic checks, and nobody edited it. The generation call for a workflow took four to five minutes and cost USD 0.57 to 1.07. Learning with these generated components raises safe completion from 102 (Initial Skill) to 137 of 174 conditions, on evaluation states and checks written by a different model (Appendix E.8). This also bears on your W1: on 22 further public tasks, a language model, not the authors, wrote the reference programs and effect checks, and the benchmarks' official evaluators scored the result. Learning still raised joint success under attack from 205 to 296 of 392 conditions.
Thank you again for the review. We hope that the new page 4 and the held-out evaluation address your concerns, and we would be happy to discuss any point further.
Best regards, The authors
Response to Reviewer LZrC (1/3): the security claim (W2, Q2) and the identical configurations (Q3)
Dear Reviewer LZrC,
Thank you for your review. Each weakness you list pointed to something we could test or change, and we have acted on all three. We start with your Q2, because its answer separates what direct execution does from what learning adds. The held-out evaluation that your Q1 asks for is in our second comment, and the third covers the diagram, the pseudocode and the wording.
Q2 / W2. Does the 0% attack success stem mainly from direct code execution, and will the abstract and Section 4.1 be reframed? Yes to both. A deployed skill is a program, and no model reads tool output while it runs, so an injected instruction has no one to instruct. This holds before any learning: Initial Skill has only 2 attack successes in 3,018 conditions (Table B3). Section 4.1 of the submission said so; the abstract now says it as well, and ends:
Section 4.1 no longer compares Rebind's zero with SkillOpt's and Progent's attack counts, and its paragraph on direct execution says that the near-zero attack success is present before any learning. Table 1 has three blocks by execution interface, so that the effect of learning can be read as the difference between Initial Skill and Rebind, both in the last block. With clean and injected conditions counted together, Table B4 gives this difference as 34.09 points of joint success [27.33, 41.13].
Rebind's lead over Precomputed Rehearsals on these benchmarks is 0.99 points, as you note, and the reason lies in the attacks: they insert text. They do not change which meeting is current or who is on it, so they rarely reach the selection rules, by which a skill decides which object to act on. It is in these rules that the two differ. When those relations do change, the two diverge and the intervals exclude zero:
These gains come from individual selection rules. Over the two original panels of Table 2 (462 and 180 conditions), Rebind is safe on 61 conditions where Precomputed Rehearsals is not. Deleting the responsible rule from Rebind's skill loses the gain in all 61 cases, and copying that rule alone into the Precomputed Rehearsals skill recovers it in 60 (Table E8).
Q3. Why are the results identical across the three (depth, distractor) configurations? Because the configurations vary two quantities that no selection rule depends on, as Section 4.2 now explains. Depth is the number of superseded versions created before the current object, and the distractor count is the number of alternative records created after it (Appendix C.7). A skill that takes the first match picks a superseded version whether there are 2, 4 or 7 of them, so it fails in all three configurations. A skill that takes the current confirmed object picks it however many distractors follow, and is safe in all three. Each skill therefore gets the same verdict in every configuration: all 60 of Rebind's skills are safe, compared with 52 and 55 of 60 for the two controls (Table C10). Appendix E.9 confirms this skill by skill for all five methods, including Initial Skill and Repair only: of 60 skills each, 13 and 23 are safe in all three configurations, and the rest in none.