- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper observes that models are not as good at using skills as they are at using tools.
They offload skill workflow steps into neuro-symbolic functions that the model can call.
These functions are separate model calls to another LLM, which the original agent has to parametrise.
They find that this results in improvements over long horizon tasks.
Critical strengths
The paper's observations on model utilizing functions/tool calling better than skills is useful. The idea of offloading intermediate steps onto a separate LLM is also good, but it is well studied.
Critical weaknesses
-
No Comparison Against multi agent system The solution revolves around calling another llm with some parametrised focussed information, this is also how coding agents etc do handoffs, but the handoff entity is also another agent in its own right, which is more powerful. Should be baselined against. Otherwise this technique is outdated. The only difference I see is that the function signatures are defined offline while in agent handoff the model does it on the fly.
-
The paper seems unfinished in many regards. I have expanded more on the following points in questions to the authors:
- Why not do the ARISE process in a loop? Get trajectories --> define subroutines --> Get trajectories with subroutines --> get higher abstraction subroutines --> till convergence. Will this converge?
- The mechanism of making labels before clustering seems arbitrary and is not justified. What is the stability of such operations. Please give examples and statistics. Are the labels obvious?
- The parametrization of text to function (llm call) is not discussed adequately. What is the size of the focus text provided in different domains? How is it capped? Won't it increase latency due to huge number of output tokens the model needs to produce for the parametrization.
Questions for authors
Lines 17–18: Chain-of-thought is the weakest baseline to headline in the abstract. Beyond AWM/ACE/ASI, prompt optimization methods (Trace2Skill, which is cited but not compared, GEPA etc.) are also useful additions, would love to know authors thoughts on this.
Lines 83–89: These sound the same to me as ARISE. The distinction (execution in a separate LLM call) is actually what agent hand-off and multi-agent orchestration systems do, which are not mentioned here.
Lines 91–95: Interesting. What I understand is that when the main agent calls a tool/subroutine, a separate single LLM call carries it out (not an agent). If that is so, a good comparison or implementation harness would be a multi-agent one with hand-offs, e.g., Claude Code. The novelty over Decomposed Prompting and meta-prompting (lines 134–137) also needs to be clearer.
Lines 187–188: Why only successful trajectories? Negative trajectories also contain useful information about what not to do, which can help guide compilation. This ablation would be good.
Line 213: What does this step mean? Filter already keeps the successful rollouts. Does Extract just split them into per-step items? Please specify.
Lines 214–215: What is the stability of such an operation? Does it always converge to the same library of labels?
Lines 221–224: It might be useful to do this in a loop, then? Sample till you get some successful trajectories, then create subroutines. Then take the trajectories arising from the subroutines and go again, till you converge. Dependence on successful trajectories needs to be studied more. Does performance improve on different task types? E.g., succeed on type A, fail on B, generate subroutines from A trajectories: do they help B? If they only help A, is this useful?
Line 230: Please expand more on this, since the executor is also an LLM. Figure 2 says the agent supplies only focus and the instance text is bound automatically. Does the main agent have to parametrize a lot of text for this? What are samples of size of text it has to give? If it is very large this is wasted output tokens, if it is low the model might be better off doing this in context without a hand-off.
Lines 230–232: If it is a single LLM call, then everything is in-context, which is quite weak. Please compare this with agent hand-off.
8 comments
Response to Reviewer Ab5q (2/4)
W1 (continued): Hand-off experiments. To further compare our method with agent hand-off for this review, we conducted additional experiments and added hand-off conditions with a tool-using delegate (see table below): the base ReAct agent gets a delegate(instruction) tool, and each call starts a sub-agent (same model, base tools, the instance input, a fresh context, up to 20 requests) that returns a report, as in coding-agent sub-agents. In a second variant, the agent is also instructed to decompose the problem and delegate. We also ran ARISE with agentic executors: each induced subroutine is executed by such a sub-agent following its specification, instead of by one LLM call.
| Gemini 2.5 Flash | murder | object | team | NBA | meeting | Avg (4 tasks w/o NBA) |
|---|---|---|---|---|---|---|
| ReAct | 78.0 | 50.9 | 76.0 | 65.2 | 71.0 | 69.0 |
| Hand-off: ReAct + delegate (tool-using sub-agent) | 74.0 | 51.9 | 69.0 | 65.2 | 65.0 | 65.0 |
| Hand-off, agent instructed to decompose and delegate | 77.0 | 58.5 | 63.0 | 69.7 | 43.0 | 60.4 |
| ARISE library, agentic executor | 85.0 | 78.3 | 68.0 | 74.2 | 75.0 | 76.6 |
| ARISE library, single-call executor (paper) | 86.0 | 75.5 | 76.0 | 71.2 | 80.0 | 79.4 |
With the ARISE library, both executors are above every hand-off variant, and the tool-using executor does not improve on the single call (76.6 vs 79.4) while costing about twice as much on murder (USD 9.43 vs 4.85 for the test split). In these runs, the gain comes from the learned library rather than from the capability of the delegate.
Q1 (lines 17–18): We appreciate this feedback and will revise the abstract to lead with the comparison to experience-adapted methods rather than CoT. That said, CoT is not a very weak baseline; it is actually the third-best-performing baseline on Gemini. We thank the reviewer for this suggestion and now add two other baselines.
GEPA and Trace2Skill:
Accuracy (%), mean of three test runs:
| murder | object | team | NBA | meeting | Avg | |
|---|---|---|---|---|---|---|
| GEPA | 84.7 | 76.1 | 70.7 | 64.6 | 81.0 | 75.4 |
| Trace2Skill | 77.7 | 75.5 | 53.3 | 67.2 | 45.3 | 63.8 |
| ARISE | 87.3 | 74.2 | 74.3 | 71.2 | 81.3 | 77.7 |
- GEPA averages 75.4 over three test runs, against 77.7 for ARISE, at about three times ARISE's total cost (USD 10.36 vs 3.63 per task).
ARISE is higher than Trace2Skill on four of five tasks and by 13.9 points on average, at a lower total cost per task (USD 3.63 vs 4.22). The closest controlled analogue in the paper is Ours as skill: the identical induced text read in context is lower than ARISE in all ten model–task settings (51.8 vs 67.4 on DeepSeek-V3, 66.1 vs 77.7 on Gemini 2.5 Flash).
Q3 (lines 134–137): Novelty over Decomposed Prompting and meta-prompting. In Decomposed Prompting, a developer writes the decomposition and each handler. In meta-prompting, the conductor invents expert instances for each query, and nothing carries over between problems. Coding-agent sub-agents are either written by the user (named, with a natural-language prompt) or described on the fly by the orchestrator. ARISE produces such named, specified procedures automatically from experience; the comparisons in W1 measure what that learning adds.
Response to Reviewer Ab5q (3/4)
W2 (also Q7): Running ARISE in a loop (lines 221–224). We ran two more rounds on Gemini 2.5 Flash: the round-r agent solves the training split, the pipeline is applied to its successful trajectories (subroutine calls become action items), and the new library is tested.
| Task | Test accuracy, rounds 1 / 2 / 3 | Training successes, rounds 1 / 2 / 3 |
|---|---|---|
| murder | 86.0 / 87.0 / 85.0 | 54 / 62 / 58 of 75 |
| object | 75.5 / 67.9 / 73.6 | 33 / 56 / 62 |
| team | 76.0 / 71.0 / 72.0 | 53 / 60 / 59 |
| NBA | 71.2 / 72.7 / 74.2 | 47 / 52 / 51 |
| meeting | 80.0 / 86.0 / 89.0 | 53 / 61 / 59 of 70 |
| Avg | 77.7 / 76.9 / 78.8 |
Iteration does not converge to a fixed library, and it does not raise average accuracy. Our reading is that a model solves a task family with a stable set of recurring procedures: correct trajectories reuse them, and this recurrence is what ARISE captures. In later rounds the agent solves more training instances (e.g., 33 → 62 on object), but the additional successes reuse the same procedures, and each round recovers procedures for the same task functions with different wording and boundaries. Specification similarity between consecutive rounds (TF-IDF cosine 0.27–0.49) is close to that between two independent inductions on the same task (0.29–0.30) and well above that between unrelated tasks (0.07–0.11). Re-capturing the procedures therefore adds little beyond one round. Continual improvement would need a different mechanism that refines what has been learned, for example optimizing the specifications with textual feedback as in TextGrad, or refining details over several adaptation epochs as ACE does, and likely a careful combination of such techniques with induced subroutines. We leave this to future work. On the dependence on successful trajectories, we also re-induced libraries from 5, 10, and 20 successful trajectories (see our response to Reviewer hvgA, Q2): on murder, 10 trajectories already match the full set, while on object and team the smaller libraries stay about 7–10 points below.
W3 (also Q6): Stability of labelling and merging (lines 214–215). Labelling items before grouping them is a standard way to induce categories from free text. In qualitative analysis, data segments are first coded and the codes are then collated into themes (Braun & Clarke, 2006); LLM topic models such as TopicGPT label each document with a topic, then merge near-duplicate topics and drop infrequent ones (Pham et al., 2024), much as our merge step and minimum support do; and AWM prompts the model to extract common sub-routines across experiences (Wang et al., 2024). Labelling first also helps the grouping itself: IDAS reports that clustering utterances by short LLM-generated labels, which keep the core content and drop non-essential detail, outperforms prior intent-discovery methods (De Raedt et al., 2023). Our step items are full of instance-specific content (suspect names, times, rule numbers), so naming the reasoning move is what makes recurrence across instances countable. In ARISE, each step item receives a short, free-form name for the reasoning move it performs (e.g., "verify alibi consistency"). Batches of 30 items are labelled by independent calls, and one merge call then maps synonymous labels onto canonical categories using their counts. On Gemini 2.5 Flash, for example, 114 murder items receive 49 distinct raw labels, which merge into 7 categories; the five most frequent become subroutines (object 13 → 5, team 22 → 6, NBA 29 → 5). Are the labels obvious? They are short verb phrases naming the move, and the merge groups near-synonyms. For example, for murder, Investigate Means And Weapon collects "Identify murder weapon" (5 items), "Identify weapon type" (4), "Check suspect weapon access" (4), and "Link suspect to weapon" (3), among 9 raw labels. They are fairly obvious. Since free-form names differ from run to run, the relevant question is whether the resulting libraries cover the same task functions. Appendix B (Table 6) addresses this: two different models, each inducing from its own trajectories, recover the same functions. For example, for object placement, both libraries track which object movements a character could observe and infer the character's belief from that (DeepSeek-V3: analyze_character_knowledge, infer_character_belief; Gemini 2.5 Flash: get_character_object_belief, formulate_character_belief).
Response to Reviewer Ab5q (1/4)
We thank the reviewer for the detailed questions. We first clarify a few points in the summary.
Relating to multi-agent hand-offs: Multi-agent systems are used for a variety of reasons: as a way of orchestrating complex agentic tasks, as a means for test-time scaling by parallelization, as a means for context management, as a means of coordinating work between stronger and weaker agents, and doubtless other reasons. While it is true that the LLM-executed subroutines are executed by something much like a multi-agent hand-off, our paper’s contribution is on learning to perform a task from experience, which is not to our knowledge a common focus for multi-agent systems. Defining the function signatures offline is precisely the mechanism ARISE uses to encode what it has learned - so that difference is critical.
Simply enabling on-the-fly ARISE-like subroutine hand-offs in a multi-agent system would not, as far as we can see, enable learning from trajectories, because nothing learned on one task instance would carry over to the next.
Since the paper asks what an agent should learn from experience, the most informative comparisons are other methods that learn from the same experience (AWM, ACE, ASI, and now GEPA and Trace2Skill) and controls that keep the learned text fixed and change only how it is used.
We understand the concern that delegation alone might lead to gains, so we added hand-off baselines with a tool-using sub-agent and ran ARISE with such sub-agents as executors. In short, hand-off does not improve on the base ReAct agent on average (65.0 and 60.4 vs 69.0), while the single-call ARISE library outperforms every hand-off variant on all five tasks: the gain comes from what is learned, not from delegation. Details are in W1 (parts 1/4 and 2/4; Appendix F, Table 8), W4 in part 4/4 (in-context controls), and Q1 in part 2/4 (GEPA, Trace2Skill). We hope these results lead the reviewer to reconsider the assessment.
Our point-by-point responses follow.
W1 (also Q2, Q3, Q9): Relation to hand-off and multi-agent systems (lines 83–95, 230–232). While this paper is not about agent hand-off and is intended as an extension of skill induction and tool induction, we will clarify this with a comparison between ARISE and agent hand-off: delegating a focused request to a separate model call is shared with hand-off and multi-agent orchestration, and we will discuss this relationship explicitly. Again, ARISE's main contribution is learning agent subroutines: the inventory of subroutines and their specifications, induced from the agent's own successful trajectories.
As the reviewer notes, in a hand-off system the orchestrator defines each sub-task on the fly. The paper already contains a single-call form of this condition: in generic tools (Appendix F, Table 8), the agent has the same number of tools and the same separate-call execution, but the tools carry no learned names or specifications, so the agent defines every delegated request at run time through focus. It averages 64.2%, below the ReAct agent (68.2%); ARISE averages 78.4%. With the mechanism held fixed, the learned definitions account for 14.2 points.