- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SkillFlow, a framework for learning multi-step LLM agent orchestration while maintaining an evolving skill library. SkillFlow jointly trains forward and backward policies using Tempered Trajectory Balance (TTB), with the goal of sampling terminal strategies in proportion to their rewards. Quantities derived from the same flow objective, including the TTB residual, forward-backward probability ratios, and skill invocation flows, are used to determine when the library should evolve, which skills should be retained or removed, and where new skills should be created.
Critical strengths
-
The problem is important and the central idea is coherent. Using a common flow signal for orchestration learning, step-level diagnosis, and skill-library evolution provides a unified perspective on an important agent-learning problem.
-
The empirical evaluation is broad. The paper evaluates 14 benchmarks and includes component ablations, OOD evaluation, transfer across executors, and replacement of the skill-creator model. The results suggest that the complete system has strong empirical performance.
Critical weaknesses
-
The experimental action space is not reconciled with the exact normalization required by TTB. TTB assumes a normalized distribution over a finite set of complete actions, , at every state. The evaluated tasks involve open-ended textual actions such as search instructions, tool arguments, and code-related operations, yet the paper does not explain how these actions are finitely enumerated, how large the candidate sets are, or how the normalizing constant is computed. It is therefore unclear whether the implemented policy corresponds to the probability model used in the theoretical analysis, and whether has the claimed flow interpretation.
-
The GRPO comparison does not clearly isolate the contribution of TTB. The paper states that GRPO replaces only the TTB objective, while SkillFlow's library evolution depends on the TTB residual, the backward policy, and . It is not explained how GRPO‡ performs the same when/where/what skill updates after TTB is removed. The comparison may therefore change both the optimization objective and the skill-evolution mechanism, preventing the observed difference from being attributed specifically to TTB.
-
The
-Backward policyablation is not sufficiently defined. TTB requires a backward distribution, but the paper does not state whether the learned is replaced by a uniform distribution, a fixed rule, or . These alternatives induce different path distributions and different training objectives. Consequently, the reported degradation cannot be interpreted directly as evidence for the benefit of step-level credit assignment. -
The diversity metric is not aligned with the paper's definition of a terminal strategy. The theory identifies different orderings of commuting actions as the same terminal strategy, whereas the experimental Div metric counts distinct step arrangements. It may therefore count multiple permutations of the same strategy as different workflows. The reported result does not establish that SkillFlow discovers a larger number of substantively distinct successful strategies.
Questions for authors
-
How is constructed for each task family? Are open-ended search instructions, tool arguments, and code-related actions fully enumerated? Please report typical candidate-set sizes and the exact computation of the forward normalizer.
-
How does GRPO‡ perform the when/where/what skill-evolution operations without a trained TTB residual and backward policy? Does it differ from SkillFlow only in the optimization objective, or also in the library-update rule?
-
What backward distribution is used in the
-Backward policyablation? Does this variant preserve the same terminal target as the full method? -
Does Div count different action orderings that belong to the same quotient terminal strategy as distinct workflows? Can diversity be reported over the equivalence classes defined by the paper?
3 comments
Response to Reviewer eV3D (2/3): The contribution of TTB, and how GRPO‡ evolves skills
W2 / Q2. The contribution of TTB, and how GRPO‡ performs when/where/what.
GRPO‡, the "− TTB" row of Table 2, measures the contribution of TTB to the whole system, covering both the objective itself and the signals it supplies to skill evolution. With the library fixed so that only the objective differs, TTB is above GRPO‡ on all 14 benchmarks (80.47 vs. 76.09 on average).
How when/where/what are performed under GRPO‡. The three flow signals are produced by TTB training: they come from the TTB residual, the backward policy and . GRPO‡ keeps the same Skill Creator (GPT-5.5), the same curation operator and the same library capacity, and GPT-5.5 decides directly from the sampled trajectories and their rewards when to evolve, where the decision gaps lie and what skills to create or prune.
A comparison that changes only the objective. To measure the objective on its own, we train both objectives on the same library: SkillFlow's final library is given from step 0 and frozen, with no skill created, refined or pruned during training. Neither run makes any when/where/what decision, so the two "Fixed final" rows differ only in the objective.
| Library | Objective | Hotpot | Trivia | AIME | MedQA | WebShop | ALFWorld | SWE | MuSiQue | NQ | MATH | GPQA | HumanEval | SciWorld | Mind2Web | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fixed final | GRPO‡ | 89.06 | 92.19 | 56.67 | 87.50 | 83.59 | 85.16 | 43.75 | 81.25 | 81.25 | 92.19 | 83.59 | 90.63 | 52.34 | 46.09 | 76.09 |
| Fixed final | TTB | 91.41 | 96.09 | 63.33 | 90.63 | 89.84 | 92.19 | 49.22 | 84.38 | 82.81 | 96.09 | 89.06 | 97.66 | 54.69 | 49.22 | 80.47 |
| Evolving | GRPO‡ (Table 2, − TTB) | 83.59 | 87.50 | 53.33 | 85.16 | 82.03 | 82.81 | 40.63 | 73.44 | 79.69 | 87.50 | 79.69 | 89.06 | 39.06 | 33.59 | 71.22 |
| Evolving | TTB (SkillFlow) | 92.19 | 96.09 | 70.00 | 92.19 | 93.75 | 96.09 | 52.34 | 85.16 | 82.81 | 96.09 | 89.84 | 98.44 | 57.81 | 51.56 | 82.45 |
The contribution of the TTB objective itself. With the same library, TTB is above GRPO‡ on all 14 benchmarks, and this gap comes from the objective alone. The reason lies in what the two objectives train toward: GRPO‡ tends to concentrate probability on one mode, whereas TTB matches the probability of each terminal strategy to its tempered reward and keeps multiple high-reward paths alive (§4.2, Fig. 5d). The same table also shows what the flow signals contribute to skill evolution: still under GRPO‡, replacing the library that GPT-5.5 evolves by its own judgment with the flow-driven one raises all 14 benchmarks. Together with the gain of co-evolution over a fixed library (the two TTB rows), these two parts make up the gap of "− TTB" in Table 2.
We have added how GRPO‡ updates its library, together with the table, to App. L.2 (Table 8), and the caption of Table 1 now points to it.
Response to Reviewer eV3D (3/3): The backward-policy ablation, and Div over equivalence classes
W3 / Q3. The backward distribution in the "− Backward policy" ablation and its terminal target.
The ablation replaces the learned by the uniform distribution over the legal parent set, and it has the same terminal target as the full method.
The backward distribution. For every child state , for , where is the legal parent set of Proposition 28. It has no trainable parameters and stays fixed during training, while the forward policy, , the form of the TTB residual and the skill evolution are unchanged. The residual becomes .
The terminal target. The uniform distribution is a positive normalized backward kernel on the same graph. For any such kernel, the path distribution targeted by TTB is (Theorem 16), whose terminal marginal is (Eq. 1, Lemma 12). The backward kernel only determines how the mass of each terminal strategy is spread over its paths, and thereby the interior flows (Lemma 18). The two variants therefore share the terminal target and differ only in the path distribution and the interior flows.
What the ablation measures. The uniform kernel splits the mass of each state equally among its parents, whereas learns this allocation from child and candidate-parent information. The step quantities and are computed from this allocation (Eqs. 7, 8), and the where and what steps of skill evolution rely on them. Since the two variants share the terminal target and differ only in this allocation, the consistent drop on all 14 benchmarks reflects the benefit of learned per-step credit.
We have added the definition and the residual above to the caption of Table 2 and App. K.4.
W4 / Q4. Div and the terminal strategies defined in the paper.
Which orderings belong to the same terminal strategy. The equivalence is generated only by swaps of adjacent independent actions (Definition 13), and the independence relation contains the pairs of distinct skill activations (Lemma 26). Pairs of act operations and pairs involving accept are dependent: they share execution state, so their observations and context order can differ between the two orders (Definition 13). Retrieving before computing and computing before retrieving are thus two different terminal strategies in the theory as well, and counting them separately is consistent with it. The only orderings the theory identifies are swaps of adjacent activations of distinct skills.
Div over equivalence classes. Div in the submitted Fig. 5(d) counts step arrangements. The new column first merges the workflows of a query that differ only by such swaps and then counts; everything else follows App. O.
| Method | Div, step arrangements (submitted Fig. 5d) | Div, equivalence classes |
|---|---|---|
| GRPO‡ | 8.75 | 7.31 |
| Tree-GRPO | 7.31 | 5.62 |
| HCAPO | 8.42 | 6.72 |
| SkillFlow | 16.79 | 13.38 |
Result. The conclusion is unchanged over equivalence classes: SkillFlow still finds 1.8× as many distinct successful strategies as the best of the three objectives, and the four methods keep their order. Merging only concerns the order of distinct skill activations and affects the four methods alike; the gap that remains after merging shows that the additional workflows of SkillFlow are substantively different strategies, not reorderings of one strategy. This agrees with the target of TTB, which matches the probability of each terminal strategy to its tempered reward and therefore keeps several high-reward strategies at once. The Pass@K gain in Fig. 5(d) does not depend on how workflows are counted and supports the same conclusion: SkillFlow gains clearly more under repeated attempts, so these distinct strategies do turn into success.
We now report Div over equivalence classes in Fig. 5(d), §5 (RQ5) and App. O (Table 9).
We thank the reviewer again. All additional experiments and revisions have been incorporated into the revised version and are marked in blue. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider your score. Thank you very much!
Response to Reviewer eV3D (1/3): The action set and the forward normalizer
We thank the reviewer for the careful review and valuable suggestions, and answer each point below.
W1 / Q1. The action set Σ(s) and the forward normalizer.
Σ(s) is the finite set of legal complete actions at a state (§3). Open-ended text is not an element of it, so both normalizers are exact sums over finite sets.
How Σ(s) is constructed. A complete action a=(α,o) has a type α∈{skill,act,accept} and parameters o (Definition 2 in §3), which name the skill to activate or the operation to dispatch. At a non-terminal state, Σ(s) contains (i) the activation of each skill retrieved for the query (at most the four best matches), (ii) each
actoperation legal at that state, i.e. one of the three to six operations the task family exposes or, in WebShop and ALFWorld, one of the admissible actions the environment returns at that state, and (iii)accept. The set is a deterministic function of the canonical state, is fixed within a phase, and holds each action once, in a unique serialization of at most Kmax tokens with an end marker (§3, App. K.3). All task families follow this rule and differ only in which operations are legal in (ii).Why open-ended text needs no enumeration. At each step, the only stochastic decision of the Supervisor is the choice in Σ(s). Once an
actoperation is chosen, its instruction text is decoded greedily, so it is a deterministic function of the canonical state and the chosen action, with unit conditional mass. It has the same status as the reasoning r=f(k)(s) (§3, App. K.2, Lemma 13) and belongs to the deterministic transition δ(k)(s,a), not to Σ(s). What the instruction produces when executed (retrieved passages, generated code, tool outputs) comes from the frozen Executor and enters the state as execution feedback through Cexec (§4.1, App. B.4). Search instructions, tool arguments and code are therefore not edges of the graph; every edge is one element of Σ(s).Typical candidate-set sizes. The table covers all non-terminal states visited in run 1.
On the non-interactive families the maximum equals the number of operations plus four skills and
accept; in the interactive environments it follows the number of admissible actions. The parent set of the backward policy is bounded by the horizon, ∣Pa(x)∣≤Tmax=12 (App. E.3).Exact computation of the forward normalizer. For each a∈Σ(s), the Supervisor computes ℓθ(a;r,s), the mean token log-likelihood of its serialization including the end marker, and then CF(s,r)=∑a′∈Σ(s)eℓθ(a′;r,s) and πθ(a∣r,s)=eℓθ(a;r,s)/CF(s,r) (Eq. 4). With at most 58 candidates this is an exact sum, with no sampling approximation. The TTB residual uses the same logπθ, with logCF kept inside Δ and differentiated (App. B.3, K.3).
Correspondence with the analysis. The action is drawn from πθ itself, and the rollout temperature of 1.0 in App. L.1 refers to this draw. The sampler and the loss therefore use the same normalized kernels on the same finite DAG, which is the probability model analyzed in Proposition 1 and Lemma 13. On this graph, logI(t) differs from the difference of consecutive state log-flows by at most twice the largest TTB residual, with equality at zero residual (Eq. 7, App. D.2), so I(t) has the stated flow interpretation.
We have added the construction of Σ(s), the decoding rule for instructions and the table to App. K.3 (Table 7).