acceptodds
Under review as a conference paper at ICLR 2027

SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration

Abstract

In recent years, a variety of powerful LLM-based agentic systems have been applied to automate complex tasks through task orchestration. However, existing orchestration methods still face key challenges, including strategy collapse under reward maximization, high gradient variance with opaque credit assignment, and unguided skill evolution whose decisions are typically made by directly prompting an LLM to judge rather than derived from principled training signals. To address these challenges, we propose SkillFlow, a flow-based framework that takes a trainable Supervisor as the agent and a structured environment with dynamic skill library and frozen executor, automating task orchestration through multi-turn interaction. SkillFlow employs Tempered Trajectory Balance (TTB), a regression-based flow-matching loss that samples orchestration strategies in proportion to reward, preserving diverse strategies rather than collapsing to a single mode. The same flow objective yields a jointly learned backward policy that provides transparent per-step credit assignment at zero additional inference cost. Building on these flow diagnostics, a recursive skill evolution mechanism determines *when* to evolve, *what* skills to create or prune, and *where* decision gaps lie—closing the loop from training signal to autonomous capability growth. Experimental results on 14 datasets show that SkillFlow significantly outperforms baselines across question answering, mathematical reasoning, code generation, and real-world interactive decision making tasks. Our code is available at https://anonymous.4open.science/r/SkillFlow-E850.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agentic orchestration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eV3DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SkillFlow, a framework for learning multi-step LLM agent orchestration while maintaining an evolving skill library. SkillFlow jointly trains forward and backward policies using Tempered Trajectory Balance (TTB), with the goal of sampling terminal strategies in proportion to their rewards. Quantities derived from the same flow objective, including the TTB residual, forward-backward probability ratios, and skill invocation flows, are used to determine when the library should evolve, which skills should be retained or removed, and where new skills should be created.

Critical strengths
  1. The problem is important and the central idea is coherent. Using a common flow signal for orchestration learning, step-level diagnosis, and skill-library evolution provides a unified perspective on an important agent-learning problem.

  2. The empirical evaluation is broad. The paper evaluates 14 benchmarks and includes component ablations, OOD evaluation, transfer across executors, and replacement of the skill-creator model. The results suggest that the complete system has strong empirical performance.

Critical weaknesses
  1. The experimental action space is not reconciled with the exact normalization required by TTB. TTB assumes a normalized distribution over a finite set of complete actions, Σ(s)\Sigma(s), at every state. The evaluated tasks involve open-ended textual actions such as search instructions, tool arguments, and code-related operations, yet the paper does not explain how these actions are finitely enumerated, how large the candidate sets are, or how the normalizing constant is computed. It is therefore unclear whether the implemented policy corresponds to the probability model used in the theoretical analysis, and whether PF/PBP_F/P_B has the claimed flow interpretation.

  2. The GRPO comparison does not clearly isolate the contribution of TTB. The paper states that GRPO replaces only the TTB objective, while SkillFlow's library evolution depends on the TTB residual, the backward policy, and PF/PBP_F/P_B. It is not explained how GRPO‡ performs the same when/where/what skill updates after TTB is removed. The comparison may therefore change both the optimization objective and the skill-evolution mechanism, preventing the observed difference from being attributed specifically to TTB.

  3. The -Backward policy ablation is not sufficiently defined. TTB requires a backward distribution, but the paper does not state whether the learned PBP_B is replaced by a uniform distribution, a fixed rule, or PB=1P_B=1. These alternatives induce different path distributions and different training objectives. Consequently, the reported degradation cannot be interpreted directly as evidence for the benefit of step-level credit assignment.

  4. The diversity metric is not aligned with the paper's definition of a terminal strategy. The theory identifies different orderings of commuting actions as the same terminal strategy, whereas the experimental Div metric counts distinct step arrangements. It may therefore count multiple permutations of the same strategy as different workflows. The reported result does not establish that SkillFlow discovers a larger number of substantively distinct successful strategies.

Questions for authors
  1. How is Σ(s)\Sigma(s) constructed for each task family? Are open-ended search instructions, tool arguments, and code-related actions fully enumerated? Please report typical candidate-set sizes and the exact computation of the forward normalizer.

  2. How does GRPO‡ perform the when/where/what skill-evolution operations without a trained TTB residual and backward policy? Does it differ from SkillFlow only in the optimization objective, or also in the library-update rule?

  3. What backward distribution is used in the -Backward policy ablation? Does this variant preserve the same terminal target as the full method?

  4. Does Div count different action orderings that belong to the same quotient terminal strategy as distinct workflows? Can diversity be reported over the equivalence classes defined by the paper?

3 comments
AuthorsOpenReview ↗

Response to Reviewer eV3D (1/3): The action set and the forward normalizer

We thank the reviewer for the careful review and valuable suggestions, and answer each point below.

W1 / Q1. The action set Σ(s)\Sigma(s) and the forward normalizer.

Σ(s)\Sigma(s) is the finite set of legal complete actions at a state (§3). Open-ended text is not an element of it, so both normalizers are exact sums over finite sets.

How Σ(s)\Sigma(s) is constructed. A complete action a=(α,o)a=(\alpha,o) has a type α∈{skill,act,accept}\alpha\in\{\texttt{skill},\texttt{act},\texttt{accept}\} and parameters oo (Definition 2 in §3), which name the skill to activate or the operation to dispatch. At a non-terminal state, Σ(s)\Sigma(s) contains (i) the activation of each skill retrieved for the query (at most the four best matches), (ii) each act operation legal at that state, i.e. one of the three to six operations the task family exposes or, in WebShop and ALFWorld, one of the admissible actions the environment returns at that state, and (iii) accept. The set is a deterministic function of the canonical state, is fixed within a phase, and holds each action once, in a unique serialization of at most Kmax⁡K_{\max} tokens with an end marker (§3, App. K.3). All task families follow this rule and differ only in which operations are legal in (ii).

Why open-ended text needs no enumeration. At each step, the only stochastic decision of the Supervisor is the choice in Σ(s)\Sigma(s). Once an act operation is chosen, its instruction text is decoded greedily, so it is a deterministic function of the canonical state and the chosen action, with unit conditional mass. It has the same status as the reasoning r=f(k)(s)r=f^{(k)}(s) (§3, App. K.2, Lemma 13) and belongs to the deterministic transition δ(k)(s,a)\delta^{(k)}(s,a), not to Σ(s)\Sigma(s). What the instruction produces when executed (retrieved passages, generated code, tool outputs) comes from the frozen Executor and enters the state as execution feedback through CexecC_{\text{exec}} (§4.1, App. B.4). Search instructions, tool arguments and code are therefore not edges of the graph; every edge is one element of Σ(s)\Sigma(s).

Typical candidate-set sizes. The table covers all non-terminal states visited in run 1.

Task familyBenchmarkMean ∣Σ(s)∣\lvert\Sigma(s)\rvertMax
QAHotpotQA7.29
QATriviaQA6.48
QAMedQA7.39
MathAIME7.79
CodeSWE-bench9.611
InteractiveWebShop15.747
InteractiveALFWorld30.358

On the non-interactive families the maximum equals the number of operations plus four skills and accept; in the interactive environments it follows the number of admissible actions. The parent set of the backward policy is bounded by the horizon, ∣Pa(x)∣≤Tmax⁡=12\lvert\mathrm{Pa}(x)\rvert\le T_{\max}=12 (App. E.3).

Exact computation of the forward normalizer. For each a∈Σ(s)a\in\Sigma(s), the Supervisor computes ℓθ(a;r,s)\ell_\theta(a;r,s), the mean token log-likelihood of its serialization including the end marker, and then CF(s,r)=∑a′∈Σ(s)eℓθ(a′;r,s)C_F(s,r)=\sum_{a\prime\in\Sigma(s)}e^{\ell_\theta(a\prime;r,s)} and πθ(a∣r,s)=eℓθ(a;r,s)/CF(s,r)\pi_\theta(a\mid r,s)=e^{\ell_\theta(a;r,s)}/C_F(s,r) (Eq. 4). With at most 58 candidates this is an exact sum, with no sampling approximation. The TTB residual uses the same log⁡πθ\log\pi_\theta, with log⁡CF\log C_F kept inside Δ\Delta and differentiated (App. B.3, K.3).

Correspondence with the analysis. The action is drawn from πθ\pi_\theta itself, and the rollout temperature of 1.0 in App. L.1 refers to this draw. The sampler and the loss therefore use the same normalized kernels on the same finite DAG, which is the probability model analyzed in Proposition 1 and Lemma 13. On this graph, log⁡I(t)\log I(t) differs from the difference of consecutive state log-flows by at most twice the largest TTB residual, with equality at zero residual (Eq. 7, App. D.2), so I(t)I(t) has the stated flow interpretation.

We have added the construction of Σ(s)\Sigma(s), the decoding rule for instructions and the table to App. K.3 (Table 7).

AuthorsOpenReview ↗

Response to Reviewer eV3D (2/3): The contribution of TTB, and how GRPO‡ evolves skills

W2 / Q2. The contribution of TTB, and how GRPO‡ performs when/where/what.

GRPO‡, the "− TTB" row of Table 2, measures the contribution of TTB to the whole system, covering both the objective itself and the signals it supplies to skill evolution. With the library fixed so that only the objective differs, TTB is above GRPO‡ on all 14 benchmarks (80.47 vs. 76.09 on average).

How when/where/what are performed under GRPO‡. The three flow signals are produced by TTB training: they come from the TTB residual, the backward policy PϕP_\phi and F^(s)\hat F(s). GRPO‡ keeps the same Skill Creator (GPT-5.5), the same curation operator Φ\Phi and the same library capacity, and GPT-5.5 decides directly from the sampled trajectories and their rewards when to evolve, where the decision gaps lie and what skills to create or prune.

A comparison that changes only the objective. To measure the objective on its own, we train both objectives on the same library: SkillFlow's final library is given from step 0 and frozen, with no skill created, refined or pruned during training. Neither run makes any when/where/what decision, so the two "Fixed final" rows differ only in the objective.

LibraryObjectiveHotpotTriviaAIMEMedQAWebShopALFWorldSWEMuSiQueNQMATHGPQAHumanEvalSciWorldMind2WebAvg
Fixed finalGRPO‡89.0692.1956.6787.5083.5985.1643.7581.2581.2592.1983.5990.6352.3446.0976.09
Fixed finalTTB91.4196.0963.3390.6389.8492.1949.2284.3882.8196.0989.0697.6654.6949.2280.47
EvolvingGRPO‡ (Table 2, − TTB)83.5987.5053.3385.1682.0382.8140.6373.4479.6987.5079.6989.0639.0633.5971.22
EvolvingTTB (SkillFlow)92.1996.0970.0092.1993.7596.0952.3485.1682.8196.0989.8498.4457.8151.5682.45

The contribution of the TTB objective itself. With the same library, TTB is above GRPO‡ on all 14 benchmarks, and this gap comes from the objective alone. The reason lies in what the two objectives train toward: GRPO‡ tends to concentrate probability on one mode, whereas TTB matches the probability of each terminal strategy to its tempered reward and keeps multiple high-reward paths alive (§4.2, Fig. 5d). The same table also shows what the flow signals contribute to skill evolution: still under GRPO‡, replacing the library that GPT-5.5 evolves by its own judgment with the flow-driven one raises all 14 benchmarks. Together with the gain of co-evolution over a fixed library (the two TTB rows), these two parts make up the gap of "− TTB" in Table 2.

We have added how GRPO‡ updates its library, together with the table, to App. L.2 (Table 8), and the caption of Table 1 now points to it.

AuthorsOpenReview ↗

Response to Reviewer eV3D (3/3): The backward-policy ablation, and Div over equivalence classes

W3 / Q3. The backward distribution in the "− Backward policy" ablation and its terminal target.

The ablation replaces the learned PϕP_\phi by the uniform distribution over the legal parent set, and it has the same terminal target as the full method.

The backward distribution. For every child state xx, PB(p∣x)=1/∣Pa(x)∣P_B(p\mid x)=1/\lvert\mathrm{Pa}(x)\rvert for p∈Pa(x)p\in\mathrm{Pa}(x), where Pa(x)\mathrm{Pa}(x) is the legal parent set of Proposition 28. It has no trainable parameters and stays fixed during training, while the forward policy, log⁡Zθ\log Z_\theta, the form of the TTB residual and the skill evolution are unchanged. The residual becomes Δ(τ)=log⁡Zθ(q)+∑tlog⁡PF(st∣st−1)−βlog⁡R~(sT)+∑tlog⁡∣Pa(st)∣\Delta(\tau)=\log Z_\theta(q)+\sum_t\log P_F(s_t\mid s_{t-1})-\beta\log\tilde R(s_T)+\sum_t\log\lvert\mathrm{Pa}(s_t)\rvert.

The terminal target. The uniform distribution is a positive normalized backward kernel on the same graph. For any such kernel, the path distribution targeted by TTB is R~(x)βPB(τ∣x)/Zβ(q)\tilde R(x)^\beta P_B(\tau\mid x)/Z_\beta(q) (Theorem 16), whose terminal marginal is πβ∗(x)∝R~(x)β\pi^*_\beta(x)\propto\tilde R(x)^\beta (Eq. 1, Lemma 12). The backward kernel only determines how the mass of each terminal strategy is spread over its paths, and thereby the interior flows (Lemma 18). The two variants therefore share the terminal target and differ only in the path distribution and the interior flows.

What the ablation measures. The uniform kernel splits the mass of each state equally among its parents, whereas PϕP_\phi learns this allocation from child and candidate-parent information. The step quantities I(t)=PF/PBI(t)=P_F/P_B and F^(s)\hat F(s) are computed from this allocation (Eqs. 7, 8), and the where and what steps of skill evolution rely on them. Since the two variants share the terminal target and differ only in this allocation, the consistent drop on all 14 benchmarks reflects the benefit of learned per-step credit.

We have added the definition and the residual above to the caption of Table 2 and App. K.4.

W4 / Q4. Div and the terminal strategies defined in the paper.

Which orderings belong to the same terminal strategy. The equivalence ≡I\equiv_I is generated only by swaps of adjacent independent actions (Definition 13), and the independence relation II contains the pairs of distinct skill activations (Lemma 26). Pairs of act operations and pairs involving accept are dependent: they share execution state, so their observations and context order can differ between the two orders (Definition 13). Retrieving before computing and computing before retrieving are thus two different terminal strategies in the theory as well, and counting them separately is consistent with it. The only orderings the theory identifies are swaps of adjacent activations of distinct skills.

Div over equivalence classes. Div in the submitted Fig. 5(d) counts step arrangements. The new column first merges the workflows of a query that differ only by such swaps and then counts; everything else follows App. O.

MethodDiv, step arrangements (submitted Fig. 5d)Div, equivalence classes
GRPO‡8.757.31
Tree-GRPO7.315.62
HCAPO8.426.72
SkillFlow16.7913.38

Result. The conclusion is unchanged over equivalence classes: SkillFlow still finds 1.8× as many distinct successful strategies as the best of the three objectives, and the four methods keep their order. Merging only concerns the order of distinct skill activations and affects the four methods alike; the gap that remains after merging shows that the additional workflows of SkillFlow are substantively different strategies, not reorderings of one strategy. This agrees with the target of TTB, which matches the probability of each terminal strategy to its tempered reward and therefore keeps several high-reward strategies at once. The Pass@K gain in Fig. 5(d) does not depend on how workflows are counted and supports the same conclusion: SkillFlow gains clearly more under repeated attempts, so these distinct strategies do turn into success.

We now report Div over equivalence classes in Fig. 5(d), §5 (RQ5) and App. O (Table 9).

We thank the reviewer again. All additional experiments and revisions have been incorporated into the revised version and are marked in blue. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider your score. Thank you very much!

Reviewer_1sLVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkillFlow, a flow-based framework for agentic task orchestration with recursive skill evolution. The framework employs a trainable Supervisor, a frozen Executor, and a dynamically maintained skill library. Its core training objective, Tempered Trajectory Balance (TTB), jointly optimizes forward and backward policies to encourage reward-proportional sampling of orchestration strategies, aiming to mitigate strategy collapse and improve credit assignment. Building on the resulting flow diagnostics, the authors introduce a recursive skill evolution mechanism that determines when to update the skill library, which skills to retain, refine, or prune, and where to generate new skills. Experiments on benchmarks demonstrate improvements over existing agent orchestration and reinforcement learning baselines. Additional ablation studies, strategy diversity analyses, executor transfer experiments, and computational cost comparisons support the effectiveness of the proposed components.

Critical strengths

1. Unified Framework for Flow-Based Policy Learning and Skill Evolution. The paper presents a well-motivated framework that integrates reward-proportional trajectory sampling with recursive skill evolution. A key strength is the reuse of backward-flow diagnostics to guide skill evolution, connecting policy optimization with skill-library management without requiring additional policy evaluations.

2. Extensive Empirical Validation. The paper evaluates SkillFlow across diverse benchmarks, demonstrating improvements over agent orchestration and RL baselines. The evaluation is further strengthened by component ablations, controlled comparisons with alternative RL objectives, and strategy diversity analyses.

Critical weaknesses

1. Insufficient Validation of Flow-Based Credit Assignment. The paper lacks convincing evidence that the proposed flow diagnostics reliably capture long-horizon decision importance or actual skill utility. Their effectiveness is primarily supported by indirect ablations, weakening the justification for a central methodological contribution.

2. Unclear Contribution of Recursive Skill Evolution. The paper lacks a controlled comparison with TTB training using a fixed, non-empty skill library. Consequently, it remains unclear whether the improvements stem from the proposed flow-driven evolution mechanism or simply from access to high-quality LLM-generated skills.

Questions for authors
  1. Validity of Flow-Based Credit Assignment: How does the proposed step importance I(t)I(t) reliably reflect long-horizon decision importance, particularly before TTB convergence? Can the authors provide direct empirical evidence supporting this interpretation?

  2. Contribution of Recursive Skill Evolution: Can the authors compare SkillFlow with TTB training using a fixed, non-empty skill library to isolate the benefits of dynamic skill evolution from those of simply having high-quality skills?

  3. Effectiveness of Flow-Guided Skill Generation: How much improvement comes specifically from flow-based guidance? Can the authors compare against a baseline that uses the same Skill Creator and success/failure trajectories but selects skill-generation positions without flow diagnostics?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1sLV (1/4): Overall Conclusions & W1 / Q1 (Theory + Step-Level Causal Interventions)

Thank you for the constructive review. To answer your questions directly: (1) Step importance I(t)I(t) tracks the single-step change in reachable terminal reward within 2max⁡∣Δ(τ)∣2\max|\Delta(\tau)| without discounting, and causally isolates outcome-determining steps—separating critical decisions from adjacent and random steps as early as 12% into training and outperforming GPT-5.5 reading full trajectories—while skill flow G(s)G(s) directly reflects individual skill utility (W1/Q1). (2) Co-evolving the skill library with TTB outperforms training on even an oracle fixed library that freezes SkillFlow's final GPT-5.5 skills from step 0 (82.45 vs. 80.47 avg.), showing that gains come from dynamic evolution synchronized with policy bottlenecks rather than static access to strong skills (W2/Q2). (3) Under the identical GPT-5.5 Skill Creator and trajectory pairs, selecting target steps via I(t)I(t) outperforms letting GPT-5.5 choose where to intervene across all 14 benchmarks (+6.94 avg., Q3).


W1 / Q1: Does I(t)I(t) capture long-horizon decision importance?

Yes—analytically, log⁡I(t)\log I(t) equals the exact single-step increment in reachable terminal log-reward up to 2max⁡∣Δ(τ)∣2\max|\Delta(\tau)| with no discounting, and causally, intervening at the highest-∣log⁡I(t)∣|\log I(t)| step shifts task success far more than adjacent (t±1t\pm 1) or random steps (44.52% vs. 55.03%–56.45% when perturbed; 84.99% vs. 80.44%–80.64% when re-executed).

Why I(t)I(t) tracks downstream reward without discounting. Step importance I(t)=PF(st∣st−1)/PB(st−1∣st)I(t) = P_F(s_t \mid s_{t-1}) / P_B(s_{t-1} \mid s_t) (Eq. 7) compares the forward policy's choice probability before execution against the hindsight backward policy's parent attribution after observing execution feedback. Even before balance, log⁡I(t)\log I(t) obeys two properties controlled directly by the TTB residual Δ(τ)\Delta(\tau) (Eq. 6):

  1. Exact trajectory telescoping: For any complete trajectory τ=(s0,…,sT)\tau = (s_0, \dots, s_T), ∑t=1Tlog⁡I(t)=βlog⁡R~(sT)−log⁡Zθ(q)+Δ(τ)\sum_{t=1}^T \log I(t) = \beta \log \tilde{R}(s_T) - \log Z_\theta(q) + \Delta(\tau). Up to the query-only term log⁡Zθ(q)\log Z_\theta(q), step log-importances sum to tempered terminal log-reward with no γt\gamma^t discounting or truncation; the gap is literally the TTB residual minimized during training.
  2. Per-step flow-increment bound: Let Fq(s)=∑x∈Desc(s)∩XR~(x)βPB(s∣x)F_q(s) = \sum_{x \in \text{Desc}(s)\cap\mathcal{X}} \tilde{R}(x)^\beta P_B(s \mid x) be the reward-matching state flow induced by the current backward policy PBP_B, summing reachable terminal rewards weighted by backward hitting probabilities (Lemma 18). Because both PFP_F and the backward-induced target forward policy are normalized over children at every state, suffix log-ratios cannot take one strict sign across all continuations, bounding each prefix flow error by max⁡∣Δ∣\max |\Delta|. Hence at every step tt, ∣log⁡I(t)−(log⁡Fq(st)−log⁡Fq(st−1))∣≤2max⁡τ′⊃(s0,…,st−1)∣Δ(τ′)∣,\big|\log I(t) - \big(\log F_q(s_t) - \log F_q(s_{t-1})\big)\big| \le 2 \max_{\tau' \supset (s_0,\dots,s_{t-1})} |\Delta(\tau')|, over complete trajectories sharing the prefix up to st−1s_{t-1}, with equality at Δ=0\Delta = 0. Large ∣log⁡I(t)∣|\log I(t)| thus marks a sharp shift in reachable terminal reward across step tt, however many steps remain.

Direct step-level causal interventions (Tables A & B). Because SkillFlow reads I(t)I(t) directly to locate decision gaps rather than only training a sampler, we test I(t)I(t) via single-step interventions on the frozen run-1 model (fixed library, inference only) across four binary-outcome tasks: WebShop, ALFWorld, SWE-bench, and AIME (414 problems total; 331 solved, 83 failed unperturbed). Tables A and B report per-task success rates (%) and their 4-task macro-average:

(A) Perturbing one step on solved trajectories: replay to t−1t-1, replace ata_t with another uniformly sampled legal action, then roll out normally (originally failed problems stay failed; lower = more critical):

Perturbed StepWebShopALFWorldSWEAIMEAvg.
None (SkillFlow, run 1)93.7596.0952.3470.0078.05
Step with largest ∣log⁡I(t)∣\lvert\log I(t)\rvert54.6957.0329.6936.6744.52
Adjacent step (t−1t-1 or t+1t+1)66.4168.7538.2846.6755.03
Uniformly random step68.7571.0935.9450.0056.45
Step with smallest ∣log⁡I(t)∣\lvert\log I(t)\rvert83.5984.3842.9760.0067.74
Step chosen by GPT-5.5 reading full trajectory59.3864.0632.0343.3349.70

(B) Re-executing from one step on failed trajectories: replay to t−1t-1, forbid original action ata_t, and re-execute once from step tt (originally solved problems stay solved; higher = key failure step). Please see Table B at the start of Comment 2 / 4 below.

AuthorsOpenReview ↗

Response to Reviewer 1sLV (2/4): W1 / Q1 Continued (Table B, Pre-Convergence Checkpoints & Direct Skill Utility)

(B) Re-executing from one step on failed trajectories (continued from Comment 1 / 4):

Re-execution StepWebShopALFWorldSWEAIMEAvg.
None (SkillFlow, run 1)93.7596.0952.3470.0078.05
Step with largest ∣log⁡I(t)∣\lvert\log I(t)\rvert97.6698.4467.1976.6784.99
Adjacent step (t−1t-1 or t+1t+1)95.3196.8857.0373.3380.64
Uniformly random step95.3196.8856.2573.3380.44
Step with smallest ∣log⁡I(t)∣\lvert\log I(t)\rvert93.7596.0953.9170.0078.44
Step chosen by GPT-5.5 reading full trajectory96.0996.8862.5076.6783.04

Tables A and B show four clear patterns: (1) replacing the highest-∣log⁡I(t)∣|\log I(t)| step breaks the most solved runs, and re-running from it rescues the most failed ones on all four tasks; (2) the adjacent step (t±1t \pm 1) sits at virtually the same depth yet behaves like a random step, ruling out a position artifact; (3) flagged steps occur early (steps 3–4 on average out of up to 12) although rewards arrive only at termination; and (4) I(t)I(t) localizes bottlenecks more accurately than GPT-5.5 reading the full transcript, with zero extra LLM calls.


W1 / Q1 (continued): Validity before TTB convergence, and direct skill utility.

Before TTB convergence (Table C): I(t)I(t) reliably separates critical decisions from adjacent and random steps as early as step 30 (just 12% into training), well before full TTB convergence. Both analytical properties above hold throughout training: the approximation gap is governed by Δ(τ)\Delta(\tau) and shrinks as TTB drives the residual down. Moreover, SkillFlow only reads I(t)I(t) to generate skills at phase boundaries when the running TTB residual enters a plateau (Definition 14), rather than while the residual is still rapidly changing. To test pre-convergence accuracy directly, we repeat both single-step interventions at intermediate checkpoints (steps 0, 30, 60, 90, 120, out of 250) on WebShop, ALFWorld, SWE-bench, and AIME. Table C reports the pooled success rate (%) across all 414 test instances (331/414 = 79.95% unperturbed at step 250):

Checkpoint (training step)0306090120250 (final)
No intervention (pooled over 414 problems)46.1466.9173.1979.7180.1979.95
Perturb step with largest ∣log⁡I(t)∣\lvert\log I(t)\rvert (lower = more critical)32.3739.1342.0347.3447.1046.38
Perturb adjacent step (t±1t \pm 1)33.8245.1748.7955.3157.4957.00
Perturb random step34.0645.4150.7257.0058.2157.97
Re-execute from largest ∣log⁡I(t)∣\lvert\log I(t)\rvert (higher = more critical)51.9371.2678.9985.0285.9986.96
Re-execute from adjacent step (t±1t \pm 1)50.0069.5775.1282.3783.8282.37
Re-execute from random step50.9768.8475.3681.4082.6182.13

At step 0 (untrained, large residual), all three step-selection rules give nearly identical results. Once TTB training starts—helped by execution feedback conditioning the hindsight backward policy PϕP_\phi—I(t)I(t) already separates critical decisions from adjacent and random steps by step 30, and that margin widens steadily across later checkpoints. Reading I(t)I(t) at intermediate TTB plateaus is therefore empirically well grounded.

Direct validation of skill utility (Table D): Removing the single invoked skill with highest G(s)G(s) causes the largest accuracy drop on every task (-10.75 points on average vs. -2.59 for the lowest-G(s)G(s) skill), directly confirming that G(s)G(s) tracks true skill utility. At the skill level, G(s)G(s) measures the mean log-flow share across the states where skill ss is invoked rather than raw invocation count (Lemma 22). To test whether G(s)G(s) reflects actual skill utility, we remove a single invoked skill from the final library at inference time and re-evaluate (reporting per-task success rates and their 4-task macro-average; lower means the removed skill contributed more):

Removed SkillWebShopALFWorldSWEAIMEAvg.
None (full library, run 1)93.7596.0952.3470.0078.05
Skill with highest G(s)G(s)85.1685.9441.4156.6767.30
Randomly chosen invoked skill89.0690.6346.0960.0071.45
Skill with lowest G(s)G(s)92.1993.7549.2266.6775.46

Across all four benchmarks, dropping the highest-G(s)G(s) skill causes the largest drop, random removal sits in between, and dropping the lowest-G(s)G(s) skill has minimal impact. Because state flow sums reachable terminal rewards (Lemma 18), high G(s)G(s) identifies skills whose invocation states carry substantial flow toward successful completions, consistent with Fig. 5c where each curation event precedes a reward gain. We have added this proposition and proof to Appendix D.2, and Tables A–D to Appendix O.1 (A and B as Table 10, C as Table 11, D as Table 12).

AuthorsOpenReview ↗

Response to Reviewer 1sLV (3/4): W2 / Q2 (Fixed Versus Co-Evolving Skill Library)

W2 / Q2: Comparison with TTB training on a fixed, non-empty high-quality skill library.

Co-evolving the skill library with TTB outperforms training on even an oracle fixed library that freezes SkillFlow's own final GPT-5.5 skills from step 0 (82.45 vs. 80.47 average across 14 benchmarks, winning on 11/14 and never lower), confirming that the gains come from dynamic evolution rather than static access to high-quality skills.

To give a static library the strongest possible test, we run an oracle fixed-library control: we hand SkillFlow's own end-of-training skill library (written by GPT-5.5) to the Supervisor at step 0 and freeze it across all 250 TTB steps, with no creation, refinement, or pruning. Thus the static baseline starts from step 0 with the exact high-quality GPT-5.5 skills that SkillFlow only finishes assembling at the end of training, isolating dynamic evolution from tip quality. Below we compare it against TTB with an empty library (− Skills, Table 2), a fixed same-size uncurated library from Qwen3.8-27B (+ Random skills, Table 2), and full SkillFlow:

Skill Library Setting (all trained with TTB)HotpotTriviaAIMEMedQAWShopALFWSWEMuSQNQMATHGPQAHEvalSciWM2WAvg.
Empty library (− Skills, Table 2)81.2578.9143.3378.1377.3477.3432.8171.0971.8889.0680.4787.5042.9740.6368.05
Fixed: uncurated (+ Random skills, Table 2)89.8494.5360.0089.8485.9486.7246.0982.8182.8191.4185.1692.9749.2244.5377.28
Fixed: final GPT-5.5 library frozen at step 091.4196.0963.3390.6389.8492.1949.2284.3882.8196.0989.0697.6654.6949.2280.47
SkillFlow (library co-evolves with TTB)92.1996.0970.0092.1993.7596.0952.3485.1682.8196.0989.8498.4457.8151.5682.45

What dynamic evolution adds beyond good skills. Even gifted with the final evolved GPT-5.5 library from step 0, fixed-library TTB trails co-evolving SkillFlow on 11 of 14 benchmarks and is never higher, with the clearest gaps on multi-step tasks (AIME 63.33 vs. 70.00, WebShop 89.84 vs. 93.75, ALFWorld 92.19 vs. 96.09, SWE-bench 49.22 vs. 52.34, ScienceWorld 54.69 vs. 57.81). In SkillFlow, each skill is created at the current policy's bottleneck when its TTB residual plateaus, so the policy immediately trains around it; dumping all end-stage skills into step 0 leaves the library out of sync with what the early policy actually fails on.

Where those high-quality skills come from. Moreover, that final library does not exist before training: it is itself produced by flow-guided evolution. A same-size uncurated fixed library (+ Random skills) scores lower on 13 of 14 benchmarks (77.28 avg.), and even with GPT-5.5 writing every tip, dropping any one of when, where, or what hurts across benchmarks (Table 2). We added this comparison and discussion to Appendix F.10 (Table 5).

Reviewer_9wH2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SkillFlow trains a Supervisor with Tempered Trajectory Balance and uses training-derived flow signals to decide when, what, and where to evolve a GPT-5.5 generated skill library.

Critical strengths
  1. This paper is well motivated. Existing dynamic-library methods manage skills with heuristic schedules or LLM-as-judge prompting. Using training signals instead is a principled alternative with little extra overhead.
  2. The experiments and analysis are solid. The paper evaluates on 7 IID and 7 OOD benchmarks, with thorough analyses covering executor transfer, Skill Creator swap, objective-only comparison, and training cost.
Critical weaknesses

My major concern is that the tips are written by GPT-5.5, and performance clearly tracks the strength of the Skill Creator (Table 3). However, the paper does not specify which LLMs the baselines use for their auxiliary components (e.g., skill distillation in SkillRL, the optimizer in AFlow), so it is unclear whether the comparison in Table 1 is matched in teacher strength. A controlled experiment using the Supervisor itself as the Skill Creator would better isolate the contribution of the flow signals.

Questions for authors
  1. Could the authors report results with the Supervisor itself (Qwen3.5-9B) serving as the Skill Creator? If SkillFlow still outperforms the baselines in this setting, it would strongly support that the gains come from the flow signals rather than from GPT-5.5.

  2. Which LLMs do the baselines use for their auxiliary components (e.g., skill distillation in SkillRL, the LLM judge in AFlow)? Do any of them have access to GPT-5.5?

I would consider raising my score if the authors show that SkillFlow retains a clear advantage when the Supervisor itself serves as the Skill Creator, and clarify that the baselines are matched in auxiliary LLM strength.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 9wH2 (1/2): Overall Conclusions & W1 / Q1 (Supervisor as Skill Creator)

Thank you for the thoughtful review. Separating the effect of our flow signals from GPT-5.5's strength as a teacher is a fair test, and your two questions get straight to the heart of it. In short: (1) when we replace GPT-5.5 with the Supervisor itself (Qwen3.5-9B) as the Skill Creator Ψ\Psi, SkillFlow still outperforms all baselines in Table 1; and (2) every baseline in Table 1 that uses an auxiliary LLM was already run with GPT-5.5, so teacher strength is matched across methods.

W1 / Q1: Using the Supervisor itself (Qwen3.5-9B) as the Skill Creator Ψ\Psi.

We replace the frozen GPT-5.5 in the Skill Creator Ψ\Psi with the Supervisor itself (Qwen3.5-9B), keeping all other training and evaluation settings unchanged. To isolate what the flow signals contribute under this 9B creator, we also run a control that fills a same-size library with random skills written by the same 9B model without flow guidance. For direct comparison, the table below includes the peer baselines from Table 1 (all sharing the same trainable Qwen3.5-9B Supervisor backbone and frozen Qwen3.5-9B Executor), along with our in-framework variant that keeps the same dynamic skill library and GPT-5.5 Skill Creator Ψ\Psi while replacing only the Tempered Trajectory Balance (TTB) training objective with GRPO (GRPO‡, Fig. 5a):

MethodAux. / Creator LLMHotpotTriviaAIMEMedQAWebShopALFWorldSWEMuSiQueNQMATHGPQAHumanEvalSciWorldMind2WebAvg.
SkillFlow, Ψ\Psi = GPT-5.5 (original)GPT-5.592.1996.0970.0092.1993.7596.0952.3485.1682.8196.0989.8498.4457.8151.5682.45
SkillFlow, Ψ\Psi = Qwen3.8-27BQwen3.8-27B89.8494.5360.0087.5087.5089.0646.8879.6982.8189.0685.9496.0952.3445.3177.61
SkillFlow, Ψ\Psi = Qwen3.5-9B (Supervisor)Qwen3.5-9B89.0694.5356.6785.9485.1686.7242.9778.9182.8189.0683.5992.9748.4443.7575.76
SkillFlow, Ψ\Psi = Qwen3.5-9B, random skillsQwen3.5-9B86.7293.7550.0083.5981.2584.3839.0674.2282.8185.9481.2591.4144.5340.6372.82
SkillFlow, without skill libraryNone81.2578.9143.3378.1377.3477.3432.8171.0971.8889.0680.4787.5042.9740.6368.05
FlowSteer (RL orchestration without skill library, Table 1)None89.8491.2562.0090.9478.9182.6643.5979.8480.3193.5984.3893.9147.8141.1075.72
AgentFlow (multi-turn planning & tool-use RL, Table 1)GPT-5.588.4490.1660.0087.3474.0680.7841.5680.0078.9192.0381.2593.5944.8440.1573.79
SkillRL (RL with heuristic skill evolution, Table 1)GPT-5.587.6689.0656.0085.7882.9785.4739.3875.4779.2291.0978.1292.3438.9135.1672.62
AFlow (MCTS workflow search, Table 1)GPT-5.588.5992.8153.3389.5359.3875.1630.1677.8175.7890.7875.6292.8129.8430.7868.74
GRPO‡ (in-framework, replacing TTB with GRPO, Fig. 5a)GPT-5.583.5987.5053.3385.1682.0382.8140.6373.4479.6987.5079.6989.0639.0633.5971.22

Even with the 9B Supervisor writing its own skills, SkillFlow stays ahead of FlowSteer, AgentFlow, SkillRL, and AFlow in overall average, and leads all four on every interactive benchmark (WebShop, ALFWorld, ScienceWorld, and Mind2Web), even though SkillRL, AgentFlow, and AFlow still rely on GPT-5.5 for their auxiliary components (see Q2). Within our own architecture, the contrast is even clearer: GRPO‡ keeps GPT-5.5 as its Skill Creator Ψ\Psi and replaces TTB with GRPO, yet trails 9B-only SkillFlow across every benchmark.

AuthorsOpenReview ↗

Response to Reviewer 9wH2 (2/2): W1 / Q1 Continued & W1 / Q2 (Flow Signals & Auxiliary LLMs of the Baselines)

W1 / Q1 (continued): What the flow signals contribute.

Why don't the baselines get more out of GPT-5.5? Because none of them gives GPT-5.5 a signal pointing to which step went wrong. SkillRL updates its library only when validation accuracy falls below a preset threshold, leaving GPT-5.5 to read whole trajectories and guess the failure point on its own; AgentFlow uses GPT-5.5 to judge whether the final answer is right; and AFlow asks GPT-5.5 to edit workflows from whole-workflow scores. Localizing the specific step that needs fixing is the job of our flow signals.

The 9B random-skill row isolates that effect directly. With the same 9B model writing the tips, adding flow guidance improves over random skills on nearly every benchmark and is never worse, and lifts the 9B self-creator variant close to the 3×3\times larger Qwen3.8-27B. Stronger Skill Creators do raise the overall ceiling (GPT-5.5 > Qwen3.8-27B > Qwen3.5-9B), as you noted. What the flow signals do is handle the hardest part before Ψ\Psi is ever called—pinpointing where a trajectory broke down:

  1. Where to generate: The forward-to-backward policy ratio (step importance I(t)I(t)) flags steps that deviate sharply from flow balance as candidate decision gaps. Ψ\Psi then receives one succeeding and one failing trajectory for the same query aligned at step tt (Appendix F.3), and only has to write local guidance for that decision.
  2. What to retain: Skills are kept, refined, or pruned according to the mean and stability of the flow measured at their invocation sites (Eq. 10), so weak tips do not pile up in the library.
  3. When to evolve: We trigger a library update only when the flow-balance training loss (TTB residual) plateaus.

Because all three signals target sequential credit assignment, the gains show up most clearly on multi-step interactive tasks. We have added this experiment and discussion to Appendix F.9 (Table 4).


W1 / Q2: Auxiliary LLMs used by the baselines.

Every trained method in our experiments uses the same trainable Qwen3.5-9B Supervisor backbone and the same frozen Qwen3.5-9B Executor (which carries out tool calls and environment actions; Appendix L.1). Apart from direct backbone fine-tuning without a skill library (SFT and GRPO† in Table 1) and FlowSteer (which relies on rule-based rewards), every baseline with an auxiliary LLM uses GPT-5.5, matching our default Skill Creator Ψ\Psi:

MethodRole of Auxiliary ComponentAuxiliary LLM
SkillFlowSkill Creator Ψ\Psi (drafting and refining candidate skills)GPT-5.5
SkillRLTeacher for skill distillation and library evolutionGPT-5.5
AFlowWorkflow optimizer and evaluator (LLM judge) in MCTS searchGPT-5.5
AgentFlowFinal-answer correctness judge for training rewards; retrieval helperGPT-5.5
FlowSteerNone: rule-based rewards; the only other LLM is the shared frozen Executor–

Thus all baselines that rely on an auxiliary LLM have the same access to GPT-5.5 as SkillFlow, and the comparison in Table 1 is matched in teacher strength. We have added an "Auxiliary LLM" row to the shared configuration table in Appendix L.1 and documented each baseline's auxiliary setup in Appendix N.

Thank you again for pushing us to test this cleanly. All new results and clarifications are now in the revised manuscript, marked in blue; if they resolve your two concerns, we hope you will consider raising your score.

Thank you for the detailed response and for running the additional experiment. Q2 is resolved.

On Q1, I would like to make sure I understand the new results correctly. With the 9B creator, SkillFlow (75.76) performs on par with FlowSteer (75.72) which does not use a skill library. Does this mean the gain over FlowSteer relies mainly on a strong Skill Creator?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.