acceptodds
Under review as a conference paper at ICLR 2027

RFlow: Recursive Self-Improvement via Recursive Skill Evolution

Abstract

LLM-based agents can improve themselves across tasks by reusing and revising the skills they orchestrate into executable procedures. Flow-based training fits this loop: it samples procedures in proportion to reward, and the flow through each skill credits it for the next library revision. Three obstacles stand in the way of making this self-improvement reliable: flow training suffers strategy collapse over tree-structured histories; nonnegative flow-based credit rewards frequent use as if it were benefit; and library edits rest on the task reward the policy optimizes. We introduce RFlow, a recursive self-improvement framework that alternates policy learning, independent verification, and versioned skill-library updates on a shared-state orchestration graph. The graph merges histories that differ only in the order of independent steps, allowing flow training to pool evidence across equivalent executions. A flow-share readout of the trained flow, invariant to the backward policy, and a separate signed utility rank which skills to change, verifier evidence decides whether an edit is warranted, and a residual-variance plateau sets when to update. Committed edits reshape the graph the next policy learns on, realizing recursive skill evolution. Across question answering, mathematical reasoning, interactive decision making, and code generation, RFlow improves task accuracy and library-edit precision over heuristic orchestration, reinforcement learning, and skill-evolution baselines, and transfers across executors. Code is available at https://anonymous.4open.science/r/r2flow-DD00.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Skill, Self-evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8UgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes R²Flow, a framework for improving an agent’s orchestration policy and skill library across successive training phases. A trainable Supervisor operates with a frozen executor on a shared-state graph that merges reorderings of independent events. The policy is trained using a subtrajectory-balance objective, R²TB. Flow share measures skill occupancy, while a separate signed utility estimates relative benefit. Independent verifier evidence and paired validation govern library edits, and a plateau in residual variance determines phase transitions.

The evaluation covers six in-distribution and six held-out datasets, additional executors, component ablations, and training-objective comparisons. Results are reported over five runs, generally on fixed 128-record evaluation subsets, with all 30 problems used for AIME 2026. The reported improvements are broad, but important questions remain about the correspondence between the formal readouts and their implementation, and the evidence supporting reliable skill evolution.

Critical strengths
  1. The framework separates quantities that are often conflated in skill evolution. A frequently used skill need not be beneficial, and a favorable task reward does not independently establish that a library edit is justified. R²Flow explicitly separates flow share, signed utility, and verifier-based eligibility (§4.2–4.3). The conditional analysis of backward-policy dependence and path-invariant skill counts makes this distinction more precise (Appendices A and D). This provides a coherent motivation for the design beyond adding another memory component.

  2. The empirical analysis covers both performance and the proposed mechanisms. Table 1 reports improvements across all twelve evaluated datasets under the stated shared-executor protocol. Table 2 includes extensive component ablations, while Figure 5 examines training objectives, phase-to-phase improvement, edit precision, and harmful-skill removal. The untrained-architecture comparison also helps separate architectural benefits from policy learning. Reporting five-run variation and training cost strengthens the evidence that the complete system is useful, even though some individual margins are small.

Critical weaknesses
  1. The formal interpretation of the main readouts is not yet reconciled with the released implementation. Corollary D.9 requires the rollout proposal and the forward probability used in the residual to agree, with positive support on every legal edge; Appendix C.3 states the same assumption explicitly ("Sampling and scoring then use the same conditional policy"). The released paper configuration declares greedy action decoding, and the rollout implementation uses temperature zero for action generation, while residual scoring uses model probabilities. Unless the reported runs used a different configuration, the consistency argument for the residual-weighted share estimator does not directly apply, and the bounded-error guarantee of Corollary D.9(iii), which also presumes positive-support sampling, does not cover the greedy case. Off-policy balance training remains possible, but the readout requires a separate justification. In addition, Eq. (10) specifies a fixed continuation policy and a reference uniform over legal events, whereas _signed_utility compares observed mean returns over observed events and leaves the utility undefined when fewer than two alternatives are observed. These differences matter because the readouts rank edits and can veto pruning; their interpretation as occupancy and relative benefit is central to the contribution.

  2. The evidence supporting reliable, independently verified evolution is insufficiently specified and calibrated. Appendix C lists tests, schema checks, assertions, and execution status, but does not adequately specify their task-level meaning or error rates across the different domains. The public repository additionally uses an external skill-author model to generate or revise skills and a separate external model to generate reference answers for verification in TriviaQA and AIME, without identifying either model or its budget in the reported experiments. Their capabilities and budgets could materially affect both the quality of edits and comparison fairness. Moreover, the released configuration uses a non-inferiority level of 0.2, absolute success/reward margins of 0.10, and token/latency margins corresponding to a factor of two. These choices need justification and should be stated in the paper. Proposition C.1 establishes that the specified gate is followed; it does not establish that verifier labels are accurate or that accepted edits reliably improve the system. The reported 33 beneficial edits out of 37 is encouraging, but needs this operational context.

Questions for authors
  1. Estimator–sampler correspondence: Which decoding configuration produced the rollouts used for flow-share estimation? How does the probability in δ0:T\delta_{0:T} correspond to the actual behavior distribution under greedy action decoding? Please explain how Corollary D.9 applies and provide the effective sample sizes of the importance weights.

  2. Signed utility: How are the fixed-continuation estimates and the uniform legal-event reference in Eq. (10) implemented? The released function appears to use observed returns and observed alternatives. What fraction of relevant states has sufficient alternative-action coverage, and how are unsupported utility estimates handled?

  3. Verification and resource parity: Which models generated skill edits and reference answers in the reported runs? Please specify the verifier signal and available evidence for each domain, provide any existing assessment of verifier errors, and clarify whether competing methods received comparable authoring, verification, and compute resources.

  4. Acceptance calibration: What motivates the released non-inferiority level and margins? Using existing paired outcomes, how sensitive are acceptance decisions and beneficial-edit precision to tighter settings? Please also clarify whether the 37 committed edits are pooled across the five runs or describe a particular run.

Reviewer_EMFMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce R2 Flow, a recursive self-improvement framework that alternates policy learning, independent verification, and versioned skill-library updates on a shared-state orchestration graph, which aims to solve three obstacles: strategy collapse, nonnegative flow-based credit rewards and library edits rest on the task reward the policy optimizes. Across 12 IID and OOD datasets, R2Flow outperforms the reported orchestration, RL, and skill-evolution baselines. And the authors provides source code for replication

Critical strengths
  1. The paper offers technically novel responses to concrete weaknesses of prior flow-based skill evolution. Shared-state merging makes the backward policy meaningful, while the split between non-negative flow share and signed utility avoids treating frequent use as evidence of benefit. Verifier-based edit gating further separates the policy's training reward from the evidence used to change the library. These ideas form a coherent contribution rather than a superficial combination of modules.

  2. The empirical study is broad and unusually diagnostic. It spans QA, mathematical reasoning, code, rubric-based evaluation, and interactive tasks; separates IID from OOD; and includes one-at-a-time component ablations, alternative training objectives, executor transfer, harmful-skill injection, and phase-by-phase edit outcomes. In particular, the reported 33 beneficial edits among 37 committed edits and the degradation under reward-derived edit labels support the use of independent verification.

Critical weaknesses
  1. The method is highly specialized and complex, while comparison to its closest flow-based predecessor is incomplete. SkillFlow appears in a harmful-skill analysis but not in the main result table, and the GRPO-in-pipeline comparison is insufficiently specified. Although Table 2 ablates individual modules, it does not show the cumulative gain from a simpler SkillFlow-like system or important interactions among shared states, the learned backward policy, the readout, signed utility, and verifier gating. This makes it difficult to judge which complexity is essential and whether the contribution is competitive with more mainstream RL or skill-evolution approaches.

  2. Practicality cannot be assessed from the current resource reporting. The paper gives per-step learner cost for the Supervisor objective, but not total training cost over eight phases, verifier calls, rollouts for signed-utility estimation and paired validation, or the model calls used to write skill edits. The asserted common inference budget is not quantified for all methods, and baseline costs are absent. These omissions are consequential because the gains may depend on substantial additional computation and verification.

Questions for authors
  1. [Suggestion] Can the authors add SkillFlow, and ideally Bayesian-Agent, to the main comparison under the same executor and budget, and provide a cumulative build-up plus selected interaction ablations? In particular, does the learned backward policy or reference readout help without shared-state merging, and does signed utility still help without verifier gating?

  2. [Clarification] What verifier is used for each open-ended domain, does it access gold answers, and how is the held-out verified score used to define the reported 0.89 edit precision kept independent from the edit gate? Please also report comparable edit-precision numbers for skill-evolution baselines if the abstract intends that comparison.

Reviewer_gQUROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes R2 Flow, a two-loop framework in which a trainable Supervisor (Qwen3.5-9B) orchestrates skills that a frozen executor runs. In the inner loop the Supervisor is trained as a GFlowNet forward policy with a sub-trajectory balance loss (R2 TB, which is SubTB plus a per-domain scalar offset b_d and a forward term conditioned on the reasoning text). States are "shared states" (committed events, dependency order, canonicalised artifacts, verified facts, observables), so that histories differing only in the order of independent steps are merged and the state space becomes a DAG instead of a prefix tree. Two readouts come from the trained flow: a "flow share" Psi^0(u) per skill, defined through a reference flow with uniform backward policy, and a signed utility A~_e that compares an event to a uniform alternative at the same state, discounted by the single-edge residual. In the outer loop, a library operator (defer/split/refine/retain/prune/consolidate/generate) proposes edits. Edits are gated by Beta-posterior credible bounds computed from verifier labels and by a paired non-inferiority test on a held-out validation set. A phase ends when the variance of the full-trajectory residual levels off. Propositions 1-3 and Appendices A-D give the formal side (DAG well-posedness, invariance of the terminal law and of Psi^0 to P_B, the residual-to-terminal-law bound, and gate invariants). Experiments cover 12 datasets (6 IID, 6 OOD), each a 128-record draw, plus AIME 2026 with 30 problems. Table 1 reports R2 Flow first on every row over SFT, GRPO, AFlow, FlowSteer, Trace2Skill and SkillOpt. The paper also reports executor transfer to six other models, a 9-row ablation (Table 2), a comparison of training objectives (Figure 5a,b), and an eight-phase self-improvement run with an edit precision of 0.89 (33/37).

Critical strengths
  1. The problem framing is good. The paper separates three questions in a skill-evolution loop, and states clearly that a frequently used skill is not necessarily a beneficial one.

    • Which skill to change: flow share and signed utility.
    • Whether the evidence supports a change: a verifier posterior with credible bounds rather than point estimates.
    • When to change it: a residual-variance plateau.
    • The share-only vs. share-plus-signed-utility comparison (Table 2; Figure 5e, where 1 of 12 injected harmful skills survives vs. 8 for share-only) is the kind of controlled experiment this area needs. If it holds up, it is the most useful finding in the paper.
  2. The ablation and objective-comparison design is extensive. The within-architecture comparison of training objectives is a much fairer test of the flow objective than the Table 1 SFT/GRPO rows.

    • Nine component removals are evaluated on all twelve datasets (Table 2).
    • Figure 5a compares GRPO, DB, TB, TTB, SubTB and R2 TB inside the same architecture and reports per-step learner and wall-clock cost; R2 TB is 7.15 IID points above GRPO (Appendix E).
Critical weaknesses
  1. The main results are hard to believe as stated, and the evaluation protocol cannot support the claimed precision. Several numbers are far above anything I know of for these benchmarks, the splits and retrieval corpus aren't specified, and the significance tests ignore test-set sampling.

    • Implausible scores for a 9B Supervisor with a frozen 9B executor (Table 1): NQ-Open EM 86.25, HotpotQA EM 91.09, MuSiQue-Ans EM 85.31, TriviaQA EM 95.00, WebShop SR 89.53, SWE-bench resolved 44.22, GPQA 84.84. Strong retrieval-augmented systems on open-domain NQ are usually in the 50s-60s EM, MuSiQue-Ans EM in the 80s is far beyond prior reports, and strong WebShop agents are typically well below 89.5 SR. The paper doesn't discuss why.
    • Unspecified setup. No split or variant is given (SWE-bench full, Lite or Verified; GPQA main or Diamond; the HotpotQA setting; the NQ-Open split), and no retrieval corpus. Without this, the numbers can't be compared to anything.
    • Small, fixed test sets. Each dataset is a single 128-record draw, and AIME 2026 has 30 problems (Appendix E). The binomial standard error at p around 0.9 on n = 128 is about 2.6 points; one AIME problem is 3.33 points.
    • Variance from reruns only. The standard deviations come from five runs on the same records, and several baselines show exactly +-0.00 (AFlow MATH-Hard 91.41, AFlow WebShop 60.16, Trace2Skill AIME 66.67). The Welch t-tests in Figure 4b (p < 1e-3 for every gain) treat run-to-run noise as the only noise. Many margins over the best baseline are within sampling error: 1.56 EM on HotpotQA, 1.10 on MATH-Hard, 0.93 on GPQA, 1.25 on SWE-bench.
    • Too-clean ablations. All 108 ablated cells in Table 2 fall below the full model, including gaps as small as 0.31 (MATH-Hard, paired non-inferiority). At this noise level I'd expect some cells to tie or cross. Per-record outputs, bootstrap CIs over records, and full-split evaluation are needed.
    • AIME 2026 leakage question. It is an IID training dataset and is evaluated on "all thirty released problems". Where did the training and validation queries for this domain come from, and are they disjoint from the 30 evaluation problems?
  2. The core claim, that library edits are admitted on "verifier evidence outside the task reward" (abstract, Section 2, Conclusion), is not substantiated, and much of the formal support is definitional. The key quantity, flow share, turns out not to need the flow machinery at all.

    • Verifiers are described only generically: "unit tests, schema validation, assertions, execution status" (Appendix C.1, E). For QA, schema validation and execution status say whether a call ran, not whether it helped. For MBPP+ and SWE-bench, if the verifiers are the benchmark's tests, they are the reward. For HealthBench and AIME, no verifier is described.
    • The commit gate still uses task reward. Paired validation tests non-inferiority on success and tempered reward (Section 4.3; Definition C.1), just on held-out queries. The "held-out verified score" behind the 0.89 edit precision (Figure 5d) is never defined.
    • Several propositions hold by construction. Proposition 3 / C.1: the gate is defined to require verifier predicates. Proposition C.2: the loop is a function of its full state. Proposition 1: a monotone rank gives a DAG.
    • Flow share is a reward-weighted usage count. Corollary D.9 gives Psi^0(u) = sum_x R_eta(x) n_u(x) / sum_x R_eta(x) t_skill(x), the reward-tilted invocation frequency of u at terminals. Its "backward-policy invariance" holds simply because every path to x commits the same events. That's fine as a quantity, but the paper presents it as a substantive contribution.
    • Missing positioning. Merging reorderings into a DAG is the standard motivation for GFlowNets over set-like or permutation-invariant states (Bengio et al. 2021; 2023; DAG-GFlowNet), and the paper doesn't position the shared-state graph against that work.
    • sigma is never implemented in the paper. It's not said how dependencies between LLM skill calls are detected, how free-text artifacts are canonicalised, or how often merging happens (e.g. the fraction of visited states with more than one in-edge). Assumption A.2 (reasoning text affects the future only through committed artifacts) is strong for a Supervisor whose P_F is explicitly conditioned on r_t, and it is untested.
Questions for authors
  1. For each of the 12 datasets, which split or variant is used (in particular SWE-bench, GPQA, NQ-Open, HotpotQA setting), what retrieval corpus and retriever are used, and does the "Qwen3.5 answering directly" baseline have access to retrieval and tools? Can you explain why NQ-Open EM 86.25, MuSiQue-Ans EM 85.31 and WebShop SR 89.53 are so far above previously reported numbers?

  2. Please describe the verifier used for each dataset, concretely. For MBPP+ and SWE-bench, are any of the verifier tests the same as (or derived from) the evaluation tests? For HotpotQA, TriviaQA, HealthBench and AIME, what labels a single skill invocation as correct? How is the "held-out verified score" in Figure 5d computed?

  3. What exactly is a "run": a fresh training seed, or repeated evaluation of one trained Supervisor? Can you report bootstrap confidence intervals over the 128 records (or full-split results), and re-run the Figure 4b tests with record-level pairing?

  4. Where do AIME 2026 training and validation queries come from, given that all 30 problems are used for evaluation?

  5. How is sigma implemented? How is independence of two skill calls decided, and how are artifacts canonicalised? What fraction of states visited during training have |In(s')| > 1? If this fraction is small on, say, AIME, how do you explain the 14.00-point drop from removing shared states there?

  6. How are the baselines run? Were FlowSteer, Trace2Skill and SkillOpt trained on the same IID queries with the same compute and the same initial skill library? SkillFlow (Zhang et al., 2026e) is the closest prior method ("flow-driven recursive skill evolution") and appears only in Figure 5e and as the TTB objective. Why is it not a main baseline in Table 1, and Bayesian-Agent likewise?

  7. Please list the missing hyperparameters: the thresholds theta_low/mid/high, n_min, kappa_u, credible level a, epsilon_b, gamma_var, h_0, window W, tau_c, the cooldown, the non-inferiority margins, the verifier budget, rollouts per query, the number of training queries, the optimizer and learning rate, and whether fine-tuning is full or LoRA. Also, how many skills are in the initial library, what form a skill takes (prompt, code, tool), and how refine/split/generate edits are produced (presumably by an LLM; which one?).

  8. The signed utility needs E_hat[R_eta | s, e'] for every legal alternative e' under a continuation policy pi_eval. What is pi_eval, how many counterfactual rollouts are used, and is that cost (and the cost of verification and paired validation) included in the "whole step" cost in Figure 5a?

  9. b_d is a single scalar per domain that, by Remark B.1, does not change the zero-loss solutions. Why would it produce a 6.67-point AIME gain over SubTB (two problems out of 30)? Does the "SubTB" baseline also include the r_t-conditioned forward term?

Suggestions (not required for my assessment):

  1. Define the "distinct strategies" metric in Figure 5f precisely. It would also help to show, for one task, the actual skills, a few committed edits, and an example of merged reorderings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.