- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes R²Flow, a framework for improving an agent’s orchestration policy and skill library across successive training phases. A trainable Supervisor operates with a frozen executor on a shared-state graph that merges reorderings of independent events. The policy is trained using a subtrajectory-balance objective, R²TB. Flow share measures skill occupancy, while a separate signed utility estimates relative benefit. Independent verifier evidence and paired validation govern library edits, and a plateau in residual variance determines phase transitions.
The evaluation covers six in-distribution and six held-out datasets, additional executors, component ablations, and training-objective comparisons. Results are reported over five runs, generally on fixed 128-record evaluation subsets, with all 30 problems used for AIME 2026. The reported improvements are broad, but important questions remain about the correspondence between the formal readouts and their implementation, and the evidence supporting reliable skill evolution.
Critical strengths
-
The framework separates quantities that are often conflated in skill evolution. A frequently used skill need not be beneficial, and a favorable task reward does not independently establish that a library edit is justified. R²Flow explicitly separates flow share, signed utility, and verifier-based eligibility (§4.2–4.3). The conditional analysis of backward-policy dependence and path-invariant skill counts makes this distinction more precise (Appendices A and D). This provides a coherent motivation for the design beyond adding another memory component.
-
The empirical analysis covers both performance and the proposed mechanisms. Table 1 reports improvements across all twelve evaluated datasets under the stated shared-executor protocol. Table 2 includes extensive component ablations, while Figure 5 examines training objectives, phase-to-phase improvement, edit precision, and harmful-skill removal. The untrained-architecture comparison also helps separate architectural benefits from policy learning. Reporting five-run variation and training cost strengthens the evidence that the complete system is useful, even though some individual margins are small.
Critical weaknesses
-
The formal interpretation of the main readouts is not yet reconciled with the released implementation. Corollary D.9 requires the rollout proposal and the forward probability used in the residual to agree, with positive support on every legal edge; Appendix C.3 states the same assumption explicitly ("Sampling and scoring then use the same conditional policy"). The released paper configuration declares greedy action decoding, and the rollout implementation uses temperature zero for action generation, while residual scoring uses model probabilities. Unless the reported runs used a different configuration, the consistency argument for the residual-weighted share estimator does not directly apply, and the bounded-error guarantee of Corollary D.9(iii), which also presumes positive-support sampling, does not cover the greedy case. Off-policy balance training remains possible, but the readout requires a separate justification. In addition, Eq. (10) specifies a fixed continuation policy and a reference uniform over legal events, whereas
_signed_utilitycompares observed mean returns over observed events and leaves the utility undefined when fewer than two alternatives are observed. These differences matter because the readouts rank edits and can veto pruning; their interpretation as occupancy and relative benefit is central to the contribution. -
The evidence supporting reliable, independently verified evolution is insufficiently specified and calibrated. Appendix C lists tests, schema checks, assertions, and execution status, but does not adequately specify their task-level meaning or error rates across the different domains. The public repository additionally uses an external skill-author model to generate or revise skills and a separate external model to generate reference answers for verification in TriviaQA and AIME, without identifying either model or its budget in the reported experiments. Their capabilities and budgets could materially affect both the quality of edits and comparison fairness. Moreover, the released configuration uses a non-inferiority level of 0.2, absolute success/reward margins of 0.10, and token/latency margins corresponding to a factor of two. These choices need justification and should be stated in the paper. Proposition C.1 establishes that the specified gate is followed; it does not establish that verifier labels are accurate or that accepted edits reliably improve the system. The reported 33 beneficial edits out of 37 is encouraging, but needs this operational context.
Questions for authors
-
Estimator–sampler correspondence: Which decoding configuration produced the rollouts used for flow-share estimation? How does the probability in correspond to the actual behavior distribution under greedy action decoding? Please explain how Corollary D.9 applies and provide the effective sample sizes of the importance weights.
-
Signed utility: How are the fixed-continuation estimates and the uniform legal-event reference in Eq. (10) implemented? The released function appears to use observed returns and observed alternatives. What fraction of relevant states has sufficient alternative-action coverage, and how are unsupported utility estimates handled?
-
Verification and resource parity: Which models generated skill edits and reference answers in the reported runs? Please specify the verifier signal and available evidence for each domain, provide any existing assessment of verifier errors, and clarify whether competing methods received comparable authoring, verification, and compute resources.
-
Acceptance calibration: What motivates the released non-inferiority level and margins? Using existing paired outcomes, how sensitive are acceptance decisions and beneficial-edit precision to tighter settings? Please also clarify whether the 37 committed edits are pooled across the five runs or describe a particular run.