- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces RECAST, a framework that expands an optimized textual Root skill into an adaptive skill tree through Router–Expert coevolution. The paper argues that continued single-skill optimization can plateau because candidate revisions repair some inputs while harming others when applied globally. RECAST instead learns specialized Expert skills for residual error families and textual Routers that determine when those Experts should be applied. Expert refinement and Router reselection are alternated, and accepted Experts can be recursively specialized. Across six benchmarks, two Qwen models, and Roots produced by GEPA or SkillOpt, RECAST improves all 24 paired configurations; an additional three-task evaluation reports similar gains with GPT-5.4.
Critical strengths
-
The problem is important and the central insight is clear. The observation that globally evaluated skill revisions can discard useful local repairs is intuitive, memorable, and directly motivates conditional specialization. Jointly optimizing what an Expert does and when it is applied is a coherent response to this failure mode.
-
The reported gains are large and unusually consistent. RECAST improves every paired Root across six tasks, two target-model scales, and two upstream skill optimizers. The Router/Expert ablations and the depth analysis also provide initial evidence that conditional specialization and recursive growth are useful.
Critical weaknesses
-
My main concern is that it is unclear where the gains come from. RECAST adds gpt-oss-120b as a Router and uses 3.48–7.24x more inference tokens than the Root baseline. The system-level gains are convincing, but without compute-matched verifier-retry, static routing, or best-of-N baselines, they cannot be attributed specifically to Router–Expert coevolution.
-
The ablation does not fully isolate the benefit of coevolution. Full coevolution may use more proposals, validation evaluations, and optimization calls than the frozen variants. A budget-matched comparison is needed to distinguish dynamic coupling from simply performing more search.
-
The method performs extensive selection on very small validation sets. Some tasks use only 18–53 validation examples for Router selection, Expert acceptance, tree growth, and stopping. Together with configuration-specific thresholds and stochastic skill generation, the absence of repeated runs makes it difficult to assess overfitting and stability.
Questions for authors
-
Under the same Router model, number of Router/Executor calls, and approximate token or FLOP budget, how does RECAST compare with a fixed verifier-and-retry pipeline, static routed Experts, and self-consistency or best-of-N execution?
-
How many Router/Expert candidates, validation evaluations, optimization tokens, and model calls are used by each condition in Table 3? Can the coevolution comparison be repeated with these budgets matched?
-
How were the task-, model-, and Root-specific profiles in Table 7 selected? Were any thresholds or budgets changed after observing test results? Can one fixed profile be transferred across models or Roots without retuning?
-
Across independent tree-construction runs with fixed hyperparameters, how variable are test performance, routing rates, tree depth, and the learned Expert families?
3 comments
Reply to Reviewer kUmB (2/3)
W2 / Q2: Optimization Budget and Coevolution Ablations
A3: We thank the reviewer for requesting detailed resource accounting for the coevolution ablations. We additionally report the actual model calls and token consumption from the original Table 3 experiments.
The three conditions follow different optimization mechanisms:
- Router-frozen: Keeps the initial Router fixed while continuing Expert refinement.
- Expert-frozen: Keeps the initial Expert pool fixed while continuing Router optimization.
- Full Coevolution: Alternates Expert refinement and Router reselection, allowing Expert capabilities and routing scopes to adapt jointly.
All conditions share a common optimization-token budget cap and follow predefined stopping criteria:
- Expert optimization: Terminates upon reaching the maximum search rounds, producing no admissible update after the minimum required rounds, or exhausting valid new candidates.
- Router–Expert alignment: Terminates when Router selection remains unchanged, validation routing behavior becomes equivalent, a selection cycle occurs, or the maximum alignment rounds are reached.
- Budget termination: Stops when the next model call would exceed the common optimization-token budget cap, retaining the best validated state obtained so far.
Any applicable stopping condition can terminate the corresponding search. Therefore, sharing the same optimization-token budget cap does not imply identical actual token consumption, since different variants may terminate before exhausting the available budget.
Furthermore, fixed-component execution results are cached and reused by design, avoiding redundant model calls. The differences in actual consumption arise from each variant's optimization mechanism, cached execution results, and stopping conditions.
Table R3. Optimization resource usage across coevolution ablations. All conditions share the same optimization-token budget cap; reported values reflect actual model calls and token consumption.
| Condition | Qwen Calls | Qwen Tokens | 120B Calls | 120B Tokens | Total Tokens |
|---|---|---|---|---|---|
| Full Coevolution | 1,343 | 2,595,018 | 1,667 | 3,166,619 | 5,761,637 |
| Router-frozen | 2,341 | 8,716,908 | 0 | 0 | 8,716,908 |
| Expert-frozen | 0 | 0 | 8,543 | 23,984,342 | 23,984,342 |
Full coevolution achieves superior performance while consuming the fewest actual aggregate optimization tokens.
Router-frozen and Expert-frozen consume 1.51× and 4.16× the total tokens of full coevolution, respectively.
Combined with Table 3, where full coevolution outperforms the stronger single-component variant by 8.54 points on 4B and 6.21 points on 35B, these results show that its performance advantage cannot be explained by greater aggregate optimization-token consumption.
The shared budget cap, predefined stopping conditions, and explicit model-specific usage statistics provide a transparent comparison of the actual optimization resources allocated by the different update strategies.
Additional evidence from recursive specialization. Figure 4 and Table 10 further examine the contribution of tree depth. On SearchQA with Qwen3.5-4B, depth-one Flat MoE deployment achieves 73.50% test accuracy, while the full depth-three RECAST tree achieves 77.64%, improving performance by 4.14 percentage points and correctly answering 58 additional test examples.
Together, these findings support the effectiveness of RECAST's two complementary mechanisms: jointly adapting Router and Expert capabilities, and recursively specializing Experts to address residual errors beyond the first layer.
Reply to Reviewer kUmB (3/3)
W3 / Q4: Validation Robustness and Independent Tree-Construction Stability
A4: Our original experiments follow the established SkillOpt train/validation/test splits to maintain comparability with upstream optimizers. To further examine the robustness of RECAST under stochastic skill generation and adaptive validation, we conducted additional validation-size and independent-run experiments.
Validation-size sensitivity. We expanded the SearchQA validation set from 200 to 500 examples, adding 300 previously unused questions without overlap with the existing train, validation, or test splits.
We kept the Qwen3.5-4B Executor, SkillOpt Root, training set, held-out test set, and hyperparameters fixed. For each validation size, we independently constructed three RECAST trees and evaluated them on the same 1,400-example test set.
Table R4. SearchQA validation-size sensitivity and independent-run stability. Test gains are measured against the same frozen SkillOpt Root (71.50% accuracy).
(a) Validation-size sensitivity: average test gains across independent constructions.
| Validation Size | # Runs | Mean Test Gain |
|---|---|---|
| 200 | 3 | +5.43 pp |
| 500 | 3 | +5.07 pp |
(b) Independent-run results for the 200-example validation setting.
| Run | Test Gain | # Experts | Max Depth | Routing Rate |
|---|---|---|---|---|
| Run 1 | +5.21 pp | 5 | 3 | 447/1400 (31.93%) |
| Run 2 | +5.64 pp | 5 | 3 | 453/1400 (32.36%) |
| Run 3 | +5.43 pp | 6 | 3 | 448/1400 (32.00%) |
| Mean ± SD | +5.43 ± 0.22 pp | — | — | 32.10% mean |
Across both validation sizes, all six independently constructed trees outperform the Root. Increasing validation size from 200 to 500 changes the mean test gain by only 0.36 percentage points.
The three runs under the original validation setting achieve consistent improvements of 5.43 ± 0.22 percentage points. Their tree structures also exhibit similar complexity, with 5–6 Experts and a maximum depth of 3.
Importantly, the overall routing rates are highly consistent across independent constructions. The three routing rates are 31.93%, 32.36%, and 32.00%, with a mean of 32.10% and a maximum difference of only 0.43 percentage points.
In RECAST, Expert families are induced from recurring training-error mechanisms, while family assignment relies on deployment-visible information (Section 4.2). Expert admission further requires minimum routed support, positive parent-relative repair/harm utility, bounded regressions, and non-decreasing composed-system validation performance.
These results demonstrate stable test improvements, overall routing rates, and learned tree complexity across independent constructions, together with consistent gains under substantially different validation-set sizes.
Q3: Hyperparameter Selection and Configuration Transfer
A5: We initialize thresholds and search budgets based on task data sizes, followed by limited validation-guided adjustments. Test results are never used for hyperparameter selection.
Importantly, many configurations already share identical settings. For example, ALFWorld uses the same routing thresholds, Expert admission criteria, and search budgets across both Qwen models and both GEPA/SkillOpt Roots, achieving consistent gains of 13.43–21.64 percentage points across all four configurations (Tables 1 and 7).
However, different models and Roots exhibit different residual error patterns, making a single threshold potentially too permissive or restrictive. For instance, OfficeQA/35B uses Root Router precision thresholds of 0.75 for GEPA and 0.65 for SkillOpt (Table 7). An overly low threshold may introduce unnecessary routing, while an overly high threshold may miss repairable errors.
We therefore use limited validation-guided adaptation to balance error coverage and routing precision, rather than independently tuning every configuration.
We thank the reviewer again for the constructive feedback. The additional experiments provide further evidence that RECAST's gains cannot be reproduced by stronger verification or additional inference computation alone, that full coevolution does not depend on greater aggregate optimization-token consumption, and that the resulting trees exhibit consistent performance, overall routing rates, and structural complexity across independent runs.
All additional experiments and analyses presented above will be incorporated into the revised manuscript.
Reply to Reviewer kUmB (1/3)
Dear Reviewer kUmB,
We sincerely thank you for the thoughtful and constructive review. We appreciate your recognition of RECAST's central insight and consistent empirical improvements. To address your concerns regarding performance attribution, we conducted additional controlled experiments on Router capacity and inference-time computation.
W1 / Q1: Router Capacity and Compute-Matched Baselines
A1: Router capacity. We chose a heterogeneous Router to reduce correlated errors between task execution and trajectory evaluation. Our preliminary observations suggested that models evaluating their own trajectories tended to accept their own outputs more readily, consistent with prior research on LLM self-preference (Panickssery et al., 2024; Pombal et al., 2026).
We selected GPT-OSS-120B as a heterogeneous Router with reasoning capabilities comparable to Qwen3.6-35B-A3B on representative benchmarks. Official evaluations report 86.0%/21.4% on GPQA Diamond/HLE for Qwen3.6-35B-A3B, compared with 80.1%/14.9% for GPT-OSS-120B (high reasoning, no tools). Thus, larger parameter counts do not necessarily imply stronger reasoning capabilities.
To further examine Router dependence, we replaced GPT-OSS-120B with the substantially smaller GPT-OSS-20B for inference-time Router execution, keeping the optimized skill tree and other settings fixed.
Table R1. Router backbone sensitivity on SearchQA and LiveMath (test accuracy, %). Qwen3.5-4B Executor with SkillOpt Roots.
With the 20B Router, RECAST retains improvements of 5.00 and 17.74 percentage points over the Root, only 1.14 and 2.42 points below the 120B setting. These results show that RECAST preserves most of its gains with a substantially smaller Router.
A2: Inference-token-matched comparisons. We additionally evaluated strong alternatives under matched total inference-token budgets, using Qwen3.5-4B as the Executor and SkillOpt Roots. All baselines requiring external verification or answer selection use the same GPT-OSS-120B model.
The comparisons cover repeated sampling, majority voting, 120B answer selection, verification-guided retry, feedback revision, self-correction, and routing among existing upstream skill candidates.
Table R2. Inference-token-matched baseline comparisons on SearchQA and LiveMath (test accuracy, %).
RECAST achieves the highest accuracy on both benchmarks under matched total inference-token budgets.
On SearchQA, RECAST outperforms the strongest alternative by 4.86 percentage points, correctly answering 68 additional questions out of 1,400. On LiveMath, the advantage reaches 17.74 points, corresponding to 22 additional correct answers out of 124.
Importantly, neither fixed nor learned routing among existing upstream candidates reproduces RECAST's gains. This directly tests whether complementary capabilities discovered during upstream optimization can be exploited through simple candidate selection.
These results demonstrate that additional inference computation, repeated execution, and access to the same 120B verification model are insufficient to reproduce RECAST's performance under the evaluated matched-token settings. They provide direct empirical support for conditional Expert specialization beyond simply increasing test-time computation.