acceptodds
Desk rejected at ICLR 2027

Beyond the Best Skill: Router–Expert Coevolution for Adaptive Skill Trees

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents can improve by revising reusable textual skills without updating model weights. However, continued single-skill search can plateau despite useful local repairs. Our analysis identifies structural interference: repairs on some inputs are offset by regressions on others when candidates are evaluated as global replacements. We introduce RECAST (Router–Expert Coevolution for Adaptive Skill Trees), which expands an optimized Root skill into an adaptive skill tree by jointly evolving Expert skills and their routing scopes. Residual errors guide specialization, and Experts are refined through repair and harm evaluations relative to their parents on routed examples. Router selection uses end-to-end validation performance with the current Experts, while the selected Router defines the scope for further Expert refinement. Validation guides acceptance and recursive growth. At inference, the frozen tree executes the Root and follows a single routed path, returning the current node's output when routing stops. Across six benchmarks, two Qwen models, and Roots from GEPA and SkillOpt, RECAST improves all 24 paired configurations, with six-task macro-average gains of 12.82 to 15.69 percentage points. On three tasks with GPT-5.4, the macro-average gain is 15.09 percentage points. Ablations support joint adaptation of Routers and Experts, and depth analysis shows further gains from recursive specialization. Across six workflow comparisons, an upstream optimization prefix followed by RECAST achieves higher test performance with fewer optimization and validation tokens than completing upstream single-skill search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, skill evolution, router–expert coevolution, skill trees, agent optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kUmBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces RECAST, a framework that expands an optimized textual Root skill into an adaptive skill tree through Router–Expert coevolution. The paper argues that continued single-skill optimization can plateau because candidate revisions repair some inputs while harming others when applied globally. RECAST instead learns specialized Expert skills for residual error families and textual Routers that determine when those Experts should be applied. Expert refinement and Router reselection are alternated, and accepted Experts can be recursively specialized. Across six benchmarks, two Qwen models, and Roots produced by GEPA or SkillOpt, RECAST improves all 24 paired configurations; an additional three-task evaluation reports similar gains with GPT-5.4.

Critical strengths
  1. The problem is important and the central insight is clear. The observation that globally evaluated skill revisions can discard useful local repairs is intuitive, memorable, and directly motivates conditional specialization. Jointly optimizing what an Expert does and when it is applied is a coherent response to this failure mode.

  2. The reported gains are large and unusually consistent. RECAST improves every paired Root across six tasks, two target-model scales, and two upstream skill optimizers. The Router/Expert ablations and the depth analysis also provide initial evidence that conditional specialization and recursive growth are useful.

Critical weaknesses
  1. My main concern is that it is unclear where the gains come from. RECAST adds gpt-oss-120b as a Router and uses 3.48–7.24x more inference tokens than the Root baseline. The system-level gains are convincing, but without compute-matched verifier-retry, static routing, or best-of-N baselines, they cannot be attributed specifically to Router–Expert coevolution.

  2. The ablation does not fully isolate the benefit of coevolution. Full coevolution may use more proposals, validation evaluations, and optimization calls than the frozen variants. A budget-matched comparison is needed to distinguish dynamic coupling from simply performing more search.

  3. The method performs extensive selection on very small validation sets. Some tasks use only 18–53 validation examples for Router selection, Expert acceptance, tree growth, and stopping. Together with configuration-specific thresholds and stochastic skill generation, the absence of repeated runs makes it difficult to assess overfitting and stability.

Questions for authors
  1. Under the same Router model, number of Router/Executor calls, and approximate token or FLOP budget, how does RECAST compare with a fixed verifier-and-retry pipeline, static routed Experts, and self-consistency or best-of-N execution?

  2. How many Router/Expert candidates, validation evaluations, optimization tokens, and model calls are used by each condition in Table 3? Can the coevolution comparison be repeated with these budgets matched?

  3. How were the task-, model-, and Root-specific profiles in Table 7 selected? Were any thresholds or budgets changed after observing test results? Can one fixed profile be transferred across models or Roots without retuning?

  4. Across independent tree-construction runs with fixed hyperparameters, how variable are test performance, routing rates, tree depth, and the learned Expert families?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer kUmB (1/3)

Dear Reviewer kUmB,

We sincerely thank you for the thoughtful and constructive review. We appreciate your recognition of RECAST's central insight and consistent empirical improvements. To address your concerns regarding performance attribution, we conducted additional controlled experiments on Router capacity and inference-time computation.

W1 / Q1: Router Capacity and Compute-Matched Baselines

A1: Router capacity. We chose a heterogeneous Router to reduce correlated errors between task execution and trajectory evaluation. Our preliminary observations suggested that models evaluating their own trajectories tended to accept their own outputs more readily, consistent with prior research on LLM self-preference (Panickssery et al., 2024; Pombal et al., 2026).

We selected GPT-OSS-120B as a heterogeneous Router with reasoning capabilities comparable to Qwen3.6-35B-A3B on representative benchmarks. Official evaluations report 86.0%/21.4% on GPQA Diamond/HLE for Qwen3.6-35B-A3B, compared with 80.1%/14.9% for GPT-OSS-120B (high reasoning, no tools). Thus, larger parameter counts do not necessarily imply stronger reasoning capabilities.

To further examine Router dependence, we replaced GPT-OSS-120B with the substantially smaller GPT-OSS-20B for inference-time Router execution, keeping the optimized skill tree and other settings fixed.

Table R1. Router backbone sensitivity on SearchQA and LiveMath (test accuracy, %). Qwen3.5-4B Executor with SkillOpt Roots.

MethodSearchQALiveMath
SkillOpt Root71.5050.00
RECAST + 120B Router77.6470.16
RECAST + 20B Router76.5067.74

With the 20B Router, RECAST retains improvements of 5.00 and 17.74 percentage points over the Root, only 1.14 and 2.42 points below the 120B setting. These results show that RECAST preserves most of its gains with a substantially smaller Router.

A2: Inference-token-matched comparisons. We additionally evaluated strong alternatives under matched total inference-token budgets, using Qwen3.5-4B as the Executor and SkillOpt Roots. All baselines requiring external verification or answer selection use the same GPT-OSS-120B model.

The comparisons cover repeated sampling, majority voting, 120B answer selection, verification-guided retry, feedback revision, self-correction, and routing among existing upstream skill candidates.

Table R2. Inference-token-matched baseline comparisons on SearchQA and LiveMath (test accuracy, %).

MethodSearchQALiveMath
SkillOpt Root71.5050.00
Repeated Sampling + Majority Vote71.9352.42
Repeated Sampling + 120B Selection72.3652.42
120B Verification + Independent Retry72.6451.61
120B Verification + Feedback Revision72.2137.10
Qwen Self-Correction72.7952.42
Upstream Skill Routing (Fixed Selector)69.3650.81
Upstream Skill Routing (Learned Selector)72.0751.61
RECAST77.6470.16

RECAST achieves the highest accuracy on both benchmarks under matched total inference-token budgets.

On SearchQA, RECAST outperforms the strongest alternative by 4.86 percentage points, correctly answering 68 additional questions out of 1,400. On LiveMath, the advantage reaches 17.74 points, corresponding to 22 additional correct answers out of 124.

Importantly, neither fixed nor learned routing among existing upstream candidates reproduces RECAST's gains. This directly tests whether complementary capabilities discovered during upstream optimization can be exploited through simple candidate selection.

These results demonstrate that additional inference computation, repeated execution, and access to the same 120B verification model are insufficient to reproduce RECAST's performance under the evaluated matched-token settings. They provide direct empirical support for conditional Expert specialization beyond simply increasing test-time computation.

AuthorsOpenReview ↗

Reply to Reviewer kUmB (2/3)

W2 / Q2: Optimization Budget and Coevolution Ablations

A3: We thank the reviewer for requesting detailed resource accounting for the coevolution ablations. We additionally report the actual model calls and token consumption from the original Table 3 experiments.

The three conditions follow different optimization mechanisms:

  • Router-frozen: Keeps the initial Router fixed while continuing Expert refinement.
  • Expert-frozen: Keeps the initial Expert pool fixed while continuing Router optimization.
  • Full Coevolution: Alternates Expert refinement and Router reselection, allowing Expert capabilities and routing scopes to adapt jointly.

All conditions share a common optimization-token budget cap and follow predefined stopping criteria:

  • Expert optimization: Terminates upon reaching the maximum search rounds, producing no admissible update after the minimum required rounds, or exhausting valid new candidates.
  • Router–Expert alignment: Terminates when Router selection remains unchanged, validation routing behavior becomes equivalent, a selection cycle occurs, or the maximum alignment rounds are reached.
  • Budget termination: Stops when the next model call would exceed the common optimization-token budget cap, retaining the best validated state obtained so far.

Any applicable stopping condition can terminate the corresponding search. Therefore, sharing the same optimization-token budget cap does not imply identical actual token consumption, since different variants may terminate before exhausting the available budget.

Furthermore, fixed-component execution results are cached and reused by design, avoiding redundant model calls. The differences in actual consumption arise from each variant's optimization mechanism, cached execution results, and stopping conditions.

Table R3. Optimization resource usage across coevolution ablations. All conditions share the same optimization-token budget cap; reported values reflect actual model calls and token consumption.

ConditionQwen CallsQwen Tokens120B Calls120B TokensTotal Tokens
Full Coevolution1,3432,595,0181,6673,166,6195,761,637
Router-frozen2,3418,716,908008,716,908
Expert-frozen008,54323,984,34223,984,342

Full coevolution achieves superior performance while consuming the fewest actual aggregate optimization tokens.

Router-frozen and Expert-frozen consume 1.51× and 4.16× the total tokens of full coevolution, respectively.

Combined with Table 3, where full coevolution outperforms the stronger single-component variant by 8.54 points on 4B and 6.21 points on 35B, these results show that its performance advantage cannot be explained by greater aggregate optimization-token consumption.

The shared budget cap, predefined stopping conditions, and explicit model-specific usage statistics provide a transparent comparison of the actual optimization resources allocated by the different update strategies.

Additional evidence from recursive specialization. Figure 4 and Table 10 further examine the contribution of tree depth. On SearchQA with Qwen3.5-4B, depth-one Flat MoE deployment achieves 73.50% test accuracy, while the full depth-three RECAST tree achieves 77.64%, improving performance by 4.14 percentage points and correctly answering 58 additional test examples.

Together, these findings support the effectiveness of RECAST's two complementary mechanisms: jointly adapting Router and Expert capabilities, and recursively specializing Experts to address residual errors beyond the first layer.

AuthorsOpenReview ↗

Reply to Reviewer kUmB (3/3)

W3 / Q4: Validation Robustness and Independent Tree-Construction Stability

A4: Our original experiments follow the established SkillOpt train/validation/test splits to maintain comparability with upstream optimizers. To further examine the robustness of RECAST under stochastic skill generation and adaptive validation, we conducted additional validation-size and independent-run experiments.

Validation-size sensitivity. We expanded the SearchQA validation set from 200 to 500 examples, adding 300 previously unused questions without overlap with the existing train, validation, or test splits.

We kept the Qwen3.5-4B Executor, SkillOpt Root, training set, held-out test set, and hyperparameters fixed. For each validation size, we independently constructed three RECAST trees and evaluated them on the same 1,400-example test set.

Table R4. SearchQA validation-size sensitivity and independent-run stability. Test gains are measured against the same frozen SkillOpt Root (71.50% accuracy).

(a) Validation-size sensitivity: average test gains across independent constructions.

Validation Size# RunsMean Test Gain
2003+5.43 pp
5003+5.07 pp

(b) Independent-run results for the 200-example validation setting.

RunTest Gain# ExpertsMax DepthRouting Rate
Run 1+5.21 pp53447/1400 (31.93%)
Run 2+5.64 pp53453/1400 (32.36%)
Run 3+5.43 pp63448/1400 (32.00%)
Mean ± SD+5.43 ± 0.22 pp——32.10% mean

Across both validation sizes, all six independently constructed trees outperform the Root. Increasing validation size from 200 to 500 changes the mean test gain by only 0.36 percentage points.

The three runs under the original validation setting achieve consistent improvements of 5.43 ± 0.22 percentage points. Their tree structures also exhibit similar complexity, with 5–6 Experts and a maximum depth of 3.

Importantly, the overall routing rates are highly consistent across independent constructions. The three routing rates are 31.93%, 32.36%, and 32.00%, with a mean of 32.10% and a maximum difference of only 0.43 percentage points.

In RECAST, Expert families are induced from recurring training-error mechanisms, while family assignment relies on deployment-visible information (Section 4.2). Expert admission further requires minimum routed support, positive parent-relative repair/harm utility, bounded regressions, and non-decreasing composed-system validation performance.

These results demonstrate stable test improvements, overall routing rates, and learned tree complexity across independent constructions, together with consistent gains under substantially different validation-set sizes.

Q3: Hyperparameter Selection and Configuration Transfer

A5: We initialize thresholds and search budgets based on task data sizes, followed by limited validation-guided adjustments. Test results are never used for hyperparameter selection.

Importantly, many configurations already share identical settings. For example, ALFWorld uses the same routing thresholds, Expert admission criteria, and search budgets across both Qwen models and both GEPA/SkillOpt Roots, achieving consistent gains of 13.43–21.64 percentage points across all four configurations (Tables 1 and 7).

However, different models and Roots exhibit different residual error patterns, making a single threshold potentially too permissive or restrictive. For instance, OfficeQA/35B uses Root Router precision thresholds of 0.75 for GEPA and 0.65 for SkillOpt (Table 7). An overly low threshold may introduce unnecessary routing, while an overly high threshold may miss repairable errors.

We therefore use limited validation-guided adaptation to balance error coverage and routing precision, rather than independently tuning every configuration.

We thank the reviewer again for the constructive feedback. The additional experiments provide further evidence that RECAST's gains cannot be reproduced by stronger verification or additional inference computation alone, that full coevolution does not depend on greater aggregate optimization-token consumption, and that the resulting trees exhibit consistent performance, overall routing rates, and structural complexity across independent runs.

All additional experiments and analyses presented above will be incorporated into the revised manuscript.

Reviewer_dHwxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the challenge of continuously optimizing a single document skill. During skill optimization, the modifications will often bring conflicting effects on different application scenarios, and coordinating the skill across different application scenarios is of great importance. The paper proposes Recast, by extending the root skill into an expert tree with a router and activating the suitable expert when applicable, the method brings significant effectiveness improvement. On various benchmarks, with two backbone LLMs and two base skill optimizers, Recast consistently improves the success rate by at least 12.82%.

Critical strengths
  1. The problem is well-defined and supported with substantial experiments. To validate the concept that the skill modification may bring conflicts, the paper presents detailed analysis and data, which is very convincing.
  2. The method is highly generalizable. The paper includes abundant experiments across different skill optimizers, backbone models, and benchmarks, demonstrating the effectiveness of the proposed method. And the supplementary experiments are also abundant.
Critical weaknesses
  1. The contribution of the gpt-oss-120b router model is not well separated. Based on the same Qwen model, a much larger and more powerful router model could be a very important cause of the improved effectiveness, yet related experiments are not sufficient.
  2. Lack of comparison of performance with the same test-time budget. In the appendix, it is summarized that the proposed method takes many more tokens to finish the task; this raises the concern that the other baselines may also gain higher performance if they are allowed to use more tokens. For example, if they could have more trials for experiments.
Questions for authors
  1. In Table 4. how is the token cost summarized? Since there is a plateau for the GEPA and SkillOpt methods, there might be useless tokens counted towards the baselines, which makes the comparison unfair. Please clarify the stopping criteria for the baselines and how the switch to Recast is determined. Please provide a detailed cost-improvement breakdown table for the baseline and Recast. Also, the tokens of a gpt-oss-120b router and qwen base models clearly have different meanings; simply counting them is not convincing.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer dHwx (1/2)

Dear Reviewer dHwx,

We sincerely thank you for your positive assessment and constructive feedback. We appreciate your recognition of RECAST's problem formulation, generalizability, and extensive experimental evaluation. To address your concerns regarding Router capacity and inference-time computation, we conducted additional controlled experiments.

W1: Contribution of the GPT-OSS-120B Router

A1: We chose a heterogeneous Router to reduce correlated errors between task execution and trajectory evaluation. Our preliminary observations suggested that models evaluating their own trajectories tended to accept their outputs more readily, consistent with prior research on LLM self-preference (Panickssery et al., 2024; Pombal et al., 2026).

To directly examine whether RECAST depends on GPT-OSS-120B, we replaced it with the substantially smaller GPT-OSS-20B for inference-time Router execution, while keeping the optimized skill tree and other settings fixed.

Table R1. Router backbone sensitivity on SearchQA and LiveMath (test accuracy, %). Qwen3.5-4B Executor with SkillOpt Roots.

MethodSearchQALiveMath
SkillOpt Root71.5050.00
RECAST + 120B Router77.6470.16
RECAST + 20B Router76.5067.74

With the 20B Router, RECAST retains improvements of 5.00 and 17.74 percentage points over the Root, only 1.14 and 2.42 points below the 120B setting.

These results demonstrate that RECAST preserves most of its gains with a substantially smaller Router, showing that its inference-time effectiveness does not depend on GPT-OSS-120B specifically.

W2: Inference-Token-Matched Baselines

A2: We agree that comparisons against a single Root execution cannot fully distinguish the benefits of RECAST from those of additional inference computation.

We therefore conducted additional evaluations under matched total inference-token budgets, using Qwen3.5-4B as the Executor and SkillOpt Roots. All baselines requiring external verification or answer selection use the same GPT-OSS-120B model. Token accounting includes task execution, routing, verification, answer selection, and correction calls.

The comparisons cover repeated sampling, majority voting, 120B answer selection, verification-guided retry, feedback revision, self-correction, and routing among existing upstream skill candidates.

Table R2. Inference-token-matched baseline comparisons on SearchQA and LiveMath (test accuracy, %).

MethodSearchQALiveMath
SkillOpt Root71.5050.00
Repeated Sampling + Majority Vote71.9352.42
Repeated Sampling + 120B Selection72.3652.42
120B Verification + Independent Retry72.6451.61
120B Verification + Feedback Revision72.2137.10
Qwen Self-Correction72.7952.42
Upstream Skill Routing (Fixed Selector)69.3650.81
Upstream Skill Routing (Learned Selector)72.0751.61
RECAST77.6470.16

RECAST achieves the highest accuracy on both benchmarks under matched total inference-token budgets.

On SearchQA, RECAST outperforms the strongest alternative by 4.86 percentage points, correctly answering 68 additional questions out of 1,400. On LiveMath, the advantage reaches 17.74 points, corresponding to 22 additional correct answers out of 124.

Importantly, neither fixed nor learned selection among existing upstream skills reproduces RECAST's improvements, demonstrating that simply exploiting the existing candidate pool is insufficient.

These results show that additional inference computation, repeated execution, and access to the same 120B verification model cannot reproduce RECAST's gains under the evaluated matched-token settings. They provide direct empirical support for conditional Expert specialization beyond simply increasing test-time computation.

AuthorsOpenReview ↗

Reply to Reviewer dHwx (2/2)

Q1: Optimization Cost Accounting and Plateau-Based Budget Allocation

A3: We thank the reviewer for requesting clarification of the token accounting, upstream stopping criteria, and the transition from single-skill optimization to RECAST.

Plateau detection and switching criterion. We identify an optimization plateau when the best validation performance fails to improve for five consecutive optimization steps. This provides a validation-based criterion for determining when to redirect optimization resources toward RECAST.

Table 4 compares two workflows starting from the same upstream optimization prefix:

  1. Continued upstream optimization: GEPA or SkillOpt continues single-skill search beyond the detected plateau.
  2. Prefix + RECAST: Optimization switches from the upstream prefix to Router–Expert coevolution, allocating subsequent resources to conditional specialization.

Baseline budget and stopping. Rather than terminating the upstream baseline immediately upon detecting a plateau, we continue its search until its token consumption reaches or exceeds the reference budget of the prefix-plus-RECAST workflow.

This design directly examines whether allocating comparable or greater optimization resources to continued single-skill search can overcome the plateau.

The additional tokens correspond to actual candidate generation and validation evaluations. They represent computation genuinely spent investigating further improvements, rather than an artificially assigned penalty for the baseline.

Cost accounting. All reported costs include optimization and validation calls, excluding final test evaluation. For RECAST, total consumption includes both the upstream prefix used to obtain the starting Root and the subsequent Router–Expert construction and validation.

We provide detailed cost and performance comparisons below.

Table R5. Optimization cost and test performance for continued SkillOpt versus prefix-plus-RECAST. Qwen3.5-4B Executor; token counts include optimization and validation.

(a) Total optimization-token consumption and test performance.

BenchmarkContinued SkillOpt TokensPrefix + RECAST TokensSkillOpt TestRECAST Test
SearchQA25,285,10214,302,73671.50%77.64%
LiveMath3,070,8012,733,36050.00%70.16%

(b) Model-specific token breakdown for the prefix-plus-RECAST workflow.

BenchmarkQwen TokensGPT-OSS-120B TokensTotal Tokens
SearchQA10,512,3823,790,35414,302,736
LiveMath2,049,680683,6802,733,360

On SearchQA, the prefix-plus-RECAST workflow reduces total optimization tokens by 43.43%, while improving test accuracy by 6.14 percentage points.

On LiveMath, it reduces total optimization tokens by 10.99%, while improving test accuracy by 20.16 percentage points.

These results show that allocating further optimization resources to RECAST achieves substantially stronger performance than continuing single-skill search, despite consuming fewer total optimization tokens in the evaluated workflows.

Accounting for heterogeneous models. We agree that tokens generated by Qwen and GPT-OSS-120B should not be interpreted as equivalent computational units. Accordingly, Table R5(b) explicitly separates their consumption, and our reported efficiency improvements refer to actual aggregate token usage, rather than proportional reductions in FLOPs, execution time, or monetary cost.

We will clarify the plateau criterion, switching procedure, baseline continuation policy, and model-specific token accounting in the revised manuscript.

We thank the reviewer again for the constructive feedback. The additional experiments demonstrate that RECAST retains substantial gains with a smaller Router, outperforms strong inference-token-matched alternatives, and achieves better optimization outcomes by reallocating resources from continued single-skill search toward conditional specialization.

All additional experiments and analyses presented above will be incorporated into the revised manuscript.

AuthorsOpenReview ↗

Follow-up to Reviewer dHwx

Dear Reviewer dHwx,

Thank you again for your positive assessment and constructive suggestions. In response to your concerns, we have added experiments with a smaller GPT-OSS-20B Router and strong baselines under matched inference-token budgets. We have also clarified the plateau detection and switching criteria in Table 4, provided detailed optimization cost breakdowns, and separately reported Qwen and GPT-OSS-120B token consumption.

These additional experiments and clarifications strengthen the evidence for RECAST's effectiveness and provide a more transparent account of its computational costs. We have presented the corresponding results and explanations in our rebuttal.

As the discussion period approaches its end, we would greatly appreciate your feedback on whether our responses adequately address your concerns. We would be happy to provide any further clarification if needed.

Thank you again for your careful review and encouraging feedback.

Authors of Submission 53330

Reviewer_oc2ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RECAST expands an optimized textual skill into a tree of specialized experts. A router inspects the current execution and decides whether to retain its output or invoke a specialist. Experts and routing scopes are optimized alternately, and the procedure recursively grows deeper branches. The motivation is that globally replacing a skill can repair some examples while introducing regressions elsewhere. Conditional specialization aims to preserve those local improvements.

Critical strengths

The method follows from the diagnosis. The paper first establishes plateaus in single-skill search (Table 8), then shows that candidates repair errors locally but fail as global replacements (Fig. 2a), and quantifies the headroom with a per-input oracle (Fig. 2b). The parent-relative repair/harm admission rule follows directly from this analysis. The insight that skill optimization should adapt where a skill is applied, not only what it says, is useful beyond this specific method.

The gains are large and consistent, and the ablations support the mechanism. RECAST improves all 24 paired configurations across QA, spreadsheet, document, math and interactive-planning tasks, with a clean train/validation/test protocol. The coevolution ablation (Table 3) shows that full Router–Expert coevolution is best or tied-best in all 12 settings. It outperforms the stronger single-component variant by 8.54 points (4B) and 6.21 points (35B). The depth analysis (Fig. 4) further shows gains from recursive specialization.

Critical weaknesses
  1. The source of the gains is not isolated. (a) A much stronger model does the routing. In the Qwen experiments, routing and Router proposals use gpt-oss-120b (Table 6). The Router reads the full execution trajectory and decides whether the Root has failed, so it effectively acts as a 120B verifier. On Qwen3.5-4B it consumes more tokens per example than the Executor itself (7,341 vs. 5,872; Table 11). The claim that trees "self-optimized using open-source backbone models" outperform GPT-5.5-optimized baselines therefore does not hold as stated. (b) The baselines get far less inference compute. RECAST uses 7.24×, 3.48× and 5.24× the total inference tokens of the frozen Root (Table 11), but it is compared only against a single Root execution. Without inference-matched alternatives, it is unclear how much of the gain comes from coevolution itself. Such alternatives include repeated sampling with answer selection, error detection followed by retry, self-correction, and routing among existing candidate skills, using the same routing model where applicable. Routing among existing candidates is especially relevant, because the motivating analysis already locates complementary capability in the upstream candidate pool. (c) The ablations may not be budget-matched. Table 3 does not report whether the variants are matched in proposal count, optimization tokens, validation queries or final tree size, so the advantage of full coevolution may partly reflect greater search effort or capacity.

  2. The evidence relies on extensive adaptive selection over very small validation sets. (a) The decisions rest on very few examples. Validation sets contain only 18 examples for LiveMath and ALFWorld and 24 for OfficeQA (Table 5). These examples repeatedly determine Router selection, Expert admission, recursive growth and best-state retention, and several configurations allow a routed validation support of a single example (n_min = 1, Table 7a). (b) Thresholds are set per configuration without explanation. Thresholds and search budgets differ across all 24 configurations (Table 7), and the paper does not explain how they were chosen. (c) The search is stochastic, yet only single runs are reported. Proposal generation uses temperature 0.7, so the search is stochastic even though task execution is deterministic. Still, the paper reports single optimization runs with no uncertainty estimates. Together, these factors raise concerns about validation overfitting and structural instability. This does not suggest test leakage, but it makes repeated-run evidence necessary. (d) The motivating analysis is partly an artifact of selection. Since the Root is the best candidate on the same validation set, it is expected that no candidate surpasses it globally. The label-based oracle shows potential complementarity, but not how much of it is identifiable from deployment-visible evidence. Also, the LiveMath/TextGrad run in Table 8 reaches 100% validation accuracy, so its plateau reflects saturation rather than stagnation.

Questions for authors

Questions that could change my assessment:

  1. How much of RECAST's advantage remains against repeated sampling with answer selection, error detection followed by retry, self-correction, and routing among existing upstream candidates, under matched inference budgets and with the same routing model?
  2. What are the results when the Router (execution and proposal) uses the same model as the Executor?
  3. Are the coevolution ablations in Table 3 matched in proposal count, optimization tokens, validation queries and final tree size?
  4. How does RECAST compare with an independently optimized, budget-matched flat expert pool?
  5. Are the gains stable across independent optimization runs and different data splits, especially for tasks with 18-example validation sets? Please report the variance or confidence intervals.
  6. How were the per-configuration thresholds and search budgets in Table 7 chosen, and how do the results change under a single shared configuration?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer oc2Z (1/3)

Dear Reviewer,

We sincerely thank you for the thoughtful and constructive review. We appreciate your recognition of RECAST's motivation, consistent empirical gains, and the value of jointly optimizing skill behavior and application scope. We address your concerns with additional experiments and clarifications below.

W1(a) / Q2: Router Model Capacity

A1: We chose a heterogeneous Router to reduce correlated errors between task execution and trajectory evaluation. In preliminary experiments, models evaluating their own trajectories tended to accept their outputs more readily. This observation is consistent with prior studies on LLM self-preference (Panickssery et al., NeurIPS 2024; Pombal et al., 2026).

We selected GPT-OSS-120B as a heterogeneous Router with reasoning capabilities comparable to Qwen3.6-35B-A3B on representative benchmarks. Official evaluations report 86.0%/21.4% on GPQA Diamond/HLE for Qwen3.6-35B-A3B, versus 80.1%/14.9% for GPT-OSS-120B (high reasoning, no tools). Thus, a larger total parameter count does not necessarily imply stronger reasoning capability.

To further examine Router dependence, we replaced GPT-OSS-120B with the substantially smaller GPT-OSS-20B for inference-time Router execution, keeping the optimized skill tree and other settings fixed.

Table R1. Router backbone sensitivity on SearchQA and LiveMath (test accuracy, %). Qwen3.5-4B Executor with SkillOpt Roots.

MethodSearchQALiveMath
SkillOpt Root71.5050.00
RECAST + 120B Router77.6470.16
RECAST + 20B Router76.5067.74

With the 20B Router, RECAST retains gains of 5.00 and 17.74 percentage points over the Root, only 1.14 and 2.42 points below the 120B setting. These results demonstrate that RECAST retains most of its improvements with a much smaller heterogeneous Router.

We also agree that the comparison with GPT-5.5-optimized baselines should not be interpreted as the smaller Executor independently surpassing GPT-5.5 optimization. We will revise this statement to accurately reflect the heterogeneous Router's role.

W1(b) / Q1: Inference-Token-Matched Baselines

A2: We conducted additional comparisons under matched total inference-token budgets, using Qwen3.5-4B as the Executor and SkillOpt Roots. All baselines requiring external verification or answer selection use the same GPT-OSS-120B model.

We evaluate repeated sampling, answer selection, verification-guided retry, feedback revision, self-correction, and routing among existing upstream skill candidates.

Table R2. Inference-token-matched comparisons on SearchQA and LiveMath (test accuracy, %).

MethodSearchQALiveMath
SkillOpt Root71.5050.00
Repeated Sampling + Majority Vote71.9352.42
Repeated Sampling + 120B Selection72.3652.42
120B Verification + Independent Retry72.6451.61
120B Verification + Feedback Revision72.2137.10
Qwen Self-Correction72.7952.42
Upstream Skill Routing (Fixed Selector)69.3650.81
Upstream Skill Routing (Learned Selector)72.0751.61
RECAST77.6470.16

RECAST achieves the highest accuracy on both benchmarks under matched total inference-token budgets.

On SearchQA, RECAST outperforms the strongest alternative by 4.86 points, correctly answering 68 additional questions out of 1,400. On LiveMath, the advantage reaches 17.74 points, corresponding to 22 additional correct answers out of 124.

Importantly, neither fixed nor learned routing among existing upstream candidates reproduces RECAST's gains. This directly tests whether complementary capabilities discovered during upstream optimization can be exploited through simple candidate selection.

These results demonstrate that additional inference computation, repeated execution, and access to the same 120B verification model are insufficient to reproduce RECAST's performance under the evaluated matched-token settings. They directly support the value of conditional Expert specialization beyond simply increasing test-time computation.

AuthorsOpenReview ↗

Reply to Reviewer oc2Z (2/3)

W1(c) / Q3: Optimization Resources in Coevolution Ablations

A3: We thank the reviewer for requesting more detailed resource accounting. We additionally report the actual model calls and token consumption from the original Table 3 ablations.

The three conditions differ in their optimization mechanisms: Router-frozen continues Expert refinement with a fixed initial Router; Expert-frozen continues Router optimization with a fixed Expert pool; and Full Coevolution alternates Expert refinement and Router reselection.

All conditions share a common optimization-token budget cap and follow predefined stopping rules:

  • Expert optimization: Stops upon reaching the maximum search rounds, producing no admissible update after the minimum required rounds, or having no valid new candidates.
  • Router–Expert alignment: Stops when Router selection remains unchanged, validation routing behavior becomes equivalent, a selection cycle occurs, or the maximum alignment rounds are reached.
  • Budget termination: Stops if the next model call would exceed the common token budget cap, retaining the best validated state.

Any applicable stopping condition can terminate the corresponding search. Thus, a common budget cap does not imply identical actual token consumption, as different variants may terminate before exhausting the budget. Fixed-component execution results are also cached and reused by design, avoiding redundant model calls.

Table R3. Optimization resource usage across coevolution ablations. All variants share the same optimization-token budget cap; the table reports actual consumption.

ConditionQwen CallsQwen Tokens120B Calls120B TokensTotal Tokens
Full Coevolution1,3432,595,0181,6673,166,6195,761,637
Router-frozen2,3418,716,908008,716,908
Expert-frozen008,54323,984,34223,984,342

Full coevolution achieves superior performance while consuming the fewest actual aggregate optimization tokens.

Router-frozen and Expert-frozen consume 1.51× and 4.16× the total tokens of full coevolution, respectively. The different consumption patterns follow from the update mechanisms, cached execution results, and predefined stopping conditions.

Combined with Table 3, where full coevolution outperforms the strongest single-component variant by 8.54 points on 4B and 6.21 points on 35B, these results show that its advantage cannot be explained by greater aggregate optimization-token consumption.

Q4: Recursive Specialization versus Flat Experts

A4: We provide two complementary analyses of RECAST's design.

First, Table 3 demonstrates the benefits of jointly evolving Router scopes and Expert capabilities. The additional resource accounting in Table R3 shows that these improvements are achieved with fewer aggregate optimization tokens than either frozen-component variant.

Second, Figure 4 and Table 10 examine the contribution of recursive specialization through a depth-controlled deployment comparison. On SearchQA with Qwen3.5-4B, depth-one Flat MoE achieves 73.50% test accuracy, while the full depth-three tree reaches 77.64%, an improvement of 4.14 percentage points, corresponding to 58 additional correct predictions over 1,400 test examples.

Together, these analyses support two complementary contributions: Router–Expert coevolution improves conditional specialization, while recursive growth enables additional repairs beyond the first Expert layer.

AuthorsOpenReview ↗

Reply to Reviewer oc2Z (3/3)

W2(a), W2(c) / Q5: Validation-Size Sensitivity and Independent-Run Stability

A5: Our original experiments follow the established SkillOpt train/validation/test splits to maintain comparability with upstream optimizers. To further investigate validation robustness, we conducted additional experiments on SearchQA.

Validation-size sensitivity. We expanded the validation set from 200 to 500 examples, adding 300 previously unused questions without overlap with the existing train, validation, or test splits.

We kept the Qwen3.5-4B Executor, SkillOpt Root, training set, held-out test set, and hyperparameters fixed. For each validation size, we independently constructed three RECAST trees and evaluated them on the same 1,400-example test set.

Table R4. Validation-size sensitivity and independent-run stability on SearchQA. All gains are measured against the same frozen SkillOpt Root (71.50% test accuracy).

(a) Validation-size sensitivity: mean test gains across three independent constructions per setting.

Validation Size# RunsMean Test Gain
2003+5.43 pp
5003+5.07 pp

(b) Run-level results for the 200-example validation setting in Panel (a).

RunTest Gain# ExpertsMax Depth
Run 1+5.21 pp53
Run 2+5.64 pp53
Run 3+5.43 pp63
Mean ± SD+5.43 ± 0.22 pp——

Across both validation sizes, all six independently constructed trees outperform the Root. Increasing validation size from 200 to 500 changes the average test gain by only 0.36 percentage points.

Panel (b) further shows that the three runs under the original 200-example validation setting exhibit consistent improvements and similar tree complexity, with a test-gain standard deviation of 0.22 points, 5–6 Experts, and a maximum depth of 3.

We also emphasize that the minimum routed-support threshold in Equation 2 is only one component of Expert admission. Candidate Experts must additionally satisfy parent-relative net repair/harm requirements and non-decreasing composed-system validation performance.

These results demonstrate that RECAST maintains comparable performance gains with a substantially larger validation set and exhibits stable performance and tree complexity across independent constructions.

W2(b) / Q6: Hyperparameter Selection

A6: The thresholds and search budgets in Table 7 were initially configured based on task data sizes, followed by a small number of manual adjustments guided by validation performance. This was a heuristic validation-based tuning procedure rather than an exhaustive grid search.

Test results were never used to select or adjust thresholds, search budgets, or tree configurations. We will clarify this initialization and validation-guided adjustment procedure in the revised manuscript.

W2(d): Plateau Interpretation and Structural Interference

A7: We clarify that our plateau observation concerns the temporal progression of best-so-far validation performance, rather than merely comparing the selected Root against other candidates.

Figure 5 and Table 8 provide direct evidence: on SearchQA/Qwen3.5-4B, SkillOpt and GEPA reach their final best validation scores at steps 64 and 61, followed by 86 and 89 additional optimization steps without improvement, respectively.

Figure 2(a) further reveals structural interference: 37 of 39 SkillOpt candidates repair at least one Root error, but these gains are offset by regressions on other inputs. Figure 2(b) quantifies their potential complementarity through a label-based oracle, while RECAST learns to exploit this complementarity using deployment-visible execution information.

We will separately identify the LiveMath/TextGrad configuration reaching 100% validation accuracy as validation saturation.

Our central insight is that global skill replacement fails to preserve complementary local repairs, whereas Router–Expert coevolution jointly adapts skill behavior and application scope to accumulate these improvements.

We thank the reviewer again for the constructive feedback. The additional experiments further strengthen the empirical evidence for RECAST's effectiveness, the benefits of conditional specialization, and the performance and structural robustness of the resulting skill trees. All additional experiments and analyses presented above will be incorporated into the revised manuscript.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Request for Reviewer Follow-Up Before the Discussion Deadline

As the discussion deadline approaches, we would appreciate your help in encouraging reviewers oc2Z and dHwx to revisit our responses.

We have provided inference-token-matched comparisons, smaller-Router experiments, and detailed coevolution resource accounting to address the concerns of oc2Z, along with additional validation robustness and independent-run stability analyses. For dHwx, we have further clarified the plateau-based switching criteria and provided detailed optimization cost breakdowns, including model-specific token consumption.

We have not yet received follow-up feedback from these two reviewers and would greatly value the opportunity to clarify any remaining concerns before the discussion closes. If possible, could you kindly remind them to review our responses?

Thank you for your time and consideration!

Authors of Submission 53330

AuthorsOpenReview ↗

General Response

General Description

Dear Area Chair and Reviewers,

Thank you for your thoughtful feedback on RECAST. We appreciate the reviewers’ recognition of the motivating evidence for structural interference, the idea of jointly adapting what a skill does and when it is applied, and the consistent improvements across all 24 paired configurations. The discussion raised three main questions about the source of these gains, the fairness of the resource comparisons, and the stability of tree construction. We summarize the new evidence and manuscript revisions below.

Additional Experiments and Analyses

  1. Router capacity. We replaced gpt-oss-120b with gpt-oss-20b at inference while keeping the optimized skill tree and Qwen3.5-4B Executor fixed. With the 20B Router, RECAST still improves over the SkillOpt Root by 5.00 points on SearchQA and 17.74 points on LiveMath, within 1.14 and 2.42 points of the 120B setting. (Response Table R1 and revised Table 6.)

  2. Inference-token-matched baselines. We compared RECAST with repeated sampling, majority voting, answer selection, verification and retry, feedback revision, self-correction, and fixed or learned routing among existing upstream skills. The comparisons match total inference tokens and use the same gpt-oss-120b wherever an external verifier or selector is required. RECAST leads the strongest alternatives by 4.86 points on SearchQA and 17.74 points on LiveMath, corresponding to 68 and 22 additional correct answers, respectively. The upstream-skill routing controls test whether selecting among existing skills can recover these gains. (Response Table R2 and revised Table 7.)

  3. Coevolution resources and recursive specialization. In the SearchQA ablation, Full Coevolution uses 5.76M optimization tokens under the shared cap, compared with 8.72M for Router-frozen and 23.98M for Expert-frozen. The revised paper reports calls and tokens by model and explains the stopping and cache reuse rules. The deployment-depth analysis also shows that moving from a flat Expert layer to depth three improves accuracy from 73.50% to 77.64%, yielding 58 additional correct answers. (Response Table R3 and revised Tables 3, 5, and 13.)

  4. Validation size and construction stability. On SearchQA, we increased the validation set from 200 to 500 examples while keeping the Root, training and test sets, and hyperparameters fixed. Across three independent constructions per setting, mean gains over the Root were 5.43 and 5.07 points, respectively. At Val200, the gain was 5.43 ± 0.22 points. The three trees reached depth 3, retained 5–6 Experts, and routed 31.93–32.36% of test inputs to Experts. (Response Table R4 and revised Table 14.)

  5. Optimization workflow and cost accounting. We clarified that five consecutive steps without a new best validation score determine the switch to RECAST. The upstream-only baseline continues searching until it reaches or exceeds the prefix-plus-RECAST token budget. For the Qwen3.5-4B SkillOpt workflows, switching to RECAST reduced aggregate optimization and validation tokens by 43.43% on SearchQA and 10.99% on LiveMath, while improving test accuracy by 6.14 and 20.16 points. We report Qwen and gpt-oss-120b token counts separately for the prefix-plus-RECAST workflows. (Response Table R5 and revised Tables 4 and 15.)

Revisions to the Manuscript

  • Main paper. Section 5.4.2 now describes the plateau-based budget allocation and presents the model-specific ablation costs. Section 5.5 presents the smaller-Router results, the full inference-token-matched comparison, and the validation robustness results.
  • Appendices. Appendix C explains how thresholds and search budgets were initialized and adjusted using validation feedback, including the shared configuration across all four ALFWorld settings. Appendix D distinguishes continued-search stagnation from validation saturation and gives the depth and per-run stability results. Appendix E details the optimization-cost accounting and the inference baseline procedures.

Thank you again for helping us sharpen the experimental comparisons and clarify what each result establishes.

Sincerely,
Authors of Submission 53330

AuthorsOpenReview ↗

Correction to the 20B Router Experiment Setup

We would like to correct the description of the 20B Router experiment in our rebuttal. For the 20B condition, we rebuilt the skill trees from the SkillOpt Roots using gpt-oss-20b for both Router execution and Router revision proposals during optimization. We then used gpt-oss-20b for routing at test time. Qwen3.5-4B remained the Executor and Expert proposer throughout. The accuracies in response Table R1 remain 76.50% on SearchQA and 67.74% on LiveMath. We corrected Section 5.5 and Table 6 of the revised manuscript to reflect that the 20B Router was used during both construction and inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.