acceptodds
Under review as a conference paper at ICLR 2027

EvoScaffold: Distilling Executable Strategies for Small Language Models

Abstract

Reasoning distillation transfers the reasoning capabilities of large teacher models to smaller students for more affordable deployment, typically through supervised fine-tuning. Prompt-level distillation avoids student parameter updates by encoding teacher guidance in system instructions, but complex multi-step problems can still exceed what the student can reliably solve in a single call. We introduce EvoScaffold, a non-parametric distillation framework that jointly evolves problem decomposition and subproblem instructions through genetic search for a frozen student. The resulting executable strategies encode reusable reasoning programs that specify student calls and any dependencies between them. Student predictions and execution traces guide teacher-generated revisions, while student task performance governs strategy selection. Once selected, a strategy is reused on unseen instances of the same task family without teacher access or student parameter updates. Experiments across four reasoning tasks and four student sizes demonstrate the effectiveness of EvoScaffold in improving the reasoning performance of frozen students.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM, Prompt Engineering, Distillation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_kwFpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The submission is about moving a big model's reasoning ability into a small model without touching its weights. The idea: instead of fine-tuning, the teacher model designs an executable "strategy" for the student — a set of step-by-step programs (a DAG of subproblems, each with its own instruction and its own slice of the evidence). An interpreter runs the program by calling the frozen student step by step and passing intermediate results along.

Critical strengths

The paper proves where its gains come from, not just that it has gains. Its well.

Critical weaknesses
  1. The novelty claim is broader than the literature supports, and the paper's central comparison — offline structure search — is never actually made.

Each claimed ingredient already exists in prior work, uncited or unengaged: genetic operators + trace reflection + performance selection — GEPA[1]; offline search over executable workflow structure for small frozen models — AFlow[2]; trace-driven optimization of prompts and structure — Trace/OptoPrime[3], EvoFlow[4], MaAS[5]. What remains new is narrow: split/merge granularity operators co-evolved with subproblem instructions. Yet the two comparisons that would establish even this delta — ADaPT[6] (inference-time granularity adaptation) and unrestricted AFlow — are absent; AFlow appears only in a self-constructed restricted variant with incomplete disclosure.

  1. The deployment-cost story contradicts the method's own measurements.

The paper is motivated by cheap small-model deployment, but the method buys accuracy with inference-time calls: per-question total tokens exceed every prompt-level baseline (4.61k vs. 2.99k for MIPROv2 at 4B, Table 3), all calls are sequential (latency), and on the 9B student the gains largely vanish while the cost stays. The practical case is therefore only made for the two smallest students — the cheapest deployment tier now carries the highest per-question inference bill, which is the opposite of the motivation.

[1]Agrawal L A, Tan S, Soylu D, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning[C]//International Conference on Learning Representations. 2026, 2026: 8479-8565.

[2]Zhang J, Xiang J, Yu Z, et al. Aflow: Automating agentic workflow generation[C]//International Conference on Learning Representations. 2025, 2025: 34040-34077.

[3]Cheng C A, Nie A, Swaminathan A. Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llms[J]. Advances in Neural Information Processing Systems, 2024, 37: 71596-71642.

[4]Zhang G, Chen K, Wan G, et al. Evoflow: Evolving diverse agentic workflows on the fly[J]. arXiv preprint arXiv:2502.07373, 2025.

[5]Zhang G, Niu L, Fang J, et al. Multi-agent architecture search via agentic supernet[J]. arXiv preprint arXiv:2502.04180, 2025.

[6]Prasad A, Koller A, Hartmann M, et al. Adapt: As-needed decomposition and planning with language models[C]//Findings of the Association for Computational Linguistics: NAACL 2024. 2024: 4226-4252.

Questions for authors

Refer to "Critical Weaknesses"

2 comments

Thank you for the comments on the contribution, comparisons, and deployment motivation.

1. Research motivation, contribution, and comparisons.

Our central motivation is to improve a frozen student model, whose parameters remain fixed, by jointly adapting problem decomposition and task instructions to its execution capabilities. When a smaller student struggles to coordinate the original multi-step reasoning task within a single call, our hypothesis is that dividing it into simpler subproblems, each handled through a separate call, may improve performance. We evaluate recurring compositional task families, where a learned strategy can be reused on unseen instances with related reasoning structure. Methodologically, we jointly evolve explicit subproblem programs and shared solver instructions, using student execution feedback to adapt decomposition granularity and local reasoning.

AFlow, EvoFlow, and MaAS address automated workflow and architecture design; GEPA and Trace provide prompt and program optimization mechanisms. Within this area, EvoScaffold focuses on jointly evolving problem decomposition and solver instructions to match each subproblem’s difficulty to the execution capabilities of a fixed small student under restricted resource access. The resulting workflow encodes this learned problem-solving strategy. ADaPT similarly adapts decomposition during individual problem-solving instances; EvoScaffold performs this adaptation through offline search and reuses the learned strategies across instances.

For a fair comparison, AFlow is evaluated under the same task-solving resource constraints as EvoScaffold and the other baselines. As specified in Appendix A.5, network access, external tools, additional solving models, and programmatic semantic reasoning or direct answer computation are disabled. Structural search, control flow, parsing, student-based review or ensembling, and shared non-semantic evidence utilities remain available. This keeps task-relevant reasoning within the target frozen student. Table 1 already reports this AFlow comparison on COMMAQA-E, PhantomWiki, ExploreToM, and FinChain across all four student scales.

The matched fixed-program control S0 further quantifies the value of structural adaptation. With shared initialization, budget ceiling, and non-structural optimization permissions, Full improves over S0 by a mean of 7.37 pp on 0.8B and 3.19 pp on 4B across three paired COMMAQA-E searches. The corresponding 95% paired intervals are [4.26, 10.49] and [−0.26, 6.64]. Detailed controls and results appear in our response to yLEd.

2. Deployment setting and measured inference cost.

Our intended setting is local deployment where a larger model cannot be hosted and the available small model cannot be fine-tuned. The objective is to improve this frozen student on recurring tasks through offline teacher-guided strategy construction and student-only execution.

We expanded the accounting in Table 3 and Appendix D.3 from completed, auditable executions to include failed attempts, truncated generations, retries, and format fallbacks. The resulting COMMAQA-E measurements are:

StudentMethodActual inference tokens/questionEnd-to-end p95 latency (s)
0.8BGEPA9,805.853.73
0.8BMIPROv28,050.388.79
0.8BEvoScaffold4,039.229.90
4BGEPA4,281.736.72
4BMIPROv23,798.740.20
4BEvoScaffold4,620.727.95

Each row represents one frozen strategy evaluated once on all 669 test questions; token usage is the per-question mean.

At 0.8B, EvoScaffold uses fewer inference tokens and has lower p95 latency than both prompt baselines. The increased baseline usage mainly reflects truncated outputs and format-fallback attempts on failed examples; EvoScaffold’s simpler local calls incur less of this overhead. These measurements directly support the intended small-student deployment case.

At 4B, EvoScaffold uses more tokens and has higher median latency, while its p95 latency is lower. Median latency and offline optimization costs are reported in our response to GJG7. Together with the reduced accuracy gains on 9B, these results support our central motivation: EvoScaffold is particularly beneficial for small students that struggle to coordinate multi-step reasoning, while larger students can already handle these reasoning and coordination demands more effectively, leaving less room for additional gains from decomposition.

Thank you for the response. I’m not fully convinced that my main concerns have been addressed, and I’ll keep my original score.

Reviewer_aLoyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes EvoScaffold to help small language models solve reasoning tasks without updating their parameters. A teacher model designs reusable workflows that break a task into smaller steps, each handled by a separate student call. The method uses the student’s answers and execution records to revise both the workflow and the instructions for each step. Candidate workflows are evaluated and selected based on student performance. The selected workflow is then used on new questions from the same task family without further teacher involvement.

Critical strengths
  1. The proposed method shows consistent improvements on the small size models, outperforming the evaluated non-parametric baselines across all four tasks.

  2. The paper provides a lot of experimental results and detailed documentation, including complete workflows, prompts, revision examples, and failure cases in the appendix.

Critical weaknesses
  1. The novelty appears limited. Executable strategy distillation uses a teacher model to design external workflows, and it is unclear to me how this differs methodologically from existing approaches such as ADAS and AFlow.

  2. In Table 2, Cold start only already improves performance, and Prompt-only evolution performs close to Full EvoScaffold, especially on the 4B model. The larger gap is on the smallest 0.8B model. These results leave me uncertain about the added value of jointly evolving decomposition and instructions, which is presented as one of the main contributions of this work.

  3. Since the main gains are on smaller models, the latency overhead of the additional model calls is important for assessing practical feasibility. I think measuring end-to-end latency would help assess the method’s practical feasibility.

Questions for authors

See Weaknesses.

3 comments

Thank you for the constructive comments on the methodological contribution and practical feasibility.

1. Contribution and relation to ADAS/AFlow.

EvoScaffold is closely related to automated workflow search. ADAS explores code-defined agents using an archive of previous designs, and AFlow uses MCTS to optimize workflow code and prompts.

Our contribution concerns adapting the decomposition of the original reasoning problem to a frozen small student’s capabilities. The representation separates subproblem programs and dependencies from shared solver instructions. Constrained evolutionary operations adjust subproblem boundaries, execution granularity, and associated instructions using student execution feedback. This makes local task difficulty and coordination across subproblems explicit elements of strategy adaptation. We will sharpen this positioning in the manuscript.

2. Added value beyond initialization and fixed-program optimization.

Initialization already incorporates student feedback and provides useful strategies. We assessed the subsequent contribution of program adaptation using S0, a matched fixed-program population control. Full and S0 share initialization and the search-budget ceiling, and both can optimize instructions, evidence policies, and execution bounds.

The following results use three paired searches on COMMAQA-E and report mean ± sample SD:

StudentS0 EM (%)Full EM (%)Mean gain (pp)
0.8B29.90 ± 3.1437.27 ± 3.73+7.37
4B90.98 ± 0.8894.17 ± 0.83+3.19

The 95% paired intervals are [4.26, 10.49] for 0.8B and [−0.26, 6.64] for 4B. The larger observed gain on 0.8B aligns with our intended setting, where adapting the decomposition to the student offers the greatest opportunity to improve local reasoning. The detailed control specification appears in our response to yLEd.

3. End-to-end latency.

We measured per-question latency on COMMAQA-E:

StudentGEPA p50 / p95 (s)MIPROv2 p50 / p95 (s)EvoScaffold p50 / p95 (s)
0.8B23.23 / 53.7330.84 / 88.7916.39 / 29.90
4B15.26 / 36.7211.93 / 40.2018.81 / 27.95

Each entry comes from one frozen strategy evaluated on all 669 test questions. EvoScaffold has lower median and p95 latency at 0.8B. At 4B, its median latency is higher and its p95 is lower. These measurements further support the practical motivation for the smaller-student setting. Our response to GJG7 provides the accompanying token usage and offline student optimization costs.

Dear Reviewers,

With two days remaining in the rebuttal period, we would appreciate your feedback on our responses and additional experimental results. If these have addressed your concerns, we kindly ask you to reconsider your assessment and score accordingly. Please let us know if any questions remain so that we can address them before the deadline. Thank you for your time and consideration.

Thank the authors for the additional experiments. While my concerns about methodological novelty remain, I am raising my recommendation to weak accept.

Reviewer_M9zaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a non-parametric reasoning distillation approach, EvoScaffold. The method distills the teacher model’s problem-solving guidance into a reusable strategy that assists a frozen student model. The strategy consists of a routing instruction, reasoning programs, and shared solver modules. The strategy is learned through a joint evolutionary search guided by student execution feedback, that optimizes problem decomposition and subproblem instructions. The authors evaluate EvoScaffold on selected task families from four reasoning benchmarks using four student model sizes, with comparisons against parametric and non-parametric baselines and ablation studies.

Critical strengths

(S1) Jointly optimizing problem decomposition and solver instructions is an interesting methodological contribution. It allows the reasoning steps and their prompts to be adapted together to a frozen student's capabilities, rather than optimizing instructions within a fixed decomposition.

Critical weaknesses

(W1) Limit scope of generalization: The evaluation mainly tests instances within narrow task families. For example, for COMMAQA-E, the authors only retain four three-hop templates from the official movie-domain data, and FinChain uses a single cash-flow template. While the reasoning programs work in the specific families, they provide limited evidence that the learned strategies remain useful when the reasoning structure or context deviates from the original scope.

Questions for authors

(Q1) The authors choose a subset of questions in full datasets. The selected question types seem easy to express as reusable reasoning programs, and with clear, standardized subproblem solver modules. I wonder whether the reported gains of EvoScaffold could partially come from the task selection that fit the design? If this potential impact exists, how do we rule out it? (Q2) EvoScaffold uses multiple student calls per question than other baselines. Do the gains over the prompt-based baselines still hold at a comparable student inference cost?

6 comments

Thank you for recognizing the value of jointly adapting decomposition and solver instructions to the student.

1. Task selection and the scope of the conclusions.

Our evaluation targets recurring compositional task families, where a reusable strategy is learned and applied to unseen instances with related reasoning structure. This follows the problem setup in §3.1, and Appendix A.2 specifies the selected families and data construction.

We appreciate the concern that the measured gains can depend on the selected task families. Our current experiments provide evidence for student-adapted decomposition within this setting. We will make this scope more explicit in the main claims and discuss how changes in reasoning structure may affect strategy reuse.

2. Student inference cost.

Table 3 and Appendix D.3 originally reported completed, auditable executions, excluding engineering retries and output-format fallbacks. We expanded the COMMAQA-E cost accounting to include failed attempts, truncated generations, retries, and format fallbacks:

StudentGEPA tokens/questionMIPROv2 tokens/questionEvoScaffold tokens/question
0.8B9,805.88,050.34,039.2
4B4,281.73,798.74,620.7

Each entry averages inference usage over one complete 669-question evaluation of a frozen strategy.

At 0.8B, EvoScaffold uses 58.8% fewer inference tokens than GEPA and 49.8% fewer than MIPROv2. The increased baseline usage mainly reflects truncated outputs and format-fallback attempts on failed examples. At 4B, EvoScaffold’s token usage is 7.9% higher than GEPA and 21.6% higher than MIPROv2.

This resource profile supports our focus on small students: dividing a difficult task into simpler local calls can reduce failure-related overhead, improving execution efficiency in the weaker-student setting. The practical benefit therefore depends on student capacity. End-to-end latency and offline optimization costs are provided in our response to GJG7.

Dear Reviewers,

With two days remaining in the rebuttal period, we would appreciate your feedback on our responses and additional experimental results. If these have addressed your concerns, we kindly ask you to reconsider your assessment and score accordingly. Please let us know if any questions remain so that we can address them before the deadline. Thank you for your time and consideration.

Thanks the authors for the detailed response and clarifications. Here are some follow-ups:

  1. Thanks for the explanation on the dataset settings. My concern in Q1 remains because the additional experiments do not test how much gains depend on the selected task families. I appreciate the plan to make the scope of the claims more explicit.

  2. The updated cost accounting partially addressed Q2 and supports an efficiency benefit for the 0.8B student on COMMAQA-E. At 4B, token usage and median latency are higher than GEPA and MIPROv2, though P95 latency is lower. The added S0 control also strengthens the evidence for decomposition evolution, especially at 0.8B.

I still find the joint design of decomposition and solver instructions interesting. However, the results remain limited to the selected task families, and how much the gains depend on this selection is still unclear to me. I would keep my current score.

Reviewer_yLEdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EVOSCAFFOLD, a non-parametric distillation approach for frozen student LLMs. Instead of updating model parameters, it learns reusable executable strategies that combine routing instructions, decomposition programs, module-level solver prompts, and explicit dependencies across calls. Candidate strategies are improved through constrained genetic operations (mutation/crossover/split/merge), with acceptance based on student-side execution performance. The evaluation covers 4 reasoning tasks and 4 student scales (0.8B/2B/4B/9B), with both parametric and non-parametric baselines. The paper reports clear gains in several small-student settings and includes ablations on execution structure, operator variants, and cross-student transfer.

Critical strengths

The practical target is important: improving frozen small students without parameter updates is a real constraint in deployment. In that context, the gains on weaker students are meaningful. The execution-focused ablations are also a plus. The large degradation under flattened execution suggests the proposed strategy structure is not just cosmetic prompt rewriting.

Critical weaknesses

My main concern is attribution. The paper argues for the value of jointly evolving decomposition and instructions, but the key comparison does not isolate this cleanly: multiple knobs (search diversity, editable elements, and execution-related controls) move together. A second concern is reliability of strict rankings. Many decisive comparisons are single-run point estimates, and some margins are narrow. Without repeated-search variability or uncertainty estimates, the paper’s strongest comparative wording feels a bit too assertive. I also think evidence-access policy is a likely major driver of outcomes, yet this is not fully controlled in the headline narrative. As written, the paper appears to over-credit decomposition/search mechanics when some gains may come from selective evidence presentation. Finally, baseline fairness is still somewhat ambiguous in the restricted workflow setup. I understand the motivation, but it leaves room for interpretation about whether gains come from the proposed strategy search itself or from interface asymmetries.

Questions for authors

Could you provide a fully matched instruction-only population baseline (same initialization, budget, and editable controls) to isolate decomposition search without confounds? Please specify exactly which evidence-access and orchestration interfaces each baseline was allowed to use, ideally in a compact fairness-audit table. Can you report repeated-search variability (or paired uncertainty) for the close-margin comparisons used to support strict ranking claims?

2 comments

Thank you for the suggestions on matched controls, interface disclosure, and repeated-search variability.

1. Matched control for structural adaptation.

Our main setting targets frozen small students solving multi-step tasks within an 8,192-token window per call, covering input and output. Full-evidence execution preserves the learned decomposition and supplies the complete task material to every solver. Under this budget, input truncation leaves incomplete evidence, while the increased context burden further reduces local execution reliability. These effects explain the sharp performance drop in the original ablation.

To examine the role of this context constraint, we added a 32K full-evidence execution condition on COMMAQA-E. The 8K entries below reproduce the original Table 2 results.

ConfigurationContext window0.8B EM (%)4B EM (%)
EvoScaffold with learned evidence policies8K38.7193.27
Full-evidence execution8K11.6624.66
Full-evidence execution — new ablation32K36.3292.67

Expanding the context brings full-evidence execution within 2.39 pp and 0.60 pp of the original Full results. This substantial recovery demonstrates why evidence presentation matters for reliable execution under our limited-context setting.

To determine whether evolving decomposition provides additional gains when both conditions can optimize evidence presentation, we introduced S0, a matched fixed-program population control. Within each student and seed, Full and S0 share the initialization pool and starting population, data splits, search schedule, acceptance and selection rules, and teacher-budget ceiling.

Both conditions allow optimization of router and solver instructions, evidence policies, and execution bounds, together with compatible instruction crossover. S0 fixes program definitions, dependencies, and subquestion templates.

The following results use 669 COMMAQA-E test questions and three completed paired searches. Scores are mean ± sample SD; confidence intervals are paired, seed-level t intervals.

StudentS0 EM (%)Full EM (%)Mean paired gain (pp)95% CI of gain (pp)
0.8B29.90 ± 3.1437.27 ± 3.73+7.37[4.26, 10.49]
4B90.98 ± 0.8894.17 ± 0.83+3.19[−0.26, 6.64]

Full improves over S0 in all three paired runs at each scale. The larger gain on 0.8B supports our motivation for adapting decomposition and instructions to students with more limited reasoning capacity.

2. Evidence access and orchestration interfaces.

In the main comparisons, all methods share the public task evidence, data splits, scoring interface, and 8,192-token student context window. Their evidence interfaces are:

Method/configurationEvidence at inferenceHost-provided evidence interfaces
GEPA / MIPROv2Complete public materialNo chunking, bound-mention, or set-union API
PLD / SELF-DISCOVERComplete public materialNo chunking, bound-mention, or set-union API
EvoScaffold / S0Complete material or strategy-selected local viewsChunking, bound-mention views, and non-semantic set union
AFlow (restricted)Complete material, with local views availableThe same non-semantic evidence-access and aggregation utilities as EvoScaffold (Appendix A.5)

Prompt baselines can select evidence through internal reasoning; PLD’s embeddings are used during offline optimization. AFlow permits ordinary control flow, parsing, and student-based review or ensembling. External tools, network access, additional task-solving models, and programmatic semantic reasoning or direct answer computation are disabled, as specified in Appendix A.5.

3. Repeated searches for the close-margin comparisons.

Following your focus on close-margin comparisons, we repeated FinChain against MIPROv2 at 0.8B and 2B. These are the two smallest advantages in the original Table 1 over the strongest non-parametric baseline within our target 0.8B/2B regime, making them directly relevant to the small-student claim.

For each configuration, we conducted three complete optimization searches per method and evaluated each selected strategy on all 512 test questions. Scores are mean ± sample SD; differences are computed before rounding.

StudentMIPROv2 accuracy (%)EvoScaffold accuracy (%)Difference in means (pp)
0.8B80.01 ± 1.9782.49 ± 2.15+2.47
2B90.82 ± 0.7092.38 ± 1.09+1.56

EvoScaffold achieves higher mean accuracy at both scales. These results provide repeated-search variability for the closest baseline comparisons supporting our small-student claim, complementing the matched Full–S0 analysis.

Dear Reviewers,

With two days remaining in the rebuttal period, we would appreciate your feedback on our responses and additional experimental results. If these have addressed your concerns, we kindly ask you to reconsider your assessment and score accordingly. Please let us know if any questions remain so that we can address them before the deadline. Thank you for your time and consideration.

Reviewer_GJG7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EvoScaffold, a way to distill a teacher's reasoning into a frozen small student without weight updates. Instead of a single system prompt, the teacher builds an executable strategy: a router plus DAG programs whose nodes are student calls with their own instructions and evidence-access policies. Strategies are evolved with mutation, crossover, split and merge, using student traces as feedback and student accuracy as fitness. With DeepSeek-V4-Pro as teacher and Qwen3.5 0.8B–9B as students, EvoScaffold outperforms the non-parametric baselines on four tasks for the 0.8B and 2B students, with mixed results at 4B/9B.

Critical strengths
  1. For the 0.8B and 2B students, EvoScaffold outperforms all non-parametric baselines in all 8 settings (e.g., COMMAQA-E 0.8B: 38.71 vs. 29.45 for GEPA). This is the main evidence that students too weak to carry out multi-step reasoning in one call.

  2. The selection set is separate from the training data used for teacher feedback, and the teacher-token budget is matched across methods.

Critical weaknesses
  • It is unclear how much of the gain comes from evolving the decomposition, which is the paper's central claim. At 4B, the cold-start strategy with no evolution (87.74) already beats every baseline, and full vs. prompt-only evolution differs by only 1.64 points. Removing the learned evidence filtering drops accuracy from 93.27 to 24.66, so filtering may matter more than decomposition.

  • The method underperforms with larger students. On FinChain 4B/9B, EvoScaffold gets 90–93% while the prompt baselines reach 98–100%, and on ExploreToM 4B it is about 6 points below GEPA.

  • EvoScaffold uses more tokens per question than the prompt baselines, the averages exclude failed or retried executions, and latency and student-side search cost are not reported, so the efficiency claim is hard to assess.

Questions for authors

On FinChain 4B/9B, EvoScaffold gets 90–93% while the prompt baselines reach 98–100%. Why does the search not select a simpler strategy in these settings?

2 comments

Thank you for the helpful comments on the contribution of structural evolution, model capacity, and deployment cost.

1. Contribution of decomposition evolution.

Our central motivation is to improve a frozen student model, whose parameters remain fixed, by jointly adapting problem decomposition and task instructions to its execution capabilities. When a smaller student struggles to coordinate the original multi-step reasoning task within a single call, our hypothesis is that dividing it into simpler subproblems, each handled through a separate call, may improve performance.

Both the cold-start and prompt-only ablations use initial strategies generated by the teacher using student execution feedback, so their decompositions already incorporate some adaptation to the student. Cold start selects among these strategies; prompt-only further evolves the instructions while fixing the selected decomposition. The strong 4B initialization (87.74%) suggests that an effective decomposition is already available, leaving less room for further structural adaptation. The 0.8B cold-start score is 22.27%, suggesting greater scope for adapting the decomposition and instructions to this student’s more limited execution capabilities.

We also added S0, a matched fixed-program population control. Full and S0 share the initial population, budget ceiling, and selection rules; both can optimize instructions, evidence policies, and execution bounds. Across three paired searches on COMMAQA-E:

StudentS0 EM (%)Full EM (%)Mean gain (pp)95% paired CI (pp)
0.8B29.90 ± 3.1437.27 ± 3.73+7.37[4.26, 10.49]
4B90.98 ± 0.8894.17 ± 0.83+3.19[−0.26, 6.64]

Scores are mean ± sample SD. The larger observed gain on 0.8B supports our motivation for adapting decomposition to weaker students.

Evidence access also contributes to effective execution. The full-evidence ablation preserves the learned decomposition and replaces local evidence views with the complete material, showing the strategy’s dependence on these views.

2. Stronger students and simpler strategies.

Stronger students can already solve these tasks effectively with an optimized prompt, so additional decomposition may introduce intermediate errors and coordination overhead. Our finite search emphasizes decomposition-oriented candidates, which can limit its coverage of strong single-node solutions.

We examined this by adding a GEPA-optimized prompt as a single-node candidate to the initial population:

StudentSelected strategy on ExploreToMSelected strategy on FinChain
0.8BDecomposed strategyDecomposed strategy
4BGEPA single-node strategy: 85.34%GEPA single-node strategy: 99.02%

The search thus retains strong simple strategies when they are available.

3. Inference usage, latency, and offline student cost.

We expanded the cost accounting to include failed attempts, truncated generations, retries, and format fallbacks. Table 3 and Appendix D.3 originally reported completed, auditable executions and excluded engineering retries and output-format fallbacks.

The expanded COMMAQA-E measurements are:

StudentMethodInference tokens/questionEnd-to-end p50 / p95 (s)Offline student optimization tokens
0.8BGEPA9,805.823.23 / 53.7320,423,016
0.8BMIPROv28,050.330.84 / 88.7918,009,273
0.8BEvoScaffold4,039.216.39 / 29.9021,789,224
4BGEPA4,281.715.26 / 36.7213,535,070
4BMIPROv23,798.711.93 / 40.2010,262,946
4BEvoScaffold4,620.718.81 / 27.9514,521,428

Each online entry represents one frozen strategy evaluated once on all 669 test questions. Offline usage corresponds to one optimization search, including initialization, candidate evaluation, and final selection.

The larger 0.8B costs for GEPA and MIPROv2 mainly reflect truncated outputs and format-fallback attempts on failed examples. EvoScaffold’s simpler local calls incur less of this overhead. At 4B, its token usage and median latency are higher, with a lower p95 latency. This scale-dependent resource profile further supports our intended small-student setting, where decomposition can reduce failure-related execution overhead. The selected strategies can then be reused across instances of the same task family.

Dear Reviewer,

With two days remaining in the rebuttal period, we would appreciate your feedback on our responses and additional experimental results. If these have addressed your concerns, we kindly ask you to reconsider your assessment and score accordingly. Please let us know if any questions remain so that we can address them before the deadline. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.