- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The submission is about moving a big model's reasoning ability into a small model without touching its weights. The idea: instead of fine-tuning, the teacher model designs an executable "strategy" for the student — a set of step-by-step programs (a DAG of subproblems, each with its own instruction and its own slice of the evidence). An interpreter runs the program by calling the frozen student step by step and passing intermediate results along.
Critical strengths
The paper proves where its gains come from, not just that it has gains. Its well.
Critical weaknesses
- The novelty claim is broader than the literature supports, and the paper's central comparison — offline structure search — is never actually made.
Each claimed ingredient already exists in prior work, uncited or unengaged: genetic operators + trace reflection + performance selection — GEPA[1]; offline search over executable workflow structure for small frozen models — AFlow[2]; trace-driven optimization of prompts and structure — Trace/OptoPrime[3], EvoFlow[4], MaAS[5]. What remains new is narrow: split/merge granularity operators co-evolved with subproblem instructions. Yet the two comparisons that would establish even this delta — ADaPT[6] (inference-time granularity adaptation) and unrestricted AFlow — are absent; AFlow appears only in a self-constructed restricted variant with incomplete disclosure.
- The deployment-cost story contradicts the method's own measurements.
The paper is motivated by cheap small-model deployment, but the method buys accuracy with inference-time calls: per-question total tokens exceed every prompt-level baseline (4.61k vs. 2.99k for MIPROv2 at 4B, Table 3), all calls are sequential (latency), and on the 9B student the gains largely vanish while the cost stays. The practical case is therefore only made for the two smallest students — the cheapest deployment tier now carries the highest per-question inference bill, which is the opposite of the motivation.
[1]Agrawal L A, Tan S, Soylu D, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning[C]//International Conference on Learning Representations. 2026, 2026: 8479-8565.
[2]Zhang J, Xiang J, Yu Z, et al. Aflow: Automating agentic workflow generation[C]//International Conference on Learning Representations. 2025, 2025: 34040-34077.
[3]Cheng C A, Nie A, Swaminathan A. Trace is the next autodiff: Generative optimization with rich feedback, execution traces, and llms[J]. Advances in Neural Information Processing Systems, 2024, 37: 71596-71642.
[4]Zhang G, Chen K, Wan G, et al. Evoflow: Evolving diverse agentic workflows on the fly[J]. arXiv preprint arXiv:2502.07373, 2025.
[5]Zhang G, Niu L, Fang J, et al. Multi-agent architecture search via agentic supernet[J]. arXiv preprint arXiv:2502.04180, 2025.
[6]Prasad A, Koller A, Hartmann M, et al. Adapt: As-needed decomposition and planning with language models[C]//Findings of the Association for Computational Linguistics: NAACL 2024. 2024: 4226-4252.
Questions for authors
Refer to "Critical Weaknesses"
2 comments
Thank you for the response. I’m not fully convinced that my main concerns have been addressed, and I’ll keep my original score.
Thank you for the comments on the contribution, comparisons, and deployment motivation.
1. Research motivation, contribution, and comparisons.
Our central motivation is to improve a frozen student model, whose parameters remain fixed, by jointly adapting problem decomposition and task instructions to its execution capabilities. When a smaller student struggles to coordinate the original multi-step reasoning task within a single call, our hypothesis is that dividing it into simpler subproblems, each handled through a separate call, may improve performance. We evaluate recurring compositional task families, where a learned strategy can be reused on unseen instances with related reasoning structure. Methodologically, we jointly evolve explicit subproblem programs and shared solver instructions, using student execution feedback to adapt decomposition granularity and local reasoning.
AFlow, EvoFlow, and MaAS address automated workflow and architecture design; GEPA and Trace provide prompt and program optimization mechanisms. Within this area, EvoScaffold focuses on jointly evolving problem decomposition and solver instructions to match each subproblem’s difficulty to the execution capabilities of a fixed small student under restricted resource access. The resulting workflow encodes this learned problem-solving strategy. ADaPT similarly adapts decomposition during individual problem-solving instances; EvoScaffold performs this adaptation through offline search and reuses the learned strategies across instances.
For a fair comparison, AFlow is evaluated under the same task-solving resource constraints as EvoScaffold and the other baselines. As specified in Appendix A.5, network access, external tools, additional solving models, and programmatic semantic reasoning or direct answer computation are disabled. Structural search, control flow, parsing, student-based review or ensembling, and shared non-semantic evidence utilities remain available. This keeps task-relevant reasoning within the target frozen student. Table 1 already reports this AFlow comparison on COMMAQA-E, PhantomWiki, ExploreToM, and FinChain across all four student scales.
The matched fixed-program control S0 further quantifies the value of structural adaptation. With shared initialization, budget ceiling, and non-structural optimization permissions, Full improves over S0 by a mean of 7.37 pp on 0.8B and 3.19 pp on 4B across three paired COMMAQA-E searches. The corresponding 95% paired intervals are [4.26, 10.49] and [−0.26, 6.64]. Detailed controls and results appear in our response to yLEd.
2. Deployment setting and measured inference cost.
Our intended setting is local deployment where a larger model cannot be hosted and the available small model cannot be fine-tuned. The objective is to improve this frozen student on recurring tasks through offline teacher-guided strategy construction and student-only execution.
We expanded the accounting in Table 3 and Appendix D.3 from completed, auditable executions to include failed attempts, truncated generations, retries, and format fallbacks. The resulting COMMAQA-E measurements are:
Each row represents one frozen strategy evaluated once on all 669 test questions; token usage is the per-question mean.
At 0.8B, EvoScaffold uses fewer inference tokens and has lower p95 latency than both prompt baselines. The increased baseline usage mainly reflects truncated outputs and format-fallback attempts on failed examples; EvoScaffold’s simpler local calls incur less of this overhead. These measurements directly support the intended small-student deployment case.
At 4B, EvoScaffold uses more tokens and has higher median latency, while its p95 latency is lower. Median latency and offline optimization costs are reported in our response to GJG7. Together with the reduced accuracy gains on 9B, these results support our central motivation: EvoScaffold is particularly beneficial for small students that struggle to coordinate multi-step reasoning, while larger students can already handle these reasoning and coordination demands more effectively, leaving less room for additional gains from decomposition.