- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Meta-reasoning agent, a harness that makes decision-making a explicit part of test-time improving pipeline. The meta-reasoning agent is seperated into four parts for each cycle: assessing the controller state, propose the next batch of choices (i.e. experiments to run), evaluating these candidates and then dispatching workers for execution. Across 4 benchmarks and 3 models, meta-reasoning agents achieve higher score with matched budgets, demonstrating the benefits of this design decision. Furthermore, it exhibits scaling behavior with inference-time budget, and outperforms direct control agents. I find the artifact-graph analyses also quite interesting, for candidate coverage, quality assessment, and final selection.
Critical strengths
- The problem motivation is clear and has significant downstream applications. As agentic tasks get more complex and long-horizon, orchestrating these tasks to be effective and efficient is critical, and the method separates decision-making with workers, which has clear motivation and is a relatively clean concept.
- The experiments finding is insightful, providing useful information beyond final scores. I especially like the artifact graph which is a clear depiction of how meta-reasoning agents change the orchestrated structure for test-time improvments.
Critical weaknesses
- The staged approach for meta-reasoning agents deserve clear justification. Separating gains from this four-phase pipeline help compared to a one-phase alternative would help justify this component.
Questions for authors
- Can you provide an ablation on the four-phase pipeline against a single-stage controller which still maintains the same compact state and retrieval interface?
- I would like to better understand how this pipeline performances against classic est-time improving methods. Could the authors provide comparisons, especially empirically, against baselines such as AlphaEvolve and Reflexion?
3 comments
Thank you for your response and clarifications. I will maintain my positive rating.
Response to Reviewer Mfpp
We thank the reviewer for engaging with our response and for their positive rating. If any concerns remain, we would be glad to address them before the discussion period ends.
Response to Reviewer Mfpp
We thank the reviewer for the thoughtful review, for recognizing the problem's significant downstream applications, and for their appreciation of the artifact-graph analyses. We answer the first question with the requested ablation and the second by explaining how our Meta-Reasoning Agent (MRA) differs from fixed test-time methods.
Staged design (weakness and Q1). We ran the suggested ablation: a single-stage controller (SS-MRA) that keeps MRA's compact state, retrieval interface, workers, and instructions, but makes all decisions in a single agentic loop instead of four stages. MRA scores higher in 8 of the 11 settings, ties in 2, and trails by 2.1 points in 1 (General Response; Table 2, Appendix C). So, in most settings, the four-stage architecture contributes beyond the compact state and retrieval interface.
AlphaEvolve and Reflexion (Q2). We agree that comparisons with established test-time methods help place our results, but two differences make a direct comparison not like for like. First, these methods fix the structure of the computation in advance: Reflexion (Shinn et al., 2023) repeats an attempt–reflect–retry loop, Tree of Thoughts (Yao et al., 2023) runs a fixed tree search, Parallel-Distill-Refine (Madaan et al., 2025) runs a fixed number of draft-and-distill rounds, and AlphaEvolve (Novikov et al., 2025) runs an evolutionary loop. MRA instead decides at each step how many parallel branches to explore or whether to keep refining one line of work, based only on the work produced so far. Fixed-structure methods also depend on per-task settings, such as the number of trials, branches, or rounds, so a comparison at one budget would partly measure those choices. Our related-work section (Appendix A) now covers evolutionary search methods, including AlphaEvolve and ShinkaEvolve (Lange et al., 2025), alongside other fixed-structure methods.
Second, AlphaEvolve and Reflexion are built around feedback on every attempt, whereas MRA receives only the task and decides for itself when and how to check its work. AlphaEvolve needs an evaluator that scores the fitness of every candidate, but our benchmarks give the agent no evaluation signal beyond the task itself. Reflexion can use environment feedback, such as test results, which again gives it information MRA does not receive. With feedback the agent generates itself, Reflexion's loop becomes one of the strategies MRA's controller can already choose, such as asking a worker to revise an earlier attempt given a critique. We therefore compare MRA, at the same call budget, with systems that also organize their computation dynamically without an external evaluator. On all three reasoning benchmarks, MRA scores higher than RLM, a method that manages its own context and dynamically decomposes sub-tasks; on ProgramBench, it scores higher than the production coding agents Claude Code and Codex and the research harness mini-SWE-agent.