acceptodds
Under review as a conference paper at ICLR 2027

Solver.EXE: Learning to Solve Expert-level Problems via EXploration and Exploitation at Test-time

Abstract

An expert develops mastery not only from textbooks, but also through practice and refinement. We show that LLMs can do the same at test time: with the right self-evolution procedure, small- to medium-sized models can more fully realize their reasoning potential and grow into expert-level solvers. Motivated by the classical exploration-exploitation tradeoff, we introduce Solver.EXE, a multi-agent test-time learning framework built around a single frozen backbone and an online-evolving memory. Solver.EXE turns each reasoning problem into a miniature learning environment: a Reasoner exploits the current memory to produce rollouts, an Explorer expands the strategy space by surfacing new solutions, an Extractor distills insights and warnings from both kinds of rollouts, and a Curator consolidates them into a compact, evolving memory. Over several expert-level reasoning benchmarks spanning research-level mathematics and broader STEM, Solver.EXE consistently outperforms exploitation-only self-learning baselines and enables 30B-level open-source backbones (Gemma4-26B-A4B-it and Qwen3.5-35B-A3B) to match or surpass proprietary models, including Claude Opus-4.6 and GPT-5.2. Our analysis traces these gains to two latent capabilities of the backbone: bold strategy exploration and careful insight exploitation. These findings suggest that inference-time orchestration is a powerful and still largely untapped axis for advancing reasoning abilities.

Primary area:
generative models
Keywords:
Large Language Models, Test-Time Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PqBxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether test-time learning can improve a model without repeatedly reinforcing the same solution strategies. Solver.EXE combines exploitation and exploration through four agents: a Reasoner uses the evolving memory, an Explorer searches for alternative strategies, an Extractor distills insights and warnings, and a Curator maintains the memory. Experiments on mathematics, STEM, and coding tasks show improvements over the backbone and exploitation-based baselines.

Critical strengths
  • The paper presents a clear test-time learning pipeline, separating memory-guided solving, strategy exploration, memory extraction, and memory curation.
  • The paper reports results across several backbones and task types, and provides prompts, pseudocode, and local runtime measurements that help readers understand the setup.
Critical weaknesses
  • The main math benchmarks are small: Apex has 12 problems and ArxivMath has 32. Apex is also selected from problems that several frontier models fail. The four-run standard deviations only measure sampling variation. Please report per-problem uncertainty and narrow the frontier-model claims if needed.
  • The benefit of structured exploration is not fully clear. In Table 9, Solver.EXE improves over Simple Explore by only 1.4 points after two rounds, without error bars. Are these differences significant, and which problems account for the gain?
  • The ablations do not isolate all components. The paper removes the Explorer and Curator, but not the Extractor, the two memory types, or the DEVIATE/DRAFT modes. The “w/o Explorer” setting also reallocates its rollouts to the Reasoner, which mixes exploration effects with budget allocation.
  • The exploration–exploitation framing is somewhat loose. RL is not inherently exploratory, and prior methods also sample multiple stochastic trajectories. Moreover, if the frozen backbone cannot find a useful new strategy, the Explorer may produce paraphrases or invalid plans; Figure 5 suggests that exploration can hurt easy problems. Please quantify useful exploratory rollouts and consider an adaptive fallback.
  • The cost comparison is difficult to interpret because Solver.EXE uses extra calls for exploration, extraction, and curation, while reference models use different sampling budgets and API defaults. Please report total tokens, calls, and latency.
Questions for authors

See weaknesses section.

4 comments
AuthorsOpenReview ↗

Rebuttal-Part1

We sincerely appreciate your recognition of the clarity of our method and the strong empirical gains it achieves. We hope our responses below address your concerns.


W1: Please report per-problem uncertainty and narrow the frontier-model claims if needed.

A: Thank you for your suggestion.

Per-problem uncertainty. We report per-problem comparisons in Table 9 with 95% Newcombe–Wilson confidence intervals. After Holm correction, the directional Fisher exact tests favor Solver.EXE on 3 Apex and 5 ArxivMath problems, and favor Opus-4.6 on 2 and 5 problems, respectively. Notably, with test-time orchestration, Solver.EXE produced correct answers on 3 Apex problems and 2 ArxivMath problems for which Opus-4.6 produced none.

Scale of benchmark. Our paper has also included evaluations on HLE-Math and HLE-STEM with 200 problems, and GPQA-Diamond with 198 problems. Across these benchmarks, Solver.EXE improves over both the backbone and test-time learning baseline, while achieving performance comparable to the referenced model results.

MethodsHLE-MathHLE-STEMGPQA-Diamond
Opus-4.5 (High)27.226.087.0
Kimi-K2.5 (Thinking)24.725.287.4
Qwen3.5-35B-A3B14.116.680.9
w/ RSE23.022.485.2
w/ Solver.EXE27.526.488.7

Scope of claim. We have tightened the wording of the relevant claims, clarifying that open-source backbones equipped with Solver.EXE can achieve frontier-comparable performance on certain benchmarks. Our core claim remains that Solver.EXE consistently improves over both the raw backbone and prior test-time learning baselines.


W2: Are the differences between Simple Explore and Solver.EXE significant, and which problems account for the gain?

A: Yes, and the benefit of structured exploration is substantial. In Table 10, we have extended the comparison between Simple Explore and Solver.EXE from two to four evolution rounds. The results are shown as follows:

ArxivMath

MethodIter 1Iter 2Iter 3Iter 4
Fixed Prompting30.0 (0.55)32.1 (0.49)32.7 (0.62)32.8 (0.67)
Diverse Prompting30.7 (0.20)32.5 (0.54)32.9 (0.40)32.9 (0.57)
Simple Explore30.5 (0.96)32.8 (0.44)33.5 (0.33)33.6 (0.61)
Solver.EXE30.8 (0.28)34.2 (0.49)36.5 (0.41)37.2 (0.34)

Apex

MethodIter 1Iter 2Iter 3Iter 4
Fixed Prompting16.7 (1.22)26.1 (1.50)29.1 (0.86)29.4 (0.76)
Diverse Prompting14.6 (0.41)27.9 (0.45)30.8 (0.67)31.8 (0.69)
Simple Explore23.3 (0.49)30.0 (0.74)31.2 (0.81)32.2 (0.52)
Solver.EXE26.3 (1.52)31.3 (1.48)33.2 (1.12)36.5 (1.08)

After four rounds, Solver.EXE leads Simple Explore by +3.6 percentage points on ArxivMath and +4.3 points on Apex. The gains come primarily from 4 medium-difficulty problems (baseline Pass@1∈(201\in (20%, 8080%]]) and 7 challenging problems (base Pass@1 ≤20\le 20%). This suggests that our structured exploration could effectively guide the model to discover useful strategies on problems it struggles with and thus raise the performance ceiling.


W3: The ablations do not isolate all components.

A: We have added the ablation results using Qwen3.5-35B-A3B and show them below.

ArxivMath

MethodIter 1Iter 2Iter 3Iter 4
Solver.EXE30.834.236.537.2
w/o Extractor29.631.032.833.0
w/o Insights30.332.335.636.1
w/o Warnings30.032.635.736.4
w/o Deviate30.733.335.235.8
w/o Draft30.433.034.335.3
w/o Explorer (no rollout reallocation)29.930.831.931.9

Apex

MethodIter 1Iter 2Iter 3Iter 4
Solver.EXE26.331.333.236.5
w/o Extractor13.520.025.329.2
w/o Insights20.027.332.832.0
w/o Warnings20.828.630.731.2
w/o Deviate24.630.132.334.9
w/o Draft23.428.931.933.7
w/o Explorer (no rollout reallocation)15.423.426.127.1

Across both benchmarks, the ablations suggest that the Extractor, both memory types, and both exploration modes contribute to Solver.EXE's performance.

AuthorsOpenReview ↗

Rebuttal-Part2

W4: Prior methods also sample multiple stochastic trajectories. Please quantify useful exploratory rollouts and consider an adaptive fallback.

A: Thank you for your comment.

Benefit of structured exploration. The reason we borrow the concept of the exploration–exploitation trade-off from RL [1] is that we observed an important limitation in prior test-time learning (TTL) methods:

  • In RL [2], external rewards guide subsequent updates to the model's output distribution.
  • In prior TTL methods without external rewards, stochastic sampling alone offers limited guidance on how to avoid repeating earlier strategies or mistakes.

We therefore developed our structured exploration method to guide the model to discover alternative solution strategies both effectively and efficiently. To show that, we compare our structured exploration method with prior stochastic methods with Qwen3.5-35B-A3B as follows:

ArxivMath

MethodPass@16Pass@32Pass@64Pass@128
Stochastic trajectories45.347.448.450.0
Explorer48.156.2——
Reasoner50.356.2——

Apex

MethodPass@16Pass@32Pass@64Pass@128
Stochastic trajectories31.238.441.441.7
Explorer37.341.7——
Reasoner41.550.0——

On ArxivMath, our structured exploration at 32 rollouts outperforms stochastic sampling at 128 rollouts (56.2 vs. 50.0). On Apex, the Explorer matches the backbone's Pass@128 with 32 rollouts and solves two problems the backbone misses. These results demonstrate that our structured exploration has the potential to discover useful information for solving these problems.

Useful exploration quantification. To quantify this, we use GPT-6.1 Sol (medium) to annotate 704 Explorer rollouts across four iterations of Solver.EXE. We label a rollout as useful if it introduces a solution approach distinct from previously attempted strategies and either outlines an actionable plan or makes nontrivial progress toward a solution. The table reports the percentage of rollouts labeled useful at each iteration.

IterationArxivMathApex
Iter 185.2%77.1%
Iter 284.4%83.3%
Iter 391.4%79.2%
Iter 482.8%85.4%

On both benchmarks, more than 3/4 of the Explorer rollouts are labeled useful at every iteration. For reference, the corresponding rate for Simple Explore is 33.8%. This suggests that the Explorer continues to find new ways to make progress even in later rounds.

Adaptive fallback. To implement an adaptive fallback mechanism, at the exploration step, we first sample 4 Explorer rollouts and ask the model to judge whether each produces a plan that is both useful and valid. If fewer than 2/4 rollouts meet this criterion, we skip further Explorer sampling. The table below reports the results.

ArxivMath

IterationAdaptive Pass@1 (%)Solver.EXE Pass@1 (%)Token change per problem
Iter 131.330.8−70.0k
Iter 233.734.2−86.9k
Iter 336.936.5−88.3k
Iter 437.137.2−57.2k

Apex

IterationAdaptive Pass@1 (%)Solver.EXE Pass@1 (%)Token change per problem
Iter 126.026.3−86.7k
Iter 231.531.3−65.5k
Iter 332.333.2−97.2k
Iter 434.936.5−79.5k

The adaptive fallback reduces token usage per problem while keeping Pass@1 close to that of the full Solver.EXE pipeline. We hope these results will motivate further work on adaptive exploration at test time.

[1] Sutton and Barto (2018). Reinforcement Learning: An Introduction, 2nd ed.

[2] DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning


W5: Please report total tokens, calls, and latency.

A: We report API costs in the main text as a common basis for comparison because GPU-hours are unavailable for the proprietary models. We report the actual cost comparison below. For the test-time learning methods, latency was measured on 8 H200 GPUs under each method's default four-round setting.

Model or methodOutput tokens per problemMean model calls per problemMean latency per problemAPI cost per problem
GPT-5.238.3k110.4 min$0.5
Opus-4.668.4k118.5 min$1.71
Solver.EXE3.75M148.126.4 min$1.65
RSE4.09M13330.6 min$2.20
RSA3.66M25625.2 min$1.81
  • Among the TTL methods, Solver.EXE keeps inference costs within a reasonable range while achieving substantial performance gains.
  • Compared with proprietary models, Solver.EXE enables near-frontier performance on self-hosted hardware without incurring proprietary API fees.
AuthorsOpenReview ↗

A kind reminder as the deadline is approaching

Dear Reviewer PqBx,

We hope this message finds you well. As the discussion period is entering its final day, we wanted to check in to ensure that we have adequately addressed your concerns. Your constructive feedback and suggestions are very important to us, and we would greatly appreciate any further thoughts you might have.

Thank you again for your time and effort in reviewing our paper and rebuttal.

Best,
The authors

AuthorsOpenReview ↗

Rebuttal Summary and Key Revisions

Thank you for taking the time to review our paper and share your feedback. As the rebuttal period draws to a close, we have summarized our main responses and revisions below for easy reference.

  • W1. We report per-problem uncertainty in Table 9. Solver.EXE's gains over Opus-4.6 are statistically significant on multiple problems in both benchmarks. We also report frontier-comparable performance on HLE-Math and HLE-STEM (200 problems) and GPQA-Diamond (198 problems). Additionally, we used more conservative expressions in lines 26–27, 39–41, 72–75, 272–273, and 320–323.
  • W2. We extended the comparison between Simple Explore and Solver.EXE to four rounds with error bars in Table 10. Structured exploration yields substantial gains, primarily on medium-difficulty problems (base model Pass@1 in (20(20%,80, 80%]]) and harder problems (base model Pass@1 ≤20\le 20%).
  • W3. We added ablation results for all components. The results show that each component contributes to Solver.EXE's performance.
  • W4.
    • To assess the advantage of our structured exploration mechanism over prior stochastic methods, we compared Explorer trajectories with up to 128 stochastic rollouts from the base model. The Explorer found additional correct solutions to problems that the stochastic method could not solve even at a large search budget.
    • We used GPT-6.1 Sol to assess rollout usefulness. It labeled more than 75% of structured exploration trajectories as useful, compared with just 33.8% for vanilla exploration.
    • We introduced an adaptive mechanism that lets the model assess the usefulness of its initial rollouts and decide whether to continue exploring. This reduces token use while keeping Pass@1 close to that of the full Solver.EXE pipeline, suggesting an interesting direction for future work on exploration mechanism design.
  • W5. We report output tokens, calls, and latency for test-time learning methods and proprietary APIs. Solver.EXE achieves substantial performance gains over test-time learning baselines at comparable inference costs. It also achieves near-frontier performance on self-hosted hardware without incurring proprietary API fees.

We hope this summary is helpful and hope that you will take these new results and revisions into account in your final assessment. Thank you again for your time and thoughtful review.

Reviewer_uGErOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Solver.EXE, a training-free test-time learning framework that combines strategy exploration and memory-guided exploitation to improve expert-level reasoning. Four agents sharing a frozen LLM backbone generate diverse solutions, extract useful insights and warnings, and maintain an evolving problem-specific memory. Experiments on mathematical and STEM reasoning benchmarks show substantial improvements over the original backbones and existing test-time learning baselines, with some configurations achieving performance comparable to or above proprietary frontier models.

Critical strengths
  • The paper addresses an important problem in test-time reasoning: how to improve a frozen model's ability to solve challenging problems without additional training or external reward signals. The exploration-exploitation perspective is intuitive, and the overall framework is clearly presented. 
  • The reported performance gains are substantial, particularly on Apex and ArxivMath. Solver.EXE consistently improves over the original backbones and RSE across different model sizes. The additional comparisons with RSA, Self-Refine, and diverse prompting strategies also provide useful evidence for the effectiveness of the overall framework. 
  • The ablation and analysis sections offer some interesting observations about strategy diversity and experience reuse. In particular, removing the Explorer noticeably reduces performance, and the experiments suggest that even unsuccessful reasoning trajectories can provide useful information for subsequent attempts.
Critical weaknesses
  • The methodological novelty appears rather limited. Test-time exploration and exploitation, iterative reasoning refinement, and experience-based memory updates have already been extensively studied [1, 2, 3]. Solver.EXE mainly combines these established ideas through four prompt-based agent roles, while the DEVIATE and DRAFT mechanisms rely on instructing the model to avoid previous strategies or generate alternative approaches. Although the paper provides ablations against simpler exploration prompts, it remains unclear what fundamentally new mechanism is introduced beyond existing test-time search and memory-based self-evolution frameworks. 
  • The strongest claims about reaching frontier-level reasoning performance rely heavily on relatively small benchmarks. Apex contains only 12 problems and ArxivMath contains 32, making the results sensitive to individual examples despite repeated sampling. Furthermore, Solver.EXE performs multiple rounds of reasoning with many rollouts per problem, while the proprietary reference models are evaluated through direct inference rather than under matched test-time computational budgets. These comparisons demonstrate the benefits of additional inference-time orchestration, but do not fully support the broader claim that smaller models reach the reasoning capabilities of frontier models. 
  • Since insights and warnings are extracted through self-critique without external verification, incorrect information may enter the memory and influence subsequent reasoning. The memory audit reports 17.6%–20.5% erroneous items, but the analysis of how these errors propagate across multiple evolution rounds remains limited. The paper also lacks a theoretical characterization of the proposed evolution process, making it difficult to understand when or why the iterative updates should reliably improve reasoning. In addition, the code is promised upon acceptance rather than made available for review, limiting independent verification of the reported results.

[1] Ouyang S, Yan J, Hsu I, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory.

[2] Li D, Zhao X, Yu L, et al. SolverLLM: Leveraging test-time scaling for optimization problem via LLM-guided search.

[3] Venkatraman S, Jain V, Mittal S, et al. Recursive self-aggregation unlocks deep thinking in large language models.

Questions for authors

Please see the Critical Weaknesses section for the main concerns. I would particularly appreciate the authors' clarification on the methodological novelty, the fairness of compute-budget comparisons, and the reliability of the reported improvements.  Additionally, could the authors clarify the current reproducibility status of the work, given that the code is planned for release only upon acceptance?

4 comments
AuthorsOpenReview ↗

Rebuttal-Part1

We sincerely thank you for recognizing the importance of our research question, the clarity of our framework, the magnitude of our improvements, and the insights from our experimental analysis. We hope our responses below address your concerns.


W1: It remains unclear what fundamentally new mechanism is introduced beyond existing test-time search and memory-based self-evolution frameworks.

A: We summarize the novelty of our work below.

  • Structured exploration mechanism. Prior test-time learning methods primarily use stochastic sampling [1,3] for test-time search or MCTS [2] when external rewards are available. In general test-time settings without rewards, we first identify an overlooked bottleneck in prior methods: unguided search can lead the model to repeat attempted strategies or mistakes between rollouts. We therefore introduce a new test-time learning framework with an explicit and structured exploration process, the DEVIATE and DRAFT mechanisms, and compute allocation between memory-guided search and strategy-oriented exploration. The substantial gains brought by our design over these prior methods mark a non-trivial and successful attempt towards addressing this bottleneck.

  • Comprehensive experimental and theoretical analysis. We present several interesting findings on models' potential for strategy exploration and information exploitation, which has not been adequately studied and used in prior works. We also added a theoretical analysis of the expected performance gain from exploration and exploitation. We are confident that these results will inform the design of future test-time learning frameworks.

  • Strong empirical gains. Our method achieves substantial improvements over prior methods. To further demonstrate the benefit of our exploration mechanism, we compare with prior stochastic search methods[1,3] as follows:

ArxivMath

MethodPass@16Pass@32Pass@64Pass@128
Stochastic sampling45.347.448.450.0
Explorer rollouts48.156.2——
Reasoner rollouts50.356.2——

Apex

MethodPass@16Pass@32Pass@64Pass@128
Stochastic sampling31.238.441.441.7
Explorer rollouts37.341.7——
Reasoner rollouts41.550.0——

On ArxivMath, the Explorer with 32 rollouts outperforms stochastic sampling with 128 rollouts (56.2% vs. 50.0%). On Apex, the Explorer with 32 rollouts matches the solve rate of stochastic sampling with 128 rollouts (41.7%) and solves two problems that the backbone still misses after 128 rollouts. These results further underscore the importance of our work.

[1] Ouyang S, Yan J, Hsu I, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory.

[2] Li D, Zhao X, Yu L, et al. SolverLLM: Leveraging test-time scaling for optimization problem via LLM-guided search.

[3] Venkatraman S, Jain V, Mittal S, et al. Recursive self-aggregation unlocks deep thinking in large language models.


W2: The claims about reaching frontier-level performance rely on small benchmarks. How fair is the comparison between test-time orchestration and direct inference with proprietary models?

A:

Dataset scale. Apart from Apex and ArxivMath, our original submission also included evaluations on HLE-Math and HLE-STEM (200 problems) and GPQA-Diamond (198 problems). Across these diverse benchmarks, Solver.EXE consistently outperforms both the frozen backbone and the prior test-time learning baseline while achieving frontier-comparable performance.

MethodsHLE-MathHLE-STEMGPQA-Diamond
Opus-4.5 (High)27.226.087.0
Kimi-K2.5 (Thinking)24.725.287.4
Qwen3.5-35B-A3B14.116.680.9
w/ RSE23.022.485.2
w/ Solver.EXE27.526.488.7

Fairness of comparison. Our intended claim is not that smaller models, by themselves, can reach the reasoning capabilities of frontier models. Our original wording is:

Over several rounds of self-evolution, open-source backbones equipped with Solver.EXE can achieve performance comparable to that of frontier models. (Abstract ll 15-16; Introduction ll 39-41, 72-75, 81-83)

We see rollout generation as necessary for test-time learning methods to scale the performance of a frozen backbone. This also reveals a promising direction: self-hosted open-source models can provide competitive reasoning performance with additional computation while avoiding high proprietary API costs. Furthermore, with frontier-model performance serving as reference points, we can contextualize the difficulty of the benchmarks and highlight our core claim that Solver.EXE achieves substantial improvements over the frozen backbone and other TTL methods.


W4: Current reproducibility status of the work.

A: We have organized the experimental code used in our submission and submitted it for internal review. It will be ready for release before acceptance.

AuthorsOpenReview ↗

Rebuttal-Part2

W3: The analysis of how these errors propagate across multiple evolution rounds remains limited. The paper lacks a theoretical characterization of the proposed evolution process.

A: We examine three aspects of Solver.EXE's reliability.

  1. How reliable is our memory across iterations? The overall error rate stays near 20% across four rounds. Errors carried over from the previous round make up less than 5% of current memory.
  2. How reliably does the Reasoner reuse memory? The Reasoner self-verifies in at least 99% of cases involving memory use, and additional verification yields little improvement.
  3. A theoretical and empirical understanding of Solver.EXE's reliability: We derive a theoretical bound in Appendix G on Solver.EXE's expected improvement per round and provide empirical evidence supporting this bound.

SubQ1: How reliable is our memory across iterations?

We use 6.1 Sol (medium) to assess the correctness of all memory items produced by Qwen3.5-35B-A3B on ArxivMath and track error propagation.

The error ratio is shown as follows:

IterationInsight Error RateWarning Error RateMemory Error Rate
Iter 124.6%12.6%20.0%
Iter 224.7%12.3%20.0%
Iter 324.2%11.6%19.7%
Iter 423.9%11.9%19.8%

For each memory type, we calculate the error rate by dividing the number of erroneous items by the total number of items of that type. The error rates remain relatively low across iterations.

The propagation rate is shown as follows:

Iteration TransitionError Retention RateShare of Current Memory
Iter 1 → Iter 226.0%3.9%
Iter 2 → Iter 324.2%4.2%
Iter 3 → Iter 427.1%4.9%

We report retained errors as a share of all erroneous items in the previous round (retention rate) and of all memory items in the current round. Most previous-round errors are filtered out before the next round of reasoning, and the remaining errors make up less than 5% of the memory used in that round.

These results suggest that Solver.EXE's memory update and maintenance process helps preserve memory quality across iterations, supporting stable and reliable test-time evolution.

SubQ2: How reliably does the Reasoner reuse memory?

We analyze the Reasoner's behavior from the 1024 rollouts sampled across Solver.EXE's evolution rounds. The annotation results from 6.1 Sol show that:

  • In at least 99% of cases, the Reasoner performs self-verification when using memory items.
  • In 75.3% of cases, the Reasoner checks every memory item it uses for correctness before relying on it.
  • In 76.5% of cases, the Reasoner checks the validity of every result it derives using memory items.

Furthermore, Table 11 shows that adding either self- or external verification of memory items yields little additional improvement. Together, these results show that the Reasoner exhibits a degree of robustness to erroneous signals in memory, which contributes to the reliability of Solver.EXE's test-time evolution.

SubQ3: A theoretical and empirical understanding of Solver.EXE’s reliability

We provide a theoretical analysis of when and why Solver.EXE improves reasoning in Appendix G.

  • Expected improvement per round. Our analysis bounds the expected improvement for each iteration: ηct(1−pt)−ϵt\eta c_t(1-p_t)-\epsilon_t. Intuitively, the current Reasoner's performance on the given problem can improve if a correct solution has been discovered by the previous Reasoner or by the Explorer (ctc_t), the system successfully reuses information from that solution (η\eta), and the current Reasoner would otherwise fail (1−pt1-p_t). ϵt\epsilon_t bounds the probability that the actual memory update leads to a wrong answer when idealized reuse would have produced a correct one.
  • The Explorer increases the chance of a correct discovery (ctc_t). Across all four Solver.EXE iterations, the Explorer finds correct answers on problems unsolved by the previous Reasoner.
  • The Reasoner can reuse information from correct rollouts (η\eta). In section 5.3, the Reasoner achieves 96.4% accuracy using memory built from correct rollouts, supporting the assumption of effective reuse.
  • Erroneous memory has a limited effect on final accuracy (ϵt\epsilon_t). In Appendix E.2, removing flagged erroneous memory items yields little additional improvement.

We also provide empirical evidence on the reliability of memory updates. We separately measure how memory items from the current round's Explorer rollouts EtE_t and the latest Reasoner rollouts RtR_t contribute to the update from Mt−1M_{t-1} to MtM_t. We report the Reasoner's Pass@1 with each memory group.

IterationMt−1M_{t-1}Mt−1+RtM_{t-1} + R_tMt−1+EtM_{t-1} + E_tFull Memory MtM_t
Iter 231.2%33.4%32.9%34.5%
Iter 334.5%35.4%35.7%36.8%
Iter 436.8%37.2%37.0%37.6%

The results suggest that both sources contribute to the improvement of Solver.EXE.

Thank you for the detailed response and additional results. The new analyses help clarify the benefits of structured exploration and address some of my concerns about memory reliability. However, I still have reservations about the methodological novelty and the fairness of the compute comparisons, so I will maintain my original rating.

  • Novelty and contributions: Our structured exploration mechanism fills an important gap in prior test-time learning frameworks [1–5]: how to discover correct solutions and obtain useful learning signals both effectively and sustainably across evolution rounds, thereby avoiding early saturation. Our resulting framework represents the first successful attempt to address this problem and achieves substantial improvements over prior methods. Ours is also the first work to analyze the potential of backbone models and derive theoretical bounds for effective test-time learning and exploration. We believe these results provide valuable insights and contribute to future research on test-time learning.

  • Fairness of compute comparisons: The method we propose is a test-time learning framework [1–5], which inherently requires additional computation from the backbone models for sampling learning signals and online evolution. All test-time learning baselines in our experiments shared the same rollout budgets and were compared fairly. The performance of frontier models serves only as a reference point to suggest that open-source backbones enhanced through our test-time learning framework can provide competitive performance on self-hosted hardware without incurring high proprietary API fees.

We thank the reviewer for their engagement and are happy to address any remaining questions.

[1] Ouyang S, Yan J, Hsu I, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory.

[2] Li D, Zhao X, Yu L, et al. SolverLLM: Leveraging test-time scaling for optimization problem via LLM-guided search.

[3] Venkatraman S, Jain V, Mittal S, et al. Recursive self-aggregation unlocks deep thinking in large language models.

[4] Wang X, Shi J, Feng S, et al. Do not waste your rollouts: Recycling search experience for efficient test-time scaling.

[5] Hu J, Zhang Y, Shang S, et al. PaCoRe: Learning to scale test-time compute with parallel coordinated reasoning.

Reviewer_qwTwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a test-time learning method where a frozen LLM works on a problem over multiple rounds, with four prompt roles: 1) a reasoner which samples solutions conditioned on some evolving memory, 2) an explorer samples solutions that depart from already tried strategies, 3) an extractor distills insights from the generated rollouts, and 4) a curator prunes and updates the memory. The authors compare against RSE, which they argue lacks exploration. They show relatively large gains in performance on ArxivMath and Apex using Qwen 3.5 35B A3B and Gemma 4 26B A4B, at times matching GPT-5.2 and Opus 4.6.

Critical strengths

The paper is clear and interesting to read. The reported performance gains are substantial.

Critical weaknesses

The paper lacks pass@1 vs inference cost curves. I’d also add cons@k as a baseline in the main text.

Table 3 and Table 9 stop at 2 iterations, and have no error bars. For Table 3, since it is only 2 iterations, it is unclear whether explorer only speeds up convergence or also rises the ceiling. I’d recommend to also include Gemma4, where Solver.EXE lead over RSE on ArxivMath is largest.

Section 5.2 claims exploration reaches correct solutions the backbone misses (l. 402), but the evidence is embedding diversity (Table 4) and UMAP plots for two problems (Fig. 4). Diversity is not coverage, since varied but wrong strategies also raise these metrics. Please report pass@k across k for the base model, the Reasoner rollouts and the Explorer rollouts, also against inference cost. If the base model's pass@k at large k already solves these problems, the solutions are low-probability rather than outside its solution space (ll. 143–144).

Minor: there is no ablation of deviate-only vs. drift-only, nor of the split ratio. The trade-off described at ll. 199–200 is also not measured.

Minor: The paper claims to reach “expert-level solvers” throughout, it is unclear what this even means, and in my opinion the paper loses more from making this murky claim than it gains.

Minor: Figure 4 is interesting but it only comprises 2 problems, it would be much more interesting to see similar plots but for entire datasets.

Questions for authors

Minor feedback:

  • Figure 1 would be more clear if it contained the notation of Section 3.2
  • Lines 141-143 could be more precise, the difference to me seems to be how context is built rather than external signals. Solve.EXE also technically lacks the latter (all agents share the same frozen backbone, the explorer’s inputs come from the model’s own rollouts).
  • In Tables 1, 2, 6 and 7, the scores of reference models, raw backbones and majority voting are placed under the "Iter 4" column. A separate "Base" (or "Iter 0", as in Fig. 7) column would be clearer.
5 comments

We sincerely appreciate your recognition of the clarity of our framework and its strong empirical improvements. We hope our responses below address your concerns.


W1: The paper lacks pass@1 vs inference cost curves, and cons@k in the main text.

A: Thank you for the suggestion. We now report inference cost curves in Figure 7 and Cons@kk in Table 1. Solver.EXE achieves higher pass@1 than other methods at comparable inference costs. We omit Cons@kk for HLE-Math and HLE-STEM, where the answers are judged by o3-mini and cannot be reliably grouped by rule-based methods.


W2: Table 3 and Table 9 stop at 2 iterations, and have no error bars.

A: We have updated Table 3 and 10 to extend the evolution rounds to 4 iterations with error bars. At Iter 4, removing the Explorer lowers Pass@1 by 4.4 percentage points on ArxivMath and 7.1 on Apex for Qwen3.5-35B-A3B. The corresponding drops for Gemma4-26B-A4B-it are 5.6 and 6.1 points. The results suggest that our exploration mechanism effectively raises the performance ceiling.


W3: Please report pass@k across k for the base model, the Reasoner rollouts and the Explorer rollouts.

A: Thank you for your thoughtful comments. We report the pass@k results for rollouts from the base model (Qwen3.5-35B-A3B), the Reasoner, and the Explorer. To better probe low-probability regions of the base model's solution space, we increase its sampling budget to 128 rollouts.

ArxivMath

Rollout sourcePass@16Pass@32Pass@64Pass@128
Base model45.3 (0.73M)47.4 (1.46M)48.4 (2.93M)50.0 (5.86M)
Explorer48.1 (0.32M)56.2 (0.63M)——
Reasoner50.3 (0.26M)56.2 (0.52M)——

Apex25

Rollout sourcePass@16Pass@32Pass@64Pass@128
Base model31.2 (0.67M)38.4 (1.35M)41.4 (2.69M)41.7 (5.38M)
Explorer37.3 (0.30M)41.7 (0.60M)——
Reasoner41.5 (0.28M)50.0 (0.56M)——

Each cell reports pass@k (%) with output tokens per problem in parentheses. On ArxivMath, the Explorer achieves 56.2% coverage with 32 rollouts, versus 50.0% for the base model with 128 stochastic rollouts. On Apex25, both reach 41.7% at those budgets, with the Explorer solving two problems the base model misses. The results suggest that our structured exploration design effectively enlarges the search space.


W4: There is no ablation of deviate-only vs. drift-only, nor of the split ratio.

A: Thank you for the suggestion. We have added the ablation of the Explorer's Deviate and Draft modes with Qwen3.5-35B-A3B to Table 12. We further vary the split of Explorer rollouts between Deviate and Draft in Figure 9. Overall, our default 50:50 mix performs best. We attribute this advantage to the distinct conditioning contexts used by Deviate and Draft, which induce different rollout distributions and thus extend the exploration space.


W5: It is unclear what "expert-level solvers" means.

A: Our intended claim is that, on benchmarks that require the knowledge and skills of domain experts, such as ArxivMath and HLE, LLMs can go from struggling to solve these problems on their own to solving them correctly with our test-time learning methods. We agree that the phrase "expert-level solvers" is too broad. The original wording in the abstract was

With the right self-evolution procedure, small- to medium-sized models can more fully realize their reasoning potential and grow into expert-level solvers.

We have revised it to read

With the right self-evolution procedure, small- to medium-sized models can more fully realize their reasoning potential and solve challenging reasoning problems that they initially struggle with on their own.

We have also tightened related wording in the main text.


W6: Figure 4 is interesting. It would be much more interesting to see similar plots but for entire datasets.

A: Sure. We have added plots for all ArxivMath problems to Appendix E.4. We will also include the visualization code in our code release upon acceptance.


Q1: Figure 1 would be more clear if it contained the notation of Section 3.2.

A: We have updated our main figure accordingly.


Q2: Lines 141-143 could be more precise.

A: We have revised Section 3.1 accordingly. The revised sentence reads as follows.

If the rollouts repeatedly follow similar strategies, the updates may reinforce those strategies and provide little guidance for finding alternatives.


Q3: A clearer version of Table 1,2,6 and 7.

A: Thank you for pointing this out. We have revised Tables 1,2,6, and 7 so that the Iter 1–4 column labels apply only to the test-time learning rows.

AuthorsOpenReview ↗

A kind reminder as the deadline is approaching

Dear Reviewer qwTw,

As we enter the final day of the discussion period, we wanted to follow up on our rebuttal and see whether our responses have addressed your concerns. We appreciate your thoughtful feedback and would be grateful for any further thoughts you might have.

Thank you again for taking the time to review our paper and consider our responses.

Best,
The authors

AuthorsOpenReview ↗

Rebuttal Summary and Key Revisions

Thank you for taking the time to review our paper and share your feedback. As the rebuttal period draws to a close, we have summarized our main responses and the corresponding revisions below for easy reference.

  • W1. We report inference cost curves in Figure 7 and Cons@k results in Table 1. The cost curves show that Solver.EXE achieves substantial performance gains over prior test-time learning methods at comparable inference costs.
  • W2. We extended the evaluations in Tables 3 and 10 to four rounds with error bars. The gains from structured exploration persist through Round 4, suggesting that the Explorer raises the performance ceiling.
  • W3. We report the base model's Pass@k for up to 128 rollouts and compare it with the Reasoner and Explorer at 32 rollouts. The Explorer found correct solutions to problems that the base model could not solve even at large kk.
  • W4. We added the Deviate-only and Draft-only ablations in Table 12 and compared different split ratio in Figure 9. The default 50:50 split generally performs best.
  • W5. We revised the wording in lines 14–17 and 471–473.
  • W6. We included plots for the entire dataset in Appendix E.4.
  • Q1. We updated our main figure in Figure 2 to use the notation from Section 3.2.
  • Q2. We clarified the wording in lines 144–145.
  • Q3. We revised Tables 1, 2, 6, and 7 so that the Iter 1–4 column labels apply only to the test-time learning rows.

We hope these updates address your concerns. Thank you again for your valuable comments.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their time, suggestions, and comments.

Merits of Our Work

The reviewers acknowledged these strengths:

  • Our exploration–exploitation formulation and method are well motivated and clearly presented. (Reviewers uGEr, PqBx, qwTw)
  • The empirical results are strong: Solver.EXE consistently and substantially improves over frozen backbones and prior test-time learning baselines across diverse reasoning tasks. (Reviewers uGEr, PqBx, qwTw)
  • Our ablations and analyses are thorough and insightful, providing interesting observations on model behavior and performance gains. (Reviewers uGEr, qwTw)
  • The evaluation spans a broad range of tasks and backbones, including mathematics, STEM problems, and coding tasks, demonstrating broad applicability. (Reviewer PqBx)

How We Addressed the Reviewers' Concerns

  • Structured exploration boosts solution discovery

    • We compare Solver.EXE's Explorer with stochastic sampling under large search budgets.
    • On ArxivMath, the Explorer achieves Pass@32 = 56.2%, compared with 50.0% for the base model with 128 stochastic rollouts. On Apex, the Explorer with 32 rollouts finds correct solutions to 2 problems the backbone still misses after 128 rollouts.
  • Quantitative evidence of effective exploration

    • An exploration trajectory is useful if it identifies an approach distinct from attempted strategies and either outlines an actionable plan or makes nontrivial progress.
    • More than 75% of Explorer trajectories are labeled useful by GPT-6.1 Sol, compared with only 33.8% for vanilla exploration.
    • We introduce an adaptive fallback mechanism in which the model assesses whether at least two of the first four exploration rollouts are useful. If not, further exploration is skipped. This mechanism reduces token usage while keeping Pass@1 close to that of full Solver.EXE, suggesting a promising direction for future work.
  • Solver.EXE achieves competitive performance on large-scale benchmarks

    • We evaluate on HLE-Math and HLE-STEM (200 frontier problems) and GPQA-Diamond (198 graduate-level science problems).
    • Qwen3.5-35B-A3B with Solver.EXE achieves 27.5% on HLE-Math, 26.4% on HLE-STEM, and 88.7% on GPQA-Diamond, outperforming other test-time learning methods and performing comparably to its trillion-parameter counterparts, Opus-4.5 and Kimi-K2.5-Thinking.
  • Deeper understanding of Solver.EXE's reliable improvements

    • Robust memory evolution: We use GPT-6.1 Sol to assess the correctness of all memory items produced during evolution. The error rate stays near 20% across 4 rounds. Errors from the previous round account for less than 5% of current memory.
    • Self-verification behavior: The Reasoner self-verifies in at least 99% of cases involving memory use. Removing erroneous memory items yields little improvement, suggesting the Reasoner's robustness to noise.
    • Theoretical bound on Solver.EXE's improvements: Appendix G derives the bound on Solver.EXE's expected improvement per iteration based on (1) the complementary contributions of the Explorer and Reasoner to discovering correct solutions; (2) the reliability of the Reasoner's memory reuse; (3) the limited effect of memory noise on performance. We provide empirical evidence supporting each assumption underlying this bound.
    • Empirical evidence of Solver.EXE's improvements: We separately measure how Reasoner rollouts and Explorer trajectories contribute to each update, showing that both consistently improve Solver.EXE across iterations.

Contributions to the Research Community

  • The first structured exploration mechanism that addresses prior TTL bottleneck. (Reviewers uGEr, PqBx, qwTw)

    • We identify an important but understudied bottleneck in prior test-time learning methods: unguided test-time search can cause models to repeat previously attempted strategies and mistakes.
    • Our framework with explicit exploration is the first successful attempt to address this problem, providing an effective and efficient approach for future work to build on.
  • Strong empirical performance and practical value. (Reviewers uGEr, PqBx, qwTw)

    • By reframing test-time learning as an iterative process of exploration and memory-guided improvement, our method consistently and substantially improves over prior methods across challenging benchmarks and backbones, enabling frontier-competitive performance using self-hosted hardware without incurring proprietary API fees.
  • Insightful analysis into model behavior and theoretical foundations. (Reviewers uGEr, qwTw)

    • We make several interesting discoveries regarding models' understudied potential for exploration and memory exploitation, with empirical support.
    • We also propose the first theoretical bound on reliable improvements from test-time learning and exploration. We believe these results provide insights for future research on test-time learning.
AuthorsOpenReview ↗

Clarification on the Comparison Between Evolved Backbones and Proprietary Models

  • Fair comparison between test-time learning methods. All test-time learning methods require additional computation for sampling learning signals and online adaptation. Therefore, to evaluate the effectiveness of Solver.EXE, we fix the backbone and compare the performance of all TTL methods under the same rollout budgets.

  • Fair comparison between the evolved backbone and the reference models. To give a more intuitive sense of the evolved backbone's performance, we use frontier models' Pass@1 scores as reference points to illustrate the level of reasoning performance the backbone reaches after online adaptation. For Pass@1 evaluation, we use a single inference call per answer for both the evolved backbone and the reference models, without intervening in or extending the inference process.

  • Claims throughout the paper.

    • Throughout the paper, our claim concerns the performance of open-source backbones after adaptation with Solver.EXE.

    Over several rounds of self-evolution, open-source backbones equipped with Solver.EXE can achieve frontier-comparable performance on certain benchmarks (lines 23–27, 39–41, 72–75, 272–274, 320–323, and 325–327).

    These results show that such performance can be achieved on self-hosted hardware without incurring proprietary API fees. We never claim that smaller models alone can reach the reasoning capabilities of frontier models.

    • Furthermore, we want to emphasize that our main contribution remains the same:

    our framework for balancing exploration and exploitation, incorporating an explicit exploration mechanism, delivers substantial improvements over prior test-time learning methods.

    We believe our work offers valuable guidance and insights for future work on test-time learning.

AuthorsOpenReview ↗

Other Suggestions and Our Responses

Additional Ablations

  • We extended the Explorer and Curator ablations to four rounds with error bars and added Gemma4 to Table 3. The Explorer's gains persist through Round 4 on both backbones, suggesting that the Explorer raises the performance ceiling. (Reviewer qwTw)
  • We extended the Simple Explore comparison to four rounds in Table 10. Solver.EXE holds the advantage in all four rounds, with gains concentrated on medium-difficulty and harder problems. (Reviewers PqBx, qwTw)
  • We added separate ablations for the Extractor, Insights, and Warnings. We also removed the Explorer without reallocating its rollouts to the Reasoner. Each component contributes to final-round performance on both benchmarks. (Reviewer PqBx)
  • We added Deviate-only and Draft-only ablations in Table 12 and varied their rollout split in Figure 9. Both modes contribute, and our default 50:50 split performs best on both benchmarks. (Reviewers qwTw, PqBx)

Additional Comparisons and Inference Costs

  • We added Pass@1 versus inference cost curves in Figure 7 and Cons@k results in Table 1. Solver.EXE achieves higher Pass@1 than prior test-time learning methods at comparable inference costs. (Reviewer qwTw)
  • We reported output tokens, model calls, and latency per problem for test-time learning methods and the proprietary reference models. (Reviewer PqBx)
  • We reported Pass@k for the base model, Reasoner, and Explorer alongside output-token usage. The Reasoner's Pass@32 exceeds the base model's Pass@128. The Explorer found correct solutions to problems that the base model could not solve even at large kk. (Reviewer qwTw)
  • We report per-problem uncertainty in Table 9. The open-source model's gains over Opus-4.6 after Solver.EXE evolution are statistically significant on multiple problems in both benchmarks. (Reviewer PqBx)

Presentation and Reproducibility

  • We extended the strategy visualizations to all ArxivMath dataset in Appendix E.4. (Reviewer qwTw)
  • We clarified in Section 3.1 that when rollouts repeatedly follow similar strategies, the updates may reinforce those strategies and provide little guidance for finding alternatives. In the abstract, we replaced "grow into expert-level solvers" with "solve challenging reasoning problems that they initially struggle with on their own" and tightened the related wording in the main text. (Reviewer qwTw)
  • We aligned Figure 2 with the notation in Section 3.2 and revised Tables 1, 2, 6, and 7 so that the iteration labels apply only to test-time learning results. (Reviewer qwTw)
  • We prepared the experimental code and submitted it for internal review. The public release will also include the strategy visualization code. (Reviewers uGEr)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.