acceptodds
Under review as a conference paper at ICLR 2027

Towards Better Exploration in Sequential Test-Time Scaling

Abstract

Test-time scaling improves language model reasoning by spending additional compute at inference. However, both classes of existing methods often fail to continue improving over long timescales. Parallel methods repeatedly sample independent answers from the model, scaling poorly on problems the model is unlikely to solve in a single attempt. In contrast, sequential methods build on previous answers to access new ideas, yet so far have not been shown to reach answers beyond those found by parallel scaling. First, we show that sequential scaling often stops improving because it becomes prematurely trapped in an attractor: a set of answers that prevents exploration of different answers once entered. Across 27 combinations of scaling methods, models, and benchmarks, we find that 53.8% of sequential scaling trajectories enter an attractor within four iterations. Second, we show that a simple model-mixing intervention helps escape attractors. This reduces the attractor hit rate by 21.2 percentage points on average, expands solution coverage beyond a compute-matched parallel baseline, and improves accuracy of recursive self-aggregation by at least 2.2 percentage points. Our results motivate refocusing long-horizon test-time scaling from parallel methods to sequential methods that improve previous answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model reasoning, test-time scaling, inference-time compute, attractors, exploration, mixture-of-agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vni5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why the sequential sampling fails to improve the reasoning performance in long-horizon task. The work explains that the sequential sampling will get struck in a local minimum in the semantic space. The authors proposed using mixture models to deal with this case. They carried out experiments across 27 combinations of methods models and benchmarks to prove their claims empirically.

Critical strengths

The works provides a way to define the attractor in the semantic space. The empirical results show that at step 4, a large proportion of sample will stay in a small valley and could not get out.

The author proposed a simple approach to get out of the local minimum by calling for help from other model, using weaker model as perturbation.

Critical weaknesses

In Table 3, the mixed models only help to improve the overall performance slightly, the increases are mostly round 2-3%.

The idea of using mixed model is quite simple, like looking at different direction to get out of the loop you are stuck in. It lacks a new vision/angle to view the problem or bring a new insight to help solving the problem. It is not at the level of top-tier acceptance.

The concept of attraction is well defined in mathematics, but it is not clear how it will be applied to real situations, because it could differs from case to case.

Questions for authors

No

2 comments
AuthorsOpenReview ↗

Response to reviewer Vni5

Thank you reviewer Vni5 for your feedback.

Question for reviewer Vni5. You indicate that you have no questions for us to answer; is there any further information we can provide that you believe would improve our work? We have put considerable effort into this work and are eager to improve it.

[W1] Effect size

The main concern about effect size seems to come from a misreading of Table 3, which we clarify first.

In Table 3, the mixed models only help to improve the overall performance slightly, the increases are mostly round 2-3%.

Table 3 reports attractor hit rates, not task performance, and our intervention lowers the rate of hitting an attractor within four states by 21.2 percentage points on average.

[W2] Novelty

The idea of using mixed model is quite simple, like looking at different direction to get out of the loop you are stuck in.

We believe this is actually a strength of our intervention! Its simple nature helps isolate where the improvements we obtain come from.

It lacks a new vision/angle to view the problem or bring a new insight to help solving the problem. It is not at the level of top-tier acceptance.

We strongly disagree with this statement. Our novel contributions are as follows (also listed in the introduction):

  • We introduce a general formulation for test-time scaling methods, and the notion of attractors within this formulation.
  • We show that existing sequential scaling methods prematurely converge to attractors, limiting further gains in the number of problems solved.
  • We show that a simple model-mixing intervention lets existing sequential scaling methods escape these attractors, access more correct answers than parallel scaling, and improve benchmark performance.

We have not found any evidence in published literature of a similar investigation into this problem, and we believe that our work provides a useful new perspective on why sequential test-time scaling often falls short of parallel scaling.

[W3] Real situations

It is not clear how it will be applied to real situations, because it could differs from case to case.

We investigate benchmarks for mathematics, games (chess, Sudoku, ...), and ARC-AGI-1, as these tasks (1) require minimal external knowledge and (2) are easily verifiable. We have tried to present a variety of benchmarks, which we believe measure reasoning skills that are often relevant to real situations. It is of course true of any method that the results may differ from case to case.

AuthorsOpenReview ↗

Reply to reviewer Vni5's response

Thank you for your reply! You raise many of the same concerns as reviewer XBsf initially made, so we have repeated some of our replies to them here, as reviewer XBsf was satisfied with these responses.

First, there is no explicit algorithm provided in the main text, making it unclear how the proposed method is actually executed in practice.

Our intervention is described on line 320. We now have added pseudocode to Appendix M.

Secondly, relying solely on Definition 4.1 leaves the terminologies very vague, i.e. the choice of parameters, how readers imagine the states.

All parameters for identifying attractors are given in Appendix B.4 (lines 889–932). States are defined concretely for each method in Table 1. For example, in Self-Refine a state is the current answer. Following XBsf’s suggestions, we have also added:

  • a full worked example that follows a real IMO-AnswerBench problem from answers to embeddings, clusters, transitions and attractors, including one optimal and one suboptimal attractor (Appendix B.4.4)
  • an additional experiment to confirm that identified attractors satisfy Definition 4.1 (Appendix B.4.3).

The intuition about escaping local minima is very standard

Our intuition relates to escaping local minima in optimisation, but it applies to reasoning in natural language rather than model training. Prior work has not used the same framing to show that sequential test-time scaling fails in this way, as is also noted by reviewer aAfj.

I looked for a mathematical explanation or theoretical proof for why mixing models help to escape these local minima and improves overall performance, but there is none.

Our goal is to provide empirical evidence for our hypotheses rather than to prove them mathematically. It is standard for test-time scaling research to be published with empirical contributions rather than theoretical proofs [1,2,3].

[1] Wang, Xuezhi, et al. “Self-Consistency Improves Chain of Thought Reasoning in Language Models.” The Eleventh International Conference on Learning Representations (2023).

[2] Madaan, Aman, et al. "Self-refine: Iterative refinement with self-feedback." Advances in neural information processing systems 36 (2023): 46534-46594.

[3] Wang, Junlin, et al. "Mixture-of-agents enhances large language model capabilities." International Conference on Learning Representations (2025).

Reviewer_aAfjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper shows that sequential scaling potential is limited by getting trapped by attractors.
  • Authors introduce an approach that mixes inference between two different models to help escape attractors.
  • Method the method reduces attractor hit rate by 21.2%
Critical strengths
  • Finding is novel: mixing models makes sequential scaling less likely to get stuck in attractors. This allows for better performance over parallel scaling and for long timescales.
Critical weaknesses
  • Method seems impractical in a production environment for various reasons (e.g., needing to keep multiple models in memory). But, I understand from the Introduction that the primary motivation is to demonstrate the potential of sequential scaling.
  • Mixing a smaller model with a larger model can actually degrade performance (although paper mentions linear annealing to mitigate this).
Questions for authors
  • How did you arrive at using a smaller model as the right approach for this experiment? Why did you prefer that over other possible approaches like prompt-level methods to inject variation into the primary model?
3 comments
AuthorsOpenReview ↗

Response to reviewer aAfj

Thank you reviewer aAfj for your positive feedback. We are pleased to hear that you find our work novel! We have made updates to the main paper based on our responses below (changes coloured red).

[W1] Method seems impractical in a production environment

We disagree with this as a weakness for two reasons:

  1. This work is focused on the effect of attractors on sequential scaling. This problem formulation is novel and, as such, is the contribution. We emphasise that it is therefore not the primary goal of our research to iterate on a method to most efficiently / practically achieve this, but rather to motivate further investigation into avoiding attractors. We appreciate that you also mention this in your review:

But, I understand from the Introduction that the primary motivation is to demonstrate the potential of sequential scaling.

  1. Although our method requires generating multiple answers from different models, this can be practical when deployed at scale if each model is hosted on its own machine. It is already common to have the same circumstances when A) a larger model hands work to subagents that run on smaller models (e.g. as in Claude Code [8]), or B) a router sends requests to different models depending on the difficulty [9].

[W2] Mixing a smaller model with a larger model can degrade performance

As you mention, we present an annealing approach that outperforms the baselines by improving exploration without sacrificing exploitation.

[Q1] Why did you prefer that over other possible approaches like prompt-level methods to inject variation into the primary model?

Great question – this was actually our original goal when starting this research: to show that attractors can be escaped by perturbing the prompts (as you have described). However, we were not able to find any useful improvement through this method, which led us to conclude that attractors are more fundamental to the model, and can't be escaped through prompting alone, resulting in our model-mixing intervention which changes the model itself (perhaps the primary model will never output the correct answer directly from the initial problem, so the only way to make progress is by changing the model). We have therefore also added this as an additional baseline and discussion in Appendix H. Specifically, we apply the prompt-level perturbations from [10] to each sequential scaling method and find negligible impact on performance, as shown in the table below.

MethodPerturbationIMO (%)AMO (%)RGG (%)
Reasoning CacheNone81.364.177.0
Wang et al. (2025d)81.465.977.0
Self-RefineNone84.362.480.7
Wang et al. (2025d)83.162.679.3
Recursive Self-AggregationNone80.066.779.0
Wang et al. (2025d)78.462.579.9

[8] Anthropic Research system: “How We Built Our Multi-Agent Research System.” Anthropic (2025, www.anthropic.com/engineering/built-multi-agent-research-system.

[9] Microsoft Foundry Model Router: “How Model Router Works in Microsoft Foundry.” Microsoft Learn, learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-router-how-it-works. Accessed 9 Nov. 2026.

[10] Wang, Tianchun, et al. “On the Effect of Sampling Diversity in Scaling LLM Inference.” Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, pp. 7137–67 (2025).

Thank you for the clarifications. Overall I think the direction of the research is good.

If the paper's framing is studying attractors in sequential scaling, then characterizing what type of variation needs to be added in (and what are the properties of the secondary model needed to achieve this) would give the paper a stronger scientific basis. Having a clearer line of sight on this mechanism would make the paper's contribution stronger, so I'm maintaining my score.

AuthorsOpenReview ↗

Reply to reviewer aAfj's response

Thank you for your reply. We are glad you find the direction of the research good.

Based also on our discussion with you and reviewer XBsf, we added additional experiments and analysis to help characterise exactly what you highlight.

what type of variation needs to be added in

Our hypothesis is that attractors come from biases in the model itself, so changing the prompt or the sampling method should not be enough to escape them. Our updated Appendix H tests this hypothesis: rephrasing the prompt (Table 8) and raising the temperature (Figure 12) both give no clear improvement, while changing the model does.

what are the properties of the secondary model needed to achieve this

We tested 9 model combinations (Figures 4 and 1) and have expanded our analysis of them in Appendix J:

  • Models from the same family, which we expect to make more correlated errors, help less than similarly capable models from a different family. This matches prior work showing that heterogeneous model pools help multi-agent debate [11].
  • Once the secondary model is from a different family, swapping it for another such model, or adding more models, makes little difference, as long as the relative selection probabilities stay the same.
  • The secondary models do not need to be strong. On their own they do not improve coverage (Table 7), and they are best used sparingly (c = 0.75, Figure 5), because their role is to help the primary model escape attractors, not to supply solutions.

Our goal in this analysis is (as you mention) to help establish why secondary models help escape attractors. A full study of model complementarity would be a separate contribution in its own right.

[11] Chen, Justin, Swarnadeep Saha, and Mohit Bansal. "Reconcile: Round-table conference improves reasoning via consensus among diverse llms." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Reviewer_XBsfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why sequential test-time scaling often stops improving and proposes premature convergence to suboptimal attractors as an explanation. It formulates several scaling methods as state-transition processes and identifies attractors through semantic clustering and empirical transition estimates. Across 27 combinations of methods, models, and benchmarks, the authors report that 53.8% of trajectories enter detected attractors within the first four states. They introduce random model mixing to improve exploration, reporting a 21.2-percentage-point reduction in early attractor hits and an average coverage improvement of approximately 3.3 percentage points over the mixed-model parallel baseline. An annealed mixing strategy also improves RSA’s final accuracy by 2.2–2.3 percentage points on two benchmarks. Additional experiments examine costs, temperatures, model combinations, and failure cases.

Critical strengths
  1. The paper addresses an important problem through a systematic cross-method investigation. Understanding why sequential reasoning stops benefiting from additional computation is important for long-horizon inference. Studying summarization-based reasoning, critique-and-revision, and population aggregation makes the investigation broader than a diagnosis of one particular algorithm. Separating coverage from final accuracy also helps distinguish failures of exploration from failures to retain useful solutions.

  2. The intervention is simple and supported by meaningful controls. Model mixing requires no additional training and can be incorporated into existing methods. The mixed-model parallel baseline helps assess whether gains extend beyond independently collecting answers from different models, while temperature and model-combination ablations provide additional evidence. Annealing offers an initial demonstration that improved exploration can translate into final-accuracy gains. Reporting negative results also helps clarify the method’s limitations.

Critical weaknesses
  1. The central attractor-based explanation lacks sufficient independent validation. Detection relies on limited trajectories, semantic clustering, and additional context-distribution assumptions for RSA. The same-trajectory analysis and clustering-seed checks do not establish that detected sets satisfy Definition 4.1 on independent continuations. Moreover, the claimed potential accuracy improvement of at least 7.7 percentage points does not directly follow from a statistic conditioned on both trajectories terminating in attractors.

  2. Evidence for practical, long-horizon superiority remains limited. Matching answer counts does not match all computation, and the cost analysis does not provide a full parallel-scaling budget curve. Fixed mixtures do not improve final accuracy; the positive final-accuracy results rely on annealing on two benchmarks. Figure 13 does not report uncertainty for the accuracy differences, and the selection procedure for the annealing configuration is unclear.

Questions for authors
  1. Provide a verifiable example of an attractor: I am curious about what an attractor as defined in def 4.1 actually looks like. Could you provide a complete example showing the original reasoning, cluster members, transition counts and probabilities, the time of first entry, and how an attractor is determined to be suboptimal, or showing two attractors and which of them is better? The changes in numerical answers shown in Figure 8 are not sufficient.

  2. Clarify the mechanistic basis for model selection: When model capabilities and budgets are comparable, does performance vary with differences in error patterns or reasoning approaches? The current model-combination experiments show that “some mixtures work,” but do not yet establish which kinds of differences between models are most useful.

  3. The authors attribute the lack of improvement from Qwen 3.5 + GPT OSS on IMO-AnswerBench to benchmark saturation, but the positive results on AMO use a different Qwen model. What evidence can distinguish “benchmark saturation” from “complementarity of the model combination” as explanations? Does the same combination of strong models still benefit from sequential collaboration on harder, unsaturated mathematical tasks? I think the authors could try testing on research-level mathematical problems to assess the boundaries of the current method’s applicability, as the paper does not appear to have evaluated such tasks.

3 comments
AuthorsOpenReview ↗

Response to reviewer XBsf (1/2)

Thank you reviewer XBsf for your positive feedback. We are encouraged that you find the problem we investigate important! We have made updates to the main paper based on our responses below (changes coloured red).

[W1] Attractor validation

The central attractor-based explanation lacks sufficient independent validation. Detection relies on limited trajectories, semantic clustering, and additional context-distribution assumptions for RSA.

We have added further experiments to help justify our operationalisation of Definition 4.1 (definition of an attractor).

What was already in the paper. Our overall operationalisation is already relatively standard, which we believe helps its initial justifiability: we construct a Markov chain from clusters based on text embeddings, which is a common method [1,2,3]. We then validate in Appendix B.4.3 that the resulting attractors have the expected behaviour in terms of answer correctness (which was not used to define the attractors) when a trajectory enters into them, and that there is little variation in the results when identifying attractors using different random seeds.

Added improvements. We appreciate your suggestions to strengthen our validation, and we have made additions for each. We also add a worked example of an attractor in Appendix B.4.4, as discussed in our reply to your Q1.

establish that detected sets satisfy Definition 4.1 on independent continuations

This is an interesting test which we had not investigated! We have now added this experiment to Appendix B.4.3, also displayed in the table below. Specifically, for each combination of model, scaling method, and benchmark, we randomly sample 100 states that are in attractors. We resample a single next state from each one, and record the percentage of cases in which this next state lies within the same attractor. We do not recompute any clusters, and we assign points to the cluster with the closest centroid. Our results show that the next state is in the same attractor with probability >1−ϵ=0.9>1-\epsilon=0.9 in almost every case, which directly confirms that the identified attractors satisfy condition i of Definition 4.1.

MethodModelIMO (%)AMO (%)RGG (%)
GPT OSS 120B969998
Reasoning CacheGemma 3 12B IT1009697
Granite 4.1 8B949696
GPT OSS 120B9610098
Self-RefineGemma 3 12B IT959499
Granite 4.1 8B939296
GPT OSS 120B9510098
Recursive Self-AggregationGemma 3 12B IT1009795
Granite 4.1 8B969797

the claimed potential accuracy improvement of at least 7.7 percentage points does not directly follow from a statistic conditioned on both trajectories terminating in attractors.

We have updated this statement to be more accurate (line 269).

[W2] Practical evidence

Evidence for practical, long-horizon superiority remains limited […] Fixed mixtures do not improve final accuracy; the positive final-accuracy results rely on annealing on two benchmarks

We respectfully disagree that this is a weakness. The scope of this paper is to highlight a problem in current test-time scaling literature. We demonstrate final-accuracy results to show how this can lead to a useful method but our primary focus is on the analysis of attractors and their impact. We have updated the text starting on line 1421 to highlight this.

Matching answer counts does not match all computation, and the cost analysis does not provide a full parallel-scaling budget curve.

We provide a fully compute-matched baseline in Figure 10. Based on your suggestion, we have added the full parallel-scaling budget curve to this figure in grey. The added budget curves demonstrate that parallel sampling not only underperforms our method at the maximum budget, but also reaches a plateau earlier.

Figure 13 does not report uncertainty for the accuracy differences

We have added this to the paper. The uncertainty is low enough that the improvement we present over the RSA baseline is significant.

the selection procedure for the annealing configuration is unclear.

The start value c = 0.75 is the best exploration setting in Figure 5 (left), the end value c = 1 is the best exploitation setting in Figure 5 (centre), and linear interpolation is a schedule between them. We have added this information to Appendix K.

AuthorsOpenReview ↗

Response to reviewer XBsf (2/2)

[Q1] Provide a verifiable example of an attractor

We have added a full worked example of an attractor in Appendix B.4.4. We include examples of two different attractors for the same problem, where one is optimal and one is suboptimal, as well as descriptions for how the state evolves over time.

[Q2] Clarify the mechanistic basis for model selection

This is a very interesting question. We have attempted to provide some initial exploration in Figure 4 and have added new analysis of these results in Appendix J. However, a full, systematic analysis of how to select complementary sets of models unfortunately lies beyond our budget, and could even merit a full paper on its own. This is particularly true as the budget for this paper is already high at over 2,000 USD for the current experiments, of which Figure 4 takes up a reasonable portion.

[Q3a] What evidence can distinguish “benchmark saturation” from “complementarity of the model combination” as explanations?

We moved this experiment to the appendix because the baselines already scored 395/400 and 389/400, so even if our method scores 400/400, this improvement would not be statistically significant. We included these results in the appendix for transparency and will update the wording to explain this better. We did not further investigate this combination of models because, given that we already had the results in Figure 1, it would not have added much additional information.

[Q3b] I think the authors could try testing on research-level mathematical problems to assess the boundaries of the current method’s applicability

This is a great point which we are currently running experiments for, but benchmarks for research-level tasks (such as [4, 5]) generally require more complex harnesses with tool-use, which we believe merits a full paper on its own. However, please note that the benchmarks we use are the same as those used in recent prior work such as [6] and [7].

[1] Grootendorst, Maarten. "BERTopic: Neural topic modeling with a class-based TF-IDF procedure." arXiv preprint arXiv:2203.05794 (2022).

[2] Prinz, Jan-Hendrik et al. "Markov models of molecular kinetics: generation and validation." The Journal of chemical physics vol. 134.17 (2011): 174105

[3] Hu, Michael Y., et al. “Latent State Models of Training Dynamics.” Transactions on Machine Learning Research (2023).

[4] Wang, Erik Y., et al. "Horizonmath: Measuring AI progress toward mathematical discovery with automatic verification." arXiv preprint arXiv:2603.15617 (2026).

[5] Mang, Qiuyang, et al. "FrontierCS 2: A Benchmark for Computer Science R&D." arXiv preprint arXiv:2610.29932 (2026).

[6] Venkatraman, Siddarth, et al. "Recursive self-aggregation unlocks deep thinking in large language models." arXiv preprint arXiv:2509.26626 (2025).

[7] Hu, Jingcheng, et al. "Pacore: Learning to scale test-time compute with parallel coordinated reasoning." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2026).

Thank you for the thoughtful and detailed responses. I appreciate the authors' efforts to address my concerns and improve the paper. The additional experiments and analyses have addressed most of my concerns. I also understand that Q2 and Q3b may be better suited for future work. Overall, I am satisfied with the rebuttal and will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.