acceptodds
Under review as a conference paper at ICLR 2027

Thanks for Failing: Better Test-Time Search from Wrong Attempts

Abstract

Test-time scaling (TTS) methods are central to both LLM evaluation, and post-training pipelines. They not only help improve inference-time accuracy, but also TTS-inspired methods support rejection sampling and search- or verifier-guided trajectory generation for reinforcement learning. For rollout generation, TTS methods can often be decomposed into two primitives: independent sampling from a shared context, and sequential sampling conditioned on previous attempts. Recent work finds that independent sampling often outperforms sequential sampling, yet we observe that these two primitives have competing strengths. Independent sampling explores more diverse solutions, but cannot leverage prior failures. Sequential revision depends on prior failures, but becomes less diverse and is often trapped by failed reasoning. In this paper, we systematically study four methods to combine the predictions of independent and sequential sampling strategies. Through this analysis, we propose Failure Logit Extrapolation (FLEx), a simple, training-free, contrastive decoding algorithm that extrapolates the difference between token-level logits generated independently and sequentially. Across six math, coding, and instruction-following benchmarks, and eight models ranging from 4B to 235B parameters, FLEx improves pass@8 over independent sampling by 3.3% and 11.5% over sequential sampling on average. On the four mathematics benchmarks, it also increases answer diversity, measured by the fraction of unique answers, by 5.9%; on problems that all three methods fail to solve, this diversity advantage grows to 7.3%. Its gains consistently increase as more failed attempts are retained in its context and as more sampling budget is provided. Finally, we show that FLEx directly benefits post-training pipelines. When used for rejection sampling, FLEx-collected trajectories result in higher downstream accuracy than independently sampled trajectories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, test-time search, large language models, reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H84iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Failure Logit Extrapolation (FLEx), a training-free decoding method that combines predictions from a clean context and a context containing previous failed attempts for test-time scaling. Experiments across eight models and six benchmarks show improvements over independent sampling and sequential revision. The paper also shows that trajectories collected with FLEx can improve downstream performance when used for supervised fine-tuning.

Critical strengths
  • The experimental evaluation is extensive, covering eight models across multiple scales and families, as well as six benchmarks spanning mathematics, coding, and instruction following.
  • FLEx is a simple, training-free decoding method that reuses the model's own failed attempts and does not require an auxiliary model.
Critical weaknesses
  • My main concern is the overall problem setting. FLEx assumes that a reliable verifier is available to determine whether each attempt is correct, but I am not convinced that this is a realistic assumption for practical test-time scaling. For example, the math experiments use reference answers to identify failures, whereas these answers are usually unavailable when solving new problems.
  • The motivation is not fully convincing. Around line 75, the paper states: “Our experiments expose a fundamental tension between these primitives. Independent sampling produces more diverse solutions, but cannot use prior failures.” How is this conclusion supported, and how does it motivate FLEx? It seems heuristic to me.
  • Some of the design choices and analyses appear heuristic:
    • How do the authors choose the extrapolation direction? When should they use the default direction, extrapolating away from zfz_f, and when should they use the reverse direction?
    • Where are the results of the α\alpha and λ\lambda sweeps reported? Appendix A.5 describes the selection protocol, but only the selected configurations are shown.
    • If FLEx is applied to a new task, do the hyperparameters and extrapolation direction need to be tuned again? How expensive is this tuning process?
    • Are the prompt distributions in Table 2 matched? If not, could this make the comparison unfair?
Questions for authors

See the Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer H84i [1/2]

We thank the reviewer for acknowledging our extensive evaluation across models and benchmarks, as well as the simplicity of our training-free approach. We address the concerns and questions below.

[W1]: Reliance on a verifier

We thank the reviewer for this concern. We clarify that FLEx is a sampling primitive designed for verifier-guided search, a well-established setting in prior literature. For example, AlphaCode [1], FunSearch [2] and AlphaEvolve [3] all rely on automated evaluators to guide test-time search.

Our contribution is to improve how trajectories are sampled, rather than how they are verified. While our math experiments use reference answers as verifiers, this directly corresponds to practical post-training settings such as rejection sampling and RL with verifiable rewards, where such feedback is available. We will expand the related work discussion and clarify the verifier-guided setting and our contribution in the revised manuscript.

We also invite the reviewer to check out our response to Reviewer TB9N regarding the performance impact of using majority@k instead of oracle answers as this metric highlights an interesting property of FLEx when interacting naively with majority voting verification method. In short, FLEx improves majority@k by +0.2 to +2.1pp in the settings we tested, though the net effect is a balance between widened coverage and weakened vote conversion.

[1] Competition-Level Code Generation with AlphaCode

[2] Mathematical discoveries from program search with large language models

[3] AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms

[W2]: Convincingness of motivation

We thank the reviewer for this question. We clarify that independent sampling cannot leverage previous failures by construction, as each attempt is sampled independently from the same original context without access to previous attempts.

The remaining observation on sequential revision is empirically grounded. As already discussed in the Introduction (L51), multiple prior works [1, 2, 3] have identified that sequential TTS often suffers from repeated errors and reduced exploration. This motivated us to investigate the underlying reasons more closely. Through preliminary experiments, we observed that sequential revision produces significantly less diverse answers than independent sampling, as reported in Figure 3. This observation led us to explore how previous failures could be used without sacrificing diversity, ultimately motivating FLEx. We will clarify this progression in the revised manuscript.

[1] Contextual Drag: How Errors in the Context Affect LLM Reasoning

[2] Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models

[3] Old Habits Die Hard: How Conversational History Geometrically Traps LLMs

AuthorsOpenReview ↗

Response to Reviewer H84i [2/2]

[W3]: Extrapolation direction, hyperparameter sensitivity, tuning cost, and prompt matching

We thank the reviewer for raising these points. Here we address them:

  1. Extrapolation direction. We extrapolate away from the branch that is less performant, and in the vast majority of cases, the branch with failure-filled context. The reason we have used the reverse direction for a specific model family x evaluation benchmark is when informative execution feedback makes sequential revision stronger than independent sampling. We further verify this explanation in Table 7, where replacing execution feedback with binary correctness largely removes this advantage.

  2. Hyperparameter sweeps. Appendix A.5 describes our hyperparameter selection protocol, and Table 6 reports the selected configurations. We agree that showing the hyperparameter sensitivity would be useful. We now report the results of a sweep over λ and α for Qwen3 on mathematics benchmarks, as shown below.

λ / α1e-41e-31e-2mean
0.135.1734.3633.5834.37
0.236.0635.3533.3334.91
0.435.2335.6036.7335.85
0.837.3938.8936.0437.44
1.038.2436.2437.3937.29
mean36.4236.0935.41

In this combination, lambda is the dominant factor. The row mean rises from 34.37 at lambda=0.1 to 37.44 at lambda=0.8. It then flattens at lambda=1.0 (37.29), as now we additionally evaluate. Alpha matters less. Values of 1e-4 and 1e-3 are close (36.42 vs 36.09). A value of 1e-2 is clearly worse (35.41). A loose mask admits low-probability tokens, which the extrapolation then amplifies.

  1. Generalization and tuning cost. We sweep once per family x domain, not per benchmark or per model size. A selected hyperparameter transfers to any size within the same family and different benchmarks of the same domain (e.g., math). A practitioner adopting FLEx on a new domain only needs to run this once on a fraction of data.

  2. Prompt distribution. We clarify that both methods collect trajectories from the exact same 20K prompts under the same sampling budget. While the distributions of retained correct trajectories may differ, this is an intended consequence of the sampling methods. In particular, FLEx can discover correct trajectories for prompts that independent sampling fails to solve. This is precisely one of the benefits of improved solution coverage. Our comparison evaluates the downstream effectiveness of the resulting trajectory collection pipelines. It does not aim to control away the improvement in solution coverage.

Dear Reviewer H84i,

As the author-reviewer discussion period is approaching its end tomorrow, we wanted to follow up in case you have had a chance to consider our response, particularly our additional verifier-free majority@k experiments, hyperparameter sensitivity analysis, and clarifications regarding the motivation and experimental design.

We sincerely hope these additional results and explanations have helped address your concerns. We would be very grateful for any updated thoughts or remaining questions you might have, and would welcome the opportunity to provide further clarification before the discussion closes.

Thank you again for your time and thoughtful consideration!

Best,

The Authors

Dear Reviewer H84i,

As the author-reviewer discussion period comes to a close, we wanted to reach out one final time.

We have made a substantial effort to address your concerns through additional verifier-free evaluations, hyperparameter sensitivity analyses, and clarifications of the motivation and experimental design. We sincerely hope these additional results and explanations have helped address your reservations.

We would be very grateful if you could share any updated thoughts on our responses, particularly whether your main concerns have been addressed or if any remain unresolved. We remain available to respond promptly over the next few hours and would sincerely appreciate the opportunity for further discussion.

Thank you again for your time and thoughtful consideration.

Best,

The Authors

Reviewer_bcxMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper essentially proposes classifier-free guidance in LLM setting. Given a clean context H_c containing original question and a failure context Hf containing question, past attempts and verifier feedback, FLEx steers away the prediction using zFLEx=zc+λ(zc−zf)\boxed{z_{\text{FLEx}}=z_c+\lambda(z_c-z_f)}. The authors evaluate open models Qwen3, Granite 3.3, Mistral 3, ranging from 4B to 235B parameters, across six math, coding, and instruction-following benchmarks. Achieves a +3.34 improvement over independent sampling and approximately +11.5 over sequential revision.

Critical strengths
  1. This method is simple to implement, and also boosts model performance at inference time across models and benchmarks. It's a simple use of failed attempts for test-time exploration.
  2. Apart from inference improvements, the authors show that FLEx improve performance when used for rejection-sampling-based supervised fine-tuning.
Critical weaknesses
  1. Lack of compute-matched comparisons. The paper primarily compares methods under equal numbers of attempts rather than equal inference compute. FLEx requires two model evaluations per generated token.
  2. The contribution of logit extrapolation over existing contrastive decoding is not sufficiently isolated. The contrastive decoding baseline already improves a lot comparing to normal sampling methods, and improvement over the baseline is not always present.
  3. There's no signal or experiment results indicating that the method performs better as model is being trained more extensively with RL on long CoT.
Questions for authors
  1. Can authors provide a report of solve rate as a function of actual inference compute as opposed to number of attempts? Does FLEx retain its advantage when matching compute budgets with independent sampling?
  2. Since FLEx requires more compute, a way to also improve the model is to use the additional compute used by inference to provide additional RL training to models. This is especially relevant for industrial setting, as the inference compute is often the majority. Can authors run a comparison of the two ways to spend compute, either additional forward pass or additional RL training, at a reasonable experiment setting?
4 comments
AuthorsOpenReview ↗

Response to Reviewer bcxM [1/2]

We thank the reviewer for recognizing the simplicity of FLEx and its benefits for both inference and post-training. We address the concerns and questions below.

[W1, Q1]: Lack of compute-matched comparisons

We thank the reviewer for this point. We would first clarify our scope. Throughout the manuscript we emphasize sample efficiency. We never claim FLEx uses the same compute as independent sampling. Two forward passes per token is intrinsic to contrastive decoding. Our contribution is what to contrast against, and which capabilities that unlocks.

That said, we provide a compute-aware comparison by re-analyzing our k=256 experiment (Sec. 3.4; Fig. 5). We approximate the decoding compute by charging FLEx two forward passes per generated token. While this proxy does not capture full prefill-plus-decode FLOPs or latency, it offers a stricter comparison than attempt count and does not account for prefix caching.

FLOP budget (token-equiv/problem)independentFLExΔ
86k36.9534.45−2.50
163k39.5738.61−0.97
310k44.1444.91+0.77
573k47.3350.18+2.85
1,045k52.3353.97+1.64

Under this stricter accounting, FLEx trails below ~300k token-equivalents and leads above. The table stops at 1,045k. This is the point where independent sampling fully consumes its k=256 budget. The compute advantage is therefore budget-dependent, and we will state this scope explicitly in the revision.

Finally, this cost need not be paid at serving time. In Sec. 4 we post-train on FLEx-generated data. The resulting checkpoint beats its baseline under plain independent sampling. FLEx then becomes a one-off offline cost, and the served model adds no inference overhead. This is the setting where the reviewer's concern about industrial inference cost is most acute.

[W2]: Contribution of logit extrapolation over existing contrastive decoding is not sufficiently isolated

We thank the reviewer for this comment. We clarify that our contribution is not the logit guidance formula itself, as both subtraction (Li et al., 2023) and extrapolation (O'Brien & Lewis, 2023) have been explored in prior literature. Our contribution is what serves as the contrast and for what purpose. FLEx identifies the predictive shift induced by accumulated, instance-specific failed attempts as an effective signal for exploration. To the best of our knowledge, we are the first to exploit this signal through logit-level guidance, making sequential TTS competitive with independent sampling and addressing a bottleneck identified by extensive prior work.

In Table 1, we compare subtraction with and without online failures, as well as extrapolation with online failures, to isolate these design choices. While extrapolation does not always outperform subtraction, it achieves better average performance in 7 out of 8 model configurations. We will clarify this distinction in the manuscript.

AuthorsOpenReview ↗

Response to Reviewer bcxM [2/2]

[W3]: Effectiveness after extensive RL post-training

We thank the reviewer for this question. We clarify that our experiments already include models extensively trained with long-CoT RL. Importantly, unlike the smaller Qwen3 models that primarily undergo distillation, Qwen3-32B and Qwen3-235B-A22B are directly trained with GRPO-based reasoning RL and general RL, as documented in the Qwen3 Technical Report (Yang et al., 2025). FLEx improves solve@8 by +3.61pp and +3.62pp on these two models, respectively. Similarly, Granite 3.3 8B undergoes multi-stage RL with TPO and GRPO, and FLEx achieves a +3.29pp improvement. These results already demonstrate that FLEx remains effective on extensively RL-trained models across different families.

To further strengthen this evidence, we now additionally evaluate FLEx on two officially released RL-trained checkpoints from Tajwar et al. (Maximum Likelihood Reinforcement Learning, ICML 2026 Oral): Qwen3-4B-Base-GRPO-Polaris and Qwen3-4B-Base-MaxRL-Polaris. Both undergo 1,000 steps of dedicated in-domain math RL on POLARIS-53K. They share the same training pipeline but use different RL objectives.

We evaluate both checkpoints on AIME25, AIME26, HMMT26, and IMO-AnswerBench, following the same protocols and FLEx hyperparameters as Table 1. Results are shown below:

modelevalpass@8FLEx@8
Qwen3-4B-Base-GRPO-PolarisAIME2531.3332.00
AIME2630.6732.00
HMMT2628.4826.67
IMO-AnswerBench30.0032.75
mean Δ+0.73
Qwen3-4B-Base-MaxRL-PolarisAIME2532.6732.00
AIME2628.6729.33
HMMT2622.4224.85
IMO-AnswerBench33.5034.50
mean Δ+0.86

FLEx improves over pass@8 on 6 of 8 model × evaluation combinations, with mean gains of +0.73pp and +0.86pp under GRPO and MaxRL, respectively (McNemar significant at paired-level). Together with our original results on flagship Qwen3 and Granite 3.3 models, these experiments further demonstrate the effectiveness of FLEx after extensive RL post-training, across different model families and RL objectives.

[Q2]: Inference vs. training compute

We believe that training-time compute and inference-time compute are orthogonal to each other and the gains compound. The two checkpoints added in [W3] have already absorbed 1,000 steps of in-domain math RL, precisely the training compute being proposed. FLEx still improves them under two distinct RL objectives (GRPO and MaxRL).

As for the argument where industry spends most of the compute on inference rather than training, we believe our results in Sec. 4 i.e., FLEx in post-training demonstrate that we are able to curate rejection sampling training data with FLEx to train the model. Once it’s trained, the model performs significantly better under the same plain independent sampling regime. In this scenario, we get the benefit of FLEx without spending additional compute during inference. We will discuss the implications of our post-training experiment in our revised manuscript to reflect this point.

Dear Reviewer bcxM,

As the author-reviewer discussion period is approaching its end tomorrow, we wanted to follow up in case you have had a chance to consider our response, particularly the new compute-matched comparisons and additional experiments on extensively RL-trained checkpoints.

We sincerely hope these additional results have helped address your concerns. We would be very grateful for any updated thoughts or remaining questions you might have, and would welcome the opportunity to provide further clarification before the discussion closes.

Thank you again for your time and thoughtful consideration!

Best,

The Authors

Dear Reviewer bcxM,

As the author-reviewer discussion period comes to a close, we wanted to reach out one final time.

We have made a substantial effort to address your concerns, particularly through additional compute-aware comparisons and new experiments on extensively RL-trained checkpoints under both GRPO and MaxRL. We sincerely hope these results have helped address your questions.

We would be very grateful if you could share any updated thoughts on our responses, particularly whether your main concerns have been addressed or if any remain unresolved. We remain available to respond promptly over the next few hours and would sincerely appreciate the opportunity for further discussion.

Thank you again for your time and consideration.

Best,

The Authors

Reviewer_TB9NOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sequential revision often repeats old errors, while parallel sampling ignores failed attempts. FLEx (Sec. 2, Alg. 1) runs one model on both the clean prompt and prior failures, sampling from zc+λ(zc−zf)z_c + \lambda(z_c - z_f) with a clean-logit mask. Across eight models and six tasks, FLEx most often beats comparable baselines.

Critical strengths
  1. Very nice table of results. i liked the extensive model ablations, across model sizes.
  2. The baselines are meaningful. Not trivial or weak baselines. This underscores the good score that the authors' own method gets.
Critical weaknesses
  1. Main weakness: Why does test time scaling not include number of tokens, but instead works on number of turns?
  2. How can you get correct answer in case you don't already know the answer? Can you please use majority@k instead of oracle answers? What is the performance impact of doing this?
  3. I want to isolate the confounding effect of token count here. We know that longer thought traces should lead to better results. But does this model get better performance due to that or while keeping token count stable?
  4. The abstract says gains "consistently increase" with more sampling budget, but clearly something has to give as small models will plateau. Can this limit fo scaling be found? also please change the wording in the Abstract slightly to reflect the same.
  5. You are training on 20k additional prompts, so how do you know the gain does not come from that knowledge, and rather the scaling method?
Questions for authors

Q1. HPlease add accuracy vs total generated tokens, prefill-plus-decode FLOPs, and wall-clock latency? Q2. Related work also uses similar context-conditioned methods: context distillation (Snell et al., 2022), TTEL (Chitale et al., 2026), and SDPO (Hübotter et al., 2026). Discussing these in Sec. 5, along with the negative self-distillation work cited only at L204 (Pei et al., 2026), would sharpen the novelty claim.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TB9N [1/2]

We thank the reviewer for the positive assessment of our extensive model ablations and meaningful baselines. We address the concerns and questions below.

[W1, W3]: Turn-based vs. token-based scaling

We thank the reviewer for the suggestion. We wanted to clarify that scaling by the number of turns is a well-established evaluation protocol in test-time search. This is exemplified by some of the most cited existing literature e.g., codex [1], reflexion [2], and the scaling study on inference compute with repeated sampling [3]. Given FLEx's atomic unit is a turn, we believe scaling by the number of turns is the more natural setup.

Nevertheless, we agree that token efficiency is also an important dimension and a positive result can further strengthen the effectiveness of FLEx. Therefore, we re-analyze our k=256 scaling experiment (Sec 3.4; Fig 5) with generated tokens on the x-axis:

tokens/problemindependentFLExΔ
43k32.9734.45+1.49
81k36.8138.61+1.79
155k39.2644.91+5.65
287k43.8450.18+6.34
530k46.8754.09+7.22
982k51.6757.30+5.63

FLEx is better than independent sampling at every token budget. Thus the gain is not an artifact of longer traces.

[1] Evaluating Large Language Models Trained on Code

[2] Reflexion: Language Agents with Verbal Reinforcement Learning

[3] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

[W2]: Majority@k metric and the performance impact

We thank the reviewer for this suggestion. As expressed in our manuscript, FLEx focuses on verifier-guided test-time search. Our contribution is to improve how trajectories are sampled, rather than how they are verified.

Nevertheless, we now additionally evaluate a verifier-free setting. We assume that no verifier is available and every turn is appended to the contrastive branch regardless of correctness. We evaluate four Qwen3 scales (4B, 8B, 14B, 30B-A3B) on AIME25 and HMMT26 across 5 seeds, and report majority@4 through majority@64.

Two metrics we believe are critical to understand how performance changes:

Coverage = probability the correct answer appears among the k samples.

Conversion = probability the correct answer wins the vote given that it appears.

Maj@k is the product of coverage and conversion:

kpassFLExΔ maj@kΔ coverageΔ conversion
422.2922.44+0.15+2.91−6.19
824.8026.70+1.91+4.89−4.29
1626.5728.65+2.08+6.08−4.74
3227.4929.51+2.02+6.88−4.67
6427.5829.43+1.86+6.92−3.70

Δ columns are FLEx − independent sampling.

We observe a clear tradeoff. FLEx improves coverage (+2.9 to +6.9pp, growing with k) but reduces conversion (−3.7 to −6.2pp). The reduction in conversion further supports the mechanism of FLEx. It steers generation away from what is present in the contrasted branch, including potentially correct trajectories as in a completely verifier-free setup.

Despite this tradeoff, FLEx improves the majority@k by +0.2 to +2.1pp across k=4 to 64 in this setup. These results suggest that FLEx's exploration benefits could extend to verifier-free settings, although it highly depends on the balance between coverage and conversion.

[W4]: Limit of scaling

We concur that gains inevitably saturate as the sampling budget approaches infinity, regardless of the sampling method. Our result is to show that FLEx consistently outperforms independent sampling up to a fairly large budget as shown in Figure 5. We will update the abstract wording to remove the ambiguity between increasing absolute solve@k and consistent gains over independent sampling, where we meant the latter.

[W5]: Controlling for additional training prompts

We thank the reviewer for this question. Our post-training experiments are not comparing SFT with FLEx-collected data against the base model. As shown in Table 2, only the first row reflects base model performance. In every block, we compare SFT with pass@8-collected data against SFT with FLEx@8-collected data under the exact same sampling budget (i.e., 8 attempts) and the same 20K training prompts.

To further control the SFT training budget, we compare not only performance after 1 epoch, where FLEx-collected data contains more tokens, but also performance after the same number of training tokens. In both settings, models trained with FLEx-collected data achieve better downstream performance.

We believe our controlled setup rules out the confounding factor of additional training prompts; our token-matched comparison further controls for differences in SFT training compute.

AuthorsOpenReview ↗

Response to Reviewer TB9N [2/2]

[Q1]: Accuracy vs total generated tokens, prefill-plus-decode FLOPs, and wall-clock latency

We reported accuracy vs total generated tokens as part of our response to [W1, W3]. Regarding prefill-plus-decode FLOPs and wall-clock latency, we charge FLEx two forward passes per generated token.

FLOP budget (token-equiv/problem)independentFLExΔ
86k36.9534.45−2.50
163k39.5738.61−0.97
310k44.1444.91+0.77
573k47.3350.18+2.85
1,045k52.3353.97+1.64

Under this stricter accounting, FLEx trails below ~300k token-equivalents and leads above. The table stops at 1,045k. This is the point where independent sampling fully consumes its k=256 budget.

Our empirical observation on latency is that FLEx takes ~14x time to reach k=256 compared to independent sampling. Three points qualify this. First, our implementation is a research prototype built on vLLM with no system-level optimization. vLLM natively does not support contrastive decoding of two different contexts. Second, the cost is amortizable: the two requests of a turn are batched within a single engine pass, and throughput recovers substantially when many problems are served concurrently. Third, we would emphasize that FLEx's contribution is to make sequential revision outperform independent sampling. To our knowledge this is the first method to upcycle failed attempts into negative guidance instead of discarding them.

Finally, latency need not be paid at serving time. Our rejection-sampling SFT experiment (Sec. 4) shows that post-training on FLEx-generated data yields a checkpoint that outperforms its baseline under plain independent sampling at inference. In that deployment pattern, FLEx is a one-off offline data-generation cost, and the served model carries no additional inference overhead at all.

[Q2]: Comparison with Context Distillation, TTEL, SDPO and Negative Self-Distillation

We thank the reviewer for pointing out these relevant works. While these methods also leverage context-conditioned distributions, they differ from FLEx in how the contrast is constructed and used. In particular, FLEx constructs its contrast from accumulated, instance-specific failed attempts and extrapolates away from their induced logit shifts to encourage exploration. Although FLEx can also be adapted for distillation, its construction of teacher logits would differ fundamentally from methods that distill predictions conditioned on privileged information.

For conciseness, the following table summarizes the distinctions in contrast source and use of contrast:

MethodContrast sourceUse of contrast
Context distillationRich vs. reduced contextTeacher distillation
SDPOFeedback-conditioned vs. original policyPolicy distillation
TTELFeedback vs. null-feedbackError localization
NSDModel-generated negative instructionsNegative distillation
FLEx (Ours)Accumulated, instance-specific failed attempts vs. clean contextLogit extrapolation for exploration

Dear Reviewer TB9N,

As the author-reviewer discussion period is approaching its end tomorrow, we wanted to follow up in case you have had a chance to consider our response, particularly the additional token- and compute-aware comparisons, verifier-free majority@k experiments, and matched post-training controls.

We sincerely hope these additional analyses have helped address your questions. We would be very grateful for any updated thoughts or remaining concerns you might have, and would welcome the opportunity to provide further clarification before the discussion closes.

Thank you again for your time and thoughtful consideration!

Best,

The Authors

Dear Reviewer TB9N,

As the author-reviewer discussion period comes to a close, we wanted to thank you again for your thoughtful review and constructive suggestions.

We sincerely hope that our additional token- and compute-aware analyses, verifier-free evaluations, and matched post-training controls have helped clarify our contributions and strengthen the paper.

We greatly appreciate your time and consideration, and remain available for any further clarification before the discussion period concludes.

Best,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Official Reviews of Submission5583

We thank all reviewers for their careful assessment and constructive feedback. Reviewers recognized FLEx's simplicity as a training-free method and its consistent improvements across diverse models and benchmarks (Reviewers bcxM and H84i). They also highlighted the extensive evaluation across model scales, the meaningful baselines and ablations (Reviewers TB9N and H84i), and the practical benefits of FLEx-generated trajectories for post-training (Reviewers bcxM and H84i).

In response to the reviews, we have:

  • added token-matched and compute-aware comparisons, including wall-clock latency analysis;
  • evaluated verifier-free majority@k, analyzing the tradeoff between solution coverage and vote conversion;
  • demonstrated FLEx's effectiveness on additional checkpoints trained with extensive in-domain RL under GRPO and MaxRL;
  • clarified the contribution of instance-specific failure guidance relative to existing contrastive decoding and related methods;
  • reported additional hyperparameter sweeps and clarified extrapolation direction and tuning costs;
  • clarified the matched training prompts and token budgets in our post-training experiments; and
  • refined the motivation, verifier-guided setting, and scope of our test-time scaling claims.

Detailed responses and experimental results are provided under each reviewer's comments. We are committed to revising our manuscript accordingly. We sincerely thank all reviewers again for their constructive suggestions, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.