acceptodds
Under review as a conference paper at ICLR 2027

Can Language Models Learn from Failures Alone at Test Time?

Abstract

Test-time training (TTT), where language models (LMs) undergo parameter updates during inference, has emerged as a promising approach for adapting LMs to individual problems. Prior work on problem-specific TTT often operates in hill-climbing settings, where candidate solutions can be evaluated with a fine-grained scalar reward. This paper asks: how far can test-time training go when learning only from failures, without such fine-grained feedback? We adapt a range of existing optimization methods to this regime and propose Controlled Redistribution (CoRed), which treats failure-only TTT as controlled reshaping of the model's token-level predictive distribution. Across countdown games, math, and code tasks, CoRed improves both solvability and sample efficiency. On challenging math problems, CoRed enables Qwen3-4B-Instruct to solve more than as many problems as the frozen base model within 4,096 attempts, while substantially reducing the number of attempts required to reach the first correct solution on problems that the base model can eventually solve within 4,096 attempts. Our results demonstrate that substantial test-time adaptation is possible even when learning from failures alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test time training, test time adaption, distributional perspective, large language models, learning from failure

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mSrYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study the problem of test-time adaptation from binary failure feedback alone, without successful demonstrations or graded rewards. They propose CORED, which combines a smoothed variation of the unlikelihood objective with KL regularization toward the original model. Experiments on arithmetic, competition math, and coding problems show improvements in solve rates and attempts required to reach a first correct solution compared with frozen-model sampling. CORED seems to be an incremental combination of a modified unlikelihood objective and standard KL regularization. The paper’s stronger contribution appears to be its empirical investigation of failure-only test-time adaptation, rather than the optimization method itself.

Critical strengths

This paper explores an interesting and difficult scenario, which is test-time adaptation using solely binary failure feedback and no successful instances or reward functions.

Experiments cover multiple reasoning heavy domains in two different models and show that updates on failure cases alone can increase the success rate and enhance first success sample efficiency.

The effect of smoothing and KL regularization is analyzed through ablations and the need for regulating the distance from the base model is demonstrated.

Comparison with multiple adapted baselines, such as SDPO, allows separating the effects of failure-only adaptation from CORED in particular.

Critical weaknesses

CORED combines a modified unlikelihood loss with standard KL regularization. Its incremental gains over unlikelihood plus KL are modest.

Inconsistent superiority over strong baselines. SDPO is competitive and matches CORED’s final solve rate on Qwen coding.

Questions for authors

Did the authors evaluate different feedback prompts for SDPO while preserving the binary-feedback constraint? How sensitive are its results to the wording of the incorrect-answer verdict?

Beyond the component ablations, how sensitive is CORED to the smoothing parameter and KL coefficient?

How does CORED compare with frozen-model sampling when temperature is tuned for exploration and total compute is matched?

2 comments

We thank the reviewer for all of your comments. Here we will address your concerns in the following.

Q1: CORED combines a modified unlikelihood loss with standard KL regularization. Its incremental gains over unlikelihood plus KL are modest.

A1: Thank you for your comment. We see our contribution as twofold, and we respectfully disagree that the gains are modest. Clarification of contribution:

  1. A systematic study of an underexplored setting. We study test-time learning on problems the base model cannot solve at a high sampling budget (for instance, 4,096). The only feedback is that an attempt is wrong with no correct solution, reference answer, or partial credit. To our knowledge, this is the first comprehensive study of failure-only learning at this difficulty and scale. We break failure-only learning into two design choices: (i) how strongly to suppress each token of a failed attempt, and (ii) where the removed probability mass goes. We also define an evaluation protocol based on the number of attempts needed to solve a problem. Within this framework, we test six objectives. The results show that, with the right design and constraints, a model that learns only from its own failures can solve problems the frozen model cannot. Several objectives (CoRed, SDPO, NSR) make progress beyond the frozen model (Section 4). We hope this opens a new direction for test-time training.

  2. Simple by design yet effective. We see the simplicity of our method as a strength. It comes from a specific weakness of unlikelihood training (UL) (in Section 3): the per-token gradient is proportional to the token's probability, so most of the pressure falls on high-confidence tokens. In a failed attempt, these are mostly routine, correct steps that just happen to appear in a wrong response. Our smoothed objective shifts relative pressure toward low-probability tokens, reshaping the distribution more balanced than under UL.

Importance of SMO: we would like to clarify that we provided an ablation study in Section 4 (Figure 8) of our paper. On hard math problems, full CoRED improves on UL + KL by more than 25% (relative) (Section 4, Figure 8). While we agree that the KL anchor is essential to prevent degeneration, even so, the gains of SMO are substantial.

We appreciate the comment and we will state our contributions more clearly in the revised version.

Q2: Inconsistent superiority over strong baselines. SDPO is competitive and matches CORED’s final solve rate on Qwen coding.

A2: We thank the reviewer for this comment and agree that SDPO is a strong baseline. Across the five settings, however, CoRed outperforms SDPO in four and ties it in one (Qwen3-4B-Instruct on Code):

SettingCoRedSDPO
Qwen3-4B-Inst, Countdown50.0%20.0%
Qwen3-4B-Inst, Math46.7%40.0%
Llama-3.2-3B-Inst, Code42.0%38.0%
Llama-3.2-3B-Inst, Math46.7%33.3%
Qwen3-4B-Inst, Code16.7%16.7%

The tie only occurs in the hardest setting, where both methods have their lowest solve rate (16.7%). Here both methods solve only a few of the hardest problems, which leaves little room to separate them. Thanks for raising it.

Q3: Did the authors evaluate different feedback prompts for SDPO while preserving the binary-feedback constraint? How sensitive are its results to the wording of the incorrect-answer verdict?

A3: We thank the reviewer for this question. To test how sensitive SDPO is to the feedback wording, we ran it with a second, independently prompt that keeps the binary-feedback constraint. The original prompt (used in the paper) and the new one are:

Original paper

The following is feedback from your unsuccessful earlier attempt:
Verdict: WRONG ANSWER. Your program did not solve the problem.
Correctly solve the original question.

Alternative:

Evaluation of the previous attempt:
Verdict: INCORRECT. The previous program was not accepted.
Reconsider the problem from the beginning and provide a complete, correct solution.

The two prompts differ in all three components: the header, the verdict phrasing, and the closing instruction. We evaluated both with Llama3.2-3-Instruct on Code, using 15 randomly sampled problems:

PromptSolved / Total
SDPO, original prompt (paper)5 / 15
SDPO, alternative prompt4 / 15
CoRed8 / 15

The two prompts differ by a single problem, so SDPO's results are robust to the wording of the binary verdict. Our comparison does not depend on a favorable choice of prompt for SDPO. We will add this ablation and provide both prompts to the appendix in the future revision. Thanks for suggesting that.

Q4: Beyond the component ablations, how sensitive is CORED to the smoothing parameter and KL coefficient?

A4: We thank the reviewer for raising the question of hyperparameter sensitivity. We conduct additional experiments using different smoothing parameters and KL coefficient in the following.

(a) Different smoothing parameters All experiments in the paper use γ = 0.5. To test sensitivity to the smoothing parameter, we ran CoRed with γ ∈ {0.25, 0.5, 0.75} on Code with Llama-3.2-3B-Instruct. Because of compute and time limits during the rebuttal period, we used 15 problems randomly sampled from the Code set in the paper.

Llama on code:

MethodSolved / TotalSolve rate
Naive sampling2 / 1513.3%
CoRed, γ = 0.255 / 1533.3%
CoRed, γ = 0.5 (default)5 / 1533.3%
CoRed, γ = 0.754 / 1526.7%
CoRed, γ = 1 (unlikelihood training + KL)3 / 1520.0%

Across the three values of γ={0.25, 0.5, 0.75}, the number of solved problems differs by at most one. The small drop at γ = 0.75 is expected: as γ approaches 1, the objective approaches standard unlikelihood training, so the shift of pressure toward low-probability tokens weakens. Overall, CoRed is not sensitive to γ in the range 0.25–0.75 around our default of 0.5.

(b) Different KL coefficients:

In the paper we use a KL coefficient of β = 1.5 for Qwen and β = 3.0 for Llama. To test sensitivity, we ran CoRed with β ∈ {1.0, 1.5, 2.5} for Qwen3-4B-Instruct on Math, and with β ∈ {1.5, 2.0, 3.0, 4.0} for Llama-3.2-3B-Instruct on Code. Because of compute limits, each experiment uses 15 problems randomly sampled from the corresponding set in the paper.

Qwen3-4B-inst on Math

MethodSolved / Total
Naive sampling3 / 15
CoRed, β = 1.07 / 15
CoRed, β = 1.5 (default)8 / 15
CoRed, β = 2.56 / 15

Llama3.2-3B-Inst on code

MethodSolved / Total
Naive sampling2 / 15
CoRed, β = 1.54 / 15
CoRed, β = 2.05 / 15
CoRed, β = 3.0 (default)5 / 15
CoRed, β = 4.03 / 15

From above tables, the KL coefficient affects performance, but no single value is critical:

  • Qwen on Math. The default β = 1.5 does best (8/15), and β = 1.0 and β = 2.5 solve 7/15 and 6/15. Every setting solves at least twice as many problems as naive sampling (3/15).

  • Llama on Code. β = 1.5, 2.0, and 3.0 give similar results (4–5 of 15). Only β = 4.0 drops, to 3/15, which still matches or beats naive sampling (2/15).

These results suggest that within a reasonable range, however, performance is stable. The main failure mode is a coefficient that is too large: it holds the model too close to the base policy and cancels the distribution reshaping done by SMO. This is what we see at β = 4.0 on Llama.

Together with the smoothing parameter γ ablation above, these results suggest that both the smoothing parameter and the KL coefficient matter, but CoRed is robust within a broad range of each, and small changes have little effect. We will add these sensitivity experiments to the revised paper and extend them to the full problem sets. Thanks for raising this question.

Q5: How does CORED compare with frozen-model sampling when temperature is tuned for exploration and total compute is matched?

A5: We thank the reviewer for raising the question of temperature tuning. We run a temperature sweep on two tasks, Countdown and Math, using Qwen3-4B-Instruct-2507. Because of compute and time limits, we used 15 hard problems randomly sampled from the Math set in the paper, and 10 Countdown problems.

Qwen3-4B-Inst on Countdown and Math

MethodTemperatureCountdown (solved / 10)Math (solved / 15)
Naive sampling0.613
Naive sampling1.034
Naive sampling1.235
CoRed0.6 (default)58

The table shows the number of problems each method solves. As expected, higher temperatures help naive sampling. Even at its best temperature, however, naive sampling solves only 3/10 on Countdown and 5/15 on Math. CoRed at our default T = 0.6 solves 5/10 and 8/15. This comparison favors the baseline: naive sampling gets its best temperature chosen after the fact for each task, while CoRed uses a single default.

Temperature is also independent of the training method. It only controls how attempts are sampled, so a better temperature could help CoRed as well. Because of compute limits, we did not sweep temperature for CoRed. The current results already show that CoRed's gain does not come from a poorly tuned sampling temperature for the baseline.

We will add the temperature results to the revised paper. Thanks the reviewer for suggesting this point.

Reviewer_d683OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates using only failed attempts in test-time training. Rather than sampling multiple times in a "naive" approach, the authors use each sampled solution's label as a signal to update the sampling model's parameters. They propose the Controlled Redistribution (CoRed) method. They repeatedly sample candidate solutions, and when an entire batch fails, they update a new LoRA adapter using those unsuccessful trajectories. Adaptation terminates as soon as a verified solution is found. They then evaluate it on two model checkpoints across the countdown, math, and coding tasks. They compare their method to both naive sampling and existing test-time training methods. They find that CoReD can improve both sample efficiency and accuracy on hard problems.

Critical strengths
  1. The paper is very well executed and well written. The motivation, methods, and setup were clear. Every ablation and the core experiments made sense. The overarching report is well designed.
  2. The evaluation goes beyond just one model checkpoint or dataset. They thoroughly evaluated CoReD and highlighted the importance of their approach. The results are clear and objective, strengthening their contribution.
Critical weaknesses
  1. The overall novelty/impact is not as large as the paper makes it out to be. "“Learning to Reason from Feedback at Test-Time" from ACL 2025 appears to be a very similar method and setting to this work[1]. Thus the actual novelty from this paper is just the unlikelihood loss modification. By itself, I am unsure if that passes the bar for a conference contribution. I am happy to raise my score if the authors can argue their novelty is greater than just this.
  2. The comparisons with naive sampling seem somewhat underdiscussed when neither the FLOPs used nor the actual thinking models are compared. If naive sampling is more FLOP-efficient for producing a correct solution, then what tangible value does CoReD have? Further, the rise of thinking models means sample efficiency is likely very different. Further, the experiments use two relatively small instruction-tuned checkpoints. In particular, Qwen3-4B-Instruct-2507 is explicitly a non-thinking checkpoint. A same-size Qwen3-4B-Thinking-2507 checkpoint is available, allowing us to test whether the proposed adaptation remains useful for a model already trained to perform extended reasoning.

[1] Yanyang Li, Michael R. Lyu, and Liwei Wang. Learning to Reason from Feedback at Test-Time. Proceedings of ACL 2025, pages 5241–5253.

Questions for authors
  1. Do the results change if you use a newer model at the same parameter size?
  2. Do the results change if you use a thinking model?
  3. How does CoReD compare on net FLOPs used to naive sampling? Can you use a smaller model with updating to beat out larger ones?
2 comments

Thanks for all your valuable comments. We will address your concerns here.

Q1: How to position this paper against FTTT [1], which appears to be a similar method and study a similar setting?

A1: Thank you for pointing us to FTTT, which is closely related work. Although our work and FTTT share a similar interface, the two methods target vastly different problem difficulties, which is reflected in both method design and experimental results. At a high level, FTTT improves efficiency on problems within the model's reach, while we target extremely hard problems. In particular, FTTT evaluates problems the model fails on with greedy decoding, of which pass@32 already solves 47–81% (Li et al., Table 2), and it competes with test-time scaling methods under a budget of 32 attempts. We evaluate problems the base model fails to solve in 1,024 samples, under a budget of 4,096 attempts. We briefly summarize the discussion and results here and include more details in our revised submission.

Experimental results: We ran an additional comparison with FTTT in our regime (new Appendix C), using its official implementation, with and without self-reflection, on 15 Math problems from our hard sets. Neither variant solves any problem within 4,096 attempts, falling below naive sampling (3/15). CoRed solves 8/15.

Technically different methods: CoRed and FTTT target different ranges of problem difficulty, which leads to different method designs. FTTT trains the model using self-generated feedback, which requires the model to already be able to reflect on and diagnose failed solutions; this can be unreliable for extremely hard problems. CoRed instead redistributes probability mass in a controlled way, directly shifting mass away from failed solutions and toward correct solutions in the long tail. We believe this difference in method design also explains the gap between FTTT and our method on hard problems, as our experimental results suggest.

We have incorporated these results and this discussion into newly added Appendix C.1. Thank you for the valuable question!

Q2:The comparisons with naive sampling seem somewhat underdiscussed when neither the If naive sampling is more FLOP-efficient for producing a correct solution, then what tangible value does CoReD have? How does CoReD compare on net FLOPs used to naive sampling?

A2: Thank you for your question. We would like to clarify that our paper already includes GPU runtime analyses. Table 1 (Section 4.4) compares the runtime of gradient updates with that of autoregressive sampling. Figure 9 (Appendix B.2) directly compares the runtime efficiency of CoRed and naive sampling using Qwen3-4B math problems. Despite the overhead of gradient updates, CoRed achieves substantially better performance under the same GPU-time budget: at eight hours per problem, CoRed solves approximately 40% of the problems, compared with less than 20% for naive sampling. This demonstrates that the reduction in sampling attempts translates into meaningful runtime improvements. As suggested, we additionally provide a breakdown of the computational cost in FLOPs, averaged over problems, in the table below.

PFLOP = 10^15 floating-point operations.

MethodSolve RateMean samplingMean updatesMean total
Naive sampling6/30198.34 PFLOP0198.34 PFLOP
CoRed14/3061.51 PFLOP228.29 PFLOP289.80 PFLOP

As shown, CoRed substantially reduces the FLOPs spent on sampling (61.51 vs. 198.34 PFLOPs), while incurring additional computation for gradient updates. Consequently, its total FLOP count is higher than that of naive sampling. However, FLOPs do not translate directly into GPU runtime, particularly when comparing gradient updates with autoregressive sampling. Gradient updates can process tokens in parallel, whereas autoregressive generation requires sequential token decoding, resulting in substantially different runtime ( Table 1 ). Therefore, although CoRed incurs higher total FLOPs, it achieves a substantially higher solve rate under the same GPU-time budget, as demonstrated in Figure 9 (Appendix B.2).

We will add FLOP analysis in the future revision. Thanks for suggesting the FLOP issue.

Q3: How does the method work on newer model or thinking model at the same parameter size?

A3: We thank the reviewer for suggesting additional models. We ran a preliminary experiment with Qwen3.5-4B, a newer model, on 15 problems from the Math set. The goal was to check whether test-time training still improves over naive sampling on a more recent model. We followed the protocol from the paper. We first ran naive sampling and kept only the problems it did not solve within 1,024 attempts, which left 9 problems (as it is a newer, stronger model). We then gave each method a budget of 4,096 attempts on these 9 problems. Naive sampling solves 2 of the 9 problems, and CoRed solves 3 of the 9. Since the sample is small and the difference is a single problem, we see this only as preliminary evidence that the gain carries over to newer models. We will run full baselines on larger problem sets drawn from newer, more difficult problems in future versions.

We also experimented with Qwen3-4B in thinking mode on 15 problems from the Math set. With a 16k-token generation limit, about 95% of the responses were cut off before reaching a final answer (for naive sampling). Running with a higher generation limit is beyond the compute resources available to us during the rebuttal period.

We acknowledge this limitation and leave it as important future work. Thanks for suggesting the additional models.

Q4: Can you use a smaller model with updating to beat out larger ones?

A4. We thank the reviewer for this question. Our preliminary results suggest that it can. We ran Qwen3-8B (thinking disabled) with naive sampling on the same 15 Math problems used in our other ablations, with the same budget of 4,096 attempts:

Solving rate table

Model – MethodSolved / Total
Qwen3-4B-Inst – Naive sampling3 / 15
Qwen3-4B-Inst – CoRed8 / 15
Qwen3-8B – Naive sampling4 / 15

Doubling the model size adds only one solved problem under naive sampling. CoRed on the 4B model solves twice as many problems as the 8B model. The comparison also favors the 8B model, because these problems were selected as hard for the 4B model, not for the 8B model. This suggests that, on hard problems, test-time learning can be a more effective use of compute than scaling up model size.

We will add these results, and results on the full set in the future revision. Thanks for proposing this question.

Reviewer_XdiHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, a verifier rejects attempts until the first success, so every update sees only failed attempts. CORED trains a LoRA adapter on test problems during search. After some batch of failed attempts, the probability reduces from the tokens in those attempts with a damped variant of an ``unlikelihood loss''. Additionally, a KL penalty keeps the model close to the original.

The experimental results are good -- on 30 hard math problems, Qwen3-4B solves 14 with CORED versus 6 with plain repeated sampling within 4,096 attempts. Gains on Llama-3.2-3B and on code are smaller or mixed.

Critical strengths
  1. The research question posed is very clean. The failure-only setting is well posed (binary verifier, stop at first success), and five existing objectives are compared under one protocol and budget (Sec. 2.3).

  2. On the particular Qwen model, we find that learning from failures does help. CORED solves 14 of 30 problems versus 6 for repeated sampling, and two adapted baselines, SDPO and NSR, also beat repeated sampling.

Critical weaknesses

Missing References:

The most direct way to learn from failures at test time is to show them to the model, for example "these answers were wrong, try again". The paper has no such in-context or revision baseline, even though SDPO's teacher already is the model reading the failure feedback, so sampling from it directly costs no training. Training-free methods that contrast feedback-conditioned and baseline token distributions to localize errors during test-time search (TTEL, Chitale et al., 2026) would also be natural baselines. Close prior work is not discussed either: test-time optimization from feedback (FTTT, Li et al., 2025) addresses nearly the same problem, and GRPO-style training has already been extended to failures, with self-reflection under binary feedback (Reflect, Retry, Reward; Bensal et al., 2025) and with critiques of failed solutions (Critique-GRPO, Zhang et al., 2025). Can you please compare against at least the in-context baseline and FTTT, and position the work against these?


Compute = Sampling Flops + gradient flops. Cored has the latter and therefore attempts alone is not a fair comparison? Can you please include tokens as well as GPU hours for comparison?


Whether the gain is coming from the new loss or the KL anchor is not clear. For example, if you just do num_tokens maximization with KL the gain should be high as well. Is the method just providing this or something more?

Questions for authors
  1. Please can you provide token counts. I think there might be a confounder. The token length is probably increasing as you keep reducing probability of EOS on shorter length responses. Is this true? There is nothing wrong with this, but can you please report it?

  2. What was the impact of the KL term? can you provide a with KL and without KL ablation of your method?

3 comments

We thank the reviewer for all your comments. We address your concerns as follow.

Q1 (a): How does the proposed method compare against in-context baselines?

A1 (a): Thank you for this suggestion. We conducted additional preliminary experiments, comparing our method against both in-context baseline. As suggested, we prompt the model using the last-generated failed response followed by a message “the answer is wrong. Try again”, and resample another solution from the model until success. We test and provide preliminary comparisons against these new baselines. Given compute time and budget limit during the rebuttal phase, we use a random sample of 15 problems from Math (using Qwen3-4B) and Code (using LLama-3B), two settings that allow naive sampling to achieve a nontrial solve rate on small random subsets.

Math — problems solved (out of 15)

Method512102420484096
Naive1113
In-Context Retry1111
CoRed2458

Code — problems solved (out of 15)

Method512102420484096
Naive0003
In-Context Retry1114
CoRed3555

As shown in the Tables (detailed results in new Appendix C.1), in-context retry indeed outperforms naive sampling, solving one additional problem on Code. CoRed solves 5/15 on Code and 8/15 on Math. We believe the one reason for in-context retry underperforming CoRed comes from that the in-context baseline conditions on only one failure at a time, discarding all previous failure attempts. CoRed, in contrast, accumulates every failure in its weights. Thanks again for suggesting this valuable baseline. We will include a full comparison in future versions.

Q1 (b): How does the proposed method compare against FTTT, and how to position the work against FTTT?

A1 (b): Thank you for pointing us to this related work. Overall. We find FTTT is not effective on the extreme difficulty level studied in our work, as suggested by our experimental results and the design of methods.

Experiments: We use the official implementation provided by FTTT, and test two variants in [1]. FTTT with self-reflection: conditioned on a failed solution, trains the model to predict "Your answer is incorrect” and a reflection generated by the model itself. FTTT w/o self reflection: trains the model to only predict “your answer is incorrect”. We test two variants of FTTT on a random subset of 15 hard math problems using Qwen3-4B.

Naive SamplingFTTT w/ reflectionFTTT w/o reflectionCoRed
Hard Math (pass@1024 = 0)3/150/150/158/15

As shown in the Table above, FTTT solves none of the hard problems. FTTT (w/o reflection0 generally degenerates, while FTTT with self-reflection mostly exhausts the budgets of 4K attempts. We would like to note that we use the official implementation in our experiments. Given FTTT’s original experiment set up uses a max budget of 32, we additionally test FTTT on medium level set (pass@64 <= 1) and find FTTT is able to solve problems in that regime, as good as naive sampling.

CoRed and FTTT targets different scopes of problem difficulty, as suggested by our experiment's results and the evaluation set up of the original paper (32 attempts at max). We believe these gaps can be attributed to the method design. FTTT trains the model using self-generated feedback, which requires model to already possess reasonable capabilities to reflect and diagnose failure solution, which can be unreliable for extremely hard problems. CoRed instead optimize to redistribute the weights in a controlled way, which directly shifts the mass from failures solution and (potentially) incentive correct solutions in the long-tail.

We incorporate these results and suggestions in new Appendix C.1, thanks for the valuable suggestion!

[1] Yanyang Li, Michael R. Lyu, and Liwei Wang. Learning to Reason from Feedback at Test-Time. Proceedings of ACL 2025, pages 5241–5253.

Q2: Cored has the gradients FLOPs and therefore the attempt may not be a fair comparison. Can you please include tokens as well as GPU hours for comparison?

A2: Thank you for your question. We would like to clarify that our paper already includes GPU runtime analyses. Table 1 (Section 4.4) compares the runtime of gradient updates with that of autoregressive sampling. Figure 9 (Appendix B.2) directly compares the runtime efficiency of CoRed and naive sampling using Qwen3-4B math problems. Despite the overhead of gradient updates, CoRed achieves substantially better performance under the same GPU-time budget: at eight hours per problem, CoRed solves approximately 40% of the problems, compared with less than 20% for naive sampling. This demonstrates that the reduction in sampling attempts translates into meaningful runtime improvements.

As suggested, we additionally provide a breakdown of the computational cost in FLOPs, averaged over problems, in the table below. PFLOP = 10^15 floating-point operations.

MethodSolve RateMean samplingMean updatesMean total
Naive sampling6/30198.34 PFLOP0198.34 PFLOP
CoRed14/3061.51 PFLOP228.29 PFLOP289.80 PFLOP

As shown, CoRed substantially reduces the FLOPs spent on sampling (61.51 vs. 198.34 PFLOPs), while incurring additional computation for gradient updates. Consequently, its total FLOP count is higher than that of naive sampling. However, FLOPs do not translate directly into GPU runtime, particularly when comparing gradient updates with autoregressive sampling. Gradient updates can process tokens in parallel, whereas autoregressive generation requires sequential token decoding, resulting in substantially different runtime ( Table 1 ). Therefore, although CoRed incurs higher total FLOPs, it achieves a substantially higher solve rate under the same GPU-time budget, as demonstrated in Figure 9 (Appendix B.2).

Thanks for your valuable comment on FLOPs. We will add FLOP analysis in the future revision.

Q3 (a): Does the gain come from increasing response token lens? Can you provide token counts?

A3(a): Thank you for the interesting question!

We conduct analysis on the token length on Math with Qwen3-4B-Instruct. We have added a new subsection of length analysis in Appendix C.2. Overall we do not find substantially longer length contributes to the successes of CoRed. The statistics from Table 6 and Figure 12 (in added Appendix C.2) show that CoRed and Naive sampling response length distributions are highly overlapped over the sampling process across 500 batches of 8 samples, totaling 4000 samples. We added our more detailed analysis regarding response length in Appendix C.2.

Table 6 (in Appendix C.2): Response length (in tokens) statistics, averaged over problems in Math with Qwen3-4B-Instruct

Generated response lengthNaive samplingCoRed
Mean tokens6,9607,179
Median tokens7,8778,159
90th percentile11,08511,341

Q3(b) What if you just do num_tokens maximization with KL?

A3(b): We tested length maximization by performing SFT with the longest out of 8 generated samples in each batch, with the same base policy KL constraint as CoRed. We conduct experiments on 15 random problems from Math with Qwen3-4B-Instruct.

MethodSolved / Total
Naive sampling3 / 15
Length maximization4 / 15
CoRed8 / 15

The length-maximization baseline solves 4/15 problems, one more than naive sampling (3/15) but well below CoRed (8/15). Maximizing length under a KL constraint therefore yields only a modest gain.

Figure 13 in Appendix C.2 shows the mean response length rises above that of naive sampling and then keeps fluctuating at a higher response length level. However, longer responses do not lead to more effective exploration to correct solutions. We therefore conclude that length bias alone cannot explain the gains of CoRed. These details are also reflected correspondingly in our Appendix C.2.

Thanks for raising the point of length analysis.

Q4: Whether the gain is coming from the new loss or the KL anchor is not clear. Can you provide a with KL and without KL ablation of your method?

A4: Thanks for raising the KL ablation question. In Section 4.4 from our original submission, we did an ablation study for the contribution of both SMO (i.e. the proposed loss term) and the KL term. Specifically, we reported our ablation results with Qwen3-4B-Instruct-2507 in Figure 8 and Llama3.2-3B-Instruct in Appendix B.3. This table summarizes our results from Figure 8.

Ablation study of SMO and KL on CoRed final performance on Math for Qwen

MethodNaive samplingCoRedCoRed (w/o SMO)CoRed (w/o KL)
Solve rate20.0%46.7%36.7%13.3%

As shown in the table above and the original Figure 8, removing either component degrades both final solve rate and sample efficiency. In particular, removing the KL constraint leads to severe degeneration (CoRed w/o KL), and removing SMO term (CoRed w/o SMO) leads to inferior performance. So both terms jointly contribute to the final effectiveness of our approach.

Thanks for suggesting that.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.