- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors study the problem of test-time adaptation from binary failure feedback alone, without successful demonstrations or graded rewards. They propose CORED, which combines a smoothed variation of the unlikelihood objective with KL regularization toward the original model. Experiments on arithmetic, competition math, and coding problems show improvements in solve rates and attempts required to reach a first correct solution compared with frozen-model sampling. CORED seems to be an incremental combination of a modified unlikelihood objective and standard KL regularization. The paper’s stronger contribution appears to be its empirical investigation of failure-only test-time adaptation, rather than the optimization method itself.
Critical strengths
This paper explores an interesting and difficult scenario, which is test-time adaptation using solely binary failure feedback and no successful instances or reward functions.
Experiments cover multiple reasoning heavy domains in two different models and show that updates on failure cases alone can increase the success rate and enhance first success sample efficiency.
The effect of smoothing and KL regularization is analyzed through ablations and the need for regulating the distance from the base model is demonstrated.
Comparison with multiple adapted baselines, such as SDPO, allows separating the effects of failure-only adaptation from CORED in particular.
Critical weaknesses
CORED combines a modified unlikelihood loss with standard KL regularization. Its incremental gains over unlikelihood plus KL are modest.
Inconsistent superiority over strong baselines. SDPO is competitive and matches CORED’s final solve rate on Qwen coding.
Questions for authors
Did the authors evaluate different feedback prompts for SDPO while preserving the binary-feedback constraint? How sensitive are its results to the wording of the incorrect-answer verdict?
Beyond the component ablations, how sensitive is CORED to the smoothing parameter and KL coefficient?
How does CORED compare with frozen-model sampling when temperature is tuned for exploration and total compute is matched?
2 comments
Q4: Beyond the component ablations, how sensitive is CORED to the smoothing parameter and KL coefficient?
A4: We thank the reviewer for raising the question of hyperparameter sensitivity. We conduct additional experiments using different smoothing parameters and KL coefficient in the following.
(a) Different smoothing parameters All experiments in the paper use γ = 0.5. To test sensitivity to the smoothing parameter, we ran CoRed with γ ∈ {0.25, 0.5, 0.75} on Code with Llama-3.2-3B-Instruct. Because of compute and time limits during the rebuttal period, we used 15 problems randomly sampled from the Code set in the paper.
Llama on code:
| Method | Solved / Total | Solve rate |
|---|---|---|
| Naive sampling | 2 / 15 | 13.3% |
| CoRed, γ = 0.25 | 5 / 15 | 33.3% |
| CoRed, γ = 0.5 (default) | 5 / 15 | 33.3% |
| CoRed, γ = 0.75 | 4 / 15 | 26.7% |
| CoRed, γ = 1 (unlikelihood training + KL) | 3 / 15 | 20.0% |
Across the three values of γ={0.25, 0.5, 0.75}, the number of solved problems differs by at most one. The small drop at γ = 0.75 is expected: as γ approaches 1, the objective approaches standard unlikelihood training, so the shift of pressure toward low-probability tokens weakens. Overall, CoRed is not sensitive to γ in the range 0.25–0.75 around our default of 0.5.
(b) Different KL coefficients:
In the paper we use a KL coefficient of β = 1.5 for Qwen and β = 3.0 for Llama. To test sensitivity, we ran CoRed with β ∈ {1.0, 1.5, 2.5} for Qwen3-4B-Instruct on Math, and with β ∈ {1.5, 2.0, 3.0, 4.0} for Llama-3.2-3B-Instruct on Code. Because of compute limits, each experiment uses 15 problems randomly sampled from the corresponding set in the paper.
Qwen3-4B-inst on Math
| Method | Solved / Total |
|---|---|
| Naive sampling | 3 / 15 |
| CoRed, β = 1.0 | 7 / 15 |
| CoRed, β = 1.5 (default) | 8 / 15 |
| CoRed, β = 2.5 | 6 / 15 |
Llama3.2-3B-Inst on code
| Method | Solved / Total |
|---|---|
| Naive sampling | 2 / 15 |
| CoRed, β = 1.5 | 4 / 15 |
| CoRed, β = 2.0 | 5 / 15 |
| CoRed, β = 3.0 (default) | 5 / 15 |
| CoRed, β = 4.0 | 3 / 15 |
From above tables, the KL coefficient affects performance, but no single value is critical:
-
Qwen on Math. The default β = 1.5 does best (8/15), and β = 1.0 and β = 2.5 solve 7/15 and 6/15. Every setting solves at least twice as many problems as naive sampling (3/15).
-
Llama on Code. β = 1.5, 2.0, and 3.0 give similar results (4–5 of 15). Only β = 4.0 drops, to 3/15, which still matches or beats naive sampling (2/15).
These results suggest that within a reasonable range, however, performance is stable. The main failure mode is a coefficient that is too large: it holds the model too close to the base policy and cancels the distribution reshaping done by SMO. This is what we see at β = 4.0 on Llama.
Together with the smoothing parameter γ ablation above, these results suggest that both the smoothing parameter and the KL coefficient matter, but CoRed is robust within a broad range of each, and small changes have little effect. We will add these sensitivity experiments to the revised paper and extend them to the full problem sets. Thanks for raising this question.
Q5: How does CORED compare with frozen-model sampling when temperature is tuned for exploration and total compute is matched?
A5: We thank the reviewer for raising the question of temperature tuning. We run a temperature sweep on two tasks, Countdown and Math, using Qwen3-4B-Instruct-2507. Because of compute and time limits, we used 15 hard problems randomly sampled from the Math set in the paper, and 10 Countdown problems.
Qwen3-4B-Inst on Countdown and Math
| Method | Temperature | Countdown (solved / 10) | Math (solved / 15) |
|---|---|---|---|
| Naive sampling | 0.6 | 1 | 3 |
| Naive sampling | 1.0 | 3 | 4 |
| Naive sampling | 1.2 | 3 | 5 |
| CoRed | 0.6 (default) | 5 | 8 |
The table shows the number of problems each method solves. As expected, higher temperatures help naive sampling. Even at its best temperature, however, naive sampling solves only 3/10 on Countdown and 5/15 on Math. CoRed at our default T = 0.6 solves 5/10 and 8/15. This comparison favors the baseline: naive sampling gets its best temperature chosen after the fact for each task, while CoRed uses a single default.
Temperature is also independent of the training method. It only controls how attempts are sampled, so a better temperature could help CoRed as well. Because of compute limits, we did not sweep temperature for CoRed. The current results already show that CoRed's gain does not come from a poorly tuned sampling temperature for the baseline.
We will add the temperature results to the revised paper. Thanks the reviewer for suggesting this point.
We thank the reviewer for all of your comments. Here we will address your concerns in the following.
A1: Thank you for your comment. We see our contribution as twofold, and we respectfully disagree that the gains are modest. Clarification of contribution:
A systematic study of an underexplored setting. We study test-time learning on problems the base model cannot solve at a high sampling budget (for instance, 4,096). The only feedback is that an attempt is wrong with no correct solution, reference answer, or partial credit. To our knowledge, this is the first comprehensive study of failure-only learning at this difficulty and scale. We break failure-only learning into two design choices: (i) how strongly to suppress each token of a failed attempt, and (ii) where the removed probability mass goes. We also define an evaluation protocol based on the number of attempts needed to solve a problem. Within this framework, we test six objectives. The results show that, with the right design and constraints, a model that learns only from its own failures can solve problems the frozen model cannot. Several objectives (CoRed, SDPO, NSR) make progress beyond the frozen model (Section 4). We hope this opens a new direction for test-time training.
Simple by design yet effective. We see the simplicity of our method as a strength. It comes from a specific weakness of unlikelihood training (UL) (in Section 3): the per-token gradient is proportional to the token's probability, so most of the pressure falls on high-confidence tokens. In a failed attempt, these are mostly routine, correct steps that just happen to appear in a wrong response. Our smoothed objective shifts relative pressure toward low-probability tokens, reshaping the distribution more balanced than under UL.
Importance of SMO: we would like to clarify that we provided an ablation study in Section 4 (Figure 8) of our paper. On hard math problems, full CoRED improves on UL + KL by more than 25% (relative) (Section 4, Figure 8). While we agree that the KL anchor is essential to prevent degeneration, even so, the gains of SMO are substantial.
We appreciate the comment and we will state our contributions more clearly in the revised version.
A2: We thank the reviewer for this comment and agree that SDPO is a strong baseline. Across the five settings, however, CoRed outperforms SDPO in four and ties it in one (Qwen3-4B-Instruct on Code):
The tie only occurs in the hardest setting, where both methods have their lowest solve rate (16.7%). Here both methods solve only a few of the hardest problems, which leaves little room to separate them. Thanks for raising it.
A3: We thank the reviewer for this question. To test how sensitive SDPO is to the feedback wording, we ran it with a second, independently prompt that keeps the binary-feedback constraint. The original prompt (used in the paper) and the new one are:
Original paper
Alternative:
The two prompts differ in all three components: the header, the verdict phrasing, and the closing instruction. We evaluated both with Llama3.2-3-Instruct on Code, using 15 randomly sampled problems:
The two prompts differ by a single problem, so SDPO's results are robust to the wording of the binary verdict. Our comparison does not depend on a favorable choice of prompt for SDPO. We will add this ablation and provide both prompts to the appendix in the future revision. Thanks for suggesting that.