- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Based on TTRL, this paper is a combination of self-certainty weighting, weighted voting, answer entropy-based HOAS, and embedding-based clustering. The results are rich and better than the TTRL baseline.
Critical strengths
The paper identifies several weaknesses in standard TTRL and tries to address them at different levels, especially through uncertainty-aware weighting and finer-grained credit assignment.
The empirical study is quite thorough, with multiple models and benchmarks, matched multi-seed runs, ablations, and efficiency analysis, and the overall results are consistently positive.
Critical weaknesses
I am not fully convinced by the HOAS design. More ambiguous prompts are given larger weights, but these prompts may also have less reliable pseudo-targets. The paper shows positive empirical gains, but does not directly establish that higher ambiguity corresponds to more useful rather than noisier supervision.
The “reasoning credit” is built from embedding similarity and clustering of target-matching traces. This can certainly create finer-grained scores, but it is not clear that these scores actually measure better reasoning rather than just semantic similarity or stylistic similarity.
Questions for authors
If the soft vote is intended to produce a more reliable pseudo-target, why is direct pseudo-target accuracy against the available gold answer not reported? The current “reward accuracy” can count jointly negative decisions and therefore does not directly validate the claimed target-quality improvement.
2 comments
Response to Reviewer nGzv: What HOAS amplifies & Mathematical steps under equal rewards
What HOAS amplifies
Training effect. TCCA exceeds HOAS-Off by 1.18 pp in sample accuracy and 1.20 pp in majority-vote accuracy across three matched MATH-500 runs. We reset the HOAS weight to 1 when voting returned a usable pseudo-target, the group’s active answer entropy exceeded the batch center, and its original HOAS weight was greater than 1. To preserve the near-consensus treatment raised by f6oQ, we retain the original HOAS weights for groups in which exactly 31 of the 32 retained responses match the voted pseudo-target. This keeps the scaling of the lone nonmatching response in place while testing the contribution of the high-entropy boosts elsewhere. Both arms use the same voting and RC rules. At step 150, removing these boosts reduces sample accuracy from 41.29% to 39.90% and majority-vote accuracy from 43.00% to 41.00%.
Gold-based diagnostic. HOAS applies the same prompt weight to positive and negative outcome advantages. A weight above 1 can therefore strengthen positive feedback for a response that matches an incorrect pseudo-target. To quantify what happens to gold-correct responses, we use a frozen pool of 64 responses per MATH-500 problem sampled from the step-150 Qwen2.5-0.5B TCCA checkpoint trained with seed 0. We use all 64 candidates to reconstruct the voted pseudo-target and answer entropy, and compute the outcome advantages on the same first 32 responses per problem. We fix the rollouts, pseudo-targets, and offline batches of 32 prompts, then compare unit weights with HOAS weights recomputed on those batches. This replay involves no new sampling or parameter updates. Gold is used only to label targets and responses for analysis.
For prompt () and retained response (), let () be the GRPO-normalized outcome advantage before HOAS and RC, () the recomputed HOAS weight, and () an indicator that the response’s final answer matches gold. We measure the change in signed outcome coefficients on gold-correct responses:
g_{qi}(w_q A_{qi}-A_{qi}).$$ For this diagnostic, ($\mathcal{H}$) is the higher-entropy half of the prompts with valid pseudo-targets and positive entropy in the frozen pool, comprising 178 prompts. The 20 prompts with correct pseudo-targets contribute **+22.27**, and the 158 with incorrect pseudo-targets contribute **−8.17**. Both terms count only gold-correct responses, including correct responses whose negative advantage becomes more negative when the pseudo-target is wrong. The signed coefficient sum on gold-correct responses rises from **163.68 to 177.78**, a net change of **+14.10**. Since this statistic sums changes only over gold-correct responses, prompts with no such responses contribute zero. HOAS can still amplify positive advantages on those prompts when incorrect responses match the pseudo-target. ## Mathematical steps under equal rewards We show two pairs of saved rollouts. Within each pair, both responses have the same final answer and binary pseudo-reward. Two blinded model assessments judged the mathematics and the presentation separately. The scores below are the applied reasoning credit (RC) residuals from replaying the saved rollouts. - **Unicycle speed.** Both responses answer `12π` inches per second and receive reward 1. The higher-RC response correctly gives `18π` inches per revolution and `36π` inches for two revolutions in three seconds. The other uses the same arithmetic but calls `36π` the distance for one revolution before dividing by the time for two. The applied RC residuals are **+0.057918** and **−0.100000**, respectively. Both assessments favor the former on mathematics and tie the pair on presentation. - **Fibonacci series.** Both responses give the wrong answer `1/2` and receive reward 0. The higher-RC response writes the valid telescoping identity `F(n+1) / (F(n) × F(n+2)) = 1/F(n) − 1/F(n+2)`. The series sums to `2` using `F(1) = F(2) = 1`. The other response drops the numerator and uses `F(1) = 0` despite the stated initial value. The applied RC residuals are **+0.012271** and **−0.003458**, respectively. Both assessments favor the former on mathematics and the latter on presentation. The appendix reproduces both questions and their original responses. In the matched MATH-500 ablation, TCCA improves sample and majority-vote accuracy by **1.33 and 1.40 percentage points**, respectively, over RC-Off.
Response to Reviewer nGzv: Direct pseudo-target accuracy against gold
To address the reviewer’s concern about reward agreement, we directly check whether the pseudo-target used for each training prompt matches the gold answer. TCCA uses confidence-weighted voting. SoftVote-Off uses equal-count voting and recomputes the pseudo-target. Over all 150 training steps, TCCA selects 5,717/14,400 (39.70%) correct pseudo-targets, compared with 5,557/14,400 (38.59%) for SoftVote-Off, a difference of +1.11 percentage points.
At the end of the paired training runs, TCCA improves sample accuracy from 39.70% to 41.10% and majority-vote accuracy from 41.00% to 42.80% over SoftVote-Off.
Figure R1. Training-time comparison of TCCA and SoftVote-Off.
Panel A counts one selected pseudo-target per training prompt. Panel B counts each of the 32 retained responses per prompt separately. Both use gold answers. Each column pools 10 consecutive optimizer steps across three runs, and the column at 150 represents steps 141–150. Values are rounded to the nearest 2.5 percentage points for display.