- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
DiSCTT is a label-free test-time adaptation framework that allocates different optimization objectives according to answer consensus. High-consensus questions are assigned to SFT, while low-consensus questions are assigned to GRPO,. The authors report substantial accuracy improvements over existing test-time RL approaches across multiple models and reasoning benchmarks, together with reduced adaptation compute.
Critical strengths
-
The idea of allocating different optimization objectives based on instance-level reliability is interesting.
-
The paper also includes extensive experimental comparisons and routing ablations.
Critical weaknesses
- I have concerns about the reported evaluation variance. Table 1 reports exceptionally small standard deviations across 10 runs described as independent in both training and sampling. For example, Qwen3-4B reports 51.4 ± 0.06 on MATH-500 and 27.3 ± 0.16 on GPQA-Diamond, with temperature 0.9 and top-p 0.9. Let indicate whether question is answered correctly in one independently sampled generation. Assuming independent sampling across questions, accuracy in percentage points is with variance For MATH-500 (), if the reported SD of 0.06 percentage points is representative of the true sampling SD, it implies
Questions for authors
-
How can the exceptionally small SDs in Table 1 be explained?
-
Are the wall-clock times in Table 6 measured per run or aggregated across ten seeds?
-
Why does Qwen3-32B achieve only 36.7% on AIME-2024, compared with 81.4% in the official Qwen3 Technical Report? Could the 2,048-token limit explain this discrepancy? Please report truncation rates and results with larger output budgets to clarify the source of the improvements.
-
What justifies the fixed consensus threshold of 0.45? The theoretical analysis assumes monotonic conditional risk differences but does not derive this relationship or the chosen threshold. Can the authors provide stronger theoretical or empirical justification?