- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies post-training data selection for reasoning models from the angle of test-time compute utilization. They propose Compute-Responsive Data Selection (CRDS): fine-tune on the successful blind trajectories of CR problems (616 examples, LoRA SFT) and compare against size-matched controls (SFT-Wrong, SFT-Random, SFT-Already-Solved).
Critical strengths
- Careful construction of the CR signal. The paper is unusually candid about the confounds that make a naive "wrong at low budget, right at high budget" rule misleading (truncation/completion effects, deterministic audit reproducing the same error, grader-format false recoveries)
- Controlled, size-matched comparison with converging evidence. All SFT variants share the base model, 616 examples, LoRA config and optimizer, differing only in which trajectories are used, and the CR > Already-Solved > Random ordering recurs across the held-out split, the stratified mixed test set, four token budgets, per-source decomposition, an OPD objective with a 32B teacher, a second model family, and a 3-seed replication (Table 15).
Critical weaknesses
-
Several appendix sections appear unfinished ("Missing latency values will be filled from the unified timing evaluation" in Table 12; "Exact examples will be inserted after final manual selection" in Table 13; Appendix L text).
-
The central claim ("compute responsiveness" is the useful signal, not merely "train on problems the model currently fails, with correct solutions") is not isolated by the controls. The missing control is a difficulty-based or failure-based selection with correct targets that does not use the CR probe, e.g. SFT on Unresolved problems with gold/teacher solutions, or SFT on the same number of failed problems chosen by a cheaper difficulty proxy (low pass@k), or plain RFT on the full pool. Without this, the gain over SFT-Already-Solved is equally well explained by "hard, in-distribution problems with correct targets help more than easy ones", which is the known STaR/RFT result. The related-work section does not discuss STaR, RFT, expert iteration or rejection-sampling SFT.
-
(a) The "overall" +1.2-point gain on the mixed set depends entirely on the arbitrary 14% CR share of that set, while the already-solved stratum drops by ~14 points; the compute-accuracy "frontier" (Table 3) is reported only on the 106 CR problems where the base model is 0% at every budget by construction, so it does not show that the model uses compute better on a general distribution. (b) Qwen3-8B run with thinking disabled, look forward to see the results under thinking mode.
Questions for authors
Q1. Can you add a control that separates "failed problem + correct target" from "compute-responsive"? For example, SFT on 616 Unresolved problems using gold NuminaMath/DeepScaleR solutions or Qwen3-32B solutions as targets, and/or SFT on 616 failed problems selected by a difficulty proxy without the CR probe
Q2. Where exactly do the SFT-Random targets come from (random problems from the whole pool with a correct self-generated trajectory? gold solutions?
Q3. How does SFT-CR perform on standard benchmarks (e.g. MATH-500, AIME24/25, GSM8K) at the four budgets compared with the base model and SFT-Already-Solved?
Q4. Qwen3-8B run with enable_thinking=False, Do the CR phenomenon and the SFT-CR advantage persist in thinking mode, where test-time compute is actually used by the model?