- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies pre-expansion concept allocation in concept-guided test-time scaling. Concept-guided inference first generates a pool of high-level concepts, and the key question is which of them should receive solution-generation compute before any rollout outcome is observed. The proposed method (RATTS) scores each concept with a lightweight relevance model (BGE-small) trained on LLM-judge usefulness labels which measures overlap between concepts with embedding similarity. It then selects a fixed fraction of concepts by greedily maximizing a relevance-weighted soft-coverage objective.
Critical strengths
The paper has clear problem and methodology framing, with comprehensive evaluation. The paper compares selectors on identical concept pools and stored rollouts through offline replay, which isolates the allocation decision.
Critical weaknesses
-
The methodological novelty is limited and incremental. Each part of RATTS is a well-known technique added to the two-stage setup (propose concepts, then expand them into solutions) comes directly from GuidedSampling. The authors themselves say (Appendix A) that the selection algorithm is not the contribution, and that the novelty lies in framing the problem and connecting their objective to candidate recall. That connection adds little. Proposition 3.1 shows that the chance of at least one success can only grow, with diminishing returns, as more concepts are added. That is close to obvious. Theorem 3.3 first assumes a model under which recall takes exactly the same form as their objective, then bounds the gap when the parameters differ. The conclusion is largely built into the assumption. The authors also admit the actual scorer may not satisfy this model, so the guarantee does not clearly apply to the method as used. Overall, the paper applies a standard relevance-plus-diversity selection method to a new setting rather than proposing a new method.
-
The gain is small compared to training-free baselines. The gain over simple baselines is small. At the main setting (Table 5), RATTS reaches 95.65% pass-any. Simple selectors that need no training come close: Random Cluster gets 94.61%, Top Cluster Representatives 94.53%, MMR 94.55%, and uniform soft coverage 94.45%. The strongest PRM baseline, ReasonFlux (95.12%), is not significantly different. The results therefore show mainly that skipping near-duplicate concepts helps, which almost any diversity-aware method achieves. They do not show that RATTS in particular is needed. Final accuracy is about the same as expanding every concept (74.42% vs. 74.08%, CI [−0.90, 1.62]), and drops by 5.5 points on HumanEval+ with GLM-4.7-Flash. The paper reports no final-accuracy comparison against Random Cluster or MMR, so we cannot tell whether the recall gain improves the answers.
-
The cost is not justified. RATTS adds costs that the baselines avoid that are not counted. To get the marginal gain, RATTS needs a separate scorer for each generator where each scorer is trained on 2.9k–7.6k concepts labeled by GPT-5.5 Pro with extended reasoning, and extra steps for calibration and checkpoint selection. None of the labeling or training cost appears in the token accounting. At inference time, RATTS is also slower than Random Cluster.
Questions for authors
-
Can you report final-answer accuracy, not only pass-any, for Random Cluster, MMR and uniform soft coverage at matched budgets? Does RATTS’s ~1-point recall gain over these baselines carry through to final accuracy?
-
What is the total cost of building each generator-specific scorer: the number of GPT-5.5 Pro judge calls, tokens, and the training and calibration compute? At what deployment volume does it pay off compared with a training-free selector such as Random Cluster or MMR?
-
How sensitive is RATTS to the choice of judge? Would labels from a weaker, open-weight judge, or labels derived from rollout outcomes, give similar gains? Can one scorer transfer across generators without retraining?
-
Across the full sweep, and especially at small budgets () where selection matters most, how does RATTS compare with Random Cluster and MMR? Is the gap larger than at ?
4 comments
Response to the Reviewer ZQMF [Continued]
Regarding Scorer Preparation and Cost.
The scorer is trained once and then reused, so labeling and training are a one-time cost per generator. Further, in the table below, we report the recorded concept-generation tokens and training times for each scorer.
| Generator | Generated tokens | Training time |
|---|---|---|
| Qwen3.5-9B | 193,473 | 8 min 18 sec |
| Gemma4-E4B | 3,070,727 | 15 min 55 sec |
| GLM-4.7-Flash | 565,141 | 11 min 58 sec |
As shown in the table, generating the training concepts across all three generators produced approximately 3.83 million tokens. This preparation cost is incurred once and shared across subsequent inference requests. For illustration, the same output-token volume would cost less than 50 dollars at GPT-6.1 Sol's standard rate. Regarding the judge cost, unfortunately, we did not log the judge-token usage and do not have a verified judge-call count. However, labeling all the concepts across the three generators took less than 15 minutes in total.
Next, Table 18 of the revised paper also shows that a scorer can be reused across the evaluated generators without retraining or recalibration, with similar observed macro-average accuracy. Our MMR baseline uses the same learned scorer and therefore shares its preparation cost.
Compared with scorer-free uniform coverage, RATTS adds 48.33 ms of median selection time per question. In the recorded end-to-end measurement, the mean increase is just 0.26 seconds per question, which is approximately 0.08% of the uniform-coverage latency. These increases accompany observed gains of 2.69 percentage points in final-answer accuracy and 1.20 percentage points in pass-any, without increasing the expansion budget. RATTS also achieves 74.42% final-answer accuracy compared with 69.26% for Random Cluster, an observed gain of 5.16 percentage points at the same expansion budget. Thus, the benefit of the additional selection computation extends to the accuracy of the returned answers.
Regarding Judge Choice and Scorer Transfer.
RATTS uses supervised learning, so its performance depends on the quality of the training labels. If a weaker judge provides poorer or misleading labels, these can affect the scorer's performance. Labels derived from actual rollout outcomes, however, would provide direct supervision on expansion success and give the best performance, but generating these labels is computationally costly. This is why we use proxy labels provided by a judge.
Further, we also evaluate whether a scorer can transfer across generators without retraining or recalibration. The table below reports macro averages across the same nine dataset–generator settings. Using a single scorer across all three generators gives observed final-answer accuracy between 74.33% and 74.60%, compared with 74.42% when using a matching scorer for each generator. These results support reusing a scorer across the evaluated generators and sharing its preparation cost. This comparison is reported in Appendix D.6 under "Cross-generator scorer transfer" (Table 18).
| Scorer used for selection | Pass-any (%) | Final-answer accuracy (%) |
|---|---|---|
| Matching scorer for each generator | 95.65 | 74.42 |
| Qwen-trained scorer on all generators | 95.68 | 74.60 |
| Gemma-trained scorer on all generators | 95.47 | 74.45 |
| GLM-trained scorer on all generators | 95.82 | 74.33 |
Response to the Reviewer ZQMF [Continued]
Regarding Comparisons Across Expansion Budgets.
We have added comparisons with MMR and Random Cluster for expansion fractions from ρ = 0.10 to ρ = 0.50. As shown in the table below, RATTS achieves higher observed pass-any and final-answer accuracy than both methods at every budget in this range. These results are also reported in Appendix D.3, Table 12.
| ρ | RATTS pass-any (%) | MMR pass-any (%) | Random Cluster pass-any (%) | RATTS final accuracy (%) | MMR final accuracy (%) | Random Cluster final accuracy (%) |
|---|---|---|---|---|---|---|
| 0.10 | 92.59 | 91.80 | 89.66 | 71.97 | 71.23 | 65.33 |
| 0.15 | 94.04 | 93.23 | 92.09 | 73.19 | 72.86 | 66.67 |
| 0.20 | 94.68 | 93.86 | 92.99 | 73.34 | 72.02 | 67.14 |
| 0.25 | 95.28 | 94.18 | 94.15 | 74.20 | 72.23 | 67.89 |
| 0.30 | 95.65 | 94.55 | 94.61 | 74.42 | 72.49 | 69.26 |
| 0.35 | 95.84 | 94.77 | 95.17 | 74.20 | 72.30 | 68.99 |
| 0.40 | 95.97 | 94.84 | 95.51 | 74.67 | 72.44 | 69.26 |
| 0.45 | 96.13 | 95.21 | 95.65 | 74.62 | 72.56 | 69.57 |
| 0.50 | 96.15 | 95.21 | 95.75 | 74.82 | 72.34 | 69.94 |
The gap over Random Cluster is indeed larger at the smallest evaluated budget. At ρ = 0.10, RATTS improves pass-any by 2.93 percentage points and final-answer accuracy by 6.64 points, compared with 1.04 and 5.16 points at ρ = 0.30. This shows that the benefit over Random Cluster is particularly clear when only a small fraction of the concepts can be expanded.
For MMR, RATTS also has higher observed accuracy throughout the reported range, although the gap is not consistently larger at smaller budgets. At ρ = 0.10, the gains are 0.79 percentage points in pass-any and 0.74 points in final-answer accuracy, compared with 1.10 and 1.93 points at ρ = 0.30.
Follow-up on the Rebuttal
Dear Reviewer ZQMF,
Thank you again for your review. We believe that our response addresses the concerns raised in your review. As the discussion period is approaching its end, we would appreciate it if you could take a look and let us know if any concerns remain or if any further clarification would be helpful.
Best regards,
The Authors
Response to the Reviewer ZQMF
Thank you so much for your time and constructive feedback to improve the paper. Below, we address all your concerns.
Regarding the Novelty.
Our contribution is to formulate concept expansion as a budgeted selection problem and analyze its connection to candidate recall. We agree that Proposition 3.1 has a straightforward proof. However, its role is to establish that diminishing returns are a property of actual candidate recall, even when concept successes are dependent. This provides a structural property for the selection objective to preserve, without establishing that any particular surrogate approximates the recall well. Proposition 3.2 then addresses a different, but important practical question by giving a computable upper bound on how much the optimal allocation could improve coverage at the same budget.
Regarding Theorem 3.3, the success-mode assumption supplies the ideal model under which recall has the product form. The additional result is a lower bound relative to optimal concept selection when the implemented weights and similarities differ from the model probabilities. The bound accounts separately for these errors and shows how the similarity-error contribution grows with the selection budget. Thus, the analysis distinguishes the loss from subset optimization from the error introduced by approximating recall with coverage. Its practical value is to identify what must hold for the coverage guarantee to extend to recall and how imperfect estimates weaken that guarantee.
In the revised version, we also added an empirical check using actual correctness outcomes from stored rollouts. Across 2.28 million equal-budget subsets, the upper coverage quartile achieves 0.233 percentage points higher candidate retention than the lower quartile. This supports a positive relationship between the implemented objective and candidate retention (Appendix D.6, "Coverage and retained candidate recall"). This supports the conditional analysis without establishing that the model assumptions hold or estimating its approximation errors.
Regarding the Gains over Simpler Selectors.
The gains also extend to final-answer accuracy. At the same expansion budget, RATTS achieves observed macro-average gains of 5.16 percentage points over Random Cluster, 1.93 over MMR, and 2.69 over uniform coverage. The table below reports both pass-any and final-answer accuracy at K_max = 50 and ρ = 0.30.
These comparisons were already reported in Appendix D.6. In the revised version, we also compare pretrained PRM weights within the same coverage selector. We observe that RATTS achieves 74.42% final-answer accuracy, compared with 73.45% for SkyWork-weighted coverage and 73.23% for ReasonFlux-weighted coverage. This comparison tests the choice of scorer while keeping the coverage objective and expansion budget fixed. Note that the RATTS scorer has approximately 33M parameters, compared with 1.5B parameters for each evaluated PRM.
Further, compared with full expansion, the main benefit is efficiency. RATTS reduces exploration-plus-expansion tokens by 56.69%, with observed macro-average final-answer accuracy of 74.42% versus 74.08%. As rightly observed, final accuracy is higher in seven settings, equal in one, and lower in one. The exception is HumanEval+ with GLM-4.7-Flash, where accuracy decreases from 77.44% to 71.95%. This programming benchmark is outside the scorer's training domain of mathematical concept judgments and hence acts as an out-of-distribution task.