- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CARS combines self-consistency with trajectory scores through a simple rule: retain the majority answer unless the two leading candidates have similar support and the runner-up has a higher mean score. Using the same sampled responses for all methods, it improves pooled accuracy by about one percentage point across three model families. The analysis shows why trajectory scores can help in close contests but cause harm when applied indiscriminately.
Critical strengths
-
S1. The main insight is clear and useful: auxiliary evidence need not be trusted equally on every problem. The resulting method is easy to understand and adds no new generation passes.
-
S2. The unconditional-reselection baseline makes a convincing case that selective intervention matters. Reporting corrected and newly broken answers helps explain the accuracy changes.
-
S3. All methods use the same responses, and the paper includes multiple model families and threshold sensitivity analyses. These are meaningful controls for a lightweight answer-selection method.
Critical weaknesses
-
W1. The separation between developing the selection rule and validating its transfer is not sufficiently clear. Tail-30% certainty is chosen following a preliminary comparison, while the competition threshold is introduced as an empirical operating point. The shared threshold, positive gains across three families, and sensitivity analyses are useful evidence, but the paper does not clearly identify which data informed these choices and which were reserved for evaluating the complete fixed rule. A held-out evaluation and paired uncertainty estimates would make the transfer claim easier to assess. The concern is the validation protocol, rather than the absolute number of additional correct answers.
-
W2. The incremental value of the specific hard-gating rule over stronger lightweight aggregation alternatives is not yet sufficiently isolated. Prior work has already explored alternative ways of combining answer frequency with trajectory-level confidence or certainty. For example, CISC uses a softmax-weighted combination of confidence scores and answer frequency, while self-certainty Borda voting combines frequency and certainty through rank-based weighting. Separately, RISC uses a candidate-to-leader support ratio as one feature in a learned candidate-ranking framework; for the runner-up candidate, this quantity corresponds to . These precedents do not establish that CARS’s exact Top-2 hard-gating rule already exists, and CARS’s simple competition-conditioned formulation remains distinct. However, the current comparisons against SC, UTRS, and raw-score TSWS show an advantage over the tested alternatives without fully isolating whether the gain specifically requires the proposed hard gate, rather than a stronger lightweight combination of candidate support and trajectory evidence. One representative comparison under a common development protocol would better substantiate the incremental contribution of the aggregation rule.
Questions for authors
-
Q1. If the signal and threshold are fixed using separate development data, does CARS still improve on self-consistency on held-out problems or benchmarks? This would directly test whether the proposed operating rule transfers.
-
Q2. How does CARS's fixed Top-2 gate differ from RISC's use of relative support in learned candidate ranking? Could the authors clarify which contribution lies in the competition-conditioned empirical finding and which lies in the particular decision rule?
-
Q3. Could the authors compare against one representative lightweight alternative, such as CISC-style softmax weighting or Borda rank weighting, using the same responses and trajectory signal? A common development protocol would help isolate the aggregation rule from differences in signal quality or tuning. A same-signal adaptation should be distinguished from a full reproduction of the original method.
-
Q4. How sensitive is CARS to the string-based answer clustering strategy? The appendix states that trajectories are clustered solely by parsed final-answer strings, while semantic or mathematical equivalence is used only for correctness evaluation. As a result, equivalent answers expressed in different forms may remain separate voting clusters. Because both Vanilla SC and CARS’s competition statistic depend directly on these cluster counts, could the authors report how often equivalent answers are split across clusters, or provide a sensitivity analysis using normalized or equivalence-aware clustering?
-
Q5. What typically distinguishes the answers CARS rescues from those it incorrectly overturns? A few contrasting examples could clarify when the trajectory signal adds useful evidence and when it reinforces a plausible but wrong solution.
-
Q6. Does the benefit remain at smaller sampling budgets? Reusing subsets of the existing responses would help show whether the competition rule is useful when fewer than 128 trajectories are available.