- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how increasing test-time sampling can replace correct answers with wrong ones, even when overall accuracy improves. It provides theoretical analysis, estimators for answer turnover, and proposes "ComputeGuard" to reduce corruption.
Critical strengths
- Interesting problem. The paper studies answer stability beyond accuracy, which is a useful direction for evaluating test-time scaling.
- Simple method. ComputeGuard is easy to implement and shows lower corruption than random-width sampling with a similar number of candidates.
Critical weaknesses
I'm in general not satisfied with the quality of the manuscript. Here are my feedbacks:
- Missing experimental records. The authors state that the original evaluation records are missing, so the proposed bootstrap analysis was not performed. This makes the main experimental results and statistical comparisons difficult to verify.
- AI-like writing and limited novelty. The writing feels AI-like to me, with many complex words and different terms used for same concept. The paper makes simple observations unnecessarily complicated with many equations and proofs. For example, the main result in Section 4 can be explained with simple intuition about replacement probability and some curves. While formal proofs may have value, I do not see much additional insight from this complexity. Overall, the contributions seem limited, and I do not think the novelty is strong enough for an ICLR main conference paper.
- Weak connection between theory and method. It is unclear how the theoretical results in Sections 3 and 4 help design ComputeGuard. The method seems to be a separate confidence-based selection rule.
- Missing verifier analysis. The paper does not report how accurate the two verifiers are. It also does not study how verifier quality affects repair, corruption, or ComputeGuard.
- Limited improvement and baselines. ComputeGuard has almost the same accuracy as Random Width, while reducing both corruption and repair. The Adaptive Raw Score baseline achieves higher accuracy, and the paper lacks a full accuracy-corruption trade-off comparison.
- Unclear reliability of the predictions. ComputeGuard estimates correctness and future gains using 20 score bins from development data. The paper does not study how reliable these estimates are with limited data or across problems with different difficulty.
- Unfair compute comparison. The experiments mainly match the number of candidates, but not generated tokens, verifier cost, or inference time. Therefore, the actual compute cost may be different.
- Limited practical motivation. The paper does not clearly explain why reducing answer changes is more important than improving final accuracy in real applications.
Questions for authors
- How can the data splits and calibration assumptions be verified?
- How does ComputeGuard perform with different verifier quality levels and different amounts of development data?
- Can the authors compare the full accuracy-corruption trade-off with stronger baselines, using the same total inference cost?
- What additional insight do the theoretical results provide for the design of ComputeGuard?