- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SE-Eval maintains an evolving benchmark to distinguish adjacent language model releases. Candidates come from a failure-driven generator and pass an executable solution check. The controller selects a bounded pool using model disagreement and retains three reference items. Fresh-seed puzzle experiments find positive gains on Qwen and Kimi and a decline on DeepSeek. Logged streams allow selection to be compared with matched random retirement. Further analyses audit item quality and scoring errors and explore transfer to Python synthesis.
Critical strengths
- A relevant evaluation objective with an auditable selection analysis. Static benchmarks can obscure differences among recent releases even when their answers remain valid. SE-Eval makes adjacent-release resolution an explicit objective and implements an interesting evaluation scheme.
Critical weaknesses
The results are inconclusive. You need better-designed experiments, not just more experiments. Your underlying approach is reasonable. However, you must establish that it measures meaningful model differences and that repeated adaptation adds value. See remarks in questions section
Questions for authors
Explain why the iterative loop is needed. Can you compare the full method with one-shot generation and selection, and with random refresh, using matched pool sizes and budgets Do experiments that justify either the existing method, a revised selection objective, or a simpler one-shot method. The present evidence does not decide among them.
Dear Reviewer riLf,
Thank you for your thoughtful feedback. We appreciate that you found the goal of distinguishing adjacent model releases relevant and the item selection analysis auditable. Your questions about reproducibility and the gains from iteration have helped us refine our interpretation of the results, and we will reflect these changes in the revised manuscript.
Thank you again for your review.