- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the code RLVR problems, where short tests make many solutions tied in deep execution, and querying a larger model can be too costly. Therefore, they propose the CodeVerifier, a generative verifier judging the codes and outputing the repair relevant code regions with a verdict. The empirical experiments show the improvements on both competitive programming and repository code on average. Also, the code is available.
Critical strengths
- The experiments are sufficient, as well as the ablations.
- The motivation is sounded, and the figure is also easy to understand and clear.
Critical weaknesses
- One thing i would like to suggest is the writing, which i believe it could be polished, and the math symbols/annotations should also be reframed, since now it is kind of difficult for people to follow.
- I am wondering did you compare the experiments with CodeScaler, AceCoder, and other reward models? I am curious to see the comparisons.
Questions for authors
See the weakness.
7 comments
Thanks for the responses. I have one further question: why you only choose CodeScaler-1.7B, which is the smallest size compared to AceCoder-7B and your model 9B. Therefore, the comparison might not be fair. If I remember, there are also 7B available size for CodeScaler.
Thank you for your discussion.
This paper studies a unified verifier interface for verdicts, repair regions, and rewards when tests are limited or execution is costly. The paper reports structured judgment, the binary CodeJudge comparison, and downstream reward-to-policy results. The initial response used the verified direct-score CodeScaler-1.7B and near-scale AceCodeRM-7B on the common problem/program pool. After the scale concern was raised, we added CodeScaler-8B in this turn.
The current Test BAcc values are 84.87% for CodeVerifier-9B, 72.46% for AceCodeRM-7B, 72.03% for CodeScaler-8B, and 65.77% for CodeScaler-1.7B.
The released CodeScaler family provides 1.7B, 4B, and 8B checkpoints The current Test BAcc values are 65.77% for CodeScaler-1.7B, 72.03% for CodeScaler-8B, 72.46% for AceCodeRM-7B, and 84.87% for CodeVerifier-9B.
Thank you again for the suggestion. We hope the added results support a favorable reassessment. :)
Response to Reviewer EVLG
Thank you for recognizing the motivation, experimental coverage, and ablation analysis. We provide a unified explanation of the notation, utilities, ablations, and reward-model comparison below.
Notation and ablation map
Sections 2--3 and the appendix sections “Evaluation metrics” and “Field utilities and the online objective” define the code states, structured outputs, utilities, token advantages, and evaluation metrics. The notation is dense because one interface spans multiple code states; for quick reference, the appendix groups the terms by output, target, utility, token advantage, and ablation:
In Stage II, the verdict advantage is applied to verdict tokens, the region advantage to the corresponding region-block tokens, and the KL term preserves Stage-I behavior for the rationale. The appendix expands every Table 1 ablation name and its corresponding utility term, so each symbol maps directly to an experimental comparison.
Reward-model comparison and scale convention
the appendix section “Operating point and direct reward-model comparison” already reports a direct reward-model comparison on the same 1,435-state test pool; the key values are:
Under the same BAcc convention, 9B CodeVerifier is 19.10% above CodeScaler and 12.41% above AceCodeRM. As a scale reference, CodeVerifier-4B reaches 84.32% overall binary accuracy on the same 1,435-state pool, while 9B reaches 87.67%; this accuracy reading is kept distinct from the BAcc reported in the table. CodeVerifier also supplies repair regions and reuses the structured interface across complete programs, editing states, and repository views; CodeRM-NT participates as a reward arm in Table 16.
We hope the unified notation and same-pool reward-model comparison make the method boundary and experimental contribution clearer, and we kindly ask that the new evidence and blue-marked revisions be considered in your final evaluation. Thank you for your attention to the paper's presentation and reproducibility.