- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
During TTRL, given a prompt, the policy generates a group of rollouts, and uses a task-specific rubric derived from a response archive to grade the responses for GRPO. The paper proposes a TTRL method to co-evolve the response evidence, query-specific rubrics, and policy parameters. The resulting model achieves 20.63 and 20.31 improvements on healthbench and ResearchQA, and shows generalization.
Critical strengths
This paper proposes a pipeline to co-evolve evaluation criterion and the actor’s parameters, achieving strong performance compared to baselines.
The paper identified an issue about majority voting in TTRL, where it tends to ignore common mistakes in responses, and provides an empirical pipeline that leverages the response-rubric-policy loop to improve model performance in label-free test time RL.
Critical weaknesses
The method section is not clearly written. I am not sure what the authors mean by “omit x when unambiguous” (L201). What exactly is a verdict? (L208). What is a candidate (L250) and what is a identifier (L251)?
The criteria still depends on the quality of the LLM, how can the method make sure that the LLM are not exploiting shortcuts when proposing criteria, and whether the self-evolving loop would lead to some sort of exploitation or collapse?
It is unclear what is novel in this work. The criteria update is the same as DR TULU, and learning rubrics from contrasting responses is the same idea from EvoLM.
The distinction between criteria and rubric is unclear. The authors do not test whether an intermediate criteria layer is needed or directly having the model propose rubrics based on the archive.
The baselines and experimental setup used in this paper is unintuitive. The baselines are just the most standard setup for TTRL, but it is unclear whether the proposed self-evolving pipeline outperforms alternative sota self-evolving methods. Also, all experiments are with thinking turned off, which differs from the standard inference procedure for the tasks included. This essentially tests the answer quality rather than the reasoning quality, and might not transfer to thinking-enabled GRPO training.
Questions for authors
I am not sure whether I follow the authors’ original motivation for this work. In the intro: “Two medical responses may recommend the same action yet differ in contraindications, uncertainty, and escalation guidance; equally safe answers may also use different wording or levels of detail. Answer voting can therefore reward shared omissions and reject valid alternatives.” If the responses differ, there should exist a strict ordering among them, so it seems like the intuitive solution is to rank both the explanation and the final decision, which some work already does and is not used as a baseline by the paper. How does this result in rewarding shared omissions and rejecting valid alternatives?
Suggestion on organizing the intro: SERPO is referred to before it’s defined, and the response archive, the “post-reasoning Pass/Fail probabilities”, and some other keywords are also not defined. Is the idea of using a response archive novel?
The paper does not clearly define the terms and components in its pipeline until section 4 of the paper. What is an archive, criterion, active rubrics?
6 comments
Response to Reviewer 2Nex (Part 2 / 3)
Q2. Why keep individual criteria? Do they reward superficial features, and how are the terms defined?
A2. We tested your alternative on Qwen3-4B-Instruct-2507 in non-thinking mode. At each refresh, the generator rewrote the whole rubric from the archive. Both runs used the same archive update rule, frozen generator and judge, refresh schedule, and direct sampling budget. SERPO kept surviving criteria and merged duplicate proposals.
R3-Table2: Rewriting the whole rubric versus updating individual criteria
| 4B method | HealthBench | MedQA | LLMEval-Med |
|---|---|---|---|
| Rewrite the whole rubric | 46.42 | 54.05 | 61.86 |
| Maintain individual criteria (SERPO) | 49.83 | 54.73 | 62.50 |
Maintaining criteria across visits scored 3.41 points higher on HealthBench than rewriting the whole rubric (Appendix C.3, Table 6). To test for a length shortcut, we edited answers to 100 HealthBench prompts in two ways. In the length test, both answers gave the same advice, and one was at least 1.5 times as long. In the content test, we removed one required detail and kept the difference in length within 10% of the complete answer’s length. The same frozen judge scored every version with each rubric.
R3-Table3: Preferences of the initial and final rubrics on edited HealthBench answers
| Answer tested (100 pairs per row) | Initial rubric, wins / ties / losses | Final rubric, wins / ties / losses |
|---|---|---|
| Longer answer, same content | 52 / 15 / 33 | 50 / 12 / 38 |
| Complete answer, similar length | 55 / 9 / 36 | 53 / 5 / 42 |
The final rubric favored the complete answer in 53 of 100 content pairs and the longer answer in 50 of 100 length pairs. The two preferences were similar in this sample. We saw little evidence that rubric evolution made the judge more sensitive to the omitted detail. Appendix D.3 describes the edits, and Table 8 gives the results for both rubrics.
Table 2 gives a separate check using external scores. With a fixed rubric, HealthBench reached 43.54 and mean answer length across the three medical benchmarks was 2.29× Base. SERPO reached 49.83 at 1.38× Base length. In the longer HealthBench run, the external score rose another 3.6% from epoch 30 to epoch 45 (Figure 3).
The submitted abstract gives the full name, Self-Evolving Rubric Policy Optimization. Sections 3.2 and 4 define the rubric, archive, and verdict used in the equations. In the revised introduction we also explain these terms where the method first appears. Line numbers in the table refer to the submitted paper.
R3-Table4: Rubric and archive notation in the submitted manuscript
| Term and location | Meaning |
|---|---|
| Prompt index (§4, lines 200–201) | The index identifies the prompt. Within a calculation for a single prompt, the archive and active rubric both belong to that same . |
| Answer archive (§4.2, lines 242–244) | Three bounded queues of recent Good, Normal, and Bad answers for that prompt. |
| Criterion and active rubric (§3.2 and §4, line 200) | A criterion is one grading rule. The active rubric is the set of rules and weights used on this visit. |
| Post-reasoning verdict (§4.1, line 208 and Eq. 1) | After giving its evidence, the judge emits a true/false decision for one rule. Eq. 1 normalizes those two token probabilities to give a continuous score. |
| Candidate and identifier (§4.3, lines 247–252) | A candidate is a proposed rule. Duplicate requirements share “one canonical statement and stable identifier” so the rule keeps its strike count. |
Response to Reviewer 2Nex (Part 3 / 3)
Q3. Do the gains carry over to thinking-enabled GRPO?
A3. The submitted 4B checkpoint, Qwen3-4B-Instruct-2507, supports only non-thinking generation. For this experiment we used the hybrid Qwen3-4B checkpoint for all three methods. The actor used thinking during rollout collection and evaluation. During SERPO adaptation, the rubric generator and judge used frozen copies of the hybrid checkpoint with thinking disabled. The judge assigned rewards from the final answer. Claim consensus and SERPO trained on HealthBench and transferred to MedQA and LLMEval-Med without further updates.
R3-Table5: Final-answer scores for Qwen3-4B with thinking enabled
| Qwen3-4B with thinking enabled | HealthBench | MedQA | LLMEval-Med |
|---|---|---|---|
| Base | 51.23 | 58.24 | 65.59 |
| Claim consensus | 55.84 | 58.65 | 65.17 |
| SERPO | 63.54 | 59.41 | 67.74 |
With thinking enabled, SERPO gained 7.70 points on HealthBench over claim consensus, 0.76 on MedQA, and 2.57 on LLMEval-Med. We evaluated final answers using the benchmark procedures in Appendix D.1, including answer extraction for MedQA. We have added the thinking setup and all three benchmark results to Appendix D.5 (Table 11).
Thank you for raising these questions. We hope you will reconsider your assessment in light of the new experiments and the definitions above. If any point remains unclear, we would be glad to discuss it further.
Response to Reviewer 2Nex (Part 1 / 3)
Thank you for examining the method closely. We tested the whole-answer ranking and whole-rubric rewrite you proposed. We also trained SERPO with thinking enabled to address your question about reasoning models.
Q1. Would ranking the explanation and decision suffice? What does SERPO add to earlier rubric methods?
A1. For the ranking baseline, we trained Qwen3-4B-Instruct-2507 in non-thinking mode. A frozen copy of the initial model ranked eight sampled answers to each prompt, taking both the explanation and the final decision into account. The judge saw each group in two opposite orders and could assign ties. For an answer with mean rank r across the two orders, the GRPO reward was (8−r)/7, with rank 1 assigned to the best answer. Both methods trained on the same 500 HealthBench prompts for 30 epochs with eight direct actor samples per visit.
On HealthBench, SERPO scored 5.62 points above whole-answer ranking under the same direct sampling budget. Two answers can recommend the same action, yet only one tells the reader when to seek further help. A vote on the action gives both equal credit. If all answers leave out the precaution, it never enters the claim-consensus vote (Appendix C.1). Conversely, two answers can express the same advice and precautions in different words and deserve the same score. An explicit rubric criterion lets either wording receive credit for the precaution.
We agree that DR Tulu also maintains a per-prompt rubric buffer and filters rules by the standard deviation of their scores on current rollouts. EvoLM learns to generate rubrics by contrasting answers from different checkpoints. In SERPO, the archive holds answers to the same prompt across visits. Eq. 4 selects one Good, one Normal, and one Bad answer based on how far apart their scores are. A frozen generator uses that history to propose criteria. Scores on the Good and Bad answers then calibrate the reward for the actor update (§§4.2–4.4). Freezing the archive lowers HealthBench from 49.83 to 43.98 (Table 2). Under the same direct sampling budget, our fixed-set adaptations of EvoLM and EvoRubrics scored 45.95 and 44.82. We have clarified the overlap with DR Tulu in §2. Table 6 in Appendix C.3 gives the ranking and rubric-learning comparisons. DR Tulu, §3.2, EvoLM