acceptodds
Under review as a conference paper at ICLR 2027

SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

Abstract

Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback. Existing methods rely on answer voting and therefore do not extend naturally to open-ended generation, where valid responses cannot be mapped to a shared canonical answer. Without external reward models or stronger judges, adaptation must instead construct reliable rewards from the model's own outputs. We introduce SERPO (Self-Evolving Rubric Policy Optimization), which replaces answer voting with a closed loop that co-evolves response evidence, query-specific rubrics, and policy parameters. Good-Normal-Bad (G-N-B) response evolution organizes maximally separated rollouts into ordered archives; rubric evolution retains criteria that distinguish archived responses; probabilistic criterion scoring converts verdict-token likelihoods into reward signals; and policy evolution optimizes the actor with the resulting signals. New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop. Across two model configurations, two in-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to 20.63 and 20.31 points over the corresponding base models, raises the six-benchmark macro-average by up to 8.06 points, and supports OOD transfer and continued cross-benchmark evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Reinforcement Learning, Self-Evolving Rubrics, Reinforcement Learning from AI Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2NexOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

During TTRL, given a prompt, the policy generates a group of rollouts, and uses a task-specific rubric derived from a response archive to grade the responses for GRPO. The paper proposes a TTRL method to co-evolve the response evidence, query-specific rubrics, and policy parameters. The resulting model achieves 20.63 and 20.31 improvements on healthbench and ResearchQA, and shows generalization.

Critical strengths

This paper proposes a pipeline to co-evolve evaluation criterion and the actor’s parameters, achieving strong performance compared to baselines.

The paper identified an issue about majority voting in TTRL, where it tends to ignore common mistakes in responses, and provides an empirical pipeline that leverages the response-rubric-policy loop to improve model performance in label-free test time RL.

Critical weaknesses

The method section is not clearly written. I am not sure what the authors mean by “omit x when unambiguous” (L201). What exactly is a verdict? (L208). What is a candidate (L250) and what is a identifier (L251)?

The criteria still depends on the quality of the LLM, how can the method make sure that the LLM are not exploiting shortcuts when proposing criteria, and whether the self-evolving loop would lead to some sort of exploitation or collapse?

It is unclear what is novel in this work. The criteria update is the same as DR TULU, and learning rubrics from contrasting responses is the same idea from EvoLM.

The distinction between criteria and rubric is unclear. The authors do not test whether an intermediate criteria layer is needed or directly having the model propose rubrics based on the archive.

The baselines and experimental setup used in this paper is unintuitive. The baselines are just the most standard setup for TTRL, but it is unclear whether the proposed self-evolving pipeline outperforms alternative sota self-evolving methods. Also, all experiments are with thinking turned off, which differs from the standard inference procedure for the tasks included. This essentially tests the answer quality rather than the reasoning quality, and might not transfer to thinking-enabled GRPO training.

Questions for authors

I am not sure whether I follow the authors’ original motivation for this work. In the intro: “Two medical responses may recommend the same action yet differ in contraindications, uncertainty, and escalation guidance; equally safe answers may also use different wording or levels of detail. Answer voting can therefore reward shared omissions and reject valid alternatives.” If the responses differ, there should exist a strict ordering among them, so it seems like the intuitive solution is to rank both the explanation and the final decision, which some work already does and is not used as a baseline by the paper. How does this result in rewarding shared omissions and rejecting valid alternatives?

Suggestion on organizing the intro: SERPO is referred to before it’s defined, and the response archive, the “post-reasoning Pass/Fail probabilities”, and some other keywords are also not defined. Is the idea of using a response archive novel?

The paper does not clearly define the terms and components in its pipeline until section 4 of the paper. What is an archive, criterion, active rubrics?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2Nex (Part 1 / 3)

Thank you for examining the method closely. We tested the whole-answer ranking and whole-rubric rewrite you proposed. We also trained SERPO with thinking enabled to address your question about reasoning models.

Q1. Would ranking the explanation and decision suffice? What does SERPO add to earlier rubric methods?

A1. For the ranking baseline, we trained Qwen3-4B-Instruct-2507 in non-thinking mode. A frozen copy of the initial model ranked eight sampled answers to each prompt, taking both the explanation and the final decision into account. The judge saw each group in two opposite orders and could assign ties. For an answer with mean rank rr across the two orders, the GRPO reward was (8−r)/7(8-r)/7, with rank 1 assigned to the best answer. Both methods trained on the same 500 HealthBench prompts for 30 epochs with eight direct actor samples per visit.

R3-Table1: Whole-answer ranking and SERPO after HealthBench adaptation

4B methodHealthBenchMedQALLMEval-Med
Whole-answer ranking44.2153.5262.13
SERPO49.8354.7362.50

On HealthBench, SERPO scored 5.62 points above whole-answer ranking under the same direct sampling budget. Two answers can recommend the same action, yet only one tells the reader when to seek further help. A vote on the action gives both equal credit. If all answers leave out the precaution, it never enters the claim-consensus vote (Appendix C.1). Conversely, two answers can express the same advice and precautions in different words and deserve the same score. An explicit rubric criterion lets either wording receive credit for the precaution.

We agree that DR Tulu also maintains a per-prompt rubric buffer and filters rules by the standard deviation of their scores on current rollouts. EvoLM learns to generate rubrics by contrasting answers from different checkpoints. In SERPO, the archive holds answers to the same prompt across visits. Eq. 4 selects one Good, one Normal, and one Bad answer based on how far apart their scores are. A frozen generator uses that history to propose criteria. Scores on the Good and Bad answers then calibrate the reward for the actor update (§§4.2–4.4). Freezing the archive lowers HealthBench from 49.83 to 43.98 (Table 2). Under the same direct sampling budget, our fixed-set adaptations of EvoLM and EvoRubrics scored 45.95 and 44.82. We have clarified the overlap with DR Tulu in §2. Table 6 in Appendix C.3 gives the ranking and rubric-learning comparisons. DR Tulu, §3.2, EvoLM

AuthorsOpenReview ↗

Response to Reviewer 2Nex (Part 2 / 3)

Q2. Why keep individual criteria? Do they reward superficial features, and how are the terms defined?

A2. We tested your alternative on Qwen3-4B-Instruct-2507 in non-thinking mode. At each refresh, the generator rewrote the whole rubric from the archive. Both runs used the same archive update rule, frozen generator and judge, refresh schedule, and direct sampling budget. SERPO kept surviving criteria and merged duplicate proposals.

R3-Table2: Rewriting the whole rubric versus updating individual criteria

4B methodHealthBenchMedQALLMEval-Med
Rewrite the whole rubric46.4254.0561.86
Maintain individual criteria (SERPO)49.8354.7362.50

Maintaining criteria across visits scored 3.41 points higher on HealthBench than rewriting the whole rubric (Appendix C.3, Table 6). To test for a length shortcut, we edited answers to 100 HealthBench prompts in two ways. In the length test, both answers gave the same advice, and one was at least 1.5 times as long. In the content test, we removed one required detail and kept the difference in length within 10% of the complete answer’s length. The same frozen judge scored every version with each rubric.

R3-Table3: Preferences of the initial and final rubrics on edited HealthBench answers

Answer tested (100 pairs per row)Initial rubric, wins / ties / lossesFinal rubric, wins / ties / losses
Longer answer, same content52 / 15 / 3350 / 12 / 38
Complete answer, similar length55 / 9 / 3653 / 5 / 42

The final rubric favored the complete answer in 53 of 100 content pairs and the longer answer in 50 of 100 length pairs. The two preferences were similar in this sample. We saw little evidence that rubric evolution made the judge more sensitive to the omitted detail. Appendix D.3 describes the edits, and Table 8 gives the results for both rubrics.

Table 2 gives a separate check using external scores. With a fixed rubric, HealthBench reached 43.54 and mean answer length across the three medical benchmarks was 2.29× Base. SERPO reached 49.83 at 1.38× Base length. In the longer HealthBench run, the external score rose another 3.6% from epoch 30 to epoch 45 (Figure 3).

The submitted abstract gives the full name, Self-Evolving Rubric Policy Optimization. Sections 3.2 and 4 define the rubric, archive, and verdict used in the equations. In the revised introduction we also explain these terms where the method first appears. Line numbers in the table refer to the submitted paper.

R3-Table4: Rubric and archive notation in the submitted manuscript

Term and locationMeaning
Prompt index xx (§4, lines 200–201)The index xx identifies the prompt. Within a calculation for a single prompt, the archive and active rubric both belong to that same xx.
Answer archive (§4.2, lines 242–244)Three bounded queues of recent Good, Normal, and Bad answers for that prompt.
Criterion and active rubric (§3.2 and §4, line 200)A criterion is one grading rule. The active rubric is the set of rules and weights used on this visit.
Post-reasoning verdict (§4.1, line 208 and Eq. 1)After giving its evidence, the judge emits a true/false decision for one rule. Eq. 1 normalizes those two token probabilities to give a continuous score.
Candidate and identifier (§4.3, lines 247–252)A candidate is a proposed rule. Duplicate requirements share “one canonical statement and stable identifier” so the rule keeps its strike count.
AuthorsOpenReview ↗

Response to Reviewer 2Nex (Part 3 / 3)

Q3. Do the gains carry over to thinking-enabled GRPO?

A3. The submitted 4B checkpoint, Qwen3-4B-Instruct-2507, supports only non-thinking generation. For this experiment we used the hybrid Qwen3-4B checkpoint for all three methods. The actor used thinking during rollout collection and evaluation. During SERPO adaptation, the rubric generator and judge used frozen copies of the hybrid checkpoint with thinking disabled. The judge assigned rewards from the final answer. Claim consensus and SERPO trained on HealthBench and transferred to MedQA and LLMEval-Med without further updates.

R3-Table5: Final-answer scores for Qwen3-4B with thinking enabled

Qwen3-4B with thinking enabledHealthBenchMedQALLMEval-Med
Base51.2358.2465.59
Claim consensus55.8458.6565.17
SERPO63.5459.4167.74

With thinking enabled, SERPO gained 7.70 points on HealthBench over claim consensus, 0.76 on MedQA, and 2.57 on LLMEval-Med. We evaluated final answers using the benchmark procedures in Appendix D.1, including answer extraction for MedQA. We have added the thinking setup and all three benchmark results to Appendix D.5 (Table 11).

Thank you for raising these questions. We hope you will reconsider your assessment in light of the new experiments and the definitions above. If any point remains unclear, we would be glad to discuss it further.

Reviewer_jjhsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SERPO, a self-evolving rubric-based framework for open-ended test-time reinforcement learning (TTRL) without reference answers. SERPO includes: (1) response evolution, which maintains G-N-B response archives; (2) rubric evolution, which dynamically generates and selects discriminative evaluation criteria; and (3) policy evolution, which uses probabilistic rubric-based rewards to optimize the actor through GRPO. The framework iteratively updates the response archives, rubrics, and policy while keeping the rubric generator and judge frozen. The authors evaluate SERPO on HealthBench and ResearchQA, and 4 OOD benchmarks with Qwen3-4B/Qwen3.5-9B. The results show substantial in-domain improvements over existing label-free baselines, positive OOD transfer, and continued improvements during extended and sequential evolution. Ablation studies further investigate the contributions of the main components.

Critical strengths

The paper addresses an important limitation of existing TTRL methods, which mainly rely on answer voting and are difficult to apply to open-ended scenarios. SERPO introduces an interesting framework that jointly evolves response archives, query-specific rubrics, and the policy, enabling self-improvement without reference answers or external reward models.

SERPO achieves substantial improvements over the base models and consistently outperforms the evaluated label-free baselines on the in-domain benchmarks. The paper also includes ablations of key components, OOD evaluations, and longer-horizon experiments, providing useful evidence for the effectiveness of the proposed framework.

Critical weaknesses

SERPO selects and weights rubric criteria based on their score variance across generated responses. However, high discriminative power does not necessarily imply that a criterion accurately reflects response quality. Important criteria may also receive low weights or be removed when their scores show little variation. The paper lacks a systematic analysis of how well the evolved rubrics align with actual response quality, particularly for safety-critical tasks.

The in-domain evaluation uses the same prompts that are seen during test-time adaptation, although fresh responses are generated for evaluation. Therefore, the substantial in-domain improvements may partly reflect adaptation to specific prompts. While the paper reports positive OOD transfer, the gains are generally much smaller. Evaluation on unseen prompts from the same distribution would help clarify the generalization ability of the learned policy.

Questions for authors

Q1. How well do the evolved rubrics align with the official evaluation rubrics? Could the variance-based criterion selection remove important but low-variance criteria, especially safety-related ones?

Q2. The in-domain evaluation uses the same prompts as test-time adaptation. Does SERPO also improve performance on unseen prompts from the same distribution?

4 comments
AuthorsOpenReview ↗

Response to Reviewer jjhs (Part 1 / 2)

Thank you for your thoughtful review. Your questions about low-variance safety rules and unseen prompts led us to check which safety requirements the final rubric still covers and how well the policy answers new questions within each domain.

Q1. Do the evolved rubrics agree with official scores, and can they lose safety criteria?

A1. A safety requirement can lose coverage as the rubric evolves. Using records from the non-thinking Qwen3-4B-Instruct-2507 run, we followed one official safety requirement on each of 100 HealthBench prompts. At visit 3, all the rules covering that requirement were in the bottom quarter by utility for that prompt.

At visit 30, 81 of the 100 requirements were covered, 12 had lost coverage, and 7 could not be classified. A reworded rule counted as long as it still expressed the same requirement. This tracks what the rubric checks. It leaves open whether the actor continues to follow a requirement after its rule drops out.

For the broader quality question, we applied both rubrics to the same archived answers from 500 HealthBench prompts. We compared their scores and coverage with the official evaluator and requirements after adaptation (Appendix D.4, Table 9).

R2-Table1: Agreement with official HealthBench rankings and coverage of clinical and safety requirements

4B HealthBench auditInitialAfter 30 epochs
Answer-pair agreement with the official ranking64.2%78.6%
Mean within-prompt Spearman correlation0.480.68
Coverage of official clinical and safety requirements51.4%74.5%

For pairwise agreement, we set aside official ties. If the official scores put one answer ahead and the rubric tied the pair, it received half credit. The coverage row counts each eligible official requirement once per prompt across all 500 questions. The 100-prompt follow-up tracks requirements that were already covered at visit 3.

For a pool of at least four rules, the bottom quarter by utility receives a strike. A rule’s count resets if it leaves that group. Three consecutive strikes trigger removal, and the pool has a separate cap of 15 rules (§4.3, Table 5). As the archive changes, existing rules can receive new weights and calibration even between proposals for new rules.

We also trained the same 4B model with two removal thresholds, one strike or three consecutive strikes. All other training settings were the same. For each fixed answer to the same prompt, we measured how much its raw reward changed in absolute value between consecutive visits (Eq. 6, on a 0–1 scale).

R2-Table2: Fixed-answer reward changes and HealthBench scores by rule removal threshold

Removal ruleMean absolute reward changeFinal HealthBench
Remove after one strike0.37245.18
Remove after three consecutive strikes0.20649.83

Requiring three consecutive strikes reduced the mean absolute reward change by 0.166. On the same training schedule, the final HealthBench score rose by 4.65 points (Appendix D.4, Table 10).

AuthorsOpenReview ↗

Response to Reviewer jjhs (Part 2 / 2)

Q2. How much of the gain carries over to unseen prompts in the same domain?

A2. In Table 1, the in-domain score comes from fresh answers to prompts seen during adaptation (§5.1). To test the distinction you raised, we selected 500 HealthBench prompts and 500 ResearchQA prompts and randomly split each set before training. We ran the split with Qwen3-4B-Instruct-2507 and Qwen3.5-9B, both in non-thinking mode. In each domain, 250 prompts were available for adaptation and the archive. We evaluated Base directly. For claim consensus and SERPO, we froze the policy after 30 epochs and generated one answer per held-out prompt.

R2-Table3: Scores on held-out HealthBench and ResearchQA prompts

ModelMethodHealthBench, 250 held outResearchQA, 250 held out
4BBase32.4557.48
4BClaim consensus39.4265.84
4BSERPO45.3873.12
9BBase44.8468.84
9BClaim consensus55.9075.62
9BSERPO60.4579.80

SERPO scored lower on held-out prompts in all four settings. For example, 4B SERPO scored 49.62 on seen HealthBench prompts and 45.38 on held-out prompts. On the held-out sets it led claim consensus by 5.96 points on HealthBench and 7.28 on ResearchQA. At 9B the margins were 4.55 and 4.18 points. Table 12 in Appendix D.6 gives the scores on both halves of each split. We have also clarified the scope of the generalization claim in §6.

Thank you for suggesting the held-out split and for asking which safety requirements survive as the rubric changes. We hope these analyses answer both questions and plan to keep them in the final paper.

Dear Reviewer jjhs,

Thank you again for your thoughtful review. With two days remaining in the discussion period, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Best regards,

The Authors

Dear Reviewer jjhs,

I hope this message finds you well. With the discussion period ending soon, we would appreciate it if you could revisit our response and let us know whether any concerns remain. We are ready to provide any further clarification before the deadline.

Thank you for your time and effort in reviewing our paper.

Best regards,

The Authors

Reviewer_tFBuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SERPO, a self-evolving rubric-based reinforcement learning framework for open-ended TTRL. SERPO maintains a Good–Normal–Bad response archive and a rubric pool that evolves based on the differences among generated responses. A frozen LLM judge evaluates responses against these rubrics and provides probabilistic rewards for GRPO. In the experiments, SERPO achieves better performance than voting-based TTRL baselines on HealthBench and ResearchQA

Critical strengths

This paper presents a practical RL framework for open-ended tasks, achieving better performance than TTRL.

Critical weaknesses
  1. The biggest weakness is the gap between SERPO and vanilla TTRL. SERPO introduces a rubric pool, which is not present in vanilla TTRL. Therefore, I think SERPO should be compared with existing rubric-based methods, such as EvoLM and EvoRubrics, rather than only with TTRL. The performance improvement over vanilla TTRL is somewhat expected, given the additional rubric-based evaluation mechanism.
  2. The paper uses the variance of rubric scores across archived responses to measure the utility of each rubric. However, high variance only indicates that a rubric can distinguish different responses, rather than that it reflects actual response quality. It is unclear why optimizing such criteria would consistently lead to better policy performance. Moreover, does this imply that the method is only stable when using multiple rollouts?
Questions for authors
  1. In Figure 3, RGSD starts with a substantially lower performance than the base model (around 27.1 vs. 32.3), and even after 30 epochs of rubric evolution, its performance only returns to approximately the base level. Could the authors explain why rubric-guided generation initially causes such a significant performance drop?
5 comments
AuthorsOpenReview ↗

Response to Reviewer tFBu (Part 1 / 3)

Thank you for highlighting the practical value of this setting and for suggesting EvoLM and EvoRubrics as baselines. We adapted both to the same unlabeled HealthBench prompts so we could compare how the three methods learn from their own answers.

Q1. How does SERPO compare with methods that also learn rubrics?

A1. We chose TTRL as the original baseline because the actor has access to a fixed set of unlabeled test prompts and its own answers during adaptation. For the new comparison, all three methods started from Qwen3-4B-Instruct-2507 in non-thinking mode and used the same 500 HealthBench prompts, 30 epochs, and eight direct actor samples per prompt per visit.

In our EvoLM adaptation, earlier and later answers to a question provided the contrast for training the rubric generator. Our EvoRubrics adaptation updated separate answer and rubric adapters and generated extra revisions to train the latter. Both used a frozen copy of the initial 4B model as judge. We matched the direct sampling quota across methods. The additional revisions and scoring calls differed. Official rubrics were used only at evaluation.

R1-Table1: HealthBench adaptation and transfer with EvoLM, EvoRubrics, and SERPO

4B methodHealthBench (adaptation)MedQA (transfer)LLMEval-Med (transfer)
EvoLM, fixed-set adaptation45.9552.5860.82
EvoRubrics, self-supervised adaptation44.8253.7359.51
SERPO49.8354.7362.50

On HealthBench, SERPO scored 3.88 points above the EvoLM adaptation and 5.01 above the EvoRubrics adaptation. Tables 1 and 2 show what happens when we hold parts of SERPO fixed. The HealthBench score falls to 43.98 with a fixed answer archive and 43.54 with a fixed rubric. It reaches 32.06 when the archive and rubric evolve but the actor stays frozen. Appendix C.3 describes the feedback available to each baseline, with scores on all three benchmarks in Table 6. EvoLM, EvoRubrics

AuthorsOpenReview ↗

Response to Reviewer tFBu (Part 2 / 3)

Q2. When does variance reflect answer quality, and how many rollouts are needed?

A2. You are right that a rule can distinguish answers and still rank them incorrectly. Eq. 5 measures how much scores on that rule vary across recent archived answers. Its polarity sets the direction of the score. When the Good and Bad archives are sufficiently separated, Eq. 6 uses their mean scores to calibrate the reward. Since the model also scored the archive, we checked its ranking against an external judgment.

After the non-thinking Qwen3-4B-Instruct-2507 run on 500 HealthBench prompts, we used the initial and final rubrics to score the same archived answers, up to four per prompt. We compared both rankings with the official scores after adaptation. The revised Appendix D.4 gives the audit protocol and results (Table 9).

R1-Table2: Agreement between rubric rankings and official HealthBench scores

Agreement with the official evaluatorInitial rubricAfter 30 epochs
Within-prompt answer-pair agreement64.2%78.6%
Mean within-prompt Spearman correlation0.480.68

An archive update requires three rollouts with distinct scores (Eq. 4). If there is no such triple, we skip the archive update for that prompt. We used eight rollouts per prompt per visit and kept up to three visits of history (Table 5). The smallest group with comparable stability remains an open question.

AuthorsOpenReview ↗

Response to Reviewer tFBu (Part 3 / 3)

Q3. Why does rubric-guided generation start below Base in Figure 3?

A3. The RGSD (TTS) curve in Figure 3 uses a frozen Qwen3-4B-Instruct-2507 actor in non-thinking mode. At the first point, the actor sees an initial rubric written from the question alone, before any candidate answers are available (Appendix B.1). The original RGSD distills a rubric-conditioned teacher’s token probabilities into a student. In our control, each new rubric changes the guidance in the prompt. Original RGSD method

R1-Table3: Scores of the frozen 4B actor with initial and evolved rubrics

Frozen 4B actorHealthBenchResearchQA
Base prompt32.3057.29
With the initial rubric27.1358.20
With the rubric after 30 epochs of archive and rubric updates32.0665.51

On HealthBench, the initial rubric costs 5.17 points. Once the generator has archived answers to draw on, the frozen actor recovers 4.93 points and ends 0.24 below Base, just as you observed. On ResearchQA the initial rubric helps slightly, and evolving it adds 7.31 points. Identifying which individual instructions hurt on HealthBench would take an item-level analysis.

Thank you for asking about the first point in Figure 3. Appendix C.2 now explains what the frozen actor sees in each condition. We hope this clarification, the baseline comparisons, and the audit against official scores address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive comments. Reviewers recognized the limitation of majority voting in open-ended TTRL, the practical value of the framework, and the consistent gains and component analyses. Their main questions concerned rubric-based baselines, reward quality and safety, unseen prompts, and training with thinking enabled.

The reward comparisons and audits use Qwen3-4B-Instruct-2507 in non-thinking mode. Thinking-enabled training uses hybrid Qwen3-4B model.

1. Simpler rewards and rubric-learning baselines (2Nex, tFBu). SERPO retains Good–Normal–Bad answers across visits to propose criteria and calibrate rewards (§§4.2–4.4). All five methods use the same 500 HealthBench prompts, 30 epochs and eight direct samples per prompt visit.

G-Table1 — HealthBench adaptation and transfer (Appendix C.3, Table 6)

MethodHealthBenchMedQALLMEval-Med
Whole-answer ranking44.2153.5262.13
Whole-rubric rewrite46.4254.0561.86
EvoLM adaptation45.9552.5860.82
EvoRubrics adaptation44.8253.7359.51
SERPO49.8354.7362.50

SERPO leads on all three benchmarks. MedQA and LLMEval-Med test transfer without further updates. A fixed answer archive scores 43.98 (Table 2). Section 2 discusses the overlap with DR Tulu.

2. Rubric quality and safety coverage (tFBu, jjhs). Criterion variance measures discrimination (Eq. 5). We checked initial and final rubrics against official scores on the same archived answers from 500 HealthBench prompts.

G-Table2 — Official HealthBench alignment (Appendix D.4, Table 9)

MeasureInitialEpoch 30
Answer-pair agreement64.2%78.6%
Mean within-prompt Spearman correlation0.480.68
Clinical and safety requirements covered51.4%74.5%

The audit uses official rubrics after adaptation. Of 100 initially covered, low-variance safety requirements, 81 remained covered at visit 30, 12 lost coverage and 7 were unclassified (D.4).

3. Rule removal and reward stability (jjhs). Only the removal threshold changed between runs. Reward change is the mean absolute change in Eq. 6 for a fixed answer across consecutive visits.

G-Table3 — Removal threshold (Appendix D.4, Table 10)

Removal ruleReward changeFinal HealthBench
One strike0.37245.18
Three consecutive strikes0.20649.83

The three-strike rule reduces reward change by 0.166 and raises HealthBench by 4.65 points.

4. Answer length and missing content (2Nex). Appendix D.3 tests two edits on 100 HealthBench answers. One preserves content while increasing length by at least 1.5×; the other removes a required detail while keeping length within 10%.

G-Table4 — Edited-answer preferences, 100 pairs per row (Appendix D.3, Table 8)

Answer testedInitial wins/ties/lossesFinal wins/ties/losses
Longer, same content52/15/3350/12/38
Complete, similar length55/9/3653/5/42

With a fixed rubric, HealthBench reaches 43.54 and mean length across the three medical benchmarks is 2.29× Base. SERPO reaches 49.83 at 1.38× (Table 2).

5. Unseen prompts (jjhs). Qwen3-4B-Instruct-2507 and Qwen3.5-9B use the same splits with thinking off. Each domain has 250 adaptation prompts and 250 held-out prompts. Training and archives use only the adaptation half.

G-Table5 — Held-out scores (Appendix D.6, Table 12)

ModelMethodHealthBenchResearchQA
4BBase32.4557.48
4BClaim consensus39.4265.84
4BSERPO45.3873.12
9BBase44.8468.84
9BClaim consensus55.9075.62
9BSERPO60.4579.80

SERPO exceeds claim consensus by 5.96 / 7.28 points at 4B and 4.55 / 4.18 at 9B on HealthBench / ResearchQA.

6. Thinking-enabled training (2Nex). All rows use the hybrid Qwen3-4B with actor thinking enabled. Claim consensus and SERPO adapt on HealthBench; the other two benchmarks test transfer.

G-Table6 — Thinking-enabled results (Appendix D.5, Table 11)

MethodHealthBenchMedQALLMEval-Med
Base51.2358.2465.59
Claim consensus55.8458.6565.17
SERPO63.5459.4167.74

SERPO gains 7.70 points on HealthBench over claim consensus. During SERPO adaptation, the generator and judge are frozen initial-model copies with thinking off. Final-answer evaluation follows D.1.

Figure 3's RGSD (TTS) actor stays frozen and starts with a rubric written from the question alone (C.2). The introduction defines the archive, criterion and verdict at first use. The revised section 5.1 points to the added experiments.

We look forward to the reviewers' favourable consideration and remain grateful for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.