acceptodds
Desk rejected at ICLR 2027

Recursive Reward Improvement via Adversarial Meta-Prompt Evolution

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recursive Self-Improvement (RSI) has gained wide attention for its ability to work with limited data. In a proposer-solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning. Such loops have succeeded in mathematics and coding, where every answer can be checked automatically. Open-ended expert tasks, like clinical report generations, have no answer key, which makes simple question-answer generation not viable. In these tasks, the proposer usually must also write a rubric that lists the content an answer should include and the errors it should avoid. However, the unique challenge behind model-written rubric is the comprehensiveness of it. For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the entire learning process. To solve this problem, we introduce RRI, Recursive Reward Improvement, which utilizes a specially designed reward hacking adversary agent as the supervision for the reward. Before the solver trains on a generated task, an adversary writes an answer that games its rubric while being as useless as possible, the same model writes an honest answer without seeing the rubric, and up to two repair steps revise the rubric to make the honest answer win. The exploits found also rewrite how future tasks and rubrics are generated. On HealthBench Professional, RRI improves the adjusted score over the Generation Only self-improvement baseline by 10.9 points at 9B and 12.4 at 27B, and by 10.3 points without any external model. In particular, the gains are largest on difficult and writing tasks. We show that the 27B model trained on the HealthBench Professional description improves by 6.6 points on PRBench Hard and 8.0 on ProfBench without seeing a task from either. We further provide detailed analysis on exploitable points of models' generated question-rubric pairs to help facilitate future research in open-ended self play methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Medical foundation models, reward design, adversarial training, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rRMdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a method to generate comprehensive synthetic rubrics. They use an adversarial agent to provide an answer that would hack the reward, and they refine the rubrics accordingly to make sure the adversarial rollout gets lower score than the vanilla response.

Critical strengths

I think this is a trivial idea, and the authors did the engineering around it to show that it works in medical domain, and the generalization to other domains is an interesting finding. The results support that their loop improves performance over a one-pass generation of rubrics.

Critical weaknesses

This paper reads heavily AI generated. Here are some major concerns that I noticed, but there might be more:

I think framing this paper in the "Recursive Self-Improvement" domain is very misleading. I understand that RSI is a relatively new domain and is not well-defined. However, this paper, in no way, is related to RSI. It doesn't involve a model improving itself (an unspecified GPT model is generating rubrics for a Qwen 27B model) and is not in the domain of AI R&D.

The paper lacks critical details. The model used for generating is consistently referred to as "GPT teacher" but is never specified. I don't think this is acceptable from an ICLR submission. The choice of a GPT model matters significantly here. I think a SOTA model (e.g., Sol, Astra in the newest generations) would already generate robust rubrics that won't improve by using this method. But they will be expensive and not cost-efficient, which would justify using a smaller/weaker model combined with this method. However, this would all require more studies that are not included in the paper. Additionally, the paper contradicts itself in the choice of the judge model. L368 mentions that regrading was done using GPT-5.4 judge, and then in the AI Use Statement, L491 mentions that GPT-5.5 was used as the judge. Also, I didn't see any computational cost analysis to understand whether the additional cost of generating adversarial samples, and looping over it, would justify the observed gains.

All the results are using a single Qwen 3.6-2.7B model, so cross-model family and scaling generalizations are all unclear.

In addition to all the above, generating robust synthetic rubrics have been a major focus in the field and there are plenty of methods already published over the last year and this paper doesn't really discuss related work, compare with them, or explain how this is different. I think the idea of using an adversarial sample to build rubrics is very obvious, and I'd be surprised if hasn't already been done.

Questions for authors

none

6 comments
AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 1

We thank the reviewer for the detailed comments, and for noting that the results support the loop over one-pass rubric generation and that the transfer to other domains is an interesting finding. Several of the concerns are about material that is in the paper. We point to each location below together with new measurements.

I think framing this paper in the "Recursive Self-Improvement" domain is very misleading. I understand that RSI is a relatively new domain and is not well-defined. However, this paper, in no way, is related to RSI. It doesn't involve a model improving itself (an unspecified GPT model is generating rubrics for a Qwen 27B model) and is not in the domain of AI R&D.

RRI does not need a GPT model. The paper reports an all-self setting in which a frozen copy of the solver's own base model, Qwen3.5-9B, fills every role, namely the proposer, the adversary, the honest answer, the repair step, the meta-optimizer, and the training and validation judge. These are reported in Section 3, Section 4.2 "Self-improvement works without a stronger teacher", Figure 2, Tables 2 and 14. No other model and no training data enter that loop. It improves the adjusted score by 10.3 points over Generation Only, and by 12.3 points when the same answers are re-graded by the benchmark's official GPT-5.4 judge (Table 5). The gain is of the same size as with the GPT teacher, 10.9 points.

We see RRI (Recursive Reward Improvement) as a larger idea, with self improving being a related but not neccessary confition. We use recursive self-improvement in the sense of the proposer–solver literature the paper builds on, where a model generates its own training tasks and learns from them. RRI extends that loop from the tasks to the reward. The exploits found during training rewrite the instructions that produce the next tasks and rubrics, so the reward the solver learns from is itself a product of the loop. Such design is agnostic to the model, and any combination of base models can work under this design.

We will make the phrasing more clear in the final manuscript.

The paper lacks critical details. The model used for generating is consistently referred to as "GPT teacher" but is never specified. I don't think this is acceptable from an ICLR submission.

Additionally, the paper contradicts itself in the choice of the judge model. L368 mentions that regrading was done using GPT-5.4 judge, and then in the AI Use Statement, L491 mentions that GPT-5.5 was used as the judge.

The teacher is GPT-5.5-Chat. In the GPT-teacher settings it fills every role other than the solver, as Section 3 defines for the teacher–student setting. The two lines the reviewer quotes refer to different roles and are consistent.

RoleModel
SolverQwen3.5-9B or Qwen3.6-27B
Proposer, adversary, honest answer, repair step, meta-optimizerGPT-5.5-Chat, or the frozen Qwen3.5-9B in the all-self setting
Validation HealthBench Professional judgeGPT-5.4
All other rolesGPT-5.5-Chat

Specifically, the official grader used in the HealthBench Pro used in the original benchmark paper is GPT-5.4. We continue to use this as the grader

L491 lists these roles in one sentence, "GPT-5.5-Chat and the frozen 9B model serve as judges during training and validation, GPT-5.4 is the official judge of HealthBench Professional". The re-grade in L368 exists precisely to show that the gains do not depend on the training judge. We will add this table to Appendix E.

AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 2

The choice of a GPT model matters significantly here. I think a SOTA model (e.g., Sol, Astra in the newest generations) would already generate robust rubrics that won't improve by using this method. But they will be expensive and not cost-efficient, which would justify using a smaller/weaker model combined with this method. However, this would all require more studies that are not included in the paper.

We have measured the adversary win rate and gap against SoTA models at the time, and further evaluated the latest models described in the review, and found the gap is indeed not closing. From the paper, we see the GPT-5.5-Chat adversary won over honest answer on 99.4% of 13,200 probed tasks in the 9B run and 99.5% of 37,600 in the 27B run. The deliberately useless answer received a mean score of 0.96, against 0.77 and 0.80 for the honest answer.

We also ran this test on the model the reviewer names. GPT-5.6 Sol wrote 200 tasks and rubrics through the same generation service, with the same prompts and the same knowledge base. We see the gap is even higher:

Rubric writerAdversary winsExploit ScoreHonest Score
GPT-5.6 Sol100%1.000.44
GPT-5.5-Chat100%1.000.79

The useless answer written with the rubric in view reaches the full score on every rubric that Sol wrote. A stronger rubric writer makes the rubric harder for an honest answer, while not necessarily making it harder to exploit.

The consequence for training is the main comparison of the paper. One-pass rubrics from this same frontier model are exactly the Generation Only baseline, and RRI improves on it by 10.9 points at 9B and 12.4 points at 27B (Figure 2). Both gains hold under the official GPT-5.4 judge, and their 95% bootstrap intervals over the 525 prompts stay above 5 points.

Also, I didn't see any computational cost analysis to understand whether the additional cost of generating adversarial samples, and looping over it, would justify the observed gains.

Each run trains on one node with four B200 GPUs. RRI-9B takes 69 hours of wall-clock time including validation, or 277 GPU-hours, and RRI-27B takes 62 hours, or 248 GPU-hours. With a GPT teacher the adversary runs as background API calls and uses no training GPU. We estimate the teacher's usage from the number and size of its calls.

RunAdversary tokensAdversary costAPI costShare of API costTotal costShare of total cost
RRI-9B0.27B$2.2k$7.5k29%$9.7k22%
RRI-27B0.16B$1.2k$4.5k27%$6.5k19%

API cost here uses the GPT-5.5-Chat list price. Total cost adds the training compute at 8 dollars per B200 GPU-hour, which is 2.2k and 2.0k.

From the table, we see the adversary is about a 27-29% of the total API cost of a full training run. If we count the full training cost, the adversary accounts for around 19-22% of the total cost. We believe these overheads are minimal if we consider the terminal performance of RRI is 10.9 points higher at 9B and 12.4 points higher at 27B, meaning that even if we spend much more on API and compute cost for the baseline methods, it would not reach a similar performance.

All the results are using a single Qwen 3.6-2.7B model, so cross-model family and scaling generalizations are all unclear.

The paper trains two solvers, Qwen3.5-9B and Qwen3.6-27B (note this is not 2.7B), which come from different releases and differ threefold in size, as shown in Section 4.1, Figure 2, and Table 2.

RRI improves over Generation Only by 10.9 points at 9B and 12.4 at 27B. Across the other components it is evaluated with two teachers, with retrieval on and off, and under two judges. The 27B model trained only on the HealthBench Professional description also gains 6.6 points on PRBench Hard and 8.0 on ProfBench, which are finance, legal, chemistry, physics, and consulting tasks, and rises from 0.143 to 0.246 on the 1,000 tasks of HealthBench Hard (Figure 4, Table 10).

AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 3

In addition to all the above, generating robust synthetic rubrics have been a major focus in the field and there are plenty of methods already published over the last year and this paper doesn't really discuss related work, compare with them, or explain how this is different. I think the idea of using an adversarial sample to build rubrics is very obvious, and I'd be surprised if hasn't already been done.

Section 2 discusses rubric rewards and adversarial checks, and Table 1 in Appendix A compares RRI with ten published methods on eight properties. The paper also trains three of the closest rubric-reward methods in the same setting and reports them in Figure 3(a), with implementations in Appendix F. RaR with synthetic tasks reaches 0.433, EvoRubrics 0.493, SibylSense 0.502, and RRI 0.533. SibylSense and EvoRubrics also received reference answers generated by the teacher in these runs, which RRI does not use.

Adversaries are established in code generation, where a test writer or a mutant attacks a test suite and execution decides the outcome, and we cite this line of work. Open-ended tasks have no executable oracle, and the idea of adversary generation has never been explored in the domain of self improving medical agents. In RRI, an exploit written with the rubric in view is compared with an honest answer written by the same model without it, before the solver trains on the task. Among the methods in Table 1, RRI is the only one that needs no reference, tests the reward before training, repairs it, and re-attacks the repair. If the reviewer has a specific prior method in mind that does this, we would be glad to compare with it.

The paper also contributes an analysis that we believe is new, a taxonomy of 12,147 exploit cases across three domains (RQ3, Figure 6). It shows, for example, that missed negatives and ambiguous criteria account for most reward failures while missed positives stay at or below 1%.

This paper reads heavily AI generated.

The research design, experiments, analysis, and claims are the authors' own, and the AI use statement discloses how language models were used. We would like to kindly note that AI assisted writing does not violate any of the submission rules, given it's been properly disclosed.

With that said, we have revised and improved the Method section further, and the revised text is attached at the end of this response. We hope this can help reviewer to better understand our method. If any part of this is unclear, please feel free to follow up.

Reviewer_yxC4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study aims to present a new framework for improving the recursive process that LLMs use for self-improvement through reinforcement learning.

Critical strengths
  • The overall design seems rational and timely.
  • The framework seems fairly flexible in the choice of the base LLMs and only requires training the solver model.
  • While the study pitches its relevance to the medical domain, it seems agnostic about it.
Critical weaknesses
  • Fig 3.a seems to be the most critical piece of evidence demonstrating the generalizability of the proposed framework. However, the current scale still seems rather limited to establish a broad generalization, as claimed in the list of contributions. Even within the current scope, the reported results do not contain variation errors. For instance, could it be that RRI’s results are within the same boundary as the closest baseline (SibylSense)?
  • Related to the above concern, there must be comparisons to other methods used for the HealthBench benchmark. This is a fairly popular benchmark. If the downstream performance is considered, non-relevant methods (like those not related to RL-based self-improvement) applied to the same task should be included in the results.
Questions for authors

What are the practical requirements for training the solver model (re. computational costs)?

2 comments
AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 1

We thank the reviewer for the careful reading and for recognizing that the design is timely, flexible in the choice of base model, and not tied to medicine. We answer each point below.

Even within the current scope, the reported results do not contain variation errors. For instance, could it be that RRI's results are within the same boundary as the closest baseline (SibylSense)?

We computed paired bootstrap intervals over the 525 HealthBench Professional prompts (10,000 resamples, 95% intervals) from the per-prompt grades of the official GPT-5.4 judge behind Table 5. The quantity is the gain of RRI over Generation Only in adjusted-score points.

SettingGain
9B, GPT teacher+11.6 [+6.6, +16.6]
27B, GPT teacher+11.3 [+7.2, +15.5]
9B, all roles served by the frozen 9B+12.3 [+8.0, +16.6]
9B, no retrieval+6.2 [+2.0, +10.3]

Zero lies outside every interval. The gain is therefore stable across two solver sizes, two teachers, and retrieval on and off. The categories the paper highlights are also clear of zero in all four settings. Writing gains 36.3 [23.9, 48.9] points at 9B, 14.0 [6.1, 22.5] at 27B, 24.8 [14.9, 35.0] in the all-self setting, and 20.4 [12.2, 29.5] without retrieval. Difficult tasks gain 19.2 [10.1, 28.3], 18.2 [11.2, 25.3], 19.7 [12.5, 27.1], and 14.7 [7.7, 21.9].

On SibylSense specifically, the three rubric-reward baselines in Figure 3(a) were trained once each under the same solver, judge, tools, and hyperparameters as RRI, and RRI is above all three, by 3.1 points over SibylSense, 4.0 over EvoRubrics, and 10.0 over RaR. In these runs SibylSense and EvoRubrics also received reference answers generated by the teacher, which RRI does not use.

Fig 3.a seems to be the most critical piece of evidence demonstrating the generalizability of the proposed framework. However, the current scale still seems rather limited to establish a broad generalization, as claimed in the list of contributions.

Figure 3(a) is one of several pieces of evidence, and we will make the full set easier to find. The paper evaluates RRI in

  • four training settings, covering two solver sizes, a frontier teacher and a self teacher, and retrieval on and off (Figure 2, Table 14),

  • two judges, with the official GPT-5.4 judge preserving every margin (Table 5),

  • three benchmarks the trained model never saw. The 27B model trained only on the HealthBench Professional description gains 6.6 points on PRBench Hard, which has 550 finance and legal tasks, 8.0 points on ProfBench, and moves HealthBench Hard, with 1,000 tasks, from 0.143 to 0.246, against 0.188 for Generation Only (Figure 4, Table 10),

  • three further target benchmarks on which the full recipe is re-run from that benchmark's description alone, with gains of 4.8 points on PRBench Hard, 4.9 on ProfBench, and 7.0 on MedXpertQA (Table 7).

The gain appears in both PRBench domains and all four ProfBench domains. We believe this supports the generalization claim in the contributions, and we will state the covered axes in one place in Section 4.1.

Related to the above concern, there must be comparisons to other methods used for the HealthBench benchmark. This is a fairly popular benchmark. If the downstream performance is considered, non-relevant methods (like those not related to RL-based self-improvement) applied to the same task should be included in the results.

These comparisons are in Appendix H, and we agree they deserve more visibility. Table 9 runs three retrieval-augmented inference methods that involve no training on the same frozen base models with the same corpus.

MethodQwen3.5-9BQwen3.6-27B
Medical RAG0.1860.268
RAG-Fusion0.1890.301
i-MedRAG0.2540.355
Generation Only0.4240.392
RRI0.5330.516

RRI leads the strongest of them by 27.9 points at 9B and 16.1 points at 27B. Table 8 scores frontier and open models under the benchmark's official protocol with the same judge.

ModelScore
GPT-5.3 chat0.478
GPT-5.5-Chat0.460
GPT-5.40.398
Gemma 4 31B0.297
Lingshu-32B0.139
RRI-27B0.516
RRI-9B0.533

Both RRI models, answering with their tool loop, are above every one of these rows. In the revision we will reference both tables from RQ1 and move a compact version next to Figure 3.

AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 2

What are the practical requirements for training the solver model (re. computational costs)?

Every arm trains on a single node with four B200 GPUs. The policy is served by vLLM for rollouts and updated with FSDP on the same GPUs (Appendix E).

ModelTotalRolloutsPolicyOtherGPU-hours
RRI-9B69 h41 h14 h14 h277
RRI-27B62 h18 h25 h19 h248

Total is the wall-clock time to the reported model. Rollouts covers generating and judging the solver's answers, Policy covers the gradient steps, and Other covers validation and checkpointing. We see the total training time is under 3 days for both runs.

On the cost of training:

RunAdversary tokensAdversary costAPI costShare of API costTotal costShare of total cost
RRI-9B0.27B$2.2k$7.5k29%$9.7k22%
RRI-27B0.16B$1.2k$4.5k27%$6.5k19%

API cost here uses the GPT-5.5-Chat list price. Total cost adds the training compute at $8 per B200 GPU-hour, which is $2.2k and $2.0k.

From the table, we see the adversary is about a 27-29% of the total API cost of a full training run. If we count the full training cost, the adversary accounts for around 19-22% of the total cost. We believe these overheads are minimal if we consider the terminal performance of RRI is 10.9 points higher at 9B and 12.4 points higher at 27B, meaning that even if we spend much more on API and compute cost for the baseline methods, it would not reach a similar performance.

Reviewer_b7qYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents RRI which addresses reward hacking in self-play on open-ended expert tasks. In the paper, the proposer must write both the task and its rubric and the solver learns to game model-written rubrics. RRI has an adversary write a deliberately useless answer that games the rubric, compares it with an honest answer written without the rubric, and repairs the rubric up to two times until the honest answer wins. A meta-optimizer reads training feedback and confirmed exploits and rewrites the generation prompts. Trained only on a text description of HealthBench Professional, Qwen3.5-9B and Qwen3.6-27B improve over a Generation Only baseline by 10.9 points and 12.4 points, and by 10.3 when the frozen 9B fills every role. The 27B model also improves on HealthBench Hard, PRBench Hard and ProfBench. The authors classify around 12k exploits which were mainly unpenalized errors and loosely worded criteria.

Critical strengths
  • Clear problem and simple repair rule: Self-play has worked where answers can be checked automatically. For open-ended tasks the rubric itself is the weak point, and the paper measures this directly. The exploit-honest optim rule needs no reference answer and is easy to reuse.
  • Useful failure analysis: The taxonomy of ~12k exploits shows that model-written rubrics fail mostly through missing penalties and loose wording.
Critical weaknesses
  • The reported test set is the meta-optimizer's objective. The 525 HealthBench Professional tasks used for validation are also the reported result. In the released code, the meta-optimizer receives this score at every step as "the objective," along with how it changed under each past guidance version, and is told to keep what raised it. Since it rewrites the prompts that generate the training tasks, this is black-box optimization of the curriculum against the test set. The baseline gets no such signal, and the prompt printed in the appendix shows only the training score.
Questions for authors

No further questions from me until the main concern (data leak) is addressed.

8 comments
AuthorsOpenReview ↗

Response to Question

Thank you for examining the released code closely. The code path you describe is deprecated and was never used for any reported result. The meta-optimizer variant that receives the HealthBench Professional overall score comes from an early prototype. It was replaced before any experiment in the paper was run. Every reported run matches exactly as described in Appendix, where the meta-optimizer sees only the training score and the confirmed exploits. We apologize for the confusion this caused. We confirm that the 525 validation tasks are evaluated with no information leaked to the training process, including optimizers. No checkpoint, guidance version, or hyperparameter was selected on them.

We hope this resolves the leakage concern. Since it was the only weakness raised, and you list the problem formulation, the repair rule, and the exploit taxonomy as strengths, we would be grateful if you would reconsider the score. We are glad to answer any further questions.

Thanks for the clarification. Outside of the concerns raised by the codebase, my main remaining concern is that the comparisons aren't like-for-like, so the gains are hard to attribute to the method itself:

  • Unequal comparisons: the meta-optimizer is told to match the benchmark ref stats, but Generation Only uses fixed prompts and is matched only on task count. The frontier models in Table 8 run without tools while RRI uses them, and the Table 9 inference baselines have reasoning turned off. The adversary ablation also adds extra generations with no compute-matched control. Please give the source of the ref stats, and either run Generation Only with the same targets or RRI without them.

  • Missing alternative modes: there's no comparison against simpler ways of getting the same behavior, such as the base Qwen model in thinking/CoT mode with the same tools, the same model prompted with the benchmark description and rubric guidance, or a frontier model given the same tool loop. Without these it's unclear how much the training adds over prompting and test-time reasoning alone.

  • Code does not match the manuscript: the released code has only the meta-optimizer variant that receives the validation score, no run configs, and no healthbench_pro.py, which the trainer imports. Please release the commit and configs behind the reported runs.

  • Other points:

  1. Gain is concentrated: at 9B writing goes 0.199 to 0.590 while consult/research move ~0.5 pts (Table 15); at 27B Generation Only barely moves (0.374 to 0.392).
  2. Rubrics: no human check; auditor rates 78-82% of accepted criteria clinically correct (Table 11). A physician sample would help.
  3. Variance: single run, last checkpoint; intervals resample prompts only. Please add seeds and an interval on the 3.1-pt margin over SibylSense; some Table 6 cells have n=2.
  4. Judge/contamination: same GPT-5.5-Chat judge for training and validation; HealthBench contamination of the Qwen solvers and GPT-5.5 teacher not discussed.
AuthorsOpenReview ↗

Response to follow up questions - Part 1

Thank you for the follow-up. We ran the comparisons you asked for and report them below. Every new score uses the same 525 HealthBench Professional prompts and the same GPT-5.5-Chat judge Table 15.

Unequal comparisons: the meta-optimizer is told to match the benchmark ref stats, but Generation Only uses fixed prompts and is matched only on task count. [...] Please give the source of the ref stats, and either run Generation Only with the same targets or RRI without them.

As we mentioned in our prior response, the benchmark validation statistics are not passed to the model during training. If we can clarify further in some way, please let us know.

The frontier models in Table 8 run without tools while RRI uses them

We reran the Table 8 models with the solver's own tool loop. They receive the same system instruction, the same two tools over the same knowledge base and web search service, and the same budget of two searches.

ModelNo toolsWith tools
GPT-5.40.3980.433
GPT-5.5-Chat0.4600.478
RRI-27B0.516
RRI-9B0.533

Tools add 1.8 to 3.5 points to the frontier models, and both RRI models stay ahead. RRI-9B leads GPT-5.5-Chat with tools by 5.5 points, and leads GPT-5.4 with tools by 10.0 points.

the Table 9 inference baselines have reasoning turned off

We reran the Table 9 baselines with the base model's reasoning turned on. Nothing else changes.

Qwen3.5-9BReasoning offReasoning on
Medical RAG0.1860.203
RAG-Fusion0.1890.222
i-MedRAG0.2540.230
RRI-9B0.533
Qwen3.6-27BReasoning offReasoning on
Medical RAG0.2680.290
RAG-Fusion0.3010.301
i-MedRAG0.3550.333
RRI-27B0.516

Reasoning moves the baselines by at most 3.3 points. The best baseline with reasoning is 30.3 points below RRI-9B and 18.3 points below RRI-27B.

The adversary ablation also adds extra generations with no compute-matched control.

The adversary's generations are never training data for the solver. Every method trains the solver on the same number of rollouts per update, and the adversary's calls only decide which rubric those rollouts are graded against.

For main experiments, each run trains on one node with four B200 GPUs. RRI-9B takes 69 hours of wall-clock time including validation, or 277 GPU-hours, and RRI-27B takes 62 hours, or 248 GPU-hours. With a GPT teacher the adversary runs as background API calls and uses no training GPU. We estimate the teacher's usage from the number and size of its calls.

RunAdversary tokensAdversary costAPI costShare of API costTotal costShare of total cost
RRI-9B0.27B$2.2k$7.5k29%$9.7k22%
RRI-27B0.16B$1.2k$4.5k27%$6.5k19%

API cost here uses the GPT-5.5-Chat list price. Total cost adds the training compute at $8 per B200 GPU-hour, which is $2.2k and $2.0k.

From the table, we see the adversary is about a 27-29% of the total API cost of a full training run. If we count the full training cost, the adversary accounts for around 19-22% of the total cost. We believe these overheads are minimal if we consider the terminal performance of RRI is 10.9 points higher at 9B and 12.4 points higher at 27B, meaning that even if we spend much more on API and compute cost for the baseline methods, it would not reach a similar performance.

Missing alternative modes: [...] the base Qwen model in thinking/CoT mode with the same tools, the same model prompted with the benchmark description and rubric guidance, or a frontier model given the same tool loop.

The Untrained rows of Table 15 are the base Qwen models in thinking mode with the same tools. We added the prompted variant. Its system prompt gives the base model everything the pipeline knows about the benchmark, which is the three use cases, how physician rubrics are written, the scoring formula with its length term, and guidance on how to answer. The frontier models with the same tool loop are in the table above.

ModelBasePromptedRRI
Qwen3.5-9B0.3280.3610.533
Qwen3.6-27B0.3620.3460.516

Base and Prompted are the frozen model in thinking mode with the same tools, rerun for this response. The Base column reproduces the Untrained rows of Table 15, which are 0.329 and 0.374. Prompting adds 3.3 points at 9B and nothing at 27B. RRI is 17.2 points above the prompted model at 9B and 17.0 points above it at 27B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.