- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper provides a method to generate comprehensive synthetic rubrics. They use an adversarial agent to provide an answer that would hack the reward, and they refine the rubrics accordingly to make sure the adversarial rollout gets lower score than the vanilla response.
Critical strengths
I think this is a trivial idea, and the authors did the engineering around it to show that it works in medical domain, and the generalization to other domains is an interesting finding. The results support that their loop improves performance over a one-pass generation of rubrics.
Critical weaknesses
This paper reads heavily AI generated. Here are some major concerns that I noticed, but there might be more:
I think framing this paper in the "Recursive Self-Improvement" domain is very misleading. I understand that RSI is a relatively new domain and is not well-defined. However, this paper, in no way, is related to RSI. It doesn't involve a model improving itself (an unspecified GPT model is generating rubrics for a Qwen 27B model) and is not in the domain of AI R&D.
The paper lacks critical details. The model used for generating is consistently referred to as "GPT teacher" but is never specified. I don't think this is acceptable from an ICLR submission. The choice of a GPT model matters significantly here. I think a SOTA model (e.g., Sol, Astra in the newest generations) would already generate robust rubrics that won't improve by using this method. But they will be expensive and not cost-efficient, which would justify using a smaller/weaker model combined with this method. However, this would all require more studies that are not included in the paper. Additionally, the paper contradicts itself in the choice of the judge model. L368 mentions that regrading was done using GPT-5.4 judge, and then in the AI Use Statement, L491 mentions that GPT-5.5 was used as the judge. Also, I didn't see any computational cost analysis to understand whether the additional cost of generating adversarial samples, and looping over it, would justify the observed gains.
All the results are using a single Qwen 3.6-2.7B model, so cross-model family and scaling generalizations are all unclear.
In addition to all the above, generating robust synthetic rubrics have been a major focus in the field and there are plenty of methods already published over the last year and this paper doesn't really discuss related work, compare with them, or explain how this is different. I think the idea of using an adversarial sample to build rubrics is very obvious, and I'd be surprised if hasn't already been done.
Questions for authors
none
6 comments
Thanks for the review and response to questions - Part 2
The choice of a GPT model matters significantly here. I think a SOTA model (e.g., Sol, Astra in the newest generations) would already generate robust rubrics that won't improve by using this method. But they will be expensive and not cost-efficient, which would justify using a smaller/weaker model combined with this method. However, this would all require more studies that are not included in the paper.
We have measured the adversary win rate and gap against SoTA models at the time, and further evaluated the latest models described in the review, and found the gap is indeed not closing. From the paper, we see the GPT-5.5-Chat adversary won over honest answer on 99.4% of 13,200 probed tasks in the 9B run and 99.5% of 37,600 in the 27B run. The deliberately useless answer received a mean score of 0.96, against 0.77 and 0.80 for the honest answer.
We also ran this test on the model the reviewer names. GPT-5.6 Sol wrote 200 tasks and rubrics through the same generation service, with the same prompts and the same knowledge base. We see the gap is even higher:
| Rubric writer | Adversary wins | Exploit Score | Honest Score |
|---|---|---|---|
| GPT-5.6 Sol | 100% | 1.00 | 0.44 |
| GPT-5.5-Chat | 100% | 1.00 | 0.79 |
The useless answer written with the rubric in view reaches the full score on every rubric that Sol wrote. A stronger rubric writer makes the rubric harder for an honest answer, while not necessarily making it harder to exploit.
The consequence for training is the main comparison of the paper. One-pass rubrics from this same frontier model are exactly the Generation Only baseline, and RRI improves on it by 10.9 points at 9B and 12.4 points at 27B (Figure 2). Both gains hold under the official GPT-5.4 judge, and their 95% bootstrap intervals over the 525 prompts stay above 5 points.
Also, I didn't see any computational cost analysis to understand whether the additional cost of generating adversarial samples, and looping over it, would justify the observed gains.
Each run trains on one node with four B200 GPUs. RRI-9B takes 69 hours of wall-clock time including validation, or 277 GPU-hours, and RRI-27B takes 62 hours, or 248 GPU-hours. With a GPT teacher the adversary runs as background API calls and uses no training GPU. We estimate the teacher's usage from the number and size of its calls.
| Run | Adversary tokens | Adversary cost | API cost | Share of API cost | Total cost | Share of total cost |
|---|---|---|---|---|---|---|
| RRI-9B | 0.27B | $2.2k | $7.5k | 29% | $9.7k | 22% |
| RRI-27B | 0.16B | $1.2k | $4.5k | 27% | $6.5k | 19% |
API cost here uses the GPT-5.5-Chat list price. Total cost adds the training compute at 8 dollars per B200 GPU-hour, which is 2.2k and 2.0k.
From the table, we see the adversary is about a 27-29% of the total API cost of a full training run. If we count the full training cost, the adversary accounts for around 19-22% of the total cost. We believe these overheads are minimal if we consider the terminal performance of RRI is 10.9 points higher at 9B and 12.4 points higher at 27B, meaning that even if we spend much more on API and compute cost for the baseline methods, it would not reach a similar performance.
All the results are using a single Qwen 3.6-2.7B model, so cross-model family and scaling generalizations are all unclear.
The paper trains two solvers, Qwen3.5-9B and Qwen3.6-27B (note this is not 2.7B), which come from different releases and differ threefold in size, as shown in Section 4.1, Figure 2, and Table 2.
RRI improves over Generation Only by 10.9 points at 9B and 12.4 at 27B. Across the other components it is evaluated with two teachers, with retrieval on and off, and under two judges. The 27B model trained only on the HealthBench Professional description also gains 6.6 points on PRBench Hard and 8.0 on ProfBench, which are finance, legal, chemistry, physics, and consulting tasks, and rises from 0.143 to 0.246 on the 1,000 tasks of HealthBench Hard (Figure 4, Table 10).
Thanks for the review and response to questions - Part 3
In addition to all the above, generating robust synthetic rubrics have been a major focus in the field and there are plenty of methods already published over the last year and this paper doesn't really discuss related work, compare with them, or explain how this is different. I think the idea of using an adversarial sample to build rubrics is very obvious, and I'd be surprised if hasn't already been done.
Section 2 discusses rubric rewards and adversarial checks, and Table 1 in Appendix A compares RRI with ten published methods on eight properties. The paper also trains three of the closest rubric-reward methods in the same setting and reports them in Figure 3(a), with implementations in Appendix F. RaR with synthetic tasks reaches 0.433, EvoRubrics 0.493, SibylSense 0.502, and RRI 0.533. SibylSense and EvoRubrics also received reference answers generated by the teacher in these runs, which RRI does not use.
Adversaries are established in code generation, where a test writer or a mutant attacks a test suite and execution decides the outcome, and we cite this line of work. Open-ended tasks have no executable oracle, and the idea of adversary generation has never been explored in the domain of self improving medical agents. In RRI, an exploit written with the rubric in view is compared with an honest answer written by the same model without it, before the solver trains on the task. Among the methods in Table 1, RRI is the only one that needs no reference, tests the reward before training, repairs it, and re-attacks the repair. If the reviewer has a specific prior method in mind that does this, we would be glad to compare with it.
The paper also contributes an analysis that we believe is new, a taxonomy of 12,147 exploit cases across three domains (RQ3, Figure 6). It shows, for example, that missed negatives and ambiguous criteria account for most reward failures while missed positives stay at or below 1%.
This paper reads heavily AI generated.
The research design, experiments, analysis, and claims are the authors' own, and the AI use statement discloses how language models were used. We would like to kindly note that AI assisted writing does not violate any of the submission rules, given it's been properly disclosed.
With that said, we have revised and improved the Method section further, and the revised text is attached at the end of this response. We hope this can help reviewer to better understand our method. If any part of this is unclear, please feel free to follow up.
Thanks for the review and response to questions - Part 1
We thank the reviewer for the detailed comments, and for noting that the results support the loop over one-pass rubric generation and that the transfer to other domains is an interesting finding. Several of the concerns are about material that is in the paper. We point to each location below together with new measurements.
RRI does not need a GPT model. The paper reports an all-self setting in which a frozen copy of the solver's own base model, Qwen3.5-9B, fills every role, namely the proposer, the adversary, the honest answer, the repair step, the meta-optimizer, and the training and validation judge. These are reported in Section 3, Section 4.2 "Self-improvement works without a stronger teacher", Figure 2, Tables 2 and 14. No other model and no training data enter that loop. It improves the adjusted score by 10.3 points over Generation Only, and by 12.3 points when the same answers are re-graded by the benchmark's official GPT-5.4 judge (Table 5). The gain is of the same size as with the GPT teacher, 10.9 points.
We see RRI (Recursive Reward Improvement) as a larger idea, with self improving being a related but not neccessary confition. We use recursive self-improvement in the sense of the proposer–solver literature the paper builds on, where a model generates its own training tasks and learns from them. RRI extends that loop from the tasks to the reward. The exploits found during training rewrite the instructions that produce the next tasks and rubrics, so the reward the solver learns from is itself a product of the loop. Such design is agnostic to the model, and any combination of base models can work under this design.
We will make the phrasing more clear in the final manuscript.
The teacher is GPT-5.5-Chat. In the GPT-teacher settings it fills every role other than the solver, as Section 3 defines for the teacher–student setting. The two lines the reviewer quotes refer to different roles and are consistent.
Specifically, the official grader used in the HealthBench Pro used in the original benchmark paper is GPT-5.4. We continue to use this as the grader
L491 lists these roles in one sentence, "GPT-5.5-Chat and the frozen 9B model serve as judges during training and validation, GPT-5.4 is the official judge of HealthBench Professional". The re-grade in L368 exists precisely to show that the gains do not depend on the training judge. We will add this table to Appendix E.