acceptodds
Under review as a conference paper at ICLR 2027

Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training

Abstract

In many settings where LLM post-training is desirable, high-quality answers are recognizable but difficult to score with explicit rewards. Reinforcement learning with verifiable rewards (RLVR) is limited to tasks where an explicit correctness signal is available. Prior adversarial approaches to training reasoning LLMs either complement verifiable rewards with co-trained critics or require reference reasoning traces. We ask whether a co-trained LLM judge can turn reference answers alone into the sole reward for on-policy post-training of reasoning LLMs. We call this method Reinforced Adversarial Learning (RAL). We first validate RAL on three verifiable tasks (integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation), using the verifiers only for evaluation. Judge-only RAL, with sufficiently capable judges and enough training, matches or exceeds direct RLVR. In the first, we ask the model to generate paper abstracts without specifying the target domain in the prompt, but the judge can infer that domain and provide feedback that guides the model toward it. In the second, we generate abstracts for U.S. National Institutes of Health grant proposals conditioned on the grant title. Human scientists and an ensemble of LLM judges find our generations harder to distinguish from real abstracts than those of frontier-LLM baselines. These results suggest a path toward post-training reasoning models in scientific and other high-value domains, where increasingly capable judges infer hard-to-specify reward functions from example answers. %We release our code and datasets.

Primary area:
reinforcement learning
Keywords:
llm, reinforcement learning, reference samples, verifier-free

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jSiGOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper introduces Reinforced Adversarial Learning (RAL), which aims to elicit reasoning in LLMs during post-training from reference samples by training a critic and the generator in an adversarial fashion. They test their approach on verifiable tasks (i.e. integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation) as well as non-verifiable tasks such as abstract generation for papers and grant applications.

Critical strengths

There are several interesting parts that I like about this paper:

  1. Their method is well explained and motivated
  2. I particularly like all the described tricks for stability, such as Polyak averaging and the thresholding update. (Having worked on a similar problem myself, I am aware of how painful these things can be)
  3. The problem overall is very interesting
Critical weaknesses

Unfortunately, I have to point out several weaknesses in this paper:

  1. Related Work: My biggest concern lies with the comparison to related work. I might have misunderstood something, but the authors mention:

GAD (Ye et al., 2025) distills a stronger teacher through a co-trained discriminator that distinguishes teacher from student responses. R-AIRL (Fanconi et al., 2025) avoids verifiers but requires expert reasoning traces, constructing process supervision from expert answers and artificially corrupted traces.

But in this case, aren't "teacher responses" for GAD, "expert reasoning traces" for R-AIRL , or the "demonstrations" from RARO the reference samples in this paper? To me, these are just different terms for the reference samples. Am I misunderstanding something?

Additional related work that should be discussed:

  • rePIRL: Learn PRM with Inverse RL for LLM Reasoning — Xian Wu, Kaijie Zhu, Ying Zhang, Lun Wang and Wenbo Guo (2026).
  • Unified Reinforcement and Imitation Learning for Vision-Language Models — Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang and Yueh-Hua Wu (2025). RIL; NeurIPS 2025.
  1. No Confidence Intervals: The tables do not report any kind of uncertainty to determine any significance. While I understand that some researchers may be compute-limited and cannot run multiple iterations, I strongly encourage the authors to at least report, e.g., the 95% confidence interval from a bootstrapped test set. This would help me determine whether, e.g., in Countdown, 0.878 vs 0.858 is a significant improvement.

  2. The Figures: I find the figures hard to read, as they are very packed, use small fonts, and look a bit like screenshots from wandb. I would encourage the authors to use matplotlib, with standard scientific plotting settings, to generate their figures, and potentially also separate them in LaTeX for improved readability

  3. Baselines: I would encourage testing at least against one adversarial baseline, e.g. GAD, but I do not expect this for the rebuttal as this requires a lot of additional work/compute.

Questions for authors
  1. Effect of their proposed methods I appreciate that the authors proposed several methods to stabilise and improve adversarial LLM learning. However, it is not clear to me what the effects of, for example, Polyak averaging, the format filtering, and threshold updating actually are. While I don't see this as an immediate necessary improvement for rebuttal, it would be interesting to include ablations that isolate the effects of these proposed methods.

  2. It is interesting to see that SFT performs best on SMILES, and the authors note that potentially less reasoning is required there. To that end, I would be interested to see how RAL performs if the base generator is trained after an initial SFT stage. This would alleviate format-matching rewards.

4 comments

We sincerely thank the reviewer for the constructive feedback. Below, we address each point and describe the changes made in the revised manuscript, which we have uploaded.

Lack of clarity describing differences with related work:

Thank you for raising this point. We agree that our original explanation did not explain well enough how RAL differs from related adversarial methods. We also agree that teacher responses, expert reasoning traces, and demonstrations are all reference examples. The key method differences are whether reference example reasoning traces are required, and the scalability and expressivity of the judge. Our experiments show that increasing judge capacity is critical to matching RLVR performance.

RAL requires only reference answers, without reasoning traces. Both R-AIRL and GAD require these traces, which are rarely available. R-AIRL uses expert reasoning traces to construct process rewards. GAD relies on supervised warmup on teacher responses, which its authors identify as critical for stable training. Imitating teacher reasoning during this warmup would also require reference reasoning traces. Our setting assumes reference reasoning traces are unavailable.

RARO shares our answer-only supervision setting but differs in judge design. However, RARO shares weights between the generator and judge, preventing independent scaling of judge capacity. It also restricts the judge to three pairwise verdicts: reference wins, policy wins, or tie. This limits its expressivity and ability to provide nuanced feedback. RAL instead uses a separate judge whose continuous reference probability serves as the reward, allowing more nuanced feedback and independent scaling of judge capacity. Our experiments identify increased judge capacity as critical to matching RLVR performance, whereas RARO explicitly reports underperforming RLVR.

We revised the related work to clarify these distinctions and included rePIRL and Unified RIL for VLM, which we agree to be relevant.

Table confidence Intervals:

We now report bootstrap 95% confidence intervals for every table 1 entry. For Countdown, RAL (32B judge) reaches 0.878 ± 0.023 versus 0.858 ± 0.024 for verifiable reward. To avoid overclaiming, we now describe RAL as matching RLVR rather than surpassing it. Other comparisons remain unchanged.

Figures readability:

Thank you for pointing this out. The plots were generated using matplotlib with data exported from W&B, although their presentation could have been clearer. We enlarged the figures and increased label and legend font sizes, improving readability.

Baselines:

Hopefully our updated related work clarified that RARO is the only related method applicable to our setting: reference answers available, but no verifier or reference reasoning traces. RARO explicitly reports underperforming RLVR on all evaluated tasks, including Countdown, whereas RAL at least matches RLVR on all tasks with a sufficiently large judge. Although this comparison is indirect, we believe the consistent results across tasks strongly suggest an advantage for RAL. RARO’s constraints on judge scaling and feedback expressivity may help explain this difference.

Effect of the judge gate update, replay buffer and polyak averaging mechanisms:

The reviewer’s suggestion prompted us to include additional results on our three main algorithmic mechanism: the judge update gate, replay buffer, and Polyak averaging. Additional targeted experiments provide evidence of their contributions in the updated manuscript.

For the judge update gate, we added a SMILES-to-formula stress test in Appendix D comparing training with and without it. Every ungated run collapsed: judge accuracy reached 100%, rewards vanished, and held-out exact match fell to 0%. With the gate, the generator kept learning. Under our more main hyperparameters, the gate still serves as a safeguard to allow for arbitrary scaling of the judge.

For replay and Polyak averaging, we added three NIH grant abstract generation variants in Appendix G: neither component, replay only, and replay with Polyak averaging. Mean fooling rate at the best evaluation increases from 0.27 to 0.32 and then 0.35. The same trend holds when averaging the top three evaluations or the best three consecutive evaluations. These results suggest complementary benefits.

RAL from an SFT-initialized generator:

Following the reviewer’s suggestion, Appendix D now includes an SFT-initialization experiment on SMILES-to-formula. Test accuracy remains around 94% for 1k steps, showing no additional benefit from RAL. Appendix D also compares continued judge training with freezing the judge midway through training.

We hope these revisions address the reviewer’s main concerns. We sincerely appreciate the feedback, which we believe has help strengthened the work, improved its clarity, and reinforced confidence in the method.

Thank you to the authors for their thorough follow-up and clarification. This has helped me understand it clearly, and they have addressed most of my concerns.

I have therefore increased my score to 3, as I believe this paper is an interesting addition to adversarial learning with LLMs. However, I currently refrain from scoring it a 4, as the proposed method does not always offer a clear added benefit, as seen, e.g., in the SMILES SFT experiment in question 2.

Best of luck to the authors!

For completeness, we wanted to share one additional result. We also ran RLVR on the same SFT-initialized generator for SMILES-to-formula and saw the same behavior as with RAL. Neither method led to further improvement.

Overall, our results suggest that, with a sufficiently large judge and continued training, RAL can match RLVR. This makes RAL a promising option in settings where RLVR would help but a verifier is unavailable. We see RAL, like SFT and RLVR, as one more tool in the post-training toolkit. It can be useful in many settings, though, as the reviewer points out, not necessarily all.

We thank the reviewer once again for the very useful feedback!

I wish to thank the authors for the follow-up experiment and for clarifying this. However, in this case, the reviewers have addressed all my concerns, and their method does improve results over the baseline where possible. I therefore think it is fair that I stick to my word and further improve the score to an acceptance.

Reviewer_EqkGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reinforced Adversarial Learning (RAL), a verifier-free reinforcement learning framework in which a policy and an LLM-based discriminator are jointly trained. The discriminator learns to distinguish reference answers from policy-generated answers, and its predicted reference probability serves as the policy's reward. Unlike approaches requiring explicit verifiers, preference labels, or reference reasoning traces, RAL uses reference final answers as its primary source of task supervision.

The authors evaluate RAL on three verifiable tasks - integer multiplication, Countdown-Code, and SMILES-to-formula prediction; where correctness can be measured independently. They additionally study biomedical paper abstract generation and NIH grant abstract generation, measuring domain fidelity, diversity, and the ability to fool human and LLM evaluators. The results show that sufficiently capable judges can support performance comparable to verifiable-reward training on the selected tasks, while the scientific-generation experiments demonstrate improvements in matching reference distributions.

Critical strengths

S1: The paper proposes constructing reward signals from reference final answers and policy-generated outputs, without requiring explicit preference annotations, manually specified rubrics, or reference reasoning traces. A notable strength is the verifier-held-out experimental design: on three tasks, correctness verifiers are withheld during training but used for independent evaluation. The resulting policy performance and correlations between judge scores and task errors (Figures 1–3) provide evidence that the learned reward captures task-relevant information. The independent scaling of judge capacity further supports this finding, with larger judges generally yielding better performance.

Critical weaknesses

W1: The benefit of co-training is not clearly established. In Section 4.2, RAL uses a co-trained Qwen3 judge with continuous pointwise rewards, while the frozen Claude Haiku baseline uses binary pairwise rewards. Since both the judge and reward formulation differ, the comparison does not isolate the benefit of co-training. An otherwise identical frozen-judge baseline (e.g., a same-size Qwen3-14B judge trained on reference and initial-policy outputs, then frozen) would provide stronger evidence. No such control is reported for the verifiable tasks either. Additionally, RARO (Cai et al., 2025) also jointly trains a policy and critic using reference demonstrations without verifiers, and reports results on Countdown and DeepMath. A controlled comparison would help establish the incremental contribution of RAL's separate judge and continuous reward formulation.

W2: Limited evidence of broader applicability and task quality. RAL is evaluated on three specialized verifiable tasks and two scientific abstract-generation settings. Evaluation on established reasoning benchmarks such as MATH or DeepMath, with verifiers withheld during training, would show whether RAL remains competitive with RLVR on standard reasoning tasks. Furthermore, the scientific-writing experiments primarily measure how closely generated abstracts resemble real ones, rather than their scientific validity or quality. Notably, the untrained generator already achieves a 44% human fooling rate compared with RAL's 55%, making it important to establish whether the improvement reflects meaningful gains in scientific quality.

Questions for authors
  • Q1: Is there any evidence that continuously updating the judge improves performance over a frozen judge with the same architecture, initialization, and reward formulation? A controlled ablation would help establish whether co-training itself is responsible for the observed improvements.

  • Q2: Given that RARO also performs adversarial learning from reference demonstrations without verifiers, what evidence establishes the empirical advantage of RAL's separate judge and continuous reward formulation? Are controlled comparisons available?

  • Q3 Are additional results available on established reasoning benchmarks such as MATH or DeepMath, with verifiers withheld during training? Such results would materially strengthen the evidence for RAL's broader applicability.

  • Q4: Does the improved NIH abstract fooling rate correspond to improvements in scientific validity, plausibility, or title relevance? Additionally, are the differences in human fooling rates (44% before training versus 55% after RAL) statistically significant, and how many human evaluations support these results?

2 comments

We thank the reviewer for the feedback. We address each point below and have uploaded our revision.

The benefit of co-training is not clearly established:

We added an ablation on the SMILES-to-formula experiment in Appendix D, where we compare three variants: continued judge updates, freezing the judge after 1k steps, and never training the judge. The architecture, initialization, reward formulation, and other settings are unchanged. At step 4k, held-out exact match reaches 40.1%, 12.4%, and 1.0%, respectively. These results support the importance of continued judge adaptation and provide direct evidence that continued judge updates improve answer content quality.

We also added a frozen, SFT-trained Qwen3-14B judge baseline for the biomedical abstract generation experiment, matching the co-trained judge architecture. This baseline achieves high biomedical alignment, but its reward saturates near 1.0 after a few hundred steps and stops providing effective feedback, while semantic diversity declines from approximately 0.59 to 0.38. With co-training, diversity recovers to approximately 0.58 while the judge is able to continue providing feedback throughout training.

What evidence establishes the empirical advantage of RAL over RARO?

Indeed, the only related method that could be tested in our setting (no verifier, no access to reference reasoning traces, and access only to reference answers) is RARO. In its paper, RARO performs worse than RLVR across all reported tasks, including Countdown. RAL has been shown to at least match RLVR on all tasks with a sufficiently large judge model. While we acknowledge the limitations of this indirect comparison, the reported results suggest that RAL outperforms RARO.

RARO shares weights between the generator and judge, preventing independent scaling of judge capacity. It also restricts the judge to three pairwise verdicts: reference wins, policy wins, or tie, limiting its expressivity and ability to provide nuanced feedback. From a theoretical perspective, these differences may help explain the different results when compared to RLVR.

Evidence of broader applicability:

We agree that evaluation on MATH or DeepMath would strengthen the evidence for broader applicability, but time and compute constraints leave this for future work. Nonetheless, our results across verifiable tasks and scientific generation provide substantial evidence of RAL’s promise across different settings. As discussed in the next point, for NIH abstract generation, the human evaluation assesses content in addition to stylistic resemblance.

Meaning of the improved NIH abstract fooling rates:

On our human evaluation of NIH grant abstracts, evaluators were explicitly told to focus on scientific content, including coherence and false information. Thus, fooling rates are intended as a proxy for plausibility and scientific quality. All evaluators hold PhDs in biomedical fields. Each model generated abstracts for the same 128 test titles, which we shuffled into mixed-model batches of size 64. Each of 10 evaluators assessed one batch, reducing confounding from evaluator differences across models.

RAL increased the human fooling rate from 44% to 55% under this content-aware expert evaluation. Although the sample size makes this comparison suggestive rather than conclusive, it provides a useful indication of improved scientific plausibility under domain-expert scrutiny. We now acknowledge this uncertainty in the manuscript.

The five-LLM ensemble shows a much larger improvement. The mean fooling rate rises from 0% for the base model to 35% (paired test, p < 10^-19). The improvement over the base model is therefore clear under LLM evaluation and directionally consistent under expert evaluation. Also, Figure 5 now also shows NIH abstract excerpts and reasoning traces before and after training. They illustrate more coherent links between hypotheses, aims, and experiments, and we consider them representative of the qualitative changes observed during training.

Finally, Appendix H adds independent corroboration with the AI-text detector Pangram (version 4.1). Pangram flags every abstract from the untrained model as AI-generated and labels every abstract after RAL training as human-written. RAL has a much higher human-label rate than every other baseline too (paired test, p < 10^-20 for each). This strengthens the evidence that RAL produces human-like writing. We encourage the reviewer to examine these results.

These complementary assessments support improved fidelity to real scientific abstracts, with expert judgments also supporting scientific plausibility. We now clearly distinguish this from verifying individual scientific claims or establishing scientific validity.

We hope these revisions address the reviewer’s main concerns. We sincerely appreciate the feedback, which we believe has strengthened the work and improved its clarity.

Reviewer_EqkGOpenReview ↗

Response to Authors

Thanks, the frozen-judge ablation answers Q1 well, and the RARO comparison and content-based human evaluation address my other concerns. I'm raising to 3. The MATH/DeepMath evaluation is still deferred, please keep that noted as a limitation in the final version.

Reviewer_riEdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Reinforced Adversarial Learning (RAL), which trains an LLM-based judge to distinguish reference answers from generator-produced responses. The judge's predicted probability that a response belongs to the reference distribution is then used as the sole reward signal for training the generator via reinforcement learning.

Critical strengths
  1. Reference-only reward learning achieves promising results on verifiable tasks. This paper demonstrates that learning reward signals solely from reference answers can be effective on verifiable tasks. Without access to a verifier during training, RAL with a sufficiently large judge achieves performance comparable to RLVR across multiple tasks, and the judge scores correlate with the hidden error metric.

  2. The paper provides useful case studies of generator behavior. Appendix F presents concrete examples of the generator's outputs. In addition, Figure 4 shows that training against a static judge leads to mode collapse, with generations converging to a single off-domain template, whereas RAL avoids this failure mode. These observations provide useful insights into how the generator's behavior changes during training.

Critical weaknesses

The paper does not adequately validate its central explanation for why RAL works.

The proposed method is very similar to existing approaches, essentially amounting to adversarial imitation learning with an LLM-based discriminator. The paper hypothesizes that, as the generator progressively learns to exploit superficial discriminative cues, the judge is pushed to identify deeper properties of the reference distribution. However, although the generator and judge continuously adapt to each other during adversarial co-training, this does not necessarily mean that the judge can identify new effective discriminative features once the generator has learned to exploit existing ones, nor that it will shift toward deeper or more task-relevant features. Therefore, it remains unclear whether this mutual adaptation actually gives rise to a curriculum that progressively emphasizes deeper, task-relevant properties.

The paper provides no direct evidence that the discriminator progressively shifts toward deeper properties of the reference responses. The current evidence supporting this claim comes exclusively from the generator side. Consequently, the paper shows that an existing methodological framework can work at scale, but it does not adequately explain why.

Questions for authors

How do the features learned by the judge evolve during training? Could the authors provide a case study or an analysis of how the judge's behavior changes over the course of training?

What happens if a well-performing judge is frozen after a certain period of post-training (i.e., when the generator has already begun shifting toward deeper features), while keeping all other experimental settings unchanged? Could the authors conduct this as an ablation study to assess the importance of continued judge updates?

How do generation length and its correlation with the judge's reward evolve throughout training?

3 comments

We thank the reviewer for the valuable feedback. Below, we address each point and describe the changes made in the revised manuscript, which we have uploaded.

The proposed method is very similar to existing approaches:

We agree that our original manuscript did not clearly explain how RAL differs from related adversarial methods, being R-AIRL, GAD and RARO the closest ones. The key differences concern supervision requirements and judge design.

RAL requires only reference answers, without reference reasoning traces. R-AIRL requires expert reasoning traces to construct process rewards. GAD relies on supervised warmup on teacher responses, identified by its authors as critical for stability. Using this warmup to imitate teacher reasoning would require traces unavailable in our setting.

RARO shares our answer-only setting but shares weights between the generator and judge, preventing independent judge scaling. Its three judge pairwise verdicts (reference wins, policy wins, or tie) also limit feedback granularity. RAL uses a separate judge whose continuous reference probability serves as the reward. This allows finer-grained feedback and independent judge scaling, which our experiments identify as critical to matching RLVR performance. RARO, by contrast, explicitly reports underperforming RLVR.

We additionally added targeted experiments examining our stabilization mechanisms. Appendix D includes a SMILES-to-formula stress test where every ungated run collapsed: judge accuracy reached 100%, rewards vanished, and held-out exact match fell to 0%. With the judge update gate, the generator kept learning. Under our main hyperparameters, the gate serves as a stability safeguard.

Appendix G reports NIH experiments where adding replay and then Polyak averaging increases mean fooling rate at the best evaluation from 0.27 to 0.32 and then 0.35, suggesting complementary benefits.

Does co-training produce a task-relevant curriculum?

We agree that mutual adaptation alone does not establish a curriculum of progressively deeper features. However, our results provide two complementary lines of evidence supporting this interpretation.

First, in NIH abstract generation, judge accuracy on current generations repeatedly falls and recovers, while accuracy on replayed generations remains high (Figure 5). This suggests adaptation to changing generator strategies while retaining the ability to distinguish earlier outputs. This sawtooth pattern appears in ALL experiments when judge accuracy is measured on generated samples alone, consistent with ongoing co-adaptation. Note that in our first three plots, averaging accuracy is reported over generated and reference samples mixed, showing a smoother pattern. Second, by the end of training, outputs improve on content quality metrics across all five settings.

Together, these findings suggest that the learned reward captures at least some task-relevant content properties, since co-training improves by measured correctness and output quality. We agree however that we can NOT directly establish the order in which features emerge. We have revised the manuscript to present the progressively deeper curriculum as a hypothesis consistent with these results rather than a directly validated mechanism.

How do the features learned by the judge evolve during training? Could the authors provide a case study or an analysis of how the judge's behavior changes over the course of training?

Figure 5 now includes selected NIH abstract excerpts and reasoning traces before and after training, illustrating more coherent links between hypotheses, aims, and experiments. We consider these examples representative of the qualitative improvements observed during training.

What happens if a well-performing judge is frozen after a certain period of post-training?

We added this ablation in Appendix D. On SMILES-to-formula, we freeze the judge at step 1k and continue generator training, keeping other settings unchanged. At step 4k, held-out exact match reaches 40.1% with continued judge updates, compared with 12.4% with the frozen judge and 1.0% with a judge never trained. Freezing halts further progress in this experiment, supporting the importance of continued judge adaptation.

How do generation length and its correlation with the judge's reward evolve throughout training?

Appendix G now plots the mean generation length alongside the mean fooling rate across five independent LLM evaluators throughout the NIH experiment training. Their correlation is weak and not statistically significant (Pearson r = −0.12), providing no clear evidence that length is exploited by the judges as a telling cue.

We hope these revisions address the reviewer’s main concerns. We sincerely appreciate the feedback, which we believe has help strengthened the work.

The paper still does not directly examine how the judge's decision mechanism changes during training. In particular, the current results do not establish whether the judge progressively shifts from superficial cues toward deeper, task-relevant features. I therefore encourage the authors to adjust the wording accordingly to avoid overstating this interpretation.

Overall, I appreciate the additional evidence and have increased my rating to 3.

We thank the reviewer and agree with the feedback. We will adjust the wording to avoid overstatement.

Comments on the paper

2 comments

We thank the reviewers for their constructive feedback. Guided by their comments, we have revised the manuscript to address several shared concerns. We summarize the main updates below.

Relationship to the closest methods: We expanded the discussion of R-AIRL, GAD, and RARO, clarifying differences in supervision requirements, judge design, and independent judge scaling. We also added discussion of rePIRL and Unified RIL.

Controlled experiments on judge co-training: We added frozen-judge controls on SMILES-to-formula and biomedical abstract generation. On SMILES-to-formula, continued judge updates achieve 40.1% exact match versus 12.4% when the judge is frozen midway through training. In biomedical generation, co-training sustains useful feedback and better preserves diversity.

Ablations of method components: New experiments examine the judge update gate, replay buffer, and Polyak averaging, supporting their contributions to stability and performance. We also evaluated SFT initialization on SMILES-to-formula, finding no additional improvement from RAL in this setting.

Additional evaluation of scientific writing: We added an independent evaluation with Pangram, which labels all evaluated RAL-trained NIH abstracts as human-written and all untrained-model abstracts as AI-generated. We also added qualitative examples and clarified the expert evaluation protocol and its uncertainty. We distinguish evidence of human-like writing and scientific plausibility from verification of scientific validity.

Statistical reporting and clarity: We added bootstrap 95% confidence intervals to tables, improved figure readability, and revised our claims to describe RAL as matching RLVR. We also clarify that a curriculum of progressively deeper features is a hypothesis consistent with results rather than a directly established mechanism.

The reviewers' feedback and the discussion that followed were genuinely helpful. We appreciate the time the reviewers took to engage with the additional experimental results we provided. Below, we briefly summarize the outcome of the discussion.

All three reviewers raised their scores.

Reviewer jSiG raised their score to 4 after the follow-up SMILES experiment and confirmed that we had addressed all their concerns.

Reviewer riEd raised their score to 3. The reviewer pointed out that our experiments leave open which features the judge relies on as training progresses, and we agree. In the NIH abstract experiment, the judge's accuracy on new generations repeatedly falls and recovers while its accuracy on earlier generations remains high. Across all five tasks, the generator's outputs also improve on measures of correctness or content quality. These observations would be consistent with a gradual shift from superficial cues to deeper, task-relevant features, but they do not establish that such a shift occurs like this necessarily. We will make clear that this interpretation remains a hypothesis.

Finally, reviewer EqkG raised their score to 3. The reviewer cautioned against drawing broad conclusions about RAL's applicability from the tasks evaluated, and we also agree. Our results across verifiable tasks and scientific writing provide a promising basis. In particular, RAL's ability to match RLVR on the verifiable tasks we studied suggests it could also help in other settings where RLVR would be useful but a verifier is unavailable. However, findings are limited to the settings studied. While our results provide a reasonable basis for exploring RAL’s broader applicability, that broader applicability remains a hypothesis. Future evaluations on reasoning benchmarks such as MATH and DeepMath could help test it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.