- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This paper introduces Reinforced Adversarial Learning (RAL), which aims to elicit reasoning in LLMs during post-training from reference samples by training a critic and the generator in an adversarial fashion. They test their approach on verifiable tasks (i.e. integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation) as well as non-verifiable tasks such as abstract generation for papers and grant applications.
Critical strengths
There are several interesting parts that I like about this paper:
- Their method is well explained and motivated
- I particularly like all the described tricks for stability, such as Polyak averaging and the thresholding update. (Having worked on a similar problem myself, I am aware of how painful these things can be)
- The problem overall is very interesting
Critical weaknesses
Unfortunately, I have to point out several weaknesses in this paper:
- Related Work: My biggest concern lies with the comparison to related work. I might have misunderstood something, but the authors mention:
GAD (Ye et al., 2025) distills a stronger teacher through a co-trained discriminator that distinguishes teacher from student responses. R-AIRL (Fanconi et al., 2025) avoids verifiers but requires expert reasoning traces, constructing process supervision from expert answers and artificially corrupted traces.
But in this case, aren't "teacher responses" for GAD, "expert reasoning traces" for R-AIRL , or the "demonstrations" from RARO the reference samples in this paper? To me, these are just different terms for the reference samples. Am I misunderstanding something?
Additional related work that should be discussed:
- rePIRL: Learn PRM with Inverse RL for LLM Reasoning — Xian Wu, Kaijie Zhu, Ying Zhang, Lun Wang and Wenbo Guo (2026).
- Unified Reinforcement and Imitation Learning for Vision-Language Models — Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang and Yueh-Hua Wu (2025). RIL; NeurIPS 2025.
-
No Confidence Intervals: The tables do not report any kind of uncertainty to determine any significance. While I understand that some researchers may be compute-limited and cannot run multiple iterations, I strongly encourage the authors to at least report, e.g., the 95% confidence interval from a bootstrapped test set. This would help me determine whether, e.g., in Countdown, 0.878 vs 0.858 is a significant improvement.
-
The Figures: I find the figures hard to read, as they are very packed, use small fonts, and look a bit like screenshots from wandb. I would encourage the authors to use matplotlib, with standard scientific plotting settings, to generate their figures, and potentially also separate them in LaTeX for improved readability
-
Baselines: I would encourage testing at least against one adversarial baseline, e.g. GAD, but I do not expect this for the rebuttal as this requires a lot of additional work/compute.
Questions for authors
-
Effect of their proposed methods I appreciate that the authors proposed several methods to stabilise and improve adversarial LLM learning. However, it is not clear to me what the effects of, for example, Polyak averaging, the format filtering, and threshold updating actually are. While I don't see this as an immediate necessary improvement for rebuttal, it would be interesting to include ablations that isolate the effects of these proposed methods.
-
It is interesting to see that SFT performs best on SMILES, and the authors note that potentially less reasoning is required there. To that end, I would be interested to see how RAL performs if the base generator is trained after an initial SFT stage. This would alleviate format-matching rewards.
4 comments
Thank you to the authors for their thorough follow-up and clarification. This has helped me understand it clearly, and they have addressed most of my concerns.
I have therefore increased my score to 3, as I believe this paper is an interesting addition to adversarial learning with LLMs. However, I currently refrain from scoring it a 4, as the proposed method does not always offer a clear added benefit, as seen, e.g., in the SMILES SFT experiment in question 2.
Best of luck to the authors!
For completeness, we wanted to share one additional result. We also ran RLVR on the same SFT-initialized generator for SMILES-to-formula and saw the same behavior as with RAL. Neither method led to further improvement.
Overall, our results suggest that, with a sufficiently large judge and continued training, RAL can match RLVR. This makes RAL a promising option in settings where RLVR would help but a verifier is unavailable. We see RAL, like SFT and RLVR, as one more tool in the post-training toolkit. It can be useful in many settings, though, as the reviewer points out, not necessarily all.
We thank the reviewer once again for the very useful feedback!
I wish to thank the authors for the follow-up experiment and for clarifying this. However, in this case, the reviewers have addressed all my concerns, and their method does improve results over the baseline where possible. I therefore think it is fair that I stick to my word and further improve the score to an acceptance.
We sincerely thank the reviewer for the constructive feedback. Below, we address each point and describe the changes made in the revised manuscript, which we have uploaded.
Lack of clarity describing differences with related work:
Thank you for raising this point. We agree that our original explanation did not explain well enough how RAL differs from related adversarial methods. We also agree that teacher responses, expert reasoning traces, and demonstrations are all reference examples. The key method differences are whether reference example reasoning traces are required, and the scalability and expressivity of the judge. Our experiments show that increasing judge capacity is critical to matching RLVR performance.
RAL requires only reference answers, without reasoning traces. Both R-AIRL and GAD require these traces, which are rarely available. R-AIRL uses expert reasoning traces to construct process rewards. GAD relies on supervised warmup on teacher responses, which its authors identify as critical for stable training. Imitating teacher reasoning during this warmup would also require reference reasoning traces. Our setting assumes reference reasoning traces are unavailable.
RARO shares our answer-only supervision setting but differs in judge design. However, RARO shares weights between the generator and judge, preventing independent scaling of judge capacity. It also restricts the judge to three pairwise verdicts: reference wins, policy wins, or tie. This limits its expressivity and ability to provide nuanced feedback. RAL instead uses a separate judge whose continuous reference probability serves as the reward, allowing more nuanced feedback and independent scaling of judge capacity. Our experiments identify increased judge capacity as critical to matching RLVR performance, whereas RARO explicitly reports underperforming RLVR.
We revised the related work to clarify these distinctions and included rePIRL and Unified RIL for VLM, which we agree to be relevant.
Table confidence Intervals:
We now report bootstrap 95% confidence intervals for every table 1 entry. For Countdown, RAL (32B judge) reaches 0.878 ± 0.023 versus 0.858 ± 0.024 for verifiable reward. To avoid overclaiming, we now describe RAL as matching RLVR rather than surpassing it. Other comparisons remain unchanged.
Figures readability:
Thank you for pointing this out. The plots were generated using matplotlib with data exported from W&B, although their presentation could have been clearer. We enlarged the figures and increased label and legend font sizes, improving readability.
Baselines:
Hopefully our updated related work clarified that RARO is the only related method applicable to our setting: reference answers available, but no verifier or reference reasoning traces. RARO explicitly reports underperforming RLVR on all evaluated tasks, including Countdown, whereas RAL at least matches RLVR on all tasks with a sufficiently large judge. Although this comparison is indirect, we believe the consistent results across tasks strongly suggest an advantage for RAL. RARO’s constraints on judge scaling and feedback expressivity may help explain this difference.
Effect of the judge gate update, replay buffer and polyak averaging mechanisms:
The reviewer’s suggestion prompted us to include additional results on our three main algorithmic mechanism: the judge update gate, replay buffer, and Polyak averaging. Additional targeted experiments provide evidence of their contributions in the updated manuscript.
For the judge update gate, we added a SMILES-to-formula stress test in Appendix D comparing training with and without it. Every ungated run collapsed: judge accuracy reached 100%, rewards vanished, and held-out exact match fell to 0%. With the gate, the generator kept learning. Under our more main hyperparameters, the gate still serves as a safeguard to allow for arbitrary scaling of the judge.
For replay and Polyak averaging, we added three NIH grant abstract generation variants in Appendix G: neither component, replay only, and replay with Polyak averaging. Mean fooling rate at the best evaluation increases from 0.27 to 0.32 and then 0.35. The same trend holds when averaging the top three evaluations or the best three consecutive evaluations. These results suggest complementary benefits.
RAL from an SFT-initialized generator:
Following the reviewer’s suggestion, Appendix D now includes an SFT-initialization experiment on SMILES-to-formula. Test accuracy remains around 94% for 1k steps, showing no additional benefit from RAL. Appendix D also compares continued judge training with freezing the judge midway through training.
We hope these revisions address the reviewer’s main concerns. We sincerely appreciate the feedback, which we believe has help strengthened the work, improved its clarity, and reinforced confidence in the method.