- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces GymHarness, a recursive self-improvement (RSI) framework in which the verifier co-evolves with the generator, thereby minimizing verifier drift. In essence, the policy is a 27B model that is fine-tuned on self-generated data accepted by the verifier. The verifier is recursively trained on the data from the new policy after passing a suite of hidden golden tests. This predictably improves performance over using a static verifier on WebShop, ALFWord, and DBBench, along with two held-out environments.
Critical strengths
None
Critical weaknesses
-
Novelty: This just seems like a standard co-evolving generator-verifier loop with more bells and whistles attached. It’s not surprising that throwing more compute at a problem at inference time would then boost end performance. A lot of the design choices for verifier safeguards (e.g., the append-only probe suite, the choice of non-writable evidence) feel ad hoc, it’s not clear to me whether these are the best possible choices to improve the evolving verifier. At present, I don't feel sufficiently compelled to believe that GymHarness could be a fundamentally new learning paradigm for how one ought to think about RSI.
-
Experimental soundness: I’m curious why the authors only examine a single Qwen 3.6 model size for the policy (27B), especially since Qwen has open-sourced model checkpoints in a wide range of different sizes. Historically, for empirical ML papers, one would examine a range of model sizes to show that their method holds across model scale. Showing results using only a particular model size makes me suspicious about whether this method is truly generalizable. Same argument follows for only using one model family (Qwen) throughout.
-
I am also quite suspicious about the main results in Table 1. Teacher-distilled SFT seems to perform very closely (within ~2%) to GymHarness, with much less moving steps and a considerably less complex training pipeline. I’m curious about whether GymHarness’s (small) improvements are statistically significant. I see that the authors have multiple seeded runs in Table 5 and it would be great if they could report statistical significance as well.
-
Presentation: The writing in this paper is very unclear and difficult to follow (to be clear, I’m not accusing the authors of over-relying on LLMs to draft the entire paper, but the writing in multiple sections does feel disjoint with poor logical flow. I had to read some parts several times in order to get a sense of what’s going on). Clear, coherent communication of scientific methodology and findings is just as important as the science itself, and I feel that this draft’s manuscript does not meet the bar for a top-tier conference such as ICLR.
Questions for authors
- What exactly is the “exogenous evidence”?
- In what real-world scenarios would verifier drift be a serious issue?
- Why did you cap the audit at 10%? Looking at Fig. 3a, there seems to still be some gain in in-domain avg. success even at 20%? Is raising the audit rate very computationally expensive?
- Your Appendices C and F have no content at all. I’m assuming this is not intentional?