acceptodds
Withdrawn from ICLR 2027

GymHarness: Recursive Self-Improvement of Agents with Recursively Verified Verifiers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A self-improving agent learns from data it generates itself, so how far it improves is decided by the verifier that labels that data. When ground truth is scarce, the verifier must be built by the system, and a fixed verifier does not stay accurate: as the policy trains, it leaves the distribution on which the verifier was calibrated, and the verifier accepts more of its errors. We present GymHarness, a recursive self-improvement framework in which a policy and a smaller verifier improve each other through supervised fine-tuning, and in which every verifier update is itself verified. A candidate verifier replaces the current one only if it passes a hidden, append-only probe suite whose labels come from execution evidence that the agent cannot write, so the recursion bottoms out in exogenous evidence rather than in the model’s own judgment. Trained only on its own verified trajectories, Qwen3.6-27B improves by 11.3 points on WebShop, ALFWorld, and DBBench and by 5.2 points on two held-out environments, recovering 89% of the gain of oracle-filtered training with fixed anchors and a 10% adaptive rollout-audit rate. A 9B verifier suffices to supervise the 27B policy, and co-evolving the verifier without the recursive gate performs worse than not evolving it at all.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Agent Environment, Long horizen task verifier, Recursive Self Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_auTtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces GymHarness, a recursive self-improvement (RSI) framework in which the verifier co-evolves with the generator, thereby minimizing verifier drift. In essence, the policy is a 27B model that is fine-tuned on self-generated data accepted by the verifier. The verifier is recursively trained on the data from the new policy after passing a suite of hidden golden tests. This predictably improves performance over using a static verifier on WebShop, ALFWord, and DBBench, along with two held-out environments.

Critical strengths

None

Critical weaknesses
  • Novelty: This just seems like a standard co-evolving generator-verifier loop with more bells and whistles attached. It’s not surprising that throwing more compute at a problem at inference time would then boost end performance. A lot of the design choices for verifier safeguards (e.g., the append-only probe suite, the choice of non-writable evidence) feel ad hoc, it’s not clear to me whether these are the best possible choices to improve the evolving verifier. At present, I don't feel sufficiently compelled to believe that GymHarness could be a fundamentally new learning paradigm for how one ought to think about RSI.

  • Experimental soundness: I’m curious why the authors only examine a single Qwen 3.6 model size for the policy (27B), especially since Qwen has open-sourced model checkpoints in a wide range of different sizes. Historically, for empirical ML papers, one would examine a range of model sizes to show that their method holds across model scale. Showing results using only a particular model size makes me suspicious about whether this method is truly generalizable. Same argument follows for only using one model family (Qwen) throughout.

  • I am also quite suspicious about the main results in Table 1. Teacher-distilled SFT seems to perform very closely (within ~2%) to GymHarness, with much less moving steps and a considerably less complex training pipeline. I’m curious about whether GymHarness’s (small) improvements are statistically significant. I see that the authors have multiple seeded runs in Table 5 and it would be great if they could report statistical significance as well.

  • Presentation: The writing in this paper is very unclear and difficult to follow (to be clear, I’m not accusing the authors of over-relying on LLMs to draft the entire paper, but the writing in multiple sections does feel disjoint with poor logical flow. I had to read some parts several times in order to get a sense of what’s going on). Clear, coherent communication of scientific methodology and findings is just as important as the science itself, and I feel that this draft’s manuscript does not meet the bar for a top-tier conference such as ICLR.

Questions for authors
  • What exactly is the “exogenous evidence”?
  • In what real-world scenarios would verifier drift be a serious issue?
  • Why did you cap the audit at 10%? Looking at Fig. 3a, there seems to still be some gain in in-domain avg. success even at 20%? Is raising the audit rate very computationally expensive?
  • Your Appendices C and F have no content at all. I’m assuming this is not intentional?
Reviewer_Yy2oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GymHarness addresses verifier drift in agent self-training by co-evolving a policy and a smaller generative verifier through SFT on accumulated data. Policy updates undergo paired validation, while verifier updates must pass a hidden, append-only probe suite labeled by execution evidence, protecting both rejection of incorrect trajectories and acceptance of legitimate alternative solutions. The method also incorporates failure attribution, adversarial probes, and golden-derived hints for near-zero-pass tasks, with hints removed before SFT. Using a Qwen3.6-27B policy, a Qwen3.5-9B verifier, and a fixed 397B research agent, GymHarness improves average success on WebShop, ALFWorld, and DBBench from 61.2% to 72.5%, compared with 67.0% for fixed-verifier training and 73.9% for oracle-filtered SFT. Success on held-out ScienceWorld and OS tasks rises from 39.0% to 44.2%. Its contribution is an execution-anchored co-training protocol with soundness and completeness diagnostics, evaluated over four rounds in environments with executable success checks.

Critical strengths
  1. The paper shows why verifier precision and completeness should be assessed together. Ungated co-evolution achieves higher data precision than a fixed verifier (0.74 vs. 0.71), yet produces a weaker policy (65.0% vs. 67.0%) and lower alternative-correct acceptance (0.76 vs. 0.83). GymHarness reaches 0.95 precision and 0.92 alternative-correct acceptance, supporting the importance of preserving legitimate solutions, although the causal contribution is not fully isolated.
  2. The experiments include informative controls for the proposed mechanism. Shared-runtime comparisons and component ablations examine verifier updates, probes, hints, and checkpoint selection, while shadow and fresh attacks test generalization beyond selection probes. A 27B self-researcher still achieves 70.4% in-domain success, versus 66.1% for its fixed-verifier control. The explicit accounting of evidence beyond the 10% rollout-audit rate also helps assess the method’s supervision requirements.
Critical weaknesses
  1. Missing baseline results leave the ratchet’s added value unclear. Accuracy/AUROC-gated co-evolution is described, but no numerical results are reported. The ungated baseline, which accepted all 12 candidates, supports the need for validated updates but does not establish an advantage over simpler selection rules. Downstream policy results across verifier sizes are also missing.
  2. The advantage of a learned verifier over direct execution checks is not established. As disclosed, the loop uses anchors, validation checks, and execution-labeled probes beyond its 10% rollout audits, reporting 274 execution calls per 100 rollouts. The described audit-only SFT control shares the execution budget, but its results are absent, leaving unclear whether learning a verifier is preferable to directly certifying training data under comparable supervision resources.
Questions for authors
  1. What are the results for accuracy/AUROC-gated co-evolution? How much additional benefit does the ratchet provide over these simpler update-selection strategies?
  2. How does audit-only SFT compare with GymHarness under a comparable execution-based supervision budget? Under what conditions is learning a verifier preferable to direct execution checks?
Reviewer_NzLyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to address the staleness problem for the fixed verifier in LLM self-improvement loops. To this end, the authors propose GymHarness, which jointly improves a policy model and a smaller verifier model through iterative SFT. Without access to the ground-truth execution outcomes, the policy learns from self-generated, verifier-accepted trajectories, with golden-hints attached for difficult tasks. The verifier learns from audited rollouts and execution-labeled probes, and the updated model is accepted only if a hidden probe confirms improved verification quality (the “ratchet”). To validate GymHarness, the authors evaluate a Qwen3.6-27B policy with a Qwen3.5-9B verifier on three training environments and two held-out environments with 4 improvement rounds. The reported results show notable gains over the initial policy and other environment-free baselines, yet GymHarness still underperforms the oracle-filtered SFT. The ablations demonstrate the importance of gated verifier updates (”recursive ratchet”) as well as verifier co-evolution designs.

Critical strengths

Well-motivated problem and informative experiments. The paper aims to address an important limitation of agent self-improvement: the verifier capability can deteriorate as the policy updates. To guarantee the verifier's improvement, the authors have designed a series of mechanisms for the verifier update with per-component ablations to validate their effectiveness. Those experiment results reveal some successful designs for verifier improvement in a constraint setting.

Critical weaknesses
  1. The verifier update mechanism appears overly complicated, and the necessity of this complexity is not sufficiently established. The framework combines multiple data sources, probe partitions, and acceptance constraints, making it difficult to understand which elements are essential for the design and why. Although the component ablations are useful, they do not establish whether the specialized ratchet rules outperform simpler accuracy- or AUROC-based verifier selection on the same hidden validation probes.

  2. The baseline comparisons do not fully establish the framework’s practical advantage. GymHarness can benefit from golden-derived hints and a stronger research agent, which does not exist in other baselines. Furthermore, GymHarness should be evaluted on open-ended tasks where the ground truths are absent to validate its practical advantage.

Questions for authors
  1. Why the main comparison only covers 4 rounds and what are the stopping criteria? What are the long-horizon stability of GymHarness on other benchmarks besides ALFWorld?

  2. Could the authors provide an comparison with the main baselines for the full operational overhead (verifier training, research-agent calls, execution checks, etc.)? Does GymHarness still have an advantage under similar operational budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.