acceptodds
Under review as a conference paper at ICLR 2027

ASSAY: RECOVERING FILTERED ENVIRONMENTS WITH VERIFIER-GROUNDED ACTOR–CRITIC RL

Abstract

Code-derived reinforcement-learning environments are often selected using a screening model's rollout outcomes. Such selection mixes two questions: whether an environment is reliable, and whether a particular training recipe can use it effectively. We study this distinction in , where an all-pass/all-fail screen removes of tasks that have passed execution-consistency, leakage, and verifier-agreement checks. A unanimous training group gives GRPO zero reward-driven advantage, but a unanimous screening group does not establish that future training groups will be degenerate. We introduce , which re-admits these tasks, uses single-rollout asynchronous actor–critic optimization, and pretrains the critic on outcome-labeled records retained during environment construction. On the retained RL task pool, improves MiMo-V2.5 from GRPO's to on SWE-bench Pro and from to on DeepSWE. Expanding the pool raises these scores to and . The same expansion changes  Val by percentage points under and under the evaluated GRPO configuration. Results on GLM-5.3-Flash show the same direction of improvement. Ablations, prompt-exposure controls, and training-cost measurements characterize the roles of value initialization, sampling, and asynchronous execution. These results support a practical conclusion: rollout screening should not be treated as an optimizer-independent verdict on training-data value.

Primary area:
reinforcement learning
Keywords:
rl

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VuZdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies an interesting problem in coding-agent reinforcement learning. Existing environment construction pipelines may filter tasks according to rollout outcomes from a screening model. In CODEMIDAS, tasks with all-pass or all-fail screening results are removed. The authors argue that this filtering rule may be too strong. Based on this observation, the paper proposes ASSAY. It recovers 2,628 filtered environments, pretrains a critic using outcome records collected during environment construction, and uses single-rollout asynchronous actor–critic optimization.

Critical strengths

The main idea is simple but important. The paper clearly separates environment quality from optimizer-dependent training utility. I think this distinction is useful for future RL data construction. The analytical motivation is also clear. The paper shows that a small screening group can easily be unanimous even when a larger future training group is not degenerate.

Critical weaknesses

1: The effect of recovered RL data and recovered-task critic supervision is not fully separated This is my main concern. When the filtered tasks are recovered, they can affect ASSAY in at least two ways. First, they are added into the RL task pool. Second, their construction-time records can also provide additional data for verifier-grounded value pretraining.

2: The quality of the recovered environments is still uncertain The recovered tasks have passed the earlier quality checks, but these checks are not perfect. One concerning result is the online leakage-guard trigger rate. The retained tasks have a trigger rate of 0.9%, while the screening-all-pass recovered tasks have a much higher rate of 6.8%. This suggests that some recovered tasks may have more exploit or verifier problems. The online guard is useful, but the paper also acknowledges that it cannot detect all verifier weaknesses.

Questions for authors

Please see above.

Reviewer_Fp6xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ASSAY recovers 2,628 CodeMidas tasks removed by a final all-pass/all-fail rollout screen, while retaining the earlier environment-quality checks. This expands the RL pool from 5,545 to 8,173 tasks. It pretrains a critic on 138,941 outcome-labelled records from environment construction, including reference solutions, execution checks, solver attempts and adversarial leakage probes.

The resulting learner uses SAO's single-rollout asynchronous actor-critic training, with skip-observation GAE, an importance-ratio mask and additional critic updates. An online guard addresses attempted solution leakage. Experiments compare retained and expanded pools, test component ablations and prompt-exposure controls, and report performance and resource use for MiMo-V2.5 and GLM-5.3-Flash.

Critical strengths

(1) Recovering tasks helps across both models under the ASSAY recipe. Moving from D_keep to D_all improves all six benchmarks for MiMo-V2.5 and GLM-5.3-Flash, giving 12 positive comparisons. DeepSWE improves by 3.2 and 1.6 points, respectively. On MiMo-V2.5, both recovered subsets help separately, with the larger gain from all-fail tasks. This is useful evidence against treating a failed screening outcome as a permanent judgment about an environment's value.

(2) The controls and cost reporting make the result easier to assess. Prompt-matched GRPO reaches 46.2 against ASSAY's 47.1 on D_keep, making clear that prompt exposure explains part of the main gap. Removing critic pretraining reduces CodeMidas Val from 49.8 to 46.6, and the ablations identify the importance-ratio mask as consequential. The resource measurements also show the trade-off: peak memory rises from 61 to 78 GB and training compute from 18 to 31 GPU-hours per step, while rollout compute falls from 412 to 268 GPU-hours per step and wall-clock time from 196 to 142 hours. Reporting these costs alongside accuracy is helpful.

Critical weaknesses

(1) The comparison does not isolate whether recovered tasks are more useful to ASSAY than to GRPO.

GRPO uses 32 prompts with 32 rollouts each per step. Across 70 steps, that is 2,240 prompt draws, enough to cover at most 40% of D_keep or 27% of D_all. ASSAY uses 71,680 prompt draws, equivalent to nearly nine passes over the expanded pool. These are substantially different exposure regimes.

Under the reported GRPO configuration, expansion changes which small fraction of the pool is sampled and introduces tasks selected for unanimous screening outcomes. A small decline is compatible with insufficient coverage or more degenerate groups; it does not establish that the recovered tasks lack value for GRPO. The reported +2.7 for ASSAY and -0.4 for GRPO on CodeMidas Val are valid descriptions of the tested configurations, but do not by themselves establish the proposed optimizer-dependent explanation.

The missing comparison is GRPO on both D_keep and D_all with substantially greater prompt coverage. The existing 1024 x 32 prompt-matched arm could be extended to D_all, or a G=4 configuration could use 256 prompts per step. A GRPO variant that filters and resamples degenerate groups online, as in DAPO-style dynamic sampling, would also address the practical issue the paper identifies. Table 11 makes these controls particularly important because prompt matching already removes much of the gap on D_keep.

(2) Several remaining uncertainties affect the empirical contribution.

The GRPO D_keep baseline is taken from Ye et al. (2026), while the ASSAY runs are new. The new GRPO D_all result of 54.1 being close to the published retained-pool number is reassuring, but it does not replace a rerun with the same infrastructure and evaluation harness. Differences of 1.7-2.6 points between recipes are small enough for such details to matter.

It is also unclear whether the D_keep critic was pretrained on all 138,941 records, including 44,676 from recovered tasks, or on retained-task records only. In the first case, D_keep already benefits from information associated with the recovered pool. In the second, moving to D_all changes both the actor's task pool and the critic's pretraining data. The cold-critic experiment is reported only on D_all, so the effect of recovery without critic pretraining remains unknown.

No training seeds or uncertainty intervals are reported. On a 200-task validation set at roughly 47% accuracy, a rough per-task standard-error calculation is about 3.5 points. That is not a substitute for a paired analysis, but it shows why isolated row differences in Tables 2 and 4 should be treated cautiously.

These controls carry particular weight because the methodological additions are limited. SAO is used as published, critic pretraining on execution outcomes follows CodeRL and SAO's own recommendation, and Section 3 applies familiar binomial probabilities. The paper can still make a useful empirical contribution, but the current experiments leave too many explanations unresolved.

Questions for authors
  1. Which construction records initialize the critic in each D_keep and D_all run? Can you provide the retained-pool cold-critic result to complete the comparison with and without pretraining?
  2. Can one of the prompt-exposure controls be run on D_all as well as D_keep? This would help separate pool size from the way prompts are sampled.
  3. Can you provide a same-harness MiMo GRPO baseline and uncertainty estimates for the key recovery comparison? Please distinguish the three evaluation attempts per CodeMidas Val task from independent training runs.
  4. The all-pass subset has a higher leakage-guard trigger rate. What do manual checks of a sample of these environments show, and which model is used by the guard? Trigger rate alone does not establish verifier quality, so an audit would make the recovery recommendation more useful.
Reviewer_4gnwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ASSAY, an actor-critic reinforcement learning framework for software engineering agents that recovers verified coding environments discarded by standard rollout-outcome filtering. Methods like GRPO rely on within-group reward variance to compute relative advantages. Automated pipelines curate the datasets by routinely throwing away valid tasks whenever the model produces all-pass or all-fail outcomes. ASSAY removes this dependency by replacing group-relative baselines with an asynchronous actor-critic that samples individual trajectories per prompt and computes step-level credit assignment using a critic warm-started via Verifier-Grounded Value Pretraining (VGVP) on historical trajectories generated during environment construction. Combined with skip-observation generalized advantage estimation, off-policy ratio masking, and an online leakage guard to block reward hacking on recovered easy tasks, the pretrained critic provides informative advantage estimates even when terminal rewards are constant. ASSAY outperforms GRPO on the standard filtered dataset and improves further when trained on the expanded pool of recovered environments, whereas adding those same unfiltered tasks to GRPO degrades its performance.

Critical strengths
  • Valid and well-motivated practical premise

  • Reusing artifacts and records is a very helpful direction to push the community towards

  • The use of a state-value function is also very important.

Critical weaknesses
  • Overstated theory section that is actually quite trivial

  • Very difficult to read as a paper. E.g. Figure 1b is messy and Figure 3 is difficult to read.

  • Limited novelty: every technical mechanism in ASSAY is taken directly from existing papers

  • No confidence intervals or error bars throughout the paper

Questions for authors

The reported scores on SWE-bench and DeepSWE appear to be single-seed point estimates. Given the high variance inherent to reinforcement learning on code generation, what is the statistical significance of the reported improvements?

Please refer to the weaknesses.

1 comment

Needless to say I will keep my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.