acceptodds
Under review as a conference paper at ICLR 2027

HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL

Abstract

Rubric-based reinforcement learning enables open-ended generation using natural-language criteria as rewards, but these signals can become ineffective as the policy evolves. We identify two failure modes: non-discriminative rewards, where scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability, and spurious rewards, where higher scores do not correspond to better responses. Both reflect a fixed training configuration that fails to adapt as the policy evolves, including what to train on, how to explore, and how to evaluate. We introduce HarnessRL, a training-time harness that co-evolves with the policy through three interfaces: task sampling, rollout guidance, and rubric criteria. HarnessRL maintains a training-time memory that retrieves within-task history and cross-task evidence. A harness evolver uses this evidence to attribute each reward failure to the interface whose update can address it, then runs a proposer and just-in-time critic loop in which the critic accepts a candidate update only after measuring its effect on the current frozen policy. We further introduce Rubric-Mix-10k, a unified five-domain training and evaluation suite for open-ended RL spanning medical, science, writing, role-playing, and instruction-following tasks. On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by 2.8 to 3.1 points across both 4B and 9B model scales and nearly halves the training steps to its best checkpoint.

Primary area:
foundation or frontier models, including LLMs
Keywords:
training-time harness, training-time self-evolving, open-ended rl

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_54QZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HarnessRL, a training-time framework that jointly adapts task sampling, privileged rollout guidance, and rubric-based rewards during reinforcement learning. An attributor diagnoses reward failures, a proposer suggests interventions, and a just-in-time critic accepts or rejects them based on measurements against the current frozen policy. Historical and cross-task memory supports diagnosis and proposal generation.

The framework is evaluated on a curated five-domain training mixture, with held-out and out-of-distribution evaluations using Qwen3.5-4B and Qwen3.5-9B. It reports approximately three-point improvements over the strongest baseline, alongside critic and memory ablations, reward–oracle agreement analysis, human audits, and compute comparisons.

Critical strengths
  1. Consistent downstream improvements with useful controls. HarnessRL improves over the strongest reported baseline across both policy scales and all evaluated domains. The appendix shows that the advantage persists at the final checkpoint and under matched compute, reducing concerns that the result depends primarily on evaluation-based checkpoint selection. These findings support the practical value of the implemented system.
  2. Additional evidence beyond the evolving training reward. The evaluation includes objective OOD scoring protocols, an independent oracle judge that does not supply training feedback, and human audits of diagnoses and rubric updates. These checks provide meaningful support that the improvements are not solely an artifact of changing the training rubric.
Critical weaknesses
  1. The distinctive contribution of the JIT Critic is insufficiently isolated, and its rubric acceptance criterion does not directly validate reward quality. For rubric updates targeting spurious rewards, acceptance requires a bounded decrease in mean reward and variation across responses for created or rewritten criteria. These properties do not establish that the revised reward favors better responses; an inappropriate criterion could satisfy both. Furthermore, proposals are constructed and validated using the same recorded response group. Although removing the critic substantially reduces performance, that ablation compares the proposed mechanism with unrestricted proposal acceptance, rather than with simpler tightening or discrimination filters. Given the substantial overlap with prior adaptive-rubric and harness-evolution approaches, the absence of such controls leaves the central methodological contribution unclear.
  2. The evaluation provides limited coverage of the claimed open-ended capabilities and generality. Despite spanning five domains, the evidence comes from one curated training mixture, two policy sizes within one family, and one judge backbone. MedQA and GPQA measure multiple-choice accuracy rather than open-ended medical or scientific response quality; RoleBench is scored using ROUGE-L, which provides limited evidence about persona consistency, engagement, or narrative quality; and WritingBench uses the training judge. The results demonstrate encouraging benchmark transfer, but provide a narrower basis for claims about broadly improving open-ended RL. This limitation is consequential because empirical breadth is particularly important for an otherwise incremental system contribution.
Questions for authors
  1. Mechanism validation: Do accepted rubric updates improve reward–quality ordering on fresh responses under independent evaluation? In particular, what evidence distinguishes improved alignment from simply lowering mean scores while retaining response discrimination?
  2. Ablation clarification: Do you have comparisons with simpler acceptance rules, such as requiring only rubric tightening and criterion variation? Which of the magnitude bounds, variation requirement, and iterative revision account for the critic’s improvement?
  3. Baseline clarification: Which memory and retrieval capabilities of the original AMARIS are retained in AMARIS and AMARIS++? The description of AMARIS++ as retaining only a per-task revision log appears narrower than the original method’s historical and semantic retrieval.
  4. Evaluation clarification: Do you have independent evaluations of open-ended medical response quality or role-playing quality beyond MedQA accuracy and RoleBench ROUGE-L? Such evidence could materially strengthen the connection between the reported gains and the capabilities motivating the paper.
6 comments
AuthorsOpenReview ↗

Author Response to Reviewer 54QZ (Part 1/4)

We thank the reviewer for recognizing the consistent gains under final-checkpoint and matched-compute controls and the evidence beyond the training reward, and address below the validation of the JIT critic's acceptance rule and the breadth of the open-ended evaluation.

W1. Whether the JIT critic's acceptance rule validates reward quality.

The JIT critic accepts an update to a spurious criterion only when it meets two standards, the reward change must agree with the attributor's diagnosis, lowering reward for the responses it cited as surface compliance and so moving toward the oracle, and the size of that change must stay in a range that is safe for the next RL step.

As stated in Section 2.3, the attributor judges whether meeting the criterion reflects quality. Without seeing the oracle, it compares rollouts that meet the criterion with those that fail it and cites responses that only appear to comply. Three annotators confirmed 86.0% of these labels. The critic then regrades the same rollouts under the proposed rubric and accepts the update only if reward falls where the diagnosis says it was unearned.

The second standard keeps the change neither too small nor too large. A drop within grading noise would mean the update does nothing, while a very large drop would change the reward so abruptly that the next RL step trains toward a different objective. Each new or rewritten criterion must also still separate the rollouts, and a rejected update goes back to the proposer, with the old rubric kept if no revision passes within three rounds. In the Qwen3.5-4B run, 57% of first-round rubric proposals failed these checks.

Updates that pass move the reward toward the oracle. At the end of training, HarnessRL's reward agreed with the oracle on 48.9% of response pairs, versus 38.4% for GRPO (Section 3.2). Annotators judged that 82.5% of accepted updates addressed the diagnosed failure, and removing the critic cost 3.9 OOD points (Table 3). Q1 and Q2 below test accepted updates on fresh responses and compare the critic with simpler rules. We will state these two standards explicitly in Section 2.3.

AuthorsOpenReview ↗

Author Response to Reviewer 54QZ (Part 2/4)

Q1 and Q2. Accepted updates on fresh responses, and simpler acceptance rules.

We would clarify that the critic's acceptance rule improves how the reward ranks responses, that is, its agreement with the oracle, not just how low it scores them.

As Section 3.2 shows, HarnessRL's reward agrees with the rubric-free oracle more often than GRPO's, and Table 3 shows that removing the critic lowers OOD performance. We ran two Qwen3.5-4B experiments to test the same point for individual updates on fresh responses and against simpler acceptance rules. For the fresh-response audit, we sampled 200 accepted, 200 rejected, and 200 unrestricted updates from each of three runs, with unrestricted updates applied without a check. For each update, we reloaded its frozen checkpoint, generated J=8J=8 fresh responses to the same task with guidance unchanged, and graded them under both the old and proposed rubrics. Claude Sonnet 5 compared all 28 pairs in both orders without seeing the rubrics. Kendall's τb\tau_b measures agreement between rubric rewards and oracle win counts. Δc\Delta_c is the drop in mean reward on the recorded or fresh group, and σ\sigma is the fresh group's reward standard deviation.

Update setnnτb\tau_b beforeτb\tau_b afterΔc\Delta_c recordedΔc\Delta_c freshσ\sigma fresh
Accepted6000.190.42+8.4+7.014.7
Accepted, rewrite / split2500.180.44+8.9+7.314.9
Accepted, create / merge / delete1600.170.47+11.6+9.515.3
Rejected6000.200.21+5.1+4.212.8
Rejected, no reward drop2460.200.17−2.6−2.113.9
Unrestricted6000.190.23−2.4−1.915.2

Accepted updates ranked fresh responses better (τb\tau_b rose from 0.19 to 0.42), while rejected and unrestricted updates made little difference. Most of the recorded reward drop persisted on fresh responses (+8.4 recorded, +7.0 fresh). Rejected updates also lowered mean reward and retained variation, so neither property alone accounted for the improvement in ranking.

For the acceptance-rule comparison, we retrained for 500 steps with three seeds under the Table 3 setup. Tightening alone checks the direction called for by the diagnosis by requiring a drop in group-mean reward. The magnitude band also requires a drop large enough to distinguish from grading noise and small enough to limit the change before the next RL step. The variants used the rubric checks listed below, B=1B=1 allowed one proposal round across all interfaces, and the Table 3 arm without the critic removed acceptance checks from all three interfaces. Avg. OOD reports the seed mean and standard deviation at retrospectively selected best checkpoints. For the final 100 steps, Δc\Delta_c averages reward drops for accepted updates to spurious criteria, and Reversed (%) counts the share of accepted rubric updates that undid a change to the same criterion from the task's previous two draws.

Rubric acceptance ruleAvg. OODΔc\Delta_cReversed (%)
w/o critic (Table 3)45.9 ± 0.8−3.214.1
Tightening only, Δc>0\Delta_c>047.3 ± 0.7+10.48.7
Variation only46.6 ± 0.8−2.512.4
Magnitude band [3.1,28.0][3.1, 28.0], no variation49.1 ± 0.5+7.02.9
Full rule, B=1B=149.0 ± 0.5+7.41.9
Full critic, B=3B=349.8 ± 0.4+7.21.6

Adding the magnitude band to tightening alone gave the largest OOD gain, from 47.3 to 49.1, with smaller gains from the variation check and additional proposal rounds. Tightening alone produced larger reward drops and more reversals (8.7%), consistent with the band limiting how far corrections go.

Together, the two experiments separate alignment from mere score lowering. A lower but still varied reward is not enough, since rejected updates have both and gain nothing in ranking, while the updates the full rule accepts rank fresh responses better and train a better policy. The gain comes from requiring both standards at once, a drop that agrees with the diagnosis and a size the next RL step can absorb. We will add both experiments beside Table 3 and in Appendix C.

AuthorsOpenReview ↗

Author Response to Reviewer 54QZ (Part 3/4)

W2 and Q4. Independent evaluation of open-ended medical and role-play quality.

We kindly note that the submission already evaluates open-ended generation on 250 held-out tasks in each of the five domains, each scored by the frozen judge under the task's original rubric, the seed rubric that no harness update can change.

As described in Section 3, the tasks cover medicine, science, writing, role-play, and instruction following. The OOD suite changes both the tasks and how they are scored, and only WritingBench uses the training judge. MedQA and GPQA-Diamond accuracy and RoleBench ROUGE-L measure transfer without directly assessing clinical response quality or how well the model plays a character. The reward audit uses the rubric-free Claude Sonnet 5 oracle to evaluate the reward, not the policies, and the oracle agrees with human consensus at Kendall's τ=0.82\tau=0.82.

Following the reviewer's suggestion, we built two evaluation sets with no tasks from the training sources. The medical set contains 300 open-ended patient questions about mental health and primary care, each with a scoring rubric written by practicing clinicians. The role-play set contains 300 characters absent from both the training mix and RoleBench, each played in a three-turn scene with a user. We reused the checkpoints selected on the Table 1 OOD sets. DeepSeek-V3.2, from a different model family than both the gemma-4-31B-it training judge and the Claude Sonnet 5 oracle, scored responses against individual rubric criteria and separately under a rubric-free guideline. The guideline score is the normalized win rate against eight fixed anchors. For each set and scale, three annotators independently ranked GRPO, AMARIS++, and HarnessRL on 100 prompts without knowing which model produced each response. Human first (%) is the share of prompts on which a majority ranked a method first, with ties split.

Model / methodMed. rubricMed. guidelineMed. human first (%)Role rubricRole guidelineRole human first (%)
Qwen3.5-4B
GRPO49.657.317.548.955.816.0
AMARIS++53.260.233.053.059.431.5
HarnessRL56.463.049.556.562.552.5
Qwen3.5-9B
GRPO55.862.419.554.661.018.0
AMARIS++59.165.135.058.464.334.5
HarnessRL61.767.645.561.367.147.5

The independent judge and human annotators favored HarnessRL on both sets at both scales. At 4B, annotators ranked HarnessRL first on 49.5% of medical prompts versus 33.0% for AMARIS++. The only exception in the per-dimension breakdown was narrative quality at 9B, where the 95% interval for the difference from AMARIS++ included zero.

The OOD lead also held with another policy family and another reward judge. We trained GRPO, AMARIS++, and HarnessRL on Llama-3.1-8B-Instruct with three seeds. HarnessRL scored 53.1 OOD versus 50.3 for AMARIS++. In a separate Qwen3.5-4B experiment, we replaced gemma-4-31B-it with gpt-oss-120b as the reward judge for all three methods and as the evolver for HarnessRL. HarnessRL led AMARIS++ by 2.8 OOD points, 49.6 versus 46.8.

Reviewer_tMLKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HarnessRL, a training-time framework that jointly adapts task sampling, rollout guidance, and rubric criteria for open-ended RL. The authors identify three types of reward signals: blind, saturated, and spurious, and introduce a harness evolver that uses training history and cross-task memory to diagnose these failures. A proposer generates candidate updates, while a JIT critic evaluates their effects on the current frozen policy before accepting them. The framework jointly evolves the training harness and the policy during GRPO training. The authors also introduce Rubric-Mix-10k, covering multiple tasks. Experiments with Qwen3.5-4B/9B show that HarnessRL consistently outperforms the evaluated baselines on held-out in-domain and OOD benchmarks. Ablation studies and training efficiency analyses further investigate the contributions of the different interfaces, memory, and the JIT critic.

Critical strengths

The paper addresses an important problem in rubric-based RL by identifying different types of reward failures and proposing a unified framework to address them. Rather than adapting only rubric criteria or rollout guidance, HarnessRL jointly adjusts task sampling, guidance, and reward criteria according to the current policy's training state. The use of training-time memory and a JIT critic provides an interesting mechanism for coordinating these updates.

The experimental evaluation is relatively comprehensive. HarnessRL consistently outperforms the evaluated baselines across multiple domains. The paper also includes multiple training seeds, component-level ablations, OOD evaluations, and computational cost analysis. In particular, the ablations demonstrate the importance of memory and the JIT critic, strengthening the empirical support for the proposed framework.

Critical weaknesses

The framework relies on the same model family for reward judging, failure attribution, proposal generation, and JIT criticism. Although the critic evaluates candidate updates before accepting them, these measurements still depend on the same reward judge. In particular, reducing the reward of existing responses does not necessarily indicate that the updated rubric better reflects actual response quality. This raises concerns about potential self-reinforcing evaluation biases, despite the independent evaluations provided in the paper.  The proposed framework involves multiple interacting components and relies on several heuristic rules and calibrated thresholds. Although the experiments cover two model scales and five domains, both policy models belong to the same family, and all experiments use the same judge backbone. It remains unclear whether the reported improvements would generalize to other model families, reward judges.

Questions for authors

Q1. The JIT critic evaluates updates using the same reward judge as the training process. How can the authors ensure that accepted rubric updates improve actual response quality rather than simply changing the reward scores?

Q2. Does HarnessRL maintain its performance advantage when using different policy model families or reward judges?

Q3. The ablation results show that removing the JIT critic significantly reduces performance. Could the authors further clarify which acceptance mechanisms contribute most to this improvement? (the reward-change thresholds, or the requirement that updated criteria remain discriminative)

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer tMLK (Part 1/3)

We thank the reviewer for recognizing the unified treatment of the three reward failures and the five-domain evaluation, and address below the validation of accepted updates outside the training judge and the generalization across policy families, reward judges, and calibrated thresholds.

W1 and Q1. Whether a lower reward under the same judge means a better rubric.

A lower reward alone does not validate a rubric update, so the JIT critic checks that the change agrees with the attributor's diagnosis and stays within a range set for the next RL step.

As stated in Section 2.3, the attributor labels a criterion spurious only when the rollouts that meet it are no better than those that fail it, and cites the responses that only appear to comply, for example through length gaming, partial coverage, or stance gaming. The critic regrades the same rollouts and requires a drop in mean reward, in line with the diagnosis and the goal of improving agreement with the oracle. The drop must exceed grading noise without being too large for the next RL step, and each new or rewritten criterion must still distinguish among the rollouts. If no update passes within three rounds of proposal and revision, the old rubric stays in place. The critic rejected 57% of first-round rubric proposals in the Qwen3.5-4B run. Because regrading uses the training judge, the evidence that accepted updates track quality comes from outside it.

Beyond the training judge, human annotation and the rubric-free oracle show that accepted updates track quality. Three annotators confirmed 86.0% of spurious labels and judged that 82.5% of accepted updates addressed the diagnosed failure (Appendix C). At the end of training, reward agreed with the rubric-free Claude Sonnet 5 oracle on 48.9% of response pairs, versus 38.4% for GRPO, and the oracle agreed with human consensus at Kendall's τ=0.82\tau=0.82. As W2b shows, HarnessRL kept its lead when we replaced the reward judge.

W2a. Heuristic rules and calibrated thresholds.

The thresholds are not hand-tuned constants. They adapt to the policy being trained, because the memory cold start calibrates them from that policy's own rollouts on the training tasks before RL begins. Each band therefore reflects how the initial policy's rewards vary.

As described in Appendix D, the lower bounds filter out changes in mean reward on the scale of grading noise for rubric updates and sampling noise for guidance updates. The cold start measures grading noise by rescoring the same rollouts and sampling noise by drawing fresh groups for the same task. Both bands cap the change in mean reward at the typical spread of rewards within a group, limiting how much the reward can change before the next RL step. The sampling levels also come from those rollouts, using quantiles of the share of criteria that distinguish among responses on each task. No held-out or OOD scores enter the calibration.

On Qwen3.5-4B, changing one bound at a time moved mean OOD by at most 0.5 points, within seed variation. Dropping the rubric band and keeping only the sign check lowered mean OOD to 47.3.

AuthorsOpenReview ↗

Author Response to Reviewer tMLK (Part 2/3)

W2b and Q2. Other policy families and reward judges.

HarnessRL does not depend on the policy family or the reward judge, because it only adapts the training harness around a rubric-based RL loop and its cold start sets the thresholds from whatever initial policy it trains.

To check this on a second policy family, we trained GRPO, AMARIS++, and HarnessRL on Llama-3.1-8B-Instruct for 500 steps with three seeds, keeping the gemma judge, original task rubrics, and evaluation protocols fixed. As in Table 1, we report three-seed means at the best checkpoint on each evaluation set.

PolicyMethodIn-domainOOD
Qwen3.5-4BGRPO57.843.9
Qwen3.5-4BAMARIS++61.547.0
Qwen3.5-4BHarnessRL64.549.8
Llama-3.1-8B-InstructGRPO61.247.6
Llama-3.1-8B-InstructAMARIS++64.150.3
Llama-3.1-8B-InstructHarnessRL67.053.1

HarnessRL also led on Llama, by 2.9 in-domain and 2.8 OOD points over AMARIS++ trained on the same policy.

On Qwen3.5-4B, we tested gpt-oss-120b as the attributor and proposer, leaving the gemma judge and JIT acceptance rules unchanged. We then used gpt-oss-120b as both the reward judge and the evolver backbone, with Claude Sonnet 5 still serving as the external oracle and GRPO and AMARIS++ trained with the same replacement judge.

Reward judgeConfigurationIn-domainOOD
gemma-4-31B-itHarnessRL, gemma evolver64.549.8
gemma-4-31B-itHarnessRL, gpt-oss evolver64.149.9
gpt-oss-120bHarnessRL, gpt-oss evolver64.049.6
gpt-oss-120bGRPO58.344.5
gpt-oss-120bAMARIS++61.646.8

With gpt-oss-120b as the reward judge, HarnessRL kept a 2.8-point OOD lead over AMARIS++, and its reward agreed with the oracle on 48.1% of pairs at the end of training. OOD performance also held when we changed only the attributor and proposer, so they no longer shared a model family with the judge and critic. In-domain scores from different judges are not directly comparable.

We will add the policy-family, judge and threshold-sensitivity results to the revision.

AuthorsOpenReview ↗

Author Response to Reviewer tMLK (Part 3/3)

Q3. Which acceptance condition of the critic matters most.

The critic's acceptance conditions work together, with one check at each turn of a single cycle. A blind criterion, on which every rollout scores 0, becomes saturated once guidance helps every rollout score 1, and once the guidance is withdrawn it may separate rollouts only by surface compliance, which makes it spurious. Its stricter rewrite can then leave it blind again on a later draw, at a higher level of response quality.

Each turn has its own interface, and the critic checks each one on the current policy. HarnessRL adds guidance for a blind criterion, samples a saturated task less often and withdraws its guidance once the policy can do without it, and rewrites a spurious criterion. The tighter rubric then opens the next turn of the cycle.

Section 3.2 shows what happens when only one interface adapts. RuscaRL lowered the share of blind pairs to 10.0% but raised the share of saturated pairs to 17.1%. AMARIS lowered the shares of saturated and spurious pairs but left 13.3% blind, above GRPO's 12.0%. With all three interfaces adapting and the critic checking each primary update against the current policy, HarnessRL lowered the blind, saturated, and spurious shares to 10.5%, 13.4%, and 27.2%, respectively.

For a rubric update targeting a spurious criterion, the sign condition requires mean reward to fall on the same rollouts, as called for by the diagnosis. The magnitude band rules out drops that are within grading noise or too large for the next RL step. The variation check requires each new or rewritten criterion to distinguish among the rollouts. Without the critic, every HarnessRL variant that updated the rubric loosened it on average (Appendix C).

We compared rubric acceptance rules by retraining Qwen3.5-4B for 500 steps with three seeds, keeping the sampling and guidance rules, memory, judge, proposer, and training recipe fixed. The w/o critic arm from Table 3 removes the critic from all interfaces and serves as a reference. The B=1B=1 arm keeps the full rule but allows one round instead of three. Avg. OOD is the mean across three seeds at the retrospectively selected best checkpoint, reported with the seed standard deviation. Over the last 100 steps, mean Δc\Delta_c is the average reward drop for accepted updates targeting spurious criteria, and Reversed is the percentage of accepted rubric updates that undo a change to the same criterion from the task's previous two draws.

Acceptance configurationAvg. OODMean Δc\Delta_cReversed (%)
w/o critic (Table 3)45.9±0.845.9\pm0.8−3.214.1
Tightening only, Δc>0\Delta_c>047.3±0.747.3\pm0.7+10.48.7
Variation only46.6±0.846.6\pm0.8−2.512.4
Magnitude band [3.1,28.0][3.1, 28.0], no variation49.1±0.549.1\pm0.5+7.02.9
Full rule, B=1B=149.0±0.549.0\pm0.5+7.41.9
Full critic, B=3B=349.8±0.449.8\pm0.4+7.21.6

The full rule scored highest, and the magnitude band made the larger difference. Adding the band to tightening alone raised OOD by 1.8 points, and adding the variation check at three rounds raised it by another 0.7. Tightening alone produced larger reward drops and more reversals than the full rule, while variation alone still allowed the rubric to loosen.

Reviewer_Hnm4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a fundamental vulnerability in reinforcement learning post-training for open-ended generation tasks, where verifiable rule-based reward functions are absent. The paper is exceptionally well-written, theoretically sound, and supported by thorough empirical experiments. It establishes a compelling paradigm shift from "static post-training" to "co-evolving training environments," making it a valuable contribution to the literature on RL for LLM alignment.

Critical strengths
  1. HarnessRL simultaneously couples task sampling, rollout guidance, and rubric refinement to eliminate the interdependent failure modes of reward saturation and sparsity that single-interface adaptations cannot resolve.

  2. Decoupling attribution, proposing, and empirical validation prevents self-confirming hallucinations by verifying candidate harness updates on a frozen policy before acceptance

  3. By dynamically adapting training difficulty and exploration support, the policy attains peak out-of-distribution performance in nearly half the training steps required by standard GRPO

Critical weaknesses
  1. The framework relies on a single open-weights model as the shared backbone for the Judge, Attributor, Proposer, and Critic.

  2. While HarnessRL converges in fewer total training steps, the per-step cost is 1.6× that of baseline GRPO

  3. The semantic replay memory lacks a pruning mechanism, retaining stale early rollouts that may degrade retrieval relevance as the policy shifts over long training runs

  4. The reported maximum gains depend heavily on intermediate checkpoint selection, which poses practical challenges in production deployments lacking dense ground-truth validation

Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer Hnm4 (Part 1/3)

We thank the reviewer for recognizing the coupled harness interfaces and the frozen-policy validation of candidate updates, and address below the external bounds on the shared judge and the cost of training and checkpoint selection.

W1. Bias from one shared judge and evolver model.

We kindly note that the submission already reports independent external checks using four OOD benchmarks scored without gemma-4-31B-it, a rubric-free Claude Sonnet 5 oracle, and human annotators.

In Table 1, HarnessRL outperformed AMARIS++ at both policy scales on MedQA and GPQA-Diamond (exact-choice scoring), IFBench (programmatic checks), and RoleBench (ROUGE-L). The oracle agreed with human consensus at Kendall's τ=0.82\tau=0.82, and no harness role saw its output. At training end, HarnessRL's reward agreed with the oracle on 48.9% of pairs, compared with 38.4% for GRPO. Three annotators confirmed 86.0% of spurious labels and judged that 82.5% of accepted rubric updates addressed the diagnosed failure.

A shared model can reinforce its own judgments by loosening the rubric it uses to grade responses, as the rubric-updating HarnessRL variants without the critic did on average (Appendix C). For an update targeting a spurious criterion, the JIT critic blocks this kind of score inflation on the recorded group. After the attributor cites responses that only appear to meet the criterion, the critic regrades the same rollouts. It accepts the update only if mean reward falls in the direction called for by the diagnosis, with the aim of bringing reward closer to the oracle, and if the drop is large enough to exceed grading noise but small enough to avoid an abrupt change in the reward for the next RL step (Section 2.3). Each new or rewritten criterion must still distinguish among the rollouts, and the old rubric stays in place if no proposal passes within three rounds. In the Qwen3.5-4B run, 57% of first-round rubric proposals failed. For a blind spot the judge applies consistently, the decisive check is replacing the judge, which we report below. We use one model across roles to keep costs down through a shared service pool.

In a three-seed Qwen3.5-4B experiment, we swapped the model used by the attributor and proposer while keeping the gemma-4-31B-it reward judge, critic rule, and other settings fixed. "Agree at best" measures reward-oracle pair agreement at each arm's best OOD checkpoint. Evolver cost is relative to 0.7 cluster hours per 100 steps for attribution and proposal. For HarnessRL, hours to best include the 0.75-hour cold start.

ArmIn-domainOODAgree at best (%)Evolver costHours to best
GRPO57.843.9--20.4
AMARIS++61.547.0--18.3
HarnessRL, gemma-4-31B-it64.549.846.41.0×17.5
HarnessRL, gemma-4-12B-it62.948.143.70.4×19.7
HarnessRL, gpt-oss-120b64.149.946.10.7×17.0
HarnessRL, GPT-5.665.450.948.63.0×21.0

HarnessRL kept its lead over AMARIS++ with every tested replacement for attribution and proposal. gpt-oss-120b matched the original model's OOD performance, GPT-5.6 scored higher at greater evolver cost, and the smaller gemma-4-12B-it still beat AMARIS++. With gpt-oss-120b in all four training roles, including the reward judge, HarnessRL still led matched AMARIS++ by 2.8 OOD points (49.6 versus 46.8), so the gain did not depend on Gemma's judgments.

AuthorsOpenReview ↗

Author Response to Reviewer Hnm4 (Part 2/3)

W2. Per-step training cost against GRPO.

HarnessRL costs more per step than GRPO, but it reaches a better policy in far fewer steps and scores higher at the same compute budget. The extra compute lets HarnessRL correct reward failures during training while GRPO keeps its rubric fixed.

At the same 17.5 cluster hours, HarnessRL reaches 49.8 OOD against 43.8 for GRPO and 46.9 for AMARIS++ (Appendix C), and the critic accounts for half of the extra 2.4 cluster hours per 100 steps over GRPO. Every adaptive baseline in Figure 5 also costs more per step than GRPO, including AMARIS++, which has no critic. HarnessRL's earlier peak is consistent with correcting the reward as it degrades while GRPO keeps its rubric fixed.

We tested a cheaper schedule by calling the attributor, proposer, and critic every fourth step when training Qwen3.5-4B for 500 steps with three seeds under the Section 3 protocol. Between calls, we kept the harness fixed and continued saving rollouts and scores to memory. The last column gives the best OOD score reached within 17.5 cluster hours.

MethodHours / 100 steps (× GRPO)Steps to best OODHours to bestBest OODOOD at 17.5 h
GRPO4.3 (1.00×)47520.443.9 ± 0.543.8
AMARIS++6.1 (1.42×)30018.347.0 ± 0.746.9
HarnessRL, every 4 steps5.1 (1.19×)35018.648.4 ± 0.547.8
HarnessRL6.7 (1.56×)25017.549.8 ± 0.449.8

Updating every fourth step retained most of the gain and still beat AMARIS++ at 1.19× GRPO's per-step cost. Calling the evolver every step raised OOD at the same budget and reached its own best checkpoint sooner. W4 shows how to choose a checkpoint without OOD labels.

W3. Stale records in an unpruned replay memory.

Old records cannot push a stale update into training. They only inform a proposal, and the JIT critic accepts or rejects every update against the current policy. In practice, the semantics index also retrieves mostly recent records, since it matches the current criterion.

The task's own long history is useful. As Section 2.2 describes, the time index returns the task's earlier records. The attributor uses them to tell a persistent failure from a one-off result, and the proposer can see which updates were already rejected or reversed. The semantics index looks for criteria from other tasks that closely match the current one and returns responses that met or missed each match, which the proposer can use in a <reference> field. If a proposed <reference> uses an old example to help the policy meet a blind criterion, the critic draws a fresh group from the frozen current policy with the proposed guidance. It compares the group's mean reward with that of the recorded group and accepts the addition only if the increase is large enough to exceed sampling noise and small enough to avoid an abrupt reward change before the next RL step.

To test whether keeping old records hurt performance, we measured the age of retrieved records at step 500 and trained three Qwen3.5-4B memory variants with three seeds under the same 500-step recipe. Recency weighting multiplied similarity by exp⁡(−age/250)\exp(-\mathrm{age}/250), while pruning removed records older than 250 or 100 steps from both indexes. We report the age of matches from the semantics index, the share of accepted references drawn from other tasks, the rubric reversal rate over the final 100 steps, and mean OOD at the best checkpoints.

MemoryRetrieved records older than 100 / 200 / 300 steps (%)Cross-task referencesReversed updatesAvg. OOD
No eviction11.8 / 3.9 / 1.258%1.6%49.8 ± 0.4
Recency-weighted, τ=250\tau=2507.4 / 1.6 / 0.458%1.6%49.8 ± 0.4
Pruned, W=250W=25010.6 / 2.3 / 057%1.7%49.7 ± 0.4
Pruned, W=100W=1000 / 0 / 049%3.4%49.1 ± 0.6

Keeping older records did not lower mean OOD over 500 steps. At step 500, only 1.2% of the unpruned matches were more than 300 steps old. Recency weighting and the 250-step cutoff made little difference to OOD or reversals. The 100-step cutoff reduced the share of references from other tasks, increased reversals, and lowered mean OOD to 49.1, a modest drop relative to seed variation.

AuthorsOpenReview ↗

Author Response to Reviewer Hnm4 (Part 3/3)

W4. Dependence on best-checkpoint selection.

HarnessRL does not need a well-chosen intermediate checkpoint. Training can stop at a step fixed in advance, and at that final checkpoint HarnessRL still leads the strongest baseline.

At the fixed step-500 checkpoint, HarnessRL leads AMARIS++ by 3.0 OOD points (Appendix C), more than the 2.8 points in Table 1. Selecting the best checkpoint separately on each evaluation set changes the absolute scores without creating the advantage, and it requires evaluation scores that may be unavailable in deployment.

We compared two ways to choose among checkpoints saved every 25 steps without using OOD scores. Judge-selected picks the checkpoint with the highest held-out in-domain score, graded by gemma-4-31B-it under each task's fixed original rubric. Evolver-settled picks the earliest checkpoint at which the previous 100 steps have fewer than one quarter as many accepted updates as the first 100 steps and the logged group-mean reward varies by less than 3.1 points.

Checkpoint ruleAMARIS++ (in-domain / OOD)HarnessRL (in-domain / OOD)OOD margin
Retrospective best on each set61.5 / 47.064.5 / 49.8+2.8
Fixed step 50061.5 / 46.464.5 / 49.4+3.0
Judge-selected61.5 / 46.564.5 / 49.5+3.0
Evolver-settled61.4 / 46.564.2 / 49.5+3.0

Both rules preserved HarnessRL's lead without using ground-truth labels for selection, and both selected checkpoints that scored 49.5 OOD. Judge-selected requires held-out in-domain tasks and their rubrics, while evolver-settled uses only training logs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks from the Authors

Dear AC, SAC, and Reviewers,

We sincerely thank the AC and SAC for coordinating the review process and the reviewers for their constructive feedback, which helped us strengthen the validation of the JIT critic and broaden our evaluation.

Summary of the Original Reviews

Major strengths include consistent gains over the strongest baselines across model scales and domains (54QZ, tMLK), thorough experiments (tMLK, Hnm4), a unified framework that adapts task sampling, rollout guidance and rubric criteria to diagnosed reward failures (tMLK, Hnm4), the JIT critic's check of each update on the frozen policy before acceptance (tMLK, Hnm4), controls beyond the training reward such as matched compute, an independent oracle and human audits (54QZ), and clear writing and faster convergence (Hnm4).

Key concerns include whether accepted rubric updates improve reward quality or only lower scores (54QZ, tMLK), which acceptance conditions explain the critic's gain (54QZ, tMLK), the use of one model as judge, attributor, proposer and critic (tMLK, Hnm4), generality beyond one policy family and one judge backbone (54QZ, tMLK), limited independent evaluation of open-ended medical and role-play responses (54QZ), and the per-step training cost and reliance on best-checkpoint selection (Hnm4).

Summary of the Rebuttal and Discussion

We addressed all substantive concerns with targeted clarifications, new experiments, and supporting evidence:

  • Clarified that the critic accepts an update to a spurious criterion only if reward drops where the attributor found it unearned, by an amount the next RL step can absorb (54QZ, tMLK, Hnm4).
  • Showed on fresh responses that accepted updates raised the reward's Kendall correlation with an independent oracle from 0.19 to 0.42, while rejected and unrestricted updates changed it little (54QZ).
  • Compared the full acceptance rule with simpler rules in retrained runs: the full rule scored highest, and the magnitude band made the largest difference (54QZ, tMLK).
  • Tested other evolvers, other reward judges (including gpt-oss-120b) and a second policy family, Llama-3.1-8B-Instruct: HarnessRL led AMARIS++ in every setting, by 2.8 OOD points on Llama (54QZ, tMLK, Hnm4).
  • Noted that the submission already evaluates open-ended responses on held-out tasks in all five domains, and added new medical and role-play sets with no training tasks, where a judge from a third model family and blind human rankings favored HarnessRL at both scales (54QZ).
  • Showed that HarnessRL beat GRPO and AMARIS++ at matched compute and kept its lead at a checkpoint fixed in advance or chosen without OOD labels (Hnm4).

Final Resolution

Reviewer 54QZ replied that our response addressed their concerns and raised their rating from 2 (Weak rejection) to 3 (Weak acceptance).

Reviewer Hnm4 replied to our response and raised their rating from 3 (Weak acceptance) to 4 (Clear acceptance).

Reviewer tMLK did not reply during the author-reviewer discussion period and maintained a rating of 3 (Weak acceptance). We addressed all their points in detail, such as human and oracle checks that accepted rubric updates track quality, results with a second policy family and another reward judge, and an ablation of the critic's acceptance rules.

We thank the AC, SAC, and reviewers again for their thoughtful engagement and hope our responses are helpful in the final decision process.

Sincerely,

The Authors of Paper 17720

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.