- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes HarnessRL, a training-time framework that jointly adapts task sampling, privileged rollout guidance, and rubric-based rewards during reinforcement learning. An attributor diagnoses reward failures, a proposer suggests interventions, and a just-in-time critic accepts or rejects them based on measurements against the current frozen policy. Historical and cross-task memory supports diagnosis and proposal generation.
The framework is evaluated on a curated five-domain training mixture, with held-out and out-of-distribution evaluations using Qwen3.5-4B and Qwen3.5-9B. It reports approximately three-point improvements over the strongest baseline, alongside critic and memory ablations, reward–oracle agreement analysis, human audits, and compute comparisons.
Critical strengths
- Consistent downstream improvements with useful controls. HarnessRL improves over the strongest reported baseline across both policy scales and all evaluated domains. The appendix shows that the advantage persists at the final checkpoint and under matched compute, reducing concerns that the result depends primarily on evaluation-based checkpoint selection. These findings support the practical value of the implemented system.
- Additional evidence beyond the evolving training reward. The evaluation includes objective OOD scoring protocols, an independent oracle judge that does not supply training feedback, and human audits of diagnoses and rubric updates. These checks provide meaningful support that the improvements are not solely an artifact of changing the training rubric.
Critical weaknesses
- The distinctive contribution of the JIT Critic is insufficiently isolated, and its rubric acceptance criterion does not directly validate reward quality. For rubric updates targeting spurious rewards, acceptance requires a bounded decrease in mean reward and variation across responses for created or rewritten criteria. These properties do not establish that the revised reward favors better responses; an inappropriate criterion could satisfy both. Furthermore, proposals are constructed and validated using the same recorded response group. Although removing the critic substantially reduces performance, that ablation compares the proposed mechanism with unrestricted proposal acceptance, rather than with simpler tightening or discrimination filters. Given the substantial overlap with prior adaptive-rubric and harness-evolution approaches, the absence of such controls leaves the central methodological contribution unclear.
- The evaluation provides limited coverage of the claimed open-ended capabilities and generality. Despite spanning five domains, the evidence comes from one curated training mixture, two policy sizes within one family, and one judge backbone. MedQA and GPQA measure multiple-choice accuracy rather than open-ended medical or scientific response quality; RoleBench is scored using ROUGE-L, which provides limited evidence about persona consistency, engagement, or narrative quality; and WritingBench uses the training judge. The results demonstrate encouraging benchmark transfer, but provide a narrower basis for claims about broadly improving open-ended RL. This limitation is consequential because empirical breadth is particularly important for an otherwise incremental system contribution.
Questions for authors
- Mechanism validation: Do accepted rubric updates improve reward–quality ordering on fresh responses under independent evaluation? In particular, what evidence distinguishes improved alignment from simply lowering mean scores while retaining response discrimination?
- Ablation clarification: Do you have comparisons with simpler acceptance rules, such as requiring only rubric tightening and criterion variation? Which of the magnitude bounds, variation requirement, and iterative revision account for the critic’s improvement?
- Baseline clarification: Which memory and retrieval capabilities of the original AMARIS are retained in AMARIS and AMARIS++? The description of AMARIS++ as retaining only a per-task revision log appears narrower than the original method’s historical and semantic retrieval.
- Evaluation clarification: Do you have independent evaluations of open-ended medical response quality or role-playing quality beyond MedQA accuracy and RoleBench ROUGE-L? Such evidence could materially strengthen the connection between the reported gains and the capabilities motivating the paper.
6 comments
Author Response to Reviewer 54QZ (Part 2/4)
Q1 and Q2. Accepted updates on fresh responses, and simpler acceptance rules.
We would clarify that the critic's acceptance rule improves how the reward ranks responses, that is, its agreement with the oracle, not just how low it scores them.
As Section 3.2 shows, HarnessRL's reward agrees with the rubric-free oracle more often than GRPO's, and Table 3 shows that removing the critic lowers OOD performance. We ran two Qwen3.5-4B experiments to test the same point for individual updates on fresh responses and against simpler acceptance rules. For the fresh-response audit, we sampled 200 accepted, 200 rejected, and 200 unrestricted updates from each of three runs, with unrestricted updates applied without a check. For each update, we reloaded its frozen checkpoint, generated fresh responses to the same task with guidance unchanged, and graded them under both the old and proposed rubrics. Claude Sonnet 5 compared all 28 pairs in both orders without seeing the rubrics. Kendall's measures agreement between rubric rewards and oracle win counts. is the drop in mean reward on the recorded or fresh group, and is the fresh group's reward standard deviation.
| Update set | before | after | recorded | fresh | fresh | |
|---|---|---|---|---|---|---|
| Accepted | 600 | 0.19 | 0.42 | +8.4 | +7.0 | 14.7 |
| Accepted, rewrite / split | 250 | 0.18 | 0.44 | +8.9 | +7.3 | 14.9 |
| Accepted, create / merge / delete | 160 | 0.17 | 0.47 | +11.6 | +9.5 | 15.3 |
| Rejected | 600 | 0.20 | 0.21 | +5.1 | +4.2 | 12.8 |
| Rejected, no reward drop | 246 | 0.20 | 0.17 | −2.6 | −2.1 | 13.9 |
| Unrestricted | 600 | 0.19 | 0.23 | −2.4 | −1.9 | 15.2 |
Accepted updates ranked fresh responses better ( rose from 0.19 to 0.42), while rejected and unrestricted updates made little difference. Most of the recorded reward drop persisted on fresh responses (+8.4 recorded, +7.0 fresh). Rejected updates also lowered mean reward and retained variation, so neither property alone accounted for the improvement in ranking.
For the acceptance-rule comparison, we retrained for 500 steps with three seeds under the Table 3 setup. Tightening alone checks the direction called for by the diagnosis by requiring a drop in group-mean reward. The magnitude band also requires a drop large enough to distinguish from grading noise and small enough to limit the change before the next RL step. The variants used the rubric checks listed below, allowed one proposal round across all interfaces, and the Table 3 arm without the critic removed acceptance checks from all three interfaces. Avg. OOD reports the seed mean and standard deviation at retrospectively selected best checkpoints. For the final 100 steps, averages reward drops for accepted updates to spurious criteria, and Reversed (%) counts the share of accepted rubric updates that undid a change to the same criterion from the task's previous two draws.
| Rubric acceptance rule | Avg. OOD | Reversed (%) | |
|---|---|---|---|
| w/o critic (Table 3) | 45.9 ± 0.8 | −3.2 | 14.1 |
| Tightening only, | 47.3 ± 0.7 | +10.4 | 8.7 |
| Variation only | 46.6 ± 0.8 | −2.5 | 12.4 |
| Magnitude band , no variation | 49.1 ± 0.5 | +7.0 | 2.9 |
| Full rule, | 49.0 ± 0.5 | +7.4 | 1.9 |
| Full critic, | 49.8 ± 0.4 | +7.2 | 1.6 |
Adding the magnitude band to tightening alone gave the largest OOD gain, from 47.3 to 49.1, with smaller gains from the variation check and additional proposal rounds. Tightening alone produced larger reward drops and more reversals (8.7%), consistent with the band limiting how far corrections go.
Together, the two experiments separate alignment from mere score lowering. A lower but still varied reward is not enough, since rejected updates have both and gain nothing in ranking, while the updates the full rule accepts rank fresh responses better and train a better policy. The gain comes from requiring both standards at once, a drop that agrees with the diagnosis and a size the next RL step can absorb. We will add both experiments beside Table 3 and in Appendix C.
Author Response to Reviewer 54QZ (Part 3/4)
W2 and Q4. Independent evaluation of open-ended medical and role-play quality.
We kindly note that the submission already evaluates open-ended generation on 250 held-out tasks in each of the five domains, each scored by the frozen judge under the task's original rubric, the seed rubric that no harness update can change.
As described in Section 3, the tasks cover medicine, science, writing, role-play, and instruction following. The OOD suite changes both the tasks and how they are scored, and only WritingBench uses the training judge. MedQA and GPQA-Diamond accuracy and RoleBench ROUGE-L measure transfer without directly assessing clinical response quality or how well the model plays a character. The reward audit uses the rubric-free Claude Sonnet 5 oracle to evaluate the reward, not the policies, and the oracle agrees with human consensus at Kendall's .
Following the reviewer's suggestion, we built two evaluation sets with no tasks from the training sources. The medical set contains 300 open-ended patient questions about mental health and primary care, each with a scoring rubric written by practicing clinicians. The role-play set contains 300 characters absent from both the training mix and RoleBench, each played in a three-turn scene with a user. We reused the checkpoints selected on the Table 1 OOD sets. DeepSeek-V3.2, from a different model family than both the gemma-4-31B-it training judge and the Claude Sonnet 5 oracle, scored responses against individual rubric criteria and separately under a rubric-free guideline. The guideline score is the normalized win rate against eight fixed anchors. For each set and scale, three annotators independently ranked GRPO, AMARIS++, and HarnessRL on 100 prompts without knowing which model produced each response. Human first (%) is the share of prompts on which a majority ranked a method first, with ties split.
| Model / method | Med. rubric | Med. guideline | Med. human first (%) | Role rubric | Role guideline | Role human first (%) |
|---|---|---|---|---|---|---|
| Qwen3.5-4B | ||||||
| GRPO | 49.6 | 57.3 | 17.5 | 48.9 | 55.8 | 16.0 |
| AMARIS++ | 53.2 | 60.2 | 33.0 | 53.0 | 59.4 | 31.5 |
| HarnessRL | 56.4 | 63.0 | 49.5 | 56.5 | 62.5 | 52.5 |
| Qwen3.5-9B | ||||||
| GRPO | 55.8 | 62.4 | 19.5 | 54.6 | 61.0 | 18.0 |
| AMARIS++ | 59.1 | 65.1 | 35.0 | 58.4 | 64.3 | 34.5 |
| HarnessRL | 61.7 | 67.6 | 45.5 | 61.3 | 67.1 | 47.5 |
The independent judge and human annotators favored HarnessRL on both sets at both scales. At 4B, annotators ranked HarnessRL first on 49.5% of medical prompts versus 33.0% for AMARIS++. The only exception in the per-dimension breakdown was narrative quality at 9B, where the 95% interval for the difference from AMARIS++ included zero.
The OOD lead also held with another policy family and another reward judge. We trained GRPO, AMARIS++, and HarnessRL on Llama-3.1-8B-Instruct with three seeds. HarnessRL scored 53.1 OOD versus 50.3 for AMARIS++. In a separate Qwen3.5-4B experiment, we replaced gemma-4-31B-it with gpt-oss-120b as the reward judge for all three methods and as the evolver for HarnessRL. HarnessRL led AMARIS++ by 2.8 OOD points, 49.6 versus 46.8.
Author Response to Reviewer 54QZ (Part 1/4)
We thank the reviewer for recognizing the consistent gains under final-checkpoint and matched-compute controls and the evidence beyond the training reward, and address below the validation of the JIT critic's acceptance rule and the breadth of the open-ended evaluation.
The JIT critic accepts an update to a spurious criterion only when it meets two standards, the reward change must agree with the attributor's diagnosis, lowering reward for the responses it cited as surface compliance and so moving toward the oracle, and the size of that change must stay in a range that is safe for the next RL step.
As stated in Section 2.3, the attributor judges whether meeting the criterion reflects quality. Without seeing the oracle, it compares rollouts that meet the criterion with those that fail it and cites responses that only appear to comply. Three annotators confirmed 86.0% of these labels. The critic then regrades the same rollouts under the proposed rubric and accepts the update only if reward falls where the diagnosis says it was unearned.
The second standard keeps the change neither too small nor too large. A drop within grading noise would mean the update does nothing, while a very large drop would change the reward so abruptly that the next RL step trains toward a different objective. Each new or rewritten criterion must also still separate the rollouts, and a rejected update goes back to the proposer, with the old rubric kept if no revision passes within three rounds. In the Qwen3.5-4B run, 57% of first-round rubric proposals failed these checks.
Updates that pass move the reward toward the oracle. At the end of training, HarnessRL's reward agreed with the oracle on 48.9% of response pairs, versus 38.4% for GRPO (Section 3.2). Annotators judged that 82.5% of accepted updates addressed the diagnosed failure, and removing the critic cost 3.9 OOD points (Table 3). Q1 and Q2 below test accepted updates on fresh responses and compare the critic with simpler rules. We will state these two standards explicitly in Section 2.3.