acceptodds
Under review as a conference paper at ICLR 2027

NoiseRub: Noise-Calibrated Dynamic Rubric Rewards for Real-Time Visual Games

Abstract

Training agents in real-time visual games from screen-only interaction requires dense feedback beyond sparse terminal outcomes. Vision-language judges can provide rubric-based rewards, but dynamically weighting these rewards is challenging because across-segment score variance can reflect behavioral differences, judge inconsistency, or limited opportunities for a rubric to apply. We introduce NoiseRub, a noise-calibrated approach to dynamic rubric rewards. Rather than treating observed score variance as evidence of an informative rubric, NoiseRub asks how much of that variation provides reliable evidence for adapting each rubric's weight and how well that evidence is supported by evaluation opportunities. It therefore treats dynamic weighting as a reliability-estimation problem: judge-induced variation should be discounted, while weight adaptation should reflect how often a rubric can be meaningfully evaluated. Each rubric's reliable variation is estimated by subtracting a within-segment noise floor, obtained from repeated audits, from its across-segment variance; the resulting nonnegative residual is then modulated by opportunity coverage. The resulting weights aggregate rubric scores into segment rewards for online policy optimization. In Black Myth: Wukong, NoiseRub achieves 63.5% agreement with human overall preferences, compared with 57.4% for fixed weights and 60.0% for POW3R-Game. Under matched initialization and interaction budgets, it achieves the highest HP reduction on all four evaluated bosses, outperforming POW3R-Game by 4.45–8.04 percentage points. Supplementary videos are available on the https://visual-game-supplement.pages.dev.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visual Game Agents, Rubric-Based Rewards, Vision-Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xSbiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NOISERUB, a dynamic rubric reward weighting method for real-time visual games. It uses repeated VLM evaluations to estimate judge noise and combines this with rubric coverage to calibrate weight updates. The method is evaluated on four Boss tasks in Black Myth: Wukong.

Critical strengths
  1. The paper addresses a clear and practically relevant problem: rubric score variance may reflect judge instability rather than useful behavioral differences. NOISERUB explicitly accounts for both judge noise and opportunity coverage when adapting rubric weights.
  2. The ablation study is well designed. Comparisons with w/o Noise Calibration, w/o Coverage, and POW3R-Game + Mean help show that the gains come from the proposed weighting mechanism rather than repeated-score averaging alone.
Critical weaknesses
  1. The cross-game evaluation is limited. In Honor of Kings: World, NOISERUB is only compared with SFT, without stronger rubric-based baselines such as POW3R-Game or RaR-Static. This provides limited evidence for cross-game generalization.
  2. The compute/audit-matched comparison is only conducted on Black Bear Guai. Since NOISERUB requires repeated VLM audits, it would be more convincing to include POW3R-Game + Mean on the other Boss tasks as well, to verify that the advantage remains under a matched repeated-evaluation protocol.
Questions for authors
  1. Why does the Honor of Kings: World evaluation only compare NOISERUB with SFT? Do you have results against stronger rubric-based baselines such as POW3R-Game or RaR-Static?
  2. Is the compute/audit-matched comparison with POW3R-Game + Mean available for the other Boss tasks? This would help clarify whether the gains come from NOISERUB itself rather than repeated VLM evaluation.
2 comments

We thank the reviewer for identifying the need for stronger cross-game baselines and repeated-scoring comparisons on the other Bosses. We have completed both sets of comparisons and report their settings and results below.

Q1&W1 · Stronger rubric-reward baselines in the second game

We have added RaR-Static, POW3R-Game, and POW3R-Game + Mean on both tasks in Honor of Kings: World. The submitted SFT comparison measures improvement over the initial policy; the added baselines compare fixed weights, variance-driven adaptation, and repeated-score averaging. All methods share the same character configuration and 20-episode evaluation protocol. All post-training methods start from the same SFT policy and use a 30-round interaction budget; SFT is evaluated without additional post-training. HP reduction includes failures, and Yuange Puppet evaluation covers Phase 3 only.

Mean Boss HP reduction in Honor of Kings: World.

MethodYuange Puppet (Phase 3)Qingyue Zhujiang
SFT47.86%36.22%
RaR-Static52.81%48.93%
POW3R-Game61.73%51.81%
POW3R-Game + Mean59.62%55.48%
NoiseRub66.85%59.23%

NoiseRub achieves the highest mean HP reduction on both tasks. Its gains over POW3R-Game are 5.12 and 7.42 percentage points, and its gains over POW3R-Game + Mean are 7.23 and 3.75 points. The added comparisons support NoiseRub relative to alternative rubric-reward schemes in a second game. The complete setup and results are included in Appendix A.8.2, Table 8, of the revised manuscript.

Q2&W2 · Repeated-scoring comparisons across all four Bosses

We have extended the POW3R-Game + Mean comparison to White-Clad Noble, Black Wind King, and Wandering Wight. Together with the submitted Black Bear Guai result, the comparison now covers all four Boss tasks in Black Myth: Wukong.

Both methods use the repeated-scoring protocol in Appendix A.5: each segment receives one evaluation and is selected with probability 0.2 for two additional evaluations. Rewards aggregate the available valid-score means for each rubric. The methods share repeated-audit sampling and valid-score averaging but differ in weight adaptation. POW3R-Game + Mean uses first-pass score variance with its original eligibility rules, whereas NoiseRub combines noise-corrected variance with applicability coverage and audit-support requirements. The comparison therefore evaluates the complete calibrated weight-update rule beyond repeated-score averaging.

Each run uses 30 rounds with 200 seconds of interaction per round, and the final policy is evaluated on 20 episodes, including failures. The table reports mean Boss HP reduction and NoiseRub’s gain over +Mean in percentage points.

Mean Boss HP reduction (%) across four Bosses. The last column gives the gain over +Mean in percentage points.

BossPOW3R-GamePOW3R-Game + MeanNoiseRubGain over +Mean (pp)
White-Clad Noble61.5762.3066.02+3.72
Black Wind King52.4654.4959.85+5.36
Black Bear Guai51.8354.6259.87+5.25
Wandering Wight70.3468.2275.14+6.92

NoiseRub improves over +Mean by 3.72, 5.36, and 6.92 percentage points on the three added tasks, alongside the 5.25-point gain on Black Bear Guai. Aggregate judge-call counts from the Black Bear Guai main experiment, normalized to RaR-Static, are reported below.

Aggregate judge calls in the Black Bear Guai main experiment, normalized to RaR-Static.

MethodRelative judge calls
RaR-Static1.00×
POW3R-Game1.02×
POW3R-Game + Mean1.41×
NoiseRub1.39×

In the Black Bear Guai main experiment, NoiseRub and +Mean use 1.39× and 1.41× the aggregate calls of RaR-Static, indicating comparable aggregate judge usage on this task. Across all four Bosses, NoiseRub achieves higher mean HP reduction under the same repeated-scoring protocol. This supports the contribution of the full noise-calibration and coverage-based weight-update rule beyond score averaging. The added comparisons are included in Appendix A.8.1, Table 7; call counts use the aggregate accounting reported here.

Dear Reviewer xSbi,

Thank you again for your feedback. As the discussion period draws to a close, we would appreciate your thoughts on the two comparisons you suggested. We added RaR-Static, POW3R-Game, and POW3R-Game + Mean on both tasks in Honor of Kings: World. We also extended the comparison with POW3R-Game + Mean to all four Bosses in Black Myth: Wukong, using the same repeated-scoring protocol and score aggregation for both methods.

Please let us know if you have any questions about the experimental setup or results. We would be happy to clarify.

Best regards, The Authors

Reviewer_eh1KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NoiseRub, a noise-calibrated dynamic rubric weighting method for training agents in real-time visual games. The method estimates judge noise through repeated evaluations and combines noise-corrected score variance with rubric applicability coverage to adjust reward weights. Experiments on Black Myth: Wukong show improvements over fixed weighting and POW3R-Game in human preference agreement and online policy performance. The paper also includes component ablations and additional experiments on Honor of Kings: World.

Critical strengths

The paper addresses a practical problem in VLM-based reward modeling. Accounting for judge noise and rubric applicability is a reasonable extension of existing dynamic rubric weighting methods.

The authors evaluate the method through actual online policy training rather than relying solely on offline reward metrics. The component ablations and comparison with POW3R-Game + Mean also help isolate the contributions of the proposed weighting mechanism.

Critical weaknesses
  1. The methodological novelty seems somewhat limited. NoiseRub mainly extends POW3R by introducing judge-noise correction and rubric coverage into the weighting formula. While the idea is reasonable, the paper does not clearly justify why this particular combination is preferable to simpler alternatives.

  2. I am not fully convinced that the proposed calibration signal reflects how useful a rubric is for policy learning. Even after subtracting judge noise, score variance across segments may still be influenced by differences in game states or boss behavior. It is unclear how the method distinguishes these factors from meaningful differences in agent performance.

  3. The offline evaluation is relatively small, with only 115 valid human preference pairs. NoiseRub improves agreement from 60.0% to 63.5% over POW3R-Game, corresponding to just four additional correctly ranked pairs. Given the wide confidence intervals, I find this evidence rather weak.

  4. The method measures judge consistency through repeated evaluations, but consistent judgments are not necessarily correct. The paper also reports frequent changes in rubric applicability across repeated calls. Since applicability directly affects both reward computation and weight calibration, I think this issue deserves more attention.

  5. The experimental validation is still somewhat limited. The cross-game experiments only compare NoiseRub against SFT, making it difficult to judge whether the proposed weighting method generalizes better than existing alternatives. Repeated VLM evaluations also introduce additional costs, but the paper does not clearly evaluate whether the performance gains justify this overhead.

Questions for authors
  1. What motivates the specific form of the weighting function in Eq. (9)? Would simpler approaches to incorporating judge uncertainty and rubric coverage achieve similar results?
  2. How can the authors determine whether the remaining score variance after noise correction reflects useful behavioral differences rather than variations in game states or task difficulty?
  3. The offline improvement over POW3R-Game corresponds to only four additional correct preference comparisons. Can the authors provide a paired statistical analysis to clarify how reliable this improvement is?
  4. How does the method handle cases where the judge consistently assigns incorrect scores or disagrees about whether a rubric is applicable? Could these errors affect the estimated calibration signal?
  5. Why does the cross-game evaluation only include SFT as a baseline? How does NoiseRub compare with POW3R-Game under a matched evaluation budget, considering the additional VLM calls required by repeated auditing?
6 comments

We thank the reviewer for the comments on the weighting function and its validation. We address these points using component ablations, noise controls, expanded offline evaluations, online comparisons, and human verification.

Q1&W1 · Contribution beyond POW3R and the design of Equation (9)

NoiseRub introduces rubric-specific noise calibration and applicability-coverage modulation into POW3R’s variance-driven weighting framework. The motivation is that judge variability contributes to across-segment variance, while few applicable segments provide weaker support for adaptation.

The terms in Equation (9) address these two issues. Repeated evaluations of the same segment estimate judge noise; rubric-specific subtraction reduces its influence on the adaptation signal; nonnegative clipping handles estimates exceeding the observed variance. Coverage modulates the support from evaluation opportunities. Taking the square root retains the standard-deviation scale for the subsequent relative normalization.

We examine this combination using component ablations and simpler alternatives. In the online ablation on Black Bear Guai, the full method achieves 59.87% mean Boss HP reduction. Removing noise calibration or coverage modulation reduces this to 56.71% and 57.44%, respectively, decreases of 3.16 and 2.43 percentage points (Section 4.4).

For simpler alternatives, we reuse the original Black Bear Guai scoring cache and hold sequential replay, valid-score averaging, and human-preference evaluation fixed. The controls shuffle noise estimates, subtract a uniform noise estimate, or match the adaptation magnitude.

Offline noise controls on Black Bear Guai.

MetricNoiseRubShuffled noise (one run)Uniform subtractionMagnitude matching
PAcc63.5%59.3%60.0%61.6%
95% CI48.7%–78.6%50.1%–67.9%50.9%–68.5%52.4%–70.1%

NoiseRub has the highest PAcc point estimate. The offline controls provide additional evidence for rubric-specific calibration, while the online ablations support the role of both noise calibration and coverage modulation in policy training. PAcc uses each method’s valid, non-tied comparisons, and the 95% confidence intervals follow the original round-clustered bootstrap protocol. The new controls are reported in Appendix A.7.2, Table 6.

Q2&W2 · Noise-corrected score differences and policy learning

Equation (9) constructs across-segment score variation calibrated for judge noise and uses it to adapt the relative weights of rubrics during training. The rubrics define the quality of behaviors such as attack timing, dodge responses, and resource use; noise calibration assesses how much scoring variability affects the observed differences.

We repeat evaluations with identical video frames, rubric definitions, prompts, and judge configuration to estimate inconsistency on the same input. Across-segment variance is computed on segments judged applicable on the first pass. Cases lacking an evaluation opportunity or visible evidence are marked N/A and excluded from numerical score statistics. Coverage further reflects how many evaluation opportunities the batch provides.

This procedure distinguishes within-segment scoring variability from across-segment differences, but segments can still differ in Boss phase, resources, and event difficulty. The calibrated residual represents noise-corrected score variation; it is not a pure behavioral effect conditioned on game state.

During training, the signal induces bounded, smoothed adjustments around the human base weights, changing segment rewards, returns, and PPO advantages, and hence each rubric’s influence on policy updates. Online performance evaluates the connection of the complete scheme to the task objective. Q3&W3 reports comparisons with POW3R-Game + Mean on four Bosses under the same repeated-scoring and reward-averaging protocol, evaluating the practical training effects of the different weight-update rules.

Q3&W3 · Paired uncertainty and the expanded evaluation

We substantially strengthen the empirical validation through paired statistical analysis, expanded offline preference evaluation, matched online comparisons across four Bosses, and independent training seeds. These experiments provide a broader assessment of NoiseRub beyond the original Black Bear Guai evaluation.

Paired statistical analysis. On the original 115 valid human-preference comparisons, NoiseRub correctly ranks 73 pairs, compared with 69 for POW3R-Game. Both methods are correct on 49 records, only NoiseRub is correct on 24, only POW3R-Game on 20, and neither on 22. This gives a paired PAcc improvement of 3.48 percentage points.

A bootstrap clustered by the training rounds associated with both endpoints of each comparison yields a 95% confidence interval of [−22.11, +26.32] percentage points. The interval includes zero, so the original offline sample alone does not establish a statistically significant difference. We therefore complement this analysis with substantially expanded evaluations.

Expanded offline preference evaluation. We extend the experiments to White-Clad Noble and Black Wind King, each with 400 ten-second combat segments, including 80 segments evaluated three times and 80 candidate human-preference comparisons. All methods share the scoring, sequential weight-update, and preference-evaluation protocols. Human annotations are used only for evaluation.

MethodBlack Bear GuaiWhite-Clad NobleBlack Wind King
RaR-Equal58.8%59.2%62.2%
RaR-Static57.4%61.8%66.7%
POW3R-Game60.0%68.4%69.2%
POW3R-Game + Mean60.9%69.7%74.4%
NoiseRub63.5%72.4%75.6%

NoiseRub achieves the highest preference agreement on all three Bosses. On the two newly evaluated tasks, it exceeds POW3R-Game + Mean by 2.63 and 1.28 percentage points using unrounded counts. These results extend the observed advantage beyond the original evaluation environment.

The 95% confidence intervals on the two additional Bosses are:

MethodWhite-Clad Noble 95% CIBlack Wind King 95% CI
RaR-Equal[47.5, 69.8][50.8, 72.4]
RaR-Static[50.6, 71.9][55.6, 76.1]
POW3R-Game[57.3, 77.8][58.3, 78.4]
POW3R-Game + Mean[58.7, 78.9][63.7, 82.7]
NoiseRub[61.4, 81.2][65.1, 83.8]

These intervals quantify uncertainty in the individual PAcc estimates; the additional offline differences are not claimed to be statistically significant. Full results are provided in Appendix A.7.1, Table 5.

Expanded online policy evaluation. To determine whether the calibrated weighting rule improves downstream learning beyond score averaging, we extend the POW3R-Game + Mean comparison to all four Bosses. Both methods share the repeated-scoring protocol and use the means of valid scores for reward aggregation.

BossPOW3R-GamePOW3R-Game + MeanNoiseRubGain over +Mean (pp)
White-Clad Noble61.5762.3066.02+3.72
Black Wind King52.4654.4959.85+5.36
Black Bear Guai51.8354.6259.87+5.25
Wandering Wight70.3468.2275.14+6.92

Every method starts from the corresponding SFT policy and uses 30 training rounds with 200 seconds of interaction per round. Final policies are evaluated over 20 episodes, including failures. NoiseRub outperforms +Mean on all four Bosses by 3.72–6.92 percentage points. Since both methods use the same repeated-scoring and valid-score averaging procedures, these results show that averaging alone does not explain the advantage of the calibrated weighting rule. The new comparisons appear in Appendix A.8.1, Table 7.

Independent training seeds provide further evidence of consistent online improvements. On Black Bear Guai, three independent training runs per method yield mean HP reductions of 58.43% for NoiseRub and 52.00% for POW3R-Game, a gain of 6.43 percentage points. NoiseRub achieves higher HP reduction under all three seeds (Appendix A.9, Table 9).

Taken together, the expanded results provide a substantially stronger empirical case for NoiseRub. While the original offline paired comparison has substantial uncertainty, the extended preference evaluations consistently favor NoiseRub, the matched online comparisons demonstrate improvements across four Bosses, and independent training seeds support the reproducibility of the online advantage over POW3R-Game. These complementary results strengthen the evidence for noise-calibrated, coverage-aware weight adaptation without relying solely on the original offline comparison.

Q4&W4 · Judge correctness and changes in applicability

NoiseRub specifically addresses stochastic inconsistency in rubric-based evaluation, rather than assuming that repeated agreement guarantees judge correctness. This distinction is central to our formulation: repeated evaluations estimate variability on identical inputs, while applicability coverage measures the support available for adapting each rubric's weight. We conduct additional human verification to examine both scoring correctness and the reliability of repeated judgments.

Human verification of judge scores and applicability. Figure 3(a) evaluates applicability using 257 human annotations, including 53 in the test split. Additional human verification includes 93 records with three evaluations, while applicability switching is measured over all 847 audited segment–rubric combinations.

Verification itemResult
Applicability F1, all annotations (Figure 3(a))94.20% (257 annotations)
Applicability F1, test split (Figure 3(a))95.12% (53 annotations)
Exact score agreement when both judge and human mark applicable142/188 (75.5%)
Mean absolute score error on these 188 records0.298
Human disagreements among records with three identical numerical scores2/20 (10.0%)
Human-applicable cases among records with three N/A judgments2/14 (14.3%)
Applicability switches across repeated evaluations163/847 (19.2%)

These results distinguish two sources of judge error that should not be conflated.

Stochastic inconsistency. Repeated evaluations of identical video segments can produce different numerical scores or applicability decisions. In our audited data, 163 of 847 segment–rubric combinations (19.2%) exhibit applicability switching. Such variability affects the set of valid evaluations and the evidence available for weight adaptation. NoiseRub explicitly accounts for these effects: N/A outputs are excluded rather than treated as zero, within-segment variance is estimated from at least two valid scores, and first-pass evaluations provide a consistent basis for computing coverage and across-segment variance.

Systematic errors. A judge may repeatedly assign the same incorrect score, producing zero within-segment variance despite disagreement with human annotations. Our verification identifies two disagreements among 20 records with three identical numerical scores. Similarly, two of 14 records receiving three N/A decisions are marked applicable by human annotators. These results confirm why scoring consistency and scoring correctness must be treated separately. NoiseRub does not claim to eliminate systematic judge errors; its contribution is to prevent observable scoring inconsistency from being mistaken for informative variance during rubric-weight adaptation.

The method also explicitly controls weight adaptation when evaluation support is insufficient. A rubric must have at least 5% first-pass coverage, four eligible audited segments, and an estimable across-segment variance to participate in calibration-based weight updates. Otherwise, its target multiplier is set to one and the current multiplier moves toward the human-assigned base weight through EMA smoothing. Available noise estimates can still update the noise EMA when fewer than four eligible audits are present, but the weight update follows the fallback rule. All multipliers remain bounded within 0.67–1.50.

The practical effectiveness of this design is supported by the online experiments despite imperfect VLM judgments. On Black Bear Guai, removing noise calibration or coverage modulation decreases mean HP reduction by 3.16 and 2.43 percentage points, respectively. Furthermore, under matched repeated-scoring and valid-score averaging protocols, NoiseRub outperforms POW3R-Game + Mean across all four evaluated Bosses (Appendix A.8.1, Table 7).

These findings clarify both the scope and effectiveness of NoiseRub. The human verification characterizes errors that repeated-scoring consistency cannot detect, while the online ablations and matched comparisons show that explicitly calibrating observable judge variability and opportunity coverage improves policy learning under the evaluated settings. Thus, NoiseRub does not require a perfectly accurate judge to provide useful improvements in rubric-based training.

Reviewer_Xg4FOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces NoiseRub, a method for dynamically weighting rubric-based rewards in screen-only visual games. It estimates judge variability through repeated evaluations, subtracts this estimate from across-segment score variance, and modulates the residual by rubric applicability coverage. The resulting signal adjusts human-defined base weights. Experiments report improved human preference agreement and higher mean Boss HP reduction than the evaluated baselines across four encounters in Black Myth: Wukong.

Critical strengths
  • The paper addresses a concrete issue: observed rubric variance may reflect judge inconsistency rather than useful behavioral differences.

  • The implementation explicitly handles non-applicable criteria, limited audit support, and bounded weight adaptation. Detailed prompts and rubric definitions are provided.

  • Evaluation includes both offline reward rankings and online policy performance. The repeated-score averaging baseline and component ablations provide useful controls.

  • The full method achieves higher HP reduction than POW3R-Game on all four primary tasks under matched interaction budgets.

Critical weaknesses
  • The substantive methodological increment is limited. Rubric rewards, variance-driven reweighting, human weight priors, clipping, and smoothing are inherited from existing approaches, particularly POW3R. The principal addition is the combination of repeated-audit noise subtraction and coverage modulation. This is a targeted extension, but the paper does not yet establish when this particular calibration rule is preferable to simpler alternatives.

  • Calibrated score variation is not established as a measure of training utility. Even with accurate noise removal, between-segment variation can reflect Boss phases, encounter difficulty, or resource availability rather than policy-controllable behavior. Reliable score differences also need not imply that increasing a criterion’s weight improves the final task objective. The variance decomposition motivates noise correction but does not establish this subsequent connection.

  • The empirical evidence does not isolate the proposed mechanism. Component ablations support the usefulness of the implemented components in the reported run, but do not distinguish accurate noise calibration from other effects of changing the weight distribution. Controls using shuffled noise estimates, uniform subtraction, or matched adaptation magnitudes would strengthen the attribution.

  • Uncertainty and generality remain insufficiently characterized. Each method–Boss combination uses one training run, so 20 evaluation episodes do not establish robustness across training seeds. Offline preference improvements are modest and accompanied by broad confidence intervals. The second-game evaluation compares only against SFT and therefore does not establish cross-game benefits of calibration over alternative reward schemes.

  • The noise estimate requires further validation. Across-segment variance and within-segment noise use differently selected applicability sets. Noise is estimated from only two or three valid evaluations per audited segment, using a finite-sample downward-biased variance estimator under standard sampling assumptions. The paper appropriately avoids claiming unbiased recovery, but does not quantify estimation accuracy or sensitivity. Total judge cost is also not matched by interaction-budget comparisons.

Questions for authors
  • Does the calibrated signal predict the benefit of increasing a rubric’s weight, after controlling for encounter phase or event difficulty?

  • Does using correctly matched rubric-specific noise estimates outperform shuffled estimates, uniform subtraction, and a baseline with matched adaptation magnitude?

  • How stable are the noise estimates with more repeated evaluations, and how sensitive are results to applicability-set selection and variance estimation?

  • Can the second-game evaluation include static rubric rewards and POW3R-Game with repeated-score averaging? Please also clarify the deviations from original POW3R and report judge-call and wall-clock costs.

9 comments

We thank the reviewer for the comments on the methodological contribution, the role of the calibration signal, and evaluation reliability. Below, we clarify the motivation and the connection between weights and policy updates, and report noise controls, statistical sensitivity, cross-game comparisons, and results across training seeds.

Q1&W2 · What does the calibrated signal tell us about rubric weighting?

NoiseRub improves rubric-weight adaptation by distinguishing reliable score variation from judge inconsistency and insufficient evaluation opportunities. We follow POW3R's principle that score variation can identify rubrics with useful discriminatory information. For example, a policy may dodge ordinary attacks reliably but repeatedly fail during combos. When the same dodge rubric applies, the score differences can reveal variation in behavioral performance. However, raw variance alone cannot distinguish these differences from inconsistent VLM judgments. NoiseRub addresses this by estimating within-segment judge noise through repeated evaluations and incorporating applicability coverage into the adaptation signal.

The calibrated signal directly influences policy learning through the resulting rubric weights. To make this connection explicit, consider Equations (3), (13), and (15). Holding rubric scores, applicability indicators, and sampled actions fixed, let JℓPPOJ_\ell^{\mathrm{PPO}} denote the per-token maximization objective. At the beginning of a PPO update, where the probability ratio equals one and clipping is inactive, its gradient is:

\qquad \left.\nabla_\theta J_\ell^{\mathrm{PPO}}\right|_{\theta_{\mathrm{old}}} =G_{t(\ell),k} \left.\nabla_\theta\log\pi_\theta(y_\ell\mid x_\ell)\right|_{\theta_{\mathrm{old}}}.$$ The segment reward in Equation (3) directly determines the discounted PPO advantage. Increasing an applicable rubric's weight moves the segment reward toward its score, thereby changing the sampled action's contribution to the policy update. Figure 4 illustrates the practical effect of calibration on weight allocation. On Black Bear Guai, POW3R-Game reduces the multiplier of the sparsely applicable transformation rubric (R6) to 0.828, whereas NoiseRub retains its base weight. For the fire-hazard rubric (R8), which exhibits substantial repeated-scoring noise, NoiseRub limits the upward adjustment to 1.022, compared with 1.130 under POW3R-Game. These examples show how calibration changes weight adaptation when raw variance is supported by limited opportunities or affected by judge inconsistency. The downstream improvements confirm the practical value of the complete calibrated weighting rule. Under the same repeated-scoring protocol, valid-score averaging, and interaction budget, NoiseRub achieves 59.87% mean Boss HP reduction on Black Bear Guai, compared with 54.62% for POW3R-Game + Mean, a gain of 5.25 percentage points (Section 4.4). The corresponding gains on White-Clad Noble, Black Wind King, and Wandering Wight are 3.72, 5.36, and 6.92 percentage points, respectively (Appendix A.8.1, Table 7). These consistent improvements demonstrate that the advantage of NoiseRub is not explained by repeated-score averaging alone. Together with the observed weight dynamics, they establish that calibrating the evidence used for weight adaptation provides a more effective training signal under the evaluated protocols. The comparisons assess the complete update rule; isolating the causal effect of an individual rubric weight would require matched Boss phases and event difficulty.

Q2&W1&W3 · Methodological contribution and noise-calibration controls

NoiseRub addresses two limitations of variance-driven rubric weighting: judge inconsistency and insufficient opportunity support. POW3R uses score variation to adapt criterion weights, but in real-time visual games, high variance does not necessarily imply informative behavioral differences. It may reflect inconsistent VLM judgments, while sparsely applicable rubrics may exhibit variation supported by very few observations. NoiseRub retains POW3R's human weight priors and bounded adaptation framework.

Figure 1 illustrates this distinction. R7 and R8 exhibit similar across-segment score variances, but their repeated-scoring noise-to-variance ratios differ substantially: 8.5% versus 69.6%. Raw variance cannot distinguish these cases. Similarly, R6 applies to only 3.26% of segments, providing limited support for adapting its weight. Equation (9) therefore subtracts each rubric's smoothed noise estimate, clips negative residuals to zero, and incorporates applicability coverage. The adaptation signal thus incorporates reliable variation and opportunity support beyond raw score variance.

Online component ablations directly validate both mechanisms' contributions. All variants share the SFT initialization, interaction budget, repeated-audit sampling, valid-score averaging, and policy optimization settings.

MethodMean Boss HP reduction
POW3R-Game + Mean54.62%
NoiseRub w/o noise calibration56.71%
NoiseRub w/o coverage modulation57.44%
Full NoiseRub59.87%

Removing noise calibration reduces performance by 3.16 percentage points, while removing coverage modulation reduces it by 2.43 points. The full method also outperforms POW3R-Game + Mean by 5.25 points. With repeated-scoring and reward-aggregation protocols matched, these comparisons directly support the calibrated weight-update rule beyond score averaging.

We further test whether simpler noise-correction strategies reproduce this advantage. Using the original Black Bear Guai scoring cache, we compare NoiseRub against shuffled-noise, uniform-subtraction, and magnitude-matched controls. All methods use the same 430 ten-second combat segments, 11 rubrics, and judge evaluations, including 77 segments with three evaluations each. Sequential replay, update thresholds, valid-score averaging, and human-preference evaluation follow the original protocol. Human labels are used only for evaluation.

Following Equation (9), let Bj,kB_{j,k} denote first-pass across-segment variance, cj,kc_{j,k} applicability coverage, and uj,ku_{j,k} the subtracted noise term:

\qquad u_{j,k}=\widehat W_{j,k}\quad\text{(NoiseRub)}.$$ The three alternatives are: - **Shuffled noise:** Randomly permute smoothed noise estimates among eligible rubrics after the noise-EMA update, using $u_{j,k}=\widehat W_{\pi_k(j),k}$. - **Uniform subtraction:** Replace rubric-specific estimates with their arithmetic mean, $u_{j,k}=\frac{1}{|C_k|}\sum_{\ell\in C_k}\widehat W_{\ell,k}$. - **Magnitude matching:** Remove noise subtraction, compute the uncorrected-variance weight update, and rescale candidate multiplier deviations from one to match NoiseRub's root-mean-square adaptation magnitude as closely as possible, subject to the same multiplier bounds of 0.67–1.50. Matching uses no human-preference labels. The offline preference-ranking results are: | Metric | NoiseRub | Shuffled noise (one run) | Uniform subtraction | Magnitude matching | | --- | ---: | ---: | ---: | ---: | | PAcc | **63.5%** | 59.3% | 60.0% | 61.6% | | 95% CI | [48.7, 78.6] | [50.1, 67.9] | [50.9, 68.5] | [52.4, 70.1] | The correct/valid comparison counts are 73/115, 67/113, 69/115, and 69/112, respectively. Confidence intervals follow the original round-clustered bootstrap protocol, and shuffled noise is reported for one permutation run. **NoiseRub achieves the highest offline preference agreement among all tested calibration strategies**, exceeding shuffled noise, uniform subtraction, and magnitude matching by 4.2, 3.5, and 1.9 percentage points, respectively. Although these offline comparisons have statistical uncertainty, their results consistently favor rubric-specific noise correction. More importantly, the matched online component ablations demonstrate that both noise calibration and coverage modulation contribute to improved downstream policy performance. These experiments demonstrate NoiseRub's practical contribution: calibrating score variation using judge reliability and opportunity support. Additional offline controls appear in Appendix A.7.2, Table 6; online component ablations appear in Section 4.4.

Q3&W5 · Sensitivity to variance estimation and applicability sets

Under the existing repeated-scoring budget, we varied the within-segment variance estimator and the applicability sets used to compute the two variances. The two changes altered 1 and 2 reward rankings, respectively, out of 115 human preference comparisons on Black Bear Guai. We reused the original 430 combat segments, including 77 evaluated three times each. Both comparisons held the scoring records, replay order, and human preference comparisons fixed and varied one statistical choice at a time.

Offline settingChanged rankingsMaximum absolute change in an effective weight multiplier
Sample variance within segments1 / 1150.0497
Common applicability set for both variances2 / 1150.0563

The first comparison changes the within-segment variance denominator from the number of valid scores to that number minus one, testing sensitivity to the choice of denominator at small sample sizes. The second computes across-segment variance and within-segment noise variance on the same segments. Each of these segments was judged applicable on the first pass and had at least two valid scores. Coverage calculation, update thresholds, and fallback rules were held fixed.

Under the existing repeated-scoring budget, these two choices had limited effects on reward rankings on the fixed offline records. Assessing the stability of the noise estimates with additional evaluations requires comparing estimates from different numbers of evaluations on the same segments. Judge-call overhead is reported in Q4&W4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.