- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes NOISERUB, a dynamic rubric reward weighting method for real-time visual games. It uses repeated VLM evaluations to estimate judge noise and combines this with rubric coverage to calibrate weight updates. The method is evaluated on four Boss tasks in Black Myth: Wukong.
Critical strengths
- The paper addresses a clear and practically relevant problem: rubric score variance may reflect judge instability rather than useful behavioral differences. NOISERUB explicitly accounts for both judge noise and opportunity coverage when adapting rubric weights.
- The ablation study is well designed. Comparisons with w/o Noise Calibration, w/o Coverage, and POW3R-Game + Mean help show that the gains come from the proposed weighting mechanism rather than repeated-score averaging alone.
Critical weaknesses
- The cross-game evaluation is limited. In Honor of Kings: World, NOISERUB is only compared with SFT, without stronger rubric-based baselines such as POW3R-Game or RaR-Static. This provides limited evidence for cross-game generalization.
- The compute/audit-matched comparison is only conducted on Black Bear Guai. Since NOISERUB requires repeated VLM audits, it would be more convincing to include POW3R-Game + Mean on the other Boss tasks as well, to verify that the advantage remains under a matched repeated-evaluation protocol.
Questions for authors
- Why does the Honor of Kings: World evaluation only compare NOISERUB with SFT? Do you have results against stronger rubric-based baselines such as POW3R-Game or RaR-Static?
- Is the compute/audit-matched comparison with POW3R-Game + Mean available for the other Boss tasks? This would help clarify whether the gains come from NOISERUB itself rather than repeated VLM evaluation.
2 comments
Dear Reviewer xSbi,
Thank you again for your feedback. As the discussion period draws to a close, we would appreciate your thoughts on the two comparisons you suggested. We added RaR-Static, POW3R-Game, and POW3R-Game + Mean on both tasks in Honor of Kings: World. We also extended the comparison with POW3R-Game + Mean to all four Bosses in Black Myth: Wukong, using the same repeated-scoring protocol and score aggregation for both methods.
Please let us know if you have any questions about the experimental setup or results. We would be happy to clarify.
Best regards, The Authors
We thank the reviewer for identifying the need for stronger cross-game baselines and repeated-scoring comparisons on the other Bosses. We have completed both sets of comparisons and report their settings and results below.
Q1&W1 · Stronger rubric-reward baselines in the second game
We have added RaR-Static, POW3R-Game, and POW3R-Game + Mean on both tasks in Honor of Kings: World. The submitted SFT comparison measures improvement over the initial policy; the added baselines compare fixed weights, variance-driven adaptation, and repeated-score averaging. All methods share the same character configuration and 20-episode evaluation protocol. All post-training methods start from the same SFT policy and use a 30-round interaction budget; SFT is evaluated without additional post-training. HP reduction includes failures, and Yuange Puppet evaluation covers Phase 3 only.
Mean Boss HP reduction in Honor of Kings: World.
NoiseRub achieves the highest mean HP reduction on both tasks. Its gains over POW3R-Game are 5.12 and 7.42 percentage points, and its gains over POW3R-Game + Mean are 7.23 and 3.75 points. The added comparisons support NoiseRub relative to alternative rubric-reward schemes in a second game. The complete setup and results are included in Appendix A.8.2, Table 8, of the revised manuscript.
Q2&W2 · Repeated-scoring comparisons across all four Bosses
We have extended the POW3R-Game + Mean comparison to White-Clad Noble, Black Wind King, and Wandering Wight. Together with the submitted Black Bear Guai result, the comparison now covers all four Boss tasks in Black Myth: Wukong.
Both methods use the repeated-scoring protocol in Appendix A.5: each segment receives one evaluation and is selected with probability 0.2 for two additional evaluations. Rewards aggregate the available valid-score means for each rubric. The methods share repeated-audit sampling and valid-score averaging but differ in weight adaptation. POW3R-Game + Mean uses first-pass score variance with its original eligibility rules, whereas NoiseRub combines noise-corrected variance with applicability coverage and audit-support requirements. The comparison therefore evaluates the complete calibrated weight-update rule beyond repeated-score averaging.
Each run uses 30 rounds with 200 seconds of interaction per round, and the final policy is evaluated on 20 episodes, including failures. The table reports mean Boss HP reduction and NoiseRub’s gain over +Mean in percentage points.
Mean Boss HP reduction (%) across four Bosses. The last column gives the gain over +Mean in percentage points.
NoiseRub improves over +Mean by 3.72, 5.36, and 6.92 percentage points on the three added tasks, alongside the 5.25-point gain on Black Bear Guai. Aggregate judge-call counts from the Black Bear Guai main experiment, normalized to RaR-Static, are reported below.
Aggregate judge calls in the Black Bear Guai main experiment, normalized to RaR-Static.
In the Black Bear Guai main experiment, NoiseRub and +Mean use 1.39× and 1.41× the aggregate calls of RaR-Static, indicating comparable aggregate judge usage on this task. Across all four Bosses, NoiseRub achieves higher mean HP reduction under the same repeated-scoring protocol. This supports the contribution of the full noise-calibration and coverage-based weight-update rule beyond score averaging. The added comparisons are included in Appendix A.8.1, Table 7; call counts use the aggregate accounting reported here.