- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RLAR, a reward system in which an LLM agent treats reward functions as callable tools. For each training query, a router decides whether the current reward library already contains a suitable tool. If it does, a selector picks one. If it does not, one of two synthesis agents builds a new tool. The main experiments train a Llama-3.1-8B and Qwen3-8B on a mixture of mathematics, coding, translation, and dialogue. RLAR is compared with three single classifier reward models, a generative critic, GPT-5 as a judge, and a hand-designed per-domain reward. The paper reports the best overall average for both models, relative gains of about ten and sixty percent over the untrained models, robustness to format and verbosity hacking, about four fifths fewer API tokens and three quarters fewer GPU hours than GPT-5 as a judge, and ablations that remove CodeVerify, WrapLLM or the selector.
Critical strengths
- The problem is practical and the idea is appealing. Many specialised reward models are publicly available, and letting an agent pick or build an appropriate reward per query, instead of training yet another general reward model, is a reasonable way to handle heterogeneous RL mixtures.
- The RewardBench-V2 experiment is a useful design choice because it isolates the selection step from RL. It includes an oracle upper bound and a logit-averaging baseline, and it shows clearly that naive averaging over more reward models degrades accuracy while selection does not.
- Some gains are large on test sets where they are clearly beyond noise. On the Llama model, RLAR solves nearly sixty more Hendrycks MATH problems out of five hundred than the GPT-5 judge baseline. On the Qwen3 model, it solves roughly fifty LeetCode problems against about twenty for the hand-designed reward. The prompts for every agent are given in full, and the limitations section openly discusses the risk of misleading model cards.
Critical weaknesses
- The positioning omits or understates the closest prior work, and the novelty is narrower than claimed. RewardAgent [1] already combines a router that decides which verification agents to call, LLM-generated Python checkers for instruction constraints, and a base preference reward model. The paper distinguishes itself mainly by using GRPO rather than DPO, which is not a methodological difference in the reward system. LASeR [2] selects among several reward models per instance during iterative LLM training and reports gains over reward ensembles on math, reasoning and instruction following. This is the same problem of choosing a suitable reward model per query in a heterogeneous mixture. Eureka [3] and Text2Reward [4], already established LLM-written reward code for RL, which contradicts the conclusion that RLAR establishes the paradigm of the LLM as reward designer. Tool-augmented reward models such as Themis [5] are also relevant. The genuinely new elements appear to be automatic retrieval and wrapping of public reward-model checkpoints and the growing library inside multi-task GRPO. The paper should state these explicitly and compare against at least RewardAgent-style routing or LASeR-style selection.
- Reproducibility is limited. The system depends on an unversioned proprietary model, on live Google search results and on the current state of Hugging Face repositories, so the library cannot be rebuilt as it was. The final library, the routing logs and the synthesized verifiers would be needed to reproduce the reward signal, and the paper does not say that they are released.
Reference: [1]: Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems, https://arxiv.org/abs/2502.19328 [2]: LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits, https://arxiv.org/abs/2410.01735 [3]: Eureka: Human-Level Reward Design via Coding Large Language Models, https://arxiv.org/abs/2310.12931 [4]: Text2Reward: Automated Dense Reward Function Generation for Reinforcement Learning, https://arxiv.org/abs/2309.11489 [5]: Tool-Augmented Reward Modeling, https://arxiv.org/abs/2310.01045
Questions for authors
-
Which tools actually rewarded the training queries in the reported main runs? Please give, for each training source, the share of rollouts scored by each tool and tool type, and say how many tools were synthesized during these runs and at which training steps. Was the appendix usage analysis produced from these runs or from a different experiment?
-
How accurate are the synthesized verifiers? On a sample of GSM8K, Hendrycks MATH and LeetCode rollouts, how often does the selected tool agree with exact numeric match against the gold answer or with the dataset's unit tests?
-
What evaluation protocol was used for decoding, thinking mode, maximum generation length and answer extraction? Why is the untrained Qwen3 model so far below published GSM8K and MATH accuracy? Would a format-agnostic answer extractor change the ranking on the math columns? Which Llama checkpoint was used, given that the citation points to a security-tuned derivative?