acceptodds
Under review as a conference paper at ICLR 2027

RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models

Abstract

Large language model alignment via reinforcement learning depends critically on reward function quality. However, generic reward models often underperform on heterogeneous task distributions due to distribution shifts, while training task-specific reward models is costly and prone to annotation difficulty, catastrophic forgetting, and loss of generalization. We present RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries. Specifically, RLAR transforms reward acquisition into a dynamic tool synthesis and invocation task. It leverages LLM agents to autonomously retrieve optimal reward models from the Internet and synthesize code-based verifiers through code generation. This allows the reward system to self-evolve with the shifting data distributions during training. On RewardBench-V2, RLAR outperforms static baselines and approaches the performance upper bound, validating the dynamic reward orchestration mechanism. In online RL post-training, RLAR yields consistent gains of 10%–60% across mathematics, coding, translation, and dialogue. Ablations confirm that synthesized code-based verifiers complement neural reward models on rule-heavy tasks, and that precise per-query reward routing outperforms monolithic reward prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reward design, large language model, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GSfqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RLAR, a reward system in which an LLM agent treats reward functions as callable tools. For each training query, a router decides whether the current reward library already contains a suitable tool. If it does, a selector picks one. If it does not, one of two synthesis agents builds a new tool. The main experiments train a Llama-3.1-8B and Qwen3-8B on a mixture of mathematics, coding, translation, and dialogue. RLAR is compared with three single classifier reward models, a generative critic, GPT-5 as a judge, and a hand-designed per-domain reward. The paper reports the best overall average for both models, relative gains of about ten and sixty percent over the untrained models, robustness to format and verbosity hacking, about four fifths fewer API tokens and three quarters fewer GPU hours than GPT-5 as a judge, and ablations that remove CodeVerify, WrapLLM or the selector.

Critical strengths
  1. The problem is practical and the idea is appealing. Many specialised reward models are publicly available, and letting an agent pick or build an appropriate reward per query, instead of training yet another general reward model, is a reasonable way to handle heterogeneous RL mixtures.
  2. The RewardBench-V2 experiment is a useful design choice because it isolates the selection step from RL. It includes an oracle upper bound and a logit-averaging baseline, and it shows clearly that naive averaging over more reward models degrades accuracy while selection does not.
  3. Some gains are large on test sets where they are clearly beyond noise. On the Llama model, RLAR solves nearly sixty more Hendrycks MATH problems out of five hundred than the GPT-5 judge baseline. On the Qwen3 model, it solves roughly fifty LeetCode problems against about twenty for the hand-designed reward. The prompts for every agent are given in full, and the limitations section openly discusses the risk of misleading model cards.
Critical weaknesses
  1. The positioning omits or understates the closest prior work, and the novelty is narrower than claimed. RewardAgent [1] already combines a router that decides which verification agents to call, LLM-generated Python checkers for instruction constraints, and a base preference reward model. The paper distinguishes itself mainly by using GRPO rather than DPO, which is not a methodological difference in the reward system. LASeR [2] selects among several reward models per instance during iterative LLM training and reports gains over reward ensembles on math, reasoning and instruction following. This is the same problem of choosing a suitable reward model per query in a heterogeneous mixture. Eureka [3] and Text2Reward [4], already established LLM-written reward code for RL, which contradicts the conclusion that RLAR establishes the paradigm of the LLM as reward designer. Tool-augmented reward models such as Themis [5] are also relevant. The genuinely new elements appear to be automatic retrieval and wrapping of public reward-model checkpoints and the growing library inside multi-task GRPO. The paper should state these explicitly and compare against at least RewardAgent-style routing or LASeR-style selection.
  2. Reproducibility is limited. The system depends on an unversioned proprietary model, on live Google search results and on the current state of Hugging Face repositories, so the library cannot be rebuilt as it was. The final library, the routing logs and the synthesized verifiers would be needed to reproduce the reward signal, and the paper does not say that they are released.

Reference: [1]: Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems, https://arxiv.org/abs/2502.19328 [2]: LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits, https://arxiv.org/abs/2410.01735 [3]: Eureka: Human-Level Reward Design via Coding Large Language Models, https://arxiv.org/abs/2310.12931 [4]: Text2Reward: Automated Dense Reward Function Generation for Reinforcement Learning, https://arxiv.org/abs/2309.11489 [5]: Tool-Augmented Reward Modeling, https://arxiv.org/abs/2310.01045

Questions for authors
  1. Which tools actually rewarded the training queries in the reported main runs? Please give, for each training source, the share of rollouts scored by each tool and tool type, and say how many tools were synthesized during these runs and at which training steps. Was the appendix usage analysis produced from these runs or from a different experiment?

  2. How accurate are the synthesized verifiers? On a sample of GSM8K, Hendrycks MATH and LeetCode rollouts, how often does the selected tool agree with exact numeric match against the gold answer or with the dataset's unit tests?

  3. What evaluation protocol was used for decoding, thinking mode, maximum generation length and answer extraction? Why is the untrained Qwen3 model so far below published GSM8K and MATH accuracy? Would a format-agnostic answer extractor change the ranking on the math columns? Which Llama checkpoint was used, given that the citation points to a security-tuned derivative?

Reviewer_a5HjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RLAR treats reward construction as tool retrieval and synthesis: an agent selects existing neural reward models or builds code-based verifiers, then routes queries to the resulting tools. The paper evaluates reward selection on RewardBench-v2 and online GRPO training across math, coding, translation, and dialogue using two policy backbones.

Critical strengths
  • Combining reusable reward models with generated verifiers addresses a practical limitation of a single reward model for heterogeneous tasks.
  • Tables 3–4 include held-out-task evaluations and multiple reward baselines, while §4.5 ablates tool synthesis and selection. The reported API-token and training-time comparisons also address deployment cost.
Critical weaknesses
  • The routing ablation compares against random tool selection. The Human baseline uses a separately specified reward system, and §4.3 attributes some RLAR gains to additional tests and formatting checks. A fixed task-aware router using the same tool library is needed to distinguish routing benefits from tool-quality differences.
  • Tool validation primarily establishes construction validity. Appendix F.2's 94.9% executable rate checks a minimal synthetic input and numeric return, rather than agreement with correctness labels. A verifier can run successfully yet reward wrong answers, leaving a central reliability claim insufficiently validated.
Questions for authors
  1. Is there an existing same-library comparison with deterministic task-aware routing, and how much do added tests/checks contribute relative to the Human baseline?
  2. Beyond executability, what validation checks synthesized tools against labeled correct and incorrect responses, including false accepts?
Reviewer_fdetOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RLAR, a reward framework for multi-domain GRPO post-training that treats reward functions as callable tools. The core idea is to use a GPT-5-based router to select appropriate reward tools from an evolving tool library or synthesize new ones through WrapLLM and CodeVerify when needed. The authors evaluate RLAR on RewardBench-v2 and report promising results. They further demonstrate that RLAR achieves better average performance than the baselines in GRPO post-training experiments with Llama and Qwen models.

Critical strengths
  1. The method is easy to understand, with a simple intuition behind dynamically selecting task-appropriate reward functions.

  2. The authors separately evaluate the effectiveness of the reward routing mechanism and assess its impact on downstream model performance through GRPO post-training experiments.

Critical weaknesses
  1. The overall idea is not particularly novel. Using an LLM-based agent to select reward tools and decide when to synthesize new ones is a relatively straightforward approach, without introducing substantial technical innovations.

  2. The paper lacks a thorough evaluation of the quality and reliability of automatically generated reward tools.

  3. The training dynamics figures in the appendix appear to be based on Qwen3-0.6B, whereas the main experiments are conducted on Qwen3-8B.

  4. The method shows only modest gains over strong baselines on many benchmarks and even degrades performance on some tasks.

  5. The paper does not compare against simple task-based routing or fixed tool-selection strategies, making it unclear whether GPT-5-based dynamic routing is truly necessary.

Questions for authors
  1. Could the authors provide a more thorough evaluation of the quality and reliability of the automatically generated reward tools?

  2. Could the authors include stronger baselines, such as rule-based task routing and LASeR-style selection, and report results across multiple random seeds with statistical significance tests?

  3. Could the authors further disentangle the contributions of tool selection and tool synthesis, particularly by comparing a fixed tool library with a dynamically expanded one under the same routing strategy?

  4. Could the authors disentangle the contributions of routing and tool synthesis by comparing fixed tool selection, GPT-5 routing, and dynamic tool synthesis under otherwise identical settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.