acceptodds
Under review as a conference paper at ICLR 2027

AReward: Synthesizing Real-World Environments with Executable State Rubrics

Abstract

Language agents learn through interaction with stateful environments, but sparse outcome rewards provide limited feedback on intermediate progress. Rubric-based rewards offer finer-grained evaluation, yet their criteria must reflect user requirements and be checked reliably against execution. We introduce AReward, which jointly synthesizes transactional environments, tasks, and executable state rubrics through a shared formulation of create, read, update, and delete (CRUD). Tool execution determines state changes, and task requirements specify the conditions that count as completion. Reference solution replay verifies task reachability, while reward auditing examines the consistency of acceptance criteria with user requirements. Evaluating these rubrics after each action exposes progress and reversals in rubric satisfaction. STG-GRPO, our state-transition graph variant of Group Relative Policy Optimization, uses this evidence and progress along the reference state chain to refine trajectory comparisons. Combined with supervised initialization, reinforcement learning with a staged curriculum, and long-tail task management, AReward provides a training framework in which reward criteria are tied to user requirements and their evaluation is grounded in actual execution. Code is available at https://anonymous.4open.science/r/stg-anonymous-248B/

Primary area:
reinforcement learning
Keywords:
Tool Agents,LLM,Environment scaling

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_5LdbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AReward and STG-GRPO that are environment-generation and post-training techniques respectively to improve agent's performance on state-based tasks that require CRUD operations (create, read, update, delete). The AReward framework generates tasks and executable state rubrics. STG-GRPO provides additional anchored advantages that are based on a common anchor state during a GRPO run to ensure more fine-grained advantage. Each step's reward combines progress along the reference state chain, change in rubric satisfaction, execution validity and outcome.

Critical strengths

Representations of task and state is unified under the database-state representation. The paper formalizes the state changes and rewards based on the basic CRUD operations.

Critical weaknesses

W1. RL results are similar or worse than SFT and not coherent.

• L077-079 in the introduction report 54.5% for SFT on VitaBench compared to 53.8% for RL.

• The reported higher rubric satisfaction criteria (L086-087, L437-435) compare a different RL checkpoint (RL-50) from the main Table 1. Even in that comparison, RL leads to a slight improvement compared to SFT with no improvement in task success rate.

W2. Weak experimental rigour.

• [W2.1] The paper does not have any baseline other than SFT. It does not even compare to vanilla GRPO with just the final task completion binary reward.

• [W2.2] There are no ablations or sensitivity analysis to show the importance of the different reward terms in Equation 7. Given that the paper is introducing these additional terms, there should be ablations and hyper-param search to justify the choices and their effects.

W3. Effect of STG-GRPO not conclusive. In addition to the missing ablation discussed in W2.1, there is one more issue with STG-GRPO advantage term introduced in Equation 9. This term requires anchors that are common states. Given that the training was done using only 4 rollouts, finding common anchors across different rollouts becomes difficult. The authors should report some statistics regarding when does this term get activated.

W4. Final advantage is not step level Although the additional STG-GRPO term is computed across common anchor steps, but the advantage in Equation 9 is distributed to the entire roll-out. This makes the signal average out over the trajectory.

W5. Manuscript issues

• The model name or experiment names are present in the paper (e.g. SFT checkpoint 402 and RL checkpoint stage10-59).

• Table 3 lists 7 stages but L719 states that there are four stages.

• Last para of page 15 and starting para of page 16 has some garbled text.

Questions for authors

I would request the authors to respond to their design choices regarding the use of STG-GRPO, such as choice of hyper-parameters, ablating the reward components, etc. The authors should also respond to the other weaknesses stated in the weakness section above.

Reviewer_7mGMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents AReward, a pipeline that synthesizes transactional tool-use environments. Each task in that environment must pass three checks, including a replay of the reference solution, a gate requiring every core criterion to be false in the initial state, and a reward audit. A Qwen3.6 model is trained with SFT on verified teacher trajectories. It is then trained with STG-GRPO, which rewards each call for progress along the reference state chain and in satisfied criteria, and adds a step-level advantage standardized among steps that reach the same post-action anchor. On VitaBenc and a suited τ-bench, which combines the Airline, Retail and Telecom domains of τ²-bench with the Banking domain of τ-Knowledge, SFT improves over the base model by about nine and a half and five points respectively. RL ends slightly below SFT on VitaBench and slightly above it on this suite.

Critical strengths
  1. Expressing environment dynamics and acceptance criteria over one shared relational state is clean and practical. Several safeguards address known failure modes of rubric rewards: replaying the reference solution to confirm that the task can be completed, gating out criteria that already hold in the initial state, and auditing for checks that fix one of several valid solutions or demand exact matches on agent-written text.

  2. The SFT stage brings a clear improvement on VitaBench. It solves thirty-eight more of the four hundred tasks, roughly two and a half to three binomial standard errors, gains in every domain, and solves ten more banking tasks. This suggests that verified trajectories from synthesized environments can transfer to an external benchmark, subject to the overlap question raised below.

Critical weaknesses
  1. Closely related peer-reviewed work is missing. Environment Tuning [1] combines a curriculum with fine-grained progress rewards for multi-turn tool use and is not cited. Comparing the final database state against a reference outcome is the core principle of the original τ-bench [2]. The paper should state precisely what is new relative to these works.

  2. The synthesis contribution is not characterized, and its relation to the evaluation benchmarks is not disclosed. The paper gives no counts of environments or tasks, no gate pass rates and no audit statistics. The training domains coincide with the evaluation domains. The released code contains a mimic mode that synthesizes structurally matching tasks from an external benchmark, including a table for translating natural-language rubrics into executable checks, and one appendix example carries a mimic source tag. The paper mentions none of this. I do not claim that test tasks were mimicked, but the sources must be disclosed. Without a control on unfiltered teacher trajectories, the SFT gain also cannot be separated from plain distillation.

  3. The reward cannot express several requirements the paper itself emphasizes. The initial-state gate excludes preservation constraints from the core criteria. As a result, instructions such as leaving a line untouched end up as bonus criteria outside the RL reward, and an unrequested write made as the final call is not penalized. Graph credit is matched on hashes of the full database snapshot, so valid alternative solutions lose credit. The released code also restricts RL to create and modify tasks, which leaves answers, clarification questions and refusals unrewarded. The paper does not state this.

Reference : [1]: Don't Just Fine-tune the Agent, Tune the Environment, https://arxiv.org/abs/2510.10197 [2]: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, https://arxiv.org/abs/2406.12045

Questions for authors
  1. Which benchmarks and which tasks served as targets for the mimic synthesis mode in the released code, and what share of the SFT trajectories and RL tasks derives from mimicked tasks? Were any VitaBench, τ²-bench or τ-Knowledge evaluation tasks used as targets, even with regenerated entities and values? If so, could you report results for models trained without the data derived from them?

  2. Could you add one matched comparison on the same SFT initialization, task pools and training budget: GRPO with the outcome-only reward, GRPO with the full summed reward but without the anchor term, and STG-GRPO? Please report per-task outcomes with a paired test, or a few evaluation trials per configuration. I am asking only for this limited comparison, not a full sweep.

  3. The settings table refers to four stage launch configurations, the curriculum table lists seven stages, and the name of the final checkpoint suggests a tenth stage. Which description is correct, and were all stages run with the same reward weights and anchor settings?

Reviewer_Kz5fOpenReview ↗
Rating
1
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of sparse outcome rewards in long-horizon agent training, which fail to capture intermediate subgoal progress and make fine-grained credit assignment challenging.

To address this challenge, the authors propose AReward, a framework that jointly synthesizes executable environments, tasks, and state-based rubrics to provide verifiable feedback on subgoal completion and intermediate progress. They further introduce STG-GRPO, which augments trajectory-level advantage estimation with step-level feedback normalized by anchors defined by reference-state progress and rubric satisfaction.

Experiments on VitaBench and τ-bench demonstrate improvements over the base model in task success and interaction efficiency, although the benefits of the RL stage over SFT are inconsistent across benchmarks and domains.

Critical strengths

S1. Well-motivated problem: The paper addresses an important limitation of sparse outcome rewards in long-horizon agent training, where intermediate subgoal completion is not adequately reflected in the learning signal.

S2. Reliable reward construction: AReward provides a well-designed framework that grounds rubric-based rewards in executable database states, enabling verifiable evaluation of subgoal completion.

S3. Fine-grained credit assignment: STG-GRPO incorporates intermediate progress and rubric satisfaction into trajectory-level advantage estimation, addressing the limitations of conventional outcome reward-based learning.

S4. Lower interaction cost: As shown in Table 1, the RL checkpoint consistently reduces the average number of tool calls compared to SFT across all evaluated domains, indicating lower interaction costs.

Critical weaknesses

W1. Unclear necessity of environment synthesis: While the paper highlights the limitations of existing rubric-based rewards, it does not sufficiently explain why addressing these limitations requires synthesizing new executable environments.

W2. Limited methodological novelty: The proposed environment synthesis pipeline overlaps with prior work [1–3], while its distinctive contributions are not clearly established.

W3. Insufficient baseline comparisons: The experiments lack comparisons with existing environment synthesis frameworks and rubric-based RL methods, making it difficult to assess the relative effectiveness of AReward and STG-GRPO.

W4. Inconsistent performance improvements: The RL checkpoint underperforms SFT on VitaBench (53.8% vs. 54.5%), raising concerns about the effectiveness and robustness of the proposed training method.

W5. Insufficient analysis of synthesized artifacts: Despite emphasizing joint environment and rubric synthesis as a key contribution, the paper lacks systematic analysis of the quality, diversity, and reliability of the generated environments and rubrics.

[1] Song et al. "EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis." In Findings of ACL, 2026.

[2] Wang et al. "Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning." In Proceedings of ICML, 2026.

[3] Dong et al. "Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence." arXiv preprint arXiv:2604.18292, 2026.

Questions for authors

Questions

Q1. Rubric-based progress: Some rubric criteria represent constraints (e.g., maintaining a total budget) rather than independently achievable subgoals; is it appropriate to treat changes in their satisfaction as measures of intermediate progress?

Q2. Reward auditing effectiveness: How effective is the proposed reward auditing pipeline in detecting and correcting invalid or misaligned rubrics, and could the authors provide a quantitative evaluation through an ablation study?

Q3. Rubric importance: Since different subgoals may have different levels of importance, have the authors considered weighting rubric criteria according to their importance or dependencies rather than treating all core criteria equally?

Suggestions

S1. Environment quality comparison: Applying the proposed validation and auditing pipeline to environments synthesized by existing methods, along with quantitative comparisons and qualitative case analyses, would help establish the advantages of AReward.

S2. Subgoal difficulty analysis: Analyzing which subgoals are most difficult to satisfy or frequently invalidated during execution would provide deeper insights into agent behavior and the effectiveness of the proposed reward mechanism.

S3. Baseline comparisons: Including comparisons with existing environment synthesis frameworks and rubric-based RL methods would provide stronger evidence of the effectiveness of AReward and STG-GRPO relative to prior approaches.

Reviewer_j14oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AReward which is a pipeline for synthesizing transactional tool-use environment (an SQLite database with tools create/read/update/delete), user task, an executable state rubric and a reference solution. To evaluate the task completion, each rubric criterion is represented in a Boolean check on the database status. These criteria are then validated by reimplementing the reference solution and a task is only kept if the solution satisfies all core criteria. In addition, the pipeline also implements reward auditing to ensure all criterion reflects correctly the user’s original request. Based on these rubrics, the authors built a per-step reward consisting of 4 components: rubric progress, progress of reference state chain, execution shaping and final outcome. To incorporate this reward into RL, STG-GRPO adds step-level advantage to trajectory advantage of GRPO. Specifically, the reward is normalized across the steps sharing the same anchor (defined by task, position on reference chain and number of unsatisfied criteria). In terms of experiment, the authors trained the model Qwen3.6-27B using SFT on teacher trajectories, followed by staged RL. The result then is compared with Base, SFT and RL checkpoint on VitaBench (400 tasks) and τ-bench suite (489 instances). SFT improves success rate on VitaBench from 45,0% to 54,5%, while RL checkpoint achieves 53,8% on VitaBench and 73,0% on τ-bench suite.

Critical strengths

A critical strength of this paper is using executable state rubrics to define particle credit for tool-use agents in a clear and reusable way. Specifically, each criterion is a typed predicate on the database (existence, absence, field value, aggregate), and is checked after each tool call to identify the signed per-criterion changes. To avoid letting agents receive credit for conditions already satisfied, the pipeline only keeps the state-changing task whose all core criteria are not satisfied in the initial state.

This approach also avoids reliance on LLM-as-a-judge, which is common in previous research but lack of consistency in evaluations. The paper’s case study also illustrates this matter when the benchmark judge applies 2 different definitions about room availability across runs. Moreover, the paper also provides a formal setup detailed enough for reimplementation, the released reward and advantage code aligns with the mathematical formulations. Additionally, the authors also transparently report less positive results including the slight RL-SFT performance gap on VitaBench and simulator mismatches across runs.

Critical weaknesses
  1. The contribution of STG-GRPO would benefit from more targeted ablations. It remains unclear whether STG-GRPO with its four-component state-grounded reward actually outperforms simpler alternative methods. Specifically, matched-data, matched-budget experiments that remove the step-level term or use outcome-only rewards, or simply apply plain GRPO would help clarify the contribution of STG-GRPO compared to simpler approach. Additionally, a comparison with GiGPO (Feng et al., 2025), a close prior work which paper cites as a basis using a two-level advantage design, would also be informative. Authors also acknowledge that identifying the contribution of each components require matched-data, matched-budget ablations The current results provide limited evidence for this concern. On VitaBench, RL has solved 215/400 tasks, slightly worse than SFT (218/400). On τ-bench, RL shows a modest improvement from 353 to 357/489 tasks, while authors acknowledge that training effects cannot be separated from simulator differences. Moreover, each instance is evaluated only once and no confidence intervals. And the improvement of rubric satisfaction comes from an earlier checkpoint (RL-50), rather than the final model. Although a null result relative to SFT might still be valuable, the paper would be strengthened if it can provide direct experimental evidence isolating its claimed methodology contribution.

  2. Benchmark might potentially be contaminated and data synthesis might not be sufficiently transparent. At least one synthesized task in Appendix B (for example, Sample 8, labeled “Source: env mimic D0812005 00”) appears to be based on task D0812005 in VitaBench’s OTA split. Even though it changes names and values, the task still preserves the original task structure, including a two-city hotel booking, price and room-view constraints,... The repository also includes a –mimic mode designed to generate structurally similar tasks from external benchmarks, yet the paper does not clarify how this mechanism was used or whether such tasks were used for training. Since evaluation covers all 400 VitaBench tasks, exposure to test-task structures could potentially have effect on the improvement of SFT, which serves as an evidence for transfer from synthesized environments. This concern is hard to assess because of missing synthesis statistics, including the numbers of seeds, tools, generated tasks and SFT trajectories as well as replay-validation discard rates and reward-auditing outcomes. Clarifying this would strengthen paper’s main claims.

Questions for authors
  1. Could the authors provide an additional RL run that use the same SFT adapter and stage pools, but with the step-level anchor term disabled, or replaced by anchor-free step-GRPO estimator? This would help to isolate the contribution of STG-GRPO by comparing this run with the reported results on VitaBench and the matched-simulator τ-bench domains. Additionally, it would be helpful to confirm whether the reported checkpoint (stage10-59) was achieved when step-level STG term was active throughout all stages. Because the appendix describes four stage launch configurations while the curriculum contains seven stages.
  2. Could the authors clarify which benchmarks were used as sources for mimic mode and whether the resulting tasks (i.e. Sample 8) were included in the SFT or RL training data? Essentially, if any training tasks were modelled on VitaBench or τ-bench evaluation tasks, reporting the numbers and the results without those tasks would help determine their influence on the reported gains. It would also be useful to explain how the SFT checkpoint (402) was selected. These clarifications would strengthen the paper.
  3. Could the authors report the number of seeds, environments, tools, generated tasks and SFT trajectories as well as the number of candidates discarded during replay validation and the number of rubrics flagged? Additionally, the paper describes auditing as a combination between rule-based scans and inspection of flagged tasks. But it is not clear whether this inspection was from human or an LLM. Providing these information would help readers assess the reliability of the synthesis pipeline and clearer contribution to SFT improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.