- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes AReward and STG-GRPO that are environment-generation and post-training techniques respectively to improve agent's performance on state-based tasks that require CRUD operations (create, read, update, delete). The AReward framework generates tasks and executable state rubrics. STG-GRPO provides additional anchored advantages that are based on a common anchor state during a GRPO run to ensure more fine-grained advantage. Each step's reward combines progress along the reference state chain, change in rubric satisfaction, execution validity and outcome.
Critical strengths
Representations of task and state is unified under the database-state representation. The paper formalizes the state changes and rewards based on the basic CRUD operations.
Critical weaknesses
W1. RL results are similar or worse than SFT and not coherent.
• L077-079 in the introduction report 54.5% for SFT on VitaBench compared to 53.8% for RL.
• The reported higher rubric satisfaction criteria (L086-087, L437-435) compare a different RL checkpoint (RL-50) from the main Table 1. Even in that comparison, RL leads to a slight improvement compared to SFT with no improvement in task success rate.
W2. Weak experimental rigour.
• [W2.1] The paper does not have any baseline other than SFT. It does not even compare to vanilla GRPO with just the final task completion binary reward.
• [W2.2] There are no ablations or sensitivity analysis to show the importance of the different reward terms in Equation 7. Given that the paper is introducing these additional terms, there should be ablations and hyper-param search to justify the choices and their effects.
W3. Effect of STG-GRPO not conclusive. In addition to the missing ablation discussed in W2.1, there is one more issue with STG-GRPO advantage term introduced in Equation 9. This term requires anchors that are common states. Given that the training was done using only 4 rollouts, finding common anchors across different rollouts becomes difficult. The authors should report some statistics regarding when does this term get activated.
W4. Final advantage is not step level Although the additional STG-GRPO term is computed across common anchor steps, but the advantage in Equation 9 is distributed to the entire roll-out. This makes the signal average out over the trajectory.
W5. Manuscript issues
• The model name or experiment names are present in the paper (e.g. SFT checkpoint 402 and RL checkpoint stage10-59).
• Table 3 lists 7 stages but L719 states that there are four stages.
• Last para of page 15 and starting para of page 16 has some garbled text.
Questions for authors
I would request the authors to respond to their design choices regarding the use of STG-GRPO, such as choice of hyper-parameters, ablating the reward components, etc. The authors should also respond to the other weaknesses stated in the weakness section above.