- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the cost and quality of trajectory-level rewards for long-horizon LLM agent RL. Its starting observation is that terminal success signals are too coarse to distinguish useful progress from wasted steps, while LLM-judge rewards provide richer assessments but require an inference call for every sampled rollout, which becomes a bottleneck for group-relative methods that compare many rollouts per update. HiRUB's proposal is to remove the judge from the training loop: an LLM (GPT-5.2) generates a hierarchical rubric once, before training, for a task family; the rubric is grounded in a task schema derived from the instruction and environment interface and compiled into deterministic, schema-conditioned checks over observations, actions and environment feedback. A lightweight scorer then evaluates every complete rollout with those checks and produces a scalar reward combining a progress component (first-hit milestone evidence such as target discovery, directed navigation, required interaction and verification), a semantic component (workflow-family and instance-level target or attribute constraints, with per-type counts capped so repeats cannot inflate credit), a terminal success component, and bounded repetition and length penalties, passed through a clip at zero so penalties cannot make a trajectory negative.
The reward is trajectory-level and is intended to be optimizer-agnostic; the paper plugs it into GRPO and GiGPO by replacing only the reward signal inside group-normalized advantage estimation. Evaluation is on ALFWorld (six task families, success rate by family and overall) and WebShop (dense score and success rate) with Qwen2.5-Instruct at 1.5B and 7B, comparing prompting baselines, PPO, RLOO, GRPO and GiGPO against their HiRUB variants. The headline numbers are ALFWorld overall success rising from 86.5 to 92.2 for GiGPO at 1.5B and from 90.8 to 94.0 at 7B, plus WebShop gains, and a matched comparison against an online LLM-judge reward in which validation success rises from 90.1 to 92.2 while total training time falls from 31.1 to 19.6 hours and per-step reward computation falls from 238.1 to 1.0 seconds. Supporting evidence includes a rubric-conditioned judge benchmark (Table C.1), an ablation over hierarchy levels and reward design, a reward-coefficient sensitivity study, a repeated-judge stability study, and failure and trajectory diagnostics.
Critical strengths
-
Removing the online judge by compiling rubric criteria into deterministic, schema-grounded checks is a real and well-targeted contribution. Placing an LLM judge inside the RL loop is the dominant cost in rubric-based agent RL, and it is paid once per rollout, which is exactly the worst case for group-relative methods that sample many rollouts per update. HiRUB attacks the problem at its source rather than by optimizing the judge call, and it does so in a way that keeps the reward interpretable and inspectable: every returned reward can be traced to named checks with listed coefficients, and the reward is a scalar, so it drops into different optimizers without changes (Section 4.6 and Algorithm G.1). The supporting evidence is concrete rather than rhetorical, namely the per-step decomposition showing reward computation dropping from 238.1 to 1.0 seconds and total wall-clock from 31.1 to 19.6 hours, together with roughly 0.7 top-5 pairwise agreement against a rubric-conditioned judge. This is the part of the paper that makes it worth accepting.
-
The empirical study is unusually thorough for its claim, and it tests the right alternatives. Beyond the main table, the paper runs a matched head-to-head against a rubric-conditioned online LLM judge (Figure 4), a reward-judge benchmark that motivates the two-stage rubric generation setup (Table C.1), ablations that separately remove the progress hierarchy, the semantic hierarchy, and the full design in favor of a minimal-rubric, step-slot, or no-repeat variant (Figure 5), a coefficient study that perturbs and rescales the shaping weights including a random-perturbation sweep (Table 2 and Table I.2), and a repeated-judge stability study that is careful to state what it does not establish (Appendix F). The coefficient study is the most convincing part, because the default setting is not a brittle optimum: uniform, inverted-hierarchy and randomly perturbed weights remain competitive, which is evidence that the gains come from the structure of the reward rather than from one lucky numeric choice. Reporting three-seed means with standard deviations at 1.5B also raises my confidence relative to the norm in this literature.
Critical weaknesses
-
The reward is non-potential-based shaping whose optimal policy is not characterized, and the reported superiority over a binary outcome baseline cannot be disentangled from reward specification. The trajectory score is a positive, non-telescoping sum of first-hit milestone indicators and constraint bonuses, clipped at zero against penalties, with coefficients that the paper states are the rubric-generating LLM's original suggestions and were not manually adjusted (Section 4.5, Appendix H, and the note in Appendix E). Shaping of this form need not preserve the optimal policy of the underlying task, and the paper never states whether it does, never verifies it empirically, and never bounds the shaping magnitude relative to the terminal success bonus of 4.0. This matters directly for how the main results should be read. The comparison HiRUB is sold on is against GiGPO with a binary sparse outcome, so a large part of any measured success gain may come from the reward having been specified rather than from better credit assignment per se. The paper's own weaker variants in Table 2 are consistent with this reading: equalized or reordered weights still reach 82.8 and 85.7 overall ALFWorld success, well above the 86.5 the binary GiGPO baseline obtains at 1.5B, which suggests that a substantial share of the gain is attributable to the mere presence of dense shaping rather than to the hierarchy the paper highlights. Since the paper's central claim is that task-aware rubric structure improves agent learning, not merely that dense shaping helps, this ambiguity is consequential.
-
The headline gains are not statistically robust, and family-level results contradict the claimed consistency. The abstract and introduction state that HiRUB achieves consistent gains over the evaluated RL baselines, and the strongest evidence offered for the 7B scale is a comparison of 90.8 against 94.0 that carries no uncertainty at all: Table 1 reports three-seed standard deviations only for the rows that have them, and in the 7B block the RLOO, GRPO and GiGPO comparisons, including the HiRUB variant, are reported as bare numbers. This is the wrong place to economize, because the 1.5B rows that do carry error bars show that the measurement noise in this setting is very large: RLOO at 1.5B reports 52.8 plus or minus 8.6 on Look, GRPO reports 33.3 plus or minus 19.1 on the same family, and HiRUB's own WebShop success gain of 4.3 points carries plus or minus 2.8. Against that background a 3.2 point gap with no interval is not interpretable. The per-family numbers also do not support consistency: at 7B, HiRUB moves Look from 82.7 to 75.0, a 7.7 point regression, and at 1.5B it moves Clean from 94.9 to 87.5 while carrying plus or minus 7.8, which means the claim of a 5.7 point overall improvement at 1.5B is partly an average over family-level movement in the opposite direction. Because acceptance here hinges on whether the reward reliably improves learning rather than reshuffling performance across families, this combination of unpreserved uncertainty and family-level regressions is the main obstacle to a stronger recommendation.
Questions for authors
Optimality of the shaped objective. Is the HiRUB trajectory reward potential-based or otherwise optimality-preserving with respect to the environment success signal, or is it intended as a heuristic shaping term? If it is not optimality-preserving, please give the empirical test: how does final success change as the terminal success bonus is swept relative to the sum of the shaping coefficients (for example bonus multiplied by 0.5, 1, 2, 4 with shaping fixed)? This bears directly on whether the reported gains reflect better optimization of the same objective or a different objective.
Is the LLM-generated rubric necessary? The checks are grounded in the task schema, and for ALFWorld the paper notes that the scorer actually inverts the environment's fixed goal-template library and fixed action space (Section 4.2). Could a hand-written dense reward with the same milestone and constraint features, but no LLM-generated rubric, match HiRUB? An instruction-to-checks ablation that bypasses the LLM rubric would isolate whether the rubric contributes anything beyond the shape of the reward. This is the same question as the equalized-weights result in Table 2, which already performs above the binary baseline.
Seeds and uncertainty for the 7B comparisons. How many seeds underlie each 7B entry in Table 1, and can you add standard deviations? Given that the 1.5B entries reach plus or minus 8.6, plus or minus 13.9, and plus or minus 19.1 on individual families, what is the confidence interval on the 90.8 to 94.0 and 72.8 to 73.7 comparisons?
The family-level regressions. The 7B Look family drops from 82.7 to 75.0 and the 1.5B Clean family drops from 94.9 to 87.5. Do these reflect systematic interference from the shaping terms, for example the family-condition, light-task inspect, or two-object coefficients interacting with the length and repetition penalties, or are they within seed noise? Please support the claim of consistent gains with a per-family test rather than an aggregate.
What is the right reading of the cost claim? The 31.1 versus 19.6 hour comparison issues judge requests at API concurrency two, and the per-step reward computation is 238.1 seconds for the judge against 1.0 second for HiRUB. Judge calls are asynchronous, so in a typical implementation they would overlap with GPU work rather than block it. What were GPU idle time and GPU utilization in the online-judge run, and what happens to the comparison at a realistic concurrency or with a locally served judge model? Please also report end-to-end API cost, and separate wall-clock savings from monetary savings.
Trajectory-level versus per-step accumulation. Section 4.6 and the related-work discussion describe credit as assigned once at the trajectory level tied to the task schema, but the progress and semantic terms are first-hit indicators over the whole rollout, so at step t the running reward already contains evidence satisfied at earlier steps. How is the reward consumed by the group-relative update in the actual implementation, as a terminal return or as a per-turn signal, and does the milestone evidence accumulate into a reward that grows with episode length in a way that interacts with the length penalty?
Alternative explanations for the gradient signal. The gain could reflect reduced variance from group normalization when rewards take many distinct values, rather than the task semantics of the checks. Did you compare against a dense but semantically uninformative reward with a comparable spread, for example a random or frequency-based milestone counter? This would separate the value of the rubric content from the value of reward diversity.
Generalization beyond templated environments. Appendix J notes the dependence on a task schema and on stable instruction, action and observation structure. For an environment in the spirit of ALFWorld but with an open-ended instruction set, what is the expected failure mode of the deterministic parser, and which component would need to change first?
Exploitation of the deterministic scorer. The reward is computed by a parser over text, and parsing and matching decisions are deterministic rather than sampled. Did you observe reward-hacking behavior, for example action sequences that satisfy target-aware checks without advancing the task, or degenerate placements that hit the two-object terms incrementally? Since the paper calls this a reward rather than a critic, a short negative-result statement would be useful.
Response to rebuttal
Thanks for your responses to the raised questions. For Q1, as was my concern, poorer rubrics lead to poorer downstream success. Similarly, for Q3, it seems very obvious the level of intensive effort required will be to design these rubrics for another task. I could anticipate these having worked on AlfWorld and similar datasets/research directions in a great depth. I acknowledge the responses for Q2 and Q4, and my concerns are addressed for those.
Since my primary concerns are with respect to the assumptions made behind the approach, and the rebuttal responses indeed verify those details - I recommend the authors to revise these assumptions to make this work stronger in subsequent submissions. I will maintain my score.