- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a reinforcement learning framework for open-ended tasks that combines rubric-based hierarchical rewards with diagnostic token weighting for GRPO. It implements the framework for web application generation, connecting requirement evaluation, source-code attribution, and policy optimization. Experiments show improvements on MiniAppBench and ArtifactsBench, with ablations supporting the contributions of hierarchical rewards and diagnostic weighting.
Critical strengths
Substantial improvements on the target task. The full method improves the MiniAppBench pass rate from 26.85% after SFT to 41.25%, with gains also observed on ArtifactsBench. Under the same hierarchical reward, diagnostic weighting adds 8.25 percentage points, providing evidence of its practical value for web application generation.
Critical weaknesses
-
The general methodological contribution is not clearly separated from domain-specific feedback engineering. The method relies on carefully designed rubrics, reward rules, and source-code attribution mechanisms. Both main benchmarks concern web application generation, leaving unclear whether the framework would remain effective in other domains without substantial redesign. The additional general-capability evaluations mainly assess capability preservation after training, rather than cross-domain applicability of the method. The evidence therefore does not sufficiently support the broader claim of a general framework for open-ended RL.
-
Diagnostic weighting does not resolve the conflict between local correctness and global advantage. The update direction for every token is still determined by the response-level advantage. Diagnosed errors in an above-average response are therefore still reinforced, while correct code in a below-average response is still discouraged. The sign-dependent weights only adjust the strength of these updates. The comparison with unconditional error penalization supports the proposed rule over that particular alternative, but does not establish that it reliably handles this conflict, limiting the evidence for its effectiveness as a local credit-assignment method.
Questions for authors
-
Evidence of generality: Can the authors clarify which components can be reused directly and which require domain-specific redesign, and provide validation beyond web application generation to distinguish transferable methodological gains from domain-specific feedback engineering?
-
Evidence for local credit assignment: Can the authors provide analysis or controlled experiments on cases where local diagnoses conflict with the global advantage, explaining why preserving the global update direction remains effective? In particular, how does continuing to reinforce diagnosed errors in positive-advantage responses affect those errors over training?
Response to Reviewer yggX
Thank you for the detailed feedback. We address both questions below.
Q1. How is a transferable framework distinguished from domain-specific feedback engineering?
We agree that the additional benchmarks primarily assess capability preservation after training. Our framework targets open-ended tasks with multidimensional success criteria, partial fulfillment, and failures that can be linked to specific output regions. Its reusable principle connects rubric-grounded comparisons across outputs with diagnostic guidance within each output through the GRPO objective. Domain adaptation requires appropriate rubrics, evaluators, and attribution mechanisms. For example, these could link requirements to claims and evidence in deep research or computations and actions in data analysis workflows. We select interactive web application generation as a representative testbed, and the two benchmark results support the effectiveness of this instantiation. Broader applicability remains a research direction requiring cross-domain training experiments.
Q2. How does retaining the global advantage sign address conflicts with local correctness?
We agree with the mathematical limitation: positive weights do not change the advantage sign. Incorrect tokens in positive-advantage responses may still be reinforced, and correct tokens in negative-advantage responses may still be suppressed. The method adjusts update allocation; it does not construct fully independent token-level advantages.
The response-level advantage compares complete responses within a group. Diagnosis identifies regions related to functional failures, rather than estimating the expected return difference between each token decision and its alternatives. A flawed implementation can outperform other candidates, and a diagnosed region may mix correct structure with erroneous operations. Directly assigning an independent negative update from a region-level diagnosis therefore requires further evidence for local credit assignment.
We use the response-level comparison with positive local weights:
A~i,t=wi,tAi,wi,t>0.
For negative advantages, diagnosed error regions receive stronger negative updates. For positive advantages, regions not implicated by diagnosis receive relatively stronger positive updates. This avoids additional amplification of diagnosed errors while retaining within-group comparative information; it does not eliminate every conflict between global sign and local correctness.
Section 4.4, Table 3 compares the following methods with the same SFT initialization, hierarchical reward, and training settings:
Unconditional error penalization applies negative updates to diagnosed regions regardless of the response-level advantage sign. It and our method use the same diagnostic localization; our method preserves the sign and adjusts regional update strength.
The full method exceeds hierarchical reward alone by 8.25 percentage points and unconditional error penalization by 4.25 points. These results support coordinating local diagnosis with within-group advantage under the tested settings, rather than applying negative signals to every diagnosed error. They demonstrate final-performance differences, without establishing that all local/global conflicts are resolved during training.