acceptodds
Under review as a conference paper at ICLR 2027

DESIGN THEFEEDBACK, NOTJUST THEREWARD: ENVIRONMENTS FOROPEN-ENDED REINFORCEMENT LEARNING

Abstract

Reinforcement learning has advanced large language models on tasks with reliableverifiers, where checking an outcome provides a clear basis for reward. Extendingthis approach to open-ended tasks requires environments that can both evaluatecomplex, partially successful outputs and turn these judgments into effective learn-ing guidance. We propose a reinforcement learning framework built around atraining environment that integrates human domain knowledge with model reason-ing. Human domain knowledge, encoded in rubrics, anchors evaluation to taskintent, while model reasoning combined with hierarchical rewards distinguishescomplex open-ended outputs and links diagnosed failures to likely responsible out-put regions. Furthermore, these diagnostic attributions guide token-level weights inthe GRPO objective, thereby translating the ambiguous notion of success in open-ended tasks into concrete guidance for policy optimization. We instantiate thisframework in interactive web application generation, a representative open-endedgeneration task. The resulting model achieves an average pass rate of 41.25% onMiniAppBench and 76.19% on ArtifactsBench. Ablation studies show the contri-butions of individual components, while additional out-of-distribution evaluationssupport transferable gains without degrading performance on other evaluated tasks.

Primary area:
reinforcement learning
Keywords:
Open-ended reinforcement learning, training environment, rubric-guided evaluation, hierarchical rewards, diagnostic attribution, group relative policy optimization, interactive web application generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yggXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning framework for open-ended tasks that combines rubric-based hierarchical rewards with diagnostic token weighting for GRPO. It implements the framework for web application generation, connecting requirement evaluation, source-code attribution, and policy optimization. Experiments show improvements on MiniAppBench and ArtifactsBench, with ablations supporting the contributions of hierarchical rewards and diagnostic weighting.

Critical strengths

Substantial improvements on the target task. The full method improves the MiniAppBench pass rate from 26.85% after SFT to 41.25%, with gains also observed on ArtifactsBench. Under the same hierarchical reward, diagnostic weighting adds 8.25 percentage points, providing evidence of its practical value for web application generation.

Critical weaknesses
  1. The general methodological contribution is not clearly separated from domain-specific feedback engineering. The method relies on carefully designed rubrics, reward rules, and source-code attribution mechanisms. Both main benchmarks concern web application generation, leaving unclear whether the framework would remain effective in other domains without substantial redesign. The additional general-capability evaluations mainly assess capability preservation after training, rather than cross-domain applicability of the method. The evidence therefore does not sufficiently support the broader claim of a general framework for open-ended RL.

  2. Diagnostic weighting does not resolve the conflict between local correctness and global advantage. The update direction for every token is still determined by the response-level advantage. Diagnosed errors in an above-average response are therefore still reinforced, while correct code in a below-average response is still discouraged. The sign-dependent weights only adjust the strength of these updates. The comparison with unconditional error penalization supports the proposed rule over that particular alternative, but does not establish that it reliably handles this conflict, limiting the evidence for its effectiveness as a local credit-assignment method.

Questions for authors
  1. Evidence of generality: Can the authors clarify which components can be reused directly and which require domain-specific redesign, and provide validation beyond web application generation to distinguish transferable methodological gains from domain-specific feedback engineering?

  2. Evidence for local credit assignment: Can the authors provide analysis or controlled experiments on cases where local diagnoses conflict with the global advantage, explaining why preserving the global update direction remains effective? In particular, how does continuing to reinforce diagnosed errors in positive-advantage responses affect those errors over training?

1 comment
AuthorsOpenReview ↗

Response to Reviewer yggX

Thank you for the detailed feedback. We address both questions below.

Q1. How is a transferable framework distinguished from domain-specific feedback engineering?

We agree that the additional benchmarks primarily assess capability preservation after training. Our framework targets open-ended tasks with multidimensional success criteria, partial fulfillment, and failures that can be linked to specific output regions. Its reusable principle connects rubric-grounded comparisons across outputs with diagnostic guidance within each output through the GRPO objective. Domain adaptation requires appropriate rubrics, evaluators, and attribution mechanisms. For example, these could link requirements to claims and evidence in deep research or computations and actions in data analysis workflows. We select interactive web application generation as a representative testbed, and the two benchmark results support the effectiveness of this instantiation. Broader applicability remains a research direction requiring cross-domain training experiments.

Q2. How does retaining the global advantage sign address conflicts with local correctness?

We agree with the mathematical limitation: positive weights do not change the advantage sign. Incorrect tokens in positive-advantage responses may still be reinforced, and correct tokens in negative-advantage responses may still be suppressed. The method adjusts update allocation; it does not construct fully independent token-level advantages.

The response-level advantage compares complete responses within a group. Diagnosis identifies regions related to functional failures, rather than estimating the expected return difference between each token decision and its alternatives. A flawed implementation can outperform other candidates, and a diagnosed region may mix correct structure with erroneous operations. Directly assigning an independent negative update from a region-level diagnosis therefore requires further evidence for local credit assignment.

We use the response-level comparison with positive local weights:

A~i,t=wi,tAi,wi,t>0.\tilde{A}_{i,t}=w_{i,t}A_i,\qquad w_{i,t} > 0.

For negative advantages, diagnosed error regions receive stronger negative updates. For positive advantages, regions not implicated by diagnosis receive relatively stronger positive updates. This avoids additional amplification of diagnosed errors while retaining within-group comparative information; it does not eliminate every conflict between global sign and local correctness.

Section 4.4, Table 3 compares the following methods with the same SFT initialization, hierarchical reward, and training settings:

Update methodMiniAppBench mean pass rate (%)
Standard GRPO without diagnostic weighting33.00
Unconditional error penalization37.00
Full method41.25

Unconditional error penalization applies negative updates to diagnosed regions regardless of the response-level advantage sign. It and our method use the same diagnostic localization; our method preserves the sign and adjusts regional update strength.

The full method exceeds hierarchical reward alone by 8.25 percentage points and unconditional error penalization by 4.25 points. These results support coordinating local diagnosis with within-group advantage under the tested settings, rather than applying negative signals to every diagnosed error. They demonstrate final-performance differences, without establishing that all local/global conflicts are resolved during training.

Reviewer_D3zoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how diagnostic feedback can support reinforcement learning for open-ended tasks, using interactive web application generation as a case study. The method decomposes task requirements into rubrics and constructs hierarchical rewards based on output format, code validity, inferred executability, and functional fulfillment. It then maps diagnosed code regions to tokens and weights the GRPO objective according to the sign of the response-level advantage. The training evaluator infers behavior from source code rather than executing browser interactions.

On Ling-3.0-Flash, the full method improves MiniAppBench performance from 26.85% after SFT to 41.25%, and ArtifactsBench performance from 71.71 to 76.19. Ablations compare binary rewards, hierarchical rewards, diagnostic weighting, and unconditional error penalization. Results on additional benchmarks show no apparent degradation in general capabilities.

Critical strengths
  1. The full system delivers practically meaningful gains, supported by results on a second benchmark. Starting from the same SFT checkpoint, the method improves MiniAppBench performance by 14.40 percentage points and ArtifactsBench performance by 4.48 points (Tables 1/2). This is more compelling than an increase in training reward alone. According to the respective benchmark papers, MiniAppBench incorporates browser interaction testing, while ArtifactsBench includes visual evaluation of rendered outputs, providing assessment beyond the source-code diagnostics used during training. These results support the framework’s practical value for application generation.

  2. The advantage-sign-aware use of feedback is well motivated and supported by a targeted comparison. Section 3.5 correctly observes that a response containing local errors can still receive a positive advantage. Amplifying updates at erroneous regions or unconditionally applying negative updates can therefore conflict with the relative-quality signal across responses. The full method achieves 41.25%, compared with 37.00% for unconditional error penalization (Table 3), providing useful evidence for how local diagnostic feedback should interact with response-level advantages.

Critical weaknesses
  1. The main ablations do not isolate the contribution of diagnostic localization itself. Appendix D.3 states that the token weights are not normalized by the sum of the diagnostic weights, so they change both the distribution of updates within a response and the response’s aggregate contribution to the objective. The improvement from 33.00% to 41.25% in Table 3 therefore does not, by itself, establish that the gains arise from correctly localizing erroneous regions. Controls should include shuffled localization that preserves each response’s total weight, as well as response-level reweighting alone. Similarly, the comparison between binary and hierarchical rewards introduces both partial credit and a hierarchical structure, without comparison against a simple reward based on the fraction of satisfied rubric items. These omissions limit the evidence that the proposed core design choices are necessary.

  2. Diagnostic reliability and localization quality are not directly validated on actual rollouts. The evaluator assesses source code without executing browser interactions, and localization relies on approximate source-level relations. The worked example is explicitly constructed rather than drawn from a training rollout. However, the paper does not report agreement between its diagnoses and browser-based or human assessments, the accuracy of the attributed regions, or the frequency of localization failures on actual samples. This leaves a potential gap between code-inferred rewards and observed application behavior. Incorrect attributions could also amplify negative updates to unrelated or correctly implemented regions in negative-advantage responses. The external benchmark gains support the effectiveness of the overall system, but do not directly establish the reliability of its diagnostic feedback or localization mechanism. An audit of actual rollouts against observed behavior, with manual inspection of attributed regions and reporting of localization coverage, would materially strengthen this central claim.

Questions for authors
  1. Contribution of localization: Does diagnostic localization still improve performance over shuffled localization that preserves each response’s total weight, or over uniform response-level weighting with the same total weight? These comparisons would help distinguish the benefit of identifying erroneous regions from that of reweighting responses.

  2. Contribution of reward hierarchy: How does the hierarchical reward compare with a simple reward based on the fraction of satisfied rubric items? Does the hierarchy provide gains beyond introducing partial credit?

  3. Diagnostic reliability: On actual policy rollouts, how closely do the evaluator’s diagnoses agree with browser-based or human assessments? Can you provide an audit of whether attributed regions are relevant to the diagnosed failures, together with the proportion of findings that cannot be localized? These results would help assess whether the diagnostic weights reliably target relevant implementation regions.

1 comment
AuthorsOpenReview ↗

Response to Reviewer D3zo

Thank you for the detailed feedback. We address all three questions below.

Q1. Does diagnostic localization add value beyond response reweighting?

We completed a shuffled-localization control with the same SFT initialization, hierarchical reward, evaluator, training budget, and evaluation protocol. For a response of TT tokens, it uses wtshuffle=wπ(t)w_t^{\mathrm{shuffle}}=w_{\pi(t)}, where π\pi permutes 1,…,T\\{1,\ldots,T\\}. It preserves the weight values, counts, total, and mean within each response, changing only their token positions.

SettingMiniAppBench mean pass rate (%)
Hierarchical reward only (paper)33.00
Shuffled localization (new control)32.50
Full diagnostic weighting (paper)41.25

The full method exceeds the shuffled control by 8.75 percentage points, consistent with a benefit from diagnostic placement in this configuration. The 0.50-point gap between shuffled localization and the baseline alone does not establish that shuffling harms training.

Two controls remain unrun: uniform response-level weighting, wtuniform=1T∑j=1Twjw_t^{\mathrm{uniform}}=\frac{1}{T}\sum_{j=1}^{T}w_j, which preserves the response's total diagnostic weight; and normalized diagnostic weighting, which rescales each response to mean weight 1. The current evidence does not fully separate diagnostic placement from response-level reweighting.

Q2. Does the hierarchical reward outperform a rubric satisfaction ratio?

We completed a comparison with the same model initialization, GRPO configuration, and other training settings, changing only the reward calculation. The fraction of satisfied rubric items yields 31.00% on MiniAppBench, versus 33.00% for the hierarchical reward: a modest 2.00-point gain that does not establish necessity.

The hierarchy checks syntax, then successful execution, and then functionality, while accounting for error severity. A flat ratio can assign equal rewards to failures with different effects on usability. This motivates validity gating when early policies produce syntax/runtime errors or blank pages. Final scores do not establish a larger early-training benefit; training curves and blank-page failure rates would be needed. The framework can instead use the satisfaction ratio while retaining diagnostic token weighting, and we do not claim universal superiority.

Q3. Are diagnosis and localization reliable on actual rollouts?

Before RL, we conducted three offline evaluations, whose settings, results, and error analysis we will add to the revised paper:

  • Human-annotated error regions: the evaluator covered the annotated errors in 445/500 samples (89.0%). This is sample-level coverage, not exact boundary agreement.
  • Manually injected errors: it located the incorrect code in 497/500 samples (99.4%), with 3 incorrect judgments.
  • Local repair: on 2,000 samples, repairs restricted to diagnosed incorrect and related regions increased reward in 1,879 cases (93.95%). Of the 121 cases without an increase, most had low initial rewards. Manual inspection found correct localization in 89 of them; the repair model failed to make the required changes correctly. Thus, repair failure is not equivalent to localization failure.

These offline tests provide evidence about diagnosis and repair, rather than a systematic audit of on-policy rollouts during training.

Additional training controls. Each full run requires 64 NVIDIA H200 GPUs for five days. The rebuttal time and compute budget prevented completion of all requested training controls; unrun controls remain limitations of the current evidence.

Reviewer_K16dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning framework for open-ended tasks that integrates rubric-based evaluation, hierarchical rewards, and diagnostic token-level weighting into GRPO. Specifically, the framework uses human-guided rubrics and an LLM-based evaluator to assess generated outputs, constructs hierarchical rewards based on validity and requirement satisfaction, and identifies potentially erroneous output regions to guide token-level policy updates. The method is evaluated on interactive web application generation, achieving substantial improvements on MiniAppBench and ArtifactsBench.

Critical strengths

Meaningful problem motivation. The paper addresses an important challenge in reinforcement learning for open-ended tasks, where evaluating complex outputs and providing informative learning signals are considerably more difficult than in tasks with deterministic verifiers.

Encouraging empirical results. The proposed framework achieves substantial improvements over its SFT initialization on MiniAppBench and ArtifactsBench. The ablation studies also demonstrate that hierarchical rewards and diagnostic token weighting contribute to the overall performance gains.

Clear presentation and implementation details. The paper is generally well organized, and the authors provide detailed descriptions of the evaluation pipeline, reward construction, diagnostic attribution, and training configuration.

Critical weaknesses
  • Limited methodological novelty beyond rubric-based reward modeling. My primary concern is that the proposed framework is largely an extension of existing rubric-based reward modeling and fine-grained feedback methods. The main components, including hierarchical evaluation, rule-based reward shaping, diagnostic attribution, and token-level gradient weighting, are combined into a unified pipeline, but the paper does not introduce a sufficiently distinct learning or optimization principle. While the overall system is reasonably designed, its methodological contribution appears relatively incremental.

  • Heuristic design with limited principled justification. Several key components rely heavily on manually specified rules and hyperparameters. For example, the hierarchical reward uses predefined validity gates and severity-dependent penalties, while the token-level optimization assigns fixed weights of 3.0, 1.5, 1.0, and 1.2 according to diagnostic categories and the sign of the response-level advantage. These choices are not derived from a principled optimization objective, and the paper provides limited analysis of their sensitivity or necessity. Consequently, it is unclear whether the proposed design reflects a generalizable learning mechanism or a set of heuristics tuned to the specific application.

  • Strong dependence on the external evaluator and task-specific rules. The effectiveness of the framework depends heavily on the quality of the generated rubrics and the capabilities of the external evaluation model. In particular, the evaluation model is responsible for assessing functional correctness, identifying implementation errors, and providing diagnostic evidence for token-level weighting. However, the paper does not sufficiently investigate how evaluation inaccuracies, weaker evaluators, or imperfect rubrics affect the resulting policy optimization. This dependence raises concerns about the robustness of the framework and whether its effectiveness can be maintained without a strong evaluator and carefully designed task-specific rules.

  • Limited scalability and generalizability to broader open-ended tasks. Although the paper is motivated by open-ended reinforcement learning, the proposed evaluation and attribution mechanisms are highly tailored to interactive web application generation. The diagnostic pipeline relies on source-code structures such as functions, event handlers, state variables, and code spans, which may not be directly applicable to other open-ended tasks. Extending the framework to different domains would likely require substantial redesign of the evaluation rules, diagnostic procedures, and attribution mechanisms. Moreover, the fixed token-weighting scheme may not generalize well across tasks, output structures, or model scales. The paper does not provide sufficient evidence that the proposed framework can scale beyond its specific application setting.

  • Insufficient validation of diagnostic attribution and token-level weighting. Although the ablation studies show that diagnostic token weighting improves downstream performance, they do not fully establish whether accurate error localization is responsible for these gains. The proposed weighting scheme is not normalized, meaning that it changes both the relative importance of tokens and the overall gradient magnitude of individual responses. Therefore, some improvements may arise from gradient scaling rather than meaningful token-level credit assignment. Comparisons with random token weighting, uniform response-level weighting, or normalized diagnostic weights would help isolate the actual contribution of the proposed attribution mechanism.

Questions for authors

Please refer to the Weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer K16d

Thank you for the detailed feedback. We address all five questions below.

Q1. What is new beyond existing rubric rewards and fine-grained feedback?

Our framework is designed for open-ended tasks that admit multiple valid solutions, involve ambiguous and multidimensional success criteria, and produce long, structured outputs with partial correctness. Its central idea is to connect task intent to policy optimization at two complementary resolutions. (1) At the sample level, rubrics make evaluation criteria explicit, while hierarchical rewards convert multidimensional judgments into scalar signals that distinguish validity failures and degrees of task fulfillment for group-relative learning. (2) At the token level, rubric-to-output attribution links unmet requirements to relevant output regions and converts these diagnoses into advantage-aware weights in the GRPO objective. This pipeline provides a reusable design for translating ambiguous task intent into comparable rewards and localized learning guidance. It addresses challenges shared by deep research report generation, data analysis workflows, and tool-using agents, where success involves multiple requirements and failures can be embedded in specific claims, computations, or actions within long outputs. We select interactive web application generation as a representative testbed. Improvements from 33.00% to 41.25% on MiniAppBench and from 71.71 to 76.19 on ArtifactsBench, together with component ablations, demonstrate the pipeline’s effectiveness in this setting and point to a promising direction for training future agentic systems, where environments provide structured, actionable feedback that directly guides policy optimization.

Q2. Are the fixed weights and reward rules task-specific heuristics?

The ordering follows diagnostic relevance. For negative advantages, direct error regions receive more weight than related context, followed by unattributed regions. For positive advantages, regions not implicated by the diagnosis receive more weight. Positive weights preserve the response-level advantage sign. Our offline tests support this diagnostic guidance: human-annotated error coverage was 89.0%, and injected-error localization success was 99.4%. These results do not validate the exact coefficients.

The values 3.0, 1.5, 1.0, and 1.2 are empirical choices. The reward checks syntax, execution, and functionality in order; its numerical severity penalties are also empirical. We will clarify these choices and add weight sensitivity tests and component ablations as compute becomes available. Each full training run requires 64 H200 GPUs for five days, limiting additional rebuttal experiments.

Q3. How dependent is the method on strong evaluators and complete rubrics?

We agree that our framework depends on reliable rubrics and evaluators. Rubrics define the reward criteria and guide the diagnoses used for token-level credit assignment. Their quality is therefore central to the learning signals. We refined the rubrics through multiple versions before training. Our binary-reward ablation also shows that how rubric outcomes are converted into rewards affects learning.

Evaluator quality is equally important. Before training, we assessed reward-ranking agreement with human judgments and the evaluator's ability to locate incorrect code regions. We adopted the evaluator only after these offline checks met our requirements for scoring and localization. These checks support our evaluator choice, but do not establish robustness to weaker evaluators.

Because of time and compute constraints, we have not conducted full training comparisons with weaker evaluators or systematically varied rubric quality. We will make this dependence explicit in the revised paper and add controlled experiments as resources become available.

Q4. How does the web implementation support broader open-ended tasks?

The reusable core connects rubric-grounded evaluation, hierarchical rewards, and diagnostic weighting in GRPO through a shared rubric-to-output mapping. Functions, event handlers, and state variables provide one concrete implementation of this mapping for web applications. In deep research, data analysis, or agent workflows, corresponding mappings could link unmet requirements to relevant claims, computations, or actions. The same optimization rule would apply. Adapting the evaluation criteria and attribution mechanisms requires domain knowledge. We select interactive web application generation as a representative setting. It combines ambiguous success criteria, partial correctness, and long, structured outputs. Results on MiniAppBench and ArtifactsBench, together with component ablations, support the effectiveness of this implementation. The numerical weights are empirical hyperparameters. Their robustness across domains, output structures, and model scales remains to be validated.

Q5. Do gains come from meaningful token-level attribution?

We completed a shuffled-localization control with the same SFT initialization, hierarchical reward, evaluator, training budget, and evaluation protocol. For a response of TT tokens, it uses wtshuffle=wπ(t)w_t^{\mathrm{shuffle}}=w_{\pi(t)}, where π\pi permutes 1,…,T\\{1,\ldots,T\\}. It preserves the weight values, counts, total, and mean within each response, changing only their token positions.

Experimental settingWeight allocation within a responseOriginal total diagnostic weight preservedMiniAppBench Mean pass rate (%)
Hierarchical reward only, no diagnostic weighting*Weight 1.0 for all generated tokensNot matched; baseline only33.00
Shuffled localizationKeep weight values and counts; randomly permute their positionsYes32.50
Full diagnostic localization weighting*Assign weights according to actual diagnostic locationsOriginal diagnostic weights retained41.25

* Results reported in the paper. Shuffled localization is a completed new control.

The full method exceeds the shuffled control by 8.75 percentage points, consistent with a benefit from diagnostic placement in this configuration. The 0.50-point gap between shuffled localization and the baseline alone does not establish that shuffling harms training.

Two controls remain unrun: uniform response-level weighting, wtuniform=1T∑j=1Twjw_t^{\mathrm{uniform}}=\frac{1}{T}\sum_{j=1}^{T}w_j, which preserves the response's total diagnostic weight; and normalized diagnostic weighting, which rescales each response to mean weight 1. The current evidence does not fully separate diagnostic placement from response-level reweighting.

Additional training controls. Each full run requires 64 NVIDIA H200 GPUs for five days. Time and compute limits prevented the remaining training controls; these remain limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.