acceptodds
Under review as a conference paper at ICLR 2027

Learning the Loop, Not Just the Page: Execution-Grounded Self-Improvement for Web Generation

Abstract

Functional Web generation is increasingly optimized with executable rewards, yet existing methods largely focus on the quality of the final page and leave the process of diagnosing and repairing imperfect implementations underexplored. We identify a central challenge in this setting: the Generator and Refiner produce executable artifacts with direct environment rewards, whereas the intermediate Critic influences downstream behavior without a directly executable outcome. We introduce WebLoop, an execution-grounded framework that jointly learns generation, critique, and refinement within a shared policy. WebLoop trains an execution-free Critic with complementary signals for requirement-level discriminability and downstream helpfulness, first establishing reliable diagnosis and then introducing consequence-aware credit, while all three roles are jointly optimized with group-relative policy learning. With Qwen3.5-9B, WebLoop reaches 41.5 Overall on WebRise and 38.9% accuracy on WebGen-Bench, improving the base model by 11.3 and 15.4 points, respectively. The gains transfer to first-pass generation, persist at 27B scale, and generalize from text-only training to multimodal inputs. Controlled analyses further show that the improvement cannot be explained by an additional refinement pass alone, highlighting the importance of learning the Critic and the loop itself.

Primary area:
reinforcement learning
Keywords:
website generation, reinforcement learning, large language model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_tKU7OpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WebLoop, a reinforcement learning framework for functional web generation that extends execution-based supervision to intermediate critique and subsequent refinement. A shared policy serves as the Generator, Critic, and Refiner, with the Critic trained using requirement-level diagnostic accuracy and downstream refinement gains. The framework is evaluated using Qwen3.5-9B and Qwen3.5-27B on two functional web generation benchmarks, WebRise and WebGen-Bench.

Critical strengths
  1. The paper addresses a clearly motivated and practically valuable problem that has received relatively limited systematic study: how to use execution feedback to optimize the intermediate Critic between generation and refinement in functional web generation. The supervision design is closely tied to the task. The paper also provides meaningful mechanism analyses and openly presents failures, distinguishing incorrect diagnoses, inaccurate fault localization, incomplete repair guidance, and Refiner-induced regressions, which helps clarify the system’s limitations.
  2. The experimental design is clear. In addition to comparisons with the original models and existing critique-learning methods, the paper includes generation-only training, refinement without critique, staged training, and joint training as controls. Furthermore, evaluations cover two model sizes, 9B and 27B, two functional web generation benchmarks, and five input modalities, providing some breadth to the findings.
Critical weaknesses
  1. Helpfulness depends on both feedback quality and the Refiner’s ability to implement the suggested changes. Figure 4(b) provides partial evidence by fixing the Generator/Refiner checkpoint and varying Critic checkpoints, but diagnostic accuracy, repair success rates, and regression rates for previously correct functionality are still missing on a fixed, independent set of drafts.
  2. Computational costs and key configurations are insufficiently documented. The paper does not adequately report actual training costs, inference latency, or gains under matched computational budgets. The rationale for selecting the warm-up duration and reward weight is also unclear.
Questions for authors
  1. Helpfulness is directly affected by the current Refiner’s capabilities. Have the authors considered fixing the Refiner during training or evaluation to compare the repair gains induced by different Critic outputs? If the same set of critiques is given to Refiners with different capabilities, do the Helpfulness rankings remain consistent?
  2. Why are the first 15 updates used to optimize only Discriminability, followed by the combined reward with λ=0.2\lambda=0.2? Were these hyperparameters selected using a validation set or based on other empirical considerations? How would shortening or extending the diagnostic warm-up, or changing the relative reward weights, affect the final results?
  3. Do the test results in Tables 1–4 use execution rewards for candidate selection, resampling, or fallback? Please specify the sampling counts, selection rules, and token limits for each role, and clarify the apparent inconsistency between the ≥20k critique-length bucket in Figure 5 and the configuration in Appendix A.1. Please also report the performance gains under matched computational budgets and the additional computational costs.
7 comments
AuthorsOpenReview ↗

Response to Reviewer tKU7 (Part 1 / 4)

We sincerely thank the reviewer for the thorough assessment and valuable suggestions. We particularly appreciate your recognition of WebLoop's clear motivation and practical significance in addressing the relatively underexplored challenge of optimizing intermediate critique through execution feedback. We are also encouraged by your positive assessment of our task-aligned supervision design, insightful mechanism and failure analyses, and comprehensive evaluations across different model scales, benchmarks, and input modalities.

To address your concerns, we provide additional experimental evidence, further analyses, and clarifications below.


Q1: How much do different Critics help a fixed Refiner? Do their rankings change across Refiners?

A1: To assess whether Critic feedback remains effective across Refiners with different capabilities, we conducted a Critic × Refiner cross-pairing experiment on WebRise. Base is the original Qwen3.5-9B model; WebLoop-20 and WebLoop-45 use the same backbone after joint training. Using the same Base-generated drafts (initial Overall = 30.18), we provide saved critiques from the Base, WebLoop-20, and WebLoop-45 Critics, unchanged, to both the Base and WebLoop-45 Refiners. WebLoop-20 and WebLoop-45 denote checkpoints at training steps 20 and 45, respectively. This setup isolates the effect of Critic feedback within each fixed-Refiner setting.

R3-Table 1: Critic feedback utility across fixed Refiners.

CriticBase Refiner OverallGain / rankWebLoop-45 Refiner OverallGain / rank
Base32.75+2.57 / 333.84+3.66 / 3
WebLoop-2033.75+3.56 / 234.84+4.66 / 2
WebLoop-4533.77+3.59 / 135.74+5.56 / 1

As shown in R3-Table 1, the feedback utility ranking remains consistent across both Refiners (WebLoop-45 > WebLoop-20 > Base). Compared with the Base Critic, WebLoop-45 feedback improves Overall by 1.02 points with the Base Refiner and 1.90 points with the WebLoop-45 Refiner, demonstrating its effectiveness even without a jointly trained Refiner. Although the Refiner affects the magnitude of these gains (e.g., the WebLoop-45–WebLoop-20 gap increases from 0.02 to 0.90 points), the ranking remains unchanged. Together with the independent diagnostic evaluation in Q4, these findings support the learned Critic's contribution to both requirement-level diagnosis and downstream refinement across different Refiners.

AuthorsOpenReview ↗

Response to Reviewer tKU7 (Part 2 / 4)

Q2: Why use a 15-step diagnostic warm-up and λ=0.2? How sensitive are the results to these choices?

A2: Our motivation for diagnostic warm-up is to establish the Critic’s ability to judge requirement satisfaction before optimizing downstream repair utility. Helpfulness depends on both the critique and the Refiner, so a failed repair does not necessarily indicate an incorrect diagnosis. We therefore begin with direct supervision from requirement-level execution labels, then introduce Helpfulness to encourage actionable feedback.

Based on our experience, we chose a 15-step warm-up and λ=0.2. All configurations use the same Qwen3.5-9B backbone, with Base denoting the original model. To assess sensitivity, we evaluated warm-up durations of 10, 15, and 20 steps and reward weights λ ∈ {0.1, 0.2, 0.4, 0.8} on both benchmarks.

R3-Table 2: Warm-up duration, with λ=0.2 fixed.

Configuration / outputWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Acc. ↑
Base23.1545.2922.1130.1818.3910.2067.0823.49
warm-up=10 (Gen)27.5349.7925.2834.2026.5810.8259.3531.99
warm-up=10 (Refine)33.0254.6130.1939.2727.9812.8356.4134.39
WebLoop (warm-up=15, Gen)31.7255.3830.0339.0528.7510.8256.8834.16
WebLoop (warm-up=15, Refine)34.5557.8931.9841.4831.6814.3748.3838.87
warm-up=20 (Gen)30.8654.0528.5937.8326.2810.5160.4331.53
warm-up=20 (Refine)33.6857.1030.9240.5731.5311.4452.8637.25

R3-Table 3: Reward weight, with a 15-step warm-up fixed.

Configuration / outputWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Acc. ↑
Base23.1545.2922.1130.1818.3910.2067.0823.49
λ=0.1 (Gen)29.0452.7526.5236.1021.7411.5963.3727.54
λ=0.1 (Refine)32.3755.7629.7639.2927.9812.6756.5734.31
WebLoop (λ=0.2, Gen)31.7255.3830.0339.0528.7510.8256.8834.16
WebLoop (λ=0.2, Refine)34.5557.8931.9841.4831.6814.3748.3838.87
λ=0.4 (Gen)27.1149.5124.7833.8021.178.0467.7025.19
λ=0.4 (Refine)30.1751.9327.3536.4829.0611.7553.6334.93
λ=0.8 (Gen)28.2751.7025.6935.2222.879.4363.8327.59
λ=0.8 (Refine)32.7355.6729.2739.2332.3011.9051.7838.25

The results show that the WebLoop configuration (15 steps, λ=0.2) achieves the highest initial-generation and final-refinement scores on both benchmarks among the tested settings. For warm-up duration, 15 steps improves final WebRise Overall and WebGen-Bench Acc. by 2.21/4.48 points over 10 steps and by 0.91/1.62 points over 20 steps. Similarly, λ=0.2 achieves the best final scores (41.48 Overall and 38.87 Acc.), while increasing or decreasing λ does not consistently improve performance.

AuthorsOpenReview ↗

Response to Reviewer tKU7 (Part 3 / 4)

Q2 (continued)

To further understand these differences, we evaluate the Critics from each configuration on the same held-out WebRise drafts with the WebLoop Refiner fixed, separating diagnostic accuracy from downstream repair and regression rates.

R3-Table 4: Critic warm-up sensitivity on fixed drafts (λ=0.2).

Critic configurationViolated-requirement accuracy ↑Satisfied-requirement accuracy ↑Repair rate ↑Regression rate ↓
warm-up=1045.6764.7913.008.92
WebLoop (warm-up=15)48.6761.5011.678.92
warm-up=2039.3362.7712.009.86

R3-Table 5: Critic reward-weight sensitivity on fixed drafts (15-step warm-up).

Critic configurationViolated-requirement accuracy ↑Satisfied-requirement accuracy ↑Repair rate ↑Regression rate ↓
λ=0.146.6754.4612.3313.62
WebLoop (λ=0.2)48.6761.5011.678.92
λ=0.450.6756.8113.678.92
λ=0.847.6755.409.337.51

An insufficient warm-up can leave the Critic underprepared to identify unmet requirements, whereas an overly long warm-up can limit its learning of feedback that guides actual repairs. With drafts and the Refiner fixed, 10 steps yields higher repair than 15 steps (13.00% vs. 11.67%), consistent with earlier Helpfulness supervision encouraging actionable suggestions. Yet violated-requirement accuracy is lower (45.67% vs. 48.67%), suggesting underdeveloped diagnosis. At 20 steps, satisfied-requirement accuracy improves slightly over 15 steps (62.77% vs. 61.50%), but violated-requirement accuracy drops to 39.33% and repair remains below the 10-step result (12.00% vs. 13.00%). Longer diagnosis-only training therefore does not consistently improve defect identification or repair guidance.

With insufficient diagnostic weight (λ), the Critic may flag satisfied requirements as defects and prompt damaging edits; excessive weight can weaken its guidance for actual repairs. Reducing λ from 0.2 to 0.1 increases the relative influence of Helpfulness: repair rises from 11.67% to 12.33%, but both diagnostic accuracies decline and regression increases from 8.92% to 13.62%. At λ=0.4, violated-requirement accuracy and repair reach 50.67% and 13.67%, respectively, while satisfied-requirement accuracy falls to 56.81%, revealing more false alarms despite stronger repair guidance. Increasing λ to 0.8 reduces the emphasis on Helpfulness, weakening the incentive for actionable suggestions; repair falls to 9.33% and regression to 7.51%, indicating fewer repairs alongside less damage.

Reviewer_2PCuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WebLoop, a framework for jointly training a Generator, Critic, and Refiner within a shared policy for functional web generation. The Generator and Refiner receive rewards from browser execution, while the execution-free Critic is trained using requirement-level diagnostic correctness and downstream refinement gains. Training initially emphasizes diagnostic discriminability and subsequently incorporates helpfulness, with role-specific comparison groups for joint GRPO optimization. Experiments on WebRise and WebGen-Bench demonstrate improvements in both first-pass generation and refined outputs, supported by ablations on training configurations and Critic rewards.

Critical strengths
  1. Meaningful improvements across benchmarks and model scales. At 9B, WebLoop improves WebRise Overall from 30.2 to 41.5 and WebGen-Bench accuracy from 23.5% to 38.9%. It also outperforms the untrained generate–critique–refine pipeline, which achieves 32.8 and 31.1%, respectively. Importantly, first-pass generation already improves to 39.1 and 34.2%, showing that the benefits extend beyond an additional refinement pass. Results at 27B and across input modalities further support the practical usefulness of the approach.

  2. Relevant ablations support the value of learning the full loop. The paper compares generation-only training, refinement without critique, staged versus joint training, and different Critic objectives. These controls address plausible alternative explanations for the gains. The improvement over refinement without critique is particularly substantial on WebGen-Bench: 38.9% versus 26.7%. The checkpoint cross-pairing analysis also shows that later Critic checkpoints improve refinement while holding the Generator/Refiner checkpoint fixed, providing additional evidence that learned critique contributes to performance.

Critical weaknesses
  1. The methodological novelty is incremental. The discriminability/helpfulness decomposition and staged Critic training follow Critique-RL, as acknowledged by the authors. The main contribution is combining these ideas with requirement-level execution supervision and joint optimization of the three roles for web generation. This is a useful integration, but it provides limited conceptual novelty in credit assignment itself. I therefore view the paper’s contribution primarily as an effective training framework and its empirical validation. Critique-RL

  2. The evidence is stronger for overall effectiveness than for some specific claims. Gains over close alternatives are relatively modest on WebRise: 41.5 versus 40.1 for staged training and 40.5 for discriminability-only training, without uncertainty estimates. Although the paper states that rollout budgets are matched, the allocation across variants and between stages is not sufficiently detailed. In addition, Figure 3 reports training diagnostic scores, but held-out diagnostic accuracy is not provided. Consequently, the claim that the Critic learns reliable diagnosis is less directly supported than the end-task performance improvements.

Questions for authors
  1. How exactly are rollout budgets matched across the Table 3 variants, particularly generation-only and staged training? How many updates are allocated to each stage of the latter?

  2. Are the gains over staged training and discriminability-only training consistent across repeated runs? Uncertainty estimates for these close comparisons would help assess their robustness.

  3. Can you report held-out Critic diagnostic performance, including separate accuracies for satisfied and violated requirements? This would help distinguish improvements in diagnostic correctness from improvements in downstream repair utility.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2PCu (Part 1 / 4)

We sincerely appreciate the reviewer’s detailed evaluation and insightful feedback. We are encouraged by your recognition of WebLoop’s meaningful improvements across benchmarks, model scales, and input modalities, particularly that these gains extend beyond refinement to initial generation. We also greatly appreciate your positive assessment of our comprehensive ablation studies and Critic checkpoint analyses, which provide evidence for the effectiveness of jointly learning generation, critique, and refinement.

In response to your valuable suggestions, we provide further controlled experiments, detailed analyses of Critic contributions, and clarifications on training configurations and computational costs below.


Q1: How are rollout budgets matched in Table 3? How many updates does each Staged phase use?

A1: To ensure a fair comparison, we match all Table 3 variants by the total number of browser evaluations used to obtain execution rewards (34,560). The detailed allocations are summarized below.

R2-Table 1: Allocation of execution-reward evaluation budgets.

Training schemeRollouts per taskTraining samplesExecution-reward evaluationsOptimizer updates
WebLoop3 drafts + 3 critiques + 9 refinements2,88034,560360
Refine w/o Critique3 drafts + 9 refinements2,88034,560360
Generation Only12 drafts2,88034,560360
Staged: stage 13 drafts2,8808,640360
Staged: stage 23 critiques + 9 refinements2,88025,920360
Staged: totalStage 2 uses frozen drafts from stage 15,76034,560720

Each execution-reward evaluation corresponds to a generated or refined application. Accordingly, WebLoop, Refine w/o Critique, and Generation Only each use 12 evaluations per training sample (2,880 × 12 = 34,560), whereas Staged Training allocates 8,640 evaluations to generation and 25,920 to refinement. With a task batch size of 64 and an optimizer mini-batch size of 8, each Staged phase uses 2,880 training samples and 360 optimizer updates, totaling 720 updates.

AuthorsOpenReview ↗

Response to Reviewer 2PCu (Part 2 / 4)

Q2: Are the gains over Staged and Discriminability-only training stable across repeated runs?

A2: To assess the stability of these performance gains, we evaluated all configurations in Tables 3–4 across three evaluation seeds, reporting the mean and standard deviation (SD) of each metric. All configurations use the Qwen3.5-9B backbone: Base is the original model, and WebLoop is our jointly trained model. All runs use the same 100-task WebRise subset and all 647 WebGen-Bench test items.

R2-Table 2: Training-scheme comparisons (mean ± SD).

MethodWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Start-Fail (%) ↓Acc. ↑
Base23.6±0.444.8±0.223.1±1.130.5±0.518.0±0.310.2±0.367.0±0.64.8±0.523.1±0.4
Generation Only (Gen)31.5±0.053.4±0.929.9±0.038.3±0.317.6±0.310.8±0.267.6±0.24.0±0.223.0±0.3
Refine w/o Critique (Gen)31.0±1.152.8±0.728.9±0.337.6±0.716.8±0.511.8±0.270.6±0.20.8±0.222.7±0.4
Refine w/o Critique (Refine)33.8±0.454.7±1.231.3±0.839.9±0.821.4±0.210.0±0.668.4±0.70.2±0.326.5±0.2
Staged Training (Gen)27.7±1.151.6±1.026.2±0.635.2±0.921.0±0.611.1±0.465.7±0.32.2±0.126.5±0.4
Staged Training (Refine)31.9±0.357.0±0.630.9±0.039.9±0.329.5±0.512.6±0.455.7±0.32.2±0.235.8±0.3
WebLoop (Gen)32.0±0.654.2±0.531.1±0.139.1±0.328.9±0.210.8±0.256.9±0.23.5±0.134.3±0.1
WebLoop (Refine)34.9±0.756.9±1.032.8±0.941.5±0.631.4±0.214.5±0.648.6±0.25.4±0.638.7±0.2

R2-Table 3: Critic-reward comparisons (mean ± SD).

MethodWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Start-Fail (%) ↓Acc. ↑
Base23.6±0.444.8±0.223.1±1.130.5±0.518.0±0.310.2±0.367.0±0.64.8±0.523.1±0.4
Discriminability Only (Gen)32.2±0.954.2±1.427.6±0.638.0±0.622.6±0.88.7±0.265.4±0.73.2±0.227.0±0.7
Discriminability Only (Refine)33.5±1.155.5±1.432.6±0.840.5±0.327.6±0.610.7±0.259.6±0.52.2±0.232.9±0.5
Helpfulness Only (Gen)27.9±0.850.9±1.327.8±0.135.5±0.618.0±0.28.7±0.368.1±0.25.2±0.522.4±0.3
Helpfulness Only (Refine)30.8±0.653.5±1.029.5±0.837.9±0.828.2±0.113.1±0.256.5±0.22.2±0.534.8±0.2
Mixed from Start (Gen)31.3±0.752.0±1.128.9±0.237.4±0.724.6±0.49.3±0.562.8±0.23.3±0.129.3±0.2
Mixed from Start (Refine)33.3±0.253.4±0.531.6±0.639.4±0.128.1±0.210.7±0.257.9±0.33.3±0.233.4±0.2
WebLoop (Gen)32.0±0.654.2±0.531.1±0.139.1±0.328.9±0.210.8±0.256.9±0.23.5±0.134.3±0.1
WebLoop (Refine)34.9±0.756.9±1.032.8±0.941.5±0.631.4±0.214.5±0.648.6±0.25.4±0.638.7±0.2

As shown in R2-Tables 2–3, the aggregate scores exhibit limited variation across evaluation seeds. In particular, WebLoop achieves 41.5 ± 0.6 on WebRise Overall and 38.7 ± 0.2 on WebGen-Bench Acc. Compared with Staged Training and Discriminability-only Training, WebLoop maintains higher average scores on both benchmarks, with gains of 1.6/1.0 points on WebRise Overall and 2.9/5.8 points on WebGen-Bench Acc., respectively. Together, these results indicate stable evaluation outcomes and consistent average performance advantages for the complete method across both ablation comparisons.

AuthorsOpenReview ↗

Response to Reviewer 2PCu (Part 3 / 4)

Q3: Can you report held-out diagnostic accuracy separately for satisfied and violated requirements?

A3: To distinguish diagnostic correctness from downstream repair utility, we conducted two complementary evaluations on WebRise: requirement-level diagnosis on fixed drafts and repair/regression analysis with a fixed Refiner. Base denotes the original Qwen3.5-9B model, and WebLoop denotes our jointly trained model with the same backbone, for draft generation, critique, and refinement alike. For diagnosis, we compare each Critic's judgments against browser execution labels, reporting accuracy separately for satisfied and violated requirements.

R2-Table 4: Requirement-level diagnosis on fixed drafts.

Draft sourceCriticViolated-requirement accuracy ↑Satisfied-requirement accuracy ↑
BaseBase34.8%46.6%
BaseWebLoop56.7%53.5%
WebLoopBase36.1%47.1%
WebLoopWebLoop53.5%55.2%

As shown in R2-Table 4, the WebLoop Critic consistently improves diagnostic accuracy across both draft sources. On Base/WebLoop-generated drafts, violated-requirement accuracy increases by 21.9/17.4 percentage points, while satisfied-requirement accuracy improves by 6.9/8.1 points. These results demonstrate more accurate requirement-level judgments independently of the Refiner's repair outcomes.

To further assess whether these diagnostic improvements translate into better refinement outcomes, we fix the corresponding Refiner within each draft group and vary only the Critic feedback, evaluating repair and regression rates over requirements.

R2-Table 5: Requirement repair and regression with drafts and Refiner fixed.

Draft source / fixed RefinerCriticRepair rate ↑Regression rate ↓
Base / BaseBase10.0%13.8%
Base / BaseWebLoop11.4%12.7%
WebLoop / WebLoopBase10.1%11.0%
WebLoop / WebLoopWebLoop13.1%10.1%

As shown in R2-Table 5, WebLoop Critic feedback consistently improves repair rates while reducing regressions under both fixed-Refiner settings. For Base/WebLoop drafts and their corresponding Refiners, repair rates increase by 1.4/3.0 percentage points, while regression rates decrease by 1.1/0.9 points, respectively. Therefore, these results demonstrate that the learned Critic provides both more accurate requirement-level diagnosis and more effective feedback for downstream repair, even when the Refiner is held fixed.

Reviewer_6iF7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WebLoop, which jointly trains a Critic via a shared policy to provide exclusively textual feedback without direct execution outcomes, enabling it to generate suggestions that genuinely facilitate subsequent refinement.

Critical strengths

The authors employ human auditing to enhance the reliability of execution contracts, with Table 5 showing reasonably strong agreement between the automatic evaluator and human reviewers. Additionally, the failure analysis in the appendix distinguishes among incorrect Critic judgments, incomplete repair guidance, and failures by the Refiner to implement correct suggestions. These examples help clarify where the improvement loop succeeds or fails.

Critical weaknesses

The contribution of Critic learning is not fully substantiated. On WebRise, the improvements over the most relevant ablations are relatively small. As the authors acknowledge in Appendix C.4, downstream helpfulness depends on both the Critic and the Refiner. A strong Refiner may repair the code even with weak feedback, while a correct critique may receive a low reward if the Refiner fails to act on it. The current experiments do not fully establish how much additional value the learned critique provides.

Questions for authors

Could the authors report results over multiple seeds, particularly for Tables 3 and 4?

Could the authors evaluate a fixed, trained Refiner under several settings: without critique, with its own Critic's feedback, with critiques from other tasks, and with verified correct critiques?

Why does Critique-GRPO perform worse than the base model on WebGen-Bench, despite using a teacher with access to execution feedback? Was this baseline tuned comparably to WebLoop?

Minor comments:

Page 4, Section 3.1: a missing period and space in “initial HTML implementationThe Critic”. Page 16, Appendix B.2: double comma in “manual filtering,,”.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 6iF7 (Part 1 / 4)

We sincerely thank the reviewer for the thoughtful assessment and valuable suggestions. We greatly appreciate your recognition of our efforts to ensure reliable execution-based evaluation through human auditing, as well as your positive assessment of our fine-grained failure analysis, which provides insights into the successes and limitations of the Critic–Refiner improvement process.

To address your concerns, we provide additional controlled experiments, further failure analyses, and methodological clarifications below.


Q1: Results across multiple random seeds

A1: We additionally evaluated all configurations in Tables 3–4 using three evaluation seeds, reporting the mean and standard deviation (SD) of each metric. All configurations use Qwen3.5-9B as the backbone, with Base denoting the original model and WebLoop denoting our jointly trained model. Evaluations use the same 100-task subset of WebRise and all 647 test items of WebGen-Bench across runs.

R1-Table 1: Training-scheme comparisons (mean ± SD).

MethodWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Start-Fail (%) ↓Acc. ↑
Base23.6±0.444.8±0.223.1±1.130.5±0.518.0±0.310.2±0.367.0±0.64.8±0.523.1±0.4
Generation Only (Gen)31.5±0.053.4±0.929.9±0.038.3±0.317.6±0.310.8±0.267.6±0.24.0±0.223.0±0.3
Refine w/o Critique (Gen)31.0±1.152.8±0.728.9±0.337.6±0.716.8±0.511.8±0.270.6±0.20.8±0.222.7±0.4
Refine w/o Critique (Refine)33.8±0.454.7±1.231.3±0.839.9±0.821.4±0.210.0±0.668.4±0.70.2±0.326.5±0.2
Staged Training (Gen)27.7±1.151.6±1.026.2±0.635.2±0.921.0±0.611.1±0.465.7±0.32.2±0.126.5±0.4
Staged Training (Refine)31.9±0.357.0±0.630.9±0.039.9±0.329.5±0.512.6±0.455.7±0.32.2±0.235.8±0.3
WebLoop (Gen)32.0±0.654.2±0.531.1±0.139.1±0.328.9±0.210.8±0.256.9±0.23.5±0.134.3±0.1
WebLoop (Refine)34.9±0.756.9±1.032.8±0.941.5±0.631.4±0.214.5±0.648.6±0.25.4±0.638.7±0.2

R1-Table 2: Critic-reward comparisons (mean ± SD).

MethodWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Start-Fail (%) ↓Acc. ↑
Base23.6±0.444.8±0.223.1±1.130.5±0.518.0±0.310.2±0.367.0±0.64.8±0.523.1±0.4
Discriminability Only (Gen)32.2±0.954.2±1.427.6±0.638.0±0.622.6±0.88.7±0.265.4±0.73.2±0.227.0±0.7
Discriminability Only (Refine)33.5±1.155.5±1.432.6±0.840.5±0.327.6±0.610.7±0.259.6±0.52.2±0.232.9±0.5
Helpfulness Only (Gen)27.9±0.850.9±1.327.8±0.135.5±0.618.0±0.28.7±0.368.1±0.25.2±0.522.4±0.3
Helpfulness Only (Refine)30.8±0.653.5±1.029.5±0.837.9±0.828.2±0.113.1±0.256.5±0.22.2±0.534.8±0.2
Mixed from Start (Gen)31.3±0.752.0±1.128.9±0.237.4±0.724.6±0.49.3±0.562.8±0.23.3±0.129.3±0.2
Mixed from Start (Refine)33.3±0.253.4±0.531.6±0.639.4±0.128.1±0.210.7±0.257.9±0.33.3±0.233.4±0.2
WebLoop (Gen)32.0±0.654.2±0.531.1±0.139.1±0.328.9±0.210.8±0.256.9±0.23.5±0.134.3±0.1
WebLoop (Refine)34.9±0.756.9±1.032.8±0.941.5±0.631.4±0.214.5±0.648.6±0.25.4±0.638.7±0.2

As shown in R1-Tables 1–2, the results exhibit limited variation across three evaluation seeds, with standard deviations of 0.1–0.9 percentage points for WebRise Overall and 0.1–0.7 for WebGen-Bench Acc. Notably, WebLoop (Refine) maintains the highest average scores on both benchmarks across the two ablation studies, achieving 41.5 Overall on WebRise and 38.7 Acc. on WebGen-Bench. These results demonstrate the stability of our findings across evaluation seeds while consistently supporting the effectiveness of joint training and the complete Critic reward design.

AuthorsOpenReview ↗

Response to Reviewer 6iF7 (Part 2 / 4)

Q2: Refinement gains from different feedback conditions with a fixed Refiner

A2: To assess whether Critic feedback provides additional refinement gains with a fixed Refiner, we evaluated four feedback conditions (without critique, own Critic feedback, critiques from other tasks, and verified correct critiques) under two settings: Base (original Qwen3.5-9B) and WebLoop (jointly trained from the same backbone). Within each setting, the generated drafts and Refiner are held fixed across feedback conditions. Both settings are evaluated on WebRise and WebGen-Bench.

R1-Table 3: Base: Feedback conditions on both benchmarks.

Feedback conditionWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Start-Fail (%) ↓Acc. ↑
Initial draft23.245.322.130.218.410.267.14.323.5
Without critique26.949.225.233.818.410.766.84.223.7
Own Critic feedback25.847.924.632.826.39.659.24.931.1
Critiques from other tasks23.745.222.730.518.211.364.56.023.9
Verified correct critiques30.052.827.336.7—————

R1-Table 4: WebLoop: Feedback conditions on both benchmarks.

Feedback conditionWebRiseWebGen‑Bench
T ↑Rₑ ↑Rᵢ ↑Overall ↑Yes (%) ↑Partial (%) ↑No (%) ↓Start-Fail (%) ↓Acc. ↑
Initial draft31.755.430.039.128.710.856.93.634.2
Without critique33.155.830.539.828.310.259.71.933.4
Own Critic feedback34.657.932.041.531.714.448.45.638.9
Critiques from other tasks32.956.130.839.922.111.663.72.627.9
Verified correct critiques38.661.535.345.1—————

Note: Because the task design of WebGen-Bench makes it difficult to reliably obtain verified correct critiques, this condition is indicated by “—”.

As shown in R1-Tables 3–4, both groups show that task-specific Critic feedback is more helpful for refinement. Feedback from the corresponding Critic consistently outperforms critiques from other tasks, improving WebRise Overall by 2.3/1.6 points and WebGen-Bench Acc. by 7.2/11.0 points for Base/WebLoop, respectively. Verified correct critiques further achieve the highest WebRise Overall scores (36.7/45.1), highlighting the value of reliable, task-relevant feedback.

More importantly, the within-group comparisons further show that the jointly trained WebLoop can provide useful feedback through its own Critic. Compared with refinement without critique, WebLoop’s Critic feedback improves WebRise Overall from 39.8 to 41.5 and WebGen Acc. from 33.4 to 38.9. By contrast, although the Base Critic, without this training, improves WebGen Acc. from 23.7 to 31.1, it does not outperform refinement without critique on WebRise (32.8 versus 33.8 Overall), highlighting the limitations of the untrained Critic’s feedback capabilities.

These results demonstrate that WebLoop's learned Critic provides consistent additional refinement gains across both benchmarks.

AuthorsOpenReview ↗

Response to Reviewer 6iF7 (Part 3 / 4)

Q3: Why does Critique-GRPO underperform Base on WebGen-Bench? Was it tuned?

A3: We adapted Critique-GRPO to our task and tuned its hyperparameters. However, its WebGen-Bench Acc. is below Base (17.6% vs. 23.5%). To investigate this gap, we conducted a failure analysis of the generated applications and compared their initially generated applications with those of WebLoop.

R1-Table 5: Failure outcome breakdown on WebGen-Bench.

Metric / outcome categoryBaseCritique-GRPO
WebGen-Bench Acc.23.50%17.60%
Code executability issues: startup failures, blank pages, or error pages61.30%64.60%
Functional implementation: failed tests16.80%19.00%
Functional implementation: partially passed tests5.30%4.60%
Correctly rendered page with passed tests16.70%11.70%

As shown in R1-Table 5, code executability issues constitute the dominant failure category, affecting 64.6% of Critique-GRPO applications versus 61.3% for Base. To identify the underlying causes, we further break down these executability failures.

R1-Table 6: Breakdown of Critique-GRPO executability issues.

Critique-GRPO executability issueShare of affected applications
Blank page57.60%
Dependency or file-reference error24.20%
Syntax or compilation error9.10%
Other runtime error7.60%
Application startup failure1.50%

As shown in R1-Table 6, blank pages (57.6%) and dependency or file-reference errors (24.2%) account for most executability failures. Our inspection revealed missing dependencies and build configurations. For example, the recruitment task imports react-router-dom without declaring the dependency, while the to-do list task lacks vite.config.js, resulting in an improperly rendered page. These cases highlight the importance of code completeness and configuration consistency. To further examine these executability issues across methods, we compare the initially generated applications from WebLoop, Base, and Critique-GRPO before refinement:

R1-Table 7: Comparison of initial application outcomes before refinement.

Initial application outcomeBaseCritique-GRPOWebLoop
Correctly rendered interface39.60%34.70%66.30%
Blank page41.60%37.60%24.80%
Dependency or file-reference error5.00%15.80%3.00%

As shown in R1-Table 7, WebLoop achieves a substantially higher proportion of correctly rendered interfaces (66.3%) than Base (39.6%) and Critique-GRPO (34.7%), even before refinement. It also produces fewer blank pages and dependency errors. For example, in the paper-cup production management task, Base references a missing main.jsx, whereas WebLoop includes the required entry file and renders successfully.

We think these differences may be attributed to their training mechanisms. Critique-GRPO relies on an external teacher for critiques while optimizing only the Actor. In contrast, WebLoop jointly trains generation, requirement-level diagnosis, and repair through execution supervision and repair feedback. This integrated learning process can strengthen the model's ability to identify implementation errors and maintain code completeness, thereby reducing executability failures even during initial generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.