acceptodds
Desk rejected at ICLR 2027

StepWeb: Trajectory-Conserving Step-Level Credit for Multi-Turn Web Refinement

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-turn webpage refinement requires agents to improve their own implementations while operating on states shaped by earlier edits. Terminal rewards can identify successful trajectories, but provide limited information about which intermediate decisions caused regression, recovered lost quality, or established genuine progress. We introduce STEPWEB, a trajectory-aware reinforcement learning framework that separates terminal trajectory comparison from within-trajectory credit assignment. Its trajectory-conserving step-level credit (TC-SLC) assigns history-aware, magnitude-sensitive signals to Regression, Repair, and New Gain events, and redistributes them across refinement turns through token-length-weighted centering while preserving the trajectory’s mean advantage before policy clipping. Experiments on Vision2Web Level 1, Design2Code, Design2Code-HARD, and Interaction2Code show consistent improvements over the Qwen3.5-9B backbone, yielding a 16.4% macro-averaged relative gain across the four benchmarks. In controlled multi-turn refinement on Vision2Web Level 1, TC-SLC further reduces the edit regression rate by 29.5% relative to terminal-only RL while improving final reconstruction quality. These results demonstrate the effectiveness of trajectory-local credit assignment for reliable iterative visual coding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Coding, Agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_iuyUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the challenge of assigning credit to intermediate edits from terminal rewards in multi-turn webpage refinement, the authors propose STEPWEB. The key component, trajectory-conserving step-level credit assignment, categorizes edits as Regression, Repair, or New Gain and redistributes credit across refinement turns. The method preserves the token-weighted mean advantage of the original trajectory before policy clipping, with the goal of providing more informative step-level supervision and improving final reconstruction quality while reducing edit regression.

Critical strengths
  1. TC-SLC distinguishes among Regression, Repair, and New Gain, and redistributes credit based on existing scores and token counts. This avoids additional interactions and does not require a learned value model, making the approach relatively simple and lightweight.

  2. Compared with Terminal RL, STEPWEB achieves higher final scores and lower edit regression rates while starting from comparable initial scores. The ablation studies further provide evidence for the contribution of the main components.

Critical weaknesses
  1. The paper does not provide a systematic discussion of existing step-level credit assignment approaches. Although several related works are cited, the methodological differences between TC-SLC and these approaches remain unclear. More importantly, the experiments do not include comparisons with closely related methods, such as WebGen-Agent [1]. As a result, it is difficult to assess how much of the reported improvement comes specifically from TC-SLC rather than from other aspects of the overall framework. Comparisons with representative step-level methods would help better establish the paper's contribution.

  2. The Avg score for STEPWEB in Table 1 appears to be inconsistent with the stated averaging procedure across the three viewports. Specifically, the mean of 45.2, 39.0, and 32.4 is approximately 38.9, whereas Table 1 reports 37.7. Appendix Table 8 reports another value, 39.9, without providing an explanation for the discrepancy. These inconsistencies make it difficult to verify the aggregate results and the reported relative improvements, and raise concerns about the reliability of the experimental reporting.

[1] Lu, Zimu, et al. "Webgen-agent: Enhancing interactive website generation with multi-level feedback and step-level reinforcement learning." ICLR 2026.

Questions for authors
  1. Could the authors clarify how the Vision2Web Avg score for STEPWEB is calculated and explain the discrepancy between the values reported in Table 1 and Appendix Table 8? More generally, it would be helpful if the authors could carefully check the reported numerical results throughout the paper and ensure that the tables and text are fully consistent.

  2. Could the authors clarify how TC-SLC differs from existing step-level credit assignment methods? In particular, comparisons with closely related approaches under the same backbone and evaluator would help isolate the contribution of TC-SLC and better demonstrate its advantage over existing methods.

Reviewer_Pkk5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

StepWeb proposes a step-level credit-assignment method for reinforcement learning in multi-turn screenshot-to-code refinement. Each sampled trajectory receives a terminal group-relative advantage. Refinement actions additionally receive local credit based on whether their score changes represent a regression, a repair that remains below the historical best, or a new gain beyond that best. The local terms are centered using token-length weights, preserving the trajectory’s weighted mean advantage before clipping.

The method is applied to a Qwen3.5-9B policy with supervised initialization and iterative refinement using rendered screenshots, scores, and feedback from a frozen model. Experiments cover Vision2Web, Design2Code, and Interaction2Code, with controlled ablations on Vision2Web examining terminal-only RL, centering, event weights, and score magnitudes. The main empirical claim is that the proposed local credit improves final visual fidelity while reducing regressions during refinement.

Critical strengths
  1. The credit rule is clear, inexpensive to implement, and addresses a meaningful distinction in iterative editing. Recovering from an earlier mistake and exceeding the best previous solution are different events, even when both produce positive immediate score changes. The proposed decomposition captures this distinction without learning a separate critic. The weighted-centering identity is straightforward and correctly scoped to advantages before clipping; the appendix explicitly avoids claiming conservation of the clipped objective or parameter update.

  2. The controlled terminal-only comparison provides substantive evidence for the proposed credit assignment. Table 3 compares Terminal RL and StepWeb under the same feedback, rollout horizon, rendering, and sampling setup. Final performance increases from 35.2 to 37.7, while the regression rate decreases from 28.1% to 19.8%, corresponding to the reported 29.5% relative reduction. The initial-turn difference is much smaller, 33.5 versus 33.8, which supports an effect on refinement rather than only a better starting policy. The centering, repair-weight, and magnitude ablations further support the usefulness of the design choices.

Critical weaknesses
  1. The main reported results contain a numerical inconsistency that directly affects the headline improvement. Table 1 reports StepWeb viewport scores of 45.2, 39.0, and 32.4. Under the averaging rule in Appendix C, their mean is 38.87, but Table 1 lists 37.7 and Appendix Table 8 lists 39.90. Recomputing the four-benchmark relative improvement using the Table 1 viewport scores and the stated aggregation rule gives approximately 15.6%, rather than the claimed 16.4%. The appendix’s explanation that the main and ablation tables use separate checkpoints does not resolve this internal Table 1 discrepancy or its disagreement with Table 8. This could be a reporting error, but the corrected underlying results are necessary to assess the headline claim. Table 7 specifies three training seeds, yet the presentation does not make clear how seeds are aggregated or provide uncertainty for the main method differences.

  2. The operational definition of the training score and the attribution of broader gains remain insufficiently documented. Equation 21 in Appendix A.3 averages normalized per-viewport scores, but I could not identify a complete specification of how each viewport score is computed, including the evaluator or metric, preprocessing, and any learned-model configuration. This score determines both terminal ranking and every local event label, so it is essential to reproducing the algorithm. In addition, the main cross-benchmark results combine supervised training, feedback, iterative inference, and RL, while the clean Terminal RL control is reported only on Vision2Web. The paper cites the closely related WebGen-Agent Step-GRPO approach, which also assigns screenshot-derived step-level credit, but does not isolate the incremental value of StepWeb against that alternative under the existing controlled setup. These omissions limit confidence in how much of the broader improvement is specifically due to the proposed credit rule.

Questions for authors
  1. [Clarification] What are the corrected StepWeb Vision2Web average and four-benchmark macro improvement? Please reconcile Table 1, Table 8, and the headline 16.4% claim, and explain checkpoint selection, seed aggregation, and uncertainty for the main and controlled results.

  2. [Clarification] Please provide the full operational definition of the per-viewport score in Equation 21: metric or evaluator, model/checkpoint if applicable, input preprocessing, normalization, and aggregation. What is its relationship to the final evaluation metrics, and is the same scalar score available to the policy during test-time refinement?

  3. [Clarification] Please specify the supervised and RL training-set sizes, total RL update budget, and the inference/refinement/feedback budgets of the main baselines. Which broader benchmark results are available for the supervised initialization and matched Terminal RL conditions?

  4. [Assessment-relevant validation] In the existing matched rollout setup, how does StepWeb compare with a Step-GRPO-style step-score normalization rule using the same backbone, training data, feedback, and sampling budget? This comparison would help establish the contribution of trajectory centering and history-dependent event credit relative to the closest existing alternative.

Reviewer_QVShOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StepWeb, a reinforcement learning framework for multi-turn webpage refinement, aiming to address the credit assignment problem in iterative webpage editing. Existing terminal-reward methods evaluate the final outcome but provide limited feedback on the contribution of individual editing steps.

The main contribution is Trajectory-Conserving Step-Level Credit (TC-SLC), which combines terminal trajectory-level advantages with history-aware step-level credit assignment. Specifically, editing steps are classified into three categories: Regression, Repair, and New Gain. The method assigns magnitude-sensitive credits to these events and redistributes them across refinement steps using token-length-weighted centering, while preserving the trajectory's mean advantage.

The authors evaluate StepWeb on four benchmarks: Vision2Web Level 1, Design2Code, Design2Code-HARD, and Interaction2Code. Using Qwen3.5-9B as the backbone, the method reports a 16.4% macro-averaged relative improvement across the four benchmarks. Additional ablation experiments show that StepWeb reduces the edit regression rate by 29.5% relative to terminal-only reinforcement learning, while improving final webpage reconstruction quality.

Critical strengths
  1. Relevant research problem. The paper addresses the step-level credit assignment problem in multi-turn webpage refinement. Distinguishing between Regression, Repair, and New Gain is a reasonable approach to capturing the effects of individual edits, particularly when later modifications depend on earlier states.

  2. Empirical evaluation and ablation studies. The paper evaluates the proposed method on four benchmarks and includes controlled comparisons with terminal-only reinforcement learning. The ablation studies examine several components of the credit assignment mechanism, providing some evidence for the effectiveness of the proposed design.

Critical weaknesses
  1. Limited methodological novelty. The proposed TC-SLC method mainly combines terminal trajectory advantages with heuristic step-level rewards based on score changes. Although distinguishing Regression, Repair, and New Gain is reasonable, the core idea of assigning intermediate rewards is not fundamentally new. The paper does not sufficiently establish how this formulation differs from or improves upon existing process-level credit assignment methods in reinforcement learning.

  2. Insufficient comparisons and inconsistencies in reported results. While the paper includes ablation studies, the key comparison is primarily against terminal-only RL and simplified variants of the proposed method. More direct comparisons with existing step-level credit assignment approaches would help establish the contribution of TC-SLC. Additionally, Table 1 reports Vision2Web scores of 45.2, 39.0, and 32.4 for the three viewports, but lists an average of 37.7, which does not match their arithmetic mean. Such inconsistencies raise concerns about the reliability and clarity of the reported experimental results.

Questions for authors
  1. Novelty of the credit assignment mechanism. How does TC-SLC fundamentally differ from existing process-level or step-level credit assignment methods in reinforcement learning? Could the authors clarify the specific advantages of the proposed history-aware event classification and trajectory-conserving redistribution over existing approaches?

  2. Effectiveness of the proposed credit assignment. The ablation studies mainly compare TC-SLC with terminal-only RL and simplified variants of the proposed method. How can the authors establish that the observed improvements are specifically attributable to TC-SLC rather than to the general introduction of intermediate reward signals? Would comparisons with existing step-level credit assignment methods lead to similar improvements?

  3. Consistency of experimental results. In Table 1, StepWeb achieves scores of 45.2, 39.0, and 32.4 on the desktop, tablet, and mobile settings of Vision2Web, respectively, but the reported average is 37.7. The arithmetic mean of these three scores is approximately 38.87. Could the authors clarify how the average is calculated and verify the consistency of the reported results?

Reviewer_32TqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEPWEB treats multi-turn screenshot-to-code refinement as a dependent trajectory. Terminal webpage scores rank complete rollouts via a group-relative advantage; each refinement transition is then labeled against the trajectory's running best as Regression, Repair, or New Gain, weighted by |Delta s|, and centered within the trajectory using token-length weights so the token-weighted mean advantage still equals the trajectory advantage (TC-SLC). Reported: +16.4% macro-averaged relative gain over a Qwen3.5-9B backbone across four benchmarks, and a 29.5% relative reduction in edit regression rate versus terminal-only RL.

Critical strengths

The core observation is correct and underappreciated: same-index intermediate states across independently sampled trajectories are not state-matched, so group-normalizing them confounds edit quality with inherited state. Restricting group normalization to terminal outcomes follows cleanly. The Repair / New Gain distinction — conditioning the local signal on the trajectory's running best rather than only the sign of Delta s — is genuinely novel and well-motivated for a setting where regressions are inherited. The method adds no inference machinery and no privileged information, and Appendices A-B are unusually thorough on protocol, failure handling, and asset provenance.

Critical weaknesses

W1 — the headline number contradicts the paper's own main table. Recomputing Table 1's Vision2Web Avg as (Desk+Tab+Mob)/3, per Appendix C, all 15 baseline rows reconcile within 0.04; the STEPWEB row does not. The same quantity appears as 38.87 (recomputed from 45.2/39.0/32.4), 37.7 (the printed Avg cell), and 39.90 (the value Table 8 uses to produce +21.4% and the 16.4% headline). This is outcome-determinative: Qwen3.8-27B scores 39.73 in that column, so STEPWEB wins by 0.17 under 39.90 but loses by 0.86 or 2.03 under the other two.

W2 — the 27B model that generates STEPWEB's feedback is itself a baseline row, and no baseline's feedback source is disclosed. Appendix A.3 states the frozen feedback generator is Qwen3.8-27B, which also appears in Tables 1-2 — yet Section 5.1 labels VisRefiner-7B ("single-step" / "self-refinement"), UI2CodeN ("SFT" / "RL") and ScreenCoder ("agentic" / "finetuned") while attaching no mode, or turn count to any Qwen3.x row. Placed side by side, STEPWEB leads the teacher by +0.26 on Design2Code, +0.12 on Design2Code-HARD, and trails on Vision2Web — margins too small on a 0-100 scale to establish a capability gain. Appendix F’s claim that VLM feedback is “held fixed” scopes only to the paper’s internal Terminal RL ablation, and does not cover the external refinement baselines.

W3 — the only mechanism-isolating experiment is one benchmark, one scalar per variant, zero variance. Section 5.1 concedes the main-table gains blend SFT with RL, so attribution rests on Table 3's Terminal RL 35.2 -> STEPWEB 37.7. That ladder spans only a small point range in per-rung steps of well under one point, while Table 7 reports 3 training seeds and Table 3 gives one scalar per variant without stating whether it is a mean, a single seed, or a best run.

W4 — Table 5's Repair sign appears to contradict Eq. 14, and the Adv. Shift column is undefined. Eq. 14 weights a Repair by lambda_repair = 0.5, halving a gain already smaller than New Gain's. Eq. 15 then subtracts a per-trajectory mean that, by Table 5's own frequencies, is necessarily clearly positive — New Gain alone contributes 0.43 x 2.0 = +0.86, against -0.36 from all regressions — so a Repair worth only +0.5 sits below the typical transition on its trajectory. Read at global frequency, the centered Repair credit is therefore negative, yet Table 5 reports a positive Adv. Shift for Repair. Because the true centering is per-trajectory and token-weighted, this is not proof of an error; but the paper never says how the Adv. Shift column is computed, so the sign cannot be checked. Please give the definition and the per-event mean and standard deviation.

Rating justification

The problem framing is sound and the Repair / New Gain idea is worth keeping, but two issues bear directly on the conclusions. The headline 16.4% rests on a Vision2Web value that contradicts Table 1 and is the most favorable of three incompatible figures; recomputing from Table 1’s own Desk./Tab./Mob. gives 15.6%, and from its printed Avg cell 14.7%, and STEPWEB turns from beating Qwen3.8-27B to losing to it — though the reverse also holds if the corrected cell turns out to be Mobile, so the comparison cannot be settled until the authors fix the table. And once the paper's own 27B feedback teacher enters the comparison, the pipeline leads the larger model already in its loop by 0.12, 0.26 and 0.46 points on a common 0-100 scale across three benchmarks, and loses on the fourth -- a margin that does not support the capability claim, especially since the 27B row's own protocol is never stated, and none of the refinement baselines' feedback sources are disclosed either. STEPWEB's own cost is at least six generation calls, five critic calls and six renders per trajectory, against whatever the single-pass baselines use. With a single-benchmark ablation that reports no variance despite three trained seeds, a substantive revision is needed. This is a judgment about the evidence chain, not the idea: if the authors resolve the inconsistency, add the compute-matched teacher comparison, and provide multi-benchmark ablations with variance, I would gladly reassess upward.

Questions for authors
  1. Which of STEPWEB's three Vision2Web figures is wrong — Desk./Tab./Mob. (45.2/39.0/32.4, mean 38.87), the printed Avg (37.7), or the 39.90 used in Table 8? Please give the corrected values and the corrected g_macro.
  2. What is the exact evaluation protocol for every Qwen3.x row in Tables 1-2 (number of turns, harness, sampling)? And under matched inference compute, what is the net gain over the frozen Qwen3.8-27B self-refining for five turns in the same pipeline?
  3. Please report the three per-seed values for every Table 3 variant, stating whether the variants share seeds and so admit a paired comparison, together with a per-task bootstrap confidence interval on Final for the Terminal RL vs. STEPWEB gap.
  4. Why is Repair's Adv. Shift positive, and how is that column computed?
  5. Does the Terminal RL vs TC-SLC result hold on Design2Code and Interaction2Code?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.