- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents suftPRM, a graph-guided framework for synthesizing process preference data for web PRMs. The authors argues that the scarcity of Grounded Minimal Contrastive Pairs (GMCPs) in existing web PRM dataset is a key deficiency, suggesting that many prior preference pairs contain shaloow cues such as hallucinated elements or mismatched action types. The proposed surfPRM aggregates successful web trajectories into an Interaction Element Graph and use its spatial, temporal, and spatiotemporal structure to construct grounded negative actions. Across several backbone models, PRMs trained on surfPRM-data outperform models trained on the WebArbiter baseline on WebPRMBench. The paper further shows that surfPRM outperforms PRM baselines as a test-time trajectory search verifier on WebArena-Lite.
Critical strengths
None
Critical weaknesses
-
The paper does not fully establish that GMCP scarcity is the key deficiency of existing Web PRM data.
- The authors identifies the low proportion of GMCP in training data as a central problem and presents it as a main contribution. However, it seems that GMCP is primarily a structural property of a preference pair rather than a direct measure of its difficulty or informativeness. A pair can satisfy GMCP definition simply by pairing the positive action with another grounded element that supports the same action type, even when the negative is trivially distinguishable from the positive.
- Most importantly, the paper does not directly verify whether the proposed confusions produce sufficiently challenging nagatives under the given instruction and state. For example, the Structurally Near confusion is based on AXTree proximity, but structural proximity does not imply semantic confusability. The example of choosing "Feb 25" instead of "Jan 28" may be trivial if the instruction specifies Jan 28. Similarly, temporal confusions may generate valid actions, since interacting with the same target multiple times is not inherently incorrect.
-
The paper treats each action in a successful trajectory as the preferred action against synthesized alternatives, while the downstream validation remains limited.
- However, a successful trajectory is not unique, and web tasks often admit multiple valid next actions and multiple successful paths. This is particularly important for confusions such as Act Too Early, Away from Goal, and Create a Cycle, whose negative labels depend on the successful trajectory.
- The teacher verification reduces false-negative rate from 30% to 12%, but 12% remains substantial for preference supervision. Moreover, surfPRM simultaneously changes several properties of the training data, including whether negatives are grounded in the current observation, how negatives are selected, the difficulty of the pairs, and the false-nagative rate. As a result, it is difficult to determine how much of the observed gain is due to the proposed graph-guided contrastive synthesis rather than simply cleaner preference labels.
- Also, the downstream evaluation is narrow given that the motivation for PRMs is to provide useful signals for policy learning. The authors evaluate only WebArena-Lite and two GPT policies with only Best-of-N. It is therefore unclear whether the gains transfer to other policy families or benchmarks. The regression on GitLab also suggests that the proposed PRM is not consistently beneficial across domains.
Questions for authors
-
Does task success imply that each demonstrated action is locaaly preferable to other valid actions available at the same state? Since web tasks can have diverse successful paths, it would be helpful to clarify whether the surfRPM-data assumes any form of local optimality or uniqueness of the successful trajectory.
-
The teacher verification appears unusually permissive. A pair is retained if the teacher agrees with the positive label in any one of up to five attempts, even if the remaining judgements prefer the synthesized negative. It seems there are many alternatives such as majority voting. It would be useful to provide the distribution of teacher votes and, if possible, an ablation over different acceptance thresholds, such as 1/5, 3/5, or 5/5 agreement, together with their effects on the human false-negative rate and downstream performance.
3 comments
Official Comment by Authors (Part 2/2)
Q2. Teacher filtering thresholds
The paper already specifies the filtering rule in Teacher Verification and Gold Supervision Synthesis: a candidate is discarded only if all five teacher judgments disagree with the trajectory label. It also reports first-attempt and retry acceptance counts in the appendix. We agree that this permissive rule warrants a direct comparison with stricter alternatives.
We therefore added the ablation in General Response S2, using five independently generated teacher judgments for each of 1,000 sampled candidates. Single-pass, majority-vote, unanimity, and any-of-five filtering retained 961, 955, 937, and 972 pairs, respectively. Their downstream results were broadly comparable, although performance varied across benchmarks and metrics.
We also audited all 11 retry-accepted pairs; 2 were false negatives. This higher observed rate supports the concern that retries can admit incorrect alternatives, although the small number of pairs makes the estimate uncertain. At the same time, we believe that in practical applications, when cost-sensitive, combined with our data synthesis pipeline, directly using single-pass can also achieve excellent results.
Official Comment by Reviewer 9wLc
Thank you for the clarification. My main concern is the central role that GMCP plays in the paper's movitation and claimed problem diagnosis. The authors agree that GMCP is a structural property rather than a direct measure of difficulty or informativeness. This makes it less clear why the scarcity of GMCP should be considered a key deficieny of existing web prm data, as emphasized throughout the paper.
The empirical results also seem to suggest that GMCP ratio alone does not provide a clear characterization of preference data quality. Increasing the GMCP ratio does not lead to monotonic improvements across metrics, particularly for BoN accuracy, and the no-graph baseline has a slightly higher GMCP ratio than surfPRM while performing worse. Moreover, the authors intentionally do not aim for 100% GMCP because some non-GMCP pairs are useful or necessary, which further suggests that there is no clear notion of what constitutes an appropriate GMCP ratio.
These results appear to support a dfferent and, in my view, more direct conclusion: surfPRM is effective because its graph-guided synthesis produces better grounded and contextually meaningful negative actions, while GMCP ratio is one descriptive property of the resulting dataset. I therefore remain unconvinced that low GMCP ratio has been established as the key deficiency of prior webprm data. Given how strongly the current motivation and contribution statements are organized around GMCP scarcity, I think the paper would benefit from substantially revising this framing and treating GMCP as one diagnostic statistic rather than the central explanation for the observed improvements.
I appreciate the authors for the detailed responses. However, my main concern remains so I am inclined to maintain my initial score.
Official Comment by Authors (Part 1/2)
Thank you for the review.
GMCP is a structural property, not a difficulty score. As defined in the paper’s Data Collection and Dataset Statistics section and appendix, a GMCP requires both targets to be present and uniquely resolvable in the current observation, and both actions to have the same operation type. Our motivation is that such pairs are more closely aligned with the decision setting in which a PRM must distinguish available actions using the task and page context. We agree that these conditions alone do not guarantee that a pair is difficult, informative, or correctly labeled. In particular, structural proximity in the AXTree does not by itself prove semantic confusability.
The task instruction is explicitly included in the state supplied to both the teacher and the PRM, as specified in the Problem Formulation and Comparative PRM Pipeline section and the prompts in the appendix. In the reviewer’s date example, whether selecting “Feb 25” is an appropriate alternative depends on the instruction and observed state. We agree that the comparison may be straightforward when the instruction explicitly specifies “Jan 28”; our claim does not require every proposed pair to be difficult.
The paper also already acknowledges the possibility of valid alternative actions. In Teacher Verification and Gold Supervision Synthesis, we identify an equally valid alternative path as a potential false negative and use teacher judgments to filter candidates. More importantly, the Quality Assurance and Iterative Refinement section reports a human audit: among 100 randomly sampled pairs from each dataset, annotators identified 30 false negatives in WebArbiter data and 12 in SURFPRM-DATA. Thus, the paper does not assume that graph-proposed alternatives are always incorrect; it explicitly measures residual label errors and compares them with the baseline. The remaining 12% is still a limitation.
The paper also contains a No-Graph ablation in In-Depth Analysis and Ablation Studies and the appendix. With similar dataset sizes and GMCP ratios, the graph-based model achieves 80.65% pairwise / 51.29% BoN, compared with 75.50% / 42.01% for No-Graph. This comparison addresses whether local, page-grounded candidate selection alone accounts for the results.
We do not assume that a successful trajectory is unique or that every demonstrated action is locally optimal. The paper uses an action from a successful demonstration as a reference positive from which to construct a candidate comparison. The graph proposes alternatives; it does not prove that they are invalid. This distinction matters for temporal and spatiotemporal proposals such as repeating an interaction or deviating from an observed path: these patterns can suggest a possible error, but another successful path may exist.
Our human-audit criteria explicitly account for this. If both actions can advance the task along comparably effective paths, the proposed negative is counted as a false negative. The same applies if only the proposed negative can advance the task while the reference positive is ineffective. We provide the complete criteria in General Response S1.
We also clarify the 30% versus 12% audit comparison. The paper reports these as human-audited false-negative rates for WebArbiter data and the final SURFPRM-DATA, respectively. It does not report a before-and-after experiment showing that teacher verification alone reduces the rate from 30% to 12%. Teacher verification is a quality-control component of the synthesis pipeline. As already reported in Dataset Curation and Final Statistics and the appendix, it removed 179 of 10,000 sampled candidates, while 9,821 were retained. The lower audited rate is therefore a comparison of the resulting datasets, not a causal estimate of the teacher filter’s contribution.
Regarding downstream evaluation, the paper’s main evaluation is WebPRM-Bench, covering WebArena, AssistantBench, and WorkArena; WebArena-Lite is a complementary online test of reward-guided trajectory search. As stated in Experimental Setup, the online experiment uses two policies and follows the Best-of-N setting used by WebArbiter. We agree that improved overall success does not imply improvement in every domain, and the GitLab regression is an important qualification.
To extend the online evidence, we have additionally evaluated on WorkArena. With the same Qwen3.5-9B backbone, SURFPRM-DATA achieves 31.11% macro-average success, compared with 24.58% for WebArbiter data. Details are provided in our response to reviewer 6tEh Q1 the online-evaluation question. Evaluation with additional policy families and search methods remains valuable future work.