acceptodds
Under review as a conference paper at ICLR 2027

From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs

Abstract

Comparative Process Reward Models (PRMs) provide critical step-level guidance for autonomous web agents by evaluating state-conditioned preferences between candidate actions. However, existing preference training data synthesized via multi-policy sampling suffers from a severe scarcity of Grounded Minimal Contrastive Pairs (GMCPs)—where competing candidates target genuine on-page elements with identical action types. In representative baselines preference data (namely, WebArbiter), GMCPs account for merely 24.19%, biasing PRMs during training to rely on shallow shortcuts (such as element hallucinations and action type mismatches) rather than acquiring genuine contextual decision semantics. To address these challenges, we propose SURFPRM, a graph-guided process preference synthesis framework for comparative Web PRMs. SURFPRM structures web demonstrations into a persistent Interaction Element Graph that acts as an environment-grounded negative action proposal mechanism, systematically synthesizing contrastive negative actions across spatial, temporal, and spatiotemporal confusion axes. This elevates the GMCP proportion from 24.19% to 74.60%, producing the curated SURFPRM-DATA dataset. Across six open-source backbones (3B to 9B parameters), PRMs trained on SURFPRM-DATA outperform baseline-trained models on average on WEBPRMBENCH and rival leading proprietary LLMs. In downstream reward-guided trajectory search on WEBARENA-LITE, SURFPRM provides step-level guidance for both GPT-4o (+14.21%) and GPT-4o-mini (+12.83%) policies, yielding substantial improvements in complex web task success rates. Our code and dataset are anonymously available at https://anonymous.4open.science/r/surfPRM-6188/.

Primary area:
datasets and benchmarks
Keywords:
web agent, Process Reward Model, data synsthesis, graph

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9wLcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents suftPRM, a graph-guided framework for synthesizing process preference data for web PRMs. The authors argues that the scarcity of Grounded Minimal Contrastive Pairs (GMCPs) in existing web PRM dataset is a key deficiency, suggesting that many prior preference pairs contain shaloow cues such as hallucinated elements or mismatched action types. The proposed surfPRM aggregates successful web trajectories into an Interaction Element Graph and use its spatial, temporal, and spatiotemporal structure to construct grounded negative actions. Across several backbone models, PRMs trained on surfPRM-data outperform models trained on the WebArbiter baseline on WebPRMBench. The paper further shows that surfPRM outperforms PRM baselines as a test-time trajectory search verifier on WebArena-Lite.

Critical strengths

None

Critical weaknesses
  1. The paper does not fully establish that GMCP scarcity is the key deficiency of existing Web PRM data.

    • The authors identifies the low proportion of GMCP in training data as a central problem and presents it as a main contribution. However, it seems that GMCP is primarily a structural property of a preference pair rather than a direct measure of its difficulty or informativeness. A pair can satisfy GMCP definition simply by pairing the positive action with another grounded element that supports the same action type, even when the negative is trivially distinguishable from the positive.
    • Most importantly, the paper does not directly verify whether the proposed confusions produce sufficiently challenging nagatives under the given instruction and state. For example, the Structurally Near confusion is based on AXTree proximity, but structural proximity does not imply semantic confusability. The example of choosing "Feb 25" instead of "Jan 28" may be trivial if the instruction specifies Jan 28. Similarly, temporal confusions may generate valid actions, since interacting with the same target multiple times is not inherently incorrect.
  2. The paper treats each action in a successful trajectory as the preferred action against synthesized alternatives, while the downstream validation remains limited.

    • However, a successful trajectory is not unique, and web tasks often admit multiple valid next actions and multiple successful paths. This is particularly important for confusions such as Act Too Early, Away from Goal, and Create a Cycle, whose negative labels depend on the successful trajectory.
    • The teacher verification reduces false-negative rate from 30% to 12%, but 12% remains substantial for preference supervision. Moreover, surfPRM simultaneously changes several properties of the training data, including whether negatives are grounded in the current observation, how negatives are selected, the difficulty of the pairs, and the false-nagative rate. As a result, it is difficult to determine how much of the observed gain is due to the proposed graph-guided contrastive synthesis rather than simply cleaner preference labels.
    • Also, the downstream evaluation is narrow given that the motivation for PRMs is to provide useful signals for policy learning. The authors evaluate only WebArena-Lite and two GPT policies with only Best-of-N. It is therefore unclear whether the gains transfer to other policy families or benchmarks. The regression on GitLab also suggests that the proposed PRM is not consistently beneficial across domains.
Questions for authors
  1. Does task success imply that each demonstrated action is locaaly preferable to other valid actions available at the same state? Since web tasks can have diverse successful paths, it would be helpful to clarify whether the surfRPM-data assumes any form of local optimality or uniqueness of the successful trajectory.

  2. The teacher verification appears unusually permissive. A pair is retained if the teacher agrees with the positive label in any one of up to five attempts, even if the remaining judgements prefer the synthesized negative. It seems there are many alternatives such as majority voting. It would be useful to provide the distribution of teacher votes and, if possible, an ablation over different acceptance thresholds, such as 1/5, 3/5, or 5/5 agreement, together with their effects on the human false-negative rate and downstream performance.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

Thank you for the review.

W1. GMCP, semantic confusability, and false negatives

GMCP is a structural property, not a difficulty score. As defined in the paper’s Data Collection and Dataset Statistics section and appendix, a GMCP requires both targets to be present and uniquely resolvable in the current observation, and both actions to have the same operation type. Our motivation is that such pairs are more closely aligned with the decision setting in which a PRM must distinguish available actions using the task and page context. We agree that these conditions alone do not guarantee that a pair is difficult, informative, or correctly labeled. In particular, structural proximity in the AXTree does not by itself prove semantic confusability.

The task instruction is explicitly included in the state supplied to both the teacher and the PRM, as specified in the Problem Formulation and Comparative PRM Pipeline section and the prompts in the appendix. In the reviewer’s date example, whether selecting “Feb 25” is an appropriate alternative depends on the instruction and observed state. We agree that the comparison may be straightforward when the instruction explicitly specifies “Jan 28”; our claim does not require every proposed pair to be difficult.

The paper also already acknowledges the possibility of valid alternative actions. In Teacher Verification and Gold Supervision Synthesis, we identify an equally valid alternative path as a potential false negative and use teacher judgments to filter candidates. More importantly, the Quality Assurance and Iterative Refinement section reports a human audit: among 100 randomly sampled pairs from each dataset, annotators identified 30 false negatives in WebArbiter data and 12 in SURFPRM-DATA. Thus, the paper does not assume that graph-proposed alternatives are always incorrect; it explicitly measures residual label errors and compares them with the baseline. The remaining 12% is still a limitation.

The paper also contains a No-Graph ablation in In-Depth Analysis and Ablation Studies and the appendix. With similar dataset sizes and GMCP ratios, the graph-based model achieves 80.65% pairwise / 51.29% BoN, compared with 75.50% / 42.01% for No-Graph. This comparison addresses whether local, page-grounded candidate selection alone accounts for the results.

W2 and Q1. Successful trajectories, label quality, and downstream validation

We do not assume that a successful trajectory is unique or that every demonstrated action is locally optimal. The paper uses an action from a successful demonstration as a reference positive from which to construct a candidate comparison. The graph proposes alternatives; it does not prove that they are invalid. This distinction matters for temporal and spatiotemporal proposals such as repeating an interaction or deviating from an observed path: these patterns can suggest a possible error, but another successful path may exist.

Our human-audit criteria explicitly account for this. If both actions can advance the task along comparably effective paths, the proposed negative is counted as a false negative. The same applies if only the proposed negative can advance the task while the reference positive is ineffective. We provide the complete criteria in General Response S1.

We also clarify the 30% versus 12% audit comparison. The paper reports these as human-audited false-negative rates for WebArbiter data and the final SURFPRM-DATA, respectively. It does not report a before-and-after experiment showing that teacher verification alone reduces the rate from 30% to 12%. Teacher verification is a quality-control component of the synthesis pipeline. As already reported in Dataset Curation and Final Statistics and the appendix, it removed 179 of 10,000 sampled candidates, while 9,821 were retained. The lower audited rate is therefore a comparison of the resulting datasets, not a causal estimate of the teacher filter’s contribution.

Regarding downstream evaluation, the paper’s main evaluation is WebPRM-Bench, covering WebArena, AssistantBench, and WorkArena; WebArena-Lite is a complementary online test of reward-guided trajectory search. As stated in Experimental Setup, the online experiment uses two policies and follows the Best-of-NN setting used by WebArbiter. We agree that improved overall success does not imply improvement in every domain, and the GitLab regression is an important qualification.

To extend the online evidence, we have additionally evaluated on WorkArena. With the same Qwen3.5-9B backbone, SURFPRM-DATA achieves 31.11% macro-average success, compared with 24.58% for WebArbiter data. Details are provided in our response to reviewer 6tEh Q1 the online-evaluation question. Evaluation with additional policy families and search methods remains valuable future work.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

Q2. Teacher filtering thresholds

The paper already specifies the filtering rule in Teacher Verification and Gold Supervision Synthesis: a candidate is discarded only if all five teacher judgments disagree with the trajectory label. It also reports first-attempt and retry acceptance counts in the appendix. We agree that this permissive rule warrants a direct comparison with stricter alternatives.

We therefore added the ablation in General Response S2, using five independently generated teacher judgments for each of 1,000 sampled candidates. Single-pass, majority-vote, unanimity, and any-of-five filtering retained 961, 955, 937, and 972 pairs, respectively. Their downstream results were broadly comparable, although performance varied across benchmarks and metrics.

We also audited all 11 retry-accepted pairs; 2 were false negatives. This higher observed rate supports the concern that retries can admit incorrect alternatives, although the small number of pairs makes the estimate uncertain. At the same time, we believe that in practical applications, when cost-sensitive, combined with our data synthesis pipeline, directly using single-pass can also achieve excellent results.

Reviewer_9wLcOpenReview ↗

Official Comment by Reviewer 9wLc

Thank you for the clarification. My main concern is the central role that GMCP plays in the paper's movitation and claimed problem diagnosis. The authors agree that GMCP is a structural property rather than a direct measure of difficulty or informativeness. This makes it less clear why the scarcity of GMCP should be considered a key deficieny of existing web prm data, as emphasized throughout the paper.

The empirical results also seem to suggest that GMCP ratio alone does not provide a clear characterization of preference data quality. Increasing the GMCP ratio does not lead to monotonic improvements across metrics, particularly for BoN accuracy, and the no-graph baseline has a slightly higher GMCP ratio than surfPRM while performing worse. Moreover, the authors intentionally do not aim for 100% GMCP because some non-GMCP pairs are useful or necessary, which further suggests that there is no clear notion of what constitutes an appropriate GMCP ratio.

These results appear to support a dfferent and, in my view, more direct conclusion: surfPRM is effective because its graph-guided synthesis produces better grounded and contextually meaningful negative actions, while GMCP ratio is one descriptive property of the resulting dataset. I therefore remain unconvinced that low GMCP ratio has been established as the key deficiency of prior webprm data. Given how strongly the current motivation and contribution statements are organized around GMCP scarcity, I think the paper would benefit from substantially revising this framing and treating GMCP as one diagnostic statistic rather than the central explanation for the observed improvements.

I appreciate the authors for the detailed responses. However, my main concern remains so I am inclined to maintain my initial score.

Reviewer_6tEhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a data-quality problem in comparative Web PRMs: the scarcity of grounded, same-action-type comparisons in WebArbiter’s training data may allow models to rely on superficial cues. It introduces SURFPRM, which builds interaction element graphs from successful trajectories and uses spatial, temporal, and spatiotemporal rules to propose negative actions targeting elements on the current page. A teacher model screens these pairs and generates judgment rationales for supervised fine-tuning. Experiments across six Qwen backbones report improved macro-average preference accuracy over matched-scale WebArbiter SFT-data baselines, while WebArena-Lite evaluations report higher average task success with GPT-4o and GPT-4o-mini.

Critical strengths
  • The paper addresses a relevant data-quality problem and supports its motivation with a concrete audit of existing preference data.
  • The proposal mechanism reuses structural and temporal information from successful demonstrations without requiring additional environment rollouts for negative generation.
  • The evaluation includes downstream browser-task execution with two policies, extending the evidence beyond offline preference accuracy
  • The paper is generally easy to follow, with figures and explicit construction rules explaining the connection between trajectory data and PRM training.
Critical weaknesses
  • Several downstream baseline scores are imported from prior work, but identical task sets and execution conditions are not sufficiently documented (Table 3).
  • The construction and matching of the intermediate GMCP-ratio datasets need clarification.
  • Teacher-side production costs are reported, but online latency and total execution costs are insufficiently characterized.
Questions for authors
  • What motivates limiting online evaluation to WebArena-Lite, while static evaluation also includes WorkArena and AssistantBench?
  • How was the human false-negative audit conducted, including the annotation criteria and handling of multiple valid action paths?
1 comment

W1: details on webarena exp

We thank the reviewer for this constructive suggestion. We self-hosted the WebArena Docker images and followed the WebArbiter implementation for reward-guided trajectory search. Our evaluation used the following protocol:

  • Tasks and environment. We evaluated on the 165 WebArena-Lite tasks across Shopping, Shopping Admin (CMS), Reddit, GitLab, and Map. Performance is measured by task success rate. Observations are text-only, using a BrowserGym-style indented AXTree; the policy and PRM receive the same AXTree. We do not use set-of-marks visual input.
  • Policies and candidate selection. We evaluated GPT-4o-mini and GPT-4o as policies. At each step, the policy generates five candidate blocks, each containing a thought and an action. A WebArbiter-style knockout tournament selects one candidate through four pairwise PRM comparisons. Only the winner’s action is executed.
  • PRM inputs and outputs. Each comparison includes the task intent, current AXTree, trajectory, start and current URLs, state difference, and the two candidate thought/action blocks. The PRM produces <State>, <Criteria>, <Analysis>, and an <Answer> identifying Response 1 or 2 as the winner.
  • State tracking and termination. After each action, the PRM generates a difference between the previous and current AXTrees for use at the next step. At the beginning of each step, the PRM also performs terminal detection to assess whether the task is complete.

W2: construction and matching of the intermediate GMCP-ratio datasets

Thank you for raising this point. We describe the construction and matching of GMCP40 and GMCP60, including the controlled factors and remaining limitations, in S3 of the General Response.

W3: online latency and execution cost

Thank you for highlighting the need to characterize inference-time overhead. The following table reports the observed latency and execution costs in our WebArena experiments.

Policy + PRMMean latency per stepPRM tournament time per stepMean time per taskMean API cost per taskMean tournament tokens per task
GPT-4o-mini + SURFPRM62.6 s31.6 s (50.5%)16.26 min$0.0402519.8K
GPT-4o-mini + WebArbiter74.0 s36.9 s (49.9%)16.47 min$0.0322393.6K
GPT-4o + SURFPRM82.7 s40.9 s (49.5%)15.23 min$0.4976358.1K
GPT-4o + WebArbiter76.3 s37.5 s (49.2%)16.69 min$0.5268372.9K

The pairwise tournament accounts for approximately half of the observed per-step latency in all four settings. Per-task time and token usage also depend on the number of steps taken, so they need not follow the same ordering as per-step latency.

Q1: clarification on online evaluation

Thank you for raising this question. Online evaluation requires substantially more environment and execution infrastructure than static evaluation. We initially reported online results only on WebArena-Lite, following the online evaluation in WebArbiter.

We have since completed an additional WorkArena online evaluation, using the same reward-guided trajectory search procedure described in our response to W1. Results are success rates (%); Δ\Delta is relative to the setting without trajectory search.

PRMDashboardFormKnowledgeList-filterList-sortMenuService CatalogMacro-AvgΔ\Delta
w/o Trajectory Search50.000.0010.000.005.0025.002.2213.17–
GPT-4o-mini55.0010.0010.000.006.6720.005.5615.32+2.14
Web-Shepherd (8B)65.0014.0020.000.0010.0020.007.7819.54+6.37
Qwen3.5-9B + WebArbiter 10k62.504.0030.000.0010.0030.0035.5624.58+11.40
Qwen3.5-9B + SURFPRM-DATA 10k60.0010.0060.000.0011.6735.0041.1131.11+17.94

SURFPRM-DATA improves macro-average success by 6.53 percentage points over the same-backbone WebArbiter data baseline. The gain is particularly visible on Knowledge and Service Catalog. One possible explanation is that Service Catalog benefits from discriminating among same-page targets and comparing local action sequences, capabilities emphasized by our synthesized data. The smaller gains on Dashboard and Form are consistent with limitations in value-level reading and aggregation, global field checking, and post-submission state assessment.

We did not run an online AssistantBench evaluation because its web servers are neither self-hosted nor hosted by a single organization in the manner of our WebArena and WorkArena environments. Changes to those external sites can introduce environment drift and make results harder to compare.

Q2: human audit criteria

Thank you for asking for clarification. We describe the annotation criteria, including how we handle multiple valid action paths, in S1 of the General Response.

Reviewer_FTU8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SURFPRM, a framework for constructing process preference data for comparative web process reward models. The authors identify a low proportion of Grounded Minimal Contrastive Pairs (GMCPs), defined as comparisons between actions with the same operation type and targets grounded in the current observation, in existing WebArbiter training data. SURFPRM aggregates successful demonstrations into persistent Interaction Element Graphs and proposes alternative actions using spatial, temporal, and spatiotemporal rules. A teacher model filters candidate pairs and generates judgment rationales for supervised fine-tuning. The resulting dataset contains 9,821 pairs and increases the GMCP proportion from 24.19% to 74.60%. Models trained on this dataset improve macro-average preference discrimination across six Qwen backbones and improve average downstream task success with two policy models.

Critical strengths
  • The paper addresses a concrete and relevant data-quality problem. Distinguishing between executable, grounded actions is an important capability for web PRMs. The analysis of hallucinated targets and action-type mismatches provides a clear motivation for improving negative-action construction.

  • The empirical improvements are reasonably consistent across model scales. SURFPRM-trained models outperform their corresponding WebArbiter-trained counterparts on macro-average Pairwise and BoN accuracy across all six evaluated backbones. The downstream experiments also provide evidence of practical utility beyond offline preference classification.

  • The evaluation includes useful controls and quality checks. In particular, the No-Graph comparison approximately matches dataset size, GMCP ratio, demonstration states, and teacher verification settings. The exclusion of the overlapping Mind2Web evaluation subset, manual false-negative audit, and repeated evaluation results are also positive aspects.

Critical weaknesses
  • Limited qualitative analysis. The paper lacks case studies showing how the six proposal rules improve PRM decisions or lead to failures. Such analysis would help explain the performance drops in Table 3, including CMS with GPT-4o-mini (38.89% → 30.56%) and MAP with GPT-4o (35.48% → 29.03%).

  • Potential action-type shortcut. Non-DOM actions appear only as positives, allowing models to infer preference labels from action types rather than task context. The paper does not assess this bias or test cases where non-DOM actions are incorrect.

Questions for authors
  1. How do label quality and downstream performance change under single-pass, majority-vote, and unanimity teacher filtering? What are the human-audited error rates for first-attempt passes and retry-accepted pairs?
  2. What is the distribution of operation types across positive and negative actions? How well can an operation-type-only classifier predict the labels, particularly for pairs containing virtual positive actions?
  3. How exactly are the GMCP40, GMCP60, and No-Graph datasets constructed? Are action distributions, demonstration-state coverage, teacher supervision, and label-noise levels controlled?
  4. Could the authors enlarge Figures 1, 3, and 5 and increase their font sizes? The labels and annotations are difficult to read at the current scale, making it hard to follow the framework in Figure 1 and inspect the results in Figures 3 and 5.
9 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/2)

W1: Case study and analysis on webarena performance

We sincerely thank the reviewer for this insightful comment. Our synthesis rules do not provide uniform supervision across web tasks. We primarily train the PRM to compare same-type actions grounded in the current page and to distinguish local process order. The rules do not systematically target numerical aggregation or geographic reasoning.

On WebPRMBench, we observe several scenarios:

  • Avoiding a repeated entry point (WorkArena, state 98, pair 1). The Problem menu was already expanded and Create New was visible in the AXTree. SURFPRM selected Create New, whereas the baseline selected the already-expanded Problem menu again. This illustrates a distinction between advancing from the current state and repeating an earlier interaction.
  • Correcting an erroneous filter (WorkArena, state 82, pair 0). The table was empty because an incorrect Disposal reason = Catherine-Greg Valdez-Swanson filter had been applied. SURFPRM selected Remove next condition to remove the filter; the baseline selected Personalize List to adjust the columns. SURFPRM thus prioritized restoring the filter state.
  • Misreading a field value (WorkArena, state 120, pair 1). The AXTree showed that Calendar integration was already set to Outlook, but candidate 2 claimed that it was still -- None -- in the THOUGHT. SURFPRM repeated this claim in its own state assessment and selected the dropdown instead of Submit; the baseline selected Submit. The judgment favored completing the configuration without checking the actual field value.
  • Misreading an action’s effect (WebArena, state 104, pair 1). Not all four Customer Groups options had been selected. SURFPRM treated a Shift-based range selection as though it had already occurred and selected the last option, whereas the baseline selected the first option to begin selecting them individually. This illustrates how a candidate’s rationale about an efficient action can be mistaken for an observed action effect.

The WebArena-Lite trajectories show related strengths, but also expose limits that can contribute to the per-domain differences in Table 3:

  • Shopping category navigation (GPT-4o, task 82). The task required navigating from Home & Kitchen to living room furniture and listing products in descending price order. The baseline PRM chose Advanced Search on the homepage; SURFPRM chose Home & Kitchen and followed the category path to the requested page. This is consistent with improved discrimination among page-level entry points.
  • CMS row-level targeting (GPT-4o, task 106). To unpublish “Ryker Tee Crew Neck” (product 478), SURFPRM identified the Edit button in the product’s row, entered its edit page, and completed the task. The baseline remained on the list page searching and filtering. Thus, some page-grounding skills also transfer to CMS.
  • CMS business-entry selection (GPT-4o-mini, task 70). To identify the customer with the most historical canceled orders, the baseline chose Sales → Orders and completed the task. Although that entry was also in SURFPRM’s candidate pool, SURFPRM judged the Customers page to be more direct and subsequently failed. This example illustrates a gap in understanding where the relevant business records reside, which may contribute to the CMS drop from 38.89% to 30.56% under GPT-4o-mini. CMS performance instead increases under GPT-4o, indicating that the policy and its candidate pool also matter.
  • MAP geographic search (GPT-4o, task 90). For hotels near Carnegie Music Hall, the baseline selected hotels near Carnegie Music Hall, while SURFPRM selected Carnegie Music Hall hotels; the latter trajectory failed. This example illustrates a weakness in judging whether a search query expresses the requested spatial relation.

More broadly, while the rules support page-grounded discrimination and local ordering, aggregate business reasoning, geographic relations, and terminal computation may require more targeted supervision. Despite these limits and the domain-specific drops, SURFPRM achieves a higher overall success rate than the same-backbone WebArbiter data baseline under both policies.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/2)

W2 & Q2: Distribution statistics of operation types and the performance of the operation-type-only classifier

Thank you for raising this concern. We agree that the absence of negative virtual actions in SURFPRM training data can create an action-type shortcut. We report the distribution of virtual actions and evaluate how much preference information can be recovered from action types alone.

The table below counts positive and negative actions separately, with each row measured over the stated number of pairs.

DatasetPairsPositive virtual actionsNegative virtual actions
SURFPRM 10k training data9,8211,601 (16.30%)0 (0.00%)
WebArbiter 10k training data9,6423,941 (40.87%)1,439 (14.92%)
WebPRM-Bench test data1,772208 (11.74%)96 (5.42%)

We trained a standard linear SVM using only the difference between the one-hot action-type vectors of the two candidates. The classifier receives no task instruction, page observation, or action parameters. This representation also avoids introducing a fixed candidate-position bias.

WebPRM-Bench subsetPairsSVM trained on SURFPRM 10k dataSVM trained on WebArbiter 10k data
All pairs1,77259.82%59.90%
Positive action is virtual20897.60%90.87%
Negative action is virtual9623.96%62.50%

The SURFPRM-trained type-only classifier performs well when the virtual action is positive, but its accuracy falls to 23.96% when the virtual action is negative. This asymmetry shows that the training-set correlation can lead to an incorrect preference for virtual actions on test pairs where they are the negative choice.

From the test set results, the absence of non-dom actions in the negative samples did not severely affect the results. We will also continue to pay attention to such issues in subsequent improvements.

Q1: Teacher filtering method ablation

Thank you for this question. We address the comparison of filtering methods and the human-audited error rates in the General Response S2.

Q3: construction and matching of the intermediate GMCP-ratio datasets and no-graph dataset

Thank you for asking us to clarify the dataset construction and controls. The construction and matching of GMCP40 and GMCP60 are described in the General Response S3. We detail No-Graph below.

  1. Shared inputs. No-Graph uses the same WPRM source data as Graph: 854 successful trajectories across 50 websites. It reuses the same targeted_event enumeration and state_id/candidate_id formulas.

  2. Local candidate proposals. Negative candidates are drawn only from the current AXTree, without loading element nodes, interaction edges, or containment edges for candidate selection. Three local rules propose candidates: local_type_near selects the same action type as the positive action; local_text_near requires text similarity of at least 0.75, without a graph-distance exclusion; and local_axtree_near selects elements within five containment hops in the current AXTree.

  3. Grounding and completion. We use the same formatter, BID remapping, and fill-value completion as in the Graph pipeline, producing 24,121 candidate pairs.

  4. GMCP stratification. The candidate pool contains 23,159 natural GMCP pairs and 962 natural non-GMCP pairs. To obtain a ratio close to the Graph dataset’s 74.60%, we sampled 10,002 pairs and changed the negative action type from click or fill to hover for 1,579 originally GMCP pairs, making them non-GMCP.

  5. Teacher supervision. We used GPT-5.6-Sol with the same prompt and up to five attempts, retaining a pair when a verdict agreed with the positive-action label. The final No-Graph dataset contains 9,643 pairs, covering 5,266 states, with a 75.56% GMCP ratio.

The construction aligns the state and positive-action source pools and the teacher-supervision procedure, and makes the dataset size and GMCP ratio close to those of Graph. It does not guarantee identical final state coverage, action distributions, or label-noise levels. However, because the data synthesis methods differ, it is impossible to keep the final state coverage, action distribution, and label noise level consistent. We could only choose several key factors for control.

Q4: enlarge figure

Thank you for carefully reviewing the figures and pointing out the readability issue. We will enlarge Figures 1, 3, and 5 and increase their label and annotation font sizes in the final version if the paper is accepted.

Thank you for the detailed response; could you report the full PRM’s accuracy separately for virtual-positive and virtual-negative pairs, clarify whether changing 1,579 No-Graph negatives to hover confounds the ablation, and confirm whether task- or instance-level overlap was checked between the 854 source trajectories or their derived training pairs and the evaluated test sets, beyond excluding the Mind2Web subset?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

S1: Human audit criteria

Annotators assessed the original negative action against the positive action in the given task state:

  • False negative: Both actions could advance the task along comparably effective paths, or only the negative action could advance it.
  • True negative: The negative action was invalid, off-task, or erroneous; was a possible but strictly worse path; or was hallucinated while the positive action was correct.

S2: Teacher filtering method ablation

To compare filtering strategies while limiting annotation cost, we randomly sampled 1,000 pairs from the original 10k candidate pool and independently generated five teacher judgments per pair. Let mm denote the number of verdicts agreeing with the original positive-action label. We applied each filtering rule to these judgments, trained a Qwen2.5-3B model on the retained pairs, and audited the retained data using the criteria described in S1.

MethodCriterionRetainedTeacher calls*Calls/pair*
Single-passFirst verdict agrees9611,0001.00
Majority votem≥3m \geq 39553,0283.03
Unanimitym=5m=59374,8044.80
Any-of-5m≥1m \geq 19721,1371.14

*Calls needed to determine the outcome with early stopping, calculated from the pre-generated judgments.

Downstream results are shown below; each cell reports pairwise accuracy / BoN.

MethodWebArenaAssistantBenchWorkArenaAverage
Single-pass79.73 / 47.2675.00 / 56.6770.28 / 33.4975.00 / 45.81
Majority vote81.47 / 54.2377.50 / 43.3371.11 / 31.6076.69 / 43.05
Unanimity79.23 / 49.2577.50 / 46.6772.17 / 41.0476.30 / 45.65
Any-of-579.85 / 55.2279.17 / 50.0071.93 / 33.9676.98 / 46.39

Considering the combined effects of average pairwise acc and average BoN, the differences among the several methods are not significant.

We also randomly audited 30 retained pairs for each method. In addition, we audited all 11 retry-accepted pairs: those rejected by the first judgment but accepted by a later judgment under any-of-5. Note that the total number of samples of retry-accepted type is only 11, so no sampling inspection was conducted.

Audited groupFalse negatives / auditedRate
Single-pass2 / 306.67%
Majority vote1 / 303.33%
Unanimity1 / 303.33%
Any-of-51 / 303.33%
Retry-accepted2 / 1118.18%

The small audits do not support a strong conclusion about differences in label quality across the four retained sets. The higher observed false-negative rate among retry-accepted pairs nevertheless suggests that retries can admit ambiguous or incorrectly labeled examples.

Overall, any-of-5 retained the most data at a comparatively low request cost and achieved the highest averages on both metrics in this experiment, supporting its use in our pipeline. At the same time, we believe that in practical applications, when cost-sensitive, combined with our data synthesis pipeline, directly using single-pass can also achieve excellent results.

S3: Construction and matching of GMCP-ratio datasets

We constructed GMCP40 and GMCP60 by mixing GMCP examples from SURFPRM-DATA with non-GMCP examples from WebArbiter. Each arm contained 9,180 pairs:

  • GMCP40: 3,672 SURFPRM-DATA pairs and 5,508 WebArbiter pairs.
  • GMCP60: 5,508 SURFPRM-DATA pairs and 3,672 WebArbiter pairs.

We used an (intent, current_url, positive action) fingerprint to ensure that, within each arm, the same positive action was drawn from only one source. The original <Answer> labels were preserved.

The two arms were constructed from the same state and positive-action source pools, with controlled dataset size and GMCP proportion. However, because the underlying synthesis methods differ, this construction does not guarantee identical final state coverage, action distributions, or label-noise levels.

We also recognized that the original ablation did not control for differences in teacher supervision, thanks to the reviewer's reminder. We therefore regenerated teacher supervision for both arms using the same GPT-5.6-Sol teacher and any-of-KK filtering as the main experiment:

Training dataGMCP ratioWebArenaAssistantBenchWorkArenaAverage
GMCP40, re-supervised40.00%83.71 / 55.7272.50 / 46.6772.52 / 38.2176.24 / 46.87
GMCP60, re-supervised60.00%83.08 / 58.7179.17 / 40.0073.82 / 36.3278.69 / 45.01
SURFPRM 10k SFT data74.60%81.59 / 55.2285.00 / 56.6775.35 / 41.9880.65 / 51.29

Cells report pairwise accuracy / BoN (%). We found that after controlling for teacher supervision consistency, the conclusions in the previous paper still hold, and it can be seen that the overall performance of the model shows an upward trend as the gmcp ratio increases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.