acceptodds
Under review as a conference paper at ICLR 2027

Same Explanations, Opposite Effects: How Data Construction Shapes WebPRM Judgment

Abstract

Web process reward models (WebPRMs) use explanations to help agents choose actions. When training labels influence how explanations are generated, models may learn to recognize labels rather than judge actions. We introduce PreAct, which reconstructs both candidates' explanations in WebPRM Collection using one generator and only pre-execution information, without access to preference labels. To evaluate how models use explanations, we introduce WebContrast, a benchmark of 300 pages across 119 websites that tests correct responses to goal changes and stability across explanation versions. On WebContrast, the same test explanations hurt action judgment on average after original-data training but help after PreAct training. Relative to each model's accuracy without explanations, their effect shifts from -3.58 to +5.58 percentage points for Qwen3-8B and from -24.25 to +5.33 for Llama-3.1-8B-Instruct. PreAct also improves mean action-selection accuracy on WebLINX and WebShop, and on Collection with pre-execution or no explanations. Yet evaluation with Collection's original explanations favors original-data training. Our findings show that data construction can change whether explanations help or hinder action judgment, while evaluation using the original explanations can obscure this difference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
web agents, process reward models, rationale construction, shortcut learning, diagnostic benchmarks

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mUjjOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits label cues in WebPRM explanations and proposes PreAct, a label-blind reconstruction procedure. It introduces WebContrast to test goal sensitivity and explanation effects. Controlled source assignments change learned preferences. PreAct makes identical test explanations more useful, while evaluation with Native explanations favors the original training recipe.

Critical strengths
  • Controlled source–label assignments provide evidence beyond input perturbations.
  • The evaluation separates goal accuracy, explanation benefit, and stability.
  • Length, paraphrase, generator, neutral-text, and readout controls strengthen the analysis.
Critical weaknesses
  • PreAct changes information access, generator choice, and factual review together. The final comparison cannot isolate which component produces the improvement.
  • The final Native/PreAct comparison uses 1,024 states and verdict-only LoRA SFT. PreAct’s training benefits have not been demonstrated at larger scale or within WebArbiter’s complete reasoning-distillation and RL pipeline.
  • WebContrast uses goal-blind explanations to keep text fixed across opposing goals. This enables controlled testing, but differs from typical goal-conditioned actor explanations. Generalization to natural explanation use therefore remains uncertain.
Questions for authors
  • Can you provide matched training ablations that separate the effects of pre-execution information access, generator choice, and factual review?
  • How sensitive are PreAct’s gains to training-set size and random seed? What evidence supports extending the findings from verdict-only LoRA SFT to WebArbiter’s complete distillation and RL pipeline?
  • Can you test whether PreAct improves the benefit of goal-conditioned actor explanations on an independent evaluation set?
4 comments
AuthorsOpenReview ↗

Response (1/4): Revised WebContrast and external PRMs

We sincerely thank you for your recommendation to accept our paper, and for recognizing our controlled source–label experiments and the separate evaluation of goal accuracy, explanation benefit and stability. Your constructive questions helped us strengthen the evidence.

We respond to your three questions as follows:

  • Q1: matched ablations of information access, teacher policy and factual review;
  • Q2: robustness across five seeds and a larger 2,048-state configuration, plus results from continued training with GRPO;
  • Q3: a closed-loop WebShop evaluation with goal-conditioned actor rationales, reporting overall task outcomes and rationale benefit separately.

Revised WebContrast. After submission, we expanded and refined the benchmark. We will publicly release the full WebContrast benchmark and the code for the paper’s experiments within one week of acceptance. We report two cohorts throughout:

  • All500: 500 pages, 1,000 goals and 130 websites, obtained by adding 200 harder pages.
  • Core300: the 300-page core of All500. It differs from the submitted benchmark by two page replacements and small revisions to some goals, actions and rationale texts.

Each page keeps shared candidate actions, two opposing goals and fixed goal-blind test rationales. A page counts as correct only if both goals receive the correct action. The rationale effect is this two-goal accuracy with the test rationales minus accuracy without them, in percentage points (pp).

External reward models. The panel below asks whether reward-model training preserves the benefit that base models obtain from the same test rationales. Entries are rationale effects (pp), not accuracies, under each model's native interface. Arrows give base → released for six web PRMs, with the protocol fixed within each pair. AgentRM-8B is reported only as released: it scores through a trained scalar value head its base model lacks, so no comparable base entry exists.

Effect = two-goal accuracy with − without rationales (pp); positive values indicate a benefit. Arrows show base → released.

ModelCore300 rationale effect (pp)All500 rationale effect (pp)
WebArbiter-3B+18.83 → −17.42+11.30 → −13.95
WebArbiter-4B+8.08 → −6.33+5.60 → −7.45
WebArbiter-7B+8.92 → −6.75+6.15 → −4.35
WebArbiter-8B+3.58 → −1.33+4.30 → −0.05
Web-Shepherd-3B+9.67 → −0.33+4.60 → 0.00
Web-Shepherd-8B+2.17 → −0.33+0.10 → −1.90
AgentRM-8B (generalist; released only)+5.83+3.20

On both cohorts, all four WebArbiter base models benefit from the same test rationales, whereas their released reward models have negative rationale effects. Reward-model training therefore need not preserve this benefit. Because WebContrast fixes pages, actions and rationale text, it exposes such regressions directly.

AgentRM, a generalist agent PRM trained with a different recipe, shows positive mean effects, so PRM training does not inevitably make rationales harmful. The matched comparisons in the following parts test how the construction of training rationales affects this benefit.

AuthorsOpenReview ↗

Response (2/4): Q1 — Matched component ablations

Q1. Matched ablations of information access, generator choice and factual review

"Can you provide matched training ablations that separate the effects of pre-execution information access, generator choice, and factual review?"

You are right that the Native→PreAct comparison changes three things at once. To see how each one matters, we trained matched arms. Within each backbone and seed, every arm uses the same 1,024 Collection states, labels, actions and 192 updates, and all arms are tested on identical WebContrast inputs; only the training rationales differ:

  • A: a Qwen teacher explains each candidate from pre-execution information (goal, page, history and that action) without seeing the label.
  • B: keeps A's teacher and reuses A's rejected rationales verbatim, but the preferred candidate's generator also sees the real post-execution observation, restoring Collection's asymmetric access.
  • C: keeps A's access but uses a GPT teacher policy (gpt-6.1-sol), which changes both the model and its prompting.
  • Raw and Full: the submitted PreAct rationales before and after factual review.

Every symmetric, label-blind arm we tested keeps a positive rationale effect, while restoring asymmetric access gives the largest accuracy drop. The first table gives accuracies and effects for each arm; the second reports the matched accuracy contrasts with intervals.

All500 two-goal accuracy (%), two-seed means. Qwen/Llama denote Qwen3-8B/Llama-3.1-8B-Instruct. “With” includes test rationales; effect = with − no rationale (pp). C uses gpt-6.1-sol.

Training armQwen no rationaleQwen withQwen effectLlama no rationaleLlama withLlama effect
Native (Collection rationales)56.6054.75−1.8538.8024.65−14.15
A: symmetric pre-execution, Qwen teacher56.3061.10+4.8052.2054.85+2.65
B: preferred side also sees post-execution50.2051.80+1.6016.4015.85−0.55
C: symmetric pre-execution, GPT teacher55.9059.55+3.6547.7049.75+2.05
Raw: PreAct before factual review55.7058.80+3.1049.4052.30+2.90
Full: submitted PreAct56.4060.15+3.7546.4051.00+4.60

Accuracy differences with rationales (pp); brackets give 95% intervals. All500 B−A and C−A use simultaneous intervals over the eight pre-specified comparisons; Full−Raw intervals are pointwise; Core300 intervals are descriptive. All are conditional on the two training seeds.

BackboneContrastCore300All500
Qwen3-8BB − A (information access)−12.42 [−15.17, −9.75]−9.30 [−12.08, −6.52]
Qwen3-8BC − A (generator / teacher policy)−1.58 [−3.08, −0.17]−1.55 [−3.18, +0.08]
Qwen3-8BFull − Raw (factual review)+2.00 [+0.83, +3.25]+1.35 [+0.35, +2.35]
Llama-3.1-8BB − A (information access)−60.33 [−64.25, −56.33]−39.00 [−43.94, −34.06]
Llama-3.1-8BC − A (generator / teacher policy)−7.08 [−9.83, −4.33]−5.10 [−7.75, −2.45]
Llama-3.1-8BFull − Raw (factual review)−2.92 [−5.00, −0.92]−1.30 [−2.75, +0.10]

Generator choice and factual review. After Native training, the same WebContrast rationales lower accuracy on both backbones; after training on A, C, Raw or Full, they raise it on both, including with a different teacher (C) and before factual review (Raw). These choices still shift absolute accuracy: C is below A, and review helps Qwen but not Llama.

Information access. B keeps A's teacher but gives only the preferred candidate's generator the post-execution observation, so the contrast includes both the extra facts and the accompanying label-correlated wording. On All500, accuracy with rationales falls by 9.30 pp on Qwen and 39.00 pp on Llama, and accuracy without rationales falls as well. Llama trained on B also shows a strong preference for one output position.

These results are consistent with the paper's principle that labels should supervise judgment rather than shape how candidate rationales are generated. We will add both tables to the paper.

AuthorsOpenReview ↗

Response (3/4): Q2 — Training seeds, data size, and GRPO

Q2. Sensitivity to training-set size and seed; extension to the full distillation and RL pipeline

"How sensitive are PreAct's gains to training-set size and random seed? What evidence supports extending the findings from verdict-only LoRA SFT to WebArbiter's complete distillation and RL pipeline?"

We address the two parts in turn: robustness of the results across seeds and a larger training configuration, and extending training beyond SFT with GRPO.

The experiments below use verdict-only LoRA SFT, as in the submission, and additionally evaluate continued training with GRPO. Candidate rationales remain part of the model’s input; SFT supervises only the preferred-action verdict, while GRPO rewards verdict correctness.

Seeds and training-set size. The submitted comparison used two seeds at 1,024 states. We extended it to five seeds at 1,024 states (192 updates) and also trained three seeds at 2,048 states (384 updates; the same three epochs, with the original 1,024 states kept as a prefix). Two results hold in every seed at both sizes: PreAct's own rationale effect is positive, and PreAct is more accurate than Native with rationales. All500, mean over seeds:

All500, seed means in pp. Rationale effect = accuracy with − without rationales; change = PreAct effect − Native effect. SD is the sample standard deviation of this paired change across seeds; fractions count positive results / total seeds.

BackboneSettingRationale effect: Native → PreActPreAct effect > 0PreAct − Native accuracy with rationalesChange in rationale effect (SD)
Qwen3-8B1,024 states, 5 seeds−1.20 → +3.365/5+4.40 (5/5 > 0)+4.56 (2.02; 5/5 > 0)
Llama-3.1-8B1,024 states, 5 seeds−9.80 → +4.645/5+24.56 (5/5 > 0)+14.44 (11.72; 4/5 > 0)
Qwen3-8B2,048 states, 3 seeds−0.63 → +3.003/3+3.23 (3/3 > 0)+3.63 (0.29; 3/3 > 0)
Llama-3.1-8B2,048 states, 3 seeds−14.83 → +3.873/3+25.63 (3/3 > 0)+18.70 (1.87; 3/3 > 0)

These repetitions test whether the findings depend on a favorable seed or on the original 1,024-state setting. Across all runs, the same test rationales help every PreAct-trained model, with effects in a narrow band (+2.5 to +6.3 pp on All500), and PreAct is more accurate than Native with rationales in every matched seed. After Native training, the same rationales lower accuracy on average.

Because this pattern recurs over five seeds at 1,024 states and three seeds in the larger 2,048-state configuration, and on Core300 as well as All500, PreAct's positive rationale effects and higher accuracy with rationales are reproducible across the tested seeds and training configurations. This strengthens the paper's central claim that how training rationales are constructed changes whether the same test rationales help action judgment.

Extension to GRPO. The original training targets include reasoning about the original candidate rationales. Our checks confirmed that this reasoning contains statements inconsistent with the reconstructed PreAct rationales. We therefore supervise only the unchanged preference verdict.

Building on this setup, we continued the Qwen3-8B PreAct checkpoint with GRPO. The starting checkpoint was trained on 1,024 states for 192 SFT updates. GRPO uses an augmented set of 516 distinct states with replay and both candidate orders, a binary verdict-correctness reward, seven samples per prompt, and a learning rate of 1.75 × 10⁻⁶. We ran 84 optimizer updates and evaluated three checkpoints.

WebContrast two-goal accuracy (%). Test rationales and the order-symmetrized verdict-margin readout are fixed across checkpoints. Δ denotes the change from SFT in percentage points. Bold marks the highest value in each column.

Training stageCore300Δ vs. SFT (pp)All500Δ vs. SFT (pp)
PreAct SFT94.33—60.60—
SFT + GRPO, 28 updates94.83+0.5061.50+0.90
SFT + GRPO, 56 updates95.00+0.6762.00+1.40
SFT + GRPO, 84 updates (final)95.17+0.8361.80+1.20

PreAct remains above its SFT accuracy at all three evaluated GRPO checkpoints on both cohorts. At the fixed final checkpoint, All500 accuracy reaches 61.80%, up from 60.60% after SFT, alongside a 0.83 pp increase on Core300. This extends our evaluation to subsequent RL optimization of a PreAct-trained model.

The final-minus-SFT 95% paired-page bootstrap intervals are [−0.17, +2.00] pp on Core300 and [−0.20, +2.60] pp on All500.

AuthorsOpenReview ↗

Response (4/4): Q3 — Independent online evaluation

Q3. Goal-conditioned actor explanations on an independent evaluation set

"Can you test whether PreAct improves the benefit of goal-conditioned actor explanations on an independent evaluation set?"

Thank you for this question. In addition to the submitted offline evidence, we added a closed-loop WebShop evaluation. With the actor's goal-conditioned rationales, PreAct achieves higher success rate and cumulative reward than Native on both backbones, and we report separately how much reading those rationales adds.

Submitted Table 6 already evaluates offline WebShop preference comparisons whose candidates carry the actor's original goal-conditioned rationales. With these rationales, PreAct's preference accuracy is 64.32% vs. 63.25% for Native on Qwen, and 64.37% vs. 46.33% on Llama.

To extend this from offline action judgments to online task performance, we ran a closed-loop evaluation on 200 WebShop tasks, separate from WebContrast's pages and rationales. At each step, GPT-4o (gpt-4o-2024-08-06) proposes candidate actions together with its own original goal-conditioned rationales. The Native- or PreAct-trained PRM reads the goal, page, candidates and rationales and selects one action. The environment executes it, and the loop repeats until the episode ends, when the environment returns terminal reward and success. We evaluate each PRM with and without the current candidate rationales:

Cumulative reward sums episode returns over 200 tasks; success rate measures terminal task success. “None” removes the current candidate rationales.

BackboneCurrent rationalesCumulative reward: NativeCumulative reward: PreActSuccess rate %: NativeSuccess rate %: PreAct
Qwen3-8BNatural actor rationales121.77125.8336.541.0
Qwen3-8BNone125.03125.5839.040.5
Llama-3.1-8BNatural actor rationales124.03124.5337.539.5
Llama-3.1-8BNone121.51125.0334.039.0

Task outcomes. With the actor's rationales, success rate rises from 36.5% to 41.0% for Qwen and from 37.5% to 39.5% for Llama. PreAct leads on both metrics in all four settings, so its advantage carries over to natural rationales and executed tasks.

Rationale benefit. The with/without comparison shows what reading the current rationales adds. For PreAct, they raise success rate by 0.5 pp on both backbones, with cumulative reward changes of +0.25 and −0.50. Relative to Native, this rationale effect on success rate is 3.0 pp larger for Qwen and 3.0 pp smaller for Llama, so the change in rationale benefit relative to Native differs by backbone. We will add this evaluation to the paper.

We sincerely thank you again for your positive recommendation and recognition of our work, and for the constructive questions that guided these additional analyses.

Reviewer_VER7OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In this work, the authors first show that WebRPM benchmarks contain label cues as preferred/chosen actions are paired with explanations based on post-execution information, while rejected actions are paired with explanations consisting only pre-action information. WebPRMs learn to favor whichever explanation type is associated with preferred actions during training, and reversing this association flips their preferences. To address this, the authors introduce PreAct, which regenerates explanations using only pre-action information. They also introduce WebContrast, a benchmark testing adaptation to changing user goals and consistency across explanation variants. Through extensive experiments and analyses, the authors isolate the effects of explanation type and training data construction. They show that PreAct improves action-selection accuracy while reducing cases where explanations overturn correct judgments across WebContrast and other benchmarks.

Critical strengths
  • This work addresses an important and real problem: the way training explanations are constructed for web process reward models can make preference labels recognizable from writing style alone. The authors show that models can learn to rely on these cues instead of judging the actions, causing explanations to mislead rather than help.
  • The authors show not only that this leak exists, but that models trained on the data learn to exploit it.
  • WebContrast is a well-designed benchmark. Pairing two opposite goals on the same page prevents a model from scoring by always favoring one action, goal-blind explanations cannot leak the answer, and two explanation versions test stability.
  • PreAct is a simple and practical fix: it generates all candidates' explanations the same way, from pre-action information only and without access to labels, so explanation style no longer signals the correct action.
  • The experiments are extremely thorough. The authors show that explanations alone predict labels almost perfectly and that released models rely on this cue. Controlled training shows that pairing explanation style with labels determines whether explanations help or hurt. With PreAct training, the same explanations improve the model's decisions, and these gains carry over to other web datasets.
  • The authors also show that standard benchmarks can fail to recognize the better model: because their explanations carry the same label cues as the training data, they reward models that exploit those cues over models that genuinely judge actions.
Critical weaknesses

I do not have major concerns with this work. However, I do believe in some cases, having goal specific explanations instead of neutral explanations could help the agent select actions better suited to its goal. A natural next step would be evaluating with goal-specific explanations. These may help the reward model select better actions in some cases, while in others they may argue confidently for the wrong one, so testing them would show how far the benefits of PreAct’s goal neutral explanation training would extend to goal specific explanations.

Questions for authors

Please see weaknesses above and clarify any misunderstanding I may have had.

2 comments
AuthorsOpenReview ↗

Response (1/2): Goal-conditioned rationales and online performance

We sincerely thank you for your recommendation to accept our paper, and for recognizing the label-cue evidence, the design of WebContrast and the practical value of PreAct.

We address your suggestion in two parts:

  • Training vs. test rationales: we clarify where the goal is used;
  • Goal-specific actor rationales: we revisit offline WebShop evidence and add a closed-loop online evaluation.

Weakness / Question

"I do believe in some cases, having goal specific explanations instead of neutral explanations could help the agent select actions better suited to its goal. A natural next step would be evaluating with goal-specific explanations. These may help the reward model select better actions in some cases, while in others they may argue confidently for the wrong one, so testing them would show how far the benefits of PreAct's goal neutral explanation training would extend to goal specific explanations."

We should have made clearer that PreAct's training rationales already use the goal; only WebContrast's test rationales are goal-blind:

  • PreAct training (Section 3.2): the generator sees the goal, page, history and that action, but not the preference label, competing candidate or execution outcomes.
  • WebContrast testing (Section 3.4): rationales are generated without the goal, so the same text can accompany two opposing goals; the PRM still sees the current goal.

We agree that goal-specific actor rationales may help or argue for the wrong action. Submitted Table 6 tests them offline: PreAct outperforms Native in WebShop preference accuracy (Qwen 63.25% → 64.32%, Llama 46.33% → 64.37%).

We now add an online evaluation on 200 tasks. At each step, GPT-4o (gpt-4o-2024-08-06) proposes actions with its own goal-conditioned rationales, and the PRM selects one for execution until the episode ends with terminal reward and success. We also test each PRM without the current rationales:

Cumulative reward sums episode returns over 200 tasks; success rate measures terminal task success. “None” removes the current candidate rationales.

BackboneCurrent rationalesCumulative reward: NativeCumulative reward: PreActSuccess rate %: NativeSuccess rate %: PreAct
Qwen3-8BNatural actor rationales121.77125.8336.541.0
Qwen3-8BNone125.03125.5839.040.5
Llama-3.1-8BNatural actor rationales124.03124.5337.539.5
Llama-3.1-8BNone121.51125.0334.039.0

PreAct exceeds Native in cumulative reward and success rate in all four settings. With the actor's rationales, success rate rises from 36.5% to 41.0% (Qwen) and from 37.5% to 39.5% (Llama). Within PreAct, these rationales add 0.5 pp of success rate on both backbones, with reward changes of +0.25 and −0.50.

We will add this evaluation and clarify the training/test distinction in Section 3.

AuthorsOpenReview ↗

Response (2/2): WebContrast update and external PRMs

Revised WebContrast. After submission, we also expanded and refined the benchmark. We will publicly release the full WebContrast benchmark and the code for the paper’s experiments within one week of acceptance. We report two cohorts:

  • All500: 500 pages, 1,000 goals and 130 websites, obtained by adding 200 harder pages.
  • Core300: the 300-page core of All500. It differs from the submitted benchmark by two page replacements and small revisions to some goals, actions and rationale texts.

Each page keeps shared candidate actions, two opposing goals and fixed goal-blind test rationales. A page counts as correct only if both goals receive the correct action. The rationale effect is this two-goal accuracy with the test rationales minus accuracy without them, in percentage points (pp).

External reward models. Entries are rationale effects (pp) under each model's native interface. Arrows give base → released for six web PRMs, with a fixed protocol within each pair. AgentRM-8B is released-only: it scores with a trained value head that its base model lacks.

Effect = two-goal accuracy with − without rationales (pp); positive values indicate a benefit. Arrows show base → released.

ModelCore300 rationale effect (pp)All500 rationale effect (pp)
WebArbiter-3B+18.83 → −17.42+11.30 → −13.95
WebArbiter-4B+8.08 → −6.33+5.60 → −7.45
WebArbiter-7B+8.92 → −6.75+6.15 → −4.35
WebArbiter-8B+3.58 → −1.33+4.30 → −0.05
Web-Shepherd-3B+9.67 → −0.33+4.60 → 0.00
Web-Shepherd-8B+2.17 → −0.33+0.10 → −1.90
AgentRM-8B (generalist; released only)+5.83+3.20

On both cohorts, all four WebArbiter base models benefit from the same test rationales, whereas their released reward models have negative rationale effects. Reward-model training can therefore lose this benefit. Because WebContrast fixes pages, actions and rationale text, it exposes such regressions directly. AgentRM, a generalist agent PRM trained with a different recipe, shows positive mean effects, so PRM training need not make rationales harmful.

We sincerely thank you again for your positive assessment of our work and for your thoughtful suggestion.

Reviewer_AF6XOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows how training-data construction affects Web Process Reward Models’ (WebPRMs) ability to use candidate-action rationales for judgment. The authors find that WebPRM Collection generates explanations for different kind of actions through different procedures. These differences may show which action is preferred. Therefore, the model may learn to use these experience instead of judging the actions themselves. They propose PreAct, which reconstructs both candidates’ rationales using a common generator with access only to pre-execution information and no preference labels. Then authors make some experiments to test this hypothesis. They also evaluate six released WebPRMs and run controlled training experiments with Qwen3-8B and Llama-3.1-8B-Instruct. As a result , they find that models may learn to prefer an action based on how its explanation was generated. Besides, the same test explanations can help or hurt the model’s action choices. Following PreAct training, the mean effect of rationales on WebContrast two-goal accuracy shifts from −3.58 to +5.58 percentage points for Qwen and from −24.25 to +5.33 for Llama. PreAct also achieves higher mean action-selection accuracy on Collection with pre-execution or no rationales, WebLINX, and offline WebShop comparisons. However, evaluation using Collection’s original rationales still favors original-data training, suggesting that this evaluation can obscure improvements in action judgment and rationale use.

Critical strengths
  1. The paper clearly addresses an important problem about how the training data are built. WebPRM Collection generates explanations differently for preferred and rejected actions. These differences make it possible for the model to learn from the labels. To support this finding, the authors set up text-based tests and evaluations of six released WebPRMs. Besides, they add some more experiments with the Aligned, Reversed, and Shuffled settings to make the results more reliable. When the way that the labels are linked to the explanations is changed, the models learn different action preferences. Even the same test explanations can then affect their choices differently. I take it as one important strength because it shows that how explanations are generated can influence action judgment, rather than just showing a correlation.

  2. An informative evaluation of explanation benefit and benchmark validity. WebContrast checks if the model will work correctly when the goal changes or keep consistent with different explanations. The authors also compare results with and without explanations. This is a valuable test because it helps to show if the model takes use of the explanations effectively or ignore them. Another important finding is that Native training performs better with the original explanations, while PreAct performs better in several other settings. This shows why the choice of evaluation matters and provides useful guidance for selecting and testing WebPRMs.

Critical weaknesses
  1. The experiments do not clearly show which parts of PreAct lead to its gains. PreAct changes the available information, explanation generation, and factual calibration together, without testing their effects separately. They also miss one model trained without the explanations as compared model ; I think removing explanations only at test time does not replace this baseline. Table 5 shows better resistance to neutral-text interference, but Qwen’s get less benefit from explanations over neutral text . Thus, the full procedure helps, but use of explanation content and the need to reconstruct explanations are not very clear.
  2. Evidence that the findings hold in realistic agent settings is not sufficient. WebContrast uses explanations generated without knowing the goal. Also, it lacks an independent human annotation study. Tests on other benchmarks are useful, but zero is included in the confidence intervals of Qwen’s WebShop gains. Besides, Table A20 does not clearly explain that explanations will bring help or disturbance under different training configuration. They should test if the results are influenced by noise or how much influence the noise bring More evidence is needed to show that the findings extend to naturally generated agent explanations and online task performance.
Questions for authors
  1. Which parts of PreAct are necessary? How does it compare with a model trained without explanations under the same conditions? Could you make some ablation about the subcomponent of PreAct to test how each part affect the whole method?
  2. Does PreAct use explanations better, or mainly resist text interference? Table 5 shows less interference from neutral text, but the benefit of real explanations over neutral text decreases for Qwen and increases only slightly for Llama. Could you test if the training process will bring changes of benefit and report the uncertainty?
  3. The paper tests other datasets, but Table A20 does not show better use of explanations clearly. Does PreAct also help models use natural agent explanations? During the validation progress, is there a new test set or let human check? Which conclusion still need online experiment to test?
6 comments
AuthorsOpenReview ↗

Response (1/5): Revised WebContrast and external PRMs

Thank you for your careful and detailed review. We appreciate your recognition that the controlled source–label experiments show how explanation construction can change action judgment, and that separating explanation benefit from accuracy matters for how WebPRMs are evaluated.

We respond to your three questions as follows:

  • Q1: component ablations of PreAct, and a model trained without rationales under the same conditions (Drop);
  • Q2: an extension of the submitted Table 5 control to all 500 pages (All500), presented more clearly, plus new training checkpoints with uncertainty intervals;
  • Q3: a closed-loop WebShop evaluation in which a real actor writes goal-conditioned rationales and the chosen actions are executed.

We first summarize the WebContrast update used in these analyses.

Revised WebContrast. After submission, we expanded and refined the benchmark. We will publicly release the full WebContrast benchmark and the code for the paper’s experiments within one week of acceptance. We report two cohorts throughout:

  • All500: 500 pages, 1,000 goals and 130 websites, obtained by adding 200 harder pages.
  • Core300: the 300-page core of All500. It differs from the submitted benchmark by two page replacements and small revisions to some goals, actions and rationale texts.

Each page keeps shared candidate actions, two opposing goals and fixed goal-blind test rationales. A page counts as correct only if both goals receive the correct action. The rationale effect is this two-goal accuracy with the test rationales minus accuracy without them, in percentage points (pp).

External reward models. The panel below asks whether reward-model training preserves the benefit that base models obtain from the same test rationales. Entries are rationale effects (pp), not accuracies, under each model's native interface. Arrows give base → released for six web PRMs, with the protocol fixed within each pair. AgentRM-8B is reported only as released: it scores through a trained scalar value head its base model lacks, so no comparable base entry exists.

Effect = two-goal accuracy with − without rationales (pp); positive values indicate a benefit. Arrows show base → released.

ModelCore300 rationale effect (pp)All500 rationale effect (pp)
WebArbiter-3B+18.83 → −17.42+11.30 → −13.95
WebArbiter-4B+8.08 → −6.33+5.60 → −7.45
WebArbiter-7B+8.92 → −6.75+6.15 → −4.35
WebArbiter-8B+3.58 → −1.33+4.30 → −0.05
Web-Shepherd-3B+9.67 → −0.33+4.60 → 0.00
Web-Shepherd-8B+2.17 → −0.33+0.10 → −1.90
AgentRM-8B (generalist; released only)+5.83+3.20

On both cohorts, all four WebArbiter base models benefit from the same test rationales, whereas their released reward models have negative rationale effects. Reward-model training therefore need not preserve this benefit. Because WebContrast fixes pages, actions and rationale text, it exposes such regressions directly.

AgentRM, a generalist agent PRM trained with a different recipe, shows positive mean effects, so PRM training does not inevitably make rationales harmful. The matched comparisons in the following parts test how the construction of training rationales affects this benefit.

AuthorsOpenReview ↗

Response (2/5): Q1 — PreAct component ablations

Q1. Which parts of PreAct are necessary, and how does it compare with training without explanations?

"Which parts of PreAct are necessary? How does it compare with a model trained without explanations under the same conditions? Could you make some ablation about the subcomponent of PreAct to test how each part affect the whole method?"

We agree that these comparisons are needed to disentangle PreAct's components. We therefore compare information access, teacher policy and factual review under matched training conditions, alongside a no-rationale training baseline (Drop).

Component ablations. Native uses WebPRM Collection's original rationales, generated differently for preferred and rejected actions. PreAct explains each candidate separately from the same pre-execution information (goal, page, history and that action), without seeing the preference label.

For each backbone, all arms use the same 1,024 states, actions and labels, 192 updates and two training seeds; only training rationale construction changes:

  • A: a Qwen teacher explains each candidate from pre-execution information, with equal access for both candidates.
  • B: the same Qwen teacher as A, but the preferred candidate's generator also sees the real post-execution observation. The rejected rationale is reused verbatim from A, restoring Collection's uneven access.
  • C: a GPT teacher (gpt-6.1-sol) uses the same information as A; both the model and prompting change.
  • Raw vs. Full: the submitted PreAct rationales before and after factual review.

The tested teacher and review variants retain a positive rationale effect on both backbones; restoring uneven access (B) causes the largest accuracy drop.

All500 two-goal accuracy (%), two-seed means. Qwen/Llama denote Qwen3-8B/Llama-3.1-8B-Instruct. “With” includes test rationales; effect = with − no rationale (pp).

Training armQwen no rationaleQwen withQwen effectLlama no rationaleLlama withLlama effect
Native56.6054.75−1.8538.8024.65−14.15
A56.3061.10+4.8052.2054.85+2.65
B50.2051.80+1.6016.4015.85−0.55
C55.9059.55+3.6547.7049.75+2.05
Raw55.7058.80+3.1049.4052.30+2.90
Full56.4060.15+3.7546.4051.00+4.60

Accuracy differences with rationales (pp), with 95% intervals conditional on the two seeds. All500 B−A and C−A use simultaneous intervals over eight pre-specified comparisons; Full−Raw intervals are pointwise; Core300 intervals are descriptive.

BackboneContrastCore300All500
Qwen3-8BB − A (information policy)−12.42 [−15.17, −9.75]−9.30 [−12.08, −6.52]
Qwen3-8BC − A (teacher policy)−1.58 [−3.08, −0.17]−1.55 [−3.18, +0.08]
Qwen3-8BFull − Raw (factual review)+2.00 [+0.83, +3.25]+1.35 [+0.35, +2.35]
Llama-3.1-8BB − A (information policy)−60.33 [−64.25, −56.33]−39.00 [−43.94, −34.06]
Llama-3.1-8BC − A (teacher policy)−7.08 [−9.83, −4.33]−5.10 [−7.75, −2.45]
Llama-3.1-8BFull − Raw (factual review)−2.92 [−5.00, −0.92]−1.30 [−2.75, +0.10]

Teacher and factual review (C, Raw). Across A, C, Raw and Full, the same test rationales increase accuracy on both backbones; under Native, they reduce it. C is below A in absolute accuracy, and factual review helps Qwen but not Llama. The positive rationale effect thus persists across the tested teachers and before factual review, pointing to candidate information access as the key design choice in these comparisons.

Information access (B). Restoring uneven access reduces All500 accuracy with rationales by 9.30 pp on Qwen and 39.00 pp on Llama relative to A. Accuracy without rationales also falls. Llama additionally develops a strong preference for one answer position, suggesting a broader decline in action judgment.

We address training without explanations (Drop) in Part 3.

AuthorsOpenReview ↗

Response (3/5): Q1 — Training without explanations (Drop)

Training without explanations (Drop). As you note, removing rationales only at test time is not the same as training without them. We therefore trained Drop with the current candidate rationales removed from the training input, using the same Collection states, labels, 192 updates and two seeds as Native and PreAct. All three models are evaluated on identical inputs.

The table shows a clear division: Drop is the most accurate without test rationales, while PreAct is the most accurate with them. Values are means over the two seeds:

Two-goal accuracy (%), two-seed means. “With” includes test rationales; effect = with − no rationale (pp).

BackboneTrainingCore300 no rationaleCore300 withAll500 no rationaleAll500 withAll500 effect
Qwen3-8BNative88.8485.6756.6054.75−1.85
Qwen3-8BDrop90.6792.2558.6059.60+1.00
Qwen3-8BPreAct89.0094.0056.4060.15+3.75
Llama-3.1-8BNative63.6739.2538.8024.65−14.15
Llama-3.1-8BDrop79.3478.8350.7050.50−0.20
Llama-3.1-8BPreAct74.8480.4246.4051.00+4.60

PreAct's lead with rationales holds in all four backbone × cohort settings, at 0.50–1.75 pp over Drop. The clearer difference is how each model responds to the same rationales. They raise PreAct's mean accuracy in every setting (+3.75 and +4.60 pp on All500), whereas Drop's response is mixed: positive on Qwen and slightly negative on Llama (+1.00 and −0.20 pp on All500).

One possible reason is a mismatch between training and test inputs: Drop is fine-tuned without current candidate rationales but receives them at test time. We will add the component and Drop results to the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

Thank you for overseeing the discussion. We appreciate the reviewers’ careful reading and the questions that guided our additional experiments.

Strengths highlighted by the reviewers

  1. A learned shortcut in WebPRM Collection. Only explanations for preferred actions were generated with access to post-execution observations. AF6X and mUjj highlighted the controlled result. With actions and labels fixed, swapping which explanation source accompanied preferred actions reversed which source the models favored. An evaluation that reuses the original explanations can reward this shortcut.

  2. Measuring whether explanations help. VER7 valued WebContrast’s requirement to choose correctly for both opposing goals on each page with the same actions and explanations. AF6X highlighted the comparison with and without explanations. mUjj valued measuring goal accuracy, explanation benefit and stability separately.

  3. A practical correction. VER7 called PreAct a “simple and practical fix” for this source of label leakage. One generator explains each candidate separately from the goal and other pre-execution information, without seeing the label.

Our responses and additional evidence

We added 200 harder pages to the revised 300-page core of WebContrast, bringing it to 500 pages, 1,000 goals and 130 websites. On the full set, we compared six WebArbiter and Web-Shepherd releases with their respective starting checkpoints. In every pair, the same explanations raised mean two-goal accuracy at the starting checkpoint and had a zero or negative mean effect at release.

  • C1. Component ablations (AF6X, mUjj) and the Drop baseline (AF6X). Among Native (original training explanations), Drop (current candidate explanations removed during training) and PreAct, PreAct with test explanations achieved the highest mean two-goal accuracy on both backbones and both page sets. With identical test explanations, its two-seed mean exceeded Drop’s by 0.50–1.75 percentage points. Across the matched changes to information access, teacher policy and factual review, accuracy with test explanations fell most under the policy that gave access to the observed outcome only when explaining preferred actions.

  • C2. Explanation benefit and text interference (AF6X). We extended Table 5 to all 500 pages. Relative to no explanations, unrelated text cost both PreAct models about 1.5 percentage points, while real explanations added 3.75 points on Qwen and 4.60 on Llama. Gains over both controls had paired 95% intervals above zero at all three Llama checkpoints.

  • C3. Training seeds, scale and RL (mUjj). Across five seeds at 1,024 training states and three at 2,048, test explanations lowered Native’s accuracy on average and raised PreAct’s accuracy by 2.5–6.3 percentage points in every Qwen and Llama run on the full benchmark. As a first step toward WebArbiter’s complete distillation and RL pipeline, we followed verdict-only fine-tuning with GRPO in a simplified Qwen3-8B run. On the 500-page benchmark, two-goal accuracy rose from 60.6% after fine-tuning to 61.8% at the final GRPO checkpoint.

  • C4. Goal-conditioned explanations (VER7), their benefit on an independent test set (mUjj) and online performance (AF6X). On 200 closed-loop WebShop tasks, GPT-4o proposed actions with its own goal-conditioned explanations, and WebShop executed each action the reward model selected. With these explanations, switching from Native to PreAct raised success from 36.5% to 41.0% on Qwen and 37.5% to 39.5% on Llama. Providing the current explanations raised PreAct’s success by 0.5 percentage points on each backbone.

The WebContrast expansion affects tables and analyses throughout the main text and appendix, so we kept the submitted PDF in place and reported both page sets in our responses. Within one week of acceptance, we will publish the revised paper, release the full updated WebContrast benchmark as open data, and open-source the code for all reported experiments.

We appreciate your time and would be happy to clarify any remaining questions.

Best wishes,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.