- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper audits label cues in WebPRM explanations and proposes PreAct, a label-blind reconstruction procedure. It introduces WebContrast to test goal sensitivity and explanation effects. Controlled source assignments change learned preferences. PreAct makes identical test explanations more useful, while evaluation with Native explanations favors the original training recipe.
Critical strengths
- Controlled source–label assignments provide evidence beyond input perturbations.
- The evaluation separates goal accuracy, explanation benefit, and stability.
- Length, paraphrase, generator, neutral-text, and readout controls strengthen the analysis.
Critical weaknesses
- PreAct changes information access, generator choice, and factual review together. The final comparison cannot isolate which component produces the improvement.
- The final Native/PreAct comparison uses 1,024 states and verdict-only LoRA SFT. PreAct’s training benefits have not been demonstrated at larger scale or within WebArbiter’s complete reasoning-distillation and RL pipeline.
- WebContrast uses goal-blind explanations to keep text fixed across opposing goals. This enables controlled testing, but differs from typical goal-conditioned actor explanations. Generalization to natural explanation use therefore remains uncertain.
Questions for authors
- Can you provide matched training ablations that separate the effects of pre-execution information access, generator choice, and factual review?
- How sensitive are PreAct’s gains to training-set size and random seed? What evidence supports extending the findings from verdict-only LoRA SFT to WebArbiter’s complete distillation and RL pipeline?
- Can you test whether PreAct improves the benefit of goal-conditioned actor explanations on an independent evaluation set?
4 comments
Response (2/4): Q1 — Matched component ablations
Q1. Matched ablations of information access, generator choice and factual review
"Can you provide matched training ablations that separate the effects of pre-execution information access, generator choice, and factual review?"
You are right that the Native→PreAct comparison changes three things at once. To see how each one matters, we trained matched arms. Within each backbone and seed, every arm uses the same 1,024 Collection states, labels, actions and 192 updates, and all arms are tested on identical WebContrast inputs; only the training rationales differ:
- A: a Qwen teacher explains each candidate from pre-execution information (goal, page, history and that action) without seeing the label.
- B: keeps A's teacher and reuses A's rejected rationales verbatim, but the preferred candidate's generator also sees the real post-execution observation, restoring Collection's asymmetric access.
- C: keeps A's access but uses a GPT teacher policy (
gpt-6.1-sol), which changes both the model and its prompting. - Raw and Full: the submitted PreAct rationales before and after factual review.
Every symmetric, label-blind arm we tested keeps a positive rationale effect, while restoring asymmetric access gives the largest accuracy drop. The first table gives accuracies and effects for each arm; the second reports the matched accuracy contrasts with intervals.
All500 two-goal accuracy (%), two-seed means. Qwen/Llama denote Qwen3-8B/Llama-3.1-8B-Instruct. “With” includes test rationales; effect = with − no rationale (pp). C uses gpt-6.1-sol.
| Training arm | Qwen no rationale | Qwen with | Qwen effect | Llama no rationale | Llama with | Llama effect |
|---|---|---|---|---|---|---|
| Native (Collection rationales) | 56.60 | 54.75 | −1.85 | 38.80 | 24.65 | −14.15 |
| A: symmetric pre-execution, Qwen teacher | 56.30 | 61.10 | +4.80 | 52.20 | 54.85 | +2.65 |
| B: preferred side also sees post-execution | 50.20 | 51.80 | +1.60 | 16.40 | 15.85 | −0.55 |
| C: symmetric pre-execution, GPT teacher | 55.90 | 59.55 | +3.65 | 47.70 | 49.75 | +2.05 |
| Raw: PreAct before factual review | 55.70 | 58.80 | +3.10 | 49.40 | 52.30 | +2.90 |
| Full: submitted PreAct | 56.40 | 60.15 | +3.75 | 46.40 | 51.00 | +4.60 |
Accuracy differences with rationales (pp); brackets give 95% intervals. All500 B−A and C−A use simultaneous intervals over the eight pre-specified comparisons; Full−Raw intervals are pointwise; Core300 intervals are descriptive. All are conditional on the two training seeds.
| Backbone | Contrast | Core300 | All500 |
|---|---|---|---|
| Qwen3-8B | B − A (information access) | −12.42 [−15.17, −9.75] | −9.30 [−12.08, −6.52] |
| Qwen3-8B | C − A (generator / teacher policy) | −1.58 [−3.08, −0.17] | −1.55 [−3.18, +0.08] |
| Qwen3-8B | Full − Raw (factual review) | +2.00 [+0.83, +3.25] | +1.35 [+0.35, +2.35] |
| Llama-3.1-8B | B − A (information access) | −60.33 [−64.25, −56.33] | −39.00 [−43.94, −34.06] |
| Llama-3.1-8B | C − A (generator / teacher policy) | −7.08 [−9.83, −4.33] | −5.10 [−7.75, −2.45] |
| Llama-3.1-8B | Full − Raw (factual review) | −2.92 [−5.00, −0.92] | −1.30 [−2.75, +0.10] |
Generator choice and factual review. After Native training, the same WebContrast rationales lower accuracy on both backbones; after training on A, C, Raw or Full, they raise it on both, including with a different teacher (C) and before factual review (Raw). These choices still shift absolute accuracy: C is below A, and review helps Qwen but not Llama.
Information access. B keeps A's teacher but gives only the preferred candidate's generator the post-execution observation, so the contrast includes both the extra facts and the accompanying label-correlated wording. On All500, accuracy with rationales falls by 9.30 pp on Qwen and 39.00 pp on Llama, and accuracy without rationales falls as well. Llama trained on B also shows a strong preference for one output position.
These results are consistent with the paper's principle that labels should supervise judgment rather than shape how candidate rationales are generated. We will add both tables to the paper.
Response (3/4): Q2 — Training seeds, data size, and GRPO
Q2. Sensitivity to training-set size and seed; extension to the full distillation and RL pipeline
"How sensitive are PreAct's gains to training-set size and random seed? What evidence supports extending the findings from verdict-only LoRA SFT to WebArbiter's complete distillation and RL pipeline?"
We address the two parts in turn: robustness of the results across seeds and a larger training configuration, and extending training beyond SFT with GRPO.
The experiments below use verdict-only LoRA SFT, as in the submission, and additionally evaluate continued training with GRPO. Candidate rationales remain part of the model’s input; SFT supervises only the preferred-action verdict, while GRPO rewards verdict correctness.
Seeds and training-set size. The submitted comparison used two seeds at 1,024 states. We extended it to five seeds at 1,024 states (192 updates) and also trained three seeds at 2,048 states (384 updates; the same three epochs, with the original 1,024 states kept as a prefix). Two results hold in every seed at both sizes: PreAct's own rationale effect is positive, and PreAct is more accurate than Native with rationales. All500, mean over seeds:
All500, seed means in pp. Rationale effect = accuracy with − without rationales; change = PreAct effect − Native effect. SD is the sample standard deviation of this paired change across seeds; fractions count positive results / total seeds.
| Backbone | Setting | Rationale effect: Native → PreAct | PreAct effect > 0 | PreAct − Native accuracy with rationales | Change in rationale effect (SD) |
|---|---|---|---|---|---|
| Qwen3-8B | 1,024 states, 5 seeds | −1.20 → +3.36 | 5/5 | +4.40 (5/5 > 0) | +4.56 (2.02; 5/5 > 0) |
| Llama-3.1-8B | 1,024 states, 5 seeds | −9.80 → +4.64 | 5/5 | +24.56 (5/5 > 0) | +14.44 (11.72; 4/5 > 0) |
| Qwen3-8B | 2,048 states, 3 seeds | −0.63 → +3.00 | 3/3 | +3.23 (3/3 > 0) | +3.63 (0.29; 3/3 > 0) |
| Llama-3.1-8B | 2,048 states, 3 seeds | −14.83 → +3.87 | 3/3 | +25.63 (3/3 > 0) | +18.70 (1.87; 3/3 > 0) |
These repetitions test whether the findings depend on a favorable seed or on the original 1,024-state setting. Across all runs, the same test rationales help every PreAct-trained model, with effects in a narrow band (+2.5 to +6.3 pp on All500), and PreAct is more accurate than Native with rationales in every matched seed. After Native training, the same rationales lower accuracy on average.
Because this pattern recurs over five seeds at 1,024 states and three seeds in the larger 2,048-state configuration, and on Core300 as well as All500, PreAct's positive rationale effects and higher accuracy with rationales are reproducible across the tested seeds and training configurations. This strengthens the paper's central claim that how training rationales are constructed changes whether the same test rationales help action judgment.
Extension to GRPO. The original training targets include reasoning about the original candidate rationales. Our checks confirmed that this reasoning contains statements inconsistent with the reconstructed PreAct rationales. We therefore supervise only the unchanged preference verdict.
Building on this setup, we continued the Qwen3-8B PreAct checkpoint with GRPO. The starting checkpoint was trained on 1,024 states for 192 SFT updates. GRPO uses an augmented set of 516 distinct states with replay and both candidate orders, a binary verdict-correctness reward, seven samples per prompt, and a learning rate of 1.75 × 10⁻⁶. We ran 84 optimizer updates and evaluated three checkpoints.
WebContrast two-goal accuracy (%). Test rationales and the order-symmetrized verdict-margin readout are fixed across checkpoints. Δ denotes the change from SFT in percentage points. Bold marks the highest value in each column.
| Training stage | Core300 | Δ vs. SFT (pp) | All500 | Δ vs. SFT (pp) |
|---|---|---|---|---|
| PreAct SFT | 94.33 | — | 60.60 | — |
| SFT + GRPO, 28 updates | 94.83 | +0.50 | 61.50 | +0.90 |
| SFT + GRPO, 56 updates | 95.00 | +0.67 | 62.00 | +1.40 |
| SFT + GRPO, 84 updates (final) | 95.17 | +0.83 | 61.80 | +1.20 |
PreAct remains above its SFT accuracy at all three evaluated GRPO checkpoints on both cohorts. At the fixed final checkpoint, All500 accuracy reaches 61.80%, up from 60.60% after SFT, alongside a 0.83 pp increase on Core300. This extends our evaluation to subsequent RL optimization of a PreAct-trained model.
The final-minus-SFT 95% paired-page bootstrap intervals are [−0.17, +2.00] pp on Core300 and [−0.20, +2.60] pp on All500.
Response (4/4): Q3 — Independent online evaluation
Q3. Goal-conditioned actor explanations on an independent evaluation set
"Can you test whether PreAct improves the benefit of goal-conditioned actor explanations on an independent evaluation set?"
Thank you for this question. In addition to the submitted offline evidence, we added a closed-loop WebShop evaluation. With the actor's goal-conditioned rationales, PreAct achieves higher success rate and cumulative reward than Native on both backbones, and we report separately how much reading those rationales adds.
Submitted Table 6 already evaluates offline WebShop preference comparisons whose candidates carry the actor's original goal-conditioned rationales. With these rationales, PreAct's preference accuracy is 64.32% vs. 63.25% for Native on Qwen, and 64.37% vs. 46.33% on Llama.
To extend this from offline action judgments to online task performance, we ran a closed-loop evaluation on 200 WebShop tasks, separate from WebContrast's pages and rationales. At each step, GPT-4o (gpt-4o-2024-08-06) proposes candidate actions together with its own original goal-conditioned rationales. The Native- or PreAct-trained PRM reads the goal, page, candidates and rationales and selects one action. The environment executes it, and the loop repeats until the episode ends, when the environment returns terminal reward and success. We evaluate each PRM with and without the current candidate rationales:
Cumulative reward sums episode returns over 200 tasks; success rate measures terminal task success. “None” removes the current candidate rationales.
| Backbone | Current rationales | Cumulative reward: Native | Cumulative reward: PreAct | Success rate %: Native | Success rate %: PreAct |
|---|---|---|---|---|---|
| Qwen3-8B | Natural actor rationales | 121.77 | 125.83 | 36.5 | 41.0 |
| Qwen3-8B | None | 125.03 | 125.58 | 39.0 | 40.5 |
| Llama-3.1-8B | Natural actor rationales | 124.03 | 124.53 | 37.5 | 39.5 |
| Llama-3.1-8B | None | 121.51 | 125.03 | 34.0 | 39.0 |
Task outcomes. With the actor's rationales, success rate rises from 36.5% to 41.0% for Qwen and from 37.5% to 39.5% for Llama. PreAct leads on both metrics in all four settings, so its advantage carries over to natural rationales and executed tasks.
Rationale benefit. The with/without comparison shows what reading the current rationales adds. For PreAct, they raise success rate by 0.5 pp on both backbones, with cumulative reward changes of +0.25 and −0.50. Relative to Native, this rationale effect on success rate is 3.0 pp larger for Qwen and 3.0 pp smaller for Llama, so the change in rationale benefit relative to Native differs by backbone. We will add this evaluation to the paper.
We sincerely thank you again for your positive recommendation and recognition of our work, and for the constructive questions that guided these additional analyses.
Response (1/4): Revised WebContrast and external PRMs
We sincerely thank you for your recommendation to accept our paper, and for recognizing our controlled source–label experiments and the separate evaluation of goal accuracy, explanation benefit and stability. Your constructive questions helped us strengthen the evidence.
We respond to your three questions as follows:
Revised WebContrast. After submission, we expanded and refined the benchmark. We will publicly release the full WebContrast benchmark and the code for the paper’s experiments within one week of acceptance. We report two cohorts throughout:
Each page keeps shared candidate actions, two opposing goals and fixed goal-blind test rationales. A page counts as correct only if both goals receive the correct action. The rationale effect is this two-goal accuracy with the test rationales minus accuracy without them, in percentage points (pp).
External reward models. The panel below asks whether reward-model training preserves the benefit that base models obtain from the same test rationales. Entries are rationale effects (pp), not accuracies, under each model's native interface. Arrows give base → released for six web PRMs, with the protocol fixed within each pair. AgentRM-8B is reported only as released: it scores through a trained scalar value head its base model lacks, so no comparable base entry exists.
Effect = two-goal accuracy with − without rationales (pp); positive values indicate a benefit. Arrows show base → released.
On both cohorts, all four WebArbiter base models benefit from the same test rationales, whereas their released reward models have negative rationale effects. Reward-model training therefore need not preserve this benefit. Because WebContrast fixes pages, actions and rationale text, it exposes such regressions directly.
AgentRM, a generalist agent PRM trained with a different recipe, shows positive mean effects, so PRM training does not inevitably make rationales harmful. The matched comparisons in the following parts test how the construction of training rationales affects this benefit.