- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes CounterFactual Repair, a training framework that improves web agents by identifying and correcting critical mistakes in their failed trajectories. CFR uses a teacher model to propose short repairs, verifies their effectiveness through counterfactual rollouts, and trains the agent on verified corrections. Experiments on WebArena-Lite, WebShop, and MiniWoB++ demonstrate improved task performance with reduced teacher supervision.
Critical strengths
S1. Clear Presentation
The paper is well-written and clearly presents the motivation and proposed methodology.
Critical weaknesses
W1. Limited Methodological Novelty and Missing Closest Baselines
The overall pipeline substantially overlaps with prior work, particularly SRC and CSO, in identifying failure states, applying teacher corrections, verifying continuations, and training on corrected trajectories. While CFR introduces counterfactual admission, this appears to be an incremental extension of existing verification methods. Moreover, direct comparisons with SRC and CSO are missing, making it difficult to assess the significance of the proposed differences.
W2. Insufficient Ablation of Counterfactual Admission
The key methodological distinction of CFR is comparing repaired rollouts against the original policy’s success rate. However, the paper lacks a controlled ablation against simpler success-based filtering. Therefore, it remains unclear whether counterfactual admission provides meaningful benefits beyond conventional verified correction.
W3. Limited Evidence of Superiority Over Existing Methods
Although CFR achieves the highest average performance among the evaluated baselines, its improvements over baselines are not statistically significant in the reported held-out comparisons.
Questions for authors
- Could the authors clarify the key methodological contributions of CFR beyond SRC and CSO, and run these closest baselines?
- Can the authors provide an ablation comparing counterfactual admission with simpler success-based filtering to isolate its effectiveness?
- Given the statistically insignificant performance differences against baselines, what evidence supports CFR’s superiority over these methods? Why should we use CFR?
3 comments
Thank you for the author response.
I appreciate the authors for doing all the experiments. Some of the concerns have been addressed, thus I have increased the score to 2. This is mainly because my main concern "incremental extension of existing verification methods" has not fully been resolved.
Thank you for reading our response and for raising your score. Below we address the remaining concern, that counterfactual admission is an incremental extension of existing verification.
Outcome verification vs. counterfactual admission
Existing verification (CSO, SRC) asks whether a corrected trajectory succeeds. Counterfactual admission asks a different question: does the repair raise this policy's success at this state over what the policy would have done on its own? It measures the repair's effect against a matched control rather than the outcome of a trajectory.
| Outcome verification (CSO, SRC) | Counterfactual admission (CFR) | |
|---|---|---|
| Question | Does the corrected trajectory succeed? | Does the repair raise success over the policy's own action at the same state? |
| Control | None | Matched no-repair arm: the policy re-decides at the same state |
| Executions | One successful rollout (CSO); final outcome (SRC) | K=4 per arm, each from websites rebuilt from fresh containers and the replayed prefix |
| Rejects | Corrections that fail | Corrections that fail, and corrections that work but are not needed |
The two criteria disagree on states the policy already solves. There a correction succeeds, so outcome verification keeps it, yet training on it teaches the policy to replace behaviour that already works. An outcome check cannot detect this, because the corrected trajectory succeeds either way; a test against the policy's own behaviour can.
Evidence that the difference matters
The success-only arm keeps everything in CFR, including K=4 repeated execution from rebuilt states, and drops only the matched control. The two arms share 249 of their 268 training rows; 15 of the 19 rows that differ are repairs verified to work at states the policy already solves (§5.4, App. D.3).
| Same 100 tasks, three training seeds | CFR | Success-only (no control) |
|---|---|---|
| Held-out SR (%) | 15.34 | 14.05 |
| All 165 tasks (%) | 25.72 | 24.04 |
| Retention set (%) | 63.25 | 56.98 (base model 62.39) |
Changing these 19 rows costs 6.27 pp on the retention set, taking the arm below the base model, and CFR is ahead on held-out tasks in every training seed. The matched control is therefore the part of admission that does the work, not a refinement of a success filter.
Contributions beyond admission
| Design choice | Evidence |
|---|---|
| The policy writes the reasoning for the verified action (CSO: expert-written; SRC: action only) | Same rows, actions and seed, teacher-written reasoning instead: policy-written is ahead by 12.82 pp on the retention set and 2.75 pp on held-out tasks (§5.4, App. A.2) |
| The teacher is spent only at the state before the decisive error, with a short repair that hands control back | 7–21× less teacher input than STeP and OEC under one teacher, with the highest held-out success in Table 3 (§5.2) |
We thank the reviewer. Section, table and figure numbers refer to the revised PDF. The general comment lists the changes common to all reviews.
W1. "The overall pipeline substantially overlaps with prior work, particularly SRC and CSO ... direct comparisons with SRC and CSO are missing." Table 19 (App. E) and §2 set out the differences:
Two of these differences have direct evidence: the matched control (without it, the success-only arm falls below the base model on retention; §5.4, W2) and the policy's own reasoning, which beats teacher-written reasoning by 12.82 pp on the retention set with the same rows, actions and training seed (§5.4, App. A.2). CFR also rebuilds stateful websites from fresh containers on every rollout. For CSO, we now report a three-seed adaptation of its recipe on the same 100 tasks (App. A.2, Table 5):
CSO has no public code, and App. A.2 lists our five deviations, including policy-written reasoning. We did not reproduce SRC: it appeared on arXiv in June 2026 (2606.12485), less than four months before the deadline; its teacher reviews each speculative segment online during collection, a different interaction regime; and, like CSO, it keeps trajectories that pass a verifier, without repeated counterfactual executions from a rebuilt state (§2).
W2. "The paper lacks a controlled ablation against simpler success-based filtering." We added this comparison (UcNL Q1a, dtYK W1). Success-based filtering keeps our counterfactual success check and drops the matched no-repair control. On the same 100 tasks, it trails CFR on held-out tasks (14.05% vs. 15.34%), on all tasks, and on the retention set, where it falls 5.41 pp below the base model (§5.4, Table 5). The matched control is therefore part of what admission contributes.
W3. "Its improvements over baselines are not statistically significant in the reported held-out comparisons." The claim of §5.2 is held-out success together with teacher cost. With one teacher for every teacher-based arm, CFR included, CFR reaches the highest held-out success in Table 3 with 7.2× less teacher input than STeP and 21.1× less than OEC (original) (Figure 2). All arms train on the same 100 tasks:
In the two-seed comparison on the main harness (App. D.3, Table 14), CFR's held-out point estimate is 1.65 pp above STeP and 6.20 pp above row-matched OEC.
Q1. "Could the authors clarify the key methodological contributions of CFR beyond SRC and CSO, and run these closest baselines?" See W1 (Table 19 in App. E; the CSO adaptation).
Q2. "Can the authors provide an ablation comparing counterfactual admission with simpler success-based filtering?" See W2 and dtYK W1.
Q3. "What evidence supports CFR's superiority over these methods? Why should we use CFR?"