acceptodds
Under review as a conference paper at ICLR 2027

CounterFactual Repair: Training Web Agents on Verified Fixes of Their Own Mistakes

Abstract

Web agents built on small language models often fail a task because of a single decisive mistake. Training on teacher demonstrations spends the teacher on every step and on states the agent never reaches, and reinforcement learning needs many interactions. We present CounterFactual Repair (CFR), which teaches an agent the right action exactly where it failed. At the last state from which a failed task can still be completed, a teacher proposes a short repair, and CFR keeps the repair only if execution shows that it helps this policy: rerun from the same state, the agent succeeds far more often with the repair than without it. The agent then learns the verified action with reasoning it writes itself. On WebArena-Lite, CFR improves a Qwen3.5-4B agent on held-out tasks by 6.7 points across three training seeds while keeping the tasks it already solves. On the same training tasks it achieves the highest gain among on-policy expert correction, step-level teacher correction and self-training, with several times less teacher input than the teacher-based alternatives. CFR also does better from the base model's own error states on unseen tasks, and the same pipeline improves agents on WebShop and MiniWoB++.

Primary area:
foundation or frontier models, including LLMs
Keywords:
web agents, language model agents, counterfactual verification, learning from mistakes, supervised fine-tuning, teacher-student distillation, WebArena

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sHz6OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes CounterFactual Repair, a training framework that improves web agents by identifying and correcting critical mistakes in their failed trajectories. CFR uses a teacher model to propose short repairs, verifies their effectiveness through counterfactual rollouts, and trains the agent on verified corrections. Experiments on WebArena-Lite, WebShop, and MiniWoB++ demonstrate improved task performance with reduced teacher supervision.

Critical strengths

S1. Clear Presentation

The paper is well-written and clearly presents the motivation and proposed methodology.

Critical weaknesses

W1. Limited Methodological Novelty and Missing Closest Baselines

The overall pipeline substantially overlaps with prior work, particularly SRC and CSO, in identifying failure states, applying teacher corrections, verifying continuations, and training on corrected trajectories. While CFR introduces counterfactual admission, this appears to be an incremental extension of existing verification methods. Moreover, direct comparisons with SRC and CSO are missing, making it difficult to assess the significance of the proposed differences.

W2. Insufficient Ablation of Counterfactual Admission

The key methodological distinction of CFR is comparing repaired rollouts against the original policy’s success rate. However, the paper lacks a controlled ablation against simpler success-based filtering. Therefore, it remains unclear whether counterfactual admission provides meaningful benefits beyond conventional verified correction.

W3. Limited Evidence of Superiority Over Existing Methods

Although CFR achieves the highest average performance among the evaluated baselines, its improvements over baselines are not statistically significant in the reported held-out comparisons.

Questions for authors
  1. Could the authors clarify the key methodological contributions of CFR beyond SRC and CSO, and run these closest baselines?
  2. Can the authors provide an ablation comparing counterfactual admission with simpler success-based filtering to isolate its effectiveness?
  3. Given the statistically insignificant performance differences against baselines, what evidence supports CFR’s superiority over these methods? Why should we use CFR?
3 comments

We thank the reviewer. Section, table and figure numbers refer to the revised PDF. The general comment lists the changes common to all reviews.

W1. "The overall pipeline substantially overlaps with prior work, particularly SRC and CSO ... direct comparisons with SRC and CSO are missing." Table 19 (App. E) and §2 set out the differences:

CFRCSOSRC
Supervised stateState just before the decisive error of the policy's own failurePRM-flagged failed stepEarliest harmful action (rollback)
Correction1–3 actions (4 in two states), then hand backOne expert actionOne action
Verified by executionK=4 counterfactual rollouts with the repair, vs. a matched no-repair controlOne rollout succeedsFinal outcome
Reasoning fromPolicyExpert (our adaptation: policy, as for CFR)None (action only)

Two of these differences have direct evidence: the matched control (without it, the success-only arm falls below the base model on retention; §5.4, W2) and the policy's own reasoning, which beats teacher-written reasoning by 12.82 pp on the retention set with the same rows, actions and training seed (§5.4, App. A.2). CFR also rebuilds stateful websites from fresh containers on every rollout. For CSO, we now report a three-seed adaptation of its recipe on the same 100 tasks (App. A.2, Table 5):

Same 100 tasks, three seedsHeld-out SR (%)vs. CFR, held-outvs. CFR, all 165Teacher input per rowObjective; verification
CFR, 268-row arm15.345.9KSFT; K=4 counterfactual rollouts
CSO adaptation (pairs only; a different teacher as expert)13.13−2.20−0.405.6KDPO; one successful rollout

CSO has no public code, and App. A.2 lists our five deviations, including policy-written reasoning. We did not reproduce SRC: it appeared on arXiv in June 2026 (2606.12485), less than four months before the deadline; its teacher reviews each speculative segment online during collection, a different interaction regime; and, like CSO, it keeps trajectories that pass a verifier, without repeated counterfactual executions from a rebuilt state (§2).

W2. "The paper lacks a controlled ablation against simpler success-based filtering." We added this comparison (UcNL Q1a, dtYK W1). Success-based filtering keeps our counterfactual success check and drops the matched no-repair control. On the same 100 tasks, it trails CFR on held-out tasks (14.05% vs. 15.34%), on all tasks, and on the retention set, where it falls 5.41 pp below the base model (§5.4, Table 5). The matched control is therefore part of what admission contributes.

W3. "Its improvements over baselines are not statistically significant in the reported held-out comparisons." The claim of §5.2 is held-out success together with teacher cost. With one teacher for every teacher-based arm, CFR included, CFR reaches the highest held-out success in Table 3 with 7.2× less teacher input than STeP and 21.1× less than OEC (original) (Figure 2). All arms train on the same 100 tasks:

MethodHeld-out SR (%)Teacher input
CFR (ours)14.510.32M (1.0×)
STeP14.467.2×
OEC, original14.3321.1×
OEC, row-matched9.5018.9×
RFT12.12no teacher
Base model11.29

In the two-seed comparison on the main harness (App. D.3, Table 14), CFR's held-out point estimate is 1.65 pp above STeP and 6.20 pp above row-matched OEC.

Q1. "Could the authors clarify the key methodological contributions of CFR beyond SRC and CSO, and run these closest baselines?" See W1 (Table 19 in App. E; the CSO adaptation).

Q2. "Can the authors provide an ablation comparing counterfactual admission with simpler success-based filtering?" See W2 and dtYK W1.

Q3. "What evidence supports CFR's superiority over these methods? Why should we use CFR?"

ResultEvidenceWhere
Held-out gain with retention kept9.6% → 18.0%, a gain in every training seed§5.1
Highest held-out success in Table 3 (one teacher)14.51% vs. 14.46% (STeP), 14.33% (OEC), at 7–21× less teacher input§5.2, Table 3
Less teacher input7–21× less than STeP and OEC; an estimated 37× less than full demonstrations§5.2, Figure 2
CSO adaptation at similar cost per rowCFR (268-row arm) +2.20 pp held-out; CSO's expert is a different teacherApp. A.2
Beyond WebArenaWebShop reward +12.2; MiniWoB++ also improves§5.5

Thank you for the author response.

I appreciate the authors for doing all the experiments. Some of the concerns have been addressed, thus I have increased the score to 2. This is mainly because my main concern "incremental extension of existing verification methods" has not fully been resolved.

Thank you for reading our response and for raising your score. Below we address the remaining concern, that counterfactual admission is an incremental extension of existing verification.

Outcome verification vs. counterfactual admission

Existing verification (CSO, SRC) asks whether a corrected trajectory succeeds. Counterfactual admission asks a different question: does the repair raise this policy's success at this state over what the policy would have done on its own? It measures the repair's effect against a matched control rather than the outcome of a trajectory.

Outcome verification (CSO, SRC)Counterfactual admission (CFR)
QuestionDoes the corrected trajectory succeed?Does the repair raise success over the policy's own action at the same state?
ControlNoneMatched no-repair arm: the policy re-decides at the same state
ExecutionsOne successful rollout (CSO); final outcome (SRC)K=4 per arm, each from websites rebuilt from fresh containers and the replayed prefix
RejectsCorrections that failCorrections that fail, and corrections that work but are not needed

The two criteria disagree on states the policy already solves. There a correction succeeds, so outcome verification keeps it, yet training on it teaches the policy to replace behaviour that already works. An outcome check cannot detect this, because the corrected trajectory succeeds either way; a test against the policy's own behaviour can.

Evidence that the difference matters

The success-only arm keeps everything in CFR, including K=4 repeated execution from rebuilt states, and drops only the matched control. The two arms share 249 of their 268 training rows; 15 of the 19 rows that differ are repairs verified to work at states the policy already solves (§5.4, App. D.3).

Same 100 tasks, three training seedsCFRSuccess-only (no control)
Held-out SR (%)15.3414.05
All 165 tasks (%)25.7224.04
Retention set (%)63.2556.98 (base model 62.39)

Changing these 19 rows costs 6.27 pp on the retention set, taking the arm below the base model, and CFR is ahead on held-out tasks in every training seed. The matched control is therefore the part of admission that does the work, not a refinement of a success filter.

Contributions beyond admission

Design choiceEvidence
The policy writes the reasoning for the verified action (CSO: expert-written; SRC: action only)Same rows, actions and seed, teacher-written reasoning instead: policy-written is ahead by 12.82 pp on the retention set and 2.75 pp on held-out tasks (§5.4, App. A.2)
The teacher is spent only at the state before the decisive error, with a short repair that hands control back7–21× less teacher input than STeP and OEC under one teacher, with the highest held-out success in Table 3 (§5.2)
Reviewer_dtYKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

CFR identifies critical states from a student model’s failure trajectories, allowing a teacher to provide short fixes. Training samples are then filtered based on the difference in subsequent success rates between the group that received these fixes and a group that relied on autonomous decision-making. Finally, SFT is performed using a combination of reasoning content generated by student paired with corrective actions provided by teacher, mixed with the student’s own successful trajectories. The paper reports performance gains on three benchmarks, while highlighting both the efficiency of teacher input and the preservation of the model's existing capabilities.

Critical strengths

This paper addresses the failure states of Web Agents by performing localized error correction, utilizing comparative execution under identical states to filter and refine fixes; this ensures that the supervisory signals are more closely aligned with the actual needs of the student model. The performance gains over baseline models on WebArena are substantiated by results across three different training seeds and a statistical sensitivity analysis, while supplementary experiments on WebShop and MiniWoB++ further demonstrate the broader applicability of this approach.

Critical weaknesses

While existing ablation studies demonstrate the utility of providing repair examples and student-generated reasoning, they fail to compare the full CFR framework against a version that retains only successful repairs without running an autonomous control group. The primary model shows an improvement of 11.53 pp on the 53 tasks that share templates with the training tasks, but only a 2.94 pp improvement on the 68 tasks that do not share templates. This supports the conclusion that transfer occurs for instances of new tasks, while transfer to entirely new types of tasks remains uncertain. The text contains numerous block quotations, making it difficult and obscure to read.

Questions for authors

Please see weaknesses above.

3 comments

We thank the reviewer. Section, table and figure numbers refer to the revised PDF. The general comment lists the changes common to all reviews.

W1. "They fail to compare the full CFR framework against a version that retains only successful repairs without running an autonomous control group." We ran this comparison. The autonomous control group is the self arm of the admission rule (§3), so the arm you describe keeps the counterfactual success check and drops the difference check. Both arms train on the same 100 tasks with three training seeds and run on one serving stack (§5.4, Table 5):

ArmHeld-out SR (%)All 165 SR (%)Retention SR (%)Retention vs. base (pp)
CFR15.3425.7263.25+0.85
Success only (no matched control)14.0524.0456.98−5.41

By training seed (App. D.3), CFR is ahead on held-out tasks in all three seeds and on the retention set in two of three:

SR (%), CFR / success onlySeed 1Seed 2Seed 3
Held-out14.88 / 13.7713.77 / 12.1217.36 / 16.25
Retention64.96 / 54.7058.12 / 47.8666.67 / 68.38

The clearest separation is on the retention set, consistent with verified repairs at already-solved states displacing behaviour that works; UcNL Q1a compares this with the no-admission arm's high-baseline rows. Dropping verification altogether (the no-admission arm) halves the held-out gain over the base model, from +4.59 to +2.30 pp. These numbers pool three training seeds of the 268-row arm against the base model re-evaluated on the second serving stack (10.74% held-out), whereas the +5.79 of Table 9 (App. B) compares one training seed against the earlier base run (9.6%).

W2. "Improvement of 11.53 pp on the 53 tasks that share templates ... only 2.94 pp on the 68 tasks that do not." Under the corrected scoring these values are +12.37 and +5.23 pp (§5.1, Table 13). The main arm therefore gains on held-out tasks with no training template as well, and that gain is positive under all four scorings we report (Table 7):

Main arm, held-out gain (pp)nRecorded (submitted)Deterministic onlyReviewed (paper)One judge for all arms
Template shared with a training task53+11.53+11.62+12.37+12.37
No shared template68+2.94+2.83+5.23+4.41

n counts all held-out tasks in each group; the deterministic-only column leaves out the judge-scored tasks among them.

The 268-row arm gains +4.55 pp on the 88 held-out tasks whose templates it never trained on. On the 68 of them that share no template with any CFR training task, its three training seeds gain +1.47, 0.00 and +3.43 pp by seed (+1.63 pooled; +0.61, 0.00 and +4.24, pooled +1.62, counting deterministic tasks only), so no seed is below zero under either scoring (§5.1, App. D.2).

W3. "The text contains numerous block quotations, making it difficult and obscure to read." We understood this to refer to the long captions above the tables and the many short paragraphs with bold run-in labels. The revision changes:

ItemSubmittedRevised
Main-text captions of Tables 1, 3, 4Long, with protocol detailsOne or two lines; protocol details moved to §4 and Table 18
§1, §3, §5, §6Bold run-in labelsTopic sentences; the main text reads as continuous prose
CFR arm names(added)Fixed short names throughout: main arm; 268-row arm on the 100 shared tasks; one-teacher arm in the baseline comparison
§5.4 ablation result(added)Short paragraphs and one compact table (Table 5)
Numbers in §5 proseOften several per sentenceOne or two per sentence; §5 opens by naming the three CFR arms; secondary values in tables and the appendix
Serving stacksRestated in the main textThe main text points to one table; Table 18 (App. D) lists serving stack, scoring, evaluation seeds and training seeds for every WebArena-Lite comparison, including Table 3

Q. "Please see weaknesses above." Answered above.

Reviewer_dtYKOpenReview ↗

Official Comment by Reviewer dtYK

Thank you for the clarification and the additional experiments, which have addressed some of my concerns. I will raise my score to 2.

Thank you for raising your score. Your comment says that some of your concerns have been addressed but not which ones remain open, so below we list, for each of your points, the evidence that we believe answers it; all of it is in the revised PDF. If any point is still unresolved, please let us know and we will respond if the discussion period allows.

Your pointEvidence in the revision
W1: retaining only successful repairs, without the autonomous control groupRun on the same 100 tasks with three training seeds. CFR is ahead on held-out tasks in every seed (15.34% vs. 14.05% pooled), and without the control the arm falls 5.41 pp below the base model on the retention set. Dropping verification altogether halves the held-out gain, from +4.59 to +2.30 pp (§5.4, Table 5, App. D.3).
W2: gain on tasks that share no template with trainingThe main arm gains +5.23 pp on the 68 held-out tasks that share no template with any training task, and the gain is positive under all four scorings we report (§5.1, Tables 7 and 13).
W3: readabilityRun-in labels replaced by continuous prose in §1, §3, §5 and §6; main-text captions cut to one or two lines; one table (Table 18) lists the evaluation settings of every comparison.

If you find that these answer your concerns, we would be very grateful if you would consider raising your score to 3. Thank you again for your time and your careful review.

Reviewer_UcNLOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CounterFactual Repair (CFR), a data-generation and supervised fine-tuning pipeline for small web agents. A teacher identifies a repairable state in a failed student trajectory and proposes a short corrective action sequence. The sequence is retained only when repeated execution from reconstructed states meets both an absolute success threshold and an improvement threshold over the student's unassisted continuation. The student supplies the reasoning associated with the verified actions, and repair examples are mixed with its own successful steps for training. With Qwen3.5-4B, CFR improves success on the 121 training-disjoint WebArena-Lite tasks from 9.9% to 16.6% across three training seeds. The paper also reports adapted OEC/STeP/RFT comparisons, evaluation from unseen-task error states, and additional training experiments on WebShop and MiniWoB++.

Critical strengths
  1. The supervision is grounded in executable, student-specific interventions. Comparing repaired and unassisted continuations from reconstructed states is a well-motivated way to avoid treating every apparent failure as useful supervision. Fresh containers, action-identity checks, and explicit population accounting address practical sources of invalid comparisons (§3; Appendix A). Table 2 reports a pooled improvement of 6.70 percentage points, with positive task-bootstrap intervals for every training seed; the two-level bootstrap in Appendix D.2 also remains positive. This supports the usefulness of the complete pipeline under the evaluated setup.

  2. The paper investigates an important training-distribution issue rather than reporting gains alone. Appendix A.2 holds prompts, actions, and training rows fixed while replacing student-generated reasoning with teacher reasoning. Student reasoning improves full-set performance by 4.85 points and retention by 17.95 points in that comparison. Together with the repair-versus-successful-row controls and additional environments, this makes the paper informative about how to construct corrective training targets without unnecessarily degrading existing behavior.

Critical weaknesses
  1. The defining admission rule is not isolated by the ablations. The reported controls remove repair rows, replace them with successful rows, or change the reasoning source, but do not directly compare the two-arm filter against a repair-success-only filter on the same candidate pool. Consequently, the experiments support the full CFR recipe more strongly than the specific benefit of subtracting the self-arm success rate. With only four rollouts per arm and repeated candidate versions, the admitted set may also contain noisy selections; Appendix A acknowledges this issue. This matters because counterfactual admission is presented as a primary contribution and incurs substantial rollout cost. A limited filter ablation or independent re-verification of admitted repairs would materially strengthen the argument.

  2. The main generalization endpoint was also used for development, and transfer to new task families is less established. Section 4 states that WebArena-Lite served as the development evaluation set. Appendix D.2 further reports +11.53 points on 53 held-out tasks sharing a training template, versus +2.94 points with an interval crossing zero on the other 68. This does not negate the aggregate improvement or imply direct training contamination, but it limits the interpretation as development-independent generalization. The same appendix includes useful counterevidence from the 100-task arm, so I would characterize the result as promising task-level transfer with uncertain broader coverage, rather than uniformly strong transfer to unseen families.

Questions for authors
  1. Limited validation of the central mechanism: On the existing candidate pool, how does two-arm admission compare with keeping repairs based only on repaired-rollout success, controlling training size and reasoning generation? If new training is impractical, can fresh verification seeds quantify how often admitted repairs retain their estimated advantage? Separately, what supports identifying the teacher-selected state as the last completable state, rather than a useful intervention point under the current policy and budget? Qualifying this claim would suffice if no stronger localization evidence is available.
  2. Clarification of evaluation independence: Which method, hyperparameter, or checkpoint choices used the 121-task development endpoint? Is there an existing result on tasks withheld from both training and development, and how should the template-stratified findings constrain the main claim?
  3. Clarification of practical efficiency and reproducibility: What teacher model/version and settings were used? Can the main comparison foreground the rejection-inclusive teacher cost alongside rollout counts? Appendix C gives roughly 2.3-fold teacher-input savings with localization included, but also 1,519 direct CFR rollouts and an amortized 4.2K–6.2K. These are different conclusions from a total-compute advantage.
3 comments

We thank the reviewer. Section, table and figure numbers refer to the revised PDF. The general comment lists the changes common to all reviews. Our answers are split over two comments (1/2: admission; 2/2: evaluation and teacher).

W1. "The defining admission rule is not isolated by the ablations ... A limited filter ablation or independent re-verification of admitted repairs would materially strengthen the argument." The two checks of the admission rule (§3) serve different roles, and we now test both. The success check (SR_rep ≥ 0.75, every rollout replayed) is the counterfactual test: rollouts replay the policy's own prefix from a rebuilt state and substitute the repair for the policy's action at s_w, so an admitted repair works for this policy at this state. Without it, the data are only as good as the teacher. The difference check (SR_rep − SR_self ≥ 0.5) compares against a matched no-repair control and keeps repair rows only where the policy would fail on its own, because verified repairs at already-solved states can displace behaviour that works (§3). We ran your filter ablation and re-verified the states (§5.4):

TestResult
Drop the difference check (success-only arm, Q1a)Trails CFR on held-out tasks, all 165 tasks and the retention set; falls below the base model on retention
Drop admission (no-admission arm, Q1a)Held-out gain over the base model halves, +4.59 to +2.30 pp
Re-run 116 of 117 admitted states on four new seeds never used for selection (Q1b)Repair 81.68% vs. 24.14% for the policy's own continuation (+57.54 pp); 74 of 116 pass both thresholds again

Q1a. "On the existing candidate pool, how does two-arm admission compare with keeping repairs based only on repaired-rollout success?" We trained this arm on the same 100 tasks, recipe and three training seeds as CFR's 268-row arm: CFR's 186 on-track rows plus 82 repair rows selected by the same procedure but admitted by the success check alone. Of those, 63 are CFR's, 15 rows from 13 states carry repairs that pass the success check but that the difference check rejected because the policy already succeeds there, and 4 carry admitted repairs that differ from CFR's (App. D.3). All arms run on the second serving stack, where the base model solves 10.74% of held-out tasks (Table 5):

Arm (three training seeds)Held-out SR (%)Δ vs. base, held-outΔ vs. base, all 165Δ vs. base, retention (39)
CFR15.34+4.59+1.68+0.85
Success only14.05+3.310.00−5.41
No admission13.04+2.30+0.81−0.28

CFR is ahead of the success-only arm on held-out tasks in every training seed. The retention gap reflects the difference check's role: the success-only arm's 15 extra rows are verified teacher actions at states the policy already solves. The no-admission arm also carries 18 high-baseline rows but stays 0.28 pp below the base model on retention; those rows are the earliest repair at each state, whose success was never established.

Q1b. "Can fresh verification seeds quantify how often admitted repairs retain their estimated advantage?" Yes (§5.4; App. A, "Verification"). Both arms regress from their admission values, as expected when selecting at K=4, and the repair keeps a large advantage:

116 admitted statesRepair SRSelf SRAdvantage (pp)
At admission (K=4)92.46%15.52%+76.94
Four fresh seeds, all states81.68%24.14%+57.54
Fresh seeds, repair ends the episode (32 states)+71.88
Fresh seeds, repair returns control to the policy (84 states)+52.08

On the fresh seeds, 74 of the 116 states pass both thresholds again; 90 meet the success threshold and 83 the difference threshold. On repeated candidate versions: each training state was judged over 1.62 versions on average (median 1), and one admitted version enters training (App. A, "Verification").

Q1c. "What supports identifying the teacher-selected state as the last completable state?" We adopted your qualification and replaced the term. The paper now refers to s_w as "the state just before the decisive error, as judged by the teacher" (§1, §3, Figure 1): an intervention point chosen by reading the episode, whose usefulness for this policy admission then tests by execution (§3). It need not be the first deviation; in 11 to 23 of the 117 training states the agent had already strayed, and the repair first undoes that deviation (§3). In 85 of the 117 states, the repair hands control back to the policy (§2, App. A).

(2/2) W2 / Q2a. "Which method, hyperparameter, or checkpoint choices used the 121-task development endpoint?" We used WebArena-Lite to set the learning rate, the number of epochs, and the training-data mix. The admission rule, K, the training seeds, and the baseline row recipes were fixed without it, and every arm is evaluated at its last checkpoint. Every baseline and ablation arm uses this same recipe (App. B), so no method was tuned on WebArena-Lite separately.

Q2b. "Is there an existing result on tasks withheld from both training and development?" Yes, on WebShop (§5.5). We kept the WebArena pipeline and training recipe unchanged and used WebShop's 200 test tasks (AgentEval's split, disjoint from the training tasks) only to choose the base model size, before any repair or training data existed (App. D.5). The reward gain meets our pre-specified criterion, a positive lower bound on the paired reward difference.

WebShop test (200 tasks)CFRBase model
Reward32.1519.97
Success (%)22.012.8

Q2c. "How should the template-stratified findings constrain the main claim?" §5.1 and Table 13 now report this split for both arms. The main arm gains on the held-out tasks that share no template with a training task under all four scorings of Table 7:

Main arm, held-out gain (pp)nRecordedDeterministic onlyReviewed (paper)One judge
Template shared with a training task53+11.53+11.62+12.37+12.37
No shared template68+2.94+2.83+5.23+4.41

n counts all held-out tasks in each group; the deterministic-only column leaves out the judge-scored tasks among them.

The 268-row arm gains +4.55 pp on the 88 held-out tasks whose templates it never trained on. On the 68 of them that share no template with any CFR training task, its three training seeds gain +1.63 pp (+1.62 counting deterministic tasks only), and no seed is below zero (§5.1, App. D.2). We describe the result as transfer to new tasks on WebArena's sites.

Q3a. "What teacher model/version and settings were used?" Every teacher follows the same written protocol; App. A ("Teachers") details what each call sees, and App. C reports the mean input per call.

ExperimentTeacher (role)Completion cap per callMean input per call, localisation / repair
Main training data; 268-row arm (App. B); CFR arms of §5.4Claude Opus 5 (localisations, repairs)2.1K / 6.3K (268-row arm, restored in App. C)
Baseline comparison (§5.2, Table 3)GPT-5.6 Sol (CFR's localisations and repairs, OEC's expert, STeP's teacher)32,0003.0K / 12.0K (CFR)
CSO adaptation (App. A.2)GPT-5.6 Sol (expert)12,000
Unseen-task test (§5.3)GPT-5.6 Sol (error-state localisation)
WebShop; current-pipeline MiniWoB++GPT-5.6 Sol (localisations, repairs)

Q3b. "Can the main comparison foreground the rejection-inclusive teacher cost alongside rollout counts?" The fair comparison applies one rule to all methods, the unified idealised count: each method is charged for the teacher calls needed to produce its training rows (§5.2, Figure 2, Table 11). App. C adds the rejection-inclusive count as a worst case, the accounting most adverse to CFR: it charges CFR for every teacher call that returned a reply, including every candidate its verification rejects, and the baselines for all their calls. For the 268-row arm (App. B), App. C now also restores the worst case, charging every verified candidate up to a state's first admitted one (9.9 per admitted state) and one localisation per failed episode. §5.2 now states the rollout counts in the main text.

CountCFRSTePOECRFTFull demos (est.)
Teacher input, unified idealised, one-teacher arm42 calls, 0.32M7.2×21.1× (original)no teacher37.0×
Teacher input, worst case, one-teacher arm194 calls, 1.41M6.6× (461 calls)6.7× (547 calls)
Teacher input, worst case, 268-row arm789 calls, 4.09M2.3×, fewer calls2.3×, fewer calls
Environment rollouts, 100 tasks (Table 12)1,519 (268-row arm; 4.2K to 6.2K amortised over training states)111232401

The rollouts run on self-hosted sites, costing compute without using teacher tokens; the self-arm rollouts among them are what aim CFR's supervision at the policy's own failures. We claim a teacher-input advantage, not a total-compute advantage.

Thank you for your explanation. I have no further questions and will retain my score.

Comments on the paper

1 comment

We thank the three reviewers. The revision makes five changes that several of you asked for.

#ChangeWhat we didWhere
1Scoring correctionEvery arm uses one reviewed final score per answer on the 32 judge tasks; reference answers and deterministic scoring are unchangedApp. A ("Scoring of judge tasks"); Table 7
2Three new tests of admissionRe-verified 116 of the 117 admitted training states on four fresh seeds (one state did not replay in the self arm); trained a no-admission arm on repairs regardless of their verdicts; trained the success-only arm you asked for, which admits repairs by the success check alone§3 (role of each check); §5.4; Tables 5, 15, 16; App. D.3
3CSO adaptationSame 100 tasks, three training seeds, teacher cost under the same count as the other methodsApp. A.2; Tables 5, 11
4PresentationShorter introduction (three contributions) and captions; related-work table moved to the appendix; admission ablations in a main-text table; topic sentences in place of run-in labels; claims about teacher input scoped to the methods they compare against§1, §5, §5.2, Table 5, Figure 2, App. E
5One teacher for the baseline comparisonOEC, STeP and RFT against a CFR arm on the same 100 tasks whose localisations and repairs come from the same teacher as OEC's expert and STeP's teacher (GPT-5.6 Sol), three training seeds for CFR. CFR has the highest held-out success, with 7–21× less teacher input than STeP and OEC§5.2; Table 3; App. A; single-seed main-harness comparison in App. B (Tables 9, 10)

The scoring correction was needed because the LLM judge backend for the 32 WebArena-Lite judge tasks changed during our evaluation window. The main held-out gain moves from +6.70 to +8.36 pp, and the gain over the base model holds under all four reported scorings and in every training seed (Table 7):

Main arm, gain over base (pp)Recorded (submitted)Deterministic tasks onlyReviewed (this paper)One judge for all arms
Held-out (121)+6.70+6.73+8.36+7.90
All 165+5.79+5.85+7.54+7.47
Retention (39)+5.70+6.94+7.41+9.69

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.