CounterFactual Repair: Training Web Agents on Verified Fixes of Their Own Mistakes

Web agents built on small language models often fail a task because of a single decisive mistake. Training on teacher demonstrations spends the teacher on every step and on states the agent never reaches, and reinforcement learning needs many interactions. We present CounterFactual Repair (CFR), which teaches an agent the right action exactly where it failed. At the last state from which a failed task can still be completed, a teacher proposes a short repair, and CFR keeps the repair only if execution shows that it helps this policy: rerun from the same state, the agent succeeds far more often with the repair than without it. The agent then learns the verified action with reasoning it writes itself. On WebArena-Lite, CFR improves a Qwen3.5-4B agent on held-out tasks by 6.7 points across three training seeds while keeping the tasks it already solves. On the same training tasks it achieves the highest gain among on-policy expert correction, step-level teacher correction and self-training, with several times less teacher input than the teacher-based alternatives. CFR also does better from the base model's own error states on unseen tasks, and the same pipeline improves agents on WebShop and MiniWoB++.