- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes biCurriculum, a framework for training agents from their failures. When an agent fails a task, a rewriter model creates a modified version of the task or its environment (or both), and the agent is trained on its successes on that modified version. The key idea is to judge each rewrite by whether training on it actually improves the agent on the original task. Results are mixed: training on rewritten tasks helps on WebArena-Lite, but on ALFWorld it helps in one batch, hurts in another, and has no effect in a third. The authors trace part of this inconsistency to rewrites that skip the exact step where the agent originally failed.
Critical strengths
- The central idea is intutive and well motivated: a rewritten task being valid and solvable does not guarantee it helps the agent learn, so measuring its value after training, against a fair replay baseline, is a meaningful way to decide which rewrites are useful.
Critical weaknesses
- The framework’s main component is a rewriter that learns over multiple rounds from the measured usefulness of its rewrites, but the authors state they do not show this loop produces any gain. The ALFWorld rewriter was trained only to produce valid programs. In the authors’ own check on six failed tasks (according to Figure 3), this training fixed formatting errors, raising valid programs from 2/6 to 6/6, but the rewriter still chose the same kind of edit for each task as before. In other words, it learned to write well-formed rewrites, not to pick more useful ones.
- The most encouraging result on WebArena-Lite is underspecified and only loosely controlled. The paper does not explain what the WebArena rewrites look like or which tasks they were built from, even though the evaluation covers all 165 Lite tasks and part of the suite was previously inspected during development. The authors also state that the training budgets of the two arms were not audited for this benchmark, and only one of the three runs has task-level results.
- The ALFWorld evidence is small and inconsistent. Each batch compares a single trained model per method on a 128-task development set, and the three batches give a loss, a tie, and a gain, with the positive batch also using a different rewriter. The corrective experiment that preserves the failing step is encouraging, but it covers only three training tasks with repeated samples and does not test held-out tasks.
- The paper reads more like a progress report than a finished study. Much of the text is devoted to qualifying what each result does not show, and the main contribution shifts between a framework, an evaluation criterion, and a set of observations.
Questions for authors
-
For WebArena-Lite, what kinds of task and environment rewrites were used? Which tasks were the failed parents drawn from?
-
Measuring usefulness requires training and evaluating two model copies for each batch of rewrites. How expensive is this in practice, and would it scale to the multi-round setting the method actually proposes?