acceptodds
Under review as a conference paper at ICLR 2027

biCurriculum: Crediting Executable Task and Environment Rewrites by Their Learning Utility

Abstract

A failed agent trajectory identifies where a policy breaks, but not which training intervention will improve the unchanged task. We introduce biCurriculum, a framework that turns failures into executable task or environment rewrites and credits them by downstream learning utility relative to matched original-task replay. The framework separates native rewrite validity, solver reachability on the child, and original-task improvement after training. On complete WebArena-Lite165, a Qwen3.5-9B rewritten-data arm solves 44/165 tasks versus 34/165 for a replay control and 29/165 for the starting solver; the paired run has 13 rewrite-only versus 3 replay-only successes (exact McNemar ). Across three completed training runs, the rewritten-data arm solves 43–44 tasks while replay solves 33–35, a mean gain of 5.9 percentage points. Controlled ALFWorld results are heterogeneous across the pre-specified matched batches: two matched batches score 107/128 versus 118/128 and 108/128 versus 110/128, while a third reaches 120/128 versus 109/128 (13 versus 2 discordant successes, ). A diagnostic audit shows that only 2/10 originally trained children exercise the parent failure step. On the same three B1 training parents, a failure-step-preserving correction raises native sampled success from 2/12 for the original curriculum to 12/12 (start 5/12; replay 10/12). Together, the results show both when executable rewrites can beat replay and why valid, reachable rewrites can fail: the child must expose the capability missing on the parent.

Primary area:
foundation or frontier models, including LLMs
Keywords:
web agents, curriculum learning, task rewriting, environment rewriting, failure-driven learning, learning utility, agent post-training, data valuation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BAzzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes biCurriculum, a framework for training agents from their failures. When an agent fails a task, a rewriter model creates a modified version of the task or its environment (or both), and the agent is trained on its successes on that modified version. The key idea is to judge each rewrite by whether training on it actually improves the agent on the original task. Results are mixed: training on rewritten tasks helps on WebArena-Lite, but on ALFWorld it helps in one batch, hurts in another, and has no effect in a third. The authors trace part of this inconsistency to rewrites that skip the exact step where the agent originally failed.

Critical strengths
  1. The central idea is intutive and well motivated: a rewritten task being valid and solvable does not guarantee it helps the agent learn, so measuring its value after training, against a fair replay baseline, is a meaningful way to decide which rewrites are useful.
Critical weaknesses
  1. The framework’s main component is a rewriter that learns over multiple rounds from the measured usefulness of its rewrites, but the authors state they do not show this loop produces any gain. The ALFWorld rewriter was trained only to produce valid programs. In the authors’ own check on six failed tasks (according to Figure 3), this training fixed formatting errors, raising valid programs from 2/6 to 6/6, but the rewriter still chose the same kind of edit for each task as before. In other words, it learned to write well-formed rewrites, not to pick more useful ones.
  2. The most encouraging result on WebArena-Lite is underspecified and only loosely controlled. The paper does not explain what the WebArena rewrites look like or which tasks they were built from, even though the evaluation covers all 165 Lite tasks and part of the suite was previously inspected during development. The authors also state that the training budgets of the two arms were not audited for this benchmark, and only one of the three runs has task-level results.
  3. The ALFWorld evidence is small and inconsistent. Each batch compares a single trained model per method on a 128-task development set, and the three batches give a loss, a tie, and a gain, with the positive batch also using a different rewriter. The corrective experiment that preserves the failing step is encouraging, but it covers only three training tasks with repeated samples and does not test held-out tasks.
  4. The paper reads more like a progress report than a finished study. Much of the text is devoted to qualifying what each result does not show, and the main contribution shifts between a framework, an evaluation criterion, and a set of observations.
Questions for authors
  1. For WebArena-Lite, what kinds of task and environment rewrites were used? Which tasks were the failed parents drawn from?

  2. Measuring usefulness requires training and evaluating two model copies for each batch of rewrites. How expensive is this in practice, and would it scale to the multi-round setting the method actually proposes?

Reviewer_NApSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

biCurriculum turns a failed agent trajectory into an executable task or environment rewrite: a child task with its own verifier and a path back to the original. It proposes training the rewriter only on rewrites with positive learning utility, meaning a solver trained on the child improves on the unchanged parent over both the starting solver and a same-start replay copy. With Qwen3.5-9B, rewritten data beats replay on WebArena-Lite in three training runs (43–44 vs. 33–35 of 165 tasks), while three matched ALFWorld batches give a loss, a tie and a win. A model-based audit links the loss to children that skip the parent's failing step, and a step-preserving correction raises success on three training parents from 2/12 to 12/12.

Critical strengths
  1. The credit criterion is well posed. A rewrite is valued after an actual solver update, on the unchanged parent, against a same-start replay copy matched on supervised decisions, target tokens and optimizer steps. This utility is kept separate from validity and reachability.

  2. Reporting is candid: the negative B1 batch is kept, and the paper gives exact paired tests and a matching audit. The failure-step audit (only 2 of 10 trained children exercise the parent's failing step) gives a concrete explanation of when valid, solvable rewrites still hurt.

Critical weaknesses
  1. The titular mechanism is never tested. No utility-gated rewriter update (Eq. 4) is evaluated as an intervention: B3 uses a validity-SFT rewriter, and the utility-surrogate rewriter makes the same six edit choices as before. The paper therefore does not show that utility credit improves rewriting or beats selection by validity or child success.

  2. The headline WebArena-Lite result is the least controlled. The paper never describes the training parents, edit types, number of children or data budget used there, and budget matching is not audited. Part of the suite was previously inspected, only Run A has paired outcomes, and there is no external baseline such as WebRL.

  3. The ALFWorld results cannot separate signal from noise. The arms differ by only 13–15 non-shared supervised decisions, each arm is one adapter trained for 13–14 steps, and the three batches net to about zero (−11, −2, +11 tasks). A historical fixed-rewrite comparison also flipped sign after a refresh. The fourth pre-registered slot (B4) is unreported, and B1–B3 use only goal-truncation edits, so environment rewrites are never tested under the matched protocol.

  4. The mechanism evidence is post hoc and anecdotal. C′ was built after diagnosing B1 and is evaluated only on the same three training parents, with 12 clustered rollouts and a different collection policy. The relevance labels come from model-based annotators on 14 children. This does not support the abstract's claim to explain why rewrites fail.

Questions for authors

Please refer to Weaknesses.

Reviewer_5ZDSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether executable rewrites of tasks or environments provide useful training data for interactive agents. It distinguishes native validity, unaided child-task reachability, and learning utility on unchanged parent tasks. The proposed utility criterion compares a solver trained on verified child trajectories with both its starting checkpoint and a replay-trained copy, with matched supervised decisions, target tokens, update counts, and retention data in the controlled ALFWorld experiments. This criterion is intended to gate supervision of the rewriter.

The empirical evidence consists of 3 WebArena-Lite training runs with a consistent advantage over replay, 3 controlled ALFWorld batches with negative, near-zero, and positive effects, and a small follow-up suggesting that removing the parent's failing step can make an otherwise valid curriculum unhelpful.

Critical strengths

1/ A consequential distinction backed by meaningful controls. Separating executable validity, unaided reachability, and post-update parent performance is a useful way to evaluate curriculum construction.

2/ Informative experiments. a) WebArena: rewritten-data exceeds replay in all three reported runs; b) ALFWorld: paper retains the substantial B1 loss alongside B3's gain, reports the conditional nature of its uncertainty estimates. These observations warrant further investigation.

Critical weaknesses

1/ The distinctive credit-assignment mechanism is not sufficiently validated. Utility-gated rewriter learning is the main methodological distinction from existing task/environment generation and learning-to-teach approaches. However, B3 uses a validity-SFT rewriter, and the construction-pool probe changes schema compliance without changing semantic choices, the later utility-surrogate probe also leaves those choices unchanged. The authors acknowledge that they have not isolated gains attributable to multi-round utility-gated updates. I appreciate this restraint, but it leaves the utility-gated rewriter update supported primarily as a proposal. B1 does demonstrate that validity and reachability are insufficient to guarantee useful training data. What remains unestablished is whether using the comparatively expensive utility criterion to select rewrites produces better subsequent outcomes than selection using these cheaper signals or failure-step relevance.

2/ Missed details for WebArena benchmark. WebArena provides the strongest result, but this paper does not adequately specify the training-parent source, executable edit vocabulary, rewriter configuration, branch-specific training budgets, or concrete evaluation configuration.

Questions for authors

Please see "Critical Weaknesses"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.