acceptodds
Under review as a conference paper at ICLR 2027

Learner-Conditioned Experience Rewriting for Language Model Agents

Abstract

A successful agent trajectory demonstrates a solution, but does not identify which parts would improve the current learner. Failed trajectories can likewise contain useful actions. We introduce Learner-Conditioned Experience Rewriting (LER), a framework for converting both kinds of experience into supervision using three distinct signals: satisfaction of a checkable requirement, the learner's unaided reliability on that requirement, and the utility of a candidate update. A shared language model audits experience with training-only references and checkpoint-specific behavioral records, proposes local corrections, and learns from complete action turns admitted by explicit environment checks. Progress-conditioned replay allocates training within this verified pool. To train the rewriting policy, LER compares verified plans through equal-budget temporary updates from a common checkpoint and uses comparisons that pass relative-advantage and positive-gain filters as preference labels. The resulting objective targets measured short-horizon learning utility without differentiating through the updates. Explicit information boundaries and behavior-level measurements connect the construction, allocation, and evaluation of supervision. This formulation separates immediate correction quality from training utility, making subsequent unaided execution the objective of experience rewriting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model agents, experience rewriting, learner-conditioned supervision, learning to teach, preference optimization, adaptive replay

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jrLpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how to make more effective use of data in agent training and proposes a new algorithm that classifies and rewrites trajectories to better exploit the diverse types of information they contain. The authors evaluate the proposed approach on three datasets and demonstrate its effectiveness through experimental results.

Critical strengths
  1. The paper is motivated by a sound premise that effectively leveraging the diverse types of information contained in different trajectories. The proposed algorithm offers a potential solution to this problem and may provide insights for researchers working on agent training.
Critical weaknesses
  1. The paper lacks an in-depth analysis of the experimental findings. Sections 4.2 and 4.3 primarily report numerical results from the tables without offering sufficient interpretation or discussion. As a result, it remains unclear why the proposed approach works, whether it generalizes to larger models, and what limitations it may have. Given that the paper is only eight pages long, excluding the “Limitations” section, nearly a page could be devoted to a more thorough analysis of these questions.

  2. The proposed algorithm is based on DPO, yet the paper does not include a direct comparison with the DPO baseline. This omission makes it difficult to determine whether the proposed modifications provide meaningful improvements over the underlying method and weakens the logical consistency of the paper’s argument.

  3. A more serious concern is that the LER results reported in Tables 1 and 2 appear to be completely inconsistent. This discrepancy raises significant concerns about the reliability and reproducibility of the experimental findings.

Questions for authors
  1. Some references cite the published versions of papers, but the corresponding URLs still link to their arXiv preprints. The authors should update these URLs to point to the official publication pages, where available, to ensure consistency and adhere to standard academic practices.
Reviewer_JfbzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Learner-Conditioned Experience Rewriting (LER), a framework that converts successful and failed agent trajectories into training supervision. A shared model audits and repairs experience using training-only references and checkpoint-specific learner records. Environment checks filter action targets, progress-conditioned replay controls their sampling, and the model receives both execution and audit SFT. The full method compares verified rewriting plans through equal-budget temporary updates from a common checkpoint. Comparisons that satisfy relative-advantage and positive-gain criteria provide preference labels for rewriter DPO. Experiments with Qwen3-4B and Qwen3-8B on AppWorld, BFCL-v3, and ALFWorld report improvements over the evaluated baselines under unaided execution. Component ablations examine the contributions of the main design choices.

Critical strengths
  1. Learning gains provide a meaningful objective for experience rewriting. LER compares verified rewriting plans through temporary learner updates from the same checkpoint under matched budgets. Preference labels require both a relative advantage and positive gain over the starting model. This connects the rewriter’s training objective to the measured effect of its generated supervision on subsequent unaided performance. The design offers a concrete way to tailor training material to the current learner.
Critical weaknesses
  1. Essential implementation details and training records are missing.

The behavior registry and local checkers determine which actions become training targets. The paper does not sufficiently explain how these components are implemented on the three benchmarks or show how an actual trajectory is converted into training samples. Key training settings are left to a run manifest that is not provided. Appendix A.3 calls for preference-admission counts, gain distributions, and preference-stability statistics; these are also missing. These gaps make it difficult to verify how the training supervision and preference labels were constructed, and how the stated protocol connects to the reported experiments.

  1. The evidence for fair comparisons and stable performance gains is insufficient.

The claimed common total budget is not supported by actual resource measurements. The supervision supplied to the adapted baselines and their tuning budgets are also insufficiently documented. The reported gains over the strongest baselines are approximately 1.7–2.4 percentage points. The paper reports Avg@8 scores without uncertainty estimates for these differences or information on independent training runs. The ablation provides evidence for the contribution of the full gain stage, but its added value over simpler, equal-cost training alternatives remains unclear. The current evidence does not adequately establish a stable advantage under comparable resource budgets.

Questions for authors
  1. How is training supervision constructed on each of the three benchmarks? Please explain how the behavior registry and local checkers are used in practice, and provide a representative example showing how an actual execution trajectory is converted into the final training targets.

  2. How much reliable preference data do the temporary learning trials produce? Please report the key trial settings, the numbers of candidate pairs that pass verification and are admitted for DPO, and empirical statistics supporting the reliability of these preferences.

  3. What does learning-gain preference training add beyond simpler training alternatives? The current ablation compares the full gain stage with its removal. Are there comparisons with equal-cost extra SFT or other targeted controls that support the added value of the learning trials and preference DPO?

  4. Are the reported gains supported by comparable resource budgets and reliable statistical evidence? Please provide actual resource usage, including temporary learning trials, verification, and probe evaluation, and clarify whether the baselines receive comparable supervision. Please also report uncertainty estimates for the performance differences and information on independent training runs.

Reviewer_Lvk3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LER constructs training supervision from successful and failed agent trajectories. A shared language model audits and repairs experience using training references and checkpoint-specific learner records. Complete action turns pass environment, protected-effect, and provenance checks before learner-dependent replay and SFT. The full method then compares teaching plans through matched temporary updates from the post-SFT checkpoint and trains plan preferences with DPO, admitting a winner only when both relative advantage and positive learning gain are supported. Experiments report improvements over SFT, GRPO, and targeted-supervision baselines on AppWorld, BFCL-v3, and ALFWorld at 4B and 8B scales. My main concerns are the absence of the actual training/resource configuration needed to assess these comparisons, insufficient isolation of the utility-learning contribution, and unnecessarily difficult exposition.

Critical strengths
  1. The formulation separates three quantities that are often conflated. A valid correction need not address an unmastered behavior, and a preferred correction need not improve the learner. The scoped verification contract, clean execution prefixes, fresh continuation after state restoration, and positive-gain filter make these distinctions explicit (§§3.1–3.4). This is a useful conceptual contribution with sensible information boundaries.

  2. The experimental comparisons target relevant alternatives and data sources. Table 1 compares success-only imitation, GRPO, their combination, and two targeted-supervision adaptations at two scales. Table 2 distinguishes removing learner history, repair generation, failed-source supervision, adaptive replay, and the gain stage. The consistent reported improvements make the framework worth investigating; the paper also carefully limits what each ablation identifies.

Critical weaknesses
  1. The actual experimental configuration is missing, preventing assessment of the comparison with SFT and GRPO. Algorithm 1 describes one LER round, but the PDF does not report how many outer rounds produced Tables 1–2. It also omits the per-method numbers of distinct training tasks and task families, rollouts per task, SFT/GRPO/DPO update steps or epochs, batch sizes, target-token totals and replay exposure, and the sizes and use of behavioral panels and utility probes. Section 4.1 states that methods share a total budget covering collection, repair, verification, temporary trials, probes, and training, but gives neither numerical caps nor measured spending. Thus I cannot determine whether the reported advantage holds under comparable task exposure and total resources.

    The matched candidate trials in §3.4 and Appendix A.3 control an internal A-versus-B teaching-plan comparison; they do not establish resource matching between complete LER and the external baselines. Even equal source-task counts and outer-round counts would not ensure comparable work, since LER additionally performs repairs, audits, behavioral measurements, repeated temporary updates, and probe evaluations. Discarded branches still consume resources, and probe feedback informs preference selection. Appendix A.4 correctly distinguishes same-data, same-token, and same-total-compute comparisons, but the supplied PDF does not provide the run manifest, actual stage-wise ledgers, or budget-performance curves needed to apply that distinction. The appropriate conclusion is that fairness cannot be verified, rather than that unequal spending has been demonstrated. Actual behavior registries, checker coverage/error audits, and restoration/provenance outcomes are also missing, leaving the supervision's validity and cost difficult to assess.

    The exposition compounds this reconstructability problem. The abstract introduces several abstract signals before explaining the two levels of learning, while §§3.3–3.4 require tracking shared execution/rewriting roles, a post-SFT anchor, temporary trial branches, and a frozen DPO reference. Figure 1 and Algorithm 1 help, but the pagination illustration (§1; Appendix A.1) never traces a complete case through repaired targets, candidate datasets, measured gains, the DPO pair, and the retained checkpoint.

  2. The learning-utility contribution is not separately validated. A4 removes both temporary trials and DPO, with losses of 1.52–1.90 points (Table 2), but shared weights allow DPO to change execution directly. Appendix A.4 describes direct-transfer and fixed-learner teaching controls without reporting their results. It is also unspecified whether resources saved by A4 are reallocated to extra SFT or collection, which is necessary to interpret the difference as a benefit at matched total cost. No preference acceptance counts, gain distributions, reversal rates, or training-run uncertainty are reported. The results therefore support a reported end-to-end design difference, but do not isolate whether short-horizon utility labels produce a better rewriter or whether the gain stage is preferable to spending the same resources on additional learner training. This matters given its extra work and the 1.67–2.43-point margins over the strongest adapted baselines.

Questions for authors
  1. Please provide a per-method configuration and resource table for each backbone/benchmark underlying Tables 1–2: distinct source tasks/families and shared splits; outer rounds; rollouts per task; SFT/GRPO/DPO steps, epochs and batch sizes; actual target tokens and repeated exposure; behavioral-panel and utility-probe task counts; and budget caps versus measured stage-wise GPU time, environment calls, and token use. What stopping/checkpoint-selection rules were applied, and were resources saved by A4 or other ablations reallocated? Which comparisons match data exposure, update tokens, or total resources?
  2. Please provide one complete example from an evaluated benchmark, tracing the execution prefix, audit/repair, admitted targets, two plans and datasets, trial outcomes, preference label, and retained checkpoint. Explicitly distinguish the shared roles, temporary copies, and DPO reference.
  3. Are results available for the controls already proposed in Appendix A.4: matched evaluation immediately before/after DPO, frozen old/new rewriters teaching identical learners with equal-budget data, and an equal-cost extra-SFT control? Do short-trial preferences predict downstream gains? Please report training-run uncertainty separately from evaluation decodes and explain the different full-8B rows in Tables 1 and 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.