acceptodds
Under review as a conference paper at ICLR 2027

SEAL: Synergistic Co-Evolution of Agents and Learning Environments

Abstract

Large Language Model (LLM) agents increasingly learn through interaction, yet their training process often evolves unevenly: the policy changes with experience while the learning environment continues to provide largely fixed forms of support. As training progresses, this creates a growing mismatch between what the agent currently struggles with and what its learning process emphasizes. We call this phenomenon Agent–Learning Environment Misalignment (ALEM). We introduce SEAL, a framework for synergistic co-evolution of agents and learning environments. SEAL turns verifier-grounded execution failures into a shared diagnostic signal that coordinates two complementary forms of adaptation: it reshapes the support available in subsequent interactions and adjusts how the policy learns from collected trajectories. In this way, the learning environment follows the agent’s evolving failure profile while both sides remain grounded in the same executable feedback. Across three backbones on BFCL V3, SEAL consistently improves over matched Vanilla RL, reaching 40.25% versus 30.75% with Qwen2.5-7B. A matched factorial ablation further yields a +5.00-point interaction estimate for the coupled configuration. The same trained checkpoints also show positive transfer to the evaluated BFCL V4 and -bench domains. These results suggest that coordinating how an agent learns with the environment it learns from is a promising direction for self-evolving tool-use agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agentic reinforcement learning, self-evolution, tool-use agents, adaptive learning environments, co-evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Gwe9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the mismatch between evolving agent capabilities and relatively fixed training-time support in tool-use agent training, and proposes SEAL. Its core mechanism uses deterministic, verifier-grounded failure diagnosis to coordinate two adaptations: adjusting interface cues and recovery guidance for subsequent training rounds, and reweighting GRPO advantages for the current trajectories. The authors train on BFCL V3 and report improvements over matched Vanilla RL. The same trained checkpoints also exhibit positive transfer to the evaluated BFCL V4 and τ²-bench domains.

Critical strengths
  1. A clear design that uses shared diagnosis to coordinate interaction support and policy optimization. SEAL uses the same verifier-grounded diagnosis to determine trajectory weights for the current policy update and select recovery guidance for subsequent training rounds. This offers a practical design for agent training systems that already have executable tool environments and verifiers.

  2. The paper is generally easy to follow. The motivating example and overview figures provide an intuitive explanation of how verifier-grounded failure diagnosis connects learning-interface evolution with diagnosis-guided advantage reweighting. The distinction between training-time interface adaptation and evaluation under the original interface is also clearly communicated.

Critical weaknesses
  1. The ablations do not fully disentangle the contributions of the components added to Vanilla RL. Appendix H.1 specifies that “Uniform” assigns a weight of 1.5 to verifier failures and 1.0 to successes, rather than assigning a weight of 1.0 to every trajectory. All four factorial conditions also share the base schema affordances. These comparisons do not fully isolate the contributions of static interface enrichment, outcome-dependent weighting, and diagnosis-aware optimization to the overall improvement over Vanilla RL. Additional controls would make this attribution clearer.

  2. The trained backbones are relatively dated, leaving the applicability to more recent models uncertain. Given recent progress in open-source models for tool use in ReAct-style agents, I would like to see whether SEAL remains effective on more recent backbones of comparable scale, such as Qwen3.5-4B and Qwen3.5-9B.

Questions for authors

Please address the concerns raised in the Critical Weaknesses section above. In addition:

  1. Implementation of the diagnostic rules. Section 3.2 describes Ψ\Psi as a deterministic classifier over executable traces, applied in a fixed priority order, and Appendix F clarifies that labels are not determined simply by keywords in error messages. Could you provide more detail about how parser checks, schema validation, execution errors, state transitions, and verifier comparisons are mapped to the operational diagnosis labels? Do this process rely on keyword or pattern matching over error messages?
  2. Selection and sensitivity of the base utilities. How were the base utilities ρ0\rho_0 in Table 1 selected? Were they chosen heuristically or tuned empirically on the development set? How sensitive is SEAL to their numerical values and relative ordering?
  3. More comprehensive cost comparisons. Could you provide a more detailed cost comparison? Does SEAL retain its advantage over Vanilla RL when the total training compute budget is matched?
Reviewer_4zNLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates the gap between LLM agent learning and a static learning environment as Agent-Learning Enviroment Misalignment (ALEM). The authors propose a training framework termed SEAL, to update environment along with policy updates. SEAL operates by transforming execution feedback from an environment into turn-level feedback based on agent rollouts labelled by a set of deterministic rules. The authors experiment on BFCLv3 and evaluate on two additional function-calling benchmarks and demonstrate performance gain over baseline methods.

Critical strengths
  1. The introduction section is easy-to-understand and well-written.
  2. This work addresses an important problem and it is well-motivated in tool-use RL. The idea of co-evolving agent and environment during training is creative and sound.
Critical weaknesses
  1. Figure 2 is not illustrative, it only shows the high-level workflow without explanation. A running example of the training dynamics or a more illustrative figure of the workflow would strengthen the presentation of this work. The methodology section is difficult to understand in terms of how the module works in details, especially in section 3.3.
  2. Many design decisions of SEAL seem to be hand-waiving or under-justified. For example, how the authors choose the exact base utility values (e.g., 1.8 for correct_abstention) are not explained.
  3. The experiments only report a single training run result without further experiment such as training across random seeds to reduce variance in reported results. Besides this, error bars are not shown in evaluation results to demonstrate statistical significance. For example, τ2\tau^2-bench uses LLM-simulated users and therefore it is important to show error bars to ensure the claims are not by chance.
  4. The experiments are limited to training on only one benchmark (i.e., BFCLv3 multi-turn), yet the central contribution of this work is the SEAL training framework -- the proposed training framework may not generalize to other domains. For instance, the deterministic rules from the failure diagnosis module may not generalize well to other tool-use tasks.
Questions for authors
  1. Line 79: "failed" should be italicized instead of "that"?
  2. Why not evaluating environment-tuning checkpoint on BFCLv4 and τ2\tau^2-bench in Table 3? Can you also include it training dynamics analysis (e.g., Figure 3) for comparison?
Reviewer_3GKAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains LLM tool-use agents with GRPO on 400 BFCL V3 multi-turn examples and argues that a fixed training environment falls out of step with the agent’s changing failures (termed ALEM). SEAL uses deterministic rules, largely built on BFCL’s checkers, to label each turn’s failure type, and uses these labels to (1) add training-time hints to the prompt, partly selected from a small fixed library, and (2) reweight each trajectory’s GRPO advantage. The main claim is that coupling the two through a shared diagnosis yields synergistic gains. In single runs, SEAL beats Vanilla RL on three backbones (40.25% vs. 30.75% on Qwen2.5-7B) and Environment Tuning by 2.5 to 3.5 points, with small OOD gains. The synergy claim rests on one factorial ablation: each component alone changes accuracy by −0.25 and +0.75 points, while the combination adds 5.50.

Critical strengths

S1. Clean and consistent comparison against Vanilla RL. Tasks, tools, rewards, and the verifier are unchanged, and all training-time additions are removed at evaluation, so the gains cannot come from a friendlier test. Under this protocol, SEAL beats the matched Vanilla RL baseline in all 12 backbone-by-category cells, including on the tool-specialized ToolACE-2. This is the main reason I believe the recipe is useful in practice, even though it is not clear which component drives the gains.

S2. Simple, cheap, and reproducible diagnosis signal. Using deterministic rules over executable checks instead of an LLM judge makes the training signal auditable and easy to reproduce, at about 7% extra step time. This makes the method easy to adopt and gives a reasonable direction for using verifier information beyond a single binary reward.

Critical weaknesses

W1. The paper does not show that SEAL addresses ALEM, or which part of SEAL produces the gains. ALEM is never measured, and the direct test points the other way: in Table 4, adaptive interface selection alone changes accuracy by −0.25 points and diagnosis-aware weighting alone by +0.75. The central synergy claim rests on a single-run +5.00 interaction, with no seed variance. Much of the gain over Vanilla RL (+4 of +9.5) already appears in the Static+Uniform control, which has no adaptive component but upweights all failures by 1.5. In addition, all case studies compare against the base model rather than Vanilla RL, and there is no task-level or per-label analysis linking the gains to the diagnosed failures. This matters because the paper’s main claim, synergistic co-evolution, is not supported by the evidence.

W2. Limited novelty and overstated positioning. Each component closely follows prior work: the error-message rewriting resembles Environment Tuning’s actionable feedback on the same benchmark, the behavioral guidelines are selected from a small fixed library, and the weighting mainly emphasizes call/no-call labels while averaging away the turn-level information it collects. Adapting the environment to the learner’s changing capability is the core idea of automatic curriculum learning and unsupervised environment design (e.g., POET, PAIRED, Prioritized Level Replay), which are not discussed, and Figure 1 presents prior environment-centric methods as uncoupled from the learner. This matters because, without strong evidence for the coupling, the contribution reduces to an engineering combination of known techniques.

Questions for authors

Q1. [Experiment] Can you measure ALEM directly? For example: the fraction of all-failure groups during training under Vanilla RL and SEAL, the number of prompts that switch from all-failure to mixed groups after a cue is introduced, how the realized utilities ρ_t and the selected cues evolve in the main Qwen2.5-7B run, and what fraction of trajectories in that run receive dynamic guidance (Appendix H.4 reports only 4.57% for the auxiliary run).

Q2. [Experiment] Which component actually produces the gains? Adaptive interface selection alone changes accuracy by −0.25 points and diagnosis-aware weighting alone by +0.75 (Table 4). Can you split the +4 points between Vanilla RL and Static+Uniform into the effect of the static schema cues and the effect of upweighting all failures by 1.5? The labels with weights far from 1 (spurious_tool_call, correct_abstention, empty_turn) all concern whether to call a tool, and correct_abstention is not a failure. How much of the gain remains if these base utilities are set to 1?

Q3. [Experiment] Is there task-level evidence that the gains come from the diagnosed failures? On the evaluation set, how does the distribution of failure labels differ between Vanilla RL and SEAL, and are the gains concentrated on tasks whose failure types received cues or higher weights during training? Can you provide case studies comparing SEAL with Vanilla RL rather than with the base model?

Q4. [Clarification and experiment] How does SEAL differ from Environment Tuning’s actionable feedback, and from unsupervised environment design and learning-progress curricula such as POET, PAIRED, and Prioritized Level Replay? How does SEAL compare with a DAPO-style dynamic sampling baseline, which directly addresses the zero-advantage groups described in Remark 1?

Q5. [Experiment] Can you report multiple seeds for Vanilla RL and the four factorial cells? The synergy claim depends on whether the +5.00 interaction holds beyond a single run.

Q6. [Experiment] How does the SEAL checkpoint perform when evaluated with the training-time cues kept in place versus removed? A large gap would indicate that the cues are not internalized.

1 comment
AuthorsOpenReview ↗

A kind reminder of the upcoming end of the discussion period

Dear Reviewer 3GKA,

Thank you again for your thoughtful review. We would like to kindly follow up on our earlier rebuttal, particularly the additional evidence addressing your central concerns about diagnosis-coupled co-evolution.

We have provided three new analyses directly relevant to W1:

  1. Training dynamics: All-failure groups decrease from 40% to 15%, with measured online guidance activation and all-failure-to-mixed transitions.
  2. Task-level evidence: SEAL recovers 58 tasks failed by Vanilla RL, with 20 reverse transitions, yielding 38 net additional successes.
  3. Cross-seed stability: The factorial interaction remains positive across all four matched seeds, with a mean of +5.625 pp.

We believe these results substantially strengthen the evidence for SEAL's effectiveness and the complementarity of its two adaptive components.

We would greatly appreciate your thoughts on whether these additional findings address your main concerns. We would also be grateful if you could reconsider your assessment in light of the new evidence.

Thank you for your time and consideration.

Comments on the paper

3 comments

Dear Reviewer 4zNL,

Thank you for clarifying the remaining concerns. We have completed additional experiments addressing W2 and are preparing a further experiment for W4.

W2. Quantitative utility sensitivity.

We evaluated three alternative utility configurations on Qwen2.5-7B under the same BFCL V3 protocol:

ConfigurationBFCL V3Δ vs. Original
Original40.25%—
Unit initialization + adaptive updates38.50%−1.75 pp
Compressed utilities39.75%−0.50 pp
Diagnosis-neutral (weight = 1)33.25%−7.00 pp

Three observations emerge:

  1. Initializing all utilities to 1.0 while retaining adaptive updates achieves 38.50%, exceeding Vanilla RL (30.75%) by 7.75 pp.
  2. Compressing the original utility values toward 1.0 yields 39.75%, only 0.50 pp below the original configuration.
  3. Fixing all trajectory weights to 1.0 reduces accuracy to 33.25%, a 7.00-point decrease.

These results suggest that SEAL is robust to moderate changes in utility initialization, while diagnosis-aware weighting contributes substantially to its performance. We will include this sensitivity analysis in the revised manuscript.

W4. Generalization of the training framework.

We agree that transferring a trained policy to another benchmark does not fully establish that SEAL itself generalizes as a training framework.

To directly address this concern, we are preparing an additional training experiment on τ²-bench Retail. We will instantiate environment-specific deterministic diagnoses using its executable tool feedback and task-verification signals, while retaining SEAL's shared-diagnosis mechanism for interface adaptation and trajectory weighting.

The experiment will compare Vanilla RL and SEAL under matched training tasks, backbone, rollout budget, and evaluation protocol. We will report held-out task success rates and relevant training-dynamics statistics once the experiment is complete.

We believe this additional experiment will provide a direct test of whether SEAL's diagnosis-coupled training principle extends beyond BFCL V3.

Thank you again for identifying these important questions. We will share the final measured results of the additional τ²-bench experiment as soon as they are available.

Thank you for providing the additional sensitivity experiment, my W2 concern has been moslty addressed. Regarding W4, I appreciate the planned τ2\tau^2-bench training experiment for evaluating framework generalization. My main concern regarding the framework generalization of SEAL as a training framework remains open. Overall, given the additional results and clarifications, I have raised my score to 2.

Dear Reviewer 4zNL,

Thank you for updating your assessment and for clarifying the remaining question in W4. We have now completed the training experiment you suggested in τ²-bench Retail. On the 40 held-out Retail tasks, SEAL achieved 42.5% success (17/40), compared with 35.0% (14/40) for Vanilla RL, a gain of 7.5 percentage points.

The two experiments answer different questions. Our earlier OOD evaluation tested policy transfer by training on BFCL V3 and evaluating the resulting policies on all 114 Retail tasks. The new experiment tests training-framework generalization. Starting from the same Qwen2.5-7B-Instruct backbone, we trained Vanilla RL and SEAL directly on the 74 Retail training tasks and evaluated both on the same 40 held-out tasks.

MethodHeld-out Retail task success
Qwen2.5-7B-Instruct before RL17.5% (7/40)
Vanilla RL35.0% (14/40)
SEAL42.5% (17/40)

Your concern about the dependence on BFCL’s verifier motivated a specific design choice. In Retail, we derived failure diagnoses from Retail’s own tool feedback and task-verification signals. SEAL then used each diagnosis to guide both interface adaptation and trajectory weighting, just as it does in BFCL V3. With this environment-specific diagnostic mapping, SEAL completed three more held-out tasks than Vanilla RL. This provides initial evidence that the training procedure can be applied effectively in a second environment.

We will include the Retail diagnostic mapping, the 74/40 split, and the held-out results in the revised manuscript. If this experiment addresses your remaining concern about W4, would you consider revising your score upward?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.