acceptodds
Under review as a conference paper at ICLR 2027

State2State: Environment-Derived Mid-Training for LLM Agents

Abstract

Training LLM agents commonly relies on supervised fine-tuning from expert trajectories or online reinforcement learning over human-specified tasks with handcrafted verifiers. Though effective, both remain bottlenecked by externally specified tasks and supervision signals, limiting the scalability and diversity of agent training. We study an environment learning paradigm in which agents acquire interaction and manipulation capabilities solely through environment interaction, without externally specified tasks. We propose State2State, an environment-derived mid-training method that converts explored environment states into training objectives, challenging agents to reach a specified target state. By deriving tasks from environment exploration and verifying success through rule-based state matching, State2State provides scalable and verifiable training objectives without expert supervision or manual task design. Experiments across textual and mobile GUI environments demonstrate the effectiveness of State2State as a standalone environment-learning stage, with performance gains observed in both interaction modalities. As initialization for downstream RL, it further improves final performance and learning efficiency, retains a 3.5 to 6.6 point margin over RL given the same total number of update steps, and provides initial evidence of positive transfer across environments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Reinforcement Learning, Agentic Mid-Training, Environment Learning, Verifiable Reward

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hhvqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes State2State, a mid-training method for LLM agent. It collects reproducible environment states through random exploration or model self-exploration, treats reaching a goal state as the training task, performs RL with rule-based state matching as the reward, and then continues with RL on human-specified tasks. The authors conduct experiments on ALFWorld and ScienceWorld using Qwen3-4B and Qwen3-8B, report that the method as a preceding stage can improve the final performance of downstream RL, and report improvement when it is used alone on two mobile GUI benchmarks.

Critical strengths

The method is simple, the reward is obtained by rule matching, it does not require manually designed tasks or a teacher model, and the main experiments also set up an RL baseline aligned with the extra training steps.

Critical weaknesses
  1. Using states already reached as goals for a goal-conditioned policy is established in unsupervised goal-conditioned RL, including RIG, Skew-Fit, Go-Explore, and Asymmetric Self-Play. The related work covers this line only briefly with Kaelbling 1993 and HER, then turns to LLM relabeling work. Of the three claimed differences, task-free goal discovery and supervision from goal states are core practices of that work, and the only new part is applying this as mid-training for LLM agents. Without comparison, the contribution is overstated.

  2. Apart from the gain over compute-aligned RL in the main table, most claims rest on single runs and small gaps. Variance is reported only for the most favorable setting, and no variance is reported for the baselines. Standalone mid-training is weak. Training steps, exploration strategies, and base models differ across experiments, and the GUI comparison rows are selected from leaderboards. Under this evidence it is difficult to judge how much of these improvements goes beyond fluctuation and configuration choices.

Questions for authors
  1. Does State2State still bring improvement under exploration that does not use this kind of prior at all, given that exploration on ALFWorld is biased toward actions such as heat, cool, and clean, that these are exactly the types of the downstream tasks, and that the initial configurations on ScienceWorld are also taken from human-specified tasks?
  2. How much of the improvement comes from the goal of reaching states itself rather than from this difference in optimization, given that the compute-aligned baseline aligns only the number of update steps, that the State2State stage additionally uses dynamic sampling, and that neither the downstream RL nor the baseline uses dynamic sampling?
  3. Is State2State still effective on unseen apps, given that the goal states in MobileWorld are taken from the same set of apps used by the test benchmark and that this is closer to adaptation on the test environment?
4 comments
AuthorsOpenReview ↗

Response to Reviewer hhvq (1/2)

Thank you for recognizing the simplicity of our approach, its independence from teacher models, and the baseline matching the additional update steps. We clarify our contribution and address the experimental concerns below.

W1. Relationship to classical goal-conditioned RL

We agree that discovering goals through exploration and learning to reach them have established foundations in prior RL research. We will expand the discussion of the cited work and avoid presenting these principles as new algorithmic ideas.

Our central contribution is to study environment learning as a distinct mid-training stage for LLM agents, complementary to task-centric RL. This stage derives objectives and verification signals from environment interaction, enabling agents to acquire interaction capabilities without human-authored task instructions, expert demonstrations, or task-specific verifiers. Subsequent task-centric RL trains agents to apply these capabilities to user-specified goals.

State2State implements this paradigm and investigates its relationship with subsequent RL and SFT. We will clarify our contributions in the training paradigm, construction of supervision, and empirical findings while acknowledging their connection to established goal-conditioned RL.

W2. Experimental stability, standalone performance, and configuration differences

We agree that most settings lack complete multi-seed statistics, limiting conclusions about the significance of small differences. However, the existing repeated runs provide evidence of repeatability in one configuration.

On ALFWorld with Qwen3-4B, three independent runs of the full pipeline achieve 92.00%, 90.15%, and 90.54%, yielding a mean of 90.90% and a standard deviation of 0.98 percentage points. All three exceed the reported update-step-aligned baseline of 87.26% (Table 9), and Table 1 reports their mean. We acknowledge that this does not replace multi-seed evaluation of the baselines and will supplement the key comparisons accordingly.

Standalone gains are heterogeneous, including a decrease for Qwen3-4B on ScienceWorld. The value of mid-training must also be assessed through subsequent task learning. In Table 1, State2State + RL exceeds update-step-aligned RL across all four model–environment combinations by 3.50–6.60 percentage points. We will qualify the standalone claims and emphasize improved initialization for downstream RL.

GUI leaderboard entries serve as contextual references rather than controlled comparisons. In particular, the MobileGym baseline of 10.16% comes from our own evaluation, against which State2State’s 16.0% is measured (Appendix J.5). We will clarify configuration differences and avoid treating rankings across different setups as the primary evidence for our method.

AuthorsOpenReview ↗

Response to Reviewer hhvq (2/2)

Q1. Exploration priors and task-derived initial configurations

We acknowledge that action sampling and target selection incorporate environment-specific heuristics, and that ScienceWorld initial configurations are inherited from existing task environments. These choices can influence the distribution of discovered states.

Learning without human-task supervision does not imply the absence of environment knowledge or exploration priors. Our question is whether useful mid-training signals can be constructed from interaction without manually specifying an objective and success verifier for each training task.

For example, using a ScienceWorld task’s initial configuration does not entail using its natural-language objective or completion reward. State2State targets are still obtained from explored observations, and rewards are determined by observation matching. ALFWorld’s action preferences encourage state-changing interactions, but their contribution to performance has not been isolated.

We will distinguish the absence of human-task supervision from independence from task-distribution priors, and investigate ablations removing the relevant action-weighting and target-selection preferences.

Q2. Dynamic sampling and computational budget

We conducted an additional ablation on MobileGym without dynamic sampling.

MethodDynamic samplingSuccess rate
Qwen3-VL-4B base model—10.16%
State2State (reported)Enabled16.0%
State2State (new ablation)Disabled15.63% (40/256)

Without dynamic sampling, State2State retains a 5.47-percentage-point improvement over the base model and remains within approximately 0.4 percentage points of the full configuration. Thus, the main improvement on MobileGym persists without dynamic sampling.

Appendix D, Table 8 already includes rollouts collected and discarded by dynamic sampling. Across the four textual settings, mid-training uses 1.11–1.46× its nominal rollout budget. These figures quantify online sampling overhead; they do not establish matched GPU costs and exclude offline exploration.

We agree that update-step-aligned more accurately describes the current baseline. The MobileGym ablation does not replace controlled comparisons in textual environments, which we will supplement with matched sampling strategies and computational budgets in future work.

Q3. Adaptation within applications versus generalization to unseen applications

We agree that the current MobileWorld experiment primarily evaluates adaptation within the environment and does not establish generalization to unseen applications.

Nevertheless, this is an important use case for environment learning. An agent’s ability to explore target applications without human-authored training tasks and improve its subsequent task performance demonstrates practical value in its own right. Using the same applications for exploration and evaluation does not entail training on the evaluation task instructions or their completion rewards; State2State mid-training uses goals derived from explored states.

We will explicitly frame the MobileWorld result as adaptation without human-task annotations and treat performance on unseen applications as a separate generalization question for future evaluation.

Thank you for the response, but it concedes the overlap with unsupervised goal-conditioned RL and defers every control I asked for to future work. I therefore keep my score.

AuthorsOpenReview ↗

Response to Reviewer hhvq

Thank you for your follow-up. We recognize the importance of the requested controls. Limited computational resources during the rebuttal period prevented us from completing all of these comparisons. With the resources available, we prioritized a dynamic-sampling ablation on MobileGym and included the results in our response: 15.63% (40/256) without dynamic sampling, compared with 16.0% with it. This supports the benefit of State2State without dynamic sampling in this setting, although it does not replace the remaining controls.

Building on established unsupervised goal-conditioned RL principles, our contribution is to develop and empirically study environment-derived mid-training as a distinct stage for LLM agents, complementary to task-centric RL; we will articulate this positioning more clearly in the revision.

Overall, we really appreciate your careful review and constructive suggestions, which will help us strengthen the experimental evidence and improve the clarity of our work.

Reviewer_5mQvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STATE2STATE, an environment-derived mid-training stage for LLM agents. Reachable observations collected through environment exploration are filtered for reproducibility and paired with their source initial configurations to form state-reaching tasks, with a unified rule-based observation-matching function serving as a task-agnostic verifier. Agents are trained on these tasks with GRPO under dynamic sampling, and the stage is positioned before downstream human-task RL.

Critical strengths
  1. The "state reaching as an RL objective" framing is well motivated.

  2. Positioning the method as a mid-training stage lets the agent build task-agnostic environment familiarity before RL.

Critical weaknesses
  1. This method combines an exploration-collected goal pool, an outcome-only matching reward, and GRPO with dynamic sampling, all established in prior unsupervised goal-conditioned RL, making the contribution an engineering adaptation rather than a new algorithmic design.

  2. Each task pairs a target observation with its source initial configuration, so training is sampled along a shared trajectory and the paper didn't verifiy that capability learned under this coupling transfers to the independently-paired regime genuine goal-reaching requires.

  3. The terminal matching reward is simultaneously too strict and too loose, producing false negatives for semantically equivalent states while the continuous GUI similarity opens an exploitable reward surface with no invariance or OOD safeguard.

Questions for authors
  1. The paper does not state what algorithmic contribution STATE2STATE makes relative to unsupervised goal-conditioned RL.

  2. The GUI verifier disagreements are not characterized by direction, leaving it unclear how many stem from semantically equivalent states missed by the matching function versus observations judged similar without actually completing the target.

  3. Moreover, while observation-level matching is at least operational for GUI, it does not naturally extend to tool-calling or pure-text tasks where state equivalence has no reliable surface signal.

  4. The cross-environment transfer is equally consistent with the simpler explanation that any exploration-based pretraining supplies OOD token and action patterns that ease downstream optimization, which the paper's human-task-RL control does not isolate.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5mQv (1/2)

Thank you for your insightful comments. We clarify our contribution, task construction, and applicability below.

W1 / Q1. Contribution and positioning

We acknowledge the established foundations of goal-reaching and the optimization algorithms used in our framework. Our central contribution is to formulate and study environment learning as a distinct mid-training stage for LLM agents, complementary to task-centric RL.

This stage derives objectives and verification signals from environment interaction, enabling agents to acquire interaction capabilities without human-authored task instructions, expert demonstrations, or task-specific verifiers. Subsequent task-centric RL trains agents to apply these capabilities to user-specified goals. State2State implements this paradigm and investigates its relationship with subsequent RL and SFT across textual and GUI environments.

We will strengthen the discussion of prior unsupervised goal-conditioned RL while clarifying our contributions in the training paradigm, construction of supervision, and empirical findings.

W2. Pairing initial configurations with goals

Independent sampling of initial states and goals is not a prerequisite for valid goal-reaching tasks. Reachability depends on the initial configuration; arbitrary independent pairing can produce infeasible tasks. Pairing a target with its source configuration provides evidence that the target is reachable.

Initial configurations can also be general-purpose. For example, a phone’s initial home screen is a common starting point for many human tasks. Using such starting states does not require tailoring the initialization to each target.

The source trajectory supports goal discovery and reproducibility checks. During training, the policy learns through fresh interactions and is not required to reproduce that trajectory. We agree that evaluation on new feasible initial-state–goal combinations would strengthen the generalization evidence and will investigate this in future work. This additional evaluation is distinct from the validity of the current goal-reaching formulation.

W3 / Q2. Verifier reliability and error analysis

We acknowledge that observation matching can produce false positives and false negatives. Appendix C reports two human audits, each containing 100 cases randomly sampled from training rollouts. Two annotators independently labeled each case, with a third adjudicating disagreements.

Verifier settingSampleAgreement with human judgments / correlation with human ratingsReported sources of disagreement
Normalized exact matching in textual environments100 cases94% agreement with adjudicated human labelsSix disagreements, mainly involving observation ordering or minor wording differences
GUI matching on MobileWorld100 casesContinuous scores: Pearson/Spearman = 0.75/0.76; thresholded decisions: 82% accuracy, Cohen’s κ = 0.64Dynamically changing content or sub-states not distinguished by the UI hierarchy

The GUI verifier also normalizes XML by removing position- and instance-specific attributes and combines structural and screenshot similarity (Appendix J.3). It therefore includes explicit measures to reduce sensitivity to some incidental observation differences.

These audits support the verifier’s reliability on the sampled cases, but do not rule out reward exploitation or establish OOD robustness. We agree that the current aggregate statistics do not provide a directional FP/FN breakdown. We will supplement this analysis with error categories and representative failure cases, and clarify the limitations in the revision.

AuthorsOpenReview ↗

Response to Reviewer 5mQv (2/2)

Q3. Applicability to textual and tool-based environments

Applicability depends on whether an environment provides reproducible configurations and observations that support reliable state comparison, rather than on whether its interface uses images, text, or tool calls. Section 3.1 explicitly states this condition.

Our evaluation already covers household manipulation in ALFWorld, scientific experimentation in ScienceWorld, and mobile application interaction in MobileWorld and MobileGym. These settings span different environment mechanisms and both textual and visual modalities.

Tool-based environments that expose inspectable state and support reproducible initialization may also satisfy these conditions. Open-ended text generation or tools without reliable state feedback require additional representations or verification mechanisms. We will clarify this boundary; a textual or tool-based interface alone does not make the approach inapplicable.

Q4. Cross-environment transfer and alternative uses of exploration data

We agree that the current transfer experiments do not fully isolate the specific contribution of state-reaching objectives from the benefits of general environment experience. Familiarity with environment feedback and action patterns can itself be part of what environment learning provides.

However, directly applying SFT to the same exploration trajectories requires attention to their behavioral quality. Random exploration discovers reachable states but may contain substantial redundancy, detours, and irrelevant actions. Direct imitation treats these actions as supervision and therefore risks reinforcing inefficient behavior.

State2State uses explored outcomes to define goals and learns how to reach them through online interaction. This separates goal discovery from the requirement for high-quality demonstrations.

This design motivation does not replace controlled comparisons. We will investigate appropriately processed goal-conditioned SFT and world-modeling using the same exploration data, followed by downstream RL, to better characterize the sources of improvement.

Thank you for the clarification and additional results. My concern regarding the incremental nature of the contribution remains. I have updated my score to 2.

AuthorsOpenReview ↗

Response to Reviewer 5mQv

Thanks for your support and constructive advice.

Reviewer_9TweOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes State2State, a mid-training method for LLM agents that constructs training objectives from environment exploration. The method collects reachable observations, filters them for diversity and reproducibility, and trains an agent to reach a specified target observation from the corresponding initial configuration. Experiments on ALFWorld, ScienceWorld, MobileWorld, and MobileGym show promising results, particularly when State2State is followed by downstream RL.

Critical strengths
  1. The paper takcles an interesting problem. Constructing training tasks and their verifiers is a substantial part of developing agents for new environments. Using explored observations as both goal specifications and the basis for reward computation provides a practical way to reduce this effort.

  2. The proposed method is simple and applicable to different interaction modalities.

  3. The experiments demonstrate good results for the complete training pipeline. State2State followed by RL outperforms both standard RL and the longer-update RL baseline across the four textual model-environment settings in Table 1.

Critical weaknesses
  1. The main comparison does not isolate the benefit of the state-reaching objective. State2State uses dynamic sampling to discard rollout groups with identical rewards and collect additional informative groups. However, Appendix D explicitly states that downstream human-task RL does not use this mechanism. Thus, the comparison changes both the training objective and the selection of training data. This is consequential for sparse binary rewards, because groups in which all attempts fail or succeed provide no reward-based relative advantage. In addition, the "compute-aligned" baseline matches policy-update counts, while Appendix H.1 acknowledges that it does not account for the additional exploration or dynamic-sampling rollouts. Table 8 also shows that updates in the two stages have different GPU costs. It would be good to see human-task RL with the same dynamic sampling, together with a comparison under a matched computational budget.

  2. The conceptual novelty of the paper needs to be clarified against closer prior work. The paper acknowledges goal-conditioned RL and hindsight learning, but the distinctions do not address earlier work on unsupervised goal-reaching. An especially relevant example is Sukhbaatar et al., Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play, in ICLR 2018. In its resettable setting, an agent generates a reachable target through interaction, another agent learns to reach that target from the initial configuration, and this training improves subsequent task learning. As such, generating goals through exploration and using goal-reaching to prepare for downstream RL are established ideas. State2State’s adaptation to LLM agents may still be valuable, but the paper should identify more precisely what this adaptation contributes and which findings go beyond the existing principle.

  3. The experiments omit important alternative uses of the same exploration data. State2State already collects trajectories that successfully reach its target observations. A natural baseline would condition on those targets and perform supervised learning on the corresponding trajectories, followed by the same downstream RL. Another useful comparison would use the collected transitions for world-modeling mid-training. These controls would help establish whether online state-reaching RL is a particularly effective way to use the available experience. The Agent Early Experience baseline does not resolve this question: Appendix H.3 implements only its self-reflection variant, and Table 1 reports it without a subsequent RL stage. Since the main contribution of the paper concerns initialization for downstream RL, comparisons between complete alternative training pipelines would be more informative.

Questions for authors

See the weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9Twe

Thank you for your insightful suggestions. We clarify our central contribution and address your concerns below.

W1. Dynamic sampling and computational budget

Dynamic sampling increases the proportion of informative rollout groups. To examine whether State2State depends on this mechanism, we conducted an additional ablation on MobileGym.

MethodDynamic samplingSuccess rate
Qwen3-VL-4B base model—10.16%
State2State (reported)Enabled16.0%
State2State (new ablation)Disabled15.63% (40/256)

Without dynamic sampling, State2State retains a 5.47-percentage-point improvement over the base model, remaining within approximately 0.4 percentage points of the reported result. The main benefit on MobileGym therefore persists without dynamic sampling.

Regarding overhead, Appendix D, Table 8 already accounts for discarded rollouts. Across the four textual settings, mid-training uses 1.11–1.46× its nominal rollout budget on average. These figures quantify rollout overhead, rather than establish matched GPU costs.

State2State also provides an expandable pool of environment-derived goals, allowing training to discard currently uninformative groups and sample other goals without manually authoring additional tasks or task-specific verifiers. This makes dynamic sampling a natural fit for the framework.

We agree that update-step-aligned more accurately describes the current baseline. The MobileGym ablation does not replace controlled comparisons in textual environments; we will supplement these with matched sampling strategies and computational budgets in future work.

W2. Relation to goal-conditioned RL and our central contribution

We acknowledge that exploration-based goal discovery and goal reaching have established foundations in unsupervised goal-conditioned RL, including Asymmetric Self-Play. We will expand this discussion and clarify the relationship.

Our central insight is that environment learning can constitute a distinct mid-training stage for LLM agents, complementary to task-centric RL. In this stage, objectives and verification signals are derived from environment interaction, enabling agents to acquire interaction capabilities without human-authored task instructions, expert demonstrations, or task-specific verifiers. Subsequent task-centric RL trains agents to apply these capabilities to user-specified goals.

State2State operationalizes this paradigm and investigates its relationship with subsequent RL and SFT. Results across textual and GUI environments support its feasibility, while downstream improvements support the value of this distinct training stage. Our contribution therefore centers on the training paradigm, construction of supervision, and empirical findings, while building on established goal-conditioned RL principles.

W3. Alternative uses of exploration data

We agree that goal-conditioned SFT and world-modeling using the same exploration data are valuable controls.

However, reaching a target does not make every intermediate action a desirable demonstration. Random or autonomous exploration may contain redundant, circuitous, or irrelevant actions. Direct SFT without filtering for action quality would treat these actions as imitation targets, potentially reinforcing inefficient behavior and impairing execution efficiency.

State2State uses explored outcomes to construct verifiable goals, allowing online RL to learn alternative trajectories. This separates goal discovery from behavior learning: exploration trajectories can supply training objectives without also serving as high-quality demonstrations.

This motivates our design but does not establish superiority over appropriately processed goal-conditioned SFT. We also acknowledge that the current Agent Early Experience baseline does not replace these comparisons. In future work, we will evaluate goal-conditioned SFT and world-modeling on the same exploration data, each followed by downstream RL.

AuthorsOpenReview ↗

Gentle Reminder

We would like to express our gratitude for reviewing our paper. Your feedback has been valuable to us. We hope the submitted response addresses the concerns you raised. Since the discussion period is ending soon, we would appreciate it if you take the time to review our responses and share your thoughts.

Overall, thanks for your constructive suggestions which help to improve our work

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.