- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes State2State, a mid-training method for LLM agent. It collects reproducible environment states through random exploration or model self-exploration, treats reaching a goal state as the training task, performs RL with rule-based state matching as the reward, and then continues with RL on human-specified tasks. The authors conduct experiments on ALFWorld and ScienceWorld using Qwen3-4B and Qwen3-8B, report that the method as a preceding stage can improve the final performance of downstream RL, and report improvement when it is used alone on two mobile GUI benchmarks.
Critical strengths
The method is simple, the reward is obtained by rule matching, it does not require manually designed tasks or a teacher model, and the main experiments also set up an RL baseline aligned with the extra training steps.
Critical weaknesses
-
Using states already reached as goals for a goal-conditioned policy is established in unsupervised goal-conditioned RL, including RIG, Skew-Fit, Go-Explore, and Asymmetric Self-Play. The related work covers this line only briefly with Kaelbling 1993 and HER, then turns to LLM relabeling work. Of the three claimed differences, task-free goal discovery and supervision from goal states are core practices of that work, and the only new part is applying this as mid-training for LLM agents. Without comparison, the contribution is overstated.
-
Apart from the gain over compute-aligned RL in the main table, most claims rest on single runs and small gaps. Variance is reported only for the most favorable setting, and no variance is reported for the baselines. Standalone mid-training is weak. Training steps, exploration strategies, and base models differ across experiments, and the GUI comparison rows are selected from leaderboards. Under this evidence it is difficult to judge how much of these improvements goes beyond fluctuation and configuration choices.
Questions for authors
- Does State2State still bring improvement under exploration that does not use this kind of prior at all, given that exploration on ALFWorld is biased toward actions such as heat, cool, and clean, that these are exactly the types of the downstream tasks, and that the initial configurations on ScienceWorld are also taken from human-specified tasks?
- How much of the improvement comes from the goal of reaching states itself rather than from this difference in optimization, given that the compute-aligned baseline aligns only the number of update steps, that the State2State stage additionally uses dynamic sampling, and that neither the downstream RL nor the baseline uses dynamic sampling?
- Is State2State still effective on unseen apps, given that the goal states in MobileWorld are taken from the same set of apps used by the test benchmark and that this is closer to adaptation on the test environment?
4 comments
Response to Reviewer hhvq (2/2)
Q1. Exploration priors and task-derived initial configurations
We acknowledge that action sampling and target selection incorporate environment-specific heuristics, and that ScienceWorld initial configurations are inherited from existing task environments. These choices can influence the distribution of discovered states.
Learning without human-task supervision does not imply the absence of environment knowledge or exploration priors. Our question is whether useful mid-training signals can be constructed from interaction without manually specifying an objective and success verifier for each training task.
For example, using a ScienceWorld task’s initial configuration does not entail using its natural-language objective or completion reward. State2State targets are still obtained from explored observations, and rewards are determined by observation matching. ALFWorld’s action preferences encourage state-changing interactions, but their contribution to performance has not been isolated.
We will distinguish the absence of human-task supervision from independence from task-distribution priors, and investigate ablations removing the relevant action-weighting and target-selection preferences.
Q2. Dynamic sampling and computational budget
We conducted an additional ablation on MobileGym without dynamic sampling.
| Method | Dynamic sampling | Success rate |
|---|---|---|
| Qwen3-VL-4B base model | — | 10.16% |
| State2State (reported) | Enabled | 16.0% |
| State2State (new ablation) | Disabled | 15.63% (40/256) |
Without dynamic sampling, State2State retains a 5.47-percentage-point improvement over the base model and remains within approximately 0.4 percentage points of the full configuration. Thus, the main improvement on MobileGym persists without dynamic sampling.
Appendix D, Table 8 already includes rollouts collected and discarded by dynamic sampling. Across the four textual settings, mid-training uses 1.11–1.46× its nominal rollout budget. These figures quantify online sampling overhead; they do not establish matched GPU costs and exclude offline exploration.
We agree that update-step-aligned more accurately describes the current baseline. The MobileGym ablation does not replace controlled comparisons in textual environments, which we will supplement with matched sampling strategies and computational budgets in future work.
Q3. Adaptation within applications versus generalization to unseen applications
We agree that the current MobileWorld experiment primarily evaluates adaptation within the environment and does not establish generalization to unseen applications.
Nevertheless, this is an important use case for environment learning. An agent’s ability to explore target applications without human-authored training tasks and improve its subsequent task performance demonstrates practical value in its own right. Using the same applications for exploration and evaluation does not entail training on the evaluation task instructions or their completion rewards; State2State mid-training uses goals derived from explored states.
We will explicitly frame the MobileWorld result as adaptation without human-task annotations and treat performance on unseen applications as a separate generalization question for future evaluation.
Thank you for the response, but it concedes the overlap with unsupervised goal-conditioned RL and defers every control I asked for to future work. I therefore keep my score.
Response to Reviewer hhvq
Thank you for your follow-up. We recognize the importance of the requested controls. Limited computational resources during the rebuttal period prevented us from completing all of these comparisons. With the resources available, we prioritized a dynamic-sampling ablation on MobileGym and included the results in our response: 15.63% (40/256) without dynamic sampling, compared with 16.0% with it. This supports the benefit of State2State without dynamic sampling in this setting, although it does not replace the remaining controls.
Building on established unsupervised goal-conditioned RL principles, our contribution is to develop and empirically study environment-derived mid-training as a distinct stage for LLM agents, complementary to task-centric RL; we will articulate this positioning more clearly in the revision.
Overall, we really appreciate your careful review and constructive suggestions, which will help us strengthen the experimental evidence and improve the clarity of our work.
Response to Reviewer hhvq (1/2)
Thank you for recognizing the simplicity of our approach, its independence from teacher models, and the baseline matching the additional update steps. We clarify our contribution and address the experimental concerns below.
W1. Relationship to classical goal-conditioned RL
We agree that discovering goals through exploration and learning to reach them have established foundations in prior RL research. We will expand the discussion of the cited work and avoid presenting these principles as new algorithmic ideas.
Our central contribution is to study environment learning as a distinct mid-training stage for LLM agents, complementary to task-centric RL. This stage derives objectives and verification signals from environment interaction, enabling agents to acquire interaction capabilities without human-authored task instructions, expert demonstrations, or task-specific verifiers. Subsequent task-centric RL trains agents to apply these capabilities to user-specified goals.
State2State implements this paradigm and investigates its relationship with subsequent RL and SFT. We will clarify our contributions in the training paradigm, construction of supervision, and empirical findings while acknowledging their connection to established goal-conditioned RL.
W2. Experimental stability, standalone performance, and configuration differences
We agree that most settings lack complete multi-seed statistics, limiting conclusions about the significance of small differences. However, the existing repeated runs provide evidence of repeatability in one configuration.
On ALFWorld with Qwen3-4B, three independent runs of the full pipeline achieve 92.00%, 90.15%, and 90.54%, yielding a mean of 90.90% and a standard deviation of 0.98 percentage points. All three exceed the reported update-step-aligned baseline of 87.26% (Table 9), and Table 1 reports their mean. We acknowledge that this does not replace multi-seed evaluation of the baselines and will supplement the key comparisons accordingly.
Standalone gains are heterogeneous, including a decrease for Qwen3-4B on ScienceWorld. The value of mid-training must also be assessed through subsequent task learning. In Table 1, State2State + RL exceeds update-step-aligned RL across all four model–environment combinations by 3.50–6.60 percentage points. We will qualify the standalone claims and emphasize improved initialization for downstream RL.
GUI leaderboard entries serve as contextual references rather than controlled comparisons. In particular, the MobileGym baseline of 10.16% comes from our own evaluation, against which State2State’s 16.0% is measured (Appendix J.5). We will clarify configuration differences and avoid treating rankings across different setups as the primary evidence for our method.