- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how the SFT objective used to initialize terminal agents affects subsequent RL. Standard agent SFT applies the loss only to agent-authored action tokens, while environment observations stay in the context but are masked from the loss. The authors propose ActObs, which unmasks these observation tokens and trains on both actions and observations with a weighted, normalized loss. The authors report that the three SFT variants perform similarly before RL but diverge after identical GRPO. At 4B, ActObs to GRPO is best at every sampling budget. At 8B, it trades pass@1 for higher pass@k. On aider-polyglot, ActObs to GRPO improves over ActionSFT to GRPO by 4.2 points pass@1 at 4B.
Critical strengths
- Simple, practical idea, which is trivial to adopt and try
- The paper is written clear and easy to follow. The experiments, in major, were conducted correctly. The overall narrative looks understandable.
- The methods were benchmarked across different scales (3b, 4b, 8b), which makes the result unbiased on the model checkpoint and scale. Different setups (obs to act, act to act, actobs-100, shuffle obs), in general, were conducted correctly, except the cases discussed in weaknesses.
- Report is honest, and the authors discuss the disadvantages, such as trade-off of performance for 8B model.
Critical weaknesses
- Key ideas, such as training RL after SFT with joint observation prediction, are partially addressed in related work. In particular, [1] explores this idea as word modeling, and in particular cover PPO training based on similar SFT approach. Authors should clarify the differences of their analysis and the one discussed in related work, and highlight the novelty. Moreover, similar approaches are broadly known in different fields, such as Visual Language Action modeling [2] (WorldVLA also regularizes observation reconstruction like L_a + L_w * alpha)
- According to the figure 5, the observation and action losses gradients are orthogonal. So, the optimization budget with ActObs pipeline is now shared with two orthogonal directions, making the effect similar to undertrained model. Table 4 partially reinforces the effect: direct SFT fine-tune with ActObs performs less than ActSFT for almost every K and every billions of parameters setup. Finally, the effects of ActObs may be explained as "less overfitting" on the actions rather than prediction of the observations.
- Shuffled obs breaks the context-action causation, which also may explain the downgrade performance in comparison to ActObs. Similar entropy of both policies means that they are "sharped" equally. Combining with the previously discussed weaknesses, and the hypothesis that main effect goes from the less overfitted policy rather than observation modeling, the experiment on shuffled observation doesn't isolate the effect.
- The paper says that "Our analyses link these gains to joint supervision preserving consequence prediction during SFT and supporting higher entropy with less policy movement during RL", whereas different ways to preserve entropy wasn't benchmarked. The related work mentions, for example, [3-5], but they wasn't benchmarked as non obs regularization methods.
[1] https://arxiv.org/pdf/2510.15047v1 [2] https://arxiv.org/abs/2506.21539 [3] https://papers.nips.cc/paper_files/paper/2025/hash/a4277440d50f1f15d2cb4c14f7e0c0d2-Abstract-Conference.html [4] https://arxiv.org/abs/2603.16206 [5] https://arxiv.org/abs/2512.16911
Questions for authors
- Can you provide some ablations on training RL from undertrained ActionSFT, in order to check whether the non-sharpness of checkpoint matters?
- How do you see ActObs relative to SPA? Can you highlight the own novelty paper brings?
- Can you rerun ActObs with separately normalized terms (actions by |A|, observations by |O|), so that the action objective has the same weight as in ActionSFT, followed by the same GRPO.
- Can you rerun ActionSFT trajectories with observations masked as usual, plus an LM loss on the same observation tokens presented as separate documents without their preceding actions (or on wild logs as separate documents). This keeps the action context intact and the amount of observation supervision equal, but removes the action–consequence relation.
- Can you benchmark several methods of preserving entropy in RL in comparison with ActObs (for example, on GRPO, one RL and non-RL based method)?
5 comments
Response to Reviewer pmM5 (2/2)
Q3: Separately normalized action and observation terms
We normalize each term by its own token count, , so the action loss has the same weight as in ActionSFT. With the observation term switched off, the loss and gradients are identical to those of ActionSFT.
After the same GRPO, this variant scores 10.9 / 18.1 / 22.1 / 25.8 at pass@1 / 4 / 8 / 16 and solves 23 tasks. It is within 1.6 points (1.0 standard error) of ActObs at every k. It also shows the same pattern against ActionSFT → GRPO, with differences of −1.4 / −0.6 / +1.1 / +2.2 points at pass@1 / 4 / 8 / 16. Its training entropy during GRPO follows that of ActObs (Appendix D, Figure 10b). The smaller weight of the action loss in Eq. 1 is therefore not what produces the gain.
W3 / Q4: Shuffled observations and unpaired observation text
We agree that shuffled observations alone do not isolate the effect, because shuffling also corrupts the context of later actions. We added the ablation proposed in Q4 and ran all three variants through GRPO:
- Observation documents (Q4). The trajectories keep the action-only loss, with observations masked as usual. Each trajectory's observations, with the actions removed, are added as a separate document with a language-modeling loss. The action context stays intact and the observation tokens are the same.
- Wild logs. The same construction with token- and format-matched production logs from Loghub.
- Shuffled observations, now also followed by GRPO.
After GRPO, observation documents score 11.2 / 18.5 / 21.0 / 23.6 and wild logs 10.7 / 18.1 / 20.9 / 23.6 at pass@1 / 4 / 8 / 16, against 11.0 / 19.3 / 23.6 / 27.0 for ActObs. Shuffled observations score 8.6 / 14.6 / 18.0 / 22.5. The same observation tokens, without the actions that produced them, therefore do not give the benefit. Together with the matched-fit result (W2 / Q1), neither a less sharpened policy nor additional observation text accounts for the gain. It appears only when each observation is predicted after the action that produced it.
W4 / Q5: Other ways to preserve entropy
We benchmarked one non-RL and one RL method, as suggested, both starting from action-only supervision:
- GEM (non-RL). GEM (Li et al., ICLR 2025) replaces the cross-entropy loss of SFT with an objective designed to preserve output diversity. We use the authors' implementation and default hyperparameters, followed by standard GRPO.
- Clip-higher (RL). This is the component of DAPO [3] that counteracts entropy collapse. We run GRPO from the ActionSFT checkpoint with the upper clipping bound raised from 0.2 to 0.28.
Both are below ActObs at every k: GEM scores 6.7 / 14.2 / 17.8 / 20.2 and clip-higher 9.4 / 17.5 / 21.1 / 24.7 at pass@1 / 4 / 8 / 16, against 11.0 / 19.3 / 23.6 / 27.0 for ActObs. The training entropy of clip-higher rises above plain GRPO in the middle of training but does not reach that of ActObs (Appendix D, Figure 10b).
These results agree with two analyses already in the paper. Raising the sampling temperature of ActionSFT → GRPO to match the evaluation entropy of ActObs does not close the gap, and ECHO reaches the highest entropy without the best pass@16. Preserving entropy alone does not reproduce ActObs.
We also cite [4] and [5] in the introduction and discuss them in Section 6. Both show that checkpoints with similar pre-RL performance can hand different policy distributions to RL, which is the premise of our study. Among non-RL methods of this kind, we benchmarked GEM.
Follow-up
Dear Reviewer pmM5,
Thank you again for the detailed review. Your concerns were concrete and testable, so during the rebuttal period we tested each of them with a dedicated experiment: seven new configurations at 8B, each followed by the same GRPO.
- W2 / Q1 (softer action fit): an early-stopped ActionSFT matched to the action fit of ActObs.
- Q3 (loss normalization): separately normalized ActObs.
- W3 / Q4 (unpaired observation text): observations as separate documents, wild logs, and shuffled observations.
- W4 / Q5 (entropy preservation): GEM and clip-higher.
The results are in Table 2 and Table 3, and our response above goes through them point by point, together with the comparison to SPA and WorldVLA (W1 / Q2).
As the discussion period ends on Nov 18, we would be grateful to know whether these results address your concerns, or whether anything is still missing that we can clarify in the remaining time. Looking forward to your response!
Response to Reviewer pmM5 (1/2)
We thank the reviewer for the detailed and concrete review. We ran all of the requested experiments, each at Qwen3-8B with the same GRPO recipe, and none of the alternative hypotheses accounts for the gain. The table collects the results on Terminal-Bench 2.0 after GRPO (pass@k in percent, 16 attempts per task), and the answers below refer to it. In the revised paper, these results are reported in Section 5.3 (Table 2) and Appendix B.2 (Table 3), and the corresponding training-entropy curves in Appendix D (Figure 10).
We thank the reviewer for these pointers. SPA's premise is that an agent should learn environment dynamics in a dedicated stage before optimizing reward. In our setting, that premise corresponds to our Obs→Act method (observations first, then actions), not to ActObs, and Obs→Act does not reproduce the RL-stage gain (Qwen3-4B after GRPO: 5.3 / 11.4 / 14.7 / 16.9 at pass@1 / 4 / 8 / 16, vs. 7.2 / 12.5 / 15.5 / 19.1 for ActObs). ActObs starts from a different perspective: consequence modeling learned jointly with action imitation, so the same SFT pass learns each expert action and the observation it produces. The two methods also differ in:
The same distinctions apply to other dynamics-first recipes (Early Experience, From Word to World, Qwen-AgentWorld), which also collect extra transitions or targets and train world modeling as a separate stage; ActObs needs neither. WorldVLA [2] jointly trains action and world-model prediction in vision-language-action models. We explore joint action and observation prediction in LLM agents, with a particular focus on how observation supervision during SFT influences subsequent RL.
We reran ActionSFT with a checkpoint saved every 25 steps and selected the checkpoint whose held-out action NLL is closest to that of ActObs. It selects step 425 of 781, with NLL 0.2788, against 0.2790 for ActObs and 0.2730 for the final ActionSFT checkpoint. After SFT, this matched-fit checkpoint scores 8.2 / 16.1 / 19.8 / 23.6 at pass@1 / 4 / 8 / 16, close to 7.9 / 16.1 / 19.7 / 22.5 for ActObs.
The same GRPO recipe does not turn it into ActObs. After GRPO, the matched-fit policy scores 9.8 / 17.2 / 20.6 / 24.7 at pass@1 / 4 / 8 / 16 and solves 22 tasks, against 11.0 / 19.3 / 23.6 / 27.0 and 24 tasks for ActObs. GRPO improves matched-fit checkpoint by 1.5 / 1.1 / 0.8 / 1.1 points, against 3.1 / 3.2 / 3.9 / 4.5 for ActObs. Two checkpoints with the same action fit and similar SFT pass@k respond differently to RL, so a softer action fit does not account for the gain.