acceptodds
Under review as a conference paper at ICLR 2027

Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

Abstract

Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, language models, supervised fine-tuning, reinforcement learning, observation supervision, world modeling, exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pmM5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how the SFT objective used to initialize terminal agents affects subsequent RL. Standard agent SFT applies the loss only to agent-authored action tokens, while environment observations stay in the context but are masked from the loss. The authors propose ActObs, which unmasks these observation tokens and trains on both actions and observations with a weighted, normalized loss. The authors report that the three SFT variants perform similarly before RL but diverge after identical GRPO. At 4B, ActObs to GRPO is best at every sampling budget. At 8B, it trades pass@1 for higher pass@k. On aider-polyglot, ActObs to GRPO improves over ActionSFT to GRPO by 4.2 points pass@1 at 4B.

Critical strengths
  1. Simple, practical idea, which is trivial to adopt and try
  2. The paper is written clear and easy to follow. The experiments, in major, were conducted correctly. The overall narrative looks understandable.
  3. The methods were benchmarked across different scales (3b, 4b, 8b), which makes the result unbiased on the model checkpoint and scale. Different setups (obs to act, act to act, actobs-100, shuffle obs), in general, were conducted correctly, except the cases discussed in weaknesses.
  4. Report is honest, and the authors discuss the disadvantages, such as trade-off of performance for 8B model.
Critical weaknesses
  1. Key ideas, such as training RL after SFT with joint observation prediction, are partially addressed in related work. In particular, [1] explores this idea as word modeling, and in particular cover PPO training based on similar SFT approach. Authors should clarify the differences of their analysis and the one discussed in related work, and highlight the novelty. Moreover, similar approaches are broadly known in different fields, such as Visual Language Action modeling [2] (WorldVLA also regularizes observation reconstruction like L_a + L_w * alpha)
  2. According to the figure 5, the observation and action losses gradients are orthogonal. So, the optimization budget with ActObs pipeline is now shared with two orthogonal directions, making the effect similar to undertrained model. Table 4 partially reinforces the effect: direct SFT fine-tune with ActObs performs less than ActSFT for almost every K and every billions of parameters setup. Finally, the effects of ActObs may be explained as "less overfitting" on the actions rather than prediction of the observations.
  3. Shuffled obs breaks the context-action causation, which also may explain the downgrade performance in comparison to ActObs. Similar entropy of both policies means that they are "sharped" equally. Combining with the previously discussed weaknesses, and the hypothesis that main effect goes from the less overfitted policy rather than observation modeling, the experiment on shuffled observation doesn't isolate the effect.
  4. The paper says that "Our analyses link these gains to joint supervision preserving consequence prediction during SFT and supporting higher entropy with less policy movement during RL", whereas different ways to preserve entropy wasn't benchmarked. The related work mentions, for example, [3-5], but they wasn't benchmarked as non obs regularization methods.

[1] https://arxiv.org/pdf/2510.15047v1 [2] https://arxiv.org/abs/2506.21539 [3] https://papers.nips.cc/paper_files/paper/2025/hash/a4277440d50f1f15d2cb4c14f7e0c0d2-Abstract-Conference.html [4] https://arxiv.org/abs/2603.16206 [5] https://arxiv.org/abs/2512.16911

Questions for authors
  1. Can you provide some ablations on training RL from undertrained ActionSFT, in order to check whether the non-sharpness of checkpoint matters?
  2. How do you see ActObs relative to SPA? Can you highlight the own novelty paper brings?
  3. Can you rerun ActObs with separately normalized terms (actions by |A|, observations by |O|), so that the action objective has the same weight as in ActionSFT, followed by the same GRPO.
  4. Can you rerun ActionSFT trajectories with observations masked as usual, plus an LM loss on the same observation tokens presented as separate documents without their preceding actions (or on wild logs as separate documents). This keeps the action context intact and the amount of observation supervision equal, but removes the action–consequence relation.
  5. Can you benchmark several methods of preserving entropy in RL in comparison with ActObs (for example, on GRPO, one RL and non-RL based method)?
5 comments
AuthorsOpenReview ↗

Response to Reviewer pmM5 (1/2)

We thank the reviewer for the detailed and concrete review. We ran all of the requested experiments, each at Qwen3-8B with the same GRPO recipe, and none of the alternative hypotheses accounts for the gain. The table collects the results on Terminal-Bench 2.0 after GRPO (pass@k in percent, 16 attempts per task), and the answers below refer to it. In the revised paper, these results are reported in Section 5.3 (Table 2) and Appendix B.2 (Table 3), and the corresponding training-entropy curves in Appendix D (Figure 10).

Configurationpass@1pass@4pass@8pass@16solved/89
ActObs (reference)11.019.323.627.024
ActionSFT (reference)12.318.721.023.621
Matched-fit ActionSFT9.817.220.624.722
Separately normalized ActObs10.918.122.125.823
Observation documents11.218.521.023.621
Wild logs10.718.120.923.621
Shuffled observations8.614.618.022.520
GEM ActionSFT6.714.217.820.218
ActionSFT with clip-higher9.417.521.124.722

W1 / Q2: Novelty relative to SPA [1] and WorldVLA [2]

We thank the reviewer for these pointers. SPA's premise is that an agent should learn environment dynamics in a dedicated stage before optimizing reward. In our setting, that premise corresponds to our Obs→Act method (observations first, then actions), not to ActObs, and Obs→Act does not reproduce the RL-stage gain (Qwen3-4B after GRPO: 5.3 / 11.4 / 14.7 / 16.9 at pass@1 / 4 / 8 / 16, vs. 7.2 / 12.5 / 15.5 / 19.1 for ActObs). ActObs starts from a different perspective: consequence modeling learned jointly with action imitation, so the same SFT pass learns each expert action and the observation it produces. The two methods also differ in:

  • Purpose of SFT. SPA's SFT trains a world model as a prior for the later PPO stage; its action targets are the base model's own exploratory actions, predicted after the ground-truth next state, so they serve as an inverse-dynamics target rather than expert behavior. ActObs learns expert actions and their consequences together in one SFT stage.
  • SFT data. SPA trains on self-play rollouts of the base model. ActObs uses existing expert trajectories from a strong teacher.
  • State supervision. SPA constructs explicit current and next state descriptions (e.g., object coordinates) from the simulator's ground-truth state. ActObs uses the raw environment observations already in the trace, with no annotation. Real terminals expose no such structured state.
  • Trajectory format and extra data. SPA rewrites the trajectory format and collects new transitions. ActObs changes only the loss mask.
  • Inference. SPA's policy explicitly predicts states at test time. An ActObs policy outputs only actions, exactly like standard SFT.
  • Setting. SPA studies 0.5B–3B models on small grid puzzles (Sokoban, FrozenLake, Sudoku) with PPO. ActObs studies 4B and 8B models on real terminal tasks (Terminal-Bench 2.0, aider-polyglot) with GRPO.

The same distinctions apply to other dynamics-first recipes (Early Experience, From Word to World, Qwen-AgentWorld), which also collect extra transitions or targets and train world modeling as a separate stage; ActObs needs neither. WorldVLA [2] jointly trains action and world-model prediction in vision-language-action models. We explore joint action and observation prediction in LLM agents, with a particular focus on how observation supervision during SFT influences subsequent RL.

W2 / Q1: Is the gain "less overfitting" of an undertrained action policy?

We reran ActionSFT with a checkpoint saved every 25 steps and selected the checkpoint whose held-out action NLL is closest to that of ActObs. It selects step 425 of 781, with NLL 0.2788, against 0.2790 for ActObs and 0.2730 for the final ActionSFT checkpoint. After SFT, this matched-fit checkpoint scores 8.2 / 16.1 / 19.8 / 23.6 at pass@1 / 4 / 8 / 16, close to 7.9 / 16.1 / 19.7 / 22.5 for ActObs.

The same GRPO recipe does not turn it into ActObs. After GRPO, the matched-fit policy scores 9.8 / 17.2 / 20.6 / 24.7 at pass@1 / 4 / 8 / 16 and solves 22 tasks, against 11.0 / 19.3 / 23.6 / 27.0 and 24 tasks for ActObs. GRPO improves matched-fit checkpoint by 1.5 / 1.1 / 0.8 / 1.1 points, against 3.1 / 3.2 / 3.9 / 4.5 for ActObs. Two checkpoints with the same action fit and similar SFT pass@k respond differently to RL, so a softer action fit does not account for the gain.

AuthorsOpenReview ↗

Response to Reviewer pmM5 (2/2)

Q3: Separately normalized action and observation terms

We normalize each term by its own token count, Lact∣A∣+Lobs∣O∣\frac{L_{act}}{|A|} + \frac{L_{obs}}{|O|}, so the action loss has the same weight as in ActionSFT. With the observation term switched off, the loss and gradients are identical to those of ActionSFT.

After the same GRPO, this variant scores 10.9 / 18.1 / 22.1 / 25.8 at pass@1 / 4 / 8 / 16 and solves 23 tasks. It is within 1.6 points (1.0 standard error) of ActObs at every k. It also shows the same pattern against ActionSFT → GRPO, with differences of −1.4 / −0.6 / +1.1 / +2.2 points at pass@1 / 4 / 8 / 16. Its training entropy during GRPO follows that of ActObs (Appendix D, Figure 10b). The smaller weight of the action loss in Eq. 1 is therefore not what produces the gain.

W3 / Q4: Shuffled observations and unpaired observation text

We agree that shuffled observations alone do not isolate the effect, because shuffling also corrupts the context of later actions. We added the ablation proposed in Q4 and ran all three variants through GRPO:

  • Observation documents (Q4). The trajectories keep the action-only loss, with observations masked as usual. Each trajectory's observations, with the actions removed, are added as a separate document with a language-modeling loss. The action context stays intact and the observation tokens are the same.
  • Wild logs. The same construction with token- and format-matched production logs from Loghub.
  • Shuffled observations, now also followed by GRPO.

After GRPO, observation documents score 11.2 / 18.5 / 21.0 / 23.6 and wild logs 10.7 / 18.1 / 20.9 / 23.6 at pass@1 / 4 / 8 / 16, against 11.0 / 19.3 / 23.6 / 27.0 for ActObs. Shuffled observations score 8.6 / 14.6 / 18.0 / 22.5. The same observation tokens, without the actions that produced them, therefore do not give the benefit. Together with the matched-fit result (W2 / Q1), neither a less sharpened policy nor additional observation text accounts for the gain. It appears only when each observation is predicted after the action that produced it.

W4 / Q5: Other ways to preserve entropy

We benchmarked one non-RL and one RL method, as suggested, both starting from action-only supervision:

  • GEM (non-RL). GEM (Li et al., ICLR 2025) replaces the cross-entropy loss of SFT with an objective designed to preserve output diversity. We use the authors' implementation and default hyperparameters, followed by standard GRPO.
  • Clip-higher (RL). This is the component of DAPO [3] that counteracts entropy collapse. We run GRPO from the ActionSFT checkpoint with the upper clipping bound raised from 0.2 to 0.28.

Both are below ActObs at every k: GEM scores 6.7 / 14.2 / 17.8 / 20.2 and clip-higher 9.4 / 17.5 / 21.1 / 24.7 at pass@1 / 4 / 8 / 16, against 11.0 / 19.3 / 23.6 / 27.0 for ActObs. The training entropy of clip-higher rises above plain GRPO in the middle of training but does not reach that of ActObs (Appendix D, Figure 10b).

These results agree with two analyses already in the paper. Raising the sampling temperature of ActionSFT → GRPO to match the evaluation entropy of ActObs does not close the gap, and ECHO reaches the highest entropy without the best pass@16. Preserving entropy alone does not reproduce ActObs.

We also cite [4] and [5] in the introduction and discuss them in Section 6. Both show that checkpoints with similar pre-RL performance can hand different policy distributions to RL, which is the premise of our study. Among non-RL methods of this kind, we benchmarked GEM.

Dear Reviewer pmM5,

Thank you again for the detailed review. Your concerns were concrete and testable, so during the rebuttal period we tested each of them with a dedicated experiment: seven new configurations at 8B, each followed by the same GRPO.

  • W2 / Q1 (softer action fit): an early-stopped ActionSFT matched to the action fit of ActObs.
  • Q3 (loss normalization): separately normalized ActObs.
  • W3 / Q4 (unpaired observation text): observations as separate documents, wild logs, and shuffled observations.
  • W4 / Q5 (entropy preservation): GEM and clip-higher.

The results are in Table 2 and Table 3, and our response above goes through them point by point, together with the comparison to SPA and WorldVLA (W1 / Q2).

As the discussion period ends on Nov 18, we would be grateful to know whether these results address your concerns, or whether anything is still missing that we can clarify in the remaining time. Looking forward to your response!

Reviewer_VXgfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a simple extension to the SFT stage learning of agentic LLM models, which is to add supervision on the environment/observation prediction tokens in addition to the action prediction. While not directly predicted by the model itself, they argue flowing these gradients back to the model will allow the model be aware of the observations it's predicted actions lead to, which naturally reinforces better learning during downstream RL stage. They also observe improved emtropy conditioning during RL stage, in addition to having stronger performance at pass@K for higher K as observation supervision grows stronger.

The experiments have focused on terminal coding tasks (TerminalBench and Aider-polyglot as the OOD domain) - both of which show improved performances over the baseline.

Critical strengths
  1. The paper tackles a very focused observation of adding observation supervision, and provides an in-depth analysis of this aspect overlooked in prior works. While observations supervision is explored in RL stages in works like ECHO (extensively cited and compared in current paper), the new insights and analysis provided for SFT stage are unique and interesting.

  2. Adopting this method in current training workflows is trivial, and straight forward, with no additional parameters or training needed.

  3. The performance at higher pass@K is strong, but precision is sacrificed due to higher entropy variance during training.

Critical weaknesses
  1. The experiments are too narrow, and focus only on terminal coding tasks. It is not clear how this model performs on a broader range of tasks like agentic tool-use or general RL tasks. Furthermore, the gains obtained after the RL stage are quite minimal, with baseline approach of ActionSFT + ECHO matching or closely trailing behind proposed approach on a wide-range of tasks. Combined with the trade-off with precision, the actual use cases of this method needs to be highlighted further.

  2. The authors claim equivalent or on-par performance of all methods after SFT stage, which is in contrast to the argument of improved entropy during RL stage. Both of these can't happen together, and higher entropy always comes with trade-off in post-SFT pass@1 accuracy.

  3. In a general setting of tool-calling agents, I wonder if following this policy and training on tool-call outputs delivered by the environment in addition to the tool call requests made by the model is a good strategy. The authors should briefly comment on the generalization ability of this method.

Questions for authors
  1. The authors should elaborate more on the generalization of the setting, although experiments on wider domains is out of scope for rebuttal.

  2. I am not sure how is staying close to init policy a necessary indication of better or convergent learning. Higher entropy will lead to diverse rollouts and stable advantage during GRPO, but the overall connection made between the entropy, gradients between observation<>action and KL div between init policy and final checkpoint is unclear. More importantly, the authors needs to clearly seperate which of these is correlation vs actually causing the improved learning.

  3. L159: "....uncertainties are one bootstrap standard error, computed by holding the task set fixed and resampling attempts within each task...." - what does this exactly mean? Also, how is this different from std@k through pass@K - and is the strategy different between pass@1 and pass@k?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VXgf (1/2)

We thank the reviewer for the careful and constructive review, and we address each point below.

W1: Scope, size of the gain, and use cases

Scope. We evaluate two task families: terminal tasks (Terminal-Bench 2.0) and cross-domain code editing (aider-polyglot), whose tasks are unseen during SFT and RL. We agree that this does not cover general tool use, and we state it as a limitation (Appendix I). Our answer on tool-calling agents is under W3 / Q1.

Size of the gain. The gain replicates across training seeds, each of which repeats SFT and GRPO. Averaged over three seeds, ActObs → GRPO minus ActionSFT → GRPO is +1.3 / +1.4 / +1.5 / +1.5 points at pass@1 / 4 / 8 / 16 at 4B and −0.3 / +1.2 / +2.4 / +2.2 points at 8B. On aider-polyglot at 4B, ActObs → GRPO scores 13.9 / 25.3 at pass@1 / 4, against 9.7 / 20.4 for ActionSFT → GRPO.

ActionSFT + ECHO. ECHO changes the RL stage and ActObs changes only SFT, so the two can be combined. Read as competing methods, ActObs → GRPO is above ActionSFT → ECHO at every k on Terminal-Bench 2.0 at both scales:

Terminal-Bench 2.0pass@1pass@4pass@8pass@16
4B ActObs → GRPO7.212.515.519.1
4B ActionSFT → ECHO5.911.614.718.0
8B ActObs → GRPO11.019.323.627.0
8B ActionSFT → ECHO9.616.520.625.8

On aider-polyglot, ActObs → GRPO is higher at 4B (13.9 / 25.3 vs. 10.2 / 20.9 at pass@1 / 4), and the two are within one standard error at 8B (14.2 / 28.9 vs. 14.8 / 29.8). When combined, ActObs → ECHO leads or ties ActionSFT → ECHO at seven of the eight Terminal-Bench 2.0 operating points.

Use cases. We added guidance to Section 4.4: ActObs is most useful when several attempts can be checked by tests or verifiers, as in coding and terminal agents, whereas a smaller observation weight λ suits single-attempt use. The pass@1 trade-off is also smaller than the original 8B run suggests. At 4B, ActObs is higher at pass@1 in all three seeds. At 8B, the difference averages −0.3 / +1.2 / +2.4 / +2.2 points at pass@1 / 4 / 8 / 16 over three seeds (Table 8).

W2: Similar SFT performance vs. higher entropy

The two statements refer to different stages. We now define the two entropy measurements in the introduction to make this explicit.

  • After SFT, the checkpoints are close but not identical, and the difference has the sign the reviewer expects. ActObs has slightly higher evaluation entropy (0.706 vs. 0.683) and slightly lower scores.
  • During and after RL, the difference grows. Training entropy separates during GRPO (Figures 3, 8, and 9), and the final policies differ more (evaluation entropy 0.779 vs. 0.716). Our entropy claim concerns this stage.

Our point is that a small difference at the SFT checkpoint becomes a larger difference under the same RL procedure.

W3 / Q1: Tool-calling agents and generality

We added this discussion to Appendix I (Domain coverage). ActObs is defined for any trajectory that interleaves model outputs with environment responses. For a tool-calling agent, tool results such as API responses, retrieved documents, or code-execution output take the place of terminal output, and only their loss mask changes.

We expect the benefit to depend on how strongly an observation is determined by the action that precedes it. Terminal output is largely a consequence of the command just issued, which makes it an informative prediction target. Tool results dominated by external content that the call does not determine, such as open-web search results, carry less of this signal, and the weight λ offers a direct way to down-weight them. Our new ablations support this: observation text that is not paired with the action that produced it (separate documents, wild logs, shuffled observations) gives no benefit after GRPO.

AuthorsOpenReview ↗

Response to Reviewer VXgf (2/2)

Q2: Policy movement, entropy, and gradients

The gradient analysis explains how the two SFT checkpoints come to differ: the action and observation gradients become orthogonal within 10 to 20 SFT steps, so ActionSFT stops improving observation prediction and falls below the base model at predicting terminal feedback, while ActObs preserves it. We read the other three quantities as one chain during RL:

  1. Policy movement. The reviewer is right that staying close to the initial policy is not by itself a sign of better or more converged learning; at 8B, the pass@1 of ActObs is in fact slightly lower. What the smaller KL shows is that GRPO moved the policy less, so it keeps more of the alternatives that the SFT policy could produce, such as different commands and arguments, and leaves more room for exploration.
  2. Entropy. This shows up as higher entropy. ActObs keeps higher training entropy during GRPO and higher evaluation entropy at the end (0.779 vs. 0.716), mainly on command arguments, flags, and paths.
  3. pass@k. With more alternatives still carrying probability, repeated attempts differ more, so pass@k at larger k is higher and more distinct tasks are solved.

The cause in this chain is observation supervision. Smaller policy movement, higher entropy and pass@k are its consequences along the way.

Q3: What the bootstrap standard error means

We rewrote this description in Section 3.1. Each checkpoint has n = 16 recorded attempts on each of the 89 tasks. We draw 20,000 replicates. Each replicate keeps the same 89 tasks, resamples the 16 attempts of every task with replacement, and recomputes pass@k. The reported ± is the standard deviation of pass@k across replicates, that is, the standard error of the estimate.

The procedure is identical for pass@1 and pass@k. Only the estimator inside it changes with k; for pass@1 it is the mean success rate. It measures how much the reported number would change if the same checkpoint were evaluated again with new attempts. It does not include variation between training runs, which we now report separately with three training seeds (Appendix C).

Dear Reviewer VXgf,

Thank you again for the constructive review. In brief, our response above covers:

  • W1 (scope and size of the gain): results over three training seeds (Table 8) and the comparison with ActionSFT → ECHO.
  • W2 (similar SFT performance vs. higher entropy): the two statements refer to different stages, which the introduction now makes explicit.
  • W3 / Q1 (tool-calling agents): a new discussion in Appendix I.
  • Q2 and Q3: how policy movement, entropy, and pass@k relate, and what the bootstrap standard error measures.

We would be glad to hear whether this addresses your questions, and we are happy to clarify anything further before the discussion period ends on Nov 18. Looking forward to your response!

Reviewer_494YOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents ActObs, a supervised fine-tuning (SFT) method for LLM agents that extends the SFT loss to also include observation tokens, which are typically masked out during SFT. The stated motivation is that the observations, i.e. the environment’s responses to the agent’s actions, provide a “free” additional supervision signal that standard SFT leaves unexploited. Empirically, ActObs and standard action-only SFT perform similarly after SFT, but ActObs provides a better initialization for subsequent GRPO, reflected in improved pass@k on Terminal-Bench 2.0 and aider-polyglot. The authors attribute this to two effects: during SFT, joint supervision expectedly preserves the model’s ability to predict observations, which action-only SFT degrades below the base model. During GRPO, the ActObs initialization retains more entropy while moving less from its SFT starting point.

Disclaimer: I have strong familiarity with standard RL but not RL for LLM, so I am feeling somewhat unfit for reviewing this work. I have notified the AC about this, but am providing my review regardless.

Critical strengths

To the best of my knowledge, performing SFT followed by RL is the standard approach to training language agents. The papers thus investigates a very relevant point, and reveals that better SFT scores don't necessarily mean better initialization for subsequent RL. I believe this to be a useful message to the field.

Moreover, the proposed method is cheap and simple, requiring no extra data, parameters, or forward passes and boils down to unmasking the observation tokens, and setting the weight λ\lambda on observation term.

Since the method solely modifies the SFT objective, it is compatible with various downstream RL methods, for example ECHO, as reported in the paper.

Critical weaknesses

The main result (table 1) doesn't report training/seed variance, the errors are only over repeated sampling from the same model.

I don't see value added by the gradient alignment analysis. Standard action-only SFT doesn't incorporate the observation gradient, so the fact that the norm ratio between the action and observation gradients respectively collapses / grows is entirely expected.

There are few baselines. The comparison is essentially ActObs vs. standard action-only SFT, plus the authors own sequential Obs$\to$Act variant. ECHO is applied on top of both initializations rather than as a competing method. There is no comparison to other ways of regularizing SFT to preserve the base model or the policy’s diversity for subsequent RL.

The first pass of the paper was frustrating, particularly the introduction, which is unclear and fails to properly introduce the method and the overall problem. Concepts that are only defined later in the paper (training entropy, self-entropy, Obs →\to Act) are referenced but not explained, key citations are missing, and detailed results are presented which can't be interpreted without having read the whole paper.

Questions for authors

Under ActObs, the model needs learn to predict the action consequences. This occupies plasticity from the network. Couldn't this hurt downstream performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 494Y (1/2)

We thank the reviewer for the thoughtful review and the helpful suggestions, and we address each point below.

W1: Variance across training runs

We added two seeds at each scale, three in total (Appendix C, Table 8). Each seed repeats the full pipeline, SFT followed by GRPO, with a different random seed. Seed 1 is the original run.

ActObs minus ActionSFT after GRPO, in pointspass@1pass@4pass@8pass@16tasks solved
4B, seed 1 / 2 / 3+1.5 / +0.4 / +2.1+1.4 / +0.9 / +1.9+1.4 / +1.4 / +1.5+1.1 / +2.2 / +1.1+1 / +2 / +1
4B, mean ± s.d. over seeds+1.3 ± 0.9+1.4 ± 0.5+1.5 ± 0.0+1.5 ± 0.6
8B, seed 1 / 2 / 3−1.3 / +0.4 / −0.1+0.7 / +1.8 / +1.2+2.7 / +2.5 / +2.2+3.4 / +2.2 / +1.1+3 / +2 / +1
8B, mean ± s.d. over seeds−0.3 ± 0.9+1.2 ± 0.6+2.4 ± 0.2+2.2 ± 1.1

At 4B, ActObs is ahead at every k in every seed. At 8B, it is ahead at pass@4, pass@8, and pass@16 in every seed and solves more tasks in every pair of runs. At pass@1 the sign varies between seeds and the mean is near zero.

W2: Value of the gradient alignment analysis

We agree that the growth of the norm ratio under action-only SFT is expected. We removed the norm-ratio panel from Figure 5. Section 5.1 now makes a narrower claim, based on the cosine similarity between the action and observation gradients. At the pretrained checkpoint the two gradients are aligned (cosine 0.83), so action-only SFT could have improved observation prediction as a by-product. The measurement shows that this stops within 10 to 20 of 781 steps. After that, an action-only step no longer reduces the observation loss. This is consistent with the result that ActionSFT ends below the base model at predicting terminal feedback, and it shows why observation prediction has to be optimized explicitly and is not inherited from action training.

W3: Baselines

We added seven comparisons at 8B, each followed by the same GRPO (Section 5.3, Table 2; Appendix B.2, Table 3). Two are methods designed to preserve diversity or entropy for RL, and one keeps the action policy closer to the base model by stopping ActionSFT early.

Qwen3-8B, after GRPOpass@1pass@4pass@8pass@16solved/89
ActObs11.019.323.627.024
ActionSFT12.318.721.023.621
GEM ActionSFT (diversity-preserving SFT loss)6.714.217.820.218
ActionSFT with clip-higher (from DAPO)9.417.521.124.722
Matched-fit ActionSFT (early-stopped to the action fit of ActObs)9.817.220.624.722
Observation documents (observations as separate text)11.218.521.023.621
Separately normalized ActObs10.918.122.125.823

GEM, clip-higher, and early stopping are all below ActObs at every k. Training on the same observations as separate text, without the actions that produced them, does not help either. Wild logs and shuffled observations are also run through GRPO and are reported in Table 3.

For ECHO, ActObs → GRPO is above ActionSFT → ECHO at every k on Terminal-Bench 2.0 at both scales, for example 11.0 / 19.3 / 23.6 / 27.0 vs. 9.6 / 16.5 / 20.6 / 25.8 at pass@1 / 4 / 8 / 16 at 8B. ActObs is also complementary to ECHO rather than a replacement for it. When RL uses ECHO, starting from ActObs instead of ActionSFT gives 5.9 / 12.7 / 16.5 / 20.2 vs. 5.9 / 11.6 / 14.7 / 18.0 at 4B and 11.4 / 18.6 / 22.0 / 24.7 vs. 9.6 / 16.5 / 20.6 / 25.8 at 8B, so ActObs leads or ties at seven of the eight operating points.

AuthorsOpenReview ↗

Response to Reviewer 494Y (2/2)

W4: Presentation of the introduction

We revised the introduction (changes in blue).

  • A new second paragraph states the problem before the method: the two-stage pipeline of SFT followed by RL, and why an SFT objective should be judged by the policy obtained after RL. It adds the citations that were missing.
  • Obs→Act is described at its first mention in the introduction, as a timing control that applies observation-only SFT before action-only SFT. Section 3.1 gives the full definition.
  • The models, the common GRPO procedure, and pass@k (the probability that at least one of k attempts solves a task) are stated before any number.
  • Training entropy and evaluation entropy are defined before they are used. We renamed "self-entropy" to "evaluation entropy" throughout the paper.

Q1: Does predicting observations use up capacity and hurt downstream performance?

It can, and we see a small cost at the SFT stage. This cost does not carry over to RL.

  • ActObs fits the teacher's actions slightly less tightly (held-out action NLL 0.279 vs. 0.273) and has lower SFT scores at 8B (7.9 / 16.1 / 19.7 / 22.5 vs. 9.2 / 18.2 / 21.9 / 24.7 at pass@1 / 4 / 8 / 16).
  • There is no cost in what RL adds. GRPO improves the ActObs checkpoint at least as much as the ActionSFT checkpoint at every k at both scales. At 8B, the gain is +3.1 / +3.2 / +3.9 / +4.5 points for ActObs and +3.1 / +0.5 / −1.0 / −1.1 for ActionSFT at pass@1 / 4 / 8 / 16. At 4B, it is +2.7 / +2.6 / +1.5 / +1.1 for ActObs and +1.1 / +0.8 / +0.1 / +1.1 for ActionSFT.
  • The capacity is not wasted. An ActionSFT checkpoint stopped early at the same action fit as ActObs gains less from GRPO. ActionSFT also loses the base model's ability to predict terminal feedback, which ActObs keeps.
  • The trade-off is adjustable. The observation weight λ sets it and a smaller λ can keep more single-attempt accuracy.

Thank you for your revision of the manuscript.

Table 8 with cross-seed variance strengthens the credibility of the main result and the additional baselines underline the performance gain from ActObs.

Unfortunately, I still feel like the overall writing of the manuscript lacks clarity, and I am still confused about section 5.1.

Taken together, I have decided to raise my score from clear to weak reject, but I want to highlight again that I somewhat outside of my area of expertise with this paper and therefore encourage the AC to take the other two reviewers more into consideration.

AuthorsOpenReview ↗

Thank you, and a short clarification of Section 5.1

Thank you for reading the revision and for updating your assessment. We are glad that the seed results and the added baselines were useful.

On Section 5.1, here is the argument in plain terms:

  • The observation loss is never used during RL, so any effect of ActObs must already be present in the SFT checkpoint. Section 5.1 asks how the two SFT checkpoints come to differ.
  • At checkpoints along SFT, we compute two gradients on the same held-out trajectories: one from the action loss and one from the observation loss. Their cosine similarity tells us whether a step that improves action prediction also improves observation prediction.
  • At the start it does (cosine 0.83). After 10 to 20 of 781 steps it no longer does: the two gradients are nearly orthogonal. From then on, action-only SFT does nothing to maintain observation prediction, which ends below the base model (Section 5.2). ActObs keeps optimizing it.
  • Figure 5(b) shows that the two policies already differ in evaluation entropy by step 100 of SFT, before any RL.

If a particular step is still unclear, or if other parts of the paper remain hard to follow, we would be grateful for a pointer and will revise them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.