acceptodds
Withdrawn from ICLR 2027

State Trace Rationale As Auxiliary Task in Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose STRAT, an auxiliary task that trains deep reinforcement learning (RL) agents to predict a short textual trace of their own state. Inspired by human spatial navigation, the description combines landmark, route, and survey knowledge, tracking the agent's position, inventory, goals, and immediate progress. Environment rules generate this text online without human labelling. Our method adds a single auxiliary head to a standard policy. Across 60 sparse-reward XLand-MiniGrid tasks, STRAT solves complex environments where standard RL fails outright, while compacting state representations and preventing rank collapse. Beyond performance gains, the predicted trace provides a readable account of agent beliefs at every step for no extra cost.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Auxiliary tasks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WDiVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STRAT, a PPO-based RL method that augments policy learning with an auxiliary task of predicting a templated textual description of the agent's state, including its position, goal, subgoal, previous action, progress, and inventory. The aim is to improve learned representations and sample efficiency in sparse-reward environments. Experiments report improvements over PPO, alongside ablations and analyses of representation capacity and linear-probe predictability.

Critical strengths

The paper proposes a potentially useful approach to sparse-reward RL. The method is conceptually simple and appears straightforward to implement, making it an interesting approach to incorporating additional task-relevant supervision into policy learning.

Critical weaknesses
  1. Inadequate baselines. Improved performance over vanilla PPO is not particularly surprising given the additional privileged symbolic information available to STRAT. The evaluation should include baselines that exploit the same information, such as an asymmetric actor–critic, to establish whether STRAT offers benefits beyond conventional uses of privileged information.

  2. Insufficient justification for textual representations. It is unclear why predicting a textual description of the state abstraction should be more effective than directly predicting its individual structured components. This distinction should be evaluated empirically using comparable auxiliary objectives.

  3. Insufficient methodological detail. The method lacks a sufficiently precise mathematical description of how abstract MDP features are used and how the environment's production rules contribute to constructing the supervision. The main paper should also provide substantially more implementation detail, particularly regarding the interaction between the textual representation and the network architectures. The description in Appendix A is insufficient to assess these design choices or reproduce the method reliably.

Questions for authors
  1. Could the authors include baselines that exploit the same privileged information, particularly an asymmetric actor–critic, and directly compare textual trace prediction against auxiliary prediction of the corresponding structured state variables?

  2. What approaches do the authors envision for overcoming the performance collapse on the Align family of tasks, where the proposed method also fails? What does this failure reveal about the limitations of the method?

Reviewer_Hv1hOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

STRAT adds an auxiliary head to a Transformer-XL PPO agent that predicts at every step a templated state trace of global position, goal, subgoal, previous action with a progress label, and inventory. Labels come from the simulator, the subgoal from a plan backward-chained through the hidden ruleset, and the progress label from the critic's value difference. Sixty XLand-MiniGrid tasks compare STRAT with an identical PPO agent.

Critical strengths

S1. The method is simple, needing one auxiliary head and online labels.

S2. The main comparison covers sixty tasks with ten seeds, and gains on Go-hold tasks are consistent.

Critical weaknesses

W1. [Method] The gains come from privileged labels rather than from the proposed rationale. XLand-MiniGrid hides the goal and rules by design, yet STRAT is supervised with the goal, a subgoal derived from the hidden rules, and unobserved global coordinates, while the only baseline receives none of this. The paper's own ablation contradicts its thesis. Predicting position and action alone matches the full trace, while predicting goal and subgoal alone collapses to the baseline. Since action and heading are already inputs, the effective signal is global coordinates plus the critic's value sign, a known localization auxiliary (Mirowski et al., 2017). The planning components contribute nothing alone, yet the paper concludes all are needed. No position-only head or plan-based shaping is compared.

W2. [Method] The interpretability claim has no evidence. Probes show that the variables the head is trained to predict are easier to decode. Trace accuracy and faithfulness to the policy are not measured. The progress slot is a sign of the critic's own value difference at τ=10⁻⁵, with critic noise fed back into the critic. BFS or Euclidean distance works as well or better, and returns change with a threshold already near zero.

W3. [Experiments] The evidence is too thin for the conclusions. PPO is the only baseline. Ablations use one or two tasks yet are generalized to all components and any backbone, although the full trace is worse than two variants on m-18 and GRU is clearly below Transformer-XL. Both methods score zero on Align and near zero on most of Beside, so evidence comes almost entirely from fifteen Go-hold tasks, without aggregate statistics. Agents are never evaluated on unseen rulesets, the benchmark's intended setting.

Questions for authors

Q1. How does STRAT compare with a position-only head and with PPO given the oracle subgoal?

Q2. How accurate are predicted traces on held-out episodes?

Reviewer_dS1FOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes STRAT, an auxiliary task that trains RL agents to predict a short textual description of their current state. These descriptions are automatically generated based on environment rules, without requiring human annotations. The method is relatively simple, only adding an auxiliary prediction head to the original policy. Experiments on 60 sparse-reward XLand-MiniGrid tasks show the effectiveness of the proposed method.

Critical strengths

This paper propose a novel method for RL training motivated by human spatial navigation. The idea itself is interesting.

Critical weaknesses

I don't think the paper in its current form meets the ICLR bar. First, the technical contribution is relatively limited. The paper mainly introduces an additional loss term into the original PPO objective, but doesn't provide a theoretical analysis to explain why this loss should help, beyond some intuition. Second, the paper seems a bit rushed. For example, in line 200, the citations for PPO and GAE are both missing. The introduction is also very short, and there is almost no discussion of related work.

Questions for authors

Please see the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.