acceptodds
Under review as a conference paper at ICLR 2027

Double Identity: Decoupling Control State and Task Belief in Reinforcement Learning

Abstract

In partially observable control problems with persistent hidden task properties, immediate control and task inference may benefit from different temporal contexts. Conditions relevant to the next action can change at every transition, while evidence identifying an unobserved goal or dynamics regime accumulates over longer horizons. Encoding both in a single representation creates conflicting demands: the representation must be highly responsive to local changes while preserving as much task-relevant information from prior interactions as possible. We introduce State-Task Adaptive Control (STAC), an actor-critic architecture that assigns these inference roles to separate causal encoders with different temporal contexts and learning objectives. A control encoder summarizes recent interactions and is trained through temporal-difference learning. At the same time, a task encoder attends causally to the full interaction history and is trained with goal supervision, reward prediction, and contrastive objectives. A trainable adapter maps the task encoder's output to belief features used by the actor and critics. Under a hidden-goal protocol using the ten Meta-World ML10 training tasks, STAC achieves mean evaluation success rates of 71% within the training region and 62% in held-out regions, exceeding the strongest evaluated baseline on each split by 20% and 18%, respectively. Architectural ablations further support limiting the control encoder to recent interactions while using a separate encoder to aggregate available task evidence. Together, these findings show that tailoring both temporal context and learning objectives to the distinct demands of control-state estimation and task inference can improve learning and generalization in the evaluated manipulation setting.

Primary area:
reinforcement learning
Keywords:
Meta-reinforcement learning, partially observed reinforcement learning, task inference, representation learning, robotic manipulation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5vfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In meta reinforcement learning, an agent should distinguish between a control representation that captures the current state and varies across time steps, and task information that remains shared across time steps within the same task. The paper introduces STAC, which uses a control encoder to map a fixed window of context into a control representation and a task encoder to map the complete rollout history into a task representation. This approach improves learning and generalization.

Critical strengths
  1. The paper provides detailed experimental results, visualizations, and architectural ablations that support the effectiveness of the double encoder architecture.
  2. The method outperforms prior methods on the evaluated benchmark while using a compact architecture.
Critical weaknesses
  1. The encoder's loss function is not sufficiently discussed or evaluated. It includes contrastive losses to encourage task-representation learning, but the contributions of individual components, such as covariance regularization and goal prediction, are not examined through ablation studies.
  2. The overall novelty is limited. Compared with prior methods, the paper uses a Transformer-based control encoder with a fixed context window to aggregate contextual information, in lack of further improvement on algorithms or concepts.
  3. STAC uses goal coordinates and task labels to train its task encoder, while the baselines do not consistently receive equivalent supervision. The extra task information may induce unequally comparison.
Questions for authors

Could the authors provide ablations removing individual task loss components, particularly goal prediction and covariance regularization to specify the influence of extra task information and regularization term, while keeping the architecture and training budget fixed?

Reviewer_AWrgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper has introduced STAC, a dual-encoder meta-RL architecture method that uses 97-token control window and full-trial task memory, trained through TD learning and auxiliary task-inference objectives, respectively.

Critical strengths
  1. Convincing temporal-context ablation: The strongest finding that I found is the improvement from 38.4 ± 4.3% to 65.4 ± 2.6% when restricting control attention, without changing parameter count or objectives (Table 9) provideing meaningful evidence that longer context can hurt learned control representations despite containing more information.

  2. Promising empirical performance: It is clear to me how STAC is improving substantially over the reported baselines,especially on manipulation tasks where several alternatives fail. Also, the evaluation includes multiple task families, held-out position regions, and behavioral interventions rather than relying exclusively on aggregate success.

Critical weaknesses
  1. Limited architectural novelty: Humplik et al. (2019) already separate supervised task-belief inference from recurrent control with blocked RL gradients, while Akuzawa et al. (2021) distinguish state and task latents.

  2. Potentially unfair baseline comparisons: STAC uses privileged goal and task-attribute supervision unlike other baselines. AMAGO-2 uses only 256-step context in comparison to STAC's 750-step task history. this difference is not negligible when trying to analyze the strength of the work.

  3. Limited theoretical justification: Propositions 1 and 2 establish standard information-theoretic identities but the 97 token parameter is still a question on why the particular number why not more or less.

  4. Incomplete architectural and representation validation: Shared-encoder ablations confound parameter sharing with history length. Removing gradient isolation slightly improves performance (65.4% to 67.6%), and individual auxiliary losses are not ablated.

Questions for authors
  1. How does STAC compare against a supervision- and capacity-matched implementation of Humplik et al. (2019), given the substantial overlap in architectural principles?

  2. Can the authors provide comparisons with equivalent privileged supervision, history lengths, and computational budgets to establish whether STAC's gains arise from temporal separation?

  3. Can the authors have a detailed ablation on different control-history lengths and environments requiring longer-term physical-state memory? What has motivated the choice of (K=97)?

  4. Can encoder separation, gradient isolation, and individual auxiliary objectives be ablated independently? What quantitative evidence demonstrates genuine state–task specialization?

  5. Does STAC generalize to unseen ML10 task families and sparse-reward settings? Can the authors provide additional seeds and per-episode adaptation results?

Reviewer_yRi3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes learning separate encoders for task and state for meta-rl tasks. The task encoders observes the full history of interactions, whereas the state encoder only observes a small chunk of it. In addition to this, the authors propose using different gradient signals for these encoders. For instance, the state encoder gets a gradient from TD learning, whereas the task encoder does not. Through experiments on the Meta-World benchmark, the authors show the efficacy of their method.

Critical strengths

The paper is easy to read. The contribution is simple and direct to understand: use different amount of context for the task and state encoders. The empirical gains on Meta-World are nice.

Critical weaknesses
  1. The contribution is a bit incremental in that it is a simple design choice that works with existing methods.
  2. The results and ablations only show that using different context lengths for the two encoders is beneficial. It does not show (or convincingly show) that different encoders are needed and that the learning signals need to be different too. The ablations have several confounds.
  3. Several comparisons are also unfair because methods such as PEARL and VariBAD do not have access to privileged goal info that STAC has.
Questions for authors
  1. Could you provide experiments where the only variable between tasks is : (1) the shared or seperate encoders, or (2) learning signals?
  2. Could you please explain why the gains cannot be explained by the presence of privileged goal/task information alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.