acceptodds
Under review as a conference paper at ICLR 2027

InterSelf: Learning World Models under Hidden Action Execution

Abstract

World models typically assume that the action selected by a policy is the action that causes the next transition. This assumption fails when an unobserved execution process repeats, drops, or remaps commands, causing the model to conflate execution dynamics with environmental dynamics. We formalize this mismatch as an action-fidelity problem and quantify the predictive information lost by conditioning only on intended actions. We introduce InterSelf, which augments DreamerV3 with a recurrent latent execution model combining categorical action remapping and temporal persistence. Its effective-action representation conditions both world-model learning and latent imagination, without executed-action labels or corruption identities. We show that this channel covers the evaluated execution failures, while its latent mixtures are identifiable only when different actions leave sufficiently separable effects. Across a 22-game Atari study with three training seeds, InterSelf achieves higher corrupted returns than DreamerV3 in 14 games; in seven action-sensitive games, it outperforms both DreamerV3 and the capacity-matched InterSelf-Identity control. Variation across the remaining games is consistent with the predicted boundary: explicit execution modeling need not help when action effects are weak or redundant. These results establish hidden action execution as a distinct source of world-model misspecification and provide a principled way to model it.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, hidden action execution, latent execution dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_rM68OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the setting where the action selection is not necessarily the action actually executed. This then makes the problem one of hidden actions. Then then ask under which conditions such actions become identifiable and propose a modification of the DreamerV3 framework to address the problem. The authors provide both theoretical and empirical results.

Critical strengths
  • The problem studied seems interesting and practically relevant.
  • The paper is well motivated
Critical weaknesses
  • The paper is not very well written and at times hard to follow. It follows a non-exhaustive list of things I have noticed:
  1. A lot of phrases are used once or twice and then never used again
  2. In the abstract is says: “We formalize this mismatch as an action-fidelity problem” but “action-fidelity problem” is not used again in the main part of the paper.
  3. Are “execution protocol”, “execution process” the same thing?
  4. About “control variable”: are these simply the actions actually performed? It appears twice and then is not used again.
  5. Are “hidden actions” and “latent actions” and “latent components” the same thing?
  6. Are “Bayes prediction gap” and “Oracle prediction gap” the same thing?
  7. “latent execution channel” is used once in contributions
  8. equations (6), (7), (8) are the same as the ones of (17), (18), (19)?
  • “The oracle gap in Theorem 1 does not imply that a hidden channel is recoverable.” Why would the reading think that Theorem 1 would imply that a hidden channel is recoverable?

  • Theorem 1 and Corollary 2 seem trivial and in particular do not establish "A useful surrogate must therefore preserve execution-relevant history not already contained in S".

  • The remaining theoretical results have while not trivial are quite basic and seem to have little novelty.

  • There is significant related work missing on hidden action identification (e.g. "Nikulin, A., Zisman, I., Tarasov, D., et al. (2025). Latent Action Learning Requires Supervision in the Presence of Distractors", "Schmidt, D., & Jiang, M. (2024). Learning to Act without Actions", "Schur, F. (2026). Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversity"). Schur (2026) work seems especially related: If one views the intended action as a the demonstrator one might be able to apply (under additional assumptions) their results and get a meaningful theoretical result.

Questions for authors
  • In what respect are the theoretical results novel/non-basic?
  • How do these results relate to established works on identifying hidden actions?
Reviewer_BZzdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes InterSelf, a world-model reinforcement learning framework designed to handle hidden action execution, where intended actions may be repeated, dropped, or remapped before being executed. The method augments DreamerV3 with a recurrent latent execution model that predicts effective-action distributions for world-model learning and imagination. The authors provide theoretical analyses of execution uncertainty, identifiability, and rollout stability, and evaluate the approach on 22 Atari games with synthetic action corruptions.

Critical strengths
  1. Meaningful problem formulation and structured modeling. The paper explicitly distinguishes intended actions from executed actions and models hidden execution through action remapping and temporal persistence. Applying the same execution representation during world-model learning and imagination provides a coherent approach to handling structured action uncertainty.

  2. Well-designed experimental controls and broad evaluation. The capacity-matched InterSelf-Identity baseline is a useful design that helps isolate the contribution of action mediation from additional model capacity. The evaluation covers 22 Atari games under multiple execution corruptions, with three independent training seeds. The reported improvements in several games suggest that explicit execution modeling can be beneficial in certain settings.

Critical weaknesses
  1. Insufficient justification for explicit execution modeling. Theorem 1 quantifies the predictive benefit of observing the actual executed action, but InterSelf only predicts an execution distribution from the intended action and interaction history. A sufficiently expressive history-conditioned world model can already represent the resulting stochastic transitions. While explicit factorization may provide a useful inductive bias, the theoretical analysis does not establish why the proposed execution channel should improve prediction or control over standard recurrent world models.

  2. Lack of direct evidence for the proposed mechanism. Despite learning an explicit execution channel, the paper does not evaluate whether its predicted action distributions or persistence probabilities correspond to the underlying execution process. Such evaluation is feasible because the corruption mechanisms are known in the simulator. Moreover, the auxiliary effect-prediction loss uses post-transition latent features, potentially allowing the prediction target to be recovered without relying on the inferred effective action. Consequently, the experiments do not convincingly establish that the intended execution mechanism is actually learned.

  3. Inconsistent empirical benefits and weak support for the theoretical explanation. InterSelf outperforms the capacity-matched InterSelf-Identity baseline in only 9 of 22 games. The seven games highlighted in Table 2 are selected based on favorable performance differences, rather than independently measured action-effect separability. The paper also does not evaluate whether the identifiability conditions in Theorem 4 correlate with the observed improvements. These limitations substantially weaken the empirical support for the proposed mechanism.

Questions for authors
  1. Necessity of the execution channel. Since the inferred effective-action distribution is computed from available interaction history and intended actions, what predictive information does it provide beyond a sufficiently expressive recurrent world model? How does Theorem 1, which assumes access to the actual executed action, justify the benefit of the proposed learned surrogate?

  2. Validation of execution modeling. Can the authors directly compare the inferred execution distributions and persistence probabilities against the known corruption mechanisms? Can they also demonstrate improvements in world-model prediction accuracy or imagination fidelity, independently of policy returns?

  3. Empirical consistency. Why does InterSelf underperform the capacity-matched identity baseline in 13 of 22 games? Can the authors provide per-game uncertainty estimates and demonstrate whether the performance differences correlate with independently measured action-effect separability, as suggested by Theorem 4?

Reviewer_6RKzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes InterSelf, a method based on DreamerV3 for the action-fidelity problem caused by the inconsistency between intended action and executed action. InterSelf introduces a recurrent latent execution channel to combine categorical action remapping and temporal persistence. The generated effective-action representation conditions both world-model learning and latent imagination. The theoretical analysis of the proposed method is provided, covering oracle prediction gap, coverage of the evaluated execution channels, identifiability and stability from action effects, and recovery-to-imagination stability. Experiments in 22 Atari games show that InterSelf achieves higher corrupted returns than DreamerV3 in 14 games. The findings suggest that explicit execution modeling can improve control, with benefits varying across environments.

Critical strengths

This paper proposes a meaningful execution-aware inductive bias, and uses reasonable identity control to demonstrate the control gains. The problem formulation and the theoretical analysis appear to be sound.

Critical weaknesses

The experiment design cannot sufficiently justify the proposed method. It does not isolate the effects of mixed-execution training and evaluation-time corruption. It appears that all methods are trained using mixed-mode training, lacking a baseline that uses E=A for both training and evaluation. In addition, the experiments focus on the returns and identity control, which shows that the proposed method improves control performance, but does not sufficiently show that it improves the action-fidelity problem. For example, InterSelf improves clean returns in several games (e.g., Demon Attack and Seaquest), where execution modeling should not be needed. It would be better to include some behavior analysis for the policy, to distinguish whether the improvements come from the execution modeling or just a more robust policy. For example, whether the learned channel tracks the active corruption mode, or how the world-model prediction error changes.

The "Our setting is complementary: ..., rather than only a robust policy" justification is not sufficient. To achieve the objective of allowing an agent to maintain return performance under hidden execution corruption, a robust policy is also suitable. Specifically, the paper lacks an analysis or discussion of whether the proposed explicit execution modeling performs better than other existing robust methods focused on action uncertainty, beyond the DreamerV3 baseline that is itself trained under mixed corruption.

Questions for authors
  1. Can you provide the results that fully use E=A for training and evaluation, and compare them with the existing mixed training approach?
  2. Can you provide some analyses to demonstrate that the improvement corresponds to execution modeling, or just policy behavior?
  3. Please clarify the algorithm choice between InterSelf and other robust methods. How does the current evidence support the extra benefits of the explicit execution modeling?
Reviewer_BmxnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InterSelf for settings where the action the environment executes can differ from the action the policy chose. Built on DreamerV3, it adds a recurrent latent execution channel that infers the executed action from the intended one and feeds it to the world model. On 22 Atari games with synthetic execution corruptions, it beats DreamerV3 under corruption in 14.

Critical strengths
  • The paper makes explicit an assumption that action-conditioned world models such as DreamerV3 leave implicit, that the intended action is the executed action. It then tests this in a controlled hidden-execution setting.
Critical weaknesses
  • The novelty is thin. The method adds a small execution module on top of DreamerV3. The experiments use only the four synthetic corruptions the model was built to represent, at the same rate as in training. Adding more experiments, such as held-out failure types, or real-world settings such as robots with faulty actuators, to support the effectiveness of the proposed method would be helpful.
  • There are many undefined notations in the paper, which makes it hard to follow
    • E_t and K_* are only introduced implicitly
    • d_{t+1} and d_x in Eq. (22) are not defined
    • T_\phi is not defined
    • The values of the loss weights and gate offsets are not reported
Questions for authors
  • The paper acknowledges that a recurrent model like DreamerV3 can already learn the marginal transition under hidden execution (l. 125–126). What evidence shows that isolating the execution channel helps, e.g., lower prediction loss, or a learned channel that tracks the true executed action?
  • In Table 2, InterSelf’s corrupted mean is below its clean mean in all seven games. By contrast, DreamerV3 scores higher under corruption than under clean execution in Chopper Command and Seaquest, as does InterSelf-Identity in Space Invaders and Seaquest. From a broader perspective, does InterSelf’s advantage under corruption come from robustness to hidden execution, or from higher clean performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.