acceptodds
Withdrawn from ICLR 2027

Hawkes-Augmented World Model for Event-Driven Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models enable reinforcement learning agents to improve policies through imagined latent trajectories. In environments with temporally clustered events, recent event history can provide useful context for predicting subsequent dynamics and rewards. We introduce the Hawkes-Augmented World Model (HAWM), which equips a recurrent stochastic state-space model with learned event inference and a Hawkes-style occurrence prior. An observation-conditioned posterior infers latent event occurrences and types from observed transitions, while a latent-conditioned prior predicts them during imagination. Both pathways share an exponentially decaying memory of binary event occurrences. The prior combines this memory with latent context to predict subsequent occurrence probabilities, while event types are predicted separately to condition the world model's prediction heads. The event-memory embedding also conditions recurrent dynamics and actor-critic learning, allowing accumulated event history to inform prediction and control. An auxiliary future-reward objective encourages the event representation to capture information relevant to subsequent rewards. HAWM thus introduces an explicit event-history state whose evolution can be inferred from observations and predicted during observation-free imagination.

Primary area:
reinforcement learning
Keywords:
Model-based Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mJyUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Hawkes-Augmented World Model (HAWM), which augments DreamerV3's recurrent state-space model (RSSM) with explicit event-occurrence variables, event types, and a Hawkes-style exponentially decaying event memory. An observation-conditioned posterior infers events from observed trajectories, while a latent-conditioned prior predicts events during imagination. An auxiliary future-reward objective encourages the event representation to capture information relevant to subsequent task progress. The method is evaluated on seven robotic manipulation tasks from ManiSkill3 and MS-HAB.

Critical strengths
  1. The paper is reasonably motivated by the influence of consequential interaction events on long-term prediction and control. The overall approach offers some novelty.

  2. The integration of posterior–prior event modeling with world-model imagination is natural and coherent. Event history continues to inform prediction and policy learning throughout imagined rollouts.

  3. The empirical evaluation is relatively thorough. The main experiments, ablations, and generalization results consistently show substantial advantages for HAWM on multi-stage robotic manipulation tasks.

Critical weaknesses
  1. The experiments primarily focus on robotic manipulation tasks with distinct stages and temporally clustered events. These tasks naturally involve phases such as contact, grasping, alignment, and insertion, making them well matched to the proposed Hawkes inductive bias. The current evaluation does not provide sufficient evidence that the method is also effective on tasks without a pronounced clustered-event structure.

  2. Event discovery relies heavily on future-reward supervision. When rewards are very sparse or substantially delayed, consequential interactions may fall outside the fixed prediction horizon, weakening the learning signal for event discovery.

  3. Parts of the paper, particularly the experimental section in Appendix C.4, show clear signs of AI-generated writing or experimental reporting. This substantially undermines the manuscript's professionalism and credibility.

Questions for authors
  1. The current evaluation primarily covers robotic manipulation tasks with distinct stages and temporally clustered events. Could the authors include broader visual-control benchmarks, such as Atari-100K, to determine whether HAWM remains effective in environments without explicit event structures such as contact, grasping, and alignment?

  2. Could the authors clarify the description of the experiments in lines 1194–1195 of Appendix C.4?

Overall, I basically agree with the overall approach proposed in this paper for event history modeling. However, some of the experiments and textual descriptions in the paper, especially in Appendix C.4, have significantly affected my judgment on the overall professionalism and credibility of the manuscript. For this reason, I give a rejection score.

Reviewer_26eDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Hawkes-Augmented World Model (HAWM), a modification of DreamerV3 that explicitly represents sparse event occurrences and maintains a decaying memory of their history. The motivation is that important interactions—such as grasping an object, engaging a tool, or establishing contact with an articulated object—can influence subsequent dynamics and rewards even when they are no longer visually apparent. The model is trained without event annotations. Its objectives combine the standard RSSM losses with observation, reward, and continuation prediction; an auxiliary future-reward prediction objective; an event-rate regularizer; and posterior-to-prior matching for event occurrences and types. The auxiliary reward target is intended to encourage events that capture task-relevant progress rather than merely large visual changes.The authors evaluate HAWM on seven robotic manipulation tasks from ManiSkill3 and MS-HAB, comparing it with DreamerV3, EADream, and TD-MPC2. They also report component ablations, event visualizations, and experiments involving unseen scenes and illumination changes.

Critical strengths

S1 A clear and well-motivated modeling idea

The paper identifies a meaningful limitation of conventional latent world models: the model may need to preserve the consequences of an earlier interaction even after the corresponding visual evidence has disappeared. Explicit event occurrences and a decaying history provide a reasonable inductive bias for this setting. The use of a shared memory update for observation-conditioned inference and observation-free imagination is particularly appropriate. It connects event discovery during training with the event dynamics that the policy must rely on when optimizing imagined trajectories. The approach also retains primitive-step control rather than requiring a hierarchical policy or temporally extended actions.

S2 Promising empirical results across diverse manipulation tasks

The evaluation covers seven tasks spanning tabletop manipulation, tool use, insertion, mobile navigation, and articulated-object manipulation. The reported gains on tasks involving sequences of interactions are encouraging, and the unseen-scene and illumination experiments provide useful evidence beyond performance in the training configuration. The ablation results are also suggestive: removing the event-memory mechanism is associated with a substantial performance drop, and removing future-reward supervision is similarly damaging. These results motivate further investigation into event history as a useful representation for model-based control.

S3 The method is accompanied by useful component analysis and implementation detail

The paper specifies the event inference and prediction pathways, the memory update, the training objectives, and important implementation choices. It also examines the number of event types rather than assuming that a larger event vocabulary is necessarily better. The reported results favor K=3K=3 over the evaluated alternatives K=2,4,5K=2,4,5, which is useful evidence about the configuration studied. The authors also acknowledge that a discrete-time Hawkes-style trace is not a full point-process likelihood and identify free-running prior errors as a relevant limitation. These qualifications are valuable because they help define the actual scope of the contribution.

Critical weaknesses

W1 The specific contribution of Hawkes-style memory is not yet isolated convincingly

The central claim is that an explicit self-exciting event history improves world-model prediction and control. However, HAWM combines several changes: a learned event detector, event-type representations, an additional memory embedding, future-reward supervision, and a prior that predicts events during imagination. The reported ablations are encouraging, but the comparison labeled “w/o Hawkes memory” replaces the memory mechanism with an MLP event predictor. This changes more than the presence of an exponential trace, so the performance gap does not cleanly establish that Hawkes-style temporal decay is the decisive ingredient. I would like to see a more controlled comparison against a same-backbone model with a comparable learned memory but without positive self-excitation—for example, a learned exponential trace without the occurrence-probability excitation term, or a generic recurrent memory with matched capacity. This would help distinguish the benefit of explicitly representing event history from the benefit of adding parameters and auxiliary supervision. There is also a conceptual qualification: the method uses a discrete-time binary event process with a positive exponential-history term, not a full Hawkes point-process likelihood. This is a reasonable design choice, but the paper should carefully distinguish the benefits of the proposed parameterization from claims about Hawkes processes more generally.

W2 Event discovery and event semantics are supported primarily by qualitative evidence

A central motivation is that HAWM discovers consequential events rather than merely detecting visually surprising transitions. The paper provides illustrative trajectories in which events appear aligned with contact, object displacement, alignment, and task completion. These examples are intuitive, but they do not by themselves establish that the learned events are consistently meaningful. The event vocabulary is learned without event-level annotations, and the manuscript does not establish that the learned types correspond to stable, semantically distinct categories across episodes, random seeds, or task variants. Moreover, because future-reward prediction explicitly shapes event features, alignment with reward progression is partly an intended training outcome rather than wholly independent evidence of interpretability. A stronger evaluation would quantify event quality against independently defined interaction milestones, assess the precision and recall of event detection where annotations can be obtained, or measure how stable event types remain across runs. It would also be useful to establish whether event representations identify meaningful intermediate interactions before reward changes, rather than simply correlating with successful trajectories. This is especially important because interpretability and predictive usefulness are distinct claims: a representation can improve control without providing semantically consistent event categories.

W3 The evidence for robustness and generalization needs more careful qualification

The results on unseen scenes and altered illumination are promising, but the reported generalization study is concentrated on a single NavigateAndPick configuration and a fixed target object. The scene experiment uses five training scene configurations and 50 unseen configurations, while the illumination experiment evaluates dim and bright rendering conditions on the training scenes. These are useful controlled tests, but they provide limited evidence for broad generalization across different tasks, objects, camera configurations, or physical dynamics. In addition, the illumination results demonstrate robustness to particular rendering changes, not necessarily robustness to changes in the underlying environment. The reported means and standard deviations also leave some questions about the statistical strength of the comparisons. In particular, the number of independent training seeds, the evaluation episode counts used for the main results, and the statistical testing procedure should be made explicit wherever they are not already clear in the paper. For stochastic RL methods, the uncertainty across training runs is important when judging whether the improvements are reliable. Finally, the manuscript's own conclusion notes that event-prior errors can compound during free-running imagination. This is a significant concern for a method whose policy is optimized on imagined trajectories. The paper should make clear how accurately the prior reproduces event frequency and temporal structure over increasingly long rollouts, and how sensitive the control gains are to errors in those predictions.

Questions for authors

Q1. How much of the performance gain is attributable specifically to Hawkes-style self-excitation? Can the authors compare HAWM against a same-backbone model with a matched-capacity memory that does not use the positive exponential excitation term? For example, could the authors replace the event-history contribution to the prior with a learned non-self-exciting memory while keeping event inference, event types, and future-reward supervision unchanged? This would help isolate the proposed temporal inductive bias from the benefits of auxiliary objectives and additional model capacity.

Q2. How are the semantic quality and consistency of learned events validated independently of reward prediction? Beyond selected trajectory visualizations, can the authors provide quantitative event diagnostics against independently annotated interaction milestones, or measure event-type consistency across random seeds and task variants? In particular, how often do detected events correspond to meaningful physical interactions rather than latent-state fluctuations, and do the learned event types retain consistent meanings across runs?

Q3. How well does the latent-conditioned event prior remain calibrated during free-running imagination? Can the authors report event-frequency calibration and temporal prediction metrics over multiple rollout horizons, alongside comparisons between posterior-inferred and prior-generated event sequences? How much do these errors affect imagined returns and policy performance as the rollout length increases? This would help determine whether the event history remains reliable in the observation-free setting where it is most consequential.

Q4. How robust are the reported gains across independent runs and broader generalization settings? Could the authors clarify the number of independent training seeds, evaluation episodes, and uncertainty estimates for the main comparisons? If feasible, can they provide additional evidence on different objects, scene configurations, or task variants to establish whether the improvements extend beyond the current NavigateAndPick generalization setting?

Reviewer_5uhFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Hawkes-Augmented World Model (HAWM), which extends DreamerV3 with a Hawkes-style event memory to capture temporally dependent events. An observation-conditioned posterior infers event occurrences and types, while a latent-conditioned prior predicts them during imagination. The model learns event representations without event-level annotations through predictive objectives and future-reward supervision. Experiments on seven robotic manipulation tasks demonstrate substantial performance improvements over existing baselines.

Critical strengths
  • Effective posterior–prior design. The observation-conditioned posterior and latent-conditioned prior enable consistent event-history modeling across observed trajectories and imagined rollouts.
  • Annotation-free event learning. The method learns task-relevant event representations without explicit event annotations by leveraging world-model prediction and future-reward signals.
Critical weaknesses
  • Methodological complexity. The framework introduces multiple interacting modules and auxiliary objectives, increasing optimization complexity and potentially limiting its applicability across different domains.
  • Insufficient evidence for Hawkes self-excitation. The ablation studies do not adequately isolate the benefits of Hawkes-style self-excitation from those of additional event memory or representational capacity.
  • Limited quantitative event evaluation. Event discovery is primarily supported by qualitative visualizations, with insufficient quantitative evidence of the learned events' relevance and temporal structure.
  • Potential distribution shift in event prediction. The Hawkes prior is trained using changes in posterior latent distributions, (Δzt=vec⁡(ζtpost−ζt−1post))(\Delta z_t=\operatorname{vec}(\zeta_t^{post}-\zeta_{t-1}^{post})), but operates on dynamics-prior distributions during imagination, (Δzt=vec⁡(ζtprior−ζt−1prior))(\Delta z_t=\operatorname{vec}(\zeta_t^{prior}-\zeta_{t-1}^{prior})). This mismatch may lead to compounding errors, yet the accuracy and calibration of the prior under free-running imagination are not sufficiently evaluated.
  • Limited benchmark coverage and baseline comparisons. The evaluation is restricted to robotic manipulation tasks, without standard benchmarks such as Atari, DeepMind Control Suite (DMC), or Crafter. Although DreamerV3 and TD-MPC2 are included, comparisons with other world-model approaches, such as TWISTER, STORM, and DIAMOND, are missing.
Questions for authors

See strengths and weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.