acceptodds
Under review as a conference paper at ICLR 2027

CogWAM: Aligning Semantic Cognition with World Action Modeling via Event-Driven Interfaces

Abstract

Robot policies increasingly incorporate semantic reasoning and future-world prediction, yet combining these capabilities does not guarantee that local predictions and actions remain aligned with task progress. We introduce CogWAM, a cognition-guided world-action model that establishes an explicit semantic interface between task reasoning and world-action learning through a persistent Semantic State, which stores completed task events and the active subtask. CogWAM updates this state only when observations indicate semantic transitions, allowing task-level context to persist across multiple action chunks. To bridge semantic context with physical prediction and control, CogWAM employs progress-conditioned WORLD and ACTION queries that selectively extract task-relevant information for future-world prediction and action generation. During training, the Semantic State provides shared task-progress context for both branches, while inference removes the future-prediction branch and directly generates actions from observations and the maintained state. We further introduce semantic training strategies to improve transition learning and closed-loop conditioning. Without additional robot-action pretraining, CogWAM achieves 15.56/11.70% Score/SR on RoboDojo and state-of-the-art performance on BiCoord, while real-world experiments demonstrate closed-loop dual-arm manipulation with 16.4× fewer Semantic State regenerations than step-wise updating.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Event-driven Semantic Planning, Latent World Action Model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AeKsOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CogWAM maintains a semantic state containing completed events and the active subtask, updating it when observations indicate a task transition. The paper reports improvements across simulation and real-robot evaluations, and says event-triggered updates reduce semantic-state regenerations by 16.4× while keeping average inference latency at 146.7 ms.

Critical strengths
  1. The persistent state connects task-level progress to both future-world prediction and action generation.
  2. Evaluations include simulated benchmarks, real-robot tasks, and tests under distribution shifts.
  3. Transferring the same semantic interface to Fast-WAM raises its average RoboDojo Score / SR from 3.48 / 2.03% to 13.33 / 9.40%, suggesting the approach can benefit architectures beyond CogWAM itself.
Critical weaknesses
  1. The semantic state is append-only and cannot revise an incorrect committed event.
  2. Novel-object generalization remains limited, with 14 successes in 30 trials.
Questions for authors

Please see the weaknesses.

Reviewer_dQRJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper maintains a persistent Semantic State that records completed events and active subtasks. KEEP/UPDATE decisions trigger semantic state updates upon event detection. WORLD and ACTION queries extract distinct representations from a shared vision-language backbone, supporting future DINO feature prediction and action generation respectively. Future prediction acts only as an auxiliary training objective and the world branch is discarded at inference time.

Critical strengths
  1. Benefits of the semantic state interface transfer across multiple models. CogWAM achieves Score/SR of 15.56/11.70 on RoboDojo and SSR/SR of 58.0/43.8% on BiCoord. After porting the interface to Fast-WAM, RoboDojo average Score/SR improves from 3.48/2.03 to 13.33/9.40. This cross-architecture transfer provides stronger evidence for the practical utility of the interface than gains within a single model.
  2. It offers a deployable solution for the timescale mismatch between semantic reasoning and low-level control. State change checks run at high frequency while autoregressive semantic generation runs at low frequency. Real-robot deployment reports average inference latency reduction from 927.8 ms to 146.7 ms, suggesting this design may alleviate online control budget constraints.
Critical weaknesses
  1. The unique contribution of coupled semantic–world action modeling is not isolated. The Action MoT→MoT→CogWAM ablation in the appendix is useful, but the key baseline of “action model + identical semantic state without world prediction supervision” is missing. It is therefore impossible to confirm whether semantic state and future prediction deliver complementary gains beyond two independent enhancements. Independent ablations for cumulative history, shared vs separate query heads, and the two training strategies are also absent. Gradient analysis offers interpretive clues but cannot substitute for performance ablations.
  2. Evidence for the reliability of event-driven updates is weaker than evidence for efficiency. Appendix disclosure notes state supervision is sourced from predefined stage checklists and offline boundary annotations. This means gains from structured supervision must be separated from gains of the online update mechanism. State updates are append-only, so erroneous completion judgements may corrupt all downstream decisions. Meanwhile, the transition delay metric in Table 4 primarily measures when the policy can observe a boundary event, not the latency of correct boundary identification. Real-robot results lack matched baselines and sufficient tail latency statistics.
Questions for authors
  1. With matched backbone, semantic annotation and training budgets, how does “action model + semantic state” compare against full CogWAM? Can the authors provide a 2×2 ablation crossing semantic supervision and world prediction supervision?
  2. What are the practical precision, recall and correct-update latency of the KEEP/UPDATE detector? How does the system recover after a false positive update?
  3. Does the reported 146.7 ms include the full generation cost for UPDATE steps? Please report average, P95 latency conditioned on UPDATE, alongside matched task success rates across the compared policies.
Reviewer_t9vMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CogWAM, which uses a VLM to maintain semantic memory and guide action generation in a downstream WAM. Specifically, the upstream VLM maintains a Semantic State that summarizes completed events and identifies the current subtask. The paper introduces an event-triggered mechanism that allows the VLM to determine when to update the Semantic State based on current observations. Additionally, two special queries, WORLD and ACTION, are introduced into the VLM, whose corresponding hidden states serve as conditions for the downstream WAM. The WAM adopts a MoT architecture to generate future DINO features and actions, with each stream trained using flow matching. Future DINO feature prediction is used only during training. Through simulation and real-world evaluation, the paper demonstrates the effectiveness of the proposed method.

Critical strengths
  1. The paper introduces a semantic state into WAM, leveraging a VLM for event summarization and subtask generation. It also introduces an event-triggered mechanism that allows the VLM to automatically determine when to update the semantic state, improving WAM performance on long-horizon tasks.
  2. The proposed framework demonstrates extensibility, as applying it to FastWAM also leads to performance improvements.
Critical weaknesses

The experiments primarily demonstrate that adding the semantic memory improves a WAM. However, the completed-event summary and current subtask are still generated by an upstream VLM, while existing memory-augmented methods typically condition a downstream VLA or action expert on such VLM-generated guidance. The paper doesn't demonstrate why a WAM is needed in place of these existing alternatives or whether the proposed method provides benefits beyond memory-conditioned action generation. Especially on the long-horizon benchmark BiCoord, comparisons against standard VLAs like pi_0 and pi_05, which lack comparable memory mechanisms, are not fully convincing.

Questions for authors
  1. As mentioned in the Weaknesses, how does CogWAM compare with existing memory-augmented VLAs? In particular, do the performance improvements come from the use of WAM or from the proposed memory mechanism?
  2. How are the VLM outputs H^w and H^a from the WORLD and ACTION queries incorporated into the downstream WAM? Are they concatenated with the noisy token sequence and processed through self-attention? The attention mask in Figure 6 seems to suggest this, while Figure 2 shows these outputs being introduced after the "Multi-Modal Self-Attention" module.
  3. What is the exact inference procedure? According to Eq. (5), H^w and H^a depend on S_t. Does this mean that the VLM first performs one forward pass to obtain S_t, followed by a second forward pass to generate H^w and H^a conditioned on S_t, which are then used to guide generation in the downstream WAM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.