acceptodds
Under review as a conference paper at ICLR 2027

Learning Recurrent Belief Representations for Diffusion Policies with Predictive World Modeling

Abstract

Diffusion policies are commonly conditioned on a fixed observation window, restricting their effectiveness under partial observability, occlusion, and tasks requiring long-horizon memory. In a partially observable Markov decision process (POMDP), an effective policy should maintain a belief state that integrates historical information into a compact, structured world representation. We propose Belief-Inferring Recurrent Diffusion Policy (\ours), which integrates recurrence with attention mechanisms to maintain a latent belief representation that conditions the diffusion policy. In addition to the imitation objective, the belief encoder is trained with an auxiliary world-modeling loss that predicts latent representations of future observations from the current belief and a future action sequence, encouraging the latent state to capture the underlying world state. Across six simulated 2D manipulation tasks, achieves a mean peak success rate of 74%, a improvement over the strongest baseline, and outperforms diffusion policies that lack a belief state or are trained solely with the imitation objective.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Imitation Learning, Robotics, Robot Learning, POMDP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D5yNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BIRD conditions an action-chunk policy on a recurrent representation of observation history. A frozen MAE encodes images, and a GRU integrates visual features, proprioception, and the previous action. A discrete stochastic latent augments the recurrent state, and cross-attention reads recent belief representations. An auxiliary decoder predicts future visual tokens during training and is omitted at deployment. Across six simulated 2D tasks, BIRD reports 74% mean peak success versus 44% for the strongest baseline. The system improves control on tasks that require memory. However, future prediction provides no clear added benefit over current-feature reconstruction, and its two-step prediction objective does not establish how cues needed for later decisions are retained. Evaluation remains limited to a small 2D suite, leaving applicability to more demanding robot manipulation uncertain.

Critical strengths
  1. BIRD achieves strong results on tasks that require memory. It reaches 73% success on transient-cue Push-Box, 100% on Counting, and 87% on Drawer, versus 51%, 31%, and 47% for the strongest baseline on each task (Figure 4).

  2. Auxiliary visual supervision substantially improves the policy. Removing this loss lowers mean success from 80% to 58% on the partially observable pair and from 65% to 49% on the fully observable pair (Table 1). This supports training recurrent features with both visual and action supervision.

  3. Attention over past beliefs improves memory-dependent control. On the partially observable ablation pair, the readout raises mean success from 61% to 80% compared with conditioning only on the current recurrent state (Table 1). This supports retaining access to earlier beliefs alongside the current state.

Critical weaknesses
  1. The added benefit of future prediction is unclear. In Table 1, reconstructing current visual features achieves comparable or better performance. At the tested horizon, the results support auxiliary visual supervision, but do not establish what predicting the future adds.

  2. The predictive objective does not directly address the stated problem of retaining information for later decisions. It predicts visual features only two steps ahead, while the action head predicts 16-step chunks. A cue needed later, such as which drawer to choose, may be irrelevant to the next two observations. With the intervening expert actions also provided, the predictor can succeed without retaining that cue. Accurate short-term prediction therefore does not establish that the objective encourages the long-term memory used to motivate it.

  3. The evaluation leaves applicability to more realistic manipulation uncertain. All six tasks use simulated 2D manipulation with 96×9696\times96 images and 2D positional actions. Transient cues, counting, and hidden drawer contents provide meaningful controlled memory tests. However, there is no 3D or real-robot evaluation to show whether the learned memory remains useful under richer visual and physical complexity, including changing viewpoints, occlusion, and object interactions.

Questions for authors
  1. Given the comparable performance of current-feature reconstruction, what benefit motivates retaining future prediction in the final method?

  2. How does the two-step prediction objective, given the intervening expert actions, encourage retention of cues needed only later? What evidence shows that it teaches this form of memory?

  3. Can you provide evidence that the memory benefit extends to more realistic 3D or real-robot manipulation? Please explain the visual, physical, and memory challenges covered, so the evidence can be related to the intended robotics setting.

4 comments

We thank the reviewer for the time and feedback and answer the questions below:

The added benefit of future prediction is unclear. In Table 1, reconstructing current visual features achieves comparable or better performance. At the tested horizon, the results support auxiliary visual supervision, but do not establish what predicting the future adds.

The future observation prediction is motivated by maximizing mutual information with the POMDP state as in equation 15 and as shown in [2]; we also ablate this on the MIKASA-Robo [1] environments and find that it helps retain past information.

No forward prediction on MIKASA-Robo (success, mean ± std over 3 seeds; Table 4)

TaskBIRDNo forward prediction
ShellGame-Touch0.97 ± 0.010.42 ± 0.09
RememberColor30.97 ± 0.020.70 ± 0.13
ShellGame-Pick0.91 ± 0.010.83 ± 0.09

Does the memory keep the cue? We show the cue (e.g. which cup hides the ball), then hide it, and check whether it can still be read out of the model's recurrent state with a simple linear classifier. The arm is kept still, so the cue cannot be inferred from the image or the arm. Accuracy shortly after the cue is hidden (chance is 0.33; Appendix C):

TaskBIRDNo forward prediction
ShellGame-Touch0.990.36
ShellGame-Pick0.960.78
RememberColor31.000.83

The predictive objective does not directly address the stated problem of retaining information for later decisions. It predicts visual features only two steps ahead, while the action head predicts 16-step chunks. A cue needed later, such as which drawer to choose, may be irrelevant to the next two observations. With the intervening expert actions also provided, the predictor can succeed without retaining that cue. Accurate short-term prediction therefore does not establish that the objective encourages the long-term memory used to motivate it.

The two losses serve different purposes, the auxiliary objective is only there as a proxy of maximizing the mutual information between the latent state and the true state of the POMDP, i.e., the loss helps us maintain needed information from past observation in the latent state of the POMDP. On the other hand the action chunking loss is only a BC loss concerned with future actions. We show in the ablations on MIKASA-Robo that this improves task memory. We have added evaluation on the MIKASA-Robo benchmark and on real world experiments.

The evaluation leaves applicability to more realistic manipulation uncertain. All six tasks use simulated 2D manipulation with images and 2D positional actions. Transient cues, counting, and hidden drawer contents provide meaningful controlled memory tests. However, there is no 3D or real-robot evaluation to show whether the learned memory remains useful under richer visual and physical complexity, including changing viewpoints, occlusion, and object interactions.

In addition to our 2D results, we show our performance on the 3D MIKASA-Robo environment and on real world experiments in Fig. 4, Table 1 and Fig. 5.

MIKASA-Robo (success, mean over 3 seeds, 300 episodes each; Fig. 4)

TaskBIRDMarkovian PolicyPTPBC-RNN
ShellGame-Touch0.970.320.980.86
RememberColor30.970.330.930.69
Intercept-Medium0.790.730.830.83
ShellGame-Pick0.910.280.920.94
TakeItBack0.820.840.810.50

Real robot, UR5 (success rate; Table 1)

TaskBIRDPTPMarkovian Policy
Counting (four transfers)0.750.500.30
Cube under cup0.800.500.30

Given the comparable performance of current-feature reconstruction, what benefit motivates retaining future prediction in the final method?

Beyond the motivation in Eq. (15), the benefit shows on MIKASA-Robo. Removing forward prediction drops success from 0.97 to 0.42 on ShellGame-Touch, from 0.97 to 0.70 on RememberColor3 and from 0.91 to 0.83 on ShellGame-Pick (Table 4, Appendix D). A linear probe with the arm held still, so that neither the image nor the arm carries the cue, shows the cue fading much faster from the recurrent state without forward prediction, e.g. 0.99 against 0.36 on ShellGame-Touch three steps after the cue is hidden (Appendix C).

How does the two-step prediction objective, given the intervening expert actions, encourage retention of cues needed only later? What evidence shows that it teaches this form of memory?

The motivation is shown in Eq. (15). The MIKASA-Robo results show improved information retention, as demonstrated by the probe (0.99 vs 0.36 on ShellGame-Touch, Appendix C) and MIKASA drops (Table 4).

Can you provide evidence that the memory benefit extends to more realistic 3D or real-robot manipulation? Please explain the visual, physical, and memory challenges covered, so the evidence can be related to the intended robotics setting.

We show in the MIKASA-Robo environments that our method is able to solve 3D POMDPs covering visual, physical, and memory challenges including wrist camera views, occlusion, and 3D control. For real-robot evaluations, we cover real images, contact dynamics, counting, and cube-under-cup tasks. Generally we see that in imitation learning 2D environments such as pushT are useful in designing performant visual imitation learning algorithms.

References

[1] Cherepanov, Egor, et al. "Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning." arXiv preprint arXiv:2502.10550 (2025). [2] Gangwani, Tanmay, et al. "Learning belief representations for imitation learning in pomdps." uncertainty in artificial intelligence. PMLR, 2020.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Dear reviewer, as the discussion period comes to an end, please let us know if there are any further questions or if anything in our rebuttal is unclear.

Reviewer_Pa8DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes augmenting Diffusion Policy with latent belief states to account for partial observability. By leveraging a recurrent model that integrates observations across the episode, the obtained hidden state aggregates information without being bounded by context length. To supervise belief state learning, the proposed method uses a world-modeling objective to predict future visual latents. A memory constituted by a window of belief states conditions the diffusion policy through cross-attention. Experiments show that the proposed method achieves strong performance on both Markovian and non-Markovian tasks.

Critical strengths
  1. The proposed method builds a memory through recurrence, leading to compact representations for long-context history. Maintaining a window of belief states further retains the information that can be crucial for future task execution.
  2. The integrated belief states effectively improve performance on three Non-Markovian tasks, and the method achieves the strongest performance on two out of three Markovian tasks.
Critical weaknesses
  1. The evaluations are mainly conducted on Markovian and Non-Markovian tasks in a 2D cartoon environment, in which both the observation and action spaces are much simpler compared to actual robotic control settings. It remains unclear how the proposed method performs under more complex and realistic robotic scenarios.
  2. In ablation, removing the history belief context greatly reduces the performance. This highlights the importance of maintaining context but also questions the effectiveness of the “recurrence” mechanism for aggregating useful history information. It’d be helpful to (1) provide a detailed ablation on how varying context length WW affects performance and (2) compare with a baseline where the memory for cross-attention consists of a window of encoded historical frames while everything else stays the same. To cover longer history within the same context length, the memory can sample skipped historical frames at certain intervals.
Questions for authors
  1. In ablation, removing forward prediction seems to have little impact on Non-Markovian performance. For non-Markovian scenarios, as shown in PTP[1], would it be equally or even more important to predict the past accurately from the belief state? Since W=1W=1 degrades performance, it could be that the latest belief state fails to capture sufficient information from the past, and adding a past prediction loss to belief modeling would be a crucial experiment/ablation.

[1] Torne et al. Learning long-context diffusion policies via past-token prediction. 2025.

4 comments

We thank the reviewer for their time and feedback, and answer the questions below:

The evaluations are mainly conducted on Markovian and Non-Markovian tasks in a 2D cartoon environment, in which both the observation and action spaces are much simpler compared to actual robotic control settings. It remains unclear how the proposed method performs under more complex and realistic robotic scenarios.

We show the method on real-world experiments and on MIKASA-Robo [1]. In MIKASA-Robo, we see performance similar to PTP [2], mainly because the environments don't require long memory, and we see significant improvement on real-world tasks. Further, we show an ablation of varying the context length. We show the importance of recurrence in partially observable environments. We investigate the forward prediction loss in MIKASA-Robo and see that performance drops when only predicting the current embedding; although predicting future observations is required to maximize mutual information with the POMDP state (as in Eq. (15) and in [3]), empirical performance is domain-dependent.

We also find that predicting past actions is not as beneficial due to the fact that predicting future observations maximizes the mutual information to the underlying state of the POMDP, whereas predicting past actions doesn't improve state representation; it is slightly lower than BIRD (0.75 vs 0.80, Fig. 8), an additional reason here that unlike PTP we input the action into the recurrent cell and, as such, predicting past actions is unlikely to provide the latent states with new information.

Real robot, UR5 (success rate; Table 1)

TaskBIRDPTPMarkovian Policy
Counting (four transfers)0.750.500.30
Cube under cup0.800.500.30

MIKASA-Robo (success, mean over seeds, 300 episodes each; Fig. 4)

TaskBIRDMarkovian PolicyPTPBC-RNN
ShellGame-Touch0.970.320.980.86
RememberColor30.970.330.930.69
Intercept-Medium0.790.730.830.83
ShellGame-Pick0.910.280.920.94
TakeItBack0.820.840.810.50

Past-action prediction (peak success, mean ± std over seeds; Fig. 8)

TaskBIRDBIRD + past-action loss
Push-Box (transient cue)0.73 ± 0.030.67 ± 0.03
Drawer0.87 ± 0.050.84 ± 0.06
Mean0.800.75

In ablation, removing the history belief context greatly reduces the performance. This highlights the importance of maintaining context but also questions the effectiveness of the “recurrence” mechanism for aggregating useful history information. It’d be helpful to (1) provide a detailed ablation on how varying context length affects performance and (2) compare with a baseline where the memory for cross-attention consists of a window of encoded historical frames while everything else stays the same. To cover longer history within the same context length, the memory can sample skipped historical frames at certain intervals.

We ablate the effect of recurrence by applying attention directly to the observation embedding. We find that removing recurrence leads to a drop in performance in the POMDP environments. Using attention in our approach does not mean recurrence is not needed. Recurrence and attention are rather complementary. Our approach combines the benefits of compact, long-horizon memory from recurrence with the more fine-grained memory where attention is useful, similar to recent architectures that combine recurrence and attention in language modeling as in Griffin [4]. To test how the context span matters, we also re-evaluated the trained Drawer models with the readout stride changed at test time (Table 5, Appendix D): success is highest at the span the model was trained with and drops when the span is shortened or lengthened. However, this is likely due to the model being trained on that span,

Context ablations on the 2D tasks (peak success, mean ± std over seeds; Fig. 6)

VariantFully observablePartially observable
BIRD0.65 ± 0.030.80 ± 0.03
No recurrence0.64 ± 0.050.68 ± 0.08
No attention0.66 ± 0.040.61 ± 0.05

Readout span on Drawer (trained at S = 3, stride changed at test time, 3 seeds × 300 episodes; Table 5)

Stride S (span)134
Success0.40 ± 0.030.79 ± 0.080.64 ± 0.13

In ablation, removing forward prediction seems to have little impact on Non-Markovian performance. For non-Markovian scenarios, as shown in PTP[1], would it be equally or even more important to predict the past accurately from the belief state? Since degrades performance, it could be that the latest belief state fails to capture sufficient information from the past, and adding a past prediction loss to belief modeling would be a crucial experiment/ablation.

See Fig. 8 for past-prediction results. We show that removing the forward prediction and replacing it with predicting the same step observation leads to a drop in performance in the MIKASA-Robo environments. Ultimately, this is use-case dependent. However, performance is comparable on fully observable tasks, while there are gains to be had by predicting future observations, as shown in Fig. 6 and Table 4 (Appendix D).

No forward prediction on MIKASA-Robo (success, mean ± std over 3 seeds; Table 4)

TaskBIRDNo forward prediction
ShellGame-Touch0.97 ± 0.010.42 ± 0.09
RememberColor30.97 ± 0.020.70 ± 0.13
ShellGame-Pick0.91 ± 0.010.83 ± 0.09

References

[1] Cherepanov, Egor, et al. "Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning." arXiv preprint arXiv:2502.10550 (2025).

[2] Villasevil, Marcel Torne, et al. "Learning long-context diffusion policies via past-token prediction." 9th Annual Conference on Robot Learning. 2025.

[3] Bredis, George, et al. "Next embedding prediction makes world models stronger." ICLR 2026 Workshop on World Models. 2026.

[4] De, Soham, et al. "Griffin: Mixing gated linear recurrences with local attention for efficient language models." arXiv preprint arXiv:2402.19427 (2024).

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Dear reviewer, as the discussion period comes to an end, please let us know if there are any further questions or if anything in our rebuttal is unclear.

Reviewer_hTQLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BIRD, a diffusion policy conditioned on a learned belief state for partially observable manipulation. A frozen MAE encodes each frame, a GRU with a discrete stochastic latent (RSSM-style, without prior or KL) folds the history into a belief, and a single-query cross-attention reads a strided window of past beliefs into one conditioning vector for a Diffusion Policy U-Net. Besides behavior cloning, the belief is trained with an auxiliary loss that predicts future MAE patch embeddings from the belief and the intervening actions. On six self-designed 2D Push-T variants, three fully observable and three requiring memory, BIRD outperforms a single-frame Diffusion Policy, PTP, and BC-RNN, with the largest gains on the memory tasks.

Critical strengths
  1. A relevant problem with a clean memory test. Partial observability is a real gap for chunked diffusion policies, and the visible-cue vs. transient-cue Push-Box pair isolates the cost of having to remember in a simple, convincing way.
  2. The paper reports ablations that cut against its own framing (e.g., forward prediction not helping), which makes the results easy to interpret.
Critical weaknesses
  1. The "predictive world modeling" claim is not supported, and the method is a combination of known parts. The ablation that predicts the current token grid instead of future ones matches BIRD on the memory tasks and is better on the fully observable ones. The gain therefore comes from an auxiliary reconstruction-style loss on the belief, not from predicting the future, which is what the title and the mutual-information argument claim. The remaining components are standard: an RSSM-style recurrent core from Dreamer [1, 2], a frozen MAE [3] target and latent prediction from MWM [4] and NE-Dreamer [5], cross-attention over past states, and the Diffusion Policy head [6]. The paper does not show what is new beyond putting them together, or clearly present why this combination can resolve the key problems.

  2. The evaluation is narrow, and the baselines are not matched. All six tasks are self-designed 2D Push-T variants, with no established memory benchmark and no real robot. Results use the best checkpoint over the whole training run, which inflates all reported numbers. The baselines differ from BIRD in encoder, denoiser, and auxiliary loss at the same time: the Diffusion Policy [6] baseline is reduced to a single frame, PTP [7] uses a transformer denoiser the authors themselves call hyperparameter-sensitive, and BC-RNN [8] fails even on a fully observable task, which suggests a configuration problem.

The authors also concede that part of BIRD's advantage does not come from memory. A fair test needs the same encoder and auxiliary loss with a frame window instead of recurrence.

[1] Hafner, Danijar, et al. "Dream to control: Learning behaviors by latent imagination." International Conference on Learning Representations (2020).

[2] Hafner, Danijar, et al. "Mastering Atari with discrete world models." International Conference on Learning Representations (2021).

[3] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022).

[4] Seo, Younggyo, et al. "Masked world models for visual control." Conference on Robot Learning. PMLR, 2023.

[5] Bredis, George, et al. "Next embedding prediction makes world models stronger." arXiv preprint arXiv:2603.02765 (2026).

[6] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." Robotics: Science and Systems (2023).

[7] Torne, Marcel, et al. "Learning long-context diffusion policies via past-token prediction." arXiv preprint arXiv:2505.09561 (2025).

[8] Mandlekar, Ajay, et al. "What matters in learning from offline human demonstrations for robot manipulation." Conference on Robot Learning. PMLR, 2021.

Questions for authors
  1. Since predicting the current token grid works as well as forward prediction, what supports calling the auxiliary loss "world modeling"? Do longer horizons change this?

  2. How does a diffusion policy with the same MAE encoder and auxiliary loss, but a frame window instead of recurrence, perform on the memory tasks?

  3. Can you report final-checkpoint (not peak) success, and results on an established memory benchmark or a real robot?

6 comments

We thank the reviewer for the time and feedback and answer the questions below:

The "predictive world modeling" claim is not supported, and the method is a combination of known parts. The ablation that predicts the current token grid instead of future ones matches BIRD on the memory tasks and is better on the fully observable ones. The gain therefore comes from an auxiliary reconstruction-style loss on the belief, not from predicting the future, which is what the title and the mutual-information argument claim. The remaining components are standard: an RSSM-style recurrent core from Dreamer [1, 2], a frozen MAE [3] target and latent prediction from MWM [4] and NE-Dreamer [5], cross-attention over past states, and the Diffusion Policy head [6]. The paper does not show what is new beyond putting them together, or clearly present why this combination can resolve the key problems.

There is a gap in current generative control policies regarding partially observable environments. Recent baselines such as PTP [1] resort to auxiliary losses that help in only solving some environments and do not necessarily help the estimation of the hidden state of the POMDP as they rely on fixed context windows. In our paper we show that a combination of recurrence and attention and auxiliary loss aimed at maximizing the mutual information between the latent state and the underlying state of the POMDP can allow the method to outperform baselines under partial observability. As far as we know, our method is the first generative (diffusion/flow) control policy combining recurrence and attention. The predictive world modeling is backed by the fact that optimizing future observation prediction can lead to maximizing mutual information to the underlying state of the POMDP as shown in Eq. (15). Ultimately the results are domain dependent. We do the ablation on the MIKASA-Robo [2] domains and we find that the predictive objective improves success rate. We use the BC-RNN [3] and PTP default settings. Ablations on MIKASA-Robo show that future prediction is needed, and we add an ablation removing recurrence that shows recurrence is required. We add comparisons on the MIKASA-Robo benchmark and on real world experiments. BC-RNN is a weak baseline, because the handling of the recurrent state in BC-RNN doesn’t help in POMDPs, as the hidden state is reset every 10 steps. The reason being that the network is trained on 10 step windows in training which leads to the removal of information outside that window. In addition to the latent state reset, the lack of action chunking in BC-RNN harms it in challenging manipulation tasks. In the mikasa environments, a chunk of one is used for all baselines where memory requirements are low, which explains their comparable performance. Ultimately combining different components in an effort to extend diffusion policies to POMDPs is not a downside rather a useful contribution to the field.

Ablations on the 2D tasks (peak success, mean ± std over 3 seeds; Fig. 6)

VariantFully observablePartially observable
BIRD0.65 ± 0.030.80 ± 0.03
No recurrence0.64 ± 0.050.68 ± 0.08
No attention0.66 ± 0.040.61 ± 0.05
No forward prediction (current embedding prediction)0.72 ± 0.020.78 ± 0.04
No predictive loss0.49 ± 0.040.58 ± 0.17
No stochastic latent0.61 ± 0.020.82 ± 0.02
Conv. encoder0.66 ± 0.020.81 ± 0.04

No forward prediction on MIKASA-Robo (success, mean ± std over 3 seeds; Table 4)

TaskBIRDNo forward prediction
ShellGame-Touch0.97 ± 0.010.42 ± 0.09
RememberColor30.97 ± 0.020.70 ± 0.13
ShellGame-Pick0.91 ± 0.010.83 ± 0.09

The evaluation is narrow, and the baselines are not matched. All six tasks are self-designed 2D Push-T variants, with no established memory benchmark and no real robot. Results use the best checkpoint over the whole training run, which inflates all reported numbers. The baselines differ from BIRD in encoder, denoiser, and auxiliary loss at the same time: the Diffusion Policy [6] baseline is reduced to a single frame, PTP [7] uses a transformer denoiser the authors themselves call hyperparameter-sensitive, and BC-RNN [8] fails even on a fully observable task, which suggests a configuration problem.

We evaluated our method on the MIKASA-Robo tasks, a 3D robotic benchmark. Unfortunately, the MIKASA-Robo benchmark has the issue that the cue is partly recoverable from the arm motion, so our real-robot tasks test longer memory. We also design two real world robotic tasks where the agent needs to have longer memory, and evaluate the memory with a counting task and an active perception task where the agent needs to look for an object hidden underneath two cups. Diffusion Policy's [4] single frame is used on purpose as a memoryless reference. PTP uses the authors' published configuration. Peak success is common in imitation learning papers, and peak is applied equally to all methods. Table 6 (Appendix E) gives final-checkpoint/last-10 results where BIRD stays ahead.

MIKASA-Robo (success, mean over seeds, 300 episodes each; Fig. 4)

TaskBIRDMarkovian PolicyPTPBC-RNN
ShellGame-Touch0.970.320.980.86
RememberColor30.970.330.930.69
Intercept-Medium0.790.730.830.83
ShellGame-Pick0.910.280.920.94
TakeItBack0.820.840.810.50

Real robot, UR5 (success rate; Table 1)

TaskBIRDPTPMarkovian Policy
Counting (four transfers)0.750.500.30
Cube under cup0.800.500.30

Peak / final checkpoint / mean of last 10 evaluations on the 2D tasks (mean over seeds; Table 6)

TaskBIRDMarkovian PolicyPTPBC-RNN
Push-T.50 / .39 / .39.54 / .41 / .44.49 / .24 / .31.43 / .30 / .33
Push-Box (visible cue).80 / .76 / .68.66 / .63 / .46.57 / .51 / .48.00 / .00 / .00
Push-T (two goals).53 / .49 / .43.42 / .38 / .26.25 / .15 / .13.43 / .30 / .27
Push-Box (transient cue).73 / .59 / .61.41 / .35 / .25.51 / .46 / .42.00 / .00 / .00
Counting1.00 / .99 / .98.24 / .10 / .14.31 / .25 / .19.21 / .13 / .13
Drawer.87 / .68 / .65.38 / .26 / .22.25 / .21 / .15.47 / .01 / .16

The first three tasks are fully observable, the last three partially observable.

Since predicting the current token grid works as well as forward prediction, what supports calling the auxiliary loss "world modeling"? Do longer horizons change this?

Predicting future observations increases the mutual information between the latent state and the underlying state of the POMDP as shown in [5]. Ultimately, the result is use-case dependent; we evaluated the forward prediction loss on the MIKASA-Robo tasks and show that it’s necessary to retain the past observations in the recurrent state.

How does a diffusion policy with the same MAE encoder and auxiliary loss, but a frame window instead of recurrence, perform on the memory tasks?

Removing the recurrent cell implements this baseline (same encoder, auxiliary loss, and readout), showing a performance drop in partially observable environments (Fig. 6) and shown above. In fully observable environments, we see no degradation in performance. We also perform ablations on the MAE encoder and the auxiliary losses above.

Can you report final-checkpoint (not peak) success, and results on an established memory benchmark or a real robot?

We added Table 6 (Appendix E) showing that the ranking holds with final-checkpoint results, and we report results on MIKASA (Fig. 4) and a real robot (Table 1, Fig. 5). We also added the results to this comment to make it easier to check.

[1] Villasevil, Marcel Torne, et al. "Learning long-context diffusion policies via past-token prediction." 9th Annual Conference on Robot Learning. 2025.

[2] Cherepanov, Egor, et al. "Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning." arXiv preprint arXiv:2502.10550 (2025).

[3] Mandlekar, Ajay, et al. "What matters in learning from offline human demonstrations for robot manipulation." Conference on Robot Learning. 2021.

[4] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." Robotics: Science and Systems. 2023.

[5] Gangwani, Tanmay, et al. "Learning belief representations for imitation learning in pomdps." uncertainty in artificial intelligence. PMLR, 2020.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Comments on the paper

3 comments

We thank the reviewers for their time and constructive suggestions. All changes in the revised manuscript are marked in blue. Below, we summarize the main concerns shared across the reviews and the experiments and results we added to address them. Detailed answers are given in the individual replies.

  1. Evaluation beyond 2D tasks: All three reviewers asked whether BIRD extends beyond the 2D Push-T variants. We added two evaluations.
  • MIKASA-Robo [1] (Fig. 4): five 3D manipulation memory tasks with an overhead camera, a wrist camera whose viewpoint moves with the arm, occlusion by the gripper and the objects, and 3D arm control. BIRD and PTP [2] perform similarly on most tasks, while the Markovian Policy (Diffusion Policy [3]) struggles when the cue is hidden before the policy acts on it. Part of the cue in these tasks can be recovered from arm motion, so a limited observation window can suffice, which motivated our real-robot tasks and our 2D POMDP environments with longer memory requirements.
  • Real robot (Table 1, Fig. 5): two real world memory tasks on a UR5. In the counting task, the arm carries a cube between two plates exactly four times, from scenes that look the same between transfers. In the active perception task, the cube is hidden under one of two cups and, once a cup is put back, the scene no longer shows what was under it. BIRD outperforms both PTP and the Markovian policy.
  1. The role of forward prediction: The reviewers noted that predicting the current token grid performs comparably to forward prediction on the 2D tasks. Ultimately, the forward objective is motivated by the mutual-information bound in Eq. (15). To test it where the cue must persist far longer than the prediction horizon, we ran the ablation on three MIKASA-Robo tasks (Table 4, Appendix D). Removing forward prediction lowers success from 0.97 to 0.42 on ShellGame-Touch, from 0.97 to 0.70 on RememberColor3, and from 0.91 to 0.83 on ShellGame-Pick. A linear probe with the arm held still, so that neither the image nor the arm carries the cue, shows that the cue fades much faster from the recurrent state without forward prediction, 0.99 against 0.36 on ShellGame-Touch three steps after the cue is hidden (Appendix C). These results are merged into the No forward prediction bar of Fig. 6. As we see, the forward prediction effect is domain-dependent, but for general POMDPs, it improves the latent representation as also noted by [5].

  2. Recurrence against a window of frames: No recurrence (Fig. 6): this is a baseline that was asked for by the reviewers. It keeps the same frozen MAE [4] encoder, auxiliary loss, and cross-attention readout, but attends over per-frame embeddings instead of recurrent beliefs. It matches BIRD on the fully observable tasks and trails it on the partially observable ones, so a strided window of frames does not replace the recurrent belief.

  3. Past prediction: Following the reviewers' suggestion, we added a head that predicts the eight past actions from the current belief, as in PTP [2] (Fig. 8, Appendix D). It reaches 0.75 against 0.80 for BIRD on the partially observable pair, and its loss falls close to zero early in training. Since past actions are an input to the recurrent cell, the belief already retains the recent past, and the explicit past-prediction loss adds no benefit.

  4. Peak and final checkpoint: Peak selection is applied identically to every method. Table 6 (Appendix E) reports the peak, the final checkpoint, and the mean of the last 10 evaluations for all methods. BIRD stays ahead of every baseline on all three partially observable tasks under all three measures.

  5. Contribution: Current generative control policies condition on a fixed window of observations and do not maintain a belief over the hidden state. BIRD combines a recurrent architecture with attention to provide a strong sequential model for the latent state, along with a predictive auxiliary loss to maximize the mutual information between the latent state and the underlying state of the POMDP. The ablations show that each part contributes under partial observability, and the new MIKASA-Robo and real-robot results show that the combination transfers to 3D and real-world memory tasks.

Summary of changes to the manuscript

  • Added MIKASA-Robo results (Fig. 4) and two real-robot tasks (Table 1, Fig. 5), with supplementary videos.
  • Added the No recurrence ablation and merged the MIKASA-Robo No forward prediction results into Fig. 6.
  • Added Appendix C (probe details, including the MIKASA-Robo memory probe), Appendix D (past prediction, Fig. 8; per-task No forward prediction on MIKASA-Robo, Table 4; readout span, Table 5) and Appendix E (final-checkpoint results, Table 6).
  • Moved the task illustrations to Appendix A to stay within the page limit.
  • Updated the abstract to mention the MIKASA-Robo and real-robot results.

[1] Cherepanov, Egor, et al. "Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning." arXiv preprint arXiv:2502.10550 (2025).

[2] Villasevil, Marcel Torne, et al. "Learning long-context diffusion policies via past-token prediction." 9th Annual Conference on Robot Learning. 2025.

|[3] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." Robotics: Science and Systems. 2023.

[4] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[5] Bredis, George, et al. "Next embedding prediction makes world models stronger." ICLR 2026 Workshop on World Models. 2026.

As the end of the reviewing phase is approaching, we have added the results of the additional experiments directly in the comments to make it easier for the reviewers, we also want to highlight that video comparing our method to a markovian policy on real-world tasks are submitted in the supplementary material. We are looking forward for any feedback from the reviewers and are happy to answer any further questions or perform more experiments if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.