- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
PARM-WM constructs a persistent episode-level memory of visual responses to actions and uses it to condition video forecasting through a bounded, zero-preserving low-rank residual. The full model combines this memory-conditioned pathway with a separately trained JEPA predictor. Given seven observed frames, it predicts the next twenty and reports pixel-MSE reductions of 49.3%, 3.8%, and 12.1% relative to JEPA-WM on RoboMimic Lift, LIBERO New, and LanguageTable.
Critical strengths
- PARM-WM treats action–response history as a persistent representation of environment dynamics, offering a distinctive perspective on visual world modeling and future-state prediction.
Critical weaknesses
- The experiments establish gains in visual prediction, but the method's effect on robot planning or task success remains untested.
- The paper analyzes forecast steps from H1 to H20, yet these span only about 1–2 seconds; it does not vary the duration of the stored action–response history.
Questions for authors
- Could the authors supplement the offline prediction results with robot evaluations reporting planning performance and task success?
- How does forecasting quality vary with the duration of the stored action–response history, and does the memory benefit persist beyond the evaluated 1–2 seconds?
Thank you for your questions about robot planning, history length, and longer forecasts.
PARM-WM achieved the highest success rate, used the fewest actions, and had the lowest goal error in our planning test. We tested four planners on the same 40 held-out RoboMimic Lift episodes in simulation. The methods shared initial states, goal images, and the first six actions.
At each decision, we retrieved 16 candidate action sequences from training trajectories. The model-based planners predicted 20 steps for each candidate and ranked them by the distance between the predicted final observation and the goal image. The robot then executed five actions, received new camera observations, and planned again. All methods had the same candidate budget and a 100-action limit. We used the environment's cube-lifting criterion to determine success.
*Mean actions include all episodes, with failures assigned 100 actions. Goal error is the DINO feature distance between the final executed observation and the goal image.
PARM-WM reached 62.5% success, 10.0 percentage points higher than JEPA and 22.5 points higher than retrieval alone. It reduced final goal error by 11.80% and 33.04%, respectively, and used 6.16% and 9.77% fewer actions on average.
The zero-memory control keeps both trained branches and their mixing weight, with the explicit state set to zero. Compared with this control, PARM-WM increased success by 5.0 points, reduced goal error by 8.16%, and used 4.40% fewer actions. We used PARM-WM models trained with the paper's final method and training settings, keeping their weights fixed during evaluation.
The best results came from eight or twelve observed events, and the longer forecasts showed larger relative gains. Both tables report Motion MSE (x10^-3, lower is better), which measures image error in changing regions.
History length. K counts completed action-response events. The paper's K≤C−1 bound applies when writing within the C-frame context. Here we extend the writer's input to earlier observed events while keeping the same seven-frame predictor input, H20 targets and 64-dimensional memory fixed. We tested K=4/8/12/20 on the same windows from the paper's test sets, all with enough observed history for K20.
K=4–20 spans 0.2–1.0 s of history in RoboMimic/LIBERO and 0.4–2.0 s in LanguageTable. The lowest error occurs at K=8 for RoboMimic and LanguageTable, and K=12 for LIBERO. Compared with K=4 on the same windows, error fell by 5.08% on RoboMimic, 4.29% on LIBERO, and 1.73% on LanguageTable. These results show that a moderate increase in action-response history further improves prediction accuracy.
Longer predictions. We extended prediction to H40/H60 with K=4 and fixed model weights. H20 uses paper results for PARM-WM and LIBERO/LanguageTable JEPA, with supplementary results for the other controls. For each H, we used trajectories long enough for that forecast and compared methods on the same queries. Errors are averaged over steps 1 through H.
The last two columns give relative error reductions. PARM-WM has lower error in all eight settings, with larger relative gains at H40/H60 than in the H20 reference rows. Against zero memory, the reduction grows from 32.45% to 33.70% on RoboMimic (H40), from 11.43% to 20.47% on LIBERO (H60, 3 s), and from 9.91% to 15.19% on LanguageTable (H60, 6 s). The gains over JEPA grow from 20.71%/1.70%/8.52% to 24.79%/21.45%/12.64%, respectively.
RoboMimic trajectories support evaluation up to H40. LanguageTable H60 uses 10 episodes.
Thank you again for your helpful feedback. We would be happy to discuss any further questions : )