acceptodds
Under review as a conference paper at ICLR 2027

PARM-WM: Persistent Action–Response Memory for Long-Horizon Visual World Models

Abstract

Long-horizon visual prediction must retain how a scene responds to earlier actions as real observations leave the rolling context. We introduce PARM-WM, a world model with Persistent Action–Response Memory (PARM). An observable response encoder combines visual feature changes with aligned actions to form a compact state. A zero-preserving, low-rank residual then reads this state throughout the rollout. A validation-selected latent interpolation combines the response branch with a JEPA continuation using one mixing weight across sources. We evaluate the complete predictor against seven action-conditioned baselines on three robotic trajectory sources, using seven observed frames to predict twenty future frames. PARM-WM achieves the best values on all four metrics for RoboMimic Lift, LIBERO New, and LanguageTable, reducing pixel MSE relative to JEPA-WM by 49.3%, 3.8%, and 12.1%, respectively. Globally Holm-corrected paired tests confirm improvements over every baseline on all four metrics on all three sources. These results establish persistent action–response memory as a useful interface for sustained visual forecasting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Models, Long-Horizon, Bounded Action–Response Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dr6YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

PARM-WM constructs a persistent episode-level memory of visual responses to actions and uses it to condition video forecasting through a bounded, zero-preserving low-rank residual. The full model combines this memory-conditioned pathway with a separately trained JEPA predictor. Given seven observed frames, it predicts the next twenty and reports pixel-MSE reductions of 49.3%, 3.8%, and 12.1% relative to JEPA-WM on RoboMimic Lift, LIBERO New, and LanguageTable.

Critical strengths
  1. PARM-WM treats action–response history as a persistent representation of environment dynamics, offering a distinctive perspective on visual world modeling and future-state prediction.
Critical weaknesses
  1. The experiments establish gains in visual prediction, but the method's effect on robot planning or task success remains untested.
  2. The paper analyzes forecast steps from H1 to H20, yet these span only about 1–2 seconds; it does not vary the duration of the stored action–response history.
Questions for authors
  1. Could the authors supplement the offline prediction results with robot evaluations reporting planning performance and task success?
  2. How does forecasting quality vary with the duration of the stored action–response history, and does the memory benefit persist beyond the evaluated 1–2 seconds?
1 comment

Thank you for your questions about robot planning, history length, and longer forecasts.

W 1 / Q 1: Planning performance and task success beyond offline prediction.

PARM-WM achieved the highest success rate, used the fewest actions, and had the lowest goal error in our planning test. We tested four planners on the same 40 held-out RoboMimic Lift episodes in simulation. The methods shared initial states, goal images, and the first six actions.

At each decision, we retrieved 16 candidate action sequences from training trajectories. The model-based planners predicted 20 steps for each candidate and ranked them by the distance between the predicted final observation and the goal image. The robot then executed five actions, received new camera observations, and planned again. All methods had the same candidate budget and a 100-action limit. We used the environment's cube-lifting criterion to determine success.

PlannerTask successActions used*Final goal error
PARM-WM25/40 (62.5%)70.380.7346
Same ensemble, zero memory23/40 (57.5%)73.620.7999
JEPA21/40 (52.5%)75.000.8329
Retrieval only16/40 (40.0%)78.001.0970

*Mean actions include all episodes, with failures assigned 100 actions. Goal error is the DINO feature distance between the final executed observation and the goal image.

PARM-WM reached 62.5% success, 10.0 percentage points higher than JEPA and 22.5 points higher than retrieval alone. It reduced final goal error by 11.80% and 33.04%, respectively, and used 6.16% and 9.77% fewer actions on average.

The zero-memory control keeps both trained branches and their mixing weight, with the explicit state set to zero. Compared with this control, PARM-WM increased success by 5.0 points, reduced goal error by 8.16%, and used 4.40% fewer actions. We used PARM-WM models trained with the paper's final method and training settings, keeping their weights fixed during evaluation.

W 2 / Q 2: How does prediction vary with stored-history duration, and do memory benefits persist beyond 1–2 seconds?

The best results came from eight or twelve observed events, and the longer forecasts showed larger relative gains. Both tables report Motion MSE (x10^-3, lower is better), which measures image error in changing regions.

History length. K counts completed action-response events. The paper's K≤C−1 bound applies when writing within the C-frame context. Here we extend the writer's input to earlier observed events while keeping the same seven-frame predictor input, H20 targets and 64-dimensional memory fixed. We tested K=4/8/12/20 on the same windows from the paper's test sets, all with enough observed history for K20.

Dataset (episodes)K=4K=8K=12K=20
RoboMimic (33)4.46934.24234.24404.3231
LIBERO (40)17.128816.577916.394617.2073
LanguageTable (26)12.154111.943812.017612.1091

K=4–20 spans 0.2–1.0 s of history in RoboMimic/LIBERO and 0.4–2.0 s in LanguageTable. The lowest error occurs at K=8 for RoboMimic and LanguageTable, and K=12 for LIBERO. Compared with K=4 on the same windows, error fell by 5.08% on RoboMimic, 4.29% on LIBERO, and 1.73% on LanguageTable. These results show that a moderate increase in action-response history further improves prediction accuracy.

Longer predictions. We extended prediction to H40/H60 with K=4 and fixed model weights. H20 uses paper results for PARM-WM and LIBERO/LanguageTable JEPA, with supplementary results for the other controls. For each H, we used trajectories long enough for that forecast and compared methods on the same queries. Errors are averaged over steps 1 through H.

DatasetHJEPAZero memoryPARM-WMvs JEPAvs zero
RoboMimic205.61186.58774.449820.71%32.45%
RoboMimic406.99787.93805.262724.79%33.70%
LIBERO2017.070018.945216.77961.70%11.43%
LIBERO4022.586023.338419.706012.75%15.56%
LIBERO6028.153527.807322.114521.45%20.47%
LanguageTable2014.198014.417812.98858.52%9.91%
LanguageTable4016.706617.091214.648012.32%14.30%
LanguageTable6022.518323.195719.673012.64%15.19%

The last two columns give relative error reductions. PARM-WM has lower error in all eight settings, with larger relative gains at H40/H60 than in the H20 reference rows. Against zero memory, the reduction grows from 32.45% to 33.70% on RoboMimic (H40), from 11.43% to 20.47% on LIBERO (H60, 3 s), and from 9.91% to 15.19% on LanguageTable (H60, 6 s). The gains over JEPA grow from 20.71%/1.70%/8.52% to 24.79%/21.45%/12.64%, respectively.

RoboMimic trajectories support evaluation up to H40. LanguageTable H60 uses 10 episodes.

Thank you again for your helpful feedback. We would be happy to discuss any further questions : )

Reviewer_ifKrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PARM-WM combines observed action-response pairs into a fixed memory that modulates later visual predictions. The evaluated predictor blends this response branch with a separately trained JEPA branch.

Critical strengths

Episode-paired confidence intervals and Holm correction cover 84 comparisons. Static, shuffled, and zero-state controls help distinguish shared corrections from episode-specific conditioning.

Critical weaknesses
  1. The authors do not explain the relation between the performance benifits in the main table and the memory mechanism. RoboMimic's hybrid system includes additional joint continual training. Specifically, JEPA branch uses a version with RGB weights of 100, while the main table JEPA-WM uses a version with weights of 30; the complete system also evaluates two predictors at each step.

  2. The necessity of the "action-response" pairing is yet to be demonstrated by direct experiments. Shuffling retains 94.1% of the correct-versus-zero motion-MSE gain, and state ablations use a frozen-base validation model rather than the final ensemble. Test within-prefix action-response mismatches and repeat interventions on the final ensemble using held-out data (see table 4).

  3. The "long-term memory" concept mainly involves short-term continual predictions beyond rolling context. Appendix D.1 clearly states that H20 corresponds to 1, 1, 2 seconds respectively in RoboMimic, LIBERO, and LanguageTable, and the write window is also fixed at 4 completed transfers.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ifKr (1/2): Branch Comparisons and Action-Response Pairing

Thank you for your careful review and helpful questions about the role of action-response memory.

W 1: Main-table gains may reflect additional joint training, RGB100 versus RGB30 JEPA, and two predictors per step.

Full outperforms each branch, and memory helps with both trained branches held fixed.

The RoboMimic JEPA row below is the RGB100 companion used in Full. Full lowers pixel MSE by 25.53% against it. The paper's 49.3% compares Full with RGB30 JEPA. RoboMimic jointly trains the response base and memory interface. On LIBERO/LanguageTable, the companion is the main-table JEPA and the response base stays frozen during interface training.

We evaluated the final models on held-out data from all three datasets. We kept C7/K4/H20, the decoder and evaluation protocol. Full and LIBERO/LanguageTable JEPA use paper results. MSE is x10^-3, with lower values better. Higher PSNR and SSIM are better.

Dataset / conditionPixel MSEMotion MSEPSNRSSIM
RoboMimic / Full0.95364.449830.75480.95169
RoboMimic / zero1.47986.587728.37730.92077
RoboMimic / action-only1.26956.034129.46010.94153
RoboMimic / shuffle1.29746.271028.92200.94024
RoboMimic / JEPA, λ=01.28065.611829.09380.94099
RoboMimic / response, λ=11.31405.847128.96580.93865
LIBERO / Full3.802816.779625.17070.83218
LIBERO / zero4.113218.945224.06730.81773
LIBERO / action-only4.060717.870124.71110.82118
LIBERO / shuffle4.101518.777724.17140.81220
LIBERO / JEPA, λ=03.952017.070024.99040.8278
LIBERO / response, λ=14.174118.657124.76220.81714
LanguageTable / Full2.911412.988526.05330.88715
LanguageTable / zero3.464014.417825.75080.87164
LanguageTable / action-only3.153514.145025.77890.87205
LanguageTable / shuffle3.255813.757625.93150.88068
LanguageTable / JEPA, λ=03.312014.198025.60780.8790
LanguageTable / response, λ=13.428314.849625.48300.87620

λ=0 selects JEPA and λ=1 the response branch with memory. Full uses λ=0.4 and is best on all four metrics in each dataset. Motion MSE falls 20.71%/1.70%/8.52% versus JEPA and 23.90%/10.06%/12.53% versus the response branch (RoboMimic/LIBERO/LanguageTable). On RoboMimic, PSNR rises by 1.66/1.79 dB over JEPA/response.

Zero memory keeps the same trained two-branch system. We fix both branches' weights, λ=0.4, the decoder, actions and test queries, and set m=0 so G(z,a,0)=0. Full lowers motion MSE by 32.45%/11.43%/9.91%. With training and the ensemble held fixed, this measures the benefit of using memory at inference.

The full system runs two predictors per step. The memory interface adds about 0.109M parameters and writes a 64-value state once per query, which is then reused throughout prediction.

W 2: Shuffle retains 94.1% of the gain. Repeat Table 4 on the final ensemble and test within-prefix action-response mismatches.

Thank you for helping us identify this mismatch in how the ablations were evaluated. Table 4 tests a frozen-base response model on validation data. The main results use the final two-branch system, including additional joint training on RoboMimic. The 94.1% figure describes the Table 4 model. The table above reports these controls on the final models.

A shuffled state can still carry useful response information. It comes from another real episode of the same task, and the reader combines it with current visuals and actions. Shared response patterns can therefore contribute to the gain retained after shuffling.

In the final models, shuffle retains only 14.81%/7.73%/46.19% of the gain over zero under the same formula, (E_zero-E_shuffle)/(E_zero-E_full). Full lowers motion error by 29.04%/10.64%/5.59% compared with shuffle, in RoboMimic/LIBERO/LanguageTable order. Correct memory gives more than twice the improvement over zero that shuffled memory gives on every dataset.

The action-only writer receives historical actions without visual responses, while the predictor keeps its usual inputs. Full lowers motion error by 26.26%/6.10%/8.18% against this control, showing the value of the observed responses.

We also tested pairing by cyclically shifting only the four historical writer actions by 1, 2 or 3 positions. Responses, predictor actions, weights and λ stay fixed. This preserves the action and response sets while changing their pairings. The mean-pooled writer is invariant to reordering complete pairs, so shifting actions alone tests the correspondence within each pair. Full's motion-error reductions are:

DatasetShift 1Shift 2Shift 3
RoboMimic22.78%36.00%27.27%
LIBERO6.83%8.46%5.91%
LanguageTable3.85%4.07%3.17%

Correct action-response pairing improves prediction in all nine comparisons.

AuthorsOpenReview ↗

Response to Reviewer ifKr (2/2): Persistent Memory, Longer Histories, and Longer Forecasts

W 3: “Long-term memory” mainly concerns prediction beyond rolling context, while H20 spans only 1/1/2 seconds and writing uses four completed transitions.

Our memory keeps observed response information available after its source frames leave the visual window.

E3D-LSTM (ICLR 2019, Sec. 3.2) processes local motion and uses historical memory to capture longer temporal relations. SimVP (CVPR 2022, Table 8) studies 10-to-20/40-frame forecasts as long-term video prediction.

In our model, C=7 is the visual window, K=4 is the number of completed action-response transitions used to write memory, and H=20 is the forecast length. H20 corresponds to 1/1/2 seconds on RoboMimic/LIBERO/LanguageTable. The writer summarizes the observed transitions in a 64-dimensional state m and keeps it throughout the rollout.

From prediction 8 onward, all seven visual-context frames are generated, while m still stores the observed response information. The response branch predicts ph=Fϕ(Xh,Ah)+G(zh,ah,m)p_h=F_\phi(X_h,A_h)+G(z_h,a_h,m), where the reader adds a memory-conditioned adjustment to the base prediction:

G(zh,ah,m)=βtanh⁡(Rz(m)zh+Ra(m)ah+bm(m)).G(z_h,a_h,m)=\beta\tanh(R_z(m)z_h+R_a(m)a_h+b_m(m)).

The same memory can produce different adjustments as the scene and action change. The memory-dependent coefficients R_z(m) and R_a(m) determine how current visual features and actions affect each predicted transition. Setting m=0 gives G(z,a,0)=0, which is the zero-memory control used in W1/W2.

K4 therefore controls how much history we write, while the memory keeps those observed responses available to guide later predictions beyond the rolling visual window.

We used models trained with the paper's final method and settings and kept their weights fixed during these additional experiments. Both tables report Motion MSE (x10^-3, lower is better).

Longer histories. K counts completed action-response events. The paper's K≤C−1 bound applies when writing within the C-frame context. Here we extend the writer's input to earlier observed events while keeping the same seven-frame predictor input, H20 targets and 64-dimensional memory fixed. We tested K=4/8/12/20 on the same windows from the paper's test sets, all with enough observed history for K20.

Dataset (episodes)K=4K=8K=12K=20
RoboMimic (33)4.46934.24234.24404.3231
LIBERO (40)17.128816.577916.394617.2073
LanguageTable (26)12.154111.943812.017612.1091

K4-K20 covers 0.2-1.0 seconds of history on RoboMimic/LIBERO and 0.4-2.0 seconds on LanguageTable. The best K values are 8/12/8 on RoboMimic/LIBERO/LanguageTable, lowering error over K4 by 5.08%/4.29%/1.73%. These gains come from giving the K4-trained models more observed history at evaluation, while keeping the visual window and forecast length fixed.

Longer predictions. We tested H40/H60 with K4, C7 and model weights fixed. JEPA is the companion used in our system. Zero memory uses the same two branches and mixing weight, with the explicit memory state set to zero. Each row reports the average error over steps 1-H on matching queries. Each horizon uses trajectories long enough to support it. RoboMimic reaches H40, and LanguageTable H60 includes 10 episodes.

DatasetH (seconds)JEPAZero memoryFull
RoboMimic40 (2)6.99787.93805.2627
LIBERO40 (2)22.586023.338419.7060
LIBERO60 (3)28.153527.807322.1145
LanguageTable40 (4)16.706617.091214.6480
LanguageTable60 (6)22.518323.195719.6730

Full outperforms both controls in all five extended settings. At H60, error falls 21.45%/20.47% versus JEPA/zero memory on LIBERO (3 s), and 12.64%/15.19% on LanguageTable (6 s). Full therefore maintains better average prediction quality over rollouts of up to 60 steps, with the same seven-frame visual window and four-event write window.

Thank you again for your helpful feedback. We would be happy to discuss any further questions : )

Reviewer_YWebOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an action-conditioned latent world model on frozen DINOv2 tokens with persistent action–response memory. On RoboMimic Lift, four LIBERO tasks and LanguageTable, authors report best numbers against seven baselines with Holm-corrected paired tests, and also state-intervention ablations on RoboMimic validation.

Critical strengths
  • The proposed design is light weight, which can be added as a small residual on top of an existing predictor.
  • Results are consistent for all sources and metrics.
Critical weaknesses
  • Problem assumption: The paper starts from the premise that action–response evidence is lost when observed frames leave the rolling context (Sec. 1, Sec. 3). But there is no experiment that show existing models actually suffer from this.
  • Demostration: The main claim (persistent action–response memory is a useful interface) is not directly tested by the main results.
  • Isolated memory: In Table 4, learned static state is only 3.2% behind correct state in motion MSE, and shuffled state keep 94.1% of the gain over zero. Paper calls this "a large shared correction". For the main claim (action–response evidence is what is retained), this effect is just modest.
  • Long-horizon world model is evaluated only as short video prediction: The setting is 64×64 frames, 20 steps (about 1-2 s), and pixel metrics only. Considering that a world model's value is usually shown by downstream use like planning, policy evaluation, or at least longer rollouts, the current significance for the world-model community is unclear. Either temper the "long-horizon world model" framing or add other experiments like planning to validate.
Questions for authors

Refer to weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer YWeb (1/2): Evidence and Memory Controls

Thank you for the careful review and for recognizing our method's lightweight design and consistent prediction gains.

MSE values are x10^-3. PSNR is in dB.

W 1: Evidence that existing models lose useful historical information.

Earlier studies show that retaining past observations helps a range of world models. Four examples cover visual prediction, recall and robot interaction.

StudyHistory testReported evidence
PhysiCalWorld, 2026 preprint, Tables 1-2DINO-WM vs DINO-WM + raw interaction prefixBeerPong/FetchSlide PSNR: 27.35/25.67 → 29.10/27.14. Planning success: 44.0%/26.7% → 82.7%/54.7%
Deng et al., NeurIPS 2023, Table 5TSSM-XL cache 128 → 256 → 512, Four RoomsGeneration MSE: 224.4 → 124.4 → 38.4. Reconstruction MSE stays 2.4
CoME, ICLR 2026, App. D.1Standard diffusion vs memory experts on concealed Memory CardsRecall accuracy: 13% → 79.2%
Ctrl-World, ICLR 2026, Table 2Action-conditioned robot prediction without vs with history memoryThird-person/wrist PSNR: 23.06/18.84 → 23.56/19.18

Together, these studies show that historical evidence can improve prediction, recall and planning across different model designs and tasks. The prefix and memory variants add history inputs or training under each paper's setup.

We also tested whether observed responses still help our final model after they leave the visual window. We used a model trained with the paper's final method and settings (C7/K4/H20). For predictions 1-7, all conditions used the correct memory. Before prediction 8, we kept the correct state, set it to zero, removed observed responses while retaining past actions (action-only), or used another same-task episode's state. We kept both branches, model weights, mixing coefficient, decoder and future actions fixed. The RoboMimic scores below average predictions 11-20.

State from step 8Pixel MSE ↓Motion MSE ↓PSNR ↑SSIM ↑
Full1.18245.286229.78310.94522
Zero2.21959.477726.69660.88385
Action-only1.81218.445327.60270.92491
Same-task shuffle1.85478.843227.48430.92278

Full has the best scores on all four metrics. Compared with zero/action-only/shuffle, motion error falls by 44.22%/37.41%/40.22% and PSNR rises by 3.09/2.18/2.30 dB. The observed responses and their match to the current episode still help the model make later predictions.

The second table in W4 also shows that PARM-WM achieves lower average motion error than JEPA in all five H40/H60 settings, further supporting its advantage over longer forecasts.

W 2: Direct evidence for the memory interface.

The main tables show the overall prediction gains, and Table 4 examines the memory content. We added memory controls to the final models on all three datasets, using the paper's H20/C7/K4 settings.

For zero/action-only/shuffle, we keep both trained branches, all weights, decoder and mixing coefficient fixed. Zero sets the memory state to zero. Action-only keeps past actions and removes observed responses. Shuffle uses a state from another episode of the same task.

For static, we use one learnable 64-D state shared across episodes, with the same final architecture, companion and mixing coefficient. We train this state and its reader for 512 updates at LR=10^-3, then fix both before testing.

Dataset/statePixel MSE ↓Motion MSE ↓PSNR ↑SSIM ↑
RoboMimic / Full0.95364.449830.75480.95169
RoboMimic / zero1.47986.587728.37730.92077
RoboMimic / action-only1.26956.034129.46010.94153
RoboMimic / shuffle1.29746.271028.92200.94024
RoboMimic / static1.88998.746927.75940.91536
LIBERO / Full3.802816.779625.17070.83218
LIBERO / zero4.113218.945224.06730.81773
LIBERO / action-only4.060717.870124.71110.82118
LIBERO / shuffle4.101518.777724.17140.81220
LIBERO / static4.220918.839424.70680.81479
LanguageTable / Full2.911412.988526.05330.88715
LanguageTable / zero3.464014.417825.75080.87164
LanguageTable / action-only3.153514.145025.77890.87205
LanguageTable / shuffle3.255813.757625.93150.88068
LanguageTable / static3.479414.985225.41070.87535

Full is best on all four metrics against every control on each dataset. Motion error is lower by 32.45%/11.43%/9.91% compared with zero, and 26.26%/6.10%/8.18% compared with action-only, in RoboMimic/LIBERO/LanguageTable order. These ablations demonstrate the value of the memory interface: retaining and using observed action-response information improves the final model's predictions on all three datasets.

AuthorsOpenReview ↗

Response to Reviewer YWeb (2/2): Ablation Interpretation and Downstream Evaluation

W 3: The small static-state gap (3.2%) and high retained gain after shuffling (94.1%).

Thank you for raising this point. It helped us identify a difference between the ablation and the main evaluation. Table 4 uses a frozen-base response model. The main results combine two separately trained branches. On RoboMimic, the response base and memory interface are trained jointly. The reported 3.2% and 94.1% describe the Table 4 setting. We have now repeated these controls in the final model on all three datasets.

The static state is shared across episodes. Its trained reader also receives the current visual features and action, so the correction can vary from step to step. The 3.2% gap is the extra benefit of observed history over the correction that this reader has already learned. In the final model, the static control keeps the same architecture, companion and mixing coefficient, with its state and reader trained before testing. Full lowers motion error by 49.13%/10.93%/13.32% compared with this static control, in RoboMimic/LIBERO/LanguageTable order.

Shuffling supplies a real response state from another episode of the same task. The reader can still use information shared across episodes. The ratio (E_zero-E_shuffle)/(E_zero-E_full) measures how much prediction improvement remains with shuffled memory. A shared correction can contribute to that 94.1%. Table 4 also shows 12.74% lower motion error than action-only, supporting the value of the observed responses. In the final model, shuffle retains only 14.81%/7.73%/46.19% under the same formula. Full lowers motion error by 29.04%/10.64%/5.59% compared with shuffle, in the same dataset order.

All four metrics favor Full over both controls on every dataset. Correct memory gives more than twice the improvement over zero that shuffled memory gives. Together, these results show that the final predictor benefits from observed responses beyond a learned shared correction, and from matching those responses to the current episode.

W 4: Downstream use and prediction beyond H20.

We added closed-loop robot planning and longer forecasts at H40/H60. Both use models trained with the paper's final method and training settings, with weights fixed during evaluation.

We evaluated four planners on the same 40 held-out Lift episodes in simulation, with the same initial states, goal images and six prefix actions. Model-based planners sampled 16 candidate action sequences from training trajectories and predicted 20 steps for each. They ranked candidates by the predicted endpoint's distance to the goal image, executed five actions and planned again from new observations. All methods shared the proposal budget and 100-action limit. Success follows the environment's cube-lifting criterion.

PlannerSuccessActions* ↓Goal error ↓
PARM-WM25/40 (62.5%)70.380.7346
Same ensemble, zero memory23/40 (57.5%)73.620.7999
JEPA21/40 (52.5%)75.000.8329
Retrieval only16/40 (40.0%)78.001.0970

*Actions are averaged across episodes, with failures counted as 100. Goal error is the DINO feature distance between the final observed image and the goal.

PARM-WM reaches 62.5% success, up 10.0/22.5 percentage points over JEPA/retrieval. It also lowers goal error by 11.80%/33.04% and uses 6.16%/9.77% fewer actions. Compared with the same ensemble using zero memory, success increases by 5.0 points and goal error falls by 8.16%. The predictions help the robot complete the task more often and finish closer to its goal.

For longer forecasts, SimVP (CVPR 2022, Sec. 4/Table 8) evaluates 20/40-frame forecasts as long-term video prediction. We extended our evaluation to H40/H60 with K4 fixed. Motion MSE below is averaged over steps 1-H, using the same queries for all methods within each row. We used trajectories long enough for each horizon. RoboMimic supports H40, and LanguageTable H60 includes 10 episodes.

DatasetHJEPAZero memoryPARM-WM
RoboMimic406.99787.93805.2627
LIBERO4022.586023.338419.7060
LIBERO6028.153527.807322.1145
LanguageTable4016.706617.091214.6480
LanguageTable6022.518323.195719.6730

PARM-WM performs best in all five extended settings. At H60, motion error is lower by 21.45%/20.47% than JEPA/zero on LIBERO (3 s), and 12.64%/15.19% on LanguageTable (6 s). The relative gains over JEPA and zero memory are larger at H40/H60 than at H20 in these evaluations.

Thank you again for your helpful feedback. We would be happy to discuss any further questions : )

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.