- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Mem4Act, a framework that learns fixed-size latent memory for history-dependent robot manipulation. It pretrains a memory encoder with multi-granularity temporal attention using historical QA and keyframe DINO feature reconstruction, then jointly adapts it with a pretrained policy through action and auxiliary supervision, conditioning action generation through feature modulation. The method is evaluated on RoboMME, RMBench, and four real-world manipulation tasks, with additional ablations and memory-utilization analyses.
Critical strengths
- The paper organizes memory for action policies around coverage, content, and utilization. Its corresponding components and experimental design consistently follow this structure, making the problem formulation and methodology easy to follow.
- The paper provides detailed analyses of the components used to construct and supervise latent memory. In particular, ablations of temporal structure and experiments that replace the historical context offer valuable insights into memory organization and the policy's reliance on historical information for control.
Critical weaknesses
- The method relies heavily on subgoal annotations and task-specific heuristics for QA and DINO supervision, including QA templates and keyframes selected at subgoal boundaries, potentially limiting scalability. Further discussion of annotation effort and automation across diverse tasks, together with an analysis of sensitivity to subgoal granularity, would clarify its practical scope.
- The comparison of QA and DINO supervision in Table 5(a) supports their complementary benefits but largely relies on success rates. Concrete success and failure cases across the supervision variants, showing which semantic facts or visual and spatial details are retained and used, would better support their intended roles.
Questions for authors
- How well does Mem4Act extend to other VLA backbones, and can broader Stage 1 pretraining be reused across target tasks? Such experiments could further strengthen the framework's practical value.
- Could the authors provide matched rollout examples with the corresponding historical frames, identifying the cues needed at failure points and assessing whether FrameSamp and Mem4Act retain them? This would help distinguish information lost during sampling or compression from ineffective use of retained context.
6 comments
Response to Reviewer odnQ
3. Other VLA backbones (Question 1)
We thank the reviewer for this question. To test whether Mem4Act depends on a specific backbone, we applied it to GR00T N1.7 using the same memory architecture, QA and DINO objectives, two-stage training schedule, and conditioning through cross-attention and feature modulation. Historical observations are encoded by GR00T's frozen vision encoder and pooled to 16 tokens per frame, matching our π0.5 setup. For each backbone, the base policy and Mem4Act receive the same number of training steps and are evaluated on all 16 RoboMME tasks with 50 episodes per task.
| VLA backbone | Base policy | With Mem4Act |
|---|---|---|
| GR00T N1.7 | 13.50% | 35.50% |
| π0.5 | 17.93% | 49.88% |
Mem4Act raises success from 13.50% to 35.50% on GR00T N1.7 and from 17.93% to 49.88% on π0.5. The relative gains are similar, at 2.6x and 2.8x of the base success rate. These consistent gains support its applicability as a memory module across both tested VLA backbones. We added this comparison to Supplementary Table S5, with discussion in Supplementary Sec. D.1 (Mem4Act on Other VLA Backbones).
4. Reusable Stage 1 pretraining (Question 1)
To test whether broader Stage 1 pretraining can be reused, we pretrained a shared memory latent on RoboMME and RMBench together, then fine-tuned on RoboMME only during Stage 2.
| Stage 1 pretraining | RoboMME |
|---|---|
| Per-benchmark (RoboMME) | 49.88 |
| Shared across RoboMME + RMBench | 46.75 |
The shared checkpoint achieves 46.75%, compared with 49.88% for RoboMME-specific pretraining, which supports reuse but with a modest reduction of 3.13 points. It also remains 2.24 points above the strongest baseline, MME-VLA (FrameSamp+Modul, 44.51%).
We attribute this gap mainly to benchmark heterogeneity. RoboMME uses a single arm and a third-person memory view with left and right mirrored relative to the robot. RMBench uses dual arms and a head camera without this mirroring. As a result, the same spatial description can refer to opposite sides of the image in the two memory views. Tasks and visual appearance also differ substantially. Combining only these two distinct sources may introduce conflicting supervision, limiting the benefits of shared pretraining.
We added this comparison to Supplementary Table S6, with discussion in Supplementary Sec. D.2 (Reusing Stage 1 Pretraining Across Benchmarks). We expect larger and more diverse pretraining data to improve generalization, which we also highlight as future work in the revised paper.
5. Information lost during sampling vs. ineffective use (Question 2)
We thank the reviewer for this helpful suggestion. We added two matched rollout comparisons on RoboMME, shown in Figs. S2 and S3 and discussed in Supplementary Sec. D.8 (RoboMME Failure Analysis: MME-VLA and Mem4Act). Each shows all historical frames selected by FrameSamp and supplied to Mem4Act's memory encoder at the decision point, with the critical cues highlighted. Both methods provide 512 memory tokens to the policy.
Information lost during sampling (Fig. S2). The robot must place a cube on the target that was visited third in the demonstration, but two targets later swap positions. To find the right target, the policy must see which targets moved during the swap. The episode lasts over 1,200 frames, so FrameSamp's 32 frames are about 40 frames apart, and only one lands inside the 49-frame swap. That frame shows the targets mid-swap but not where each one started or ended, so the policy picks the wrong target. Mem4Act samples every 15 frames, sees the swap from start to finish, knows the third target never moved, and succeeds.
Failure despite retained cues (Fig. S3). The robot must identify the container hiding a green cube after a swap. FrameSamp retains more observations of the uncovered cubes (six versus three) and the swap (eight versus three), yet fails while Mem4Act succeeds. Both sampled histories contain the necessary cues, suggesting that FrameSamp fails to interpret or use the available context effectively.
Together, the two cases separate the two failure sources: in the first, the needed cue is lost during sampling, and in the second, the cue is retained but not used effectively.
Thank you for the detailed response and the extensive additional experiments, which address most of my concerns (W1, W2, Q1, and the matched examples in Q2). I have one remaining question on Q2.
[Q2] The new matched examples show which cues appear in the frames sampled by each method, but both are Mem4Act successes and neither shows which cues remain accessible in its 512 latent tokens. For one or two rollouts on a counting task such as StopCube in which Mem4Act acts on the wrong count, could the authors show whether the relevant events are visible in the sampled history and whether the existing QA decoder returns the correct count at the decision point? This would help narrow down where the cue may be lost.
Overall, the rebuttal substantially strengthens the paper, and I would be glad to maintain my positive assessment once this point is also addressed.
Response to Reviewer odnQ
Thank you very much for your detailed and constructive review. We sincerely appreciate your recognition of the coverage, content, and utilization framing and of our analyses of memory structure and memory use. Below, we respond to each point.
1. Annotation effort, task-specific heuristics, and subgoal granularity (Weakness 1)
We thank the reviewer for raising this concern. We agree that Mem4Act's auxiliary supervision relies on subgoal annotations during training.
Annotation effort. In simulation, subgoal labels come at almost no cost, since scripted demonstration pipelines already track each subtask and its transitions. RoboMME and RMBench both release subgoal descriptions with temporal boundaries, from which we generate all QA pairs and DINO keyframes automatically. For datasets without such labels, automatic video segmentation and captioning offer a practical way to obtain them. To understand how much labeling effort Mem4Act actually needs, we study sensitivity to subgoal granularity below.
Task-specific heuristics. Most of our questions require no task-specific judgement. Three common questions, covering what the robot has done, what it is doing now, and what comes next, are shared across all tasks and answered directly from the completed, current, and next subgoals. Task-specific design enters only through progress questions, such as how many times the cube has reached the target in StopCube. To measure how much Mem4Act relies on these, we remove all task-specific questions in model training.
Sensitivity to subgoal granularity. We ran two tests on RoboMME. First, we removed all task-specific questions, to measure how much they matter. Second, we additionally merged consecutive subgoals, to measure sensitivity to coarser subgoals. Merging two or four subgoals leaves about half or a quarter as many boundaries to annotate, and makes the QA answers and boundary keyframes correspondingly coarser.
Removing the task-specific questions lowers success by only 1.38 points, showing that the shared questions account for most of the benefit. Coarser labels have a similarly small effect, lowering success by a further 0.37 and 1.12 points when two or four subgoals are merged. Even in the most reduced setting, without task-specific questions and with four times coarser labels, Mem4Act reaches 47.38%, which is still above the strongest baseline, MME-VLA (FrameSamp+Modul, 44.51%). These results indicate that Mem4Act is robust to both the choice of questions and the granularity of subgoal labels.
We have added this ablation and its discussion to Supplementary Sec. D.5 (Table S8). We also state the annotation requirement in the Limitations and Future Work paragraph of the revised version, together with automatic annotation as a way to reduce manual supervision.
2. Evidence beyond success rates for the roles of QA and DINO (Weakness 2)
We thank the reviewer for this suggestion. We agree that success rates alone do not show what information the memory retains. We therefore added Figure 5 in the revised version, which compares the three supervision variants on the same StopCube episode and visualizes their memory readouts, giving two failure cases and one success case. Succeeding on this task requires two kinds of information: a semantic fact, the number of times the cube has reached the target, and a visual detail, the cube's motion, which determines when it arrives. We have also added the corresponding discussion to Sec 4.3 Ablation Studies (Do semantic and perceptual objectives complement each other?)
The two single-objective variants fail in opposite ways.
QA only. The memory keeps the count: after the 4th pass, the policy correctly prepares to stop the cube at its 5th arrival. However, without visual detail of the cube's motion, it misjudges the speed and presses too early, so the cube stops short of the target on its 5th approach.
Key-frame DINO only. The memory keeps the visual detail of the cube's motion, and its reconstructions closely match the DINOv3 targets. This lets the policy judge the speed and time the press so the cube stops on the target. However, without semantic progress, it loses count and stops the cube at its 4th arrival instead of the 5th.
QA and key-frame DINO. With both objectives, the policy presses at the right moment of the right pass, and the cube stops on the target at its 5th arrival.
This case makes concrete the complementary benefits in Table 5(a): QA supervision retains semantic progress, while DINO supervision retains the visual details needed for precise timing.