- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper targets looped (repetitive) manipulation tasks, where a vision-language-action (VLA) policy must repeat a subtask a prescribed number of times and then stop. The authors propose LoopVLA, a cross-subtask event-memory interface built on top of π0.5. At each subgoal transition, learnable event latents attend to the tokens of the just-closed subtask segment and compress it into an event memory. On the action side, action features separately read the perceptual history and the joint history–event context; the two readouts are residually fused and injected into the action head through a FiLM-like MemoryRMS module. Training combines a flow-matching action objective with a next-subgoal cosine-alignment loss. Evaluation uses 16 RoboMME tasks, with emphasis on the repetition group (PickXtimes, SwingXtimes, StopCube): under an online Qwen-VL subgoal protocol the method reaches 80.22% on the repetition group versus 22.67% for a memory-off configuration, and 47.17% across all 16 tasks versus 44.51% for the strongest non-SG RoboMME variant. Three additional real-robot tasks are reported with 39/60 successes versus 21/60 for π0.5.
Critical strengths
- The problem is well chosen and under-explored. Historically conditioned looped execution and the associated stopping decision (continue / repeat / stop) receive little dedicated treatment in the VLA memory literature, and the online Qwen-VL subgoal protocol is a genuine step toward realistic deployment.
- The division of labor between event triggering and memory weaving is plausible a priori, and the repetition group improves substantially over the memory-off configuration (80.22% vs 22.67%), so the general interface shape of "where to inject information about a completed segment" is informative for follow-up work.
Critical weaknesses
W1. Attribution of the gain is confounded, and the controlling baselines are missing. LoopVLA introduces two things that no baseline receives: the Qwen-VL subgoal labels consumed by both the policy and the event trigger, and 64 additional visual tokens (32 history observations plus 32 event latents); every baseline in Table 1 is evaluated without subgoal input, and the RoboMME SG variant is not reported. Two controls decide whether the claim holds: π0.5 with the same Qwen-VL subgoal labels but without the LoopVLA memory interface, and direct concatenation of raw frames from the closed segment at an identical token budget without learnable latents. The internal ablations already point away from the paper's causal story: raising H from 10 to 32 takes the repetition group from 60.00% to 81.33%, whereas event-aligned windows improve over fixed recent windows only from 46.38% to 49.25% across all tasks. Most of the measured gain therefore comes from additional history tokens rather than from the event memory structure, and the causal claim in the abstract and conclusion that memory of the completed subtask supports repeated execution is not supported.
W2. No identifiable mechanism, and no qualitative analysis whatsoever. Because the task is fundamentally a counting problem, the most direct falsifiable test is whether the event latents encode a decodable repetition count; the authors explicitly disclaim this test while supplying no substitute evidence. There is no similarity matrix or dimensionality-reduction plot of event memories, no visualization of Weaver attention over segment tokens, no side-by-side rollout sequences for successful and failed episodes, and no breakdown of failures into over-counting, under-counting, and premature stopping. Not one of the five figures shows a qualitative result, and the real-robot study (Section 4.7) reports six numbers with no image of the platform, the scene, or a single executed trajectory. A reader therefore cannot distinguish whether the 22.67% to 80.22% improvement reflects count information carried by the memory or merely the added parameters and subgoal supervision.
W3. The evaluation is neither comparison-fair nor statistically convincing. Baselines are quoted with the aggregation protocols of their own papers while LoopVLA is averaged over three evaluation seeds, and the authors themselves describe these comparisons as descriptive, so the headline 47.17% versus 44.51% (+2.66 points, baselines carrying no error bars) cannot support a claim of improvement. Hyperparameters (H=32, M=32) are selected on the same episodes and the same seed (Oracle, seed 7) used for reporting, and Table 2, Figure 3 and Figure 4 are single-seed without error bars, leaving the 49.25 versus 46.75 gap possibly inside seed noise. On non-repetition memory tasks LoopVLA trails MemER badly (VideoUnmask 43.33 vs 81.33, ButtonUnmask 26.67 vs 72.00, PickHighlight 24.00 vs 70.67) without any comment, so "overall success improves" conceals very large per-task variance, and the real-robot study uses only three self-designed tasks with 20 trials each against a single π0.5 baseline and no memory-based baseline, which makes 11/20 versus 5/20 weak evidence at that sample size.
W4. Presentation is below the standard. There are no qualitative figures anywhere in the paper: no rollout frame sequences, no visualization of Weaver attention over segment tokens, no illustration of what the event memory actually encodes, and even the real-robot study (Section 4.7) offers no image of the platform or the scene, so no intuition for how the method behaves during execution is ever conveyed. Figure 2 is barely legible at print scale, its legend glyphs are crowded, and the mapping between its modules and Eqs. (2)–(4) is never made clear. The data visualization is equally crude: Figure 3 renders twenty cells as a heatmap while reprinting the same numbers inside every cell and changes the reported unit (PP_IT%, %, AT%) across tables, single-seed results in Figures 3–5 carry no error bars, and all figures rely on a single red/gray palette with no regard for colorblind readers.
Questions for authors
- Please report the two controls that decide the attribution: (i) π0.5 with Qwen-VL subgoal labels but without the LoopVLA memory interface, and (ii) raw frames of the closed segment concatenated directly into the context at the same token budget, without learnable latents; please also provide multi-seed error bars and state whether hyperparameter selection used a held-out split.
- Do the event latents carry a decodable repetition count, for instance under a linear probe or in an event-memory similarity matrix? If not, please explain the mechanism behind the 22.67% to 80.22% improvement, and clarify whether the large regression relative to MemER on non-repetition tasks (ButtonUnmask 26.67 vs 72.00) follows from using subgoal transitions as the only trigger.
- Please report results for the RoboMME SG variant, the end-to-end latency and control frequency of each policy query including the Qwen3-VL-4B call issued every 16 actions, and the real-robot platform setup, success criterion, and comparison against at least one memory-based baseline.
4 comments
Responses to the weaknesses
Many thanks for reviewer's effort in review. We further address W1–W4 and Q1–Q3 separately.
W1 — Attribution from the history sweep
“Most of the measured gain therefore comes from additional history tokens rather than from the event memory structure”
The review compares a gain on three repetition tasks (60.00% → 81.33%) with a gain on all 16 tasks (46.38% → 49.25%). These different aggregates cannot decompose the contribution.
We have now extended the full history sweep to three evaluation seeds (7, 21, 42), with 50 episodes per task per seed. The history sweep and Event perturbation use Oracle and report mean ± sample SD; Repetition averages PickXtimes, SwingXtimes, and StopCube. With M = 32 throughout:
| History frames | All 16 (%) | Repetition (%) |
|---|---|---|
| H = 10 | 39.58 ± 0.62 | 60.44 ± 4.68 |
| H = 20 | 42.00 ± 0.22 | 70.22 ± 3.36 |
| H = 32 | 47.71 ± 1.37 | 79.78 ± 1.39 |
| H = 40 | 45.54 ± 1.06 | 76.22 ± 1.92 |
H = 32 exceeds H = 40 on both metrics in every seed. The sweep identifies a suitable history budget for this setting: adding more history can reduce performance. Comparing different task aggregates while omitting this decline does not establish that history accounts for most of the gain. The Event perturbation below directly tests whether the learned Event representation affects execution.
W2 — The proposed counting criterion
“the most direct falsifiable test is whether the event latents encode a decodable repetition count”
This conflates the task's global requirement with one module's role. Weaver stores the just-completed segment and replaces it at the next transition; it does not accumulate multiple event memories. The policy jointly reads that segment, perceptual history, and current observations. A global counting task does not require every intermediate representation to independently encode the global count. Which claimed function of this single-segment memory would the proposed cumulative-count test actually assess?
We instead test its role in control: at the same checkpoint, we negate only the 32 normalised Event-token vectors before their K/V projections (E → −E), preserving the history-input pathway, latent budget, fusion, and MemoryRMS.
| Event representation | All 16 (%) | Repetition (%) |
|---|---|---|
| Original | 47.71 ± 1.37 | 79.78 ± 1.39 |
| E → −E | 39.92 ± 1.25 | 61.33 ± 3.06 |
Repetition success decreases by 18.44 percentage points, with a decrease on every repetition task in every seed. The policy uses the learned Event representation for subsequent control. This supplies the requested functional evidence while keeping parameters and latent capacity fixed.
W3 — Evaluation and reporting
“The evaluation is neither comparison-fair nor statistically convincing.”
The main Qwen-VL results are 47.17 ± 1.42% overall / 80.22 ± 2.04% repetition, over evaluation seeds 7, 21, and 42.
We also evaluated history-only, H = 32 under Oracle, using the same tasks, evaluation seeds, and episode budget: 44.92 ± 0.58% overall / 72.89 ± 1.68% repetition, compared with 47.71 ± 1.37% / 79.78 ± 1.39% for full LoopVLA. This supplies a history-only comparison with seed variation reported for both configurations. Answer to Q2 addresses non-repetition performance; answer to Q3 addresses the robot study.
W4 — Presentation
“Presentation is below the standard.”
We will improve Figure 2's legibility and equation mapping. We respectfully ask that the presentation assessment also take account of the existing figures, error bars, and real-robot execution videos identified above.
Responses to the questions Q1 and Q2
Q1 — Controls and configuration selection
“π0.5 with Qwen-VL subgoal labels but without the LoopVLA memory interface”
“raw frames of the closed segment concatenated directly into the context at the same token budget, without learnable latents”
Q1(i). SimpleSG+QwenVL and GroundSG+QwenVL implement the requested class of direct subgoal-text input to π0.5, scoring 29.00% and 32.70% under RoboMME's published training protocol. As stated in factual error 'W1/Q1(i) — Incorrect policy-input premise', our labels instead trigger event-memory updates. The full SG table is below. RoboMME, Sections 4.2–5.2
Q1(ii). Our completed attribution controls are the history-only comparison (response to W3) and the same-checkpoint Event-feature intervention (response to W2). The latter changes only the Event features, preserves model weights and latent capacity, and reduces repetition success by 18.44 percentage points. It directly tests whether the learned Event representation influences subsequent control.
Selection and error bars. W1–W3 report the three-seed results and sample SDs.
Q2 — Decoding, improvement, and ButtonUnmask
“Do the event latents carry a decodable repetition count”
“please explain the mechanism behind the 22.67% to 80.22% improvement”
“clarify whether the large regression relative to MemER on non-repetition tasks (ButtonUnmask 26.67 vs 72.00) follows from using subgoal transitions as the only trigger.”
Count probe. We do not adopt cumulative-count decodability as the criterion for this single-segment memory. As explained under W2, the representation contains one completed segment, not an accumulated event record. The model has no count-supervision objective for event latents; a probe would require additional aligned count targets and a separate readout-training protocol. Feature similarity alone does not establish repetition count. If the concern is accumulated progress, why exclude the perceptual history with which the segment is jointly read?
Mechanism. Multiple layers of cross-attention let the action expert use the latest completed segment alongside earlier perceptual history and current observations. This conditions subsequent execution on what has happened. The 22.67% → 80.22% comparison concerns the memory-disabled result and the full Qwen-VL interface. The additional Oracle intervention in W2 directly establishes dependence on the Event representation, with repetition success falling from 79.78% to 61.33% while model parameters and latent capacity remain fixed.
ButtonUnmask. Our claim concerns repeated execution and correct stopping, as your own summary recognises. The reported ButtonUnmask gap does not contradict that claim. Transitions govern event-memory replacement, while perceptual history remains available; Weaver compresses the entire completed segment, not just its boundary frame. The scores therefore do not establish the trigger as the cause.
Responses to the questions Q3
Q3 — SG results, runtime, and real-robot evaluation
“Please report results for the RoboMME SG variant”
| Method | All 16 (%) | Repetition (%) |
|---|---|---|
| SimpleSG + QwenVL | 29.00 | 33.63 |
| GroundSG + QwenVL | 32.70 | 33.33 |
| SimpleSG + Gemini | 23.25 | 36.67 |
| GroundSG + Gemini | 11.56 | 8.33 |
| SimpleSG + Oracle | 49.58 | 81.48 |
| GroundSG + Oracle | 84.08 | 83.22 |
Repetition averages PickXtimes, SwingXtimes, and StopCube. Baseline entries are calculated from the published task means; no uncertainty for this three-task aggregate is reported (Table 15, p. 38).
RoboMME's authors themselves state that “FrameSamp+Modul achieves the highest overall success rate (44.51%)” (Dai et al., 2026, §5.2, Q1, p. 8), and describe GroundSG+Oracle as “an upper bound of symbolic memory” (§5.2, Q2, p. 9). Their Table 3 (p. 8) explicitly distinguishes the best non-oracle result. Our choice of 44.51% follows that distinction: all non-oracle SG variants are lower, with the strongest at 32.70%, so they cannot represent the best non-oracle baseline.
Oracle uses privileged simulator-state checks—for example, whether an object has reached a target—which are unavailable directly in real deployment. This states SG variants with Oracle settings' scores are not comparable with realistic deployment setting's.
“the real-robot platform setup, success criterion, and comparison against at least one memory-based baseline.”
Figure 8 and Appendix D (p. 17) document the dual-arm ALOHA setup, and the supplementary videos show real-robot execution. Section 4.7 (p. 9) specifies the task objectives: three pull–push cycles for Drawer Push–Pull, four round trips for Bowl Transfer, and three round trips followed by placement in a cup for Battery Transfer, with stopping or switching at the requested count. The results are 39/60 versus 21/60 against π0.5, with 20 trials per task. The physical study evaluates repeated execution with π0.5 as its comparator; the memory-baseline comparisons are reported on RoboMME.
Do these clarifications and additional results address your concerns? If a substantive concern about the repetition claim remains, could you identify the specific experiment needed to resolve it and the alternative explanation it would distinguish?
Factual errors requiring correction
Factual errors requiring correction
We respectfully ask you to revisit these points against the cited locations and supplementary videos. Does this evidence address your concerns about the policy inputs, memory mechanism, and presentation? If any discrepancy remains, we would appreciate the specific passage or figure so that we can address it precisely.