- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes VEM, a WAM Architecture & System that directly keeps all historical visual information for video model while keeps only current proprioception and the observation of information comes from the video model for the action expert. The system abandoned any selectors / compactors on the history visual observation, directly uses the video model to attend all history information. VEM also support imitation learning since its history prefix can be directly appended with demonstration videos. On RoboMME, VEM achieves an average success rate of 62.5% across all 16 tasks, outperforming the strongest reported baseline (44.5%) and a fixed-budget variant (53.3%). The paper further introduces a controlled behavioral retrieval experiment, RoboNIAH, and reports strong performance on imitation tasks.
Critical strengths
1. Clear writing and well-motivated memory design
- The paper has a clear presentation. The design insight of VEM is simple and well-motivated. The idea of keeping the whole visual history and pass through only one current latent frame to the action expert is conceptually clean and architecturally reasonable (for training and inference).
2. Strong empirical results and ablations
- The improvements on RoboMME are substantial. The ablation results on the fixed-budget and full-history directly show that keeping full observations improves performance over limited context.
3. Interesting findings on demonstration-conditioned imitation
- The idea of combining demonstrations and execution observations into a unified stream is attractive. The results on InsertPeg and RouteStick indicate that visual demonstrations can be directly plug in for informative guidance.
4. A useful evaluation of long-range behavioral retrieval
- The RoboNIAH experiment is a good addition. Through varying the distance between a informative observation and the subsequent decision, and comparing with a matched key-frame-excluded experiment, we can see more targeted evidence of long-range information leveraged.
Critical weaknesses
1. Problem formulation and lack of experiments with comparable overheads
- A key reason why other methods use a memory selector/compactor is the desire to ensure their systems can run stably for a long period of time. While the cost of VEM will eventually increase (and potentially causes failures and crashes) because it continuously records and attends to all historical frames. Since their computational budgets are different, the experiment results can not directly prove that the memory design of VEM is better. I suggest that comparisons between a resource constraint VEM variant and previous works that shares a comparable overheads could be more convincible.
2. Lack of a detailed performance analysis
- The author reports the latency result on the long history inference. However, It is better to provide more details on memory usages, TOPS and output fps since the KV Cache scales both memory usage and computational budget up linearly.
3. Task-specific performance bias
- The method exhibits more powerful performance on episodic retrieval while being less powerful on some continuous state-tracking tasks. Is this a side effect resulting from attending to all historical frames? (e.g. if the model cannot distinguish the previous states from latest states well, it could use old information over new information) Could authors provide more analysis on this?
Questions for authors
-
How does VEM model the super-long historical frames that exceeds the base model's capability. (e.g. use an extrapolated positional embedding for the KV cache?)
-
Could the authors provide more explanations on why the method performs particularly well on episodic retrieval but remains weak on continuous state-tracking tasks?