- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Key-Gram, an external conditional-memory module for language-conditioned robotic manipulation. An offline language-model parser decomposes instructions into short semantic phrases, which index learnable embeddings through deterministic multi-head hashing. Retrieved embeddings are injected into selected Transformer layers through context-dependent gating and convolutional fusion. The framework also incorporates future visual-latent prediction. Its slot/head table organization is intended to support memory expansion without changing the backbone architecture. Experiments cover RoboTwin2.0, LIBERO/LIBERO-Plus, and real-world dual-arm manipulation, including compositional and continual task expansion. The reported results show substantial improvements in several challenging settings.
Critical strengths
-
Substantial empirical gains across multiple challenging settings and two backbones. The reported improvements extend across simulation and real-world manipulation. On RoboTwin2.0 Hard, success increases from 58.4% to 75.6% for pi 0 and from 76.8% to 84.4% for pi 0.5 (Table 1). Transfer from LIBERO to LIBERO-Plus without target-domain fine-tuning improves from 53.6% to 72.8% and from 83.9% to 87.7%, respectively (Table 2). The real-world continual assembly results also show appreciable improvements in final-stage completion (Table 5). These results support the practical usefulness of the proposed augmentation in difficult manipulation settings.
-
The ablations support the usefulness of correctly matched key-gram conditioning. Table 6 includes a trained, parameter-matched +VAE control, shuffled trained memories, CRF-based phrases, and KG without the visual-foresight branch. Correct KG retrieval improves the four-task weighted score from 61.950 to 77.175 when visual foresight is present; without that branch, KG achieves 68.925 compared with 51.000 for the vanilla model. Shuffled memories and CRF phrases achieve lower scores of 58.825 and 68.075. Together with the shared language augmentation described in Appendix B, these controls support the usefulness of the correctly matched KG pathway and the importance of semantic phrase construction and instruction correspondence in this ablation setting.
Critical weaknesses
-
The experiments do not isolate the storage-specific contribution of the proposed memory architecture from the benefit of semantic key-gram conditioning. The trained +VAE, noVAE, CRF, and shuffled-memory controls in Table 6 provide substantive evidence for the usefulness of the complete KG pathway and correct instruction–memory correspondence. However, none holds the cached semantic key-grams fixed while replacing hashed retrieval with a conventional phrase encoder under comparable fusion and training conditions. Because the lookup-and-fusion mechanism substantially follows Engram (Cheng et al., 2026), this missing comparison leaves the added value of the proposed store unresolved. A focused comparison within the existing four-task ablation setting is needed to assess that storage-specific contribution.
-
The visual-foresight branch lacks essential training definitions, limiting reproducibility and assessment of the training conditions. Section 3.4 describes future-latent prediction, but the manuscript and appendices do not define the joint training objective, future-target sampling horizon, VAE model, latent-loss formulation, or its weight relative to action learning. The main optimization settings are also incomplete. This is a consequential component in the reported ablation: Table 6 gives a weighted score of 77.175 for the full configuration versus 68.925 without visual foresight. Without these definitions, the reader cannot reconstruct how visual supervision is coupled to action learning. The paper needs the complete objective and training configuration.
Questions for authors
Q1 : Can you provide one control on the existing four-task ablation that uses the same cached semantic key-grams, but replaces hashed retrieval with a conventional phrase encoder while keeping the insertion layers, gating/fusion, and visual-foresight configuration comparable? Please report parameter counts and the scope of parameter matching. Q2 : What are the joint training objective, future-target sampling horizon, VAE model, and relative loss weights for the visual-foresight and action pathways? Please also provide the main optimization settings and training schedule. Additional clarifications Q3: Does memory allocation change across the four stages in Table 5? Please report the per-stage allocation. If tables or slots are added, how are keys routed while preserving existing mappings, and how are the key/value projections adapted? Q4: Which Assembly task definition and success criteria produced Table 3? Does the plate-transfer task in Figure 3 use the same evaluation protocol as the tray-pulling and bread-pouring task in Appendix D? Q5: The vanilla row in Table 6 matches all eight corresponding π₀.₅ Easy/Hard values in Table 7. Was vanilla separately trained and evaluated for the ablation, and does the disjoint-data/checkpoint statement in Section 4.4 and Appendix C apply to this row?
Dear Reviewers,
Thank you for the review. The authors have responded to the points you raised, and the response period closes on 18 November (AoE). I would be grateful if you could post a brief reply on whether your concerns are resolved, partly resolved, or still open, so that the authors have a fair chance to address them. If anything still needs clarifying, please ask now while they can still respond.
Regards,
AC