- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Physis-Embed, a unified VLA embedding model for retrieving recorded future interaction trajectories based on task intent and execution stage, and is trained on four source datasets. It outperforms general-purpose multimodal embedding on the proposed Physis-Embed-Bench. The method shows improvements on two applications: Embed2ICL in robot policy conditioning, in simulation and on real robots, and Embed2Evolve in experience-driven memory evolution, in simulation.
Critical strengths
The paper covers multiple datasets, metrics, training-free and training-based ICL approaches. It also includes modality ablations, random-reference controls, matched training budgets, and real-robot experiments.
The proposed Predictive retrieval of task- and stage-relevant experience is useful. On the benchmark, the matched DROID ablations show that native actions substantially improve retrieval.
Critical weaknesses
The benchmark retrieves the exact future clip immediately following each query state, hence theoretically it may reward trajectory continuity rather than transferable execution-stage similarity. Moreover, Physis-Embed is trained in-domain with native actions, whereas the headline baselines are zero-shot and action-free. The evaluation lacks quantitative cross-episode stage-matching results and comparably trained multimodal baselines across all four sources.
Some concerns on evidence for applications. DreamZero reuses one selected reference per task, and LIBERO evaluates only five pilot-selected tasks, with a modest retrieved-versus-random improvement (92.8% vs. 88.0%). Real-robot results involve only 12 trials per condition. Embed2Evolve independently evolves memory banks and context configurations for each embedding model, making it hard to isolate the contribution from the representation.
Questions for authors
Can the authors evaluate cross-episode retrieval at matched execution stages, excluding immediate temporal continuity, and compare against a vision–language embedder trained on the same data and preprocessing pipeline?
For Embed2ICL, how much of the improvement comes from retrieval rather than GPT reranking or additional policy adaptation? Can the authors clarify the LIBERO task-selection protocol and provide statistical uncertainty for the retrieved-versus-random comparison?
For Embed2Evolve, can the authors compare embedding models using a fixed memory bank and context recipe, including random retrieval with identical RGB, action, and feedback fields? How sensitive are the results to the out-of-distribution ManiSkill action encoding?
2 comments
Q2 / W2. Embed2ICL
Retrieval, reranking, and policy adaptation. Tables 8 and 10 separate the three.
- Retrieval alone. Physis-Embed Top-1, without the Reranking Agent, reaches 50.3%, against 36.0% for random references and 43.3% for Omni-Embed-Nemotron-3B Top-1.
- Reranking. The Reranking Agent adds 6.0 points (56.3%). With the same agent, Omni-Embed-Nemotron-3B reaches 51.0%, so the embedding still accounts for 5.3 points.
- Policy adaptation. On LIBERO, the retrieved and random conditions use the same fine-tuned checkpoint and the same initial states, and differ only in the Context Tokens: 92.8% versus 88.0%. Training the policy for the same 21,700 updates without ICL stays at 86.4%.
LIBERO task selection. The pilot ran the official Fast-WAM release checkpoint on all 40 tasks of the four LIBERO suites, with 50 trials per task. We kept every task on which it succeeded at most 46 times. This gave five tasks, which were fixed before any ICL experiment.
One reference per task. DreamZero's interface warms the visual cache once, so one reference per task is by design. Retrieval is repeated during execution in Embed2Evolve, every one to three policy decisions (Appendix D.1).
Real-robot trials. The real-robot evaluation follows the protocol of CAIP (Sharma et al., 2026, "Contrastive Action-Image Pre-training for Visuomotor Control", arXiv:2606.17256): 12 rollouts per task and condition, with partial credit for completed sub-stages.
Q3. Embed2Evolve
Fixed memory bank and context recipe (new). We froze the Embed2Evolve harness, including its memory bank and context recipe, and changed only how memory is retrieved, on the same 20 test initializations per task:
| Retrieval in the fixed harness | Success (%) |
|---|---|
| Physis-Embed | 68 |
| Physis-Embed, action withheld from the query | 58 |
| Omni-Embed-Nemotron-3B | 56 |
| Random retrieval with the same harness | 48 |
With the memory and recipe fixed, Physis-Embed is 12 points above Omni-Embed-Nemotron-3B and 20 points above random retrieval. The harness that Omni-Embed-Nemotron-3B evolves for itself reaches 33% (Table 12), so a better embedding also evolves a better harness.
Sensitivity to the out-of-distribution action encoding. ManiSkill actions are encoded with the LIBERO action codec, which is out of distribution (Appendix D.1). The action input still carries signal: withholding it from the query lowers success from 68% to 58%.
We thank the reviewer again for the careful and constructive review.
We thank the reviewer for the positive assessment and the constructive questions. We are glad that the reviewer finds predictive retrieval of task- and stage-relevant experience useful and that the matched ablations show the contribution of native action. This is the core idea of the paper. The questions led us to run several additional analyses, which we report below.
Q1 / W1. Cross-episode stage matching and a comparably trained baseline
A comparably trained VL embedder on all four sources. We retrained the model on the full four-source corpus without one modality at a time, with the same backbone, data, preprocessing, and budget. The variant without action is the vision–language embedder the reviewer asks for. On Physis-Embed-Bench it reaches 26.10% clip R@1 and 52.63% episode R@1, against 47.33% and 81.77% for the full model. Clip R@1 falls on every source: EgoDex 36.70 to 27.20, DROID 74.10 to 54.10, LIBERO 38.50 to 12.00, and RoboMIND-FD 40.00 to 11.10. This matched VL model is close to the best external VL embeddings in Table 1 (24.5% and 25.2%), so in-domain training alone does not explain the margin in Table 1.
Cross-episode retrieval at matched stages. We ran the evaluation the reviewer suggests with the full model and these retrained variants. We use DROID, the only source that provides task-stage metadata, which lets us build approximate stage labels without new human labeling.
AUC is the probability that a positive is ranked above a negative, so 0.5 is chance. The full model retrieves stage-matched clips from other episodes well above chance and has the highest Hit@5, Hit@10, and AUC. The matched VL embedder is at chance. In this test, cross-episode stage matching depends on action. The stage labels are approximate: they come from DROID's existing task-stage metadata and action signals, not from new human labeling of semantic stages.
Continuity in the paired-clip benchmark. The reviewer is right that the query and its positive come from the same episode. Half of the gallery consists of nearby clips from that same episode (Appendix B.1). They share the same continuity and differ in stage, so continuity alone does not identify the positive.