acceptodds
Withdrawn from ICLR 2027

Physis-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieving context for a robot requires matching both its task and its stage of execution. We introduce Physis-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. Predictive training aligns these endpoints for physical-experience retrieval; at inference, the model retrieves historical continuations as context. We construct multi-source training data and introduce Physis-Embed-Bench, on which our model achieves 47.33% clip-level and 81.77% episode-level R@1, surpassing action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. In our retrieval studies, removing native action degrades retrieval more than removing vision. This retrieval space supports two applications. Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset. Both interfaces also improve task progress in real-robot evaluations. Embed2Evolve uses execution feedback to grow task-local memory and refine its use around a frozen GPT-6 Astra policy. Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Embeddings, Robot In-Context Learning, Robot Context Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kxT5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Physis-Embed, a unified VLA embedding model for retrieving recorded future interaction trajectories based on task intent and execution stage, and is trained on four source datasets. It outperforms general-purpose multimodal embedding on the proposed Physis-Embed-Bench. The method shows improvements on two applications: Embed2ICL in robot policy conditioning, in simulation and on real robots, and Embed2Evolve in experience-driven memory evolution, in simulation.

Critical strengths

The paper covers multiple datasets, metrics, training-free and training-based ICL approaches. It also includes modality ablations, random-reference controls, matched training budgets, and real-robot experiments.

The proposed Predictive retrieval of task- and stage-relevant experience is useful. On the benchmark, the matched DROID ablations show that native actions substantially improve retrieval.

Critical weaknesses

The benchmark retrieves the exact future clip immediately following each query state, hence theoretically it may reward trajectory continuity rather than transferable execution-stage similarity. Moreover, Physis-Embed is trained in-domain with native actions, whereas the headline baselines are zero-shot and action-free. The evaluation lacks quantitative cross-episode stage-matching results and comparably trained multimodal baselines across all four sources.

Some concerns on evidence for applications. DreamZero reuses one selected reference per task, and LIBERO evaluates only five pilot-selected tasks, with a modest retrieved-versus-random improvement (92.8% vs. 88.0%). Real-robot results involve only 12 trials per condition. Embed2Evolve independently evolves memory banks and context configurations for each embedding model, making it hard to isolate the contribution from the representation.

Questions for authors

Can the authors evaluate cross-episode retrieval at matched execution stages, excluding immediate temporal continuity, and compare against a vision–language embedder trained on the same data and preprocessing pipeline?

For Embed2ICL, how much of the improvement comes from retrieval rather than GPT reranking or additional policy adaptation? Can the authors clarify the LIBERO task-selection protocol and provide statistical uncertainty for the retrieved-versus-random comparison?

For Embed2Evolve, can the authors compare embedding models using a fixed memory bank and context recipe, including random retrieval with identical RGB, action, and feedback fields? How sensitive are the results to the out-of-distribution ManiSkill action encoding?

2 comments

We thank the reviewer for the positive assessment and the constructive questions. We are glad that the reviewer finds predictive retrieval of task- and stage-relevant experience useful and that the matched ablations show the contribution of native action. This is the core idea of the paper. The questions led us to run several additional analyses, which we report below.

Q1 / W1. Cross-episode stage matching and a comparably trained baseline

A comparably trained VL embedder on all four sources. We retrained the model on the full four-source corpus without one modality at a time, with the same backbone, data, preprocessing, and budget. The variant without action is the vision–language embedder the reviewer asks for. On Physis-Embed-Bench it reaches 26.10% clip R@1 and 52.63% episode R@1, against 47.33% and 81.77% for the full model. Clip R@1 falls on every source: EgoDex 36.70 to 27.20, DROID 74.10 to 54.10, LIBERO 38.50 to 12.00, and RoboMIND-FD 40.00 to 11.10. This matched VL model is close to the best external VL embeddings in Table 1 (24.5% and 25.2%), so in-domain training alone does not explain the margin in Table 1.

Cross-episode retrieval at matched stages. We ran the evaluation the reviewer suggests with the full model and these retrained variants. We use DROID, the only source that provides task-stage metadata, which lets us build approximate stage labels without new human labeling.

  • Gallery. For each query we keep only candidates with the same task operation, and we remove every clip from the query's own episode, clips that overlap in time in the same source video, and exact duplicates. Immediate temporal continuity is excluded.
  • Stage match. Stage labels are built from DROID's task-stage metadata and recorded actions. A candidate is positive if its future four-second signature matches the query's: the same gripper change (open, close, or stable), and at most one difference among gripper start state, gripper end state, number of gripper threshold crossings, and dominant translation direction. The other candidates are negatives.
  • Scale. 923 queries from 743 episodes have at least one positive and one negative, with a median of 206 candidates and 28 positives per query. Ranking uses cosine similarity only.
ModelHit@1Hit@5Hit@10AUC
Full VLA36.0777.8690.380.630
No language (VA)36.4175.5788.290.596
No vision (LA)33.2474.3688.490.591
No action (VL)22.1463.2683.380.500

AUC is the probability that a positive is ranked above a negative, so 0.5 is chance. The full model retrieves stage-matched clips from other episodes well above chance and has the highest Hit@5, Hit@10, and AUC. The matched VL embedder is at chance. In this test, cross-episode stage matching depends on action. The stage labels are approximate: they come from DROID's existing task-stage metadata and action signals, not from new human labeling of semantic stages.

Continuity in the paired-clip benchmark. The reviewer is right that the query and its positive come from the same episode. Half of the gallery consists of nearby clips from that same episode (Appendix B.1). They share the same continuity and differ in stage, so continuity alone does not identify the positive.

Q2 / W2. Embed2ICL

Retrieval, reranking, and policy adaptation. Tables 8 and 10 separate the three.

  • Retrieval alone. Physis-Embed Top-1, without the Reranking Agent, reaches 50.3%, against 36.0% for random references and 43.3% for Omni-Embed-Nemotron-3B Top-1.
  • Reranking. The Reranking Agent adds 6.0 points (56.3%). With the same agent, Omni-Embed-Nemotron-3B reaches 51.0%, so the embedding still accounts for 5.3 points.
  • Policy adaptation. On LIBERO, the retrieved and random conditions use the same fine-tuned checkpoint and the same initial states, and differ only in the Context Tokens: 92.8% versus 88.0%. Training the policy for the same 21,700 updates without ICL stays at 86.4%.

LIBERO task selection. The pilot ran the official Fast-WAM release checkpoint on all 40 tasks of the four LIBERO suites, with 50 trials per task. We kept every task on which it succeeded at most 46 times. This gave five tasks, which were fixed before any ICL experiment.

One reference per task. DreamZero's interface warms the visual cache once, so one reference per task is by design. Retrieval is repeated during execution in Embed2Evolve, every one to three policy decisions (Appendix D.1).

Real-robot trials. The real-robot evaluation follows the protocol of CAIP (Sharma et al., 2026, "Contrastive Action-Image Pre-training for Visuomotor Control", arXiv:2606.17256): 12 rollouts per task and condition, with partial credit for completed sub-stages.

Q3. Embed2Evolve

Fixed memory bank and context recipe (new). We froze the Embed2Evolve harness, including its memory bank and context recipe, and changed only how memory is retrieved, on the same 20 test initializations per task:

Retrieval in the fixed harnessSuccess (%)
Physis-Embed68
Physis-Embed, action withheld from the query58
Omni-Embed-Nemotron-3B56
Random retrieval with the same harness48

With the memory and recipe fixed, Physis-Embed is 12 points above Omni-Embed-Nemotron-3B and 20 points above random retrieval. The harness that Omni-Embed-Nemotron-3B evolves for itself reaches 33% (Table 12), so a better embedding also evolves a better harness.

Sensitivity to the out-of-distribution action encoding. ManiSkill actions are encoded with the LIBERO action codec, which is out of distribution (Appendix D.1). The action input still carries signal: withholding it from the query lowers success from 68% to 58%.

We thank the reviewer again for the careful and constructive review.

Reviewer_b4aGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Physis-Embed jointly encodes vision, language, and native actions to retrieve historical continuations from a current state. Built on Cosmos3-Edge 4B, it uses source-specific action projectors and combines predictive contrastive learning, modality-discrimination constraints, and representation regularization. Two applications connect retrieval to control: Embed2ICL conditions policies through reference videos or context tokens, while Embed2Evolve accumulates task-local memory and searches context configurations around a frozen policy.

Critical strengths

Strengths

  1. The paper evaluates whether retrieval helps control rather than relying only on retrieval scores. Random-reference controls, context replacement at fixed policy weights, and matched-update training without context strengthen the downstream evidence.
  2. The matched retraining study in Table 6 supports the contribution of actions. The paper distinguishes retraining from inference-time masking and documents data splits, gallery construction, and the evolution procedure.
Critical weaknesses
  1. The benchmark primarily measures recovery of a specified continuation from its original episode, which differs from retrieving useful control experience. Section 3.3 and Appendix B.1 assign one exact paired clip as the clip-level positive, even when another episode might provide a reasonable continuation. Episode-level retrieval accepts any indexed clip from the same episode without requiring stage compatibility. Same-instruction galleries and downstream experiments partially address this limitation, but do not establish how strongly exact-window matching predicts control utility. The metrics therefore support recorded-clip identification and temporal localization more directly than general experience suitability.
  2. The main comparison in Table 1 combines differences in retrieval training, input modalities, and backbone. Physis-Embed is trained on approximately 1.7 million current–future pairs, whereas external general-purpose embeddings are evaluated through their existing interfaces without action inputs. Matched DROID retraining in Table 6 isolates an action benefit, but this control does not cover the complete four-source benchmark. The results establish an advantage for the complete trained system over these available embeddings, rather than attributing the full margin to unified VLA representation or action input.
  3. Embed2ICL provides limited direct validation of execution-stage matching. Appendix C.1 uses one fixed DreamZero reference per task across all evaluation trials, and the Top-1 conditions recut a successful stage from the retrieved episode. Appendix C.4 retains the first retrieved Fast-WAM reference until the rollout ends. These experiments support selecting useful task demonstrations, but do not directly test the benefit of rematching continuations as execution progresses. The ICL evidence is therefore more directly about task-level reference selection than the task-and-stage matching emphasized in the motivation.
  4. Embed2Evolve compares independently evolved systems rather than isolating the retriever’s immediate contribution. Appendices D.2 and D.4 allow the embedding models to produce different memory banks, refresh schedules, and history lengths. This reasonably evaluates their overall contribution through evolution, but the 68% versus 33% gap also incorporates memory accumulation and configuration selection. It is not a fixed-memory retrieval comparison. The action-free baseline additionally differs in training background, so the experiment does not isolate action-conditioned retrieval within evolution.
  5. The efficiency claims need to retain their statistical scope. Execution-time and decision-count comparisons in Appendix D.5 use only 19 shared-success pairs, including one for PullCubeTool and two for StackCube, while weighting tasks equally. Token accounting covers all test rollouts and is better supported, but Direct retains full history whereas the evolved systems use selected bounded histories. Savings therefore also reflect context management. Appendix D.6 explicitly excludes development consumption, so the reported results establish test-time costs of the selected systems rather than lower total costs after memory construction and configuration search.
Questions for authors
  1. Is there existing analysis connecting exact-clip retrieval rank to downstream reference utility? How should valid continuations from other episodes be interpreted under the current evaluation?
  2. Is a matched-data, matched-budget action-free model available for the four-source benchmark? If the controlled result is limited to DROID, should the action-contribution claim be scoped accordingly?
  3. Which query selects each fixed DreamZero reference? How many candidates does reranking examine, and does selection use any feedback from test rollouts?
  4. Are there existing fixed-memory, fixed-configuration comparisons, or development results using a matched action-free variant? Otherwise, is the intended conclusion limited to the embedding model’s contribution through the complete evolution process?
  5. Was development-stage token consumption recorded for both evolved systems? For tasks with few shared successes, can existing per-rollout timing and decision counts clarify the variability of the efficiency estimates?
3 comments

We thank the reviewer for recognizing that the paper evaluates whether retrieval helps control, with random-reference controls, context replacement at fixed policy weights, and matched-update training, and that the matched retraining study supports the contribution of action. These controlled comparisons are how the paper establishes its core claim.

W1 / Q1. What the benchmark measures

Physis-Embed-Bench measures predictive retrieval: locating the recorded continuation of the current state. Control utility is measured separately in Section 5.

Exact-clip positives. The paired future clip is the one relevance label that needs no subjective judgment. A valid continuation retrieved from another episode is scored as a miss, so clip-level R@1 is a lower bound on how often the model retrieves a suitable continuation.

Stage sensitivity. Half of the gallery consists of nearby clips from the query's own episode (Appendix B.1). They share the instruction, objects, and scene and differ in execution stage, so the model must localize the stage to score. Restricting the gallery to candidates with the same instruction still gives 47.62% R@1, against 21.63% for random ranking. On LIBERO and RoboMIND-FD it is 38.60% and 40.10%, against 1.90% and 0.24% (Table 21).

Link to downstream utility. The link is at the model level. Physis-Embed has higher clip R@1 than Omni-Embed-Nemotron-3B (47.3% versus 24.5%, Table 1). Under the same policy it also gives higher success: 50.3% versus 43.3% with Top-1 references and 56.3% versus 51.0% with the same reranker (Table 8), and 68% versus 33% in Embed2Evolve (Table 12). Random references lower DreamZero success to 36.0%, below the 47.0% without warm-up.

W2 / Q2. Table 1 and the attribution to action

Table 1 compares complete models. The available embedding models do not accept native action, so each is evaluated through its supported interface (Appendix B.1). The attribution to action comes from controlled removal, which now covers the complete benchmark.

Training-time removal on all four sources (new). We retrained the model without one modality at a time on the full four-source corpus, with the same backbone, data, and budget. Results on Physis-Embed-Bench (R@1, %):

Training inputsClipEpisode
Full VLA47.3381.77
No action (VL)26.1052.63
No vision (LA)28.1850.89
No language (VA)16.9830.88

Without action, clip R@1 falls by 21.2 points and episode R@1 by 29.1 points. Clip R@1 falls on every source: EgoDex 36.70 to 27.20, DROID 74.10 to 54.10, LIBERO 38.50 to 12.00, and RoboMIND-FD 40.00 to 11.10. The action-free model is trained on the same data and reaches 26.10%, close to the best external VL embeddings in Table 1 (24.5% and 25.2%). In-domain retrieval training alone does not explain the margin in Table 1.

Existing controls. On DROID, matched retraining without action drops from 78.7% to 59.8% (Table 6). With the final checkpoint and the full-VLA gallery fixed, withholding action from the query lowers four-source clip R@1 from 47.33% to 20.08% (Table 7).

The action contribution holds on the complete benchmark and is not limited to DROID.

W3 / Q3. Stage matching in the applications

Stage matching enters the applications in two ways.

DreamZero (Table 8). Top-1 uses the highest-similarity retrieved clip as returned, which is task-level selection. The Reranking Agent adds stage matching: among the top-10 retrieved candidates, it selects the one relevant to the query and recuts the four-second segment that shows completion evidence. With the Reranking Agent, success rises from 50.3% to 56.3% for Physis-Embed and from 43.3% to 51.0% for Omni-Embed-Nemotron-3B. The wording of Appendix C.1 is ambiguous on this point: the successful-stage recut is performed by the Reranking Agent and is not part of the Top-1 conditions. We will clarify the sentence.

Embed2Evolve. Re-matching during execution is exercised here. The current VLA query retrieves again every one to three policy decisions and whenever the active arm changes (Appendix D.1, Table 33).

Q3. The query is the task instruction, the initial frame, and the initial action. Retrieval ranks candidates by embedding similarity alone, and the Reranking Agent examines the top-10 candidates. No outcome of any test rollout is used at either step. The selected reference then stays fixed for all trials of that task (Appendix C.1).

W4 / Q4. What the Embed2Evolve comparison shows

The comparison shows two things.

A better embedding evolves a better system. Under identical evolution settings (policy, prompt, initial memory, seeds, proposal and stopping rules, and round cap), the harness evolved with Physis-Embed reaches 68% and the one evolved with Omni-Embed-Nemotron-3B reaches 33% (Table 12).

With the memory system fixed, the better embedding still reaches higher average success (new). We ran the fixed-memory comparison the reviewer asks for. We froze the harness evolved with Physis-Embed, including its memory bank and context recipe, and replaced only the retrieval embedding with Omni-Embed-Nemotron-3B, on the same 20 test initializations per task. Success is 56%, against 68% with Physis-Embed.

TaskOmni-Embed, own evolved harnessOmni-Embed, Embed2Evolve harnessPhysis-Embed, Embed2Evolve harness
Poke Cube455080
Stack Cube203060
Lift Peg Upright556065
Pull Cube with Tool205050
Two-Robot Stack Cube259085
Average335668

The first two columns use the same embedding with different harnesses (33% versus 56%). The last two use the same harness with different embeddings (56% versus 68%).

W5 / Q5. Scope of the efficiency results

The primary efficiency result is policy tokens over all 100 test rollouts per condition, failures included: 0.921M per rollout for Direct and 0.518M for Embed2Evolve, a 43.7% reduction, with a reduction on every task (Table 39).

The saving is not a by-product of context management. Both evolved systems manage context with bounded histories under the same search rules. With Omni-Embed-Nemotron-3B the result is about 2.531M tokens per rollout: 2.7 times Direct and 4.9 times Embed2Evolve, at 33% success. Managing context with a weaker embedding raises the cost, so the saving depends on which embedding manages the context.

Per-rollout variability on shared successes. Across the 19 pairs, policy decisions decrease in 17 and execution time decreases in 15, with a median time reduction of 48.9%. Pooling all 19 pairs without task weighting gives 62.0% fewer decisions and 35.1% less time, against 64.1% and 41.4% with equal task weights. For the tasks with few pairs: Pull Cube with Tool goes from 80 to 26 decisions and from 962.27 s to 489.55 s; the two Stack Cube pairs go from 40 and 50 decisions to 10 each, and from 384.63 s and 613.43 s to 170.27 s and 202.42 s. Both pairs with more decisions are Poke Cube initializations that Direct solved in 16 and 18 decisions. We will add the per-pair table to Appendix D.5.

Development cost. Development is a one-time cost per task. For Embed2Evolve, excluding the initial Direct rollouts that seed the memory, it used 48.70M, 35.50M, 32.99M, 44.23M, and 73.06M policy tokens on the five tasks in Table 12 order, 234.48M in total, or USD 2,438.54 at the reference rates of Appendix D.6. Relative to Direct, the test-time saving recovers this cost after about 71 to 187 rollouts per task. In deployment this is a very small number of rollouts, so the development cost is controlled and reasonable.

Dear Reviewer b4aG,

Thank you again for your detailed review. The discussion period closes on November 18, and we would like to check whether our response addresses your concerns. We ran the two controls you asked about: a matched action-free model on the full four-source benchmark (26.10% versus 47.33% clip R@1), and a fixed-memory comparison in Embed2Evolve (56% versus 68%). We also reported the per-rollout efficiency statistics and the development cost.

If any concern remains, we would be glad to address it before the deadline. If the response resolves your concerns, we would appreciate it if you could consider updating your assessment.

Reviewer_586nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Physis-Embed, a method that determines future actions by retrieving relevant past experiences—specifically, past behaviors—that are suitable for the task currently being performed. Physis-Embed functions as a retriever that encodes vision, language, and action (VLA) data into a unified representation for both the current state (query) and recorded past experiences (key). The current state is represented through instructions, images, and native actions, while past states are represented via instructions, videos, and actions. Built upon the Cosmos3-Edge 4B model, the framework is trained using a combination of predictive retrieval alignment loss, multimodal grounding and discrimination loss, and representation regularization loss. Authors further present two frameworks that leverage the VLA states retrieved by Physis-Embed to generate actions: Embed2ICL, which utilizes these states through in-context learning or as context tokens, and Embed2Evolve, which iteratively refines memory and context recipes through repeated execution, without requiring additional training.

Critical strengths
  1. The paper argues that encoding vision, language, and action information when generating vector embeddings for current and past states leads to more effective retrieval of relevant clips for the current situation—a claim further supported by ablation experiments.
  2. Experiments have demonstrated that Physis-Embed is not limited to serving merely as a VLA retriever; when integrated into frameworks (Embed2ICL and Embed2Evolve), it proves effective for action generation by retrieving experiences that are genuinely more useful.
Critical weaknesses
  1. Overall, I find the methodology difficult to reproduce. The paper does not clearly define the criteria for the ignored category for the predictive retrieval alignment loss. Also the three loss terms are not clearly defined in mathematical terms, and no codebase has been made publicly available.
  2. When reporting the performance of Embed2ICL and Embed2Evolve, I believe it is necessary to compare them against frameworks such as RA-VLA (Jang et al., ICML 2026) and Retrieve, Don't Retrain (Park et al., 2026), which incorporate retrieved past experiences into robot policies through adaptation.
Questions for authors
  1. While existing methods for action-aware retrieval—such as CLASP (Abootorabi et al., 2024) and VLA-R (Seong et al., 2024)—already exist, I believe the technical novelty of Embed-Physis lies in its simultaneous encoding of a VLA for both the query and the key. I am curious whether it can be concluded that this simultaneous encoding of the VLA into both the query and the key is the specific factor that led to the performance improvements.
  2. It seems that performance in retrieving past experiences would be quite sensitive to the choice of anchors, yet this aspect is not discussed. I am curious to know how performance might change if a different method for setting these anchors were used.
3 comments

We thank the reviewer for recognizing the two central points of our paper: jointly encoding vision, language, and action for both the current state and the recorded experience retrieves more relevant clips, and the retrieved experience is useful for action generation in Embed2ICL and Embed2Evolve. This retrieval effectiveness is the core idea of our paper.

W1. Reproducibility

Criteria for ignored pairs. Appendix A.2 defines them. For each query:

  • Positive: the one candidate that shares the exact future-clip ID.
  • Negative: a candidate with a different clip ID and an explicit semantic conflict with the query (task, object, action, or joint). Conflicts are decided by deterministic rules over a structured parse of the instruction and over action primitives and phases. For action negatives, same-episode future windows with temporal IoU above 0.25 are excluded as near-duplicates, and cross-episode negatives must differ in future gripper phase.
  • Ignored: every other non-positive candidate, i.e., semantically compatible, uncertain-label, or potentially duplicate ones. They are masked out of the InfoNCE denominator and are never mined as hard negatives.

Loss definitions and release. Appendix A.2 gives the masked bidirectional InfoNCE (Eq. 5), the modality-swap hinge loss (Eq. 6), and the hard-negative selection rule, and Table 14 lists the weight and margin of every term. In the revision we will give the complete mathematical definition of every loss term. We will open-source the code, the model, and the benchmark.

W2. Comparison with RA-VLA and Retrieve, Don't Retrain

Our contribution is the VLA embedding model. We do not claim that Embed2ICL is the state of the art among retrieval-augmented policies. Embed2ICL and Embed2Evolve test one claim: under the same downstream policy, a more accurate VLA embedding retrieves better context and gives a larger gain. Each experiment holds the policy fixed and changes only what is retrieved:

  • DreamZero (Table 8): with the same frozen policy, average success is 36.0% with random references, 47.0% without warm-up, 43.3% with Omni-Embed-Nemotron-3B Top-1, and 50.3% with Physis-Embed Top-1. With the same reranker, it is 51.0% versus 56.3%.
  • Fast-WAM (Tables 10 and 30): with the same policy checkpoint, retrieved Context Tokens reach 92.8% versus 88.0% for random tokens. A no-ICL continuation with the same number of updates stays at 86.4%.
  • Embed2Evolve (Table 12): 68% versus 33% for a harness evolved independently with Omni-Embed-Nemotron-3B under the same protocol. With the harness fixed and only the embedding replaced, it is 68% versus 56% (new result).

RA-VLA is a complete retrieval-plus-policy system for test-time adaptation to unseen tasks. It fine-tunes a GR00T N1.5 policy to use retrieved expert segments, and its retrieval encoder takes only the visual observation and the language instruction as input. A comparison with it would change the policy backbone, the training, and the evaluation protocol together. It would compare two policy systems, not two embeddings.

Retrieve, Don't Retrain retrieves with hand-designed distances over object pose, proprioception, and image features. It has no learned retrieval embedding and is not related to our core claim, a VLA embedding model. It is also an arXiv-only preprint. Under the ICLR 2027 Reviewer Guidelines, authors are not required to compare to papers that are solely on arXiv, and the lack of such a comparison cannot be a basis for rejection.

We will cite both works.

Q1. Is encoding VLA on both the query and the key the factor behind the improvement?

Yes. Encoding vision, language, and native action jointly on both the query and the key is the source of the improvement, and it is the key idea of our paper: Physis-Embed is, to our knowledge, the first unified VLA embedding model.

  1. Training-time removal. We retrain the model without action at both the query and the key, with the same backbone, data, and budget. On the full four-source benchmark, clip R@1 drops from 47.33% to 26.10% and episode R@1 from 81.77% to 52.63% (new result). On DROID alone, clip R@1 drops from 78.7% to 59.8% (Table 6).
  2. Inference-time removal. With the final checkpoint and the full-VLA gallery fixed, withholding action from the query lowers four-source clip R@1 from 47.33% to 20.08% (Table 7), and from 78.7% to 50.7% on DROID (Table 25).

Prior action-aware methods place action on one side only (Table 40): CLASP retrieves state–action behavior from a language query, and VLA-R matches vision–language scene queries against action tokens.

Citation correction. "CLASP (Abootorabi et al., 2024)" is not an action-aware retrieval method. That paper is "CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval", a language–speech retrieval model with no relation to robotics or action. The action-related CLASP is Rana et al. (2023), "Contrastive Language, Action, and State Pre-training for Robot Learning", which we cite in Section 2. VLA-R is Seong et al. (2025).

Q2. Sensitivity to the choice of anchors

Retrieval is not sensitive to the choice of anchors. Anchors are not selected keyframes: we enumerate dense anchors at every frame that has a complete four-second future window, and we keep those in which SAM3 localizes the instruction's interaction object and Qwen3-VL verifies it (Appendix A.3). Replacing the SAM3 interaction crop with the full frame changes DROID clip R@1 only from 78.7% to 77.9% (Table 5). The benchmark also evaluates queries at anchors throughout held-out episodes, against a gallery in which half of the clips are nearby same-episode windows (Section 3.3, Appendix B.1).

Dear Reviewer 586n,

Thank you again for your review. The discussion period closes on November 18, and we would like to check whether our response addresses your concerns. In brief: retraining without action on the full four-source benchmark lowers clip R@1 from 47.33% to 26.10%, and with the memory harness fixed, replacing only the embedding changes success from 68% to 56%. We will also add the complete loss definitions and release the code, model, and benchmark.

If any concern remains, we would be glad to address it before the deadline. If the response resolves your concerns, we would appreciate it if you could consider updating your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.