- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CARE-VLA, a training-free framework that improves agentic VLA planning by retrieving step-level execution experience from a read-only multimodal bank. The key idea is to decompose reference trajectories into primitive-level transitions, each linking the pre-decision execution context, the executed primitive, its outcome, and before–after observations.
At each decision boundary, CARE-VLA performs context-compatibility filtering, dual-route ranking (task+context semantics via Qwen3-Embedding and visual similarity via SigLIP2), reciprocal rank fusion, and diversity-constrained selection to produce a compact "Memory Card" of at most three records. This card supplements the planner's existing context without updating any parameters.
Evaluated on LIBERO-Pro (80.3% vs. 72.1% for Harness VLA with Codex, +8.1 pp), RoboTwin C2R (61.6% vs. 58.0%, +3.6 pp), and three real-robot pick-and-place tasks (85.0% vs. 73.3%, +11.7 pp), CARE-VLA demonstrates consistent improvements while also reducing average primitive executions by 8.9% on shared successful episodes.
Critical strengths
-
The identification of decision-level misalignment between memory granularity and planning granularity (§1, p.1–2) is a genuine and underappreciated problem in agentic VLA systems. The step-level decomposition (§3.2) directly addresses this by aligning memory records with primitive decision boundaries, enabling local evidence retrieval across tasks and episodes. The training-free design, which preserves the planner and VLA parameters while only augmenting the planner's context, is both principled and practically valuable. This is consequential because it demonstrates that meaningful improvements can be achieved through better experience organization alone, without the cost and risk of retraining.
-
The paper demonstrates improvements on two simulation benchmarks (LIBERO-Pro: +8.1 pp; RoboTwin C2R: +3.6 pp) and real-robot tasks (+11.7 pp), all using the same frozen planner and VLA (Tables 1–4). The ablation studies, particularly the diversity ablation (Table 3: 84.3% → 72.5% without primitive diversity) and the experience-source study (Table 3: 79.8% → 84.3% from 1-suite to 8-suite), provide meaningful evidence for the individual design choices. The efficiency result (8.9% fewer primitive executions, Table 6) further supports that the gains come from better decision-making rather than simply more computation.
Critical weaknesses
W1. Uncontrolled experience-budget asymmetry. CARE-VLA builds its bank from 80 reference episodes (1,093 records; Table 5), whereas Harness VLA's task-specific memory uses one trace and one summary per task. No baseline gives Harness VLA an equivalent amount of structured experience (e.g., more traces per task or richer summaries). Without this control, the observed gains cannot be cleanly attributed to the step-level retrieval mechanism rather than to having substantially more experience available. The paper acknowledges this asymmetry only in the real-robot section (§4.5) but does not address it in the simulation experiments.
W2. Absence of statistical significance reporting. No confidence intervals, standard deviations, or significance tests are reported for any experiment. On RoboTwin C2R (250 episodes), the 3.6 pp improvement over Harness VLA corresponds to roughly 9 episodes, which may fall within random variation. On LIBERO-Pro (800 episodes), the 8.1 pp gap is larger but still unverified. Without at least a paired bootstrap or McNemar test on the shared task–seed pairs, the reliability of these improvements—especially on RoboTwin—remains uncertain.
W3. Execution-context filtering is never independently ablated. The paper motivates context-compatibility filtering as essential for distinguishing scenes that look similar but differ in execution state (§3.3). However, no ablation removes only the filtering step while keeping ranking, fusion, and diversity selection unchanged. The existing ablations (text-only in Table 1; w/o diversity in Table 3) change different components. Since filtering is presented as a core design principle, its isolated contribution should be directly demonstrated.
W4. Numerous hyperparameters without sensitivity analysis. The retrieval pipeline introduces at least ten tunable constants: , , , , , , , , and history weights . The paper does not explain how these were selected, whether they were tuned on a held-out set or on the evaluation tasks, or how sensitive results are to their values. Even a brief sweep of and would substantially strengthen confidence.
W5. Limited real-robot validation scope. The real-robot evaluation covers only 3 pick-and-place tasks with 20 trials each (60 trials total; Table 4). All tasks share the same primitive structure (approach → grasp → transport → place), leaving untested whether the retrieval mechanism helps with more diverse manipulation (e.g., assembly, tool use, or contact-rich tasks). No per-trial outcomes, failure-mode breakdown, or cross-day robustness are reported. While the 11.7 pp gain over Harness VLA is encouraging, the small and homogeneous task set limits the strength of the real-world claim.
Questions for authors
-
Can you provide a baseline where Harness VLA is given additional reference experience (e.g., multiple task-specific traces per task instead of one, or more detailed summaries) matched to the information content of CARE-VLA's 80-episode bank? Alternatively, can you run CARE-VLA with a reduced bank (e.g., 10 reference episodes, ~130 records) to assess how much of the improvement persists? This would help isolate whether the gain comes from the retrieval mechanism or from having more experience.
-
Could you provide confidence intervals or results of statistical significance tests (e.g., paired bootstrap, McNemar's test) for the main results on LIBERO-Pro and RoboTwin C2R? In particular, is the 3.6 pp improvement on RoboTwin C2R (approximately 9 episodes out of 250) statistically significant?
-
Could you report an ablation that removes only the execution-context filtering step (allowing all records to enter ranking, without requiring previous primitive/outcome/failure type/gripper state to match), while keeping all other components unchanged? This would directly test the paper's central claim that context compatibility is essential.
-
What is the success rate of the 80 reference episodes used to construct the LIBERO-Pro experience bank? Are all reference episodes successful, or do they include failure–recovery sequences? How does the quality of reference episodes affect retrieval quality?
-
Could you provide a brief analysis of failure cases where CARE-VLA underperforms or where retrieved evidence is misleading? For instance, are there cases where context-compatible records suggest an inappropriate next primitive due to subtle scene differences not captured by the visual encoder?
-
For RoboTwin C2R, the bank uses "one non-evaluation initialization seed per task across all 50 tasks" (§4.1). Could you clarify the total number of step-level records in this bank and whether the cross-task retrieval (across all 50 tasks) is a significant factor in the observed improvement?
7 comments
3. Expanding experience sources can further improve performance
We progressively expand the experience sources from Long-T/S to Goal-T/S and Long-T/S, then to all eight LIBERO-Pro T/S suites, and finally add reference experience from the four task suites under Object (Obj) perturbations. These configurations contain 20, 40, 80, and 120 reference episodes, respectively.
The retrieval parameters and card budget remain unchanged. Evaluation is fixed to the same 200 task–seed pairs on Long-T/S, consisting of 2 suites × 10 tasks × 10 seeds.
| Total reference episodes | 20 | 40 | 80 | 120 |
|---|---|---|---|---|
| CARE-VLA success rate | 82.0% | 83.5% | 84.5% | 85.0% |
The success rate increases from 82.0% to 85.0%, showing the value of expanding experience sources for further performance improvements. In future work, we will investigate the relationships among experience scale, diversity, and retrieval effectiveness on larger benchmarks.
In summary, directly expanding access to Harness-format experience does not yield gains, whereas CARE-VLA outperforms the retrieval-disabled control by 3.5 percentage points with the same step-level bank. With only 10 reference episodes per suite, it still improves by 7.0 percentage points and retains approximately 61% of the full-bank gain. Expanding experience sources further improves performance. These results indicate that the gains arise from both experience accumulation and effective experience organization and retrieval.
Q2 and W2: Statistical significance and uncertainty of the main results
Q2 — Reviewer’s question: Could confidence intervals or statistical significance tests, such as paired bootstrap or McNemar’s test, be provided for LIBERO-Pro and RoboTwin C2R? In particular, is the 3.6-percentage-point improvement on RoboTwin C2R, corresponding to approximately nine out of 250 episodes, statistically significant?
W2 — Reviewer’s concern: The paper does not report confidence intervals, standard deviations, or significance tests. The RoboTwin C2R gain may reflect random variation, and the larger LIBERO-Pro gap is also unverified. Paired analysis on shared task–seed pairs is needed to assess the reliability of the improvements.
Our response to Q2 and W2:
Thank you for your suggestion. Using the historical execution logs, we conducted episode-level paired bootstrap analysis and two-sided exact McNemar tests on success/failure outcomes for the same task–seed pairs. The supplementary results are reported in Appendix L, Table 17.
1. Statistical tests use the internally reproduced baseline
The published Harness VLA results provide only aggregate success rates, which cannot be used for paired tests. We therefore use the internally reproduced baseline with matched configurations and per-episode execution records. Its success rates on LIBERO-Pro and RoboTwin C2R are 71.9% and 57.2%, respectively, close to the published 72.1% and 58.0%.
2. The paired improvements on both benchmarks are statistically significant
| Quantity | LIBERO-Pro | RoboTwin C2R |
|---|---|---|
| Matched evaluation episodes | 800 | 250 |
| Harness VLA successes | 575 | 143 |
| CARE-VLA successes | 642 | 154 |
| Both methods succeed | 548 | 142 |
| Only CARE-VLA succeeds | 94 | 12 |
| Only Harness VLA succeeds | 27 | 1 |
| Success-rate gain (percentage points) | 8.4 | 4.4 |
| Paired bootstrap 95% CI (percentage points) | [5.75, 11.00] | [1.60, 7.20] |
| Two-sided exact McNemar test |
For RoboTwin C2R, which is of particular concern to the reviewer, statistical significance depends not only on the net increase in successful episodes but also on the differences between the methods on the same episodes: 12 episodes succeed only with CARE-VLA, while just one succeeds only with Harness VLA. This paired difference is statistically significant (), and the 95% confidence interval for the gain excludes zero. LIBERO-Pro similarly shows a paired advantage of 94 versus 27 episodes, with the statistical test also supporting the reliability of the improvement.
In summary, compared with the internally reproduced Harness VLA baseline under matched configurations, CARE-VLA improves by 8.4 and 4.4 percentage points on LIBERO-Pro and RoboTwin C2R, respectively. Both improvements are statistically significant (), and both paired-bootstrap 95% confidence intervals exclude zero, further supporting the reliability of the gains.
Q3: Ablation of execution-context compatibility filtering
Reviewer’s question: Could an ablation remove only execution-context filtering, allowing all records to enter ranking without matching the previous primitive, outcome, failure type, or gripper state, while leaving all other components unchanged? This would directly test the central claim about context compatibility.
Our response:
Thank you for your suggestion. We have added an ablation that removes only execution-context compatibility filtering while keeping all other components unchanged, directly testing its contribution (Appendix E.2, Table 9).
The experiment uses GPT-6 Astra with low reasoning on Goal-T/S and Long-T/S in LIBERO-Pro. Both settings share the same original memories, step-level experience bank constructed from 40 reference episodes, retrieval parameters, Memory Card budget, and evaluation settings. The ablation no longer requires historical records to match the current previous primitive, outcome, failure type, and gripper state, allowing all records to enter subsequent ranking. Multimodal ranking and diversity-based selection remain unchanged.
| Method | Goal-T | Goal-S | Long-T | Long-S | Overall |
|---|---|---|---|---|---|
| CARE-VLA without context compatibility filtering | 88 | 97 | 86 | 74 | 86.3 |
| CARE-VLA | 92 | 98 | 89 | 78 | 89.3 |
Removing filtering reduces the average success rate from 89.3% to 86.3%. This shows that context compatibility filtering provides additional gains even when multimodal ranking and diversity-based selection are retained. Historical records with similar tasks or visual observations may differ in action outcomes, failure types, and gripper states. Filtering out incompatible records before ranking helps provide experience better suited to the current execution state.
In summary, with the other components and experience budget unchanged, context compatibility filtering provides an average gain of 3.0 percentage points, supporting its importance for retrieving decision-relevant experience.
Q4: Reference-episode success, failure–recovery sequences, and experience quality
Reviewer’s question: What is the success rate of the 80 reference episodes used to construct the LIBERO-Pro bank? Are they all successful, or do they include failure–recovery sequences? How does reference-episode quality affect retrieval quality?
Our response:
Thank you for your attention to reference-experience quality. All 80 stored reference episodes ultimately complete their tasks, giving a task success rate of 100%, and are segmented into 1,093 step-level records. We follow Harness VLA’s reference initialization procedure to collect one ultimately successful trajectory per task. This 100% refers to the selected trajectories stored in the bank, not to every attempt during collection.
An ultimately successful episode can still contain local failures and recovery, such as adjusting the gripper after an empty grasp or changing the action after stalled movement. We retain these steps, their execution feedback, subsequent actions, and before–after observations. Thus, the bank contains failure–recovery sequences, rather than only successful actions.
Reference-experience quality depends not only on final success but also on record accuracy, state diversity, and applicability to the current decision. Accurate feedback and before–after observations help distinguish effective actions from failed attempts, while local failure and recovery provide experience absent from smoothly executed trajectories. Conversely, even when the source episode ultimately succeeds, highly repetitive records or records mismatched to the current state may occupy limited retrieval slots and reduce the value of the evidence. CARE-VLA therefore uses context compatibility filtering, multimodal ranking, and diversity-based selection to provide complementary experience applicable to the current state.
In future work, the bank could also include ultimately unsuccessful reference episodes, using explicit failure sequences as negative examples to help the planner identify actions to avoid. For example, under similar contact-induced stalling, a historical record showing that repeatedly issuing the same movement command produces no progress could prompt the planner to avoid repeating ineffective actions and instead observe again or adjust its strategy. We will study combinations of successful experience, recovery experience, and negative failure examples, and their effects, on larger benchmarks.
In summary, the current 80 reference trajectories all ultimately succeed but include local failures and recovery. The value of experience should be assessed by the execution evidence it provides. Including negative failure examples in the future could allow the bank to support decisions about both how to act and which actions to avoid.
Experience Controls, Statistical Reliability, Retrieval Analysis, and Real-Robot Evaluation
Q1 and W1: Experience-budget fairness and the contribution of retrieval
Q1 — Reviewer’s question: Could Harness VLA receive additional reference experience, such as multiple traces per task or richer summaries, matched to the information content of CARE-VLA’s 80-episode bank? Alternatively, could CARE-VLA be evaluated with a reduced bank of approximately 10 reference episodes and 130 records to distinguish retrieval gains from the benefits of more experience?
W1 — Reviewer’s concern: CARE-VLA uses 80 reference episodes and 1,093 records, whereas Harness VLA uses one task-specific trace and summary per task. Without a baseline given an equivalent amount of structured experience, the gains cannot be cleanly attributed to step-level retrieval. This asymmetry is acknowledged for the real robot but not addressed in simulation.
Our response to Q1 and W1:
Thank you for your attention to experience budgets and performance attribution. We address Q1 and W1 through matched-experience controls, small-bank experiments, and experience-expansion experiments to distinguish the effects of experience volume from those of retrieval.
1. Providing Harness VLA with additional experience from the same sources
The original bank contains one reference trajectory from each of 80 tasks. CARE-VLA segments these trajectories into step-level records that can be retrieved across tasks. As GPT-5.5 was no longer available during the rebuttal period, the supplementary experiments use GPT-6 Astra with low reasoning on Goal-T/S and Long-T/S and are reported separately from the original GPT-5.5 results.
All groups use matched planners, VLA backends, software, task–seed pairs, execution budgets, and evaluation protocols. The added experience is collected by Astra, with one non-evaluation reference episode for each of 40 tasks, and is provided to the planner in different forms.
Expanding access to Harness-format experience decreases the success rate from 83.5% to 76.8%. This setting covers the same reference tasks as CARE-VLA but organizes and uses the experience differently, showing that more experience does not automatically yield gains. Irrelevant experience and the additional selection burden may affect current decisions.
Providing the full step-level experience bank increases the success rate to 85.8%, supporting the value of step-level organization for using added experience. With exactly the same original memories and step-level experience bank, CARE-VLA further achieves 89.3%, outperforming the retrieval-disabled control by 3.5 percentage points, directly supporting the role of retrieval (Appendix E.2, Table 9).
2. Substantial gains remain with only 10 reference episodes
Table 3 of the original paper already includes a small-bank setting that uses only 10 reference episodes for each target suite. The record counts for Spat-T, Spat-S, Goal-T, and Goal-S are 152, 137, 108, and 135, respectively, averaging 133 records, consistent with the scale suggested by the reviewer.
The following results are averaged over these four suites using GPT-5.5 with medium reasoning. All CARE-VLA settings use the same retrieval parameters and a Memory Card budget of at most three records.
With only 10 reference episodes, CARE-VLA still improves by 7.0 percentage points, retaining approximately 61% of the full-bank gain. Thus, the complete 80-episode bank is not necessary to obtain improvements.