- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CoDA couples a simulation data engine and an action policy through shared memory. A coding agent constructs factored scene programs with executable sparse success predicates. A vision–language critic diagnoses policy failures by scene factor, trajectory phase, and supporting evidence. Targeted scene variants and controls test these diagnoses before recurring difficulty modes are compiled into two kinds of procedural memory: generation programs that recreate the difficulty and training programs that help the policy overcome it. Learning-progress feedback guides subsequent environment generation.
Text2potential converts a diagnosis into subgoals and bounded potentials over simulator state. Potential-based shaping supplies dense targets to an action-chunk value critic. Its chunkwise telescoping identity is
=\sum_{i=0}^{h-1}\gamma^i r_{t+i} +\gamma^h\Phi(s_{t+h})-\Phi(s_t).$$ The learned critic guides data collection and advantage-weighted policy updates; planner-certified demonstrations add imitation supervision when available. The final policy acts without executing the critic or memory programs. Across four simulated manipulation families, CoDA reports 27.6% OOD success versus 18.1% for scalar-feedback generation and 19.2% for a critic without memory at the main generation budget. Extended runs improve further, and LIBERO-Pro results reach 84% position and 76% instruction-perturbation success after 15 rounds. Additional studies address memory structure, shaping, critic choice, program transfer, backbone transfer, and computational cost.Critical strengths
-
The shared diagnosis-to-generation and diagnosis-to-training interface is a substantive systems idea. The method does not stop at generating physically valid scenes or ranking tasks by current success rate. A localized failure record is used both to create a difficulty and to generate a response to it, with targeted/control evaluation before the record becomes a reusable program (§§3.1–3.4). This supplies a concrete connection between the data engine and policy learning. The separation of a sparse task verifier from dense shaping is also sensible: the critic's interpretation is not itself the reported success criterion. The distinction between generation programs and training programs is supported by Table 5, where retaining both outperforms either alone and the single-store alternatives.
-
The experiments provide several mutually supporting tests of the system. The main OOD advantage is not solely against the weak SFT baseline: CoDA improves by 9.5 points over scalar-feedback co-evolution and 8.4 points over the same critic without persistent memory (Table 1). Text2potential improves over sparse-only training from 22.3% to 27.6%, while the task-description and no-refinement controls help isolate the value of diagnosis and refinement (Table 4). The LIBERO-Pro improvements retain standard LIBERO performance, and the cross-backbone experiment shows that accumulated environments/programs remain useful for another policy without new generation. The paper also reports diagnosis refutation, human agreement, and costs instead of treating the language-model components as infallible or free (Tables 3, 8, and 9). Collectively, these make the empirical systems contribution interesting.
Critical weaknesses
-
The core learning-progress loop and the implementation conditions for shaping need to be specified more precisely. Section 3.2 introduces and mode importance , while §3.3 says that an upper-confidence rule combines hardness and learning progress. Appendix B supplies a success band, exploration coefficient, and round triggers, but not the complete allocation equation, the estimator/update for progress, or how changes in sampled environments are separated from improvements in the policy. These choices define the central co-evolution mechanism and affect whether a mode is informative, merely noisy, or persistently hard. An executable pseudocode description and one worked mode update are needed to reproduce this part of the contribution.
The shaping identity itself is correct, and §3.5 appropriately cites Ng et al. (1999) and restricts policy invariance to fixed bounded potentials with suitable terminal handling. The remaining issue is whether the implemented training loop meets those conditions. Omitting the critic bootstrap at termination is not, by itself, sufficient: for an episode ending at , telescoping still leaves
Thus, the terminal contribution must be handled explicitly, for example by zero terminal potentials or a consistent absorbing continuation. The paper does not specify this rule for successful termination, failure, and time limits, nor how replay targets remain consistent when a potential is refined between rounds. The shaping scale in Appendix B should also be reconciled with Eqs. (3)–(4), and Eq. (2) needs an explicit definition when no subgoal is satisfied, since it otherwise indexes although the listed functions are . These are concrete implementation/notation questions, not a demonstrated counterexample to the authors' code. However, they must be settled before the policy-invariance discussion can be taken as a property of the actual implementation.
-
Equal accepted-environment budget does not isolate equal-resource efficiency or all aspects of memory transfer. Appendix C usefully reports the additional expense: per 1,000 accepted environments, adaptive CoDA uses 7.4M tokens and 66 GPU-hours, versus 4.6M and 51 for scalar feedback (Table 9). Hence the 9.5-point OOD improvement is demonstrated at matched accepted-environment count, not at matched total compute. The extra controls, diagnosis, compilation, and training can be worthwhile, but the experiment does not establish how much advantage remains after the comparator receives equivalent resources. The accounting also needs to make explicit whether the 64 targeted and 64 control environments per round, rejected proposals, and unsuccessful planner attempts count toward each displayed budget (Appendix B).
Likewise, Table 7 compares a target backbone trained with transferred memory against an SFT baseline with much less accumulated experience. This demonstrates useful reuse, but does not fully isolate the value of structured memory from access to the same previously generated environments and demonstrations. An archive-reuse control with identical training data but without semantic/procedural retrieval would make that interpretation stronger. Finally, targeted variants being harder than a different-factor control is evidence for a useful intervention, but not necessarily identification of a unique cause; matching perturbation severity and reporting uncertainty around the validation decision would help distinguish causal diagnosis from finding a harder perturbation. These concerns limit attribution and efficiency claims rather than erase the reported OOD gains.
-
No real-world robot experiments.
Questions for authors
-
Specify the co-evolution update. Please give the exact learning-progress estimator, its evaluation population, the mode-level update rule, and the upper-confidence allocation formula. How are estimates refreshed when the policy changes, and how are noisy or negative progress estimates treated? A worked example following one failure record through validation, allocation, program compilation, and retirement would make the central mechanism much more reproducible.
-
Verify the shaping implementation. What potential is used on success, failure, and timeout transitions? Are past replay transitions re-scored with the current potential, or versioned so that each critic update uses a consistent fixed potential? Please provide the actual terminal TD target, the placement of the shaping coefficient, and the zero-completed-subgoal convention. If these already follow the stated invariance conditions, documenting them would resolve the main soundness concern without changing the method.
-
Separate resource and memory effects. Can the authors report a matched-compute/token comparison with scalar-feedback generation, or a success-versus-total-cost curve, including control and rejected-scene costs? For cross-backbone reuse, how does structured memory compare with simply reusing the identical generated scene/demonstration archive under the same training budget? These tests would clarify whether the proposed organization contributes beyond the value of additional verified experience.
6 comments
W2/Q3 Resource matching and memory attribution
We agree that matching accepted-environment counts does not establish equal computational efficiency. Our primary comparison controls the number of accepted environments, while Table 9 reports the additional computational expenses of diagnosis, validation, and program compilation.
Table R7. Computational cost and OOD performance.
| Method | LLM tokens (M) ↓ | GPU-hours ↓ | OOD success (%) ↑ |
|---|---|---|---|
| Uniform variants | 3.1 | 38 | 8.5 |
| Scalar-feedback | 4.6 | 51 | 18.1 |
| CoDA, fixed schedule | 8.3 | 74 | 26.1 |
| CoDA, adaptive schedule | 7.4 | 66 | 27.6 |
Computational costs are normalized per 1,000 accepted environments.
CoDA improves OOD success by 9.5 percentage points over scalar-feedback generation at the matched accepted-environment budget. This improvement incurs additional computational cost, which we acknowledge. Nevertheless, adaptive scheduling reduces both LLM token usage and GPU-hours by approximately 11% relative to fixed scheduling, while increasing OOD success from 26.1% to 27.6%.
To further assess compute-normalized efficiency, we do a matched-resource comparison in which each method is subject to the same LLM-token and GPU-hour ceilings, including the costs of rejected proposals, targeted/control evaluations, and planner attempts. Accepted-environment totals follow Appendix B's training-environment budget: 320 per family, or 1,280 across four families, at five rounds; control environments are additional and their costs are included.
Table R8. OOD success under matched computational budgets.
| Budget | Method | Accepted envs | LLM tokens (M) | GPU-hours | OOD success (%) ↑ |
|---|---|---|---|---|---|
| Low | Scalar-feedback | 1,280 | ~5.89 | ~65.3 | 18.1 |
| Low | CoDA, cost-capped | 795 | 5.88 | 52.5 | 20.8 |
| High | Scalar-feedback, expanded | 1,650 | 7.59 | 84.2 | 22.4 |
| High | CoDA, original | 1,280 | ~9.47 | ~84.5 | 27.6 |
Memory attribution. We further agree that cross-backbone transfer may benefit from access to accumulated environments and demonstrations, independently of the semantic/procedural organization of memory.
The original cross-backbone experiments demonstrate that memory collected with improves OpenVLA-OFT OOD success from 3.8% to 21.1%, without additional environment generation, approaching the 22.7% achieved when running CoDA directly with OpenVLA-OFT.
To isolate the contribution of structured memory, we evaluate an archive-only reuse baseline, retaining the same generated scenes, demonstrations, and training budget while disabling semantic and procedural memory retrieval.
Table R9. Structured memory versus archive-only transfer.
| Memory source | Target policy | OOD success (%) ↑ |
|---|---|---|
| No memory (SFT) | OpenVLA-OFT | 3.8 |
| , archive-only | OpenVLA-OFT | 16.0 |
| , structured memory | OpenVLA-OFT | 21.1 |
| OpenVLA-OFT, full CoDA | OpenVLA-OFT | 22.7 |
| No memory (SFT) | 5.3 | |
| OpenVLA-OFT, archive-only | 19.8 | |
| OpenVLA-OFT, structured memory | 25.8 | |
| , full CoDA | 27.6 |
Finally, regarding diagnosis validation, targeted/control interventions test whether changing the diagnosed factor produces greater difficulty than changing an alternative factor.
W3 Real-world experiment
We have added a real-world experiment using a UR5 arm across eight conditions varying target location, clutter, support surface, and the number of required targets. Across 160 held-out trials per method, CoDA achieves 88.1% average success, compared with 66.3% for both the initial policy and uniform adaptation. CoDA and uniform adaptation use the same initial checkpoint, 80 additional demonstrations, and 2,000 policy-update steps. These results support the effectiveness of diagnosis-guided configuration selection for real-robot adaptation under a matched training budget. Section 4.1 (Figure 5 and Table 6) reports the results, and Appendix F provides the protocol and execution sequences.
Q2 Terminal potentials, shaping coefficient, and replay consistency
We appreciate the reviewer's careful distinction between the shaping identity and its implementation.
Terminal-state handling. The implementation explicitly sets the successor shaping contribution to zero for successful terminal transitions.
The implemented chunk-level TD target is
(1-d_t)\gamma^h\left[ \beta\Phi(s_{t+h})+ \bar Q(s_{t+h},a_{t+h}) \right] -\beta\Phi(s_t),$$ where $d_t$ indicates a terminal transition and $\beta=0.5$. Thus, for a terminal chunk, the target reduces to $$y_t=R_t^{(h)}-\beta\Phi(s_t).$$ The implementation distinguishes successful termination from unsuccessful time-limit truncation. Successful transitions are terminal, with zero successor potential and no bootstrap. Unsuccessful rollouts reaching the maximum horizon are treated as truncated and retain successor potential and bootstrapping. We will clarify this convention and the assumptions required for theoretical policy invariance. **Replay consistency.** When a training program is compiled or refined, the implementation recomputes stored endpoint potentials for replay transitions associated with that difficulty mode. Thus, subsequent critic updates use the current potential rather than mixing stored potential values from different versions. **Zero-completed-subgoal convention.** If no completion predicate is satisfied, $k^*(s)=0$, and the implementation evaluates the first subgoal potential $\phi_1(s)$. If all subgoals are satisfied, the fractional term is zero. We will correct Eq. (2) to make these conventions explicit. **Empirical effectiveness of Text2potential.** Beyond the theoretical analysis, we evaluate how different reward and potential formulations affect policy learning. **Table R10. Effect of potential-based shaping on policy learning.** | **Training signal** | **OOD success (%) ↑** | **Steps to 25% OOD (k) ↓** | | --- | --- | --- | | Sparse verifier only | 22.3 ± 0.8 | 31 | | Potential from task description | 24.6 ± 0.7 | 29 | | Free-form dense reward | 23.5 ± 1.0 | 33 | | Text2potential without refinement | 26.1 ± 0.6 | 27 | | **Text2potential (ours)** | **27.6 ± 0.5** | **21** | Text2potential improves OOD success by **5.3 percentage points** over sparse-only learning and requires approximately **32% fewer policy updates** to reach 25% success. Furthermore, using task descriptions rather than rollout-grounded failure diagnoses reduces success to 24.6%, while disabling potential refinement reduces success to 26.1%. These findings support the practical benefits of diagnosis-conditioned potential shaping and iterative refinement. We thank the reviewer for recognizing the contribution of coupling environment generation and policy learning through shared failure diagnoses. In our response, we have specified the learning-progress updates and scheduling rule, clarified the shaping implementation and replay consistency, and added resource-budget and archive-only transfer comparisons. Structured memory improves OOD success over reuse of the same scene and demonstration archive by 5.1 and 6.0 percentage points across the two transfer directions. Our new real-robot evaluation further achieves 88.1% success versus 66.3% for uniform adaptation under the same demonstration and policy-update budgets. These additions substantially strengthen the paper’s reproducibility and empirical support. We respectfully ask the reviewer to reconsider their score in light of this new evidence and the accompanying clarifications.
We thank the reviewer for their feedback and for recognizing the value of using shared failure diagnoses to guide both environment generation and policy training, together with the supporting evidence from our ablations and transfer experiments. We address the concerns below.
W1/Q1 Learning progress details
CoDA measures learning progress through paired evaluations of the same generated environments before and after policy updates, rather than comparing success rates on different environment populations. For environment s, we define
LP(s)=p^θr+1(s)−p^θr(s),
where both estimates use N=16 rollouts on the same environment. For a difficulty mode Kj, progress is aggregated over its targeted/replay environments:
LPj=∣Ej∣1∑s∈EjLP(s).
The implementation then updates the mode's importance and success estimates:
wj=LPj−ηI[no subgoal in target band],
p^j=∣Ej∣1∑s∈Ejp^θr+1(s),
where η=0.1. The generation scheduler uses
+c\sqrt{\frac{\log(N_{\mathrm{sel}}+2)}{n_j+1}},$$ where $n_j$ counts selections of mode $j$, $N_{\mathrm{sel}}$ is the total number of eligible-mode selections, and $c=1$. Modes are ranked by this score, with 15% of the generation budget reserved for novel environments and 10% for replay. Negative learning progress is retained rather than clipped, reducing the priority of modes on which training was ineffective. Previously encountered modes are periodically re-evaluated under the updated policy. This separates measured policy improvement from the difficulty of newly sampled environments, although finite-rollout estimation noise remains. **Table R6. Difficulty-mode validation and learning-progress update.** | **Stage** | **Metric** | **Example mode $K_j$** | | --- | --- | --- | | Diagnosis | Factor / phase | Spatial / Approach | | Validation | Targeted success (%) | 37.50 | | Validation | Control success (%) | 62.50 | | Before training | Success (%) | 37.50 | | After training | Success (%) | 56.25 | | Progress update | $LP_j$ (percentage points) | +18.75 | | Scheduling | Updated importance $w_j$ | 0.1875 | | Scheduling | UCB score $U_j$ | 1.344 | The example shows the complete update mechanism: the targeted perturbation lowers success relative to the control, supporting the diagnosed difficulty; subsequent training improves success on the same environment; and the updated learning-progress and hardness estimates determine its subsequent generation priority. We have added the learning-progress estimator, mode-importance update, and scheduling formula to Appendix B, together with the paired evaluation protocol and treatment of negative progress.