- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Probe–Diagnose–Adapt (PDA), an LLM-guided robot co-design framework that uses behavioral evidence and experimental memory to guide morphology modifications through testable predictions and iterative evaluation. Experiments on six Evolution Gym tasks show improved search performance over existing baselines under matched morphology-evaluation budgets, while additional results on two UNIMAL tasks demonstrate applicability to articulated robots.
Critical strengths
- The paper is well-motivated. The use of behavioral evidence beyond task returns provides additional information for guiding morphology search.
- The evaluation covers diverse task types, including soft voxel-based robots and articulated rigid-body robots, supporting the applicability of the method across different robot representations.
- PDA achieves the highest mean performance among the compared methods on all six Evolution Gym tasks at the common evaluation budget.
Critical weaknesses
- Efficiency is measured by morphology evaluations, without LLM costs or wall-clock time. It would be helpful to report these costs and consider the parallelism available to population-based baselines.
- Limited LLM generalization. All experiments use DeepSeek-Flash. Testing additional LLMs or unseen tasks would help assess backbone dependence and potential reliance on prior knowledge of Evolution Gym.
Questions for authors
A key distinction from Auto-Robotist is the prediction and verification of behavioral changes, but this mechanism is not independently ablated. Could the authors provide an ablation that removes prospective prediction and verification while retaining the same behavioral evidence and memory?
5 comments
Response to Reviewer cXTW [2/3]
[Weakness 2]: LLM backbone dependence and benchmark knowledge
[A2]: We followed your suggestion to test additional LLMs on UpStepper-v0: all four added backbones improve designs during online search. We used the same 53-evaluation budget, with 6 different searches per added backbone. Improvement is the mean within-search gain and final performance is mean +/- across-search SD (Sec. 4.6; App. L.2, Table 22 and Fig. 11).
| LLM | Initial performance | Final performance | Improvement |
|---|---|---|---|
| DeepSeek-Flash | 1.455 | 4.663 +/- 1.087 | +3.208 |
| GLM-5.3-Flash | 1.820 | 4.930 +/- 0.231 | +3.109 |
| Qwen3.8-Flash | 1.500 | 3.898 +/- 0.060 | +2.399 |
| Gemini 3.8 Flash | 1.452 | 3.766 +/- 0.885 | +2.313 |
| GPT-6 Luna | 1.417 | 3.711 +/- 0.618 | +2.293 |
For a quick visual comparison: PDA's search curves with different LLM backbones.
GLM has the highest final mean, 4.930, while DeepSeek has the largest mean gain, 3.208. The other backbones also make online gains of 2.293-2.399. Performance varies across models, but all tested backbones improve their initial designs through online search. These results show that PDA's improvement on this task is not limited to DeepSeek-Flash. Since the models may already know Evolution Gym, testing unfamiliar tasks is an important next step.
Response to Reviewer cXTW [3/3]
[Question 1]: New ablation: Prediction and verification with factual memory retained
[A3]: We implemented exactly this ablation as PDA-NoPred: it retains current behavioral measurements and factual histories of bodies, edits, returns, and measured changes, but removes directional predictions and their assessments. Initialization, controller budgets, retrieval, and selection rules are shared with PDA-R, the full-information reference under this common ablation protocol. The table below summarizes 6 independent searches per task and variant, paired by initialization, after 20 online edits. Values are mean +/- across-search SD (Sec. 4.3, Fig. 4; App. I.3, Table 11).
| Task | PDA-R | PDA-NoPred | Relative mean gain |
|---|---|---|---|
| UpStepper-v0 | 3.611 +/- 0.328 | 2.481 +/- 0.037 | 45.6% |
| Pusher-v0 | 9.852 +/- 0.199 | 8.965 +/- 0.795 | 9.9% |
With behavioral measurements and factual history available to both methods, PDA-R achieves higher mean returns than PDA-NoPred on both tasks. Search trajectories: EvoGym ablation curves. The corresponding NoPred comparison on UNIMAL also favors PDA in mean: 25.9% on Incline and 30.4% on Push Box Incline (Sec. 4.5, Fig. 5; App. K.4, Table 20). For a visual comparison: UNIMAL search curves.
Furthermore, we introduced PDA-Facts to test what information should be carried forward between trials. It retains behavioral measurements and still generates and checks predictions for the current trial, but provides subsequent decisions with only factual history: bodies, edits, returns, and measured changes. PDA-R additionally retains past hypotheses, predictions, and their assessments. PDA-R's higher mean returns support reusing these records to guide later edits, beyond generating and checking predictions within each trial (Sec. 4.3; App. I.3–I.4).
For a quick, systematic comparison: Anonymous ablation overview. It brings together the original and added ablations, their information matrices, performance curves, and compact examples of the actual decision inputs. We hope this makes it easy to see both what each ablation isolates and which conclusions the results support.
Thank you again for these suggestions. They helped us test whether predictions and their assessments improve later design decisions, and show the computational cost alongside the performance gains.
Response to Reviewer cXTW [1/3]
Thank you for encouraging us to connect PDA's performance to its practical cost and to test its dependence on the LLM backbone and prediction mechanism. Your questions really helped us distinguish controller-training efficiency from wall-clock efficiency and make the mechanism comparisons more direct.
[A1]: First, we report the LLM usage and end-to-end wall-clock cost of PDA. Morphology evaluations and wall-clock time capture different costs, and your comment prompted us to report both. The table below covers all 6 tasks at 53 morphology evaluations and 512K PPO steps per body (Sec. 4.6; App. N.1, Table 25).
Each row is the measured task-level mean across 6 independent searches. End-to-end wall-clock time includes initialization, controller training, evaluation, and LLM calls through the final search evaluation. Note that the LLM-call column is included in that total.
Second, we compare how many morphology evaluations and how much wall-clock time each method needs to reach the same performance target. For each task, this target is the strongest baseline's mean return at 53 morphology evaluations. This comparison allows population-based baselines 4 concurrent candidate trainings, while PDA remains sequential. Each entry reports median morphology evaluations / wall-clock time (h) to reach the target return, and NR means the target was not reached within the available records (App. N.2, Table 26).
The table makes the trade-off clear: among target-reaching searches, PDA requires the fewest morphology evaluations on all 6 tasks, but population parallelism can shorten wall-clock time. For example, LASeR is faster in the UpStepper, Carrier, and Thrower comparisons. We recognize this practical advantage.
PDA’s value is therefore in reducing costly controller-training trials, rather than always finishing sooner. With 512K PPO steps per body, each avoided controller training saves 512K training interactions; parallel execution does not eliminate those interactions. Additional reasoning can be worthwhile when controller training and candidate testing are the limiting resources. Although these results are from simulation, this distinction also motivates future applications involving costly physical interaction.