- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how CAD agents can improve by changing their instructions and tools while keeping the base model fixed. The system checks whether past advice fits a task, whether the agent follows it, whether it carries it out correctly, and whether it helps. It then revises the advice or turns some steps into code. On 200 Artiverse test tasks, it scores 0.743, compared with 0.683 for conventional evolution. The learned harness also helps on other objects and base models.
Critical strengths
All evolving methods start with the same basic knowledge of the CAD software. This is a useful control. It helps separate learning to use the software from learning better ways to build objects. The full system improves several measures of CAD quality. The authors also test the frozen harness on other data and models. The evaluation uses code-based checks, surface measures, and an agent judge. Two human raters check the judge on 50 outputs. Together, these results give reasonable evidence that the system helps in this setting.
Critical weaknesses
-
The experiments do not clearly show how much the proposed failure checks add. The two gaps describe intuitive problems: using advice in the wrong setting and carrying out suitable advice incorrectly. These are useful distinctions, but their value as a method needs stronger evidence. The main comparison adds several things together, including a verifier, extra trials, and LLM-based routing that decides which advice each task receives. Table 4 shows that choosing advice and using executable tools help. It does not isolate the value of the four-step diagnosis or local contracts. Also, replacing tools with older text may put text at a disadvantage, since the final harness evolved around those tools.
-
The paper leaves uncertainty about both repeatability and cost. It does not state how many independent evolution runs support the results, and its wording (“in our run”) suggests there was only one. It also shows no error bars. I therefore cannot tell how much the scores vary across runs. All evolving methods share the same wall-clock limit for each outer-loop round and the same token cap for each Actor execution. However, token use inside each outer-loop round is explicitly not limited (Appendix D1). Actor token use is reported only for the proposed method: it grows from about 60K to 200K tokens on average with the final harness. Outer-loop token use is not reported for any method. This does not prove that the method costs more than the baselines, but the comparison needs them.
Questions for authors
-
Can you test the four-step diagnosis while keeping the verifier, extra trials, advice selection, and tool creation unchanged? This would help show what the diagnosis itself adds.
-
How many independent evolution runs were used? Please report variation across runs and actual token use, for both the Actor and the outer loop, for each main method.
-
Figure 2 sketches one case (a flipped hinge axis and a trial with and without the experience). Can you show one complete example from an actual run: a failed step, the resulting change to advice or code, and the local-contract checks used to test that change?