acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Guided Harness Evolution for Articulated CAD Agents

Abstract

Articulated computer-aided design (CAD) agents rely on modeling knowledge written into their harnesses by hand, and harness evolution could acquire this knowledge from experience without updating the base model. Examining what conventional harness evolution learns, we find that most of its early gain comes from environment knowledge shared across tasks; adding this knowledge to the initial harness by hand recovers the gain. The modeling experience accumulated afterwards helps less reliably, for two reasons: it can reach tasks it does not fit (an applicability gap), or the agent can follow it and still carry out the operation incorrectly (a knowing–doing gap). The two failures look the same in a task score but call for different revisions. We therefore propose evidence-guided harness evolution, which localizes failures before revising experience. The outer loop tests hypotheses about which experience to add or change through geometric and kinematic checks against training references and targeted trials. Evidence traces each use of an experience through four questions—whether it fit the task, was followed, was carried out correctly, and improved the outcome—and the first that fails decides what to revise. Applicability conditions are narrowed or widened, and operations the agent repeatedly gets wrong become programs checked against local contracts. Starting from an environment-informed harness, so that gains measure what evolution adds beyond it, our method raises the test score from 0.626 to 0.743, above conventional evolution (0.683) and prior harness optimizers. The frozen harness also improves unseen categories, another dataset, and other base models. These results support our view that the scope and realization of experience should be revised with evidence from its use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, harness evolution, experience reuse, articulated CAD

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mA7FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how CAD agents can improve by changing their instructions and tools while keeping the base model fixed. The system checks whether past advice fits a task, whether the agent follows it, whether it carries it out correctly, and whether it helps. It then revises the advice or turns some steps into code. On 200 Artiverse test tasks, it scores 0.743, compared with 0.683 for conventional evolution. The learned harness also helps on other objects and base models.

Critical strengths

All evolving methods start with the same basic knowledge of the CAD software. This is a useful control. It helps separate learning to use the software from learning better ways to build objects. The full system improves several measures of CAD quality. The authors also test the frozen harness on other data and models. The evaluation uses code-based checks, surface measures, and an agent judge. Two human raters check the judge on 50 outputs. Together, these results give reasonable evidence that the system helps in this setting.

Critical weaknesses
  1. The experiments do not clearly show how much the proposed failure checks add. The two gaps describe intuitive problems: using advice in the wrong setting and carrying out suitable advice incorrectly. These are useful distinctions, but their value as a method needs stronger evidence. The main comparison adds several things together, including a verifier, extra trials, and LLM-based routing that decides which advice each task receives. Table 4 shows that choosing advice and using executable tools help. It does not isolate the value of the four-step diagnosis or local contracts. Also, replacing tools with older text may put text at a disadvantage, since the final harness evolved around those tools.

  2. The paper leaves uncertainty about both repeatability and cost. It does not state how many independent evolution runs support the results, and its wording (“in our run”) suggests there was only one. It also shows no error bars. I therefore cannot tell how much the scores vary across runs. All evolving methods share the same wall-clock limit for each outer-loop round and the same token cap for each Actor execution. However, token use inside each outer-loop round is explicitly not limited (Appendix D1). Actor token use is reported only for the proposed method: it grows from about 60K to 200K tokens on average with the final harness. Outer-loop token use is not reported for any method. This does not prove that the method costs more than the baselines, but the comparison needs them.

Questions for authors
  1. Can you test the four-step diagnosis while keeping the verifier, extra trials, advice selection, and tool creation unchanged? This would help show what the diagnosis itself adds.

  2. How many independent evolution runs were used? Please report variation across runs and actual token use, for both the Actor and the outer loop, for each main method.

  3. Figure 2 sketches one case (a flipped hinge axis and a trial with and without the experience). Can you show one complete example from an actual run: a failed step, the resulting change to advice or code, and the local-contract checks used to test that change?

Reviewer_fuAMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how LLM agents that build articulated CAD models can improve their harness over time without updating the base model. It first shows that most early gains from conventional harness evolution come from basic environment knowledge, and that writing this knowledge into the starting harness by hand recovers those gains. The proposed method adds a Verifier agent that checks generated parts and joints against training references and runs targeted trials. It traces each use of an experience through four questions, then revises the experience’s content, where it applies, or how it is carried out, including turning error-prone operations into programs checked against local contracts. It raises the test score from 0.626 to 0.743, compared with 0.683 for conventional evolution, and achieves the best mean on every reported metric, including the agent judge and surface metrics that do not depend on the LLM-annotated requirements. However, the central claims are not well supported. No variance estimates are reported, and total token usage during evolution is neither limited nor reported, even though the method uses an extra agent and additional trial runs. The four-question diagnosis is never ablated, so the gains may come largely from the Verifier’s reference checks. The baselines omit closely related methods such as ACE, AHE, and ArtiCAD. Task descriptions and most scoring requirements were written by GPT-5.5, and the method is not evaluated on any existing external benchmark.

Critical strengths

The problem picked by the paper is important within the greater literature with ongoing research needed for leveraging useage without retraining.

  • One of the paper's most solid findings is though showing much of early improvement comes from basic knowledge about the environment, not from learning how to model objects. Writing that knowledge into the starting harness by hand gives the same score that conventional evolution reaches after six rounds.
  • The method scores higher than the baselines on every reported measure, including ones that don't depend on the LLM-written scoring requirements. This makes it less likely that the improvement is just a quirk of how the tasks were scored, although the paper doesn't show how large or reliable the improvement really is.
  • The authors take some steps to keep test information away from the agents such as keeping the scoring program is hidden, the agents only see data from training tasks, and the harness is locked before testing.
Critical weaknesses

Critical Weaknesses

Experimental Rigor

No statistical evidence for the main result

  • The paper reports no variance estimates, seeds, or confidence intervals for any method.
  • The 0.060 gap over conventional evolution may reflect random variation across tasks rather than a genuine improvement.

Unmatched compute

  • Appendix H states that all methods share the same models, batches, rounds, one-hour limit per round, and per-execution Actor token cap.
  • However, Appendix D.1 states that token usage within a round is not limited.
  • The proposed method runs two outer-loop agents (Improver and Verifier) and launches additional Actor executions through targeted trials and step replays, which Appendix H acknowledges.
  • Appendix E.6 reports that Actor usage grows from about 60K tokens per task at the start of evolution to about 0.2M with the final harness, but test-time token usage for the baselines is never reported.
  • Without matched total token and rollout budgets, the comparison is not controlled.

Unreliable generalization results

  • In Table 2, all four evolved methods gain more over the environment-informed harness on unseen categories than on the seen categories they were trained on. GEPA, for example, gains +0.069 on unseen categories while losing 0.004 on seen ones.
  • This pattern is implausible and suggests noise in the 33-task unseen tier.
  • The proposed method's margin over Meta-Harness on unseen categories is only 0.007.
  • These results do not support the claimed transfer advantage.

Attribution of Gains

Confounded comparison and missing ablations

  • According to Appendix E.5, conventional evolution lacks a dedicated Verifier, hypotheses, targeted trials, localization, and explicit conditions and realizations all at once, so the 0.060 gap cannot be attributed to any single component.
  • The F1–F4 diagnosis, the paper's central idea, is never ablated; Table 4 removes only conditions and executable realizations.
  • The gain may come largely from the Verifier acting as a strong verification oracle rather than from the diagnosis itself.

Unclear contribution of each component over reflect-and-retry

  • The method adds targeted trials, step replays checked against local contracts, and explicit conditions to a standard reflect-and-edit loop.
  • The paper never shows that these components, rather than the added Verifier, produce the gain.
  • It presents no case study of a revision the method made that a reflect-and-retry loop with the same Verifier could not have made.

Baselines

Weak and possibly misconfigured GEPA baseline

  • GEPA can only modify the harness's prompts, not its tools, routing, or workflow, so it cannot test the paper's claims about scope and realization.
  • The paper does not state what feedback GEPA received, even though GEPA's performance depends heavily on feedback quality.
  • All methods share the paper's validation-based acceptance rule, which appears to replace GEPA's Pareto-based candidate selection.
  • The +0.006 result therefore says little about GEPA or about the proposed method's advantage.

Missing key baselines

  • The paper does not compare against ACE, AHE, MAC, LLMto3D, or ArtiCAD.
  • ACE and AHE evolve reusable agent experience and harnesses, and ArtiCAD generates articulated CAD assemblies; all are closer to this setting than GEPA.
  • The only domain baseline is LAM, run with substituted inputs.

Tasks and Evaluation

Self-built, LLM-generated tasks with possible leakage

  • GPT-5.5 wrote the task descriptions from the ground-truth dataset records. A human checked them, but Figure 1's example states part counts directly ("two articulated arms," "four independently rotatable adjustment knobs"), so descriptions may leak scored requirements.
  • The authors hand-annotated requirements for only 30 tasks; GPT-5.5 annotated the remaining 470, and no accuracy check on those annotations is reported.
  • The program metrics (Score, Struct, Motion, Assembly) all depend on these annotations.

No external validation

  • The harness is frozen before testing, yet it is never evaluated on an existing benchmark such as ArtiCAD-Bench or ACD, compared against published results, or compared with a human CAD designer.
  • The PartNet-Mobility transfer uses tasks built with the authors' own pipeline, so it does not provide independent validation.
  • Transfer to other base models is reported only for the proposed method, so it cannot show that its experience is more reusable than the baselines'.

Framing and Presentation

Unclear framing and contributions

  • The paper's third contribution is an evaluation on a new task set, but the reproducibility statement promises only source code, not the tasks, annotations, or evaluator, and the task set is not externally validated.
  • The main paper does not explain the tasks in enough detail, leaving most of this to the appendix.
  • The related work section is too brief to situate the paper within the CAD generation literature.
Questions for authors
  1. The paper reports no variance, seeds, or confidence intervals. What is the bootstrap confidence interval on the 0.060 gap over conventional evolution, and does the gap hold across independent evolution runs?

  2. What were each method’s total outer-loop tokens, Actor executions (including trials and replays), and test-time tokens per task, and does the gain survive at matched compute?

  3. How does conventional evolution score with only the Verifier added, and how does your method score without the F1–F4 labeling step?

  4. Why do all four evolved methods gain more on unseen categories than on the seen categories they were trained on (Table 2), and with only 33 tasks per unseen tier, how do you rule out noise?

  5. Given the same Verifier and compute, which component accounts for the gain over a reflect-and-retry loop, and can you show a revision that such a loop could not have made?

  6. What feedback did GEPA receive, and was its Pareto-based selection preserved?

7, Why did you build your own tasks rather than evaluating on an existing benchmark such as ArtiCAD-Bench or ACD, and how does the frozen harness perform there? Separately, why were ACE, AHE, MAC, LLMto3D, and ArtiCAD not compared or considered as baselines?

  1. Beyond the human check of descriptions and the 30 hand-annotated tasks, how did you verify that the GPT-5.5 descriptions do not leak scored requirements, and what is the error rate of the LLM-annotated requirements on the remaining 470 tasks?
Reviewer_rCvuOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for improving articulated CAD agents with evidence-guided harness evolution. The training procedure consists of an inner loop actor agent and outer loop that evolves the harness by accumulating experiences. Experiences are structured to have specific applicability conditions and operations that can be converted into executable actions with correctness checks. The outer loop tests hypotheses and revises experiences based on accumulated evidence. This evidence-guided evolution improves performance over other evolution strategies and hand-designed harnesses.

Critical strengths
  1. For articulated object generation, an evolving harness is a good method to capture useful knowledge that can be learned from existing datasets. By revising experiences based on evidence, the outer loop model can compensate for weaknesses in the actor model by giving more comprehensive guidance and applicability criteria.
  2. The experimental validation is reasonably thorough. The paper compares against several harness optimization methods and hand-designed harnesses. They also evaluate generalization, ablate components of their harness, and evaluate different base models.
Critical weaknesses
  1. It is unclear exactly what the harness is learning. The paper adequately describes the procedure of the harness and gives relevant metrics on the quality of the assets. However, the qualitative characteristics of what the harness is learning remain unclear. It would be useful to show examples or otherwise characterize the experiences beyond simply “environment” and “modeling.” This could demonstrate that the model is learning useful generalizable skills and not items specific to a particular dataset
  2. The computational cost is not sufficiently analyzed. Section E.6 describes the maximum budget and mentions that the actor execution uses 60K tokens at the start of evolution and 0.2M tokens with the final harness. It is unclear how different harnesses affect the average token length. Moreover, it would be best to show a cost-matched comparison to demonstrate that the learned rules are useful. The token cost of the outer loop is also not discussed adequately.
Questions for authors
  1. What are the token usage and inference costs for the methods in table 1, both during harness evolution and inference execution? If possible, could the authors provide a comparison between the final evidence-guided harness and a reasonable baseline at a matched cost?
  2. Could the authors provide examples of the learned experiences? How do the updates of the evidence-guided harness compare to the conventional harness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.