- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tasks coding agents to construct articulated assets, and aims for them to improve after each asset by accumulating information as prompts/skills to help with future assets. This forms a type of skill-learning, where a checkpoint is a set of skills that has been developed by working on a set of training tasks, during which the skills change but the LLM weights are unchanged. AdaCraft is a process for updating skills after each task that preserves correctness of articulated assets and limits the rate of change. The authors show that the final AdaCraft-optimized skill set produces more frequently correct articulated assets than either the base agent or an agent with unstructured self-adaptation of its skills.
Critical strengths
Testing the ability of agents to look at past errors and develop skills that help it in the future is interesting, and the work shows that agents' ability to do this can be significantly improved, over and above their default behavior when doing this.
Critical weaknesses
ArtiCraft is only included in qualitative results not quantitative, despite that it is to my knowledge the strongest harness for articulated object generation, and includes similar functionality checks as are used to develop AdaCraft. I could interpret the paper as a more nuanced contribution studying specifically the ability to make prompts from scratch, but at the moment its components seem very similar to an articulated object harness like articraft, just retained across sessions.
Questions for authors
Would you be able to summarize what the agent discovered to be useful? What does the final AdaCraft configuration contain after gathering across many tasks? How do they compare to the manually designed prompts of ArtiCraft?
Can AdaCraft be compared quantitatively to ArtiCraft? I would hope this can be done using its paper results or its asset corpus without running its harness. If AdaCraft is not expected to be better than ArtiCraft, then is the
Where does ArtiCraft fall on the Fig5a pareto front? Would it be a point with high functional yield and 0 optimization tokens?
NOTE - Questions from this line and onwards are non-essential:
How many tokens total were spent accumulating the skills used in the final AdaCraft system? Is it possible that if ArtiCraft were given the same additional tokens at train time it might outperform AdaCraft, which already spent its extra budget on other tasks? At what number of test examples would the upfront investment in AdaCraft pay off? Note: this question is not essential.
Would it be reasonable to have a baseline similar to Naive accumulation, but where agents are just told to test if their skill is good by writing code? Testing the skill via code is a generalization of the current harness. Or is this already present in naive accumulation and just doesnt occur successfully?
3 comments
Q2. How AdaCraft differs from ArtiCraft:
The “Fixed agent” baseline in Table 1 evaluates ArtiCraft under our shared model, benchmark, and resource settings, without our additional task-skill library. Here, “fixed” means that no persistent task-skill library is updated across objects; ArtiCraft’s testing and repair within each generation episode remain active. We agree that the original label obscures the ArtiCraft comparison and have renamed it to “ArtiCraft harness (fixed agent)” in Table 1 in the revised paper.
Our contribution is reliable cross-episode skill evolution, enabled by three complementary components: (1) Motion Memory, which retains successful repairs and counterexamples with explicit mechanical applicability conditions; (2) Homological Motion Signatures, which capture motion connectivity and identify clearance bottlenecks beyond individual pose checks; and (3) Adaptive Edit Budgets, which regulate revision scope based on transfer variability, execution uncertainty, geometric dependencies, and motion clearance. Together, these components enable reusable skill improvement while preserving previously acquired capabilities. Overall, the idea is to give an "Adam optimizter" for the coding agent.
Table R11. Quantitative comparison under the same evaluation protocol.
| Method | Functional yield ↑ | Calls / functional asset ↓ | Optimization tokens |
|---|---|---|---|
| ArtiCraft harness (fixed agent, no task skills) | 36.2% | 48.1 | 0 |
| ArtiCraft + fixed initial task skills | 40.7% | 42.5 | 0 |
| SkillAdam + shared gate | 58.1% | 28.9 | 214.7M |
| ArtiCraft + AdaCraft skill evolution | 75.1% | 16.8 | 85.4M |
AdaCraft improves functional yield by 38.9 percentage points relative to the fixed-agent condition and reduces deployment calls per functional asset from 48.1 to 16.8.
More importantly, against SkillAdam with the same acceptance gate, AdaCraft improves yield by 17.0 percentage points, demonstrating benefits beyond the shared generation harness or candidate-validation mechanism.
Q3 Where does ArtiCraft fall on the Figure 5(a) optimization–performance curve?
The fixed ArtiCraft condition requires zero cross-episode optimization tokens. It would sit on the Y axis. However, it does incur its normal generation and repair costs during deployment.
Q4 Is the additional optimization cost justified, and could ArtiCraft benefit from equivalent compute?
We agree that training-time cost and deployment-time efficiency should be distinguished.
AdaCraft uses 85.4M optimization tokens in the reported experiments, including diagnosis, candidate generation, validation, and rejected revisions. This is upfront expenditure. Table R12 summarizes optimization and deployment cost:
Table R12. Optimization and deployment cost.
| Measure | ArtiCraft fixed agent | SkillAdam + gate | AdaCraft |
|---|---|---|---|
| Optimization tokens | 0 | 214.7M | 85.4M |
| Functional yield (%) | 36.2 | 58.1 | 75.1 |
| Deployment tokens / functional asset | 1.443M | 0.867M | 0.504M |
| Successful objects needed to recover optimization cost (vs. ArtiCraft) | — | 373 | 91 |
ArtiCraft could also benefit from additional generation or repair compute. However, SkillAdam + shared gate already optimizes skills using the same harness and consumes 214.7M optimization tokens, yet achieves 58.1% functional yield, compared with 75.1% for AdaCraft. This supports the efficiency of our skill-evolution strategy.
Q5 Code-based testing?
Yes, code-based testing is definitely possible. And, in fact, they are already available through the underlying ArtiCraft harness! However, these local tests frequently fail to produce functionally valid articulated assets and do not establish whether a skill learned from one object will generalize to others.
As shown in Table 1, Naive Accumulation achieves only 39.4% functional yield, meaning that 60.6% of generation attempts fail the prescribed structural or motion checks. Simply accumulating corrections, even with access to execution feedback, is therefore insufficient.
Table R13. Functional yield and failure rate under code-based testing.
| Method | Functional Yield ↑ | Failure Rate ↓ |
|---|---|---|
| Fixed agent (ArtiCraft harness) | 36.2% | 63.8% |
| Naive Accumulation | 39.4% | 60.6% |
| SkillAdam + shared gate | 58.1% | 41.9% |
| AdaCraft | 75.1% | 24.9% |
Importantly, testing an individual generated asset is different from validating a reusable skill across objects. A correction that passes local code-based tests may still fail under different joint configurations or invalidate previously successful mechanisms.
AdaCraft addresses this limitation through mechanically conditioned Motion Memory, adaptive revision budgets, and cross-object validation.
Dear Reviewer bXqy,
Thank you again for your thoughtful questions and constructive feedback. Your input has helped us significantly strengthen our work, clarify the relationship to ArtiCraft, and explain what AdaCraft learns across tasks. As the discussion period is nearing its end, we wanted to follow up and invite your thoughts on these clarifications.
Thank you for your time and consideration!
We thank the reviewer for recognizing the value of learning reusable skills from past errors and AdaCraft’s improvements over agents’ default skill-accumulation behavior. Below, we clarify what the system learns, its relationship to ArtiCraft, and the corresponding performance and computational costs.
Q1. What does the final AdaCraft configuration contain and how useful they are
The initial task-skill library has four modules: geometry, joint structure and frames, motion clearance, and physical attributes/export. These supplement the underlying ArtiCraft harness without changing its frozen generation model or fixed tools.
The initial rules cover basic operations such as part naming, selecting joint types from required motions, choosing hinge pivots, setting gaps, inspecting motion endpoints, and assigning inertia.
AdaCraft subsequently revises these instructions based on cross-object evidence. The important distinction is that the revised library contains persistent applicability-conditioned procedures, whereas ArtiCraft's original harness performs construction and verification within an individual episode.
Table R10 summarizes representative changes from the initial task-skill library to the final selected library. The accepted seed-42, round-3 J2/J3 revision in Figure 7 and Table R4 provides a concrete example: it restricts hinge placement to appropriate mechanisms, aligns the hinge axis with the stationary seam, and specifies limits that preserve the required travel. Its compatible-objective gain is ΔQ = 0.021, and the revised rules entered the active library after acceptance.
Table R10. Evolution of the construction-skill library.
We have added a summary of the learned procedures in Section 4, ‘What the learned library contains,’ covering geometry, joint structure, motion clearance, and physical attributes/export.