acceptodds
Under review as a conference paper at ICLR 2027

ArtifactArena: Evaluating Models by What They Build in the Physical World

Abstract

To evaluate the frontier, we must measure models not by what they say, but by what they can engineer and build in grounded physical environments. We introduce ArtifactArena, an open-ended platform where models face a physically grounded hardware-software co-design challenge: engineering fully functional robots to compete in a simulated arena. We evaluate a frontier model's zero-shot, verifier guided refinement, and open-ended physical design capabilities through three harnesses that refine their bot artifacts based on text descriptions, physics simulator feedback, and gameplay data. We benchmark these capabilities with an Elo ranking of frontier models derived from head-to-head tournaments between their artifacts. By releasing this framework and tournament infrastructure for ongoing community submissions, we establish a living, non-saturating testbed to continuously measure the expanding limits of open-ended intelligence in the physical world.

Primary area:
datasets and benchmarks
Keywords:
Large language model evaluation, open-ended agentic search, robot hardware–software co-design, physically grounded reasoning, competitive benchmarking.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zEsDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

ArtifactArena asks LLM to build MJCF robot body in MuJoCo and a Python controller to help control the LLM-designed robot body. Then the author compare LLMs' robot design and controling ability by evaluating them in Last Bot Standing game. The author use ELO score to rank all model. To help LLM design the robots, the author design three harness: Sampling Harness (SH), Verifier-Grounded Refinement Harness (VGH), and Design Lab Harness (DLH). The author analysis the 'Hardware creativity', 'Controller creativity', and running examples based on the experiment.

Critical strengths
  • The Last Bot Standing game evaluates robot artifacts through head-to-head competition, which is a reasonal game setting to compare the robot's ability in a relative limited perspective. Based on robot's battle performance, it is a good way to estimate the creativity of LLM's robot artifact design and streatgy design. Section 4 distinguishes this competitive setting from prescribed construction objectives and acknowledges prior co-design and competitive-evolution work. This part of paper contribution is clear as a benchmark design.
Critical weaknesses
  • The benchmark claims is limited validated. The abstract describes the paper a "living, non-saturating testbed to continuously measure the expanding limits of open-ended intelligence in the physical world". However, the paper's experiment and analysis only based on Last Bot Standing under MuJoCo simulation, which is hard to support the evaluation of "open-ended intelligence".
  • The comparison supports the reported performance of three configurations (SH, VGH and DLH), but does not isolate which design choices explain the differences. The author may consider add the ablation study to compare the harness difference acroess fixed set of models and the model difference with the fixed configuration of harness. Figure 2 already compares models across SH, VGH, and DLH. However, the harnesses change feedback, tool access, and artifact selection together. Component ablations and measured token and simulation costs would help explain the performance differences.
  • My another major concern is about paper writing and presentation. Figure 7, captioned 'Dichotomous key of SH generated bots', is difficult to read at normal page size: its dense branches, small labels, and robot thumbnails do not provide an accessible explanation of the taxonomy or classification criteria (Appendix 5.5). Section 3.3 visibly leaves dollar signs around code identifiers such as is_upside_down. The full prompts in Section 10 are useful for reproducibility, but the main text needs a concise summary of the consequential prompt and harness choices, with precise references.
Questions for authors
  • I am queries about the token/dollar cost for a LLM to complete each task. Would different harness's cost differ across models? And would stronger models have better performance is because they are better than reasoning and creating or it just have longer reasoning? Besides the leaderboard, author may consider offer a figure that shows the perato frontier lines to help better understand the relation between elo performance and token cost.

  • Which findings remain stable when the Last Bot Standing rules, MuJoCo parameters, or design objectives change? Beyond the ArtifactArena Leaderboard, what specific capability or empirical insight is revealed by this hardware-software co-design evaluation, relative to the work discussed in Section 4? What evidence supports the 'non-saturating testbed' claim?

3 comments

We thank the reviewer for their review and address their questions and weaknesses below.

The benchmark claims is limited validated. The abstract describes the paper a "living, non-saturating testbed to continuously measure the expanding limits of open-ended intelligence in the physical world". However, the paper's experiment and analysis only based on Last Bot Standing under MuJoCo simulation, which is hard to support the evaluation of "open-ended intelligence".

We will adjust the language in our abstract to be more precise. By “open-ended” we were referring to the vast solution space of possible robot morphologies and controllers that models can design towards accomplishing the goal of winning the last bot standing task. We are not measuring a model’s ability to discover its own tasks, but in our benchmarking paradigm we are asking the models to solve a problem for which, even we the benchmark designers, do not know the optimal solution. The massive design space and absence of a known optimal require models to explore, reason about, and develop novel solutions, making this a test of open-ended physical design and problem-solving capabilities.

The comparison supports the reported performance of three configurations (SH, VGH and DLH), but does not isolate which design choices explain the differences. The author may consider add the ablation study to compare the harness difference acroess fixed set of models and the model difference with the fixed configuration of harness. Figure 2 already compares models across SH, VGH, and DLH. However, the harnesses change feedback, tool access, and artifact selection together. Component ablations and measured token and simulation costs would help explain the performance differences.

Figure 2 is the comparison the reviewer describes. All 21 models are run under all three harnesses, so harness differences can be read with the model fixed, and model differences with the harness fixed.

Artifact selection does not change across harnesses. Each harness generates multiple candidate bots and the best-performing bot is selected through tournaments among those candidates as described in (Secs. 2.3 and 6.2)

In this work we make progress towards answering how does the performance of frontier models change when they are given different levels of control over the process of designing physical artifacts? Feedback and tool access do change together, and this is intentional. Each harness represents a level of control the model has over the design loop: no feedback (SH), verifier and match feedback (VGH), and full agency with tools (DLH). We do not claim to have exhaustively evaluated every possible harness, but we believe the three harness we strategically designed to include in the paper help measure how different levels of model autonomy and access to feedback affect the quality of generated artifacts, while establishing a systematic baseline for future research into more advanced harness designs. Our arena is harness-agnostic, and we release the harnesses so the community can build and compare even better ones.

Also, just to clarify we match the number of api calls made per harness and all three harnesses are derived from the exact same base prompt that fully describes the environment and game rules. VGH and DLH add additional sanctions to this prompt to describe their harness specific descriptions such as feedback and tool use.

My another major concern is about paper writing and presentation. Figure 7, captioned 'Dichotomous key of SH generated bots', is difficult to read at normal page size: its dense branches, small labels, and robot thumbnails do not provide an accessible explanation of the taxonomy or classification criteria (Appendix 5.5). Section 3.3 visibly leaves dollar signs around code identifiers such as is_upside_down. The full prompts in Section 10 are useful for reproducibility, but the main text needs a concise summary of the consequential prompt and harness choices, with precise references.

We thank the reviewer for catching the dollar signs in Sec. 3.3. This is a typesetting error and we will fix it. We respectfully note that Figure 7 is an appendix figure. It is a supplementary vector graphic that classifies every sampled bot and is meant to be zoomed, and none of the paper’s claims depend on reading it at page size. We will add text to Appendix 5.5 that lists the key questions and explains how bots are classified. On the summary of prompt and harness choices, the main text already contains this: Sec. 2.1 (“Model Task”) summarizes what the prompt tells the model, and Sec. 2.3 summarizes the three harnesses. Our guiding principle in designing our prompts was to be as descriptive as possible providing complete information about the game rules and environment while not biasing the model towards any specific bot design, controller functions, or high level game strategies. We allowed models to derive those entirely on their own from first principals based on our description of the task and environment.

We would like to ask the reviewer to be more precise about the presentation concern. The review rates the paper “difficult to follow despite careful reading”, but the specific issues listed are an appendix figure, a typesetting error in one sentence, and a summary that is already present in the main text. Reviewer 4dby writes that our paper “is well written and easy to follow”. If there are sections of the main text that the reviewer found hard to follow, we would be grateful if they could point to them, and we will revise them. We will of course correct small mistakes such as the dollar signs, but we do not believe that they, or the legibility of an appendix figure, justify treating the presentation as a major weakness of the paper.

I am queries about the token/dollar cost for a LLM to complete each task. Would different harness's cost differ across models? And would stronger models have better performance is because they are better than reasoning and creating or it just have longer reasoning? Besides the leaderboard, author may consider offer a figure that shows the perato frontier lines to help better understand the relation between elo performance and token cost.

In our experiments, we allocate the same budget of 10 model API calls per run to each harness, with three independent runs per model–harness combination (Sec. 2.3). Matching the number of API calls provides a standardized interaction budget for comparing models and harnesses, although reasoning time and dollar costs may differ.

Our benchmark evaluates the quality of artifacts produced under this standardized API call budget, rather than attempting to isolate reasoning ability from computational effort. Such isolation is particularly difficult for closed-source models, where internal reasoning processes and computational effort are not fully observable or controllable. Moreover, our evaluation reflects how these models are used in practice, where what ultimately matters is the quality of the artifacts they produce rather than how much internal reasoning they perform.

To address the reviewer's question about computational efficiency, we provide an additional cost–performance analysis figure comparing the Elo performance of different model–harness combinations against their generation costs. Cost–Performance Analysis Figure

Which findings remain stable when the Last Bot Standing rules, MuJoCo parameters, or design objectives change? Beyond the ArtifactArena Leaderboard, what specific capability or empirical insight is revealed by this hardware-software co-design evaluation, relative to the work discussed in Section 4? What evidence supports the 'non-saturating testbed' claim?

Our primary goal is to establish an open-ended, non-saturating benchmarking paradigm that can continuously evaluate the expanding capabilities of frontier models through the physical artifacts they create. Last Bot Standing is one instantiation of this paradigm, chosen for its simple rules, non-saturating competition design, enormous design space, and objectively verifiable outcomes. We intentionally hold the simulator, physical parameters, and game rules fixed (Sec. 2) so that differences in performance reflect the capabilities of the models and the artifacts they produce, rather than changes in the evaluation task itself. Varying the rules or physical environment would help answer a different, equally interesting question: how well do model-generated designs generalize across different physical challenges? While this is beyond the scope of our current work, we believe expanding ArtifactArena to additional environments and design objectives is an exciting direction for future research. Within our current environment, we demonstrate that the leaderboard rankings are highly robust to changes in initial conditions. Replaying the 21 best-performing robots on previously unseen random seeds, which vary their starting positions and orientations, preserves 202 of 210 pairwise model rankings, with a Spearman rank correlation of 0.981 (Sec. 5.1).

What capability or insight is revealed beyond the leaderboard, relative to Sec. 4? The works in Sec. 4 either score construction against a prescribed objective or train bodies and policies through optimization. We evaluate what a general-purpose model builds from a text description when the only target is another model’s robot. This shows three things. 1) Frontier models can zero-shot working robots: with no simulator feedback, GPT-6 Astra’s sampled bot reaches 1374 Elo. 2) More agency does not help yet: DLH (feedback and tool use) trails VGH (iterative feedback) for most models and produces no bot at all for three models. 3) Feedback narrows exploration: SH gives a median of 19 structurally distinct designs against 12 for VGH.

What evidence supports the non-saturating claim? ArtifactArena is designed to be non-saturating through its competitive evaluation paradigm. Unlike traditional benchmarks with fixed questions and a maximum achievable score, ArtifactArena evaluates models relative to an evolving population of competing artifacts. As models become more capable and produce stronger bots, these bots enter the arena as increasingly challenging opponents for future submissions. Thus, improvements in model capabilities raise the difficulty of the benchmark itself, rather than bringing models closer to a fixed performance ceiling. Our current results also both quantitatively and qualitatively indicate that we are far from a ceiling and that models are far from exhausting their design capabilities. The more recently released models continue to demonstrate new creative insights that result in better tournament performance such as gpt-6 astra designing a split flipper.

Reviewer_4dbyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents ArtifactArena, a benchmark for evaluating and ranking AI models' abilities in robot hardware-software co-design. The platform is a simulated environment based on MuJoCo, where robots and their control policies designed by different AI models and harnessing strategies compete in 1-on-1 fighting matches. The results of repeated competitions are used to compute Elo rankings that reflect the overall capabilities and relative rankings of the AI models. Based on this platform, the paper tests a number of frontier AI models under three harnessing strategies. The main contributions of the paper are the design of the platform, the evaluation of the AI models, and the analysis of the results.

Critical strengths

the paper presents a new benchmark focusing on robot hardware-software co-design for AI models. This is a timely and important task.

The paper conducts systematic experiments on several recent frontier models, covering three representative harnessing strategies. The findings are useful for understanding the capabilities of these models.

The paper is well written and easy to follow.

Critical weaknesses

My main concern is that the rationale behind the benchmark and the specific capabilities it measures are not sufficiently clear to me. There are already many benchmarks, some with very specific goals, such as evaluating the correctness of image understanding, reasoning, 3D understanding, coding correctness, etc. The paper frames the proposed ArtifactArena benchmark as an evaluation of physical design capability, which combines several abilities, but the current interface for both design and control is coding-based, it is not easy to disentangle physical and 3D reasoning from code-generation, and interpret its rankings in terms of the underlying capabilities of the models and their harnesses.

The evaluation of different AI models is valuable, but to justify the value and uniqueness of the framework, I think it is more important to explain why this specific game is chosen and what makes it particularly suitable for the intended evaluation. For example, what conclusions would change if the game were replaced with a less competitive task (e.g., each robot completing the same random task separately), a collaborative task, or even if the simulation were replaced with a coding environment (such as CodeClash)?

Questions for authors

see Weaknesses.

1 comment

We thank the reviewer for appreciating that this is a timely and important task and we are glad they found our paper well written and easy to follow.

My main concern is that the rationale behind the benchmark and the specific capabilities it measures are not sufficiently clear to me. There are already many benchmarks, some with very specific goals, such as evaluating the correctness of image understanding, reasoning, 3D understanding, coding correctness, etc. The paper frames the proposed ArtifactArena benchmark as an evaluation of physical design capability, which combines several abilities, but the current interface for both design and control is coding-based, it is not easy to disentangle physical and 3D reasoning from code-generation, and interpret its rankings in terms of the underlying capabilities of the models and their harnesses. The evaluation of different AI models is valuable, but to justify the value and uniqueness of the framework, I think it is more important to explain why this specific game is chosen and what makes it particularly suitable for the intended evaluation. For example, what conclusions would change if the game were replaced with a less competitive task (e.g., each robot completing the same random task separately), a collaborative task, or even if the simulation were replaced with a coding environment (such as CodeClash)?

We believe the reviewer's concern actually highlights an important strength of our benchmarking approach. As AI models become increasingly general-purpose and capable, we believe evaluations must move beyond measuring isolated capabilities toward assessing how models combine diverse skills to solve complex problems. While traditional benchmarks often measure individual capabilities such as coding, visual understanding, or mathematical reasoning, real-world engineering problems require these capabilities to work together. We intentionally designed ArtifactArena to evaluate this integrated problem-solving ability, rather than attempting to disentangle physical reasoning, spatial understanding, coding, and strategic planning. We agree that ArtifactArena's rankings cannot be interpreted as isolated measurements of physical reasoning, spatial understanding, or coding ability. Instead, they measure a model's end-to-end ability to integrate these capabilities into successful physical designs.

We chose physical artifact design for several reasons. First, engineering functional physical systems is closely connected to real-world applications where increasingly capable AI models could provide substantial value. Second, physical design requires models to integrate a diverse set of skills, including spatial reasoning, physics understanding, coding, mechanical design, and control. Third, the enormous space of possible robot morphologies and controllers provides an open-ended design challenge with no prescribed optimal solution, allowing increasingly capable models to demonstrate novel approaches. Our results reveal that frontier models can already generate surprisingly sophisticated and functional physical systems, even without simulator feedback. We believe this emerging capability is both important to measure and particularly interesting to study.

We chose Last Bot Standing because it combines simple and objectively verifiable rules with a vast design space and head-to-head competition. The competitive setting is central to our goal of establishing a non-saturating benchmark: as stronger artifacts enter the arena, they become increasingly challenging opponents for future submissions. A noncompetitive task could also evaluate physical design capabilities, but would not provide this same mechanism for evolving competitive difficulty. Similarly, coding competitions such as CodeClash share our philosophy of head-to-head evaluation, but do not require models to engineer physical morphologies and controllers whose success depends on their interaction under simulated physics. Last Bot Standing therefore serves as one instantiation of our broader goal: evaluating the expanding capabilities of general-purpose models through the functional physical artifacts they can create.

Reviewer_a8kROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ArtifactArena, a benchmark in which language models generate rigid-body robots and feedback controllers for a fixed sumo-style game in MuJoCo. Robots compete by pushing opponents off a platform or immobilizing them under an inactivity rule. Their geometry, materials and actuation must satisfy the arena's constraints, and controllers receive detailed simulator observations of both robots. The paper compares independent sampling, verifier-guided refinement and an autonomous design-lab workflow under a fixed model-call budget. Tournaments rank the strongest generated artifacts, while additional analyses examine generation failures, mechanisms, controllers and design histories.

Critical strengths
  • ArtifactArena demonstrates that language models can generate compatible robot bodies and feedback controllers that function in physical competition. This adds an opponent-dependent objective to the construction tasks in relevant papers on LLM-generated 3D mechanical designs: a design has to contend with another generated system whose behavior affects its success. The executable artifacts can be tested against further opponents without regenerating them, making the platform useful for comparing design-generation methods under a shared simulator and game.

  • Some design histories show useful reasoning from observed failures. Kimi identifies a turning problem and changes its drive arrangement while preserving the wedge strategy. In the design-lab example, Fable creates probe opponents that defeat its current wedge, then adopts changes to its lip and plate. These examples demonstrate both physical debugging and local improvement through adversarial testing.

  • The sampling study exposes a gap between producing a buildable artifact and passing basic physical qualification: only about half of the buildable samples qualify. Reporting these rates alongside champion scores helps distinguish occasional successful designs from dependable generation, which matters when judging how reliably a model can support a design workflow.

  • Replaying the selected champions from unseen starting positions and headings largely preserves their ranking. This provides evidence that the measured differences between those robots persist beyond the original tournament conditions, giving more confidence in their relative competitive performance.

Critical weaknesses
  • The comparison with earlier competitive co-design is incomplete. For instance, CompetEvo [1] jointly optimizes morphology and fighting policies in adversarial games. The paper gives little comparison of the design spaces, use of opponents during generation, or behavioral findings needed to position its evaluation of LLM-generated artifacts.

  • The claim of full creative freedom, limited only by physics, is too strong. Robots must follow fixed sumo rules, use connected rigid parts and preset materials, and their controllers receive detailed information about their opponents. Stronger motors are assigned extra mass, but the arena sets no limits on motor speed, power or energy use. These assumptions shape the design problem, yet the paper gives little analysis of how they influence which designs succeed.

  • The comparison across models describes robot mechanisms and programmed behaviors, but does not explain which choices help them win against different opponents. Counting types of robot structures does not answer this, and the paper does not clearly describe how designs were grouped or which samples were compared. Its detailed failure analysis focuses on a preliminary test against a block with no active controller. We therefore learn little about whether higher-ranked models build more reliable robots, develop designs that work well across opponents, or respond better to particular opponents.

  • While the design-lab example shows that testing against self-created opponents can improve a robot, the study does not show adaptation to robots designed by other models. Each design-lab run is isolated, and the final tournament compares completed robots without further changes. The authors also report that most revisions keep the original strategy and correct implementation problems. The changing pool of opponents emphasized in the motivation therefore plays little part in the design process studied here.

  • Repeating the tournament with new random starting conditions supports the ranking of the selected robots. However, a model's ranking also depends on which robots are generated and selected to represent it. The reported error bars account for variation among finalists and match outcomes, but omit the uncertainty from selecting a finalist within each generation run. As the authors acknowledge, these estimates capture only part of the uncertainty, so small ranking differences between models or design workflows need cautious interpretation.

  • The paper is inconsistent about which models were tested and which robots won. MiniMax appears in the configuration table and larger sampling study but not the leaderboard. The text names Gemini 3.1 Pro's robot as the tournament winner, although the leaderboard puts GPT-6 Astra first. For Gemini 3.1 Pro, the text also favors the design-lab workflow (DLH), while Figure 2 favors independent sampling (SH). The paper does not explain whether these statements refer to different experiments or definitions.

  • While Figure 4 shows useful examples of improvement through feedback, the paper gives limited evidence that these design histories are representative. The appendix adds brief examples, but there is little comparison of how different models’ designs and results change over successive revisions. Histories from more models and runs would help establish whether the reported patterns are common or specific to the selected examples.

References:

[1] CompetEvo: Towards Morphological Evolution from Competition. Kangyao Huang et al. IJCAI, 2024. https://www.ijcai.org/proceedings/2024/0010.pdf

Questions for authors
  1. What do the recorded tournament outcomes and trajectories show about the design choices or player decisions responsible for the main performance differences?

  2. How were the structural categories assigned and checked, and what population was used for the sampling-versus-refinement diversity comparison?

  3. Which models belong to each experiment, and how should the conflicting statements about the winning model and generation method be interpreted?

4 comments

We thank the reviewer for their detailed review and for recognizing the contributions of ArtifactArena as a physically grounded competitive benchmark. We address their concerns and questions below.

The comparison with earlier competitive co-design is incomplete. For instance, CompetEvo [1] jointly optimizes morphology and fighting policies in adversarial games. The paper gives little comparison of the design spaces, use of opponents during generation, or behavioral findings needed to position its evaluation of LLM-generated artifacts.

We thank the reviewer for pointing us to CompetEvo and will add it to our related work. While both works investigate competitive robot co-design, their objectives and methodologies differ significantly. CompetEvo focuses on optimizing robot morphologies and fighting policies through reinforcement learning, whereas ArtifactArena focuses on evaluating the physical design capabilities of general-purpose language models. Our design space is also substantially more open-ended: CompetEvo optimizes 20–40 parameters of predefined robot morphologies, whereas ArtifactArena allows models to construct diverse robot morphologies and generate their own Python controllers, rather than training neural-network control policies.

The claim of full creative freedom, limited only by physics, is too strong. Robots must follow fixed sumo rules, use connected rigid parts and preset materials, and their controllers receive detailed information about their opponents. Stronger motors are assigned extra mass, but the arena sets no limits on motor speed, power or energy use. These assumptions shape the design problem, yet the paper gives little analysis of how they influence which designs succeed.

We will adjust the language in our paper to be more precise. We intended to convey that the morphology and controller design spaces are huge, allowing for the study of models' abilities to creatively solve open-ended problems. A brute force search through all the possibilities of the design space would be impossible.

The comparison across models describes robot mechanisms and programmed behaviors, but does not explain which choices help them win against different opponents. Counting types of robot structures does not answer this, and the paper does not clearly describe how designs were grouped or which samples were compared. Its detailed failure analysis focuses on a preliminary test against a block with no active controller. We therefore learn little about whether higher-ranked models build more reliable robots, develop designs that work well across opponents, or respond better to particular opponents.

We can learn from the overall benchmarking results. We release the full set of bots generated which will allow others to investigate this question without spending thousands of dollars to generate bots. In our paper, we provide several case studies of some of the most surprising and innovative morphologies that models came up with as well as the most surprising and innovative controller strategies. In the appendix, we included a dichotomous key, a tool that can be used to identify organisms or objects in the natural world applied to model-generated bots. It provides a categorization of morphology patterns just as biologies would group and classify species of animals by phenotypes.

In a broader sense, we believe the reviewer is getting at the question -- it looks like the models are getting more “intelligent” but what is more “intelligent” and how is that externalized into creating a better bot? This is an incredibly difficult question to answer which our community of AI researchers as a whole is grappling with. In this work we contribute to the endeavor of understanding model intelligence by creating a non-saturating relative ranking of model abilities based on a highly open-ended complex task with a verifiable outcome determined by the simulation environment state. We then 1) generate fully interpretable artifacts (python programs that control the bots, and xml files that fully define the bot morphology) which are vastly easier to investigate than sequences of LLM generated actions or trained RL policies in prior works and provide in our paper a case study of some of some of the most innovative and surprising model artifacts. 2) we release everything that the models generated allowing others to study them further 3) We establish through our benchmark an overall ranking of model abilities on our task. It does appear that models that are more recently released and that have ranked higher on other benchmarks of intelligence perform better on our task. Much like the reviewer we find that absolutely fascinating. What does a more performant model know and reason through that a less performant one does not? Could more intelligence models be thinking more steps into the future to anticipate enemy strategies? Could they understand physics or 3D spatial reasoning at a higher level allowing them to create more robust or exploitative design? Could the models to some degree be expressing their unique personalities through their tradeoff between loss aversion and gain seeking (for example, fable never used SDFs to create custom meshes describing the added complexity as too risky)? Much like a biologist, we set up the conditions for studying our "organisms" through a principled innovative framework where we can push them to demonstrate their abilities and surface their uniqueness. Also like in biological research, our approach advances the field, and provides insights, creating important artifacts for other scientists to study further, but it is impossible to fully solve the field's largest mysteries. We believe this work offers significant benefit to the research community with its current level of analysis and open sourcing of results, and our paper's largest contributions are in its principled benchmark design methodology which allows for ongoing further analysis of how intelligence translates to functional artifacts to be possible.

While the design-lab example shows that testing against self-created opponents can improve a robot, the study does not show adaptation to robots designed by other models. Each design-lab run is isolated, and the final tournament compares completed robots without further changes. The authors also report that most revisions keep the original strategy and correct implementation problems. The changing pool of opponents emphasized in the motivation therefore plays little part in the design process studied here.

We clarify that the evolving pool of opponents serves as the basis for our competitive evaluation, rather than as feedback during artifact generation. By evaluating completed robots against independently designed opponents, we can compare model capabilities under a common, increasingly challenging benchmark. We agree that allowing models to iteratively improve their designs against robots generated by other models would be an exciting extension of ArtifactArena, and we would like to pursue it in the future. However, studying this form of competitive co-evolution is beyond the scope of the current work due to its significantly increased complexity and computational requirements. In particular, models would need to optimize their designs against a diverse population of opponents to develop broadly competitive robots, rather than designs that exploit the weaknesses of a single opponent.

Repeating the tournament with new random starting conditions supports the ranking of the selected robots. However, a model's ranking also depends on which robots are generated and selected to represent it. The reported error bars account for variation among finalists and match outcomes, but omit the uncertainty from selecting a finalist within each generation run. As the authors acknowledge, these estimates capture only part of the uncertainty, so small ranking differences between models or design workflows need cautious interpretation.

We agree with the reviewer that our reported confidence intervals do not capture all sources of uncertainty as we do not account for the selection of the best robot within each generation run. We acknowledge this in Sec. 2.1. Fully accounting for this additional uncertainty would require substantially more computation than our budget allowed.

The paper is inconsistent about which models were tested and which robots won. MiniMax appears in the configuration table and larger sampling study but not the leaderboard. The text names Gemini 3.1 Pro's robot as the tournament winner, although the leaderboard puts GPT-6 Astra first. For Gemini 3.1 Pro, the text also favors the design-lab workflow (DLH), while Figure 2 favors independent sampling (SH). The paper does not explain whether these statements refer to different experiments or definitions.

We thank the reviewer for catching these. The statements refer to different experiments and different rating fits, and we did not make this clear.

(1) Models. The zero-shot sampling study (Sec. 3.2) covers 23 models and 5,750 calls: the 21 leaderboard models plus MiniMax M3 and Claude Opus 5. The text incorrectly says 21 models and 5,250 calls; the reported totals of 4,384 built and 2,242 qualified are for all 23. The harness runs and final tournament (Figure 2) cover 21 models. MiniMax M3 and Claude Opus 5 were included in the broader zero-shot sampling study but excluded from the main 21-model harness comparison due to generation failures. Claude Opus 5 frequently exhausted its token budget on reasoning without producing a robot artifact, while 227 of MiniMax M3's 250 samples failed validation, with only one passing qualification. Given these issues, we excluded both models from the main harness comparison and leaderboard. We will clarify this in the final paper.

(2) Winner. The sentence in Sec. 3.2 refers to the tournament among the sampled bots, where Gemini 3.1 Pro’s bot ranked first. In the final cross-model tournament, the winner is GPT-6 Astra’s VGH artifact, as Figure 2 and Sec. 3.1 state. We will rewrite the sentence to say which tournament it means.

(3) Harness. Sec. 3.1 reported the strongest harness using ratings fit over all finalists, while Figure 2 shows ratings refit over each model’s champion only. Under the first, VGH is strongest for 16 models and DLH for Gemini 3.1 Pro; under the second, VGH is strongest for 15 and SH for Gemini 3.1 Pro. For Gemini 3.1 Pro and GPT-5.5, the gap between the top two harnesses is a few Elo points, far inside the error bars, so the ordering is not resolved. We will use the champion fit throughout and will not name a best harness where intervals overlap.

We will also add a table listing the models and sample counts for every experiment.

While Figure 4 shows useful examples of improvement through feedback, the paper gives limited evidence that these design histories are representative. The appendix adds brief examples, but there is little comparison of how different models’ designs and results change over successive revisions. Histories from more models and runs would help establish whether the reported patterns are common or specific to the selected examples.

We computed the histories for all 63 VGH runs (21 models × 3 runs × 10 revisions). The patterns in Figure 4 hold in aggregate. The number of runs with a qualifying robot rises from 28 of 63 at revision 1 to 49 of 63 at revision 10. In 60 of 63 runs, the selected revision outscores revision 1 in the run’s own round robin, and in 38 runs, the selected revision comes from revisions 6 to 10. Revision 1 is selected in only 3 runs. 19 of 21 models reach a qualifying design in every run. We will add a per-model table to the appendix, and the full history of every run is released in the Bot Zoo.

What do the recorded tournament outcomes and trajectories show about the design choices or player decisions responsible for the main performance differences?

Two results in the paper speak to this. First, performance is dominated by overall strength and not by matchup-specific counters: the top artifact wins 92.5% of its matches across all opponents. Second, the differences come mostly from how well a strategy is implemented, not from which strategy is chosen. Most models choose similar strategies (208 of the 509 buildable sampled robots use powered scoops or plows), and during refinement models mostly keep their initial strategy and use feedback to fix its physical implementation. Kimi K3 kept the same wedge strategy for ten revisions and went from winning 2 of 15 qualification rounds to 3 of 3 by changing only the drive layout. For additional examples of specific strategies, design choices, and how they evolve during refinement, please see Section 3.4 What makes some models creative? and Appendix 7.8 Physical implementation across revisions

How were the structural categories assigned and checked, and what population was used for the sampling-versus-refinement diversity comparison?

For the sampling-versus-refinement diversity comparison, we used the same population of robots as in the Figure 2 leaderboard. Structural categories were developed iteratively using an LLM to identify distinguishing design features and define classification criteria. We then manually spot-checked both the criteria and the resulting category assignments for qualitative consistency and correctness.

Which models belong to each experiment, and how should the conflicting statements about the winning model and generation method be interpreted?

Please see our response above on inconsistent models and winners.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.