- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
ArtifactArena asks LLM to build MJCF robot body in MuJoCo and a Python controller to help control the LLM-designed robot body. Then the author compare LLMs' robot design and controling ability by evaluating them in Last Bot Standing game. The author use ELO score to rank all model. To help LLM design the robots, the author design three harness: Sampling Harness (SH), Verifier-Grounded Refinement Harness (VGH), and Design Lab Harness (DLH). The author analysis the 'Hardware creativity', 'Controller creativity', and running examples based on the experiment.
Critical strengths
- The Last Bot Standing game evaluates robot artifacts through head-to-head competition, which is a reasonal game setting to compare the robot's ability in a relative limited perspective. Based on robot's battle performance, it is a good way to estimate the creativity of LLM's robot artifact design and streatgy design. Section 4 distinguishes this competitive setting from prescribed construction objectives and acknowledges prior co-design and competitive-evolution work. This part of paper contribution is clear as a benchmark design.
Critical weaknesses
- The benchmark claims is limited validated. The abstract describes the paper a "living, non-saturating testbed to continuously measure the expanding limits of open-ended intelligence in the physical world". However, the paper's experiment and analysis only based on Last Bot Standing under MuJoCo simulation, which is hard to support the evaluation of "open-ended intelligence".
- The comparison supports the reported performance of three configurations (SH, VGH and DLH), but does not isolate which design choices explain the differences. The author may consider add the ablation study to compare the harness difference acroess fixed set of models and the model difference with the fixed configuration of harness. Figure 2 already compares models across SH, VGH, and DLH. However, the harnesses change feedback, tool access, and artifact selection together. Component ablations and measured token and simulation costs would help explain the performance differences.
- My another major concern is about paper writing and presentation. Figure 7, captioned 'Dichotomous key of SH generated bots', is difficult to read at normal page size: its dense branches, small labels, and robot thumbnails do not provide an accessible explanation of the taxonomy or classification criteria (Appendix 5.5). Section 3.3 visibly leaves dollar signs around code identifiers such as is_upside_down. The full prompts in Section 10 are useful for reproducibility, but the main text needs a concise summary of the consequential prompt and harness choices, with precise references.
Questions for authors
-
I am queries about the token/dollar cost for a LLM to complete each task. Would different harness's cost differ across models? And would stronger models have better performance is because they are better than reasoning and creating or it just have longer reasoning? Besides the leaderboard, author may consider offer a figure that shows the perato frontier lines to help better understand the relation between elo performance and token cost.
-
Which findings remain stable when the Last Bot Standing rules, MuJoCo parameters, or design objectives change? Beyond the ArtifactArena Leaderboard, what specific capability or empirical insight is revealed by this hardware-software co-design evaluation, relative to the work discussed in Section 4? What evidence supports the 'non-saturating testbed' claim?
3 comments
My another major concern is about paper writing and presentation. Figure 7, captioned 'Dichotomous key of SH generated bots', is difficult to read at normal page size: its dense branches, small labels, and robot thumbnails do not provide an accessible explanation of the taxonomy or classification criteria (Appendix 5.5). Section 3.3 visibly leaves dollar signs around code identifiers such as is_upside_down. The full prompts in Section 10 are useful for reproducibility, but the main text needs a concise summary of the consequential prompt and harness choices, with precise references.
We thank the reviewer for catching the dollar signs in Sec. 3.3. This is a typesetting error and we will fix it. We respectfully note that Figure 7 is an appendix figure. It is a supplementary vector graphic that classifies every sampled bot and is meant to be zoomed, and none of the paper’s claims depend on reading it at page size. We will add text to Appendix 5.5 that lists the key questions and explains how bots are classified. On the summary of prompt and harness choices, the main text already contains this: Sec. 2.1 (“Model Task”) summarizes what the prompt tells the model, and Sec. 2.3 summarizes the three harnesses. Our guiding principle in designing our prompts was to be as descriptive as possible providing complete information about the game rules and environment while not biasing the model towards any specific bot design, controller functions, or high level game strategies. We allowed models to derive those entirely on their own from first principals based on our description of the task and environment.
We would like to ask the reviewer to be more precise about the presentation concern. The review rates the paper “difficult to follow despite careful reading”, but the specific issues listed are an appendix figure, a typesetting error in one sentence, and a summary that is already present in the main text. Reviewer 4dby writes that our paper “is well written and easy to follow”. If there are sections of the main text that the reviewer found hard to follow, we would be grateful if they could point to them, and we will revise them. We will of course correct small mistakes such as the dollar signs, but we do not believe that they, or the legibility of an appendix figure, justify treating the presentation as a major weakness of the paper.
I am queries about the token/dollar cost for a LLM to complete each task. Would different harness's cost differ across models? And would stronger models have better performance is because they are better than reasoning and creating or it just have longer reasoning? Besides the leaderboard, author may consider offer a figure that shows the perato frontier lines to help better understand the relation between elo performance and token cost.
In our experiments, we allocate the same budget of 10 model API calls per run to each harness, with three independent runs per model–harness combination (Sec. 2.3). Matching the number of API calls provides a standardized interaction budget for comparing models and harnesses, although reasoning time and dollar costs may differ.
Our benchmark evaluates the quality of artifacts produced under this standardized API call budget, rather than attempting to isolate reasoning ability from computational effort. Such isolation is particularly difficult for closed-source models, where internal reasoning processes and computational effort are not fully observable or controllable. Moreover, our evaluation reflects how these models are used in practice, where what ultimately matters is the quality of the artifacts they produce rather than how much internal reasoning they perform.
To address the reviewer's question about computational efficiency, we provide an additional cost–performance analysis figure comparing the Elo performance of different model–harness combinations against their generation costs. Cost–Performance Analysis Figure
Which findings remain stable when the Last Bot Standing rules, MuJoCo parameters, or design objectives change? Beyond the ArtifactArena Leaderboard, what specific capability or empirical insight is revealed by this hardware-software co-design evaluation, relative to the work discussed in Section 4? What evidence supports the 'non-saturating testbed' claim?
Our primary goal is to establish an open-ended, non-saturating benchmarking paradigm that can continuously evaluate the expanding capabilities of frontier models through the physical artifacts they create. Last Bot Standing is one instantiation of this paradigm, chosen for its simple rules, non-saturating competition design, enormous design space, and objectively verifiable outcomes. We intentionally hold the simulator, physical parameters, and game rules fixed (Sec. 2) so that differences in performance reflect the capabilities of the models and the artifacts they produce, rather than changes in the evaluation task itself. Varying the rules or physical environment would help answer a different, equally interesting question: how well do model-generated designs generalize across different physical challenges? While this is beyond the scope of our current work, we believe expanding ArtifactArena to additional environments and design objectives is an exciting direction for future research. Within our current environment, we demonstrate that the leaderboard rankings are highly robust to changes in initial conditions. Replaying the 21 best-performing robots on previously unseen random seeds, which vary their starting positions and orientations, preserves 202 of 210 pairwise model rankings, with a Spearman rank correlation of 0.981 (Sec. 5.1).
What capability or insight is revealed beyond the leaderboard, relative to Sec. 4? The works in Sec. 4 either score construction against a prescribed objective or train bodies and policies through optimization. We evaluate what a general-purpose model builds from a text description when the only target is another model’s robot. This shows three things. 1) Frontier models can zero-shot working robots: with no simulator feedback, GPT-6 Astra’s sampled bot reaches 1374 Elo. 2) More agency does not help yet: DLH (feedback and tool use) trails VGH (iterative feedback) for most models and produces no bot at all for three models. 3) Feedback narrows exploration: SH gives a median of 19 structurally distinct designs against 12 for VGH.
What evidence supports the non-saturating claim? ArtifactArena is designed to be non-saturating through its competitive evaluation paradigm. Unlike traditional benchmarks with fixed questions and a maximum achievable score, ArtifactArena evaluates models relative to an evolving population of competing artifacts. As models become more capable and produce stronger bots, these bots enter the arena as increasingly challenging opponents for future submissions. Thus, improvements in model capabilities raise the difficulty of the benchmark itself, rather than bringing models closer to a fixed performance ceiling. Our current results also both quantitatively and qualitatively indicate that we are far from a ceiling and that models are far from exhausting their design capabilities. The more recently released models continue to demonstrate new creative insights that result in better tournament performance such as gpt-6 astra designing a split flipper.
We thank the reviewer for their review and address their questions and weaknesses below.
We will adjust the language in our abstract to be more precise. By “open-ended” we were referring to the vast solution space of possible robot morphologies and controllers that models can design towards accomplishing the goal of winning the last bot standing task. We are not measuring a model’s ability to discover its own tasks, but in our benchmarking paradigm we are asking the models to solve a problem for which, even we the benchmark designers, do not know the optimal solution. The massive design space and absence of a known optimal require models to explore, reason about, and develop novel solutions, making this a test of open-ended physical design and problem-solving capabilities.
Figure 2 is the comparison the reviewer describes. All 21 models are run under all three harnesses, so harness differences can be read with the model fixed, and model differences with the harness fixed.
Artifact selection does not change across harnesses. Each harness generates multiple candidate bots and the best-performing bot is selected through tournaments among those candidates as described in (Secs. 2.3 and 6.2)
In this work we make progress towards answering how does the performance of frontier models change when they are given different levels of control over the process of designing physical artifacts? Feedback and tool access do change together, and this is intentional. Each harness represents a level of control the model has over the design loop: no feedback (SH), verifier and match feedback (VGH), and full agency with tools (DLH). We do not claim to have exhaustively evaluated every possible harness, but we believe the three harness we strategically designed to include in the paper help measure how different levels of model autonomy and access to feedback affect the quality of generated artifacts, while establishing a systematic baseline for future research into more advanced harness designs. Our arena is harness-agnostic, and we release the harnesses so the community can build and compare even better ones.
Also, just to clarify we match the number of api calls made per harness and all three harnesses are derived from the exact same base prompt that fully describes the environment and game rules. VGH and DLH add additional sanctions to this prompt to describe their harness specific descriptions such as feedback and tool use.