acceptodds
Under review as a conference paper at ICLR 2027

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

25 views

Abstract

Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: **Engine Grounding**, **Artifact Completeness**, and **Interactive Verification**. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as **GameCraft-Bench**, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 68.44%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. All code, data, demos are provided in the supplementary material.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, Benchmark, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3b4uOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GameCraft-Bench presents a benchmark, or in my opinion more accurately a methodology, towards evaluating end to end game development. It focuses on godot and evaluates using actual execution of the game rather than code analysis.

Critical strengths
  1. End to end game development evaluation is challenging and this provides a study into a seemingly reasonable framework.
  2. The human judgement comparison with LLM judge is useful to establishing the effectiveness of the judge. This is often missing from benchmarks that use LLM judges so I do appreciate this.
  3. The findings and insights are interesting and useful, especially in this niche domain.
Critical weaknesses
  1. Why are the three desiderata necessary? The paper spends significant effort for 6 pages claiming the three desiderata are important, but does not justify these claims. There are no experiments backing these claims up nor are there any citations / references to game development literature actually proving this point.
  2. The paper states this: "The central claim is that game generation cannot be evaluated as ordinary code synthesis". Yet, I see no experimentation suggesting that these results could not be done with pure code evaluation? There is no head to head comparison anywhere. Essentially, it's a similar problem to the first weakness. [edit] FYI, this seems like a similar weakness (1) by vtFk
  3. A major problem with LLM judge papers is that the scores themselves have no real meaning. It's unclear to me what any X% of functional visual correctness means. From the appendix, it seems that there's a rubric of y values each of which has its own mean, which are then combined into X. But what does it mean if the model scores 100% on visuals? Is it something like halo? like cyberpunk? are the graphics like real life? Some sort of scale would provide better insights.
  4. Another concern is the size of the dataset. 140 tasks / games is small for a benchmark.

Ultimately, a lot of information and justification is currently missing. Some of this may simply be a writing exercise. For example, for (2), I would suggest referencing recent work that has gone through the effort to show that interactive evaluation is better than static code llm as a judge. Both of these are perhaps relevant: [1] Interaction-as-a-Judge: https://arxiv.org/abs/2610.17895 [2] The Verification Horizon: https://arxiv.org/abs/2606.26300

For points (1) and (3) I would hope to see some sort of justification or, ideally, additional experiments to support these in the rebuttal.

Questions for authors
  1. How are the rubric items generated? Auto-generated via LLM? Human designed?
  2. How are the task instructions generated?
5 comments

Thank you for recognizing the challenges of end-to-end game development evaluation and the value of our framework, human calibration, and empirical findings. We appreciate your suggestions and address them below.


Weakness 1: Necessity of the Three Desiderata

We appreciate this concern and agree that the motivation deserves stronger justification.

Our three desiderata define complementary requirements for evaluating end-to-end game generation in a real engine, rather than universal requirements for all game benchmarks:

  • Engine Grounding: Ensures that generated games function in their intended engine environment, rather than merely producing plausible code.
  • Artifact Completeness: Requires a complete, launchable game rather than isolated mechanics or partial implementations.
  • Interactive Verification: Evaluates actual player-facing behavior, capturing failures that static inspection may miss.

Weakness 2: Static Code Evaluation vs. Interactive Evaluation

Thank you for suggesting a direct comparison. We randomly selected 10 tasks from our 140-task benchmark and evaluated the corresponding games generated by GPT-5.6-sol high. We compared Code-only Judge and Agent-authored Replay Judge against human gameplay assessments, keeping the same game artifacts and task-specific rubrics across all evaluation methods.

Evaluation MethodMDVATask-level MAE ↓
Code-only Judge66.0063.2561.2558.758.15
Replay-based Judge (Ours)71.9254.7752.5049.633.78
Human Gameplay68.2152.2453.2248.73—

Replay-based evaluation reduces task-level MAE by approximately 54% compared with Code-only judging. Notably, Code-only judging overestimates Content Depth, Functional Visuals, and Art, whereas Replay-based judging more closely matches human assessments. This provides empirical support for the value of evaluating executable gameplay rather than relying solely on source code.

We also thank the reviewer for recommending Interaction-as-a-Judge and The Verification Horizon. We will cite these works to strengthen our motivation.

Weakness 3: Interpretation of Evaluation Scores

We agree that the score semantics should be clarified. GameCraft-Bench measures specification-relative rubric satisfaction, not absolute game quality or commercial production fidelity.

Each task contains explicit requirements evaluated on a 0–1 scale:

ScoreInterpretationExample: Functional Visuals
0Requirement not demonstrated or contradictedCritical gameplay states are visually indistinguishable
0.5Partially demonstrated or ambiguousSome visual feedback works, but important states remain unclear
1.0Clearly demonstratedAll specified visual states and feedback are clearly distinguishable

Thus, 100% Functional Visuals means that all task-specific visual functionality requirements receive full credit under the evaluated evidence. It does not imply photorealism or AAA-level graphics. Similarly, Art & Presentation measures consistency with the requested art direction, rather than comparison against commercial games such as Halo or Cyberpunk. We will clarify this scoring interpretation and provide concrete examples from generated games at different score levels.

Weakness 4: Dataset Size

We acknowledge that 140 tasks cannot exhaustively represent all game-development scenarios. However, our benchmark contains 140 end-to-end game-generation tasks across 15 game families, rather than isolated code-editing problems. Each task requires constructing a complete project, including gameplay mechanics, content, assets, and UI. This makes task construction and evaluation substantially more demanding than localized code-editing benchmarks. We will clarify the benchmark's intended coverage and acknowledge its current limitations, including uneven task counts across game families.

Question 1 & Question 2: Rubric and Task Instruction Construction

Both task instructions and evaluation rubrics were developed by 12 human annotators, as described in Appendix F. Annotators were allowed to use AI tools for assistance, but were required to independently draft the game specifications and define the rubric criteria themselves, rather than relying on AI-generated requirements.

For each task, annotators prepared:

  • A natural-language game specification describing the Core Vision, gameplay loop, and intended player experience.
  • A hidden rubric translating the specification into observable requirements across Mechanics, Depth, Functional Visuals, and Art & Presentation.
  • A minimal playable Godot oracle to validate feasibility and ensure that the requirements could be demonstrated and evaluated.

Dear Reviewer 3b4u,

As the discussion deadline approaches, we would be very grateful for any further thoughts or feedback on our rebuttal and additional experiments. We sincerely appreciate the time and effort you have devoted to reviewing our work and would be happy to address any remaining concerns or questions.

Best regards,
Authors

  1. I'm glad that you agree this deserves stronger justification, but your response doesn't answer my question nor address my point. This is just restating what's already in the paper. I'm looking for actual evidence that these 3 are what's important and not others. For example, why is "Visual Fidelity" not a desiderata? Or "Solvability" of the game? I'm not saying these are desiderata, but it's completely unclear to me why the three you picked should be the three desiderata for game development.
  2. This helps, but a) the sample size is very small and b) how exactly human gameplay annotation is done is very unclear. Also, I was expecting something related to your table 5 but it seems this is an entirely different experiment? Didn't you already do some human vs judge comparisons? Is this on building on top of that or something different?
  3. Again, helpful and should be in the paper. Unfortunately, it's still very unclear to me what 100% on these rubric items means. It's not photorealistic or AAA I get, but... what is it? Is it at the level of an AA game? an indie developer who spent a year? What exactly is the ceiling here? Is 100% something I could expect to see sold on any game store? Something I see on a flash games website? There is just no grounding for quality of the game that I can see

lastly, I don't see any of these edits actually in the manuscript?

Unfortunately, I will keep my score as is.

Reviewer_8qHeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The benchmark tries to make end-to-end game development the task for AI models. The benchmark has an impressive list of 140 tasks.

Critical strengths

The desiridata around Engine Grounding, Artifact Completeness, and Interactive Verification cleanly exposes why prior benchmarks (e.g., localized edits in GameDevBench or static/web-based OpenGame-Bench) fall short of testing end-to-end game engineering.

I also like how the agents have to submit behavior traces of the gameplay as part of the evaluation.

Critical weaknesses

The benchmark seems prone to reward hacking, if the agent is creating the code and also the gameplay data for evaluation, it could make its own custom scenarios to fool the LLM judge. Could it jump straight to states or between states rather than using true physical simulation. It seems like the authors need to characterize how this could be hacked, as modern agentic systems reward hack like crazy.

Ideally, we could use human data both at the gameplay level and the judge level for calibration to identify if the games are working correctly. The authors do include a small human judge calibration in Section 5.2 (5 annotators grading Kimi-K2.6 across 3 of the 15 genres), but humans only graded the agent's pre-recorded traces rather than actually playing the games themselves. Even in that narrow check, GPT-5.5 overscored Depth by +12.19 points and Art by +11.50 points on Idle games while underscoring Functional Visuals by -5.87 points (likely because videos are compressed to 480p even though the rubric asks to judge readability at 1280x720), and the paper only reports mean differences without task-level correlation or inter-annotator agreement.

A weird part of the eval that I didn't notice at first: Agents never actually play the games interactively; instead, they have to blindly write static scripts of timed inputs, which conflates game development with open-loop choreography. A fully playable game gets a zero if the agent mistimes a scripted click, while a broken game can score high by hardcoding visual states to match its own script.

Also Table 3 mixes four different harnesses (Claude Code, Codex, Kimi Code, and Code Buddy) and compares vision models that can use screenshot.sh against text-only models that cannot, despite 70% of the score coming from visual Depth and Art. Without holding the harness constant across models or testing how much the scaffold itself drives performance (e.g., running the same base models across Claude Code, Codex, or Antigravity), we cannot tell how much of the leaderboard gap is the model versus the harness.

Questions for authors

Since the visual judge only reviews 2 FPS compressed frames from agent-created traces, what prevents an agent from completely hardcoding a visual sequence instead of programming real game mechanics?

How do you justify comparing models on completely different developer harnesses, given how significantly scaffolding affects coding success rates?

Why were the top 4 models (Opus-5, Fable-5, GPT-5.6-sol, Kimi-K3) entirely omitted from the build and trace reliability diagnostics in Figure 9?

3 comments

Thank you for your constructive feedback and for recognizing the importance of evaluating end-to-end game development and the value of agent-authored gameplay traces. We appreciate your concerns regarding reward hacking, human calibration, demonstration reliability, and harness fairness.

We address them below with additional experiments and analyses.


Weakness 1 & Question 1: Reward Hacking and Hardcoded Gameplay

We agree that agent-authored demonstrations introduce a potential reward-hacking risk, particularly if the evaluation protocol is directly used as a reward signal for RL training. Our design represents a trade-off between evaluation scalability and exhaustive gameplay verification. To investigate demonstration-selection effects, we randomly selected 10 tasks from the benchmark and compared agent-authored replay with independently generated verifier-authored replay on the corresponding games generated by GPT-5.6-sol high. The game artifacts and task-specific rubrics were kept fixed across evaluation methods.

Evaluation MethodMDVATokens per problem
Agent-authored Replay (Ours)71.9254.7752.5049.63123,191
Verifier-authored Replay77.0561.2154.6552.62334,234

Our interesting finding is that Verifier-authored Replay, where an independent verifier reads the game source code and rubrics before writing traces, may actually introduce greater selection bias. The verifier tends to select traces that specifically satisfy the rubric requirements, resulting in higher scores.

We acknowledge that replay-based visual judging alone cannot categorically prevent hardcoded visual sequences or artificial state transitions. Motivated by this limitation, we have also been exploring Code + Test Cases + Replay judging for RL-oriented evaluation, where source-code evidence complements observed gameplay to reduce reliance on visual demonstrations alone.

Weakness 2: Human Calibration

We apologize for the ambiguity in our original description. Human annotators did directly launch and play the generated games, rather than merely watching prerecorded traces.

Unfortunately, the original task-level annotation records were lost on our legacy server, preventing retrospective agreement analysis. To address this concern, we conducted an additional 10-task human gameplay evaluation, where one annotator directly played each game and scored it against the same rubrics.

MetricMDVA
Human Gameplay68.2152.2453.2248.73
Judge71.9254.7752.5049.63
Judge − Human+3.71+2.53−0.73+0.90
Task-level Pearson r0.9690.9590.9790.971
Task-level MAE3.963.773.423.33

The judge achieves strong task-level correlations with human gameplay assessments across all four dimensions (r = 0.959–0.979), with mean absolute errors of 3.33–3.96 percentage points. We acknowledge that this additional calibration is limited to 10 tasks and a single annotator due to the limited time available during the discussion phase.

Weakness 3: Open-loop Choreography

We acknowledge that fixed input traces introduce a demonstration-authoring requirement. However, gameplay testing and QA are integral parts of game development, and our protocol offers a scalable, reproducible, and low-cost way to verify interactive behavior.

Importantly, agents are not restricted to blindly writing traces. Our analysis of actual development trajectories shows that frontier agents frequently run and interactively test their games before submission:

Agent ConfigurationGame ExecutionInteractive Input / Replay TestingAvg. Interactive Calls / Task
Opus-5 xhigh (Claude Code)97.9%90.0%22.82
Fable-5 high (Claude Code)100.0%79.5%10.56
GPT-5.6-sol high (Codex)100.0%26.4%0.75

This demonstrates that interactive gameplay validation is already part of agents' development workflows. We also investigated the impact of trace authoring by comparing the original agent-authored traces with independently generated verifier-authored traces on the same 10 games, as discussed in Weakness 1. Verifier-authored replay scores higher on 9/10 tasks, confirming that demonstration selection influences the observed score. We acknowledge that fixed traces cannot exhaustively cover all gameplay paths, which remains a limitation of our scalable evaluation protocol.

Weakness 4 & Question 2: Harness Fairness and Text-only Models

Thank you for raising this concern. We clarify that GameCraft-Bench evaluates end-to-end agent configurations, rather than isolated base models. We prioritize each model provider's official harness to reflect its intended deployment setting:

Model ProviderHarness
AnthropicClaude Code
OpenAICodex
MoonshotAIKimi Code
Other modelsCompatible available harnesses

For example, DeepSeek did not have an official coding harness available when our evaluation was conducted. We agree that harness differences can affect performance. Therefore, our leaderboard measures agent configuration performance, and we do not attribute score differences solely to underlying models.

We also clarify that text-only models have the same access to screenshot.sh. Although they cannot directly interpret images, some agents actively convert screenshots into textual representations to obtain visual feedback. In cardgame-spire-descent, DeepSeek-V4-Flash-0731 (Claude Code) independently implemented ascii_render.py to convert screenshots obtained via screenshot.sh into ASCII representations. After inspecting the ASCII map, the agent identified that locked nodes were too dim to reveal the path structure. It then modified map_node.gd to reduce node darkening and captured another screenshot, confirming that the map structure became visible.

Question 3: Missing Top-4 Execution and Trace Diagnostics

Thank you for pointing out this omission. We have recovered the available execution records and extended Figure 9 with the following results:

ConfigurationBuild Pass RateValid Trace RateDemo CountDemo Duration (s)
Opus-5 xhigh (Claude Code)97.8%97.7%7.4918.0
Fable-5 high (Claude Code)100.0%94.9%7.0016.4
GPT-5.6-sol high (Codex)100.0%100.0%5.6614.0
Kimi-K3 (Kimi Code)————

Kimi-K3's leaderboard results were provided by the MoonshotAI team through their official internal self-evaluation. Since we do not have access to their execution trajectories or generated game artifacts, we cannot independently recover the corresponding reliability statistics.

Reviewer_8qHeOpenReview ↗

Response to Rebuttal

Thank you to the authors for the detailed response and the additional analyses.

Overall, I continue to view this as a valuable benchmark contribution, and I am maintaining my score.

Reviewer_vtFkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GameCraft-Bench, which evaluates whether coding agents can construct complete Godot games from natural language specifications. The benchmark comprises 140 Godot tasks across 15 game families. The evaluation aims to cover three aspects: Engine Grounding, Artifact Completeness and Interactive Verification. It grounds development in a real game engine (Godot), evaluates complete launchable game projects rather than partial artifacts, and verifies generated games through replayed interactions instead of static inspection alone. With the benchmark, this paper evaluates 13 agent configurations, reports a best weighted score of 68.44%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation.

Critical strengths
  1. The benchmark addresses an important problem by jointly considering Engine Grounding, Artifact Completeness, and Interactive Verification. Together, these desiderata shift evaluation from code correctness to interactive system correctness, assessing whether coding agents can transform natural-language game specifications into playable game experiences.
  2. The evaluation protocol combines agent-authored demonstrations with verifier-side replay and generator-blind judging. The builder agent, which knows its controls and progression, submits demonstrations targeting relevant gameplay phases, while the verifier performs standardized replay.
  3. The evaluation of 13 state-of-the-art coding-agent configurations highlights substantial remaining challenges in game generation and provides useful baselines and diagnostic insights for improving future systems.
  4. The judge is also tested with human alignment to demonstrate its robustness and effectiveness.
  5. This paper is well written and easy to follow.
Critical weaknesses
  1. The design of combining agent-authored demonstrations with verifier-side replay and generator-blind judging is essential, but no ablation is performed to support that this judge design works better.
  2. Although the paper reports judge repeatability and preliminary human calibration, it does not adequately distinguish judgment errors from information lost through sparse frame sampling. Comparisons using different sampling rates and human/model judgments on identical evidence would help determine whether evaluation errors arise from the judge or from insufficient visual evidence.
  3. With the given judge design, it's also unclear how much of the score is influenced by the demonstration provided to the judge. A game may work on the demonstrated path but fail during another ordinary action that the trace never exercises.
Questions for authors

N/A

1 comment

Thank you for your positive feedback and for recognizing the value of the problem we study, our evaluation framework, and the clarity of our writing! We also understand your concerns regarding judge design and stability. We address them below.


Weakness 1: Lack of Ablation on Judge Design

Thank you for pointing this out. We agree that this is an important ablation. We conducted comparisons among Code-only Judge, Agent-authored Replay, and Verifier-authored Replay to demonstrate that Agent-authored Replay achieves reasonably consistent judgments with human evaluations while offering a substantial advantage in token efficiency.

We randomly selected 10 tasks from our 140-task benchmark: strategy-rogue-joker-poker, rhythm-garden, visualnovel-arcaneacademy, simulation-air-control, platformer-momentum-lab, openworld-bounty, horror-lighthouse, openworld-archaeology, openworld-seasons-witch, and platformer-knight-quest.

We evaluated games generated by GPT-5.6-sol high, with the following results:

Evaluation MethodMDVATokens per Problem
Code-only Judge66.0063.2561.2558.75207,202
Agent-authored Replay (Ours)71.9254.7752.5049.63123,191
Verifier-authored Replay77.0561.2154.6552.62334,234
Human Gameplay Evaluation68.2152.2453.2248.73—

We observe that:

  • Code-only Judge is relatively reliable for Mechanics (M), but performs worse on Depth (D), Visuals (V), and Art (A). In particular, without access to rendered gameplay, it tends to overestimate Visuals and Art.
  • Verifier-authored Replay, where an independent verifier reads the game source code and rubrics before writing demonstration traces, produces noticeably higher scores for Mechanics and Depth. Interestingly, the verifier tends to select traces that specifically demonstrate rubric requirements, potentially introducing additional selection bias.

Weakness 2: Sensitivity to Frame Sampling

We agree that this is an important ablation that can provide useful diagnostic insights. Following the same experimental setting as W1, we varied the frame rate and resolution while keeping the game artifacts, demonstration traces, and judge model fixed.

Evaluation SettingMechanics (M)Depth (D)Visuals (V)Art (A)Overall
2 FPS / 480p71.9254.7752.5049.6355.26
1 FPS / 480p67.2252.1751.2149.1653.27
4 FPS / 480p70.1653.5850.5949.2854.14
2 FPS / 720p71.2555.9054.7452.0256.70
Human Gameplay Evaluation68.2152.2453.2248.7353.55

We observe that:

  • Reducing FPS from 2 to 1 decreases the scores, particularly for Mechanics and Depth, as some requirements involving temporal changes can be misjudged when visual evidence is too sparse.
  • Increasing FPS from 2 to 4 leads to relatively small changes in the overall evaluation results.
  • Increasing resolution from 480p to 720p has little impact on Mechanics and Depth, but noticeably improves the scores for Visuals and Art.

Weakness 3: Demonstration Selection Bias

We agree that this is an important risk. To investigate it, we compared Agent-authored Replay with Verifier-authored Replay on the same generated games:

Evaluation MethodMDVATokens per Problem
Agent-authored Replay (Ours)71.9254.7752.5049.63123,191
Verifier-authored Replay77.0561.2154.6552.62334,234

Our interesting finding is that Verifier-authored Replay may actually introduce stronger selection bias. When an independent verifier reads the game source code and rubrics before writing traces, it tends to select demonstrations that specifically satisfy the rubric requirements. Consequently, Verifier-authored Replay produces substantially higher scores, especially for Mechanics and Depth, even though the underlying games remain unchanged.

This suggests that having an independent verifier author the demonstrations does not necessarily eliminate selection bias. In contrast, our Agent-authored Replay achieves more balanced evaluation results with substantially lower token consumption.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We sincerely appreciate all reviewers' time and effort in reviewing our paper. We are pleased that the reviewers recognize the following strengths of our work:

  • An Important and Challenging Domain: Reviewers recognize the importance and challenges of evaluating complete, playable games in real game engines [vtFk, 8qHe, 3b4u].
  • A Valuable Evaluation Framework: The agent-authored gameplay traces and human-calibrated LLM judging are appreciated as valuable contributions [8qHe, 3b4u].
  • Insightful Findings and Clear Presentation: Reviewers highlight the usefulness of our empirical findings [vtFk, 3b4u] and the clarity of our writing [vtFk].

We sincerely thank the reviewers for their constructive suggestions. Our additional experiments and clarifications are summarized below:

1. Judge Design and the Necessity of Interactive Evaluation [vtFk, 3b4u]

  • Judge Ablation: We compare Code-only, Agent-authored Replay, and Verifier-authored Replay against human gameplay assessments on 10 randomly selected tasks, providing direct empirical evidence for the value of interactive evaluation.

2. Evaluation Robustness and Human Calibration [vtFk, 8qHe]

  • Sampling and Demonstration Robustness: We examine how evidence sampling and trace selection affect evaluation reliability. Scores are relatively stable across 2–4 FPS, while higher resolution improves Visuals and Art. Interestingly, verifier-authored traces score higher on 9/10 games, revealing that independent demonstration authoring does not necessarily eliminate selection bias.
  • Human Gameplay Calibration: Our additional 10-task human evaluation shows strong task-level correlations (r = 0.959–0.979) and low MAE (3.33–3.96 points) across all four evaluation dimensions.

3. Agent Trajectory Analysis and Harness Fairness [8qHe]

  • Interactive Development Behaviors: We analyze agent trajectories to quantify game execution and interactive testing, including a concrete example of a text-only agent converting screenshots into ASCII for visual debugging.
  • Harness and Reliability Clarifications: We supplement the top-model execution diagnostics and clarify that our leaderboard evaluates agent configurations, not isolated base models. Kimi-K3's results originate from MoonshotAI's official self-evaluation, for which trajectories are unavailable.

4. Benchmark Design, Score Interpretation, and Limitations [8qHe, 3b4u]

  • Evaluation Motivation and Score Semantics: We clarify the three desiderata and emphasize that scores measure specification-relative requirement satisfaction, rather than absolute commercial game quality.
  • Dataset Construction: We clarify that 12 human annotators developed the task specifications and rubrics, with AI assistance permitted, covering 140 tasks across 15 game families.
  • Evaluation Limitations: We acknowledge the limitations of fixed demonstrations, reward-hacking risks, and incomplete gameplay-path coverage, and discuss Code + Test Cases + Replay judging for future RL-oriented evaluation.

We have incorporated all revisions and additional experimental results into the updated manuscript, with all changes highlighted in blue for easy reference.


We are happy to provide any further clarifications to help the reviewers better understand and assess our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.