- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GameCraft-Bench presents a benchmark, or in my opinion more accurately a methodology, towards evaluating end to end game development. It focuses on godot and evaluates using actual execution of the game rather than code analysis.
Critical strengths
- End to end game development evaluation is challenging and this provides a study into a seemingly reasonable framework.
- The human judgement comparison with LLM judge is useful to establishing the effectiveness of the judge. This is often missing from benchmarks that use LLM judges so I do appreciate this.
- The findings and insights are interesting and useful, especially in this niche domain.
Critical weaknesses
- Why are the three desiderata necessary? The paper spends significant effort for 6 pages claiming the three desiderata are important, but does not justify these claims. There are no experiments backing these claims up nor are there any citations / references to game development literature actually proving this point.
- The paper states this: "The central claim is that game generation cannot be evaluated as ordinary code synthesis". Yet, I see no experimentation suggesting that these results could not be done with pure code evaluation? There is no head to head comparison anywhere. Essentially, it's a similar problem to the first weakness. [edit] FYI, this seems like a similar weakness (1) by vtFk
- A major problem with LLM judge papers is that the scores themselves have no real meaning. It's unclear to me what any X% of functional visual correctness means. From the appendix, it seems that there's a rubric of y values each of which has its own mean, which are then combined into X. But what does it mean if the model scores 100% on visuals? Is it something like halo? like cyberpunk? are the graphics like real life? Some sort of scale would provide better insights.
- Another concern is the size of the dataset. 140 tasks / games is small for a benchmark.
Ultimately, a lot of information and justification is currently missing. Some of this may simply be a writing exercise. For example, for (2), I would suggest referencing recent work that has gone through the effort to show that interactive evaluation is better than static code llm as a judge. Both of these are perhaps relevant: [1] Interaction-as-a-Judge: https://arxiv.org/abs/2610.17895 [2] The Verification Horizon: https://arxiv.org/abs/2606.26300
For points (1) and (3) I would hope to see some sort of justification or, ideally, additional experiments to support these in the rebuttal.
Questions for authors
- How are the rubric items generated? Auto-generated via LLM? Human designed?
- How are the task instructions generated?
5 comments
Dear Reviewer 3b4u,
As the discussion deadline approaches, we would be very grateful for any further thoughts or feedback on our rebuttal and additional experiments. We sincerely appreciate the time and effort you have devoted to reviewing our work and would be happy to address any remaining concerns or questions.
Best regards,
Authors
- I'm glad that you agree this deserves stronger justification, but your response doesn't answer my question nor address my point. This is just restating what's already in the paper. I'm looking for actual evidence that these 3 are what's important and not others. For example, why is "Visual Fidelity" not a desiderata? Or "Solvability" of the game? I'm not saying these are desiderata, but it's completely unclear to me why the three you picked should be the three desiderata for game development.
- This helps, but a) the sample size is very small and b) how exactly human gameplay annotation is done is very unclear. Also, I was expecting something related to your table 5 but it seems this is an entirely different experiment? Didn't you already do some human vs judge comparisons? Is this on building on top of that or something different?
- Again, helpful and should be in the paper. Unfortunately, it's still very unclear to me what 100% on these rubric items means. It's not photorealistic or AAA I get, but... what is it? Is it at the level of an AA game? an indie developer who spent a year? What exactly is the ceiling here? Is 100% something I could expect to see sold on any game store? Something I see on a flash games website? There is just no grounding for quality of the game that I can see
lastly, I don't see any of these edits actually in the manuscript?
Unfortunately, I will keep my score as is.
Thank you for recognizing the challenges of end-to-end game development evaluation and the value of our framework, human calibration, and empirical findings. We appreciate your suggestions and address them below.
Weakness 1: Necessity of the Three Desiderata
We appreciate this concern and agree that the motivation deserves stronger justification.
Our three desiderata define complementary requirements for evaluating end-to-end game generation in a real engine, rather than universal requirements for all game benchmarks:
Weakness 2: Static Code Evaluation vs. Interactive Evaluation
Thank you for suggesting a direct comparison. We randomly selected 10 tasks from our 140-task benchmark and evaluated the corresponding games generated by GPT-5.6-sol high. We compared Code-only Judge and Agent-authored Replay Judge against human gameplay assessments, keeping the same game artifacts and task-specific rubrics across all evaluation methods.
Replay-based evaluation reduces task-level MAE by approximately 54% compared with Code-only judging. Notably, Code-only judging overestimates Content Depth, Functional Visuals, and Art, whereas Replay-based judging more closely matches human assessments. This provides empirical support for the value of evaluating executable gameplay rather than relying solely on source code.
We also thank the reviewer for recommending Interaction-as-a-Judge and The Verification Horizon. We will cite these works to strengthen our motivation.
Weakness 3: Interpretation of Evaluation Scores
We agree that the score semantics should be clarified. GameCraft-Bench measures specification-relative rubric satisfaction, not absolute game quality or commercial production fidelity.
Each task contains explicit requirements evaluated on a 0–1 scale:
Thus, 100% Functional Visuals means that all task-specific visual functionality requirements receive full credit under the evaluated evidence. It does not imply photorealism or AAA-level graphics. Similarly, Art & Presentation measures consistency with the requested art direction, rather than comparison against commercial games such as Halo or Cyberpunk. We will clarify this scoring interpretation and provide concrete examples from generated games at different score levels.
Weakness 4: Dataset Size
We acknowledge that 140 tasks cannot exhaustively represent all game-development scenarios. However, our benchmark contains 140 end-to-end game-generation tasks across 15 game families, rather than isolated code-editing problems. Each task requires constructing a complete project, including gameplay mechanics, content, assets, and UI. This makes task construction and evaluation substantially more demanding than localized code-editing benchmarks. We will clarify the benchmark's intended coverage and acknowledge its current limitations, including uneven task counts across game families.
Question 1 & Question 2: Rubric and Task Instruction Construction
Both task instructions and evaluation rubrics were developed by 12 human annotators, as described in Appendix F. Annotators were allowed to use AI tools for assistance, but were required to independently draft the game specifications and define the rubric criteria themselves, rather than relying on AI-generated requirements.
For each task, annotators prepared: