- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper focuses on the problem of game engine optimization across multi-game platforms, and introduces a benchmark dataset of 5193 Roblox games spanning 17 genres and 59 genre-subgenre pairs named SAMA-GEO. It further proposes a semantic-aware multi-agent framework that decomposes semantic and policy reasoning across specialized agents. The game-level semantic understanding provides an explicit intermediate signal for predicting engine policies, bridging the gap that genre alone cannot close. The paper develops an eight-metric evaluation suite spanning semantic and policy prediction to measure performance. Experimental results demonstrate that semantic grounding improves policy prediction.
Critical strengths
- Modeling semantic requirements across games is a worthwhile research problem, and the introduced dataset has potential value.
- The authors consider that the same game may be suitable for multiple reasonable configurations and evaluates the prediction results from different dimensions.
Critical weaknesses
- Of the 5193 games, only the gold set of 152 games is manually play-tested and expert-annotated. The silver set is annotated by a model using gold annotations of the same genre as reference. The paper does not report inter-annotator agreement or quantify the results of manual verification on the silver set, making it difficult to assess their reliability as ground truth for evaluation.
- The contribution of multi-agent design remains unclear. The constituent agents mainly pass and refine predictions through a predefined workflow, making it unclear how this differs from decomposing the task into several modules. Moreover, adding agents increases model calls, and the Image Analysis Agent introduces additional information. It is therefore difficult to determine whether the performance gains arise from coordination among agents.
- The experiments are limited to label prediction and do not establish whether the predicted policies improve actual game performance or player experience.
Questions for authors
- How many parameters need to be configured in the game engine under study, and how do the three configuration attributes (target FPS, resolution, and render quality) considered in this paper map to them? If these attributes are high-level abstractions of numerous low-level parameters, does translating the predictions into executable engine configurations still require substantial parameter tuning?
- Could the authors clarify what additional benefits the multi-agent design offers over a single model following the same sequence of steps?
Weakness 1: Silver set annotations
10% of the data samples in the silver set were verified manually across different genre-subgenre categories. The average error rate per semantic label was approximately ±0.2 on a scale from 0 to 1.
Manual play-testing requires substantial time per game, which limits it to a subset of games (Section 3.4). The silver set extends coverage across all 17 genres and 59 genre-subgenre pairs at a scale that manual annotation could not reach. Silver annotations are generated individually per game, with the gold annotations of the same genre-subgenre pair given to the annotator model as reference, so they follow the patterns of the human-annotated gold set. We agree that model-generated labels do not replace human labels, so the fully human-annotated gold set is our primary evaluation set, and Tables 2 and 3 show consistent trends on both.
Weakness 2: The contribution of multi-agent design
The goal of this work is not to study dynamic inter-agent coordination, but rather role specialization, where each agent is assigned a distinct information source and is restricted to refining a single policy dimension while treating upstream predictions as a prior (Fig. 8). This follows the role-specialized, predefined-workflow sense of “multi-agent” used in prior work (e.g., MetaGPT, AutoGen). Regarding whether gains come from coordination, we note two pieces of evidence. First, the gains are component-aligned rather than uniform: the Genre Analysis Agent improves CVR, PRS and GJSD, while the Image Analysis Agent primarily improves SPCS, a pattern not expected from additional model calls alone. Second, all refiner configurations improve over the Game Analyst across six different backbones (Appendix A, Table 4).
Weakness 3: Prediction objective
The benchmark tests whether a model can predict an initial configuration for a game at launch, as a first step toward engine optimization. Validating predicted configurations on a real engine requires access to platform internals, which is outside the scope of this work. We will state this scope in the introduction, and revise “maximize the player’s experience” in Section 4.1 to match what is evaluated.
Question 1: Configuration of parameters for game engine
For multi-game platforms like Roblox, there is a requirement for a device-agnostic configuration prediction at the moment when a user starts a particular game. There are other configuration parameters that define the engine policy, but most are either derivative of the basic parameters which we focus on, complex to determine solely with the game information, or determined by the engine in real-time later. Holistic engine optimization would warrant access to a real multi-game engine and internal information.
Question 2: Benefits of multi-agent design
The benefits we attribute to the multi-agent design are the following.