acceptodds
Under review as a conference paper at ICLR 2027

SAMA-GEO: Semantic-Aware Multi-Agent Framework for Game Engine Optimization

Abstract

Game engine optimization across multi-game platforms presents a challenging decision-making problem due to the heterogeneous and complex characteristics of game workloads, with significant variation across and within genres. While genre provides a useful high-level characterization of games, it alone is insufficient for accurate policy prediction, as substantial variation in gameplay characteristics and engine requirements exists even among games within the same genre. We ask: do semantic-aware multi-agent systems work for this problem? To enable this analysis, we introduce SAMA-GEO, a benchmark dataset of 5,193 Roblox games spanning 17 genres and 59 genre-subgenre pairs, constructed using publicly available Roblox APIs. The dataset features a gold set built through manual play-testing and expert annotation, and a larger silver set with gold-referenced annotations, with each game annotated with 10 semantic labels capturing gameplay characteristics and 3 policy labels governing engine configuration. We formulate game engine optimization on Roblox as semantic-aware policy prediction, where game-level semantic understanding provides an explicit intermediate signal for predicting engine policies, bridging the gap that genre alone cannot close. To investigate this formulation, we propose a Semantic-Aware Multi-Agent framework that decomposes semantic and policy reasoning across specialized agents, representing the first application of multi-agent reasoning to game engine optimization. To measure performance, we develop an eight-metric evaluation suite spanning semantic and policy prediction, combining established, adapted, and novel metrics that jointly capture pointwise accuracy, structural preservation, and constraint adherence. Component-wise analysis shows consistent and complementary gains as semantic and policy reasoning capabilities are incorporated, supporting an affirmative answer to our central question.

Primary area:
datasets and benchmarks
Keywords:
MultiAgent, GameEngine, MultiGamePlatform

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ApRqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on the problem of game engine optimization across multi-game platforms, and introduces a benchmark dataset of 5193 Roblox games spanning 17 genres and 59 genre-subgenre pairs named SAMA-GEO. It further proposes a semantic-aware multi-agent framework that decomposes semantic and policy reasoning across specialized agents. The game-level semantic understanding provides an explicit intermediate signal for predicting engine policies, bridging the gap that genre alone cannot close. The paper develops an eight-metric evaluation suite spanning semantic and policy prediction to measure performance. Experimental results demonstrate that semantic grounding improves policy prediction.

Critical strengths
  1. Modeling semantic requirements across games is a worthwhile research problem, and the introduced dataset has potential value.
  2. The authors consider that the same game may be suitable for multiple reasonable configurations and evaluates the prediction results from different dimensions.
Critical weaknesses
  1. Of the 5193 games, only the gold set of 152 games is manually play-tested and expert-annotated. The silver set is annotated by a model using gold annotations of the same genre as reference. The paper does not report inter-annotator agreement or quantify the results of manual verification on the silver set, making it difficult to assess their reliability as ground truth for evaluation.
  2. The contribution of multi-agent design remains unclear. The constituent agents mainly pass and refine predictions through a predefined workflow, making it unclear how this differs from decomposing the task into several modules. Moreover, adding agents increases model calls, and the Image Analysis Agent introduces additional information. It is therefore difficult to determine whether the performance gains arise from coordination among agents.
  3. The experiments are limited to label prediction and do not establish whether the predicted policies improve actual game performance or player experience.
Questions for authors
  1. How many parameters need to be configured in the game engine under study, and how do the three configuration attributes (target FPS, resolution, and render quality) considered in this paper map to them? If these attributes are high-level abstractions of numerous low-level parameters, does translating the predictions into executable engine configurations still require substantial parameter tuning?
  2. Could the authors clarify what additional benefits the multi-agent design offers over a single model following the same sequence of steps?
1 comment

Weakness 1: Silver set annotations‾\underline{\text{Weakness 1: Silver set annotations}}

10% of the data samples in the silver set were verified manually across different genre-subgenre categories. The average error rate per semantic label was approximately ±0.2 on a scale from 0 to 1.

Manual play-testing requires substantial time per game, which limits it to a subset of games (Section 3.4). The silver set extends coverage across all 17 genres and 59 genre-subgenre pairs at a scale that manual annotation could not reach. Silver annotations are generated individually per game, with the gold annotations of the same genre-subgenre pair given to the annotator model as reference, so they follow the patterns of the human-annotated gold set. We agree that model-generated labels do not replace human labels, so the fully human-annotated gold set is our primary evaluation set, and Tables 2 and 3 show consistent trends on both.

Weakness 2: The contribution of multi-agent design‾\underline{\text{Weakness 2: The contribution of multi-agent design}}

The goal of this work is not to study dynamic inter-agent coordination, but rather role specialization, where each agent is assigned a distinct information source and is restricted to refining a single policy dimension while treating upstream predictions as a prior (Fig. 8). This follows the role-specialized, predefined-workflow sense of “multi-agent” used in prior work (e.g., MetaGPT, AutoGen). Regarding whether gains come from coordination, we note two pieces of evidence. First, the gains are component-aligned rather than uniform: the Genre Analysis Agent improves CVR, PRS and GJSD, while the Image Analysis Agent primarily improves SPCS, a pattern not expected from additional model calls alone. Second, all refiner configurations improve over the Game Analyst across six different backbones (Appendix A, Table 4).

Weakness 3: Prediction objective‾\underline{\text{Weakness 3: Prediction objective}}

The benchmark tests whether a model can predict an initial configuration for a game at launch, as a first step toward engine optimization. Validating predicted configurations on a real engine requires access to platform internals, which is outside the scope of this work. We will state this scope in the introduction, and revise “maximize the player’s experience” in Section 4.1 to match what is evaluated.

Question 1: Configuration of parameters for game engine‾\underline{\text{Question 1: Configuration of parameters for game engine}}

For multi-game platforms like Roblox, there is a requirement for a device-agnostic configuration prediction at the moment when a user starts a particular game. There are other configuration parameters that define the engine policy, but most are either derivative of the basic parameters which we focus on, complex to determine solely with the game information, or determined by the engine in real-time later. Holistic engine optimization would warrant access to a real multi-game engine and internal information.

Question 2: Benefits of multi-agent design‾\underline{\text{Question 2: Benefits of multi-agent design}}

The benefits we attribute to the multi-agent design are the following.

  1. Each agent receives only the information relevant to its policy dimension and is restricted to refining that dimension, which prevents cross-dimension interference (Fig. 8).
  2. Intermediate outputs are explicit and independently evaluable, enabling the component-wise analysis in Section 6.
  3. The design is modular: agents can use different backbones, and the improvements over the Game Analyst hold across all six backbones tested (Appendix A, Table 4).
Reviewer_DE4vOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The work introduces SAMA-GEO which comprises of both a dataset for evaluation and a multi-agent framework for predicting attributes/settings of games built within the Roblox game engine. It uses information like the game descriptions and metadata to make these predictions via a multi-agent orchestration framework. The main claim from the paper seems to be introducing the potential for multi-agent systems to improve game engine configuration through policy prediction, and evaluation is primarily done by mapping how well the multi-agent system predicts annotated game attributes/settings.

Critical strengths

One of the main strengths of this paper is the introduction of a new class of problems to be solved by ML systems -- in this paper that is instantiated by predicting attributes of game engines given their general descriptions.

In order to accomplish this, the authors introduce their dataset. I think the "Gold" standard dataset is the one artifact from this work that is the most potentially useful going forward, as it could provide a useful resource for evaluating gameplay characteristics for a variety of future methods or could be used as a part of more meta evaluation harnesses that test many different domains.

Critical weaknesses

For most real-world game development, decisions are heavily constrained by the hardware used (e.g., which console or what are the min PC specs for running the game). Appendix F describes the prediction targets as being device-agnostic. However, things like appropriate FPS/resolution are directly tied to physical systems (e.g., the GPU, CPU, and physical monitor screen) and are generally not device agnostic. From my perspective, this significantly limits the value of this dataset as currently presented. At the very least, attributes like resolution, render quality, and target FPS should be seen as levers where each can be moved at the expense of the others for a given computer system. The current framing lacks this nuance, which calls into question how useful the labels are.

In my opinion, the value of the multi-agent architecture is not sufficiently shown. The authors would greatly benefit by showing how it improves performance beyond single-model baselines that are given all relevant instructions/prompts and input data. In essence, I feel as though there is a lack of ablations and experiments that shows the specific agent roles are all necessary.

Finally, the jump from predicting configuration targets to optimizing game engines is not sufficiently established. Prediction of the chosen attributes (resolution, render quality, and target FPS) is not the same as game engine optimization.

Questions for authors
  1. Please clarify what are the device-agnostic configuration targets that are used. Why were these chosen and how are they considered optimal? Do you have hardware metadata that can be used to map between these "optimal" settings and the hardware they were used with?

  2. Is there evidence that the multi-agent architecture is necessary? How does the proposed system compare to a single model given all relevant instructions + input data?

1 comment

Weakness 1: Nuance for device-agnostic labels‾\underline{\text{Weakness 1: Nuance for device-agnostic labels}}

We agree that feasible settings depend on the hardware, and that FPS, resolution, and render quality are levers traded off against each other on a given system. Our labels are not device-specific settings. They describe a game’s relative rendering and responsiveness demands, which is what a platform can know before it has any measurements of a player’s device. Because several configurations can be acceptable, FPS and render quality are annotated as ranges, which reflects this trade-off. The task is initial, device-agnostic configuration prediction at game launch, as a first step toward engine optimization, and the engine then adjusts within each device’s limits at runtime. We will state this scope and the lever trade-off explicitly in the introduction and Section 4.1.

Weakness 2: Value of the multi-agent architecture‾\underline{\text{Weakness 2: Value of the multi-agent architecture}}

The “+ Policy-aux” row in Table 2 serves as the single-agent comparison point: one model receives all available information (description, genre-subgenre, metadata, etc) and is prompted to jointly output the semantic profile and the policy. The Game Analyst (Table 2, row 7; Table 3, row 4) is an ensemble of models that all perform this same task, with outputs combined, and can therefore be viewed as a single-agent-adjacent configuration. The subsequent agents, each refining a specific component of the prediction, improve performance over this configuration, and Appendix A (Table 4) shows that these gains hold across all backbones tested rather than being specific to one model.

Weakness 3: Initial configuration prediction‾\underline{\text{Weakness 3: Initial configuration prediction}}

The goal of the problem formulation is policy component prediction which aid in game engine optimization. Our benchmark targets one component of it, the initial configuration chosen when a game starts. Holistic optimization would also need runtime adaptation and access to a real multi-game engine, which is outside the scope of this work. We will revise the abstract, introduction, and conclusion to describe the task as initial configuration prediction toward engine optimization, rather than engine optimization itself.

Question 1: Device-agnostic configuration targets‾\underline{\text{Question 1: Device-agnostic configuration targets}}

The targets are initial values for target FPS, resolution tier, and render quality, chosen because they are the primary performance versus visual-quality levers. Semantic labels come from annotators’ play-testing experience. Policy labels are derived from the semantic labels by the GMM-based mapping (Section 3.3, Appendix F) and represent a range of acceptable configurations for a game’s demands, not a device-specific optimum. The dataset does not include hardware metadata, so the labels are not mapped to particular devices.

Question 2: Evidence for necessity of multi-agent architecture‾\underline{\text{Question 2: Evidence for necessity of multi-agent architecture}}:

Please see our response to Comment 2. In addition, the component-wise analysis (Section 6) shows that each refiner improves different metrics (RASC broadly, the Genre Analysis Agent on CVR, PRS and GJSD, the Image Analysis Agent mainly on SPCS), supporting the role of each agent.

Reviewer_hbPHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies game-level prediction of target FPS, resolution, and render quality from Roblox descriptions, genre information, metadata, and thumbnails. It introduces 5,193 games across 17 genres and 59 genre–subgenre pairs: 152 manually play-tested gold examples and 5,041 silver examples annotated using an LLM with same-genre gold references. Ten semantic attributes provide an intermediate representation. Policy targets are constructed from these attributes using equal-weight composites, GMM-derived ranges for FPS/render quality, and thresholded resolution tiers, with reported expert verification.

The framework combines a six-model Game Analyst ensemble with a semantic policy refiner and specialized genre and image agents. Eight metrics evaluate semantic prediction and agreement with the constructed policy targets. The final system improves gold-set CVR from the Game Analyst's 0.653 to 0.603 and PRS from 0.520 to 0.576; silver-set trends are similar (Tables 2–3). These results support improved annotation agreement and internal semantic–policy consistency.

Critical strengths
  1. A concrete, interpretable cross-game dataset contribution. Sections 3.1–3.3, Table 1, and Appendices B–F specify the input modalities, genre coverage, semantic attributes, and label-generation procedure. The manually play-tested gold subset provides a human-grounded semantic reference rather than relying entirely on synthetic judgments. This supports acceptance consideration as a potentially useful benchmark resource, although the large silver portion should not be equated with independently human-labeled data.

  2. Useful evidence for richer semantic inputs and policy refinement. In Table 2 (p. 8), adding descriptions to genre reduces semantic MAE from 0.147 to 0.125 and raises SSP from 0.540 to 0.667. Adding RASC to the Game Analyst improves gold CVR and PRS.

Critical weaknesses
  1. The task itself lacks a sufficiently clear practical use case. The paper defines the objective as selecting FPS, resolution, and render quality to maximize player experience. It is unclear which concrete configuration decision the output is intended to resolve: the same game can require different feasible settings under different operating conditions, and its semantic profile alone does not define that choice. Why values such as the 68 FPS / 1440p / quality-6 recommendation in Figure 4 are useful at that level of precision? Semantic game understanding may be useful, but the manuscript has not sufficiently motivated this particular device-agnostic configuration task as a meaningful engine-optimization problem.

  2. The experiments do not isolate the value of the multi-agent design, and the smallest component gains are insufficiently resolved. Tables 2–4 include constant/random baselines, incremental inputs, the Game Analyst, and additional refiners, but no baseline applying the already-defined Appendix F mapping to predicted semantics. Nor is there a single-agent comparison with the same information and comparable inference budget; the final system adds explicit policy guidance, model calls, and images.

  3. Silver-set evaluation also depends on insufficiently validated AI-generated references. The paper uses GPT 5.6 Sol annotation with gold references and random human verification, but gives neither the audit sample size nor error rates. Strong agreement may reflect consistency between AI systems, including shared errors, and cannot establish label correctness.

Questions for authors

What is the practical value of this task, and in what concrete scenarios would it be used?

1 comment

Weakness 1: Clarity in practical use case‾\underline{\text{Weakness 1: Clarity in practical use case}}

Because this is a newly formulated problem, we agree the paper should motivate the use case more explicitly. When a player launches a game on a multi-game platform such as Roblox, the engine must choose a starting frame rate, resolution, and render quality before it has any measurements of that player’s device. A common practice is a single fixed default for all games (the constant baseline in Section 6). Our benchmark tests whether a model can predict a game-specific starting configuration from game information alone, and Table 2 shows this reduces constraint violations from 0.752 to 0.603.

The annotations are device-agnostic, meaning independent of any particular hardware, so they describe a game’s rendering and responsiveness demands, not device-specific settings. We focus on three primary parameters. Other parameters are largely derived from these, determined by the engine at runtime from hardware and network conditions, or not determinable from game information alone. The predicted configuration is thus a starting point that runtime adaptation refines.

Values such as 68 FPS / 1440p / quality 6 are not meant to be exact. Ground truth for FPS and render quality is a range of acceptable values and resolution is a tier, and evaluation checks whether predictions fall within the acceptable range (CVR, PRS). The semantic profile does not define the configuration on its own. It encourages the model to interpret a game from a player’s perspective before predicting its requirements.

We will state this use case in the introduction and Section 4.1.

Weakness 2: Isolation of the value of the multi-agent design‾\underline{\text{Weakness 2: Isolation of the value of the multi-agent design}}

Single-agent comparison. The “+ Policy-aux” row in Table 2 serves as the single-agent comparison point: one model receives all available information (description, genre-subgenre, metadata, etc) and is prompted to jointly output the semantic profile and the policy. The Game Analyst (Table 2, row 7; Table 3, row 4) is an ensemble of models that all perform this same task, with outputs combined, and can therefore be viewed as a single-agent-adjacent configuration. The subsequent agents, each refining a specific component of the prediction, improve performance over this configuration, and Appendix A (Table 4) shows that these gains hold across all backbones tested rather than being specific to one model.

Appendix F mapping baseline. We appreciate the suggestion to apply the Appendix F mapping to predicted semantics. This mapping is the procedure that generates ground-truth policies from human-annotated semantics, so applying it to predicted semantics would largely measure semantic prediction accuracy, which the semantic metrics already capture, and would share its generating process with the ground truth on policy metrics. We therefore treat it as part of the annotation procedure rather than a competing method, and will clarify this in the paper.

Weakness 3: Silver set‾\underline{\text{Weakness 3: Silver set}}

10% of the data samples in the silver set were verified manually across different genre-subgenre categories. The average error rate per semantic label was approximately ±0.2 on a scale from 0 to 1.

Silver annotations are conditioned on human gold annotations of the same genre, and the silver annotator (GPT 5.6 Sol) is from a different model family than the refiner agents (Claude Opus 4.8), which reduces the chance of results reflecting self-agreement, though we do not claim it removes shared biases. The fully human-annotated gold set is our primary evaluation set, and Tables 2 and 3 show consistent trends across both.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.