acceptodds
Withdrawn from ICLR 2027

Population-Scalable Multi-Agent World Modeling across Scene Geometries

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models have recently achieved impressive progress in visual prediction and interactive generation, yet existing multi-agent systems are typically tied to predefined agent configurations and scene-specific environments. We propose Khora, a scalable multi-agent world model that supports both inference-time population expansion and generalization across geometrically distinct environments. Khora decouples shared world-state evolution from visual rendering: an action-conditioned state model maintains dynamic agents in a persistent Spatio-temporal Board (STBoard), while each observation is generated independently through a projection-based rendering interface. The renderer is trained with one target view per sample and reused across agents at inference without retraining. Crucially, scene geometry is provided to the renderer as an explicit mesh condition, allowing the same trained model to render observations that conform to different input environments without changing the underlying world dynamics or model architecture. This design enables Khora to scale along two complementary dimensions: many agents can interact within the same evolving world, while the same model can render observations across many different mesh-defined environments. Experiments demonstrate multi-agent consistency, action-conditioned interaction, generalization to unseen meshes, and scalability to large agent populations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Video World Models, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8bezOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Khora, a scalable multi-agent world model that supports both inference-time population expansion and generalization across geometrically distinct environments. Technically, the paper proposes a persistent Spatio-temporal Board (STBoard) to store scene and entity information, while geometry-conditioned rendering generates each agent’s view independently, which supports changing agent populations without modifying the renderer. Authors provide experimental results on visual quality and world consistency, runtime scalability with up to 80 agents, and dynamic agent insertion and removal, supporting the insight and effectiveness of the proposed method.

Critical strengths

Specifically, the strengths of the paper are:

  1. The paper is well motivated and architectural decomposition is clear and useful. Khora separates shared-state evolution from view generation using a persistent STBoard and geometry-conditioned rendering. This is a sensible design: consistency can be addressed at the state level, while rendering resources scale with observation demand. It also makes dynamics and visual-generation failures easier to diagnose separately.
  2. Clear improvements in both of video quality and action consistency. Compared with removing cross-agent state, FVD decreases from 51.6657 to 36.5529 for two views, 73.4917 to 38.6849 for four views, and 104.9082 to 53.6148 for eight views (approximately 29.3%, 47.4%, and 48.9% reductions, respectively.) These consistent improvements support the value of sharing dynamic information. As for action consistency, human ratings increase from 2.075 to 3.570. This is particularly relevant because visually plausible individual videos do not necessarily depict compatible interactions. These quantitative results demonstrate the effectiveness of the proposed work.
  3. Encouraging distributed runtime performance. In Table 20, across 1–80 agents, reported compute latency grows from approximately 107 to 117 ms, while per-view throughput declines only from 37.33 to 34.27 (2741.2984 / 80) FPS. Aggregate throughput reaches 2,741.3 view-FPS. These results suggest the architecture can exploit additional rendering hardware effectively within the measured range.
Critical weaknesses

Specifically, the weaknesses of the paper, or said, my main concerns about the paper are:

  1. Nearly constant latency depends on increasing hardware resources. The runtime setup uses one GPU per view (Line 2086: One GPU is assigned per agent). Consequently, the result supports distributed scalability; it does not show how performance changes when more agents share a fixed GPU budget. A fixed-resource evaluation would make the practical efficiency advantage much clearer.
  2. In table 3, world-consistency gains are comparatively modest. Ratings improve from 4.065 to 4.2133. Table 3 omits the confidence intervals promised in the text. Without uncertainty estimates, I cannot tell confidently whether this smaller improvement is reliable or perceptually meaningful.
  3. The highest-population quality evaluation has limited coverage. Eight-view evaluation uses 159 cases, versus 10,000 at two and four views. I would therefore avoid interpreting differences across population settings as a clean scaling trend. More balanced sampling and uncertainty estimates would strengthen the evidence.
  4. Strictly, a shared representation does not itself guarantee consistent generated observations. My concern is that the renderer may still contradict the conditioning state. Objective measurements of identity preservation, visibility, interaction outcomes, and long-horizon drift would be more convincing than visual quality and subjective ratings alone.
  5. The ablation study is limited to removing cross-agent state, which demonstrates the benefit of sharing information but does not isolate the contributions of the other proposed components. In particular, it remains unclear how much the learned motion residual, persistent scene memory, and geometry-based conditioning each contribute to the final performance. Targeted ablations, such as using only the deterministic motion model or replacing geometric conditioning with a simpler alternative, would help establish which design choices are essential.
Questions for authors

Here are questions for authors:

  1. Runtime clarification: With one GPU assigned per agent, do authors have measurements showing how latency, throughput, and memory usage change as the number of agents increases under a fixed GPU budget? If unavailable, please clarify whether the efficiency claim is limited to distributed scaling with proportional hardware resources.
  2. Statistical reliability: Could authors provide the confidence intervals mentioned in the user-study description, together with the numbers of participants and evaluated clips?
  3. Consistency evidence: Do existing evaluations measure agreement between generated observations and the underlying STBoard states, including identity, visibility, and interaction outcomes? Please clarify whether consistency is enforced by the rendering mechanism or encouraged through conditioning.
  4. Component contributions: Beyond removing cross-agent state, do authors have existing ablations isolating the learned motion residual, persistent scene memory, or geometry-based conditioning? These would help determine whether the benefits arise from the specific proposed design or primarily from access to shared structured state.
  5. Is it possible to provide more qualitative results including both good cases and failure cases to better demonstrate the effectiveness and limitations of the proposed method.
3 comments

We thank the reviewer for recognizing the architectural decomposition, visual-quality and action-consistency improvements, and distributed scalability. We now report fixed-budget performance and profiled inference computation.

Multiple agents can share one device and reuse the same renderer. The original one-device-per-agent allocation isolates population-dependent latency from inter-agent device contention. All new hardware measurements use PPU-ZW810E devices with 96 GB of memory and unchanged generation settings.

Profiled computation grows linearly with requested views: 15.695680 TFLOPs/request for one player and 251.130885 for 16 players, with a constant 3.923920 TFLOPs per new view-RGB frame:

PlayersNew frames/requestTotal TFLOPs/requestTFLOPs/frameDiT TFLOPs/requestVAE TFLOPs/request
1415.6956803.9239207.8929647.802710
2831.3913613.92392015.78592815.605420
41662.7827213.92392031.57185531.210839
832125.5654433.92392063.14371062.421678
1664251.1308853.923920126.287421124.843357

Each request produces four new RGB frames per player. Frame counts and total view-FPS sum over all players; Peak/GPU is the sampled peak memory per device. FLOPs count supported operators in five DiT forwards, one streaming VAE decoding call, and recognized projection/condition-construction operations. Counted operators include matrix multiplication, convolution, and Flash/Efficient Attention. A multiply-add is two FLOPs; TFLOPs denotes 10¹² operations.

On one device, the measured population ranges from 1 to 13 players:

PlayersFPS/playerTotal view-FPSP50 (ms)P95 (ms)Peak/GPU (GiB)
133.1833.18120.33121.857.58
220.9841.96190.57191.9814.64
411.1444.56214.46360.5629.55
85.7646.08416.98698.8758.51
133.5446.02677.491137.2194.31

With a fixed 16-player world, increasing the budget from 2 to 16 devices gives a 5.79× throughput improvement, or 72.42% of ideal linear strong scaling:

GPUsPlayersFPS/playerTotal view-FPSP95 (ms)
2165.7091.32711.32
41611.01176.04371.37
81620.43326.94199.51
161633.06529.05123.71

With eight players per device, increasing from 1 to 16 devices achieves 92.55% of ideal linear throughput scaling:

GPUsPlayersFPS/playerTotal view-FPSP95 (ms)Peak/GPU (GiB)
185.7646.05698.8758.51
1133.5445.991137.2194.31
2165.7091.32711.3258.08
4325.61179.22724.1658.28
8645.55355.32728.9657.96
161285.34681.81765.4559.50

The eight-players-per-device rows define this scaling calculation; the one-device/13-player row separately reports a measured population setting. Part 2 provides user-study uncertainty and objective visibility evidence; Part 3 reports component ablations and the eight-view cohort.

Continuing our response, Khora w/o Cross-Agent State removes other agents’ dynamic state conditions while retaining static scene memory, ego state, and per-view latent history. Action consistency increases from 2.075 to 3.570 (+1.4950), and world consistency increases from 4.0650 to 4.2133 (+0.1483). The larger action gain matches the role of the dynamic conditions evaluated by this ablation.

We report the 95% confidence intervals for both criteria:

MethodAction ↑ (95% CI)World ↑ (95% CI)
Gamma-World2.027 [1.658, 2.433]2.786 [2.366, 3.221]
MIRA3.170 [2.771, 3.563]4.009 [3.596, 4.330]
Solaris3.000 [2.759, 3.250]4.105 [3.658, 4.465]
Khora w/o Cross-Agent State2.075 [1.802, 2.342]4.065 [3.755, 4.352]
Khora3.570 [3.217, 3.922]4.213 [3.895, 4.508]

Paired gains compare Khora with Khora w/o Cross-Agent State. Both 95% CIs use bootstrap resampling: participants only, or participants and source cases.

MetricPaired gainParticipant 95% CIParticipant + case 95% CI
Action+1.4950[1.0217, 1.9600][1.0005, 1.9913]
World+0.1483[0.0867, 0.2200][0.0370, 0.2692]

Both bootstrap analyses give positive paired-improvement intervals for both criteria. Means are displayed to three decimals; paired improvements use unrounded means.

Eight participants completed all 150 assignments in the paired study, yielding 1,200 ratings: 600 per variant and 200 per variant at each of 2, 4, and 8 views. Within each view count, a participant rates the same 25 source cases for both variants. Completed assignments cover 198 distinct two-view cases, 196 four-view cases, and 113 eight-view cases, identically paired across variants. Each rating includes Action and World scores. For the external baselines, 12 participants provided 338 ratings over ten source cases per model: Solaris 114, Gamma-World 112, and MIRA 112. We will include the uncertainty estimates in the revised manuscript.

STBoard supplies a common evolving state, and its projections condition rendering. Table 4 directly evaluates location, image motion, body-pose alignment, and foreground appearance against GT under recorded camera poses and supplied trajectories. CVC complements these actor-level measures with reconstructed geometry and color consistency.

We additionally evaluate target localization and joint visibility across observers:

MethodTarget recall ↑ (95% CI)Joint recall ↑ (95% CI)
Khora40.42% [26.74, 61.10]26.72% [7.07, 55.84]
MIRA32.10% [18.17, 55.04]24.43% [4.07, 54.72]
Gamma26.00% [10.66, 50.29]21.84% [1.88, 51.14]

The visibility evaluation uses 500 four-player cases with two-second clips. Localization recall counts generated detections at the corresponding locations of 673 detector-confirmed GT target observations. Joint recall uses 348 common-target observer pairs: when a target is visible to both GT observers, both generated views must detect it. Two simultaneous misses count as failure.

Khora has the highest recall on both measures. Bold and italics indicate the highest and second-highest point estimates; the intervals remain explicit. Part 3 provides action-conditioned pose prediction and matched visual ablations.

Continuing our response, we isolate the learned transition correction by comparing fixed kinematics with fixed kinematics plus the learned visual residual. All four pose-error measures improve at both horizons:

Horizon / casesState transitionADE ↓FDE ↓Yaw MAE ↓Pitch MAE ↓
10 s / 1,000Fixed kinematics130.74269.4316.27°3.30°
10 s / 1,000+ learned residual102.97229.6714.16°2.56°
30 s / 1,000Fixed kinematics305.12714.7919.38°3.46°
30 s / 1,000+ learned residual259.46678.5317.12°2.87°

We evaluate 1,000 Dust2 cases at each horizon, each with two controlled players and two first-person views. RGB context is frames 0–12; predictions cover 13–332 (10 s) or 13–972 (30 s), at 32 FPS. Both variants use identical cases, initial states, actions, and per-view seeds, with no future GT-pose resets. ADE/FDE are 3D camera-position errors in original game-coordinate units; yaw uses the shortest circular angular difference. Angular errors are in degrees.

A matched visual comparison measures the downstream effect with the same renderer:

ViewsPose sourcePSNR ↑ (dB)SSIM ↑LPIPS ↓FID ↓FVD ↓
2Fixed kinematics24.79130.71040.186210.293083.2843
2+ learned residual26.24250.71300.17897.709836.5529
2GT-pose conditioning28.04820.79870.13747.163829.3661
4Fixed kinematics24.10490.69730.188914.9372105.2849
4+ learned residual25.28250.70200.17727.982138.6849
4GT-pose conditioning27.73580.76230.163711.634432.8530
8Fixed kinematics25.04820.74480.159436.8711139.9372
8+ learned residual26.57750.75730.147727.031153.6148
8GT-pose conditioning28.64970.82770.132924.376241.3781

The visual comparison uses frames 13–80: 10,000 cases at two views, 10,000 at four views, and 159 at eight views. All pose sources use matched cases and generation settings within each view count. GT-pose conditioning provides recorded future poses to the renderer, which still generates RGB. These visual cohorts are separate from the 1,000-case long-horizon experiments. The learned residual improves all five visual metrics over fixed kinematics at every view count.

We separately remove the geometry condition:

Evaluation cases: 10,000 for 2 and 4 views; 159 for 8 views. PSNR is in dB.

ViewsGeometryPSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
2ON26.24250.71300.17897.709836.5529
2OFF10.60770.42350.6758213.43001198.8300
4ON25.28250.70200.17727.982138.6849
4OFF10.92430.41930.6698207.71001178.0400
8ON26.57750.75730.147727.031153.6148
8OFF12.86530.45190.5801199.6600934.6000

ON and OFF use the original evaluation cases and otherwise matched generation settings. Removing geometry degrades all five visual metrics, supporting its contribution independently of the motion-residual comparison. Within matched settings, bold marks the best value and italics the second-best among three settings.

The smaller eight-view cohort reflects the availability of trajectories with synchronized observations and corresponding action/state annotations for all eight agents. We collected all available valid cases: 159 cases (1,272 videos). The two- and four-view evaluations each use 10,000 cases. Each eight-view model comparison uses the same 159 cases, assessing higher-population visual quality on matched data within that setting. We will state this sampling protocol explicitly in the revised manuscript.

Reviewer_g5uDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Khora, a multi-agent world model that supports changes in the number of agents at inference time. Khora separates world-state updates from visual generation. Its STBoard stores the scene mesh, appearance memory, and dynamic agent states. Agent poses are updated using a kinematic model and a learned residual. A shared diffusion renderer then generates each agent’s view from projected scene and entity conditions.

The problem is important, and the architecture is clear. However, the strongest experiments mainly test state-conditioned rendering. All training and main quantitative evaluations are also limited to CS:GO Dust2. The ablation study is narrow, and the reported scalability requires substantial hardware. Based on the current evidence, my initial score is 2: Borderline Reject. I would be open to adjusting my score based on the authors’ response

Critical strengths
  • The state-rendering decomposition is clean. The STBoard gives agents a shared world representation, while the renderer uses a fixed-dimensional interface that does not depend on the population size.

  • The framework supports useful inference-time operations. Agents can enter or leave during a rollout, and the same renderer can process new scene meshes without weight updates.

  • The system implementation is broad. The paper includes visual-quality evaluation, cross-view evaluation, dynamic-population examples, runtime measurements, and an interactive prototype. The limitations of several evaluation protocols are also discussed clearly.

Critical weaknesses
  1. Cross-domain generalization is unclear. Khora is trained only on CS:GO Dust2, and the main quantitative tests use held-out data from the same game and map. The unseen-mesh examples change the geometry but do not test new visual styles, actions, or physical rules. Please evaluate on another game or simulator, or limit the claim to geometry generalization within CS:GO.

  2. The ablation study is limited. The main ablation only removes other-agent state conditions at inference using the same checkpoint. Visual history may still contain other agents. This shows that explicit state conditions help rendering, but it does not explain which parts of Khora are important. Please separately test the STBoard components, appearance-memory updates, mesh conditioning, and the kinematic and learned transition terms.

  3. Population scaling uses a fixed and expensive design. New agents are handled by adding STBoard entries and reusing the same modules. The communication mechanism and model capacity do not change with the number of agents or the interaction density. Each view also needs a separate diffusion-rendering call, and the measured setup assigns one GPU to each agent. Please test denser interactions and report performance under a fixed hardware budget.

  4. Minor weakness: Figures 2 and 3 may confuse readers about the experimental setting. The paper is trained and quantitatively evaluated on CS:GO Dust2, but these figures use mesh assets outside the reported dataset. Readers may think these scenes were used in the experiments. Please label them as illustrations, replace them with actual experimental examples, or explain the source of the assets.

Questions for authors

see weaknesses

2 comments

We thank the reviewer for recognizing the state–rendering decomposition, population-flexible interface, and inference-time operations. Khora supports geometry-guided rendering across games and environments through externally supplied mesh priors. Its rendering interface accepts scene layouts beyond a single CS map. Our main quantitative evaluation uses CS Dust2; the generalization claim concerns rendering with externally supplied scene geometry.

Khora w/o Cross-Agent State evaluates explicit other-agent state conditions while retaining map memory, ego conditions, and visual history. We add fixed-kinematics versus learned-residual and geometry-condition OFF comparisons to evaluate learned motion correction and geometric conditioning separately.

The learned residual improves all four pose-error measures at both horizons:

Horizon / casesState transitionADE ↓FDE ↓Yaw MAE ↓Pitch MAE ↓
10 s / 1,000Fixed kinematics130.74269.4316.27°3.30°
10 s / 1,000+ learned residual102.97229.6714.16°2.56°
30 s / 1,000Fixed kinematics305.12714.7919.38°3.46°
30 s / 1,000+ learned residual259.46678.5317.12°2.87°

We evaluate 1,000 Dust2 cases at each horizon, each with two controlled players and two first-person views. RGB context is frames 0–12; predictions cover 13–332 (10 s) or 13–972 (30 s), at 32 FPS. Both variants use identical cases, initial states, actions, and per-view seeds, with no future GT-pose resets. ADE/FDE are 3D camera-position errors in original game-coordinate units; yaw uses the shortest circular angular difference. Angular errors are in degrees.

Removing the geometry condition degrades all five visual metrics at every view count:

Evaluation cases: 10,000 for 2 and 4 views; 159 for 8 views. PSNR is in dB.

ViewsGeometryPSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
2ON26.24250.71300.17897.709836.5529
2OFF10.60770.42350.6758213.43001198.8300
4ON25.28250.70200.17727.982138.6849
4OFF10.92430.41930.6698207.71001178.0400
8ON26.57750.75730.147727.031153.6148
8OFF12.86530.45190.5801199.6600934.6000

ON and OFF use the original paper’s evaluation cases and otherwise matched generation settings. OFF removes the geometry condition. These two ablations separately support the learned transition correction and geometric conditioning.

We also evaluate whether generated views recover visible targets in four-player scenes:

MethodTarget recall ↑ (95% CI)Joint recall ↑ (95% CI)
Khora40.42% [26.74, 61.10]26.72% [7.07, 55.84]
MIRA32.10% [18.17, 55.04]24.43% [4.07, 54.72]
Gamma26.00% [10.66, 50.29]21.84% [1.88, 51.14]

The visibility evaluation uses 500 four-player cases with two-second clips. Localization recall counts generated detections at the corresponding locations of 673 detector-confirmed GT target observations. Joint recall uses 348 common-target observer pairs: when a target is visible to both GT observers, both generated views must detect it. Two simultaneous misses count as failure.

Khora achieves the highest localization recall and joint recall, at 40.42% and 26.72%. Bold and italics indicate the highest and second-highest point estimates; intervals are reported alongside them.

The non-CS mesh assets are supported by actual qualitative experiments, including Eiffel Tower-like scenes shown in Fig. 6. These examples demonstrate geometry-guided rendering on meshes beyond the training environment. They are additional qualitative demonstrations, distinct from the Dust2 training dataset and quantitative evaluation. We will revise the figure layouts and captions to clearly distinguish illustrative diagrams, actual qualitative results, and quantitative settings, and identify the mesh sources. Part 2 provides the fixed-budget and computational-scaling results.

Khora supports multiple agents per GPU. The original one-GPU-per-agent allocation evaluates population-dependent latency without contention between agents for a shared device. It is an experimental allocation; the same renderer can serve multiple agents on one device. The new measurements use PPU-ZW810E devices with 96 GB of memory and unchanged generation settings across budgets.

With fixed per-view settings, each requested view adds a call to the same renderer. Cross-agent information enters through projected shared-state conditions. The profiled rendering computation scales linearly: 15.695680 TFLOPs for one player and 251.130885 for 16, with 3.923920 TFLOPs per new view-RGB frame.

PlayersNew frames/requestTotal TFLOPs/requestTFLOPs/frameDiT TFLOPs/requestVAE TFLOPs/request
1415.6956803.9239207.8929647.802710
2831.3913613.92392015.78592815.605420
41662.7827213.92392031.57185531.210839
832125.5654433.92392063.14371062.421678
1664251.1308853.923920126.287421124.843357

Each request produces four new RGB frames per player. Frame counts and total view-FPS sum over all players; Peak/GPU is the sampled peak memory per device. FLOPs count supported operators in five DiT forwards, one streaming VAE decoding call, and recognized projection/condition-construction operations. Counted operators include matrix multiplication, convolution, and Flash/Efficient Attention. A multiply-add is two FLOPs; TFLOPs denotes 10¹² operations.

Under a fixed one-device budget, the measurements cover 1–13 players:

PlayersFPS/playerTotal view-FPSP50 (ms)P95 (ms)Peak/GPU (GiB)
133.1833.18120.33121.857.58
220.9841.96190.57191.9814.64
411.1444.56214.46360.5629.55
85.7646.08416.98698.8758.51
133.5446.02677.491137.2194.31

For a fixed 16-player world, 2→16 devices increases aggregate throughput from 91.32 to 529.05 view-frames/s, a 5.79× speedup:

GPUsPlayersFPS/playerTotal view-FPSP95 (ms)
2165.7091.32711.32
41611.01176.04371.37
81620.43326.94199.51
161633.06529.05123.71

Keeping eight players per device, 1→16 devices increases throughput from approximately 46.04 to 681.81 view-frames/s, reaching 92.55% of ideal linear throughput scaling:

GPUsPlayersFPS/playerTotal view-FPSP95 (ms)Peak/GPU (GiB)
185.7646.05698.8758.51
1133.5445.991137.2194.31
2165.7091.32711.3258.08
4325.61179.22724.1658.28
8645.55355.32728.9657.96
161285.34681.81765.4559.50

The scaling efficiency uses the eight-players-per-device rows. The one-device/13-player row additionally reports a measured population setting. These results characterize shared-device operation, fixed-population acceleration, and population growth with additional hardware using the same model.

Reviewer_Msr7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a system for the task of multi-agent world model. Specifically, it aims for the problem that previous multi-agent/-player world model can't deal with adaptive numbers of players, or extend freely in the inference time. The authors propose to explictly maintain a scene state, and let a video model based world model to do the final rendering for each player in parallel. Thus the whole system is not bounded and restricted by the number of players. The authors also propose a game-based multi-player dataset to train the model and run extensive experiments to compare with various baselines.

Critical strengths

The paper is with good intuition on trying to disentangle the state evolving and rendering process for the task of multi-agent/-player world model. It makes total sense that there should be two compoents, one being responsible for the scene state updates and is irrelevant to the number of views or players, which operates in the 3D / 4D latent space. Another model component being used to "render" the scene state or latent representations into view space, which can be shared across different views.

Critical weaknesses

The main weakness of this paper, in my perspective, is that the paper is overclaiming the scope of multi-agent / -player world model with current proposed system. The proposed Khora is actually a simple neural rendering model, and the entire scene state, scene transition are built using the explicit system with basic game kinematics. Then the "world model" part is essentially doing only the rendering part, by transforming G-Buffer style data to the desired image. The emphasized multi-agent consistency and population scalability therefore are ensured from the explicit state by construction. The part that would make this actually a world model in literature, which could be learned world transition model, is barely specified and never evaluated.

Thus the shared scene states are supplied basically from recorded CS:GO game data, rasterized into each target view, and rendered independently per view. Where the "model" is only doing the rendering and already given strong conditional cues. The only part in the paper that seems related to a "world model" is handling the scene transition instead of exlicit fixed rules, is the transition model of Eq. 2 and 4, which consists of a known kinematic mapping plus a residual. But this residual's architecture, training and details are then never reported.

Questions for authors
  1. Are there any pipeline results that uses the so called "predicted poses" instead of poses directly read from game states? If so, how is the performance, what is the pose error rate and what about the visual appearance degradation?
  2. What is the architecture, training details of the proposed residual model?
  3. Is there any support or model prediction that autonomously drives the agents, or all agents can be only controlled by explicitly commands?
  4. Could the authors provide the ablation results for the pipeline that doesn't have appearance memory updates?
  5. How were the baselines training configuration being set, they seems are using different training iterations?
2 comments

We thank the reviewer for the thoughtful critique. Khora combines persistent, explicit world state with generative video modeling. State evolution and observation generation have complementary roles. Structured priors are also used in DriveDreamer (ECCV 2024), which incorporates traffic constraints. Code World Model (2026) separates executable world evolution from neural visual generation.

STBoard maintains one evolving world state and projects it into individual viewpoints. A shared renderer synthesizes observations from these common spatial conditions and each view’s history. This decomposition supports cross-agent consistency and inference-time population expansion with the same trained parameters.

Khora recursively predicts future poses and dynamic states from initial observations/state, coarse scene geometry, and the supplied action/semantic stream. The state model combines a fixed kinematic motion proposal with a learned visual residual. Geometry supplies structure, while the diffusion model generates appearance and realistic observations. The original Table 2 results and the additional pose results use this visual residual. Table 4 separately evaluates rendering fidelity under recorded camera poses and supplied trajectories.

The residual reads each agent’s own state, actions, first-person observations, and history. A frozen DINOv3 ViT-L/16 extracts a CLS token and a 4×4 pooled spatial grid. Visual and state/action features are projected into a shared space and fused by cross-attention. Temporal feature differences and spatial correlations provide motion cues. A four-layer causal Transformer predicts local x/y/z and pitch/yaw corrections for the next four-frame block. The visual branch augments a frozen nonvisual residual base. Combined position corrections are rotated into world coordinates and added to the kinematic prediction. Parameters are shared across agents; histories remain per view.

We first train the visual branch on supervised local-motion residuals, then fine-tune with recursively propagated states. Losses supervise position, orientation, final-frame position, motion consistency, and residual regularization. Training combines synchronized recorded-image features with auxiliary supervision from cached generated observations. The nonvisual base, visual encoder, and renderer remain frozen. At inference, an observed prefix initializes state/history; subsequent corrections use the agent’s own generated images and supplied actions, without future GT images or pose resets.

The residual improves all four pose-error measures at both horizons:

Horizon / casesState transitionPosition ADE ↓Final-frame FDE ↓Yaw MAE ↓Pitch MAE ↓
10 s / 1,000Fixed kinematics130.74269.4316.27°3.30°
10 s / 1,000+ learned residual102.97229.6714.16°2.56°
30 s / 1,000Fixed kinematics305.12714.7919.38°3.46°
30 s / 1,000+ learned residual259.46678.5317.12°2.87°

We evaluate 1,000 Dust2 cases at each horizon, each with two controlled players and two first-person views. RGB context is frames 0–12; predictions cover 13–332 (10 s) or 13–972 (30 s), at 32 FPS. Both variants use identical cases, initial states, actions, and per-view seeds, with no future GT-pose resets. ADE/FDE are 3D camera-position errors in original game-coordinate units; yaw uses the shortest circular angular difference. Angular errors are in degrees.

Continuing our response, we compare fixed kinematics, kinematics plus the visual residual, and GT-pose conditioning using the same renderer. The learned residual improves all five visual metrics over fixed kinematics at 2, 4, and 8 views:

ViewsPose sourcePSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
2Fixed kinematics24.79130.71040.186210.293083.2843
2+ learned residual26.24250.71300.17897.709836.5529
2GT-pose conditioning28.04820.79870.13747.163829.3661
4Fixed kinematics24.10490.69730.188914.9372105.2849
4+ learned residual25.28250.70200.17727.982138.6849
4GT-pose conditioning27.73580.76230.163711.634432.8530
8Fixed kinematics25.04820.74480.159436.8711139.9372
8+ learned residual26.57750.75730.147727.031153.6148
8GT-pose conditioning28.64970.82770.132924.376241.3781

The visual comparison uses frames 13–80, with 10,000 cases at two views, 10,000 at four views, and 159 at eight views. Within each view count, all three pose sources use identical cases, renderer checkpoint, initial RGB, seeds, and generation settings. GT-pose conditioning supplies recorded future poses to the same renderer, which generates the RGB output. These visual cohorts are separate from the 1,000-case long-horizon pose experiments.

Relative to GT-pose conditioning, predicted poses reduce PSNR by 1.8057/2.4533/2.0722 dB and increase LPIPS by 0.0415/0.0135/0.0148 at 2/4/8 views. This quantifies the end-to-end effect of pose prediction, including trajectory misalignment with recorded GT. The pose errors in Part 1 and this visual comparison jointly evaluate state prediction and its downstream rendering effect.

Within each matched comparison, bold marks the best value and italics the second-best among three settings. GT-pose results provide a supplied-pose reference.

Khora is a World Action Model (WAM): the controller chooses actions, while Khora predicts their consequences and generates the corresponding observations. Our quantitative evaluation uses externally provided action/semantic streams. We have also completed preliminary RL-based autonomous-control explorations, allowing agents to navigate the generated world without human intervention. Human actions, predefined trajectories, and learned policies use the same action-conditioned transition and rendering interface. Integrating learned policies therefore provides a route toward large-scale autonomous multi-agent simulation.

All baselines were retrained on the same dataset as Khora, preserving their original architectures, training objectives, and recommended hyperparameter configurations as faithfully as possible. We followed the respective authors’ training procedures and continued training until convergence. Different iteration counts reflect different architectures, training strategies, and convergence behavior. Evaluating each method after convergence under its intended configuration gives a meaningful comparison across these pipelines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.