acceptodds
Under review as a conference paper at ICLR 2027

MoLan: Motion-Language Understanding and Generation for Traffic Simulation

Abstract

Realistic, reactive, and controllable multi-agent traffic simulation is essential for the scalable development and evaluation of autonomous driving systems. Natural language provides an intuitive interface both for controlling how traffic scenes unfold and for querying the behaviors and interactions represented in them. Existing approaches, however, typically specialize in either language-conditioned motion generation or motion-grounded scene understanding. We introduce MoLan, a Motion-Language model that connects pretrained traffic and language towers through lightweight bidirectional bridge blocks. The resulting architecture supports native motion simulation and text generation, language-controlled closed-loop multi-agent rollouts, and motion-grounded traffic-scene understanding within a single model. MoLan outperforms task-specific baselines on both language-controlled simulation and motion-grounded question answering, while joint training across both directions further improves closed-loop realism, indicating that motion-understanding supervision can strengthen language-controlled generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Large Language Models, Autonomous Driving, Scenario Generation, Traffic Simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rsLTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a straightforward mechanism that allows for joint modeling of language and motion inputs for multi-agent traffic simulation. The architecture adopts a cross-modality attention layer over a pre-trained language model and a motion simulator, allowing for joint learning of text and motion tasks. Experiments cover various modalities using large-scale datasets and standard metrics, where consistent improvements are shown to support the main claims.

Critical strengths
  1. The paper adopts a simple and straightforward bridge-layer design that connects pre-trained language models to pre-trained SMART motion simulator while preserving the native abilities of the backbones.
  2. Decent amount of evaluations and ablations cover various within- and cross-modality tasks against matching baselines, where improvements are reported over the standard metrics supported by the chosen datasets.
Critical weaknesses

The main concern is regarding the lack of actual semantic correctness metric to verify "creating targeted scenarios and understanding the resulting agent interactions" as claimed in Contribution-1.

Currently, the performance of language-conditioned motion generation is reported by distributional kinematics such as RMM and mADE, while the question-answering tasks are reported by text similarity such as BELU and BERTScore. These metrics do not seem to reflect semantic instruction satisfaction or relation realization, e.g.

  • An erroneous rollout that does not semantically satisfy the input instruction of "stop"/"yield" (if theses notions are defined);
  • An answer that is textually close but semantically incorrect ("The relation between A1 & A2 is x" and "The relation between A1 & A2 is y").

While the current metrics are appropriate for assessing overall pipeline performance, it would be good if a semantic metric such as instruction satisfaction rate could be reported (e.g. on a subset of the test cases).

Questions for authors

Main Questions:

  1. Do the current metrics reflect semantic correctness as stated in the weakness section?
  2. According to Tab-4, the M2T- and T2M-only variants already seem to demonstrate strong bidirectional performances without specific bidirectional training. Does this imply that the backbones already have the ability to generalize across modalities?

Other Clarifications:

  1. Sec-4.2 adopts learned agent-ID embedding for cross-modality feature correspondence. Why is this chosen over a fixed embedding, and does it affect the overall performance?
  2. Does the bridge-layers require re-training when migrating to other backbones in Sec-5.2?
5 comments

We thank the reviewer for the positive feedback and for recognizing our simple bridge-layer design. We also appreciate the acknowledgment of our extensive evaluations and ablations. We address the remaining concerns below.

Weakness 1:

  1. The main concern is regarding the lack of actual semantic correctness metric to verify "creating targeted scenarios and understanding the resulting agent interactions" as claimed in Contribution-1.
  2. Currently, the performance of language-conditioned motion generation is reported by distributional kinematics such as RMM and mADE, while the question-answering tasks are reported by text similarity such as BELU and BERTScore. These metrics do not seem to reflect semantic instruction satisfaction or relation realization, e.g.
    a. An erroneous rollout that does not semantically satisfy the input instruction of "stop"/"yield" (if theses notions are defined); b. An answer that is textually close but semantically incorrect ("The relation between A1 & A2 is x" and "The relation between A1 & A2 is y").
  3. While the current metrics are appropriate for assessing overall pipeline performance, it would be good if a semantic metric such as instruction satisfaction rate could be reported (e.g. on a subset of the test cases).

Question 1: Do the current metrics reflect semantic correctness as stated in the weakness section?

Instruction-following: We agree that RMM and minADE alone do not directly establish whether a rollout follows its instruction. However, we would like to clarify that we do report controllability, which measures how well the model follows the prompt by comparing the model’s realism gain against the unconditional model rollout. Following ProSim and ECoSim, we implement this metric by comparing relative improvement in rollout minADE with and without prompt conditioning. Tab. 2 shows that MoLan achieves the highest %Gain among language-conditioned methods (21.22% vs. 16.39% for ECoSim, both using CAT-k as the motion model).

To further strengthen the controllability evaluation, we additionally introduce Instruction-Success Rate (ISR), which measures whether a referenced agent performs the requested behavior. We first define 8 high-level action types: longitudinal (accelerate, decelerate, maintain speed, stopping, and parked) and lateral (left turn, right turn, and straight). We implement heuristic functions to detect each of the actions in a trajectory. We calculate ISR as the percentage of requested actions that are detected in the generated rollouts for the corresponding agents.

For each initial scene, we evaluate both original and counterfactual instructions. Original instructions describe actions extracted from the logged future. For the counterfactual setting, we randomly select 50% of simulated agents and change their longitudinal and/or lateral actions (e.g., replacing a left turn with a right turn, or accelerating with stopping), then convert the modified actions into text instructions using a prompt template. Because randomly modified instructions are not guaranteed to be feasible, we additionally report collision, off-road, and traffic-light violation rates to assess the safety of the resulting rollouts.

We compute ISR by first averaging action accuracy across agents within each scene, then averaging across all scenes.

The table below shows that MoLan achieves the highest ISR under both original and counterfactual instructions: 88.27% and 66.82%, respectively. Under counterfactual instructions, it exceeds ECoSim by 16.59 percentage points (66.82% versus 50.24%), while also achieving lower collision (3.25% versus 4.24%), off-road (2.24% versus 2.98%), and traffic-light violation rates (4.82% versus 8.03%).

Instruction-Success Rate (ISR) ↑Collision rate (%) ↓Off-road (%) ↓Traffic-light violation (%) ↓
Original instructions
ProSim69.2783.9283.4525.299
ECoSim76.2893.1392.3733.327
MoLan (Ours)88.2682.2721.9782.529
Counterfactual instructions (50% of agents)
ProSim10.8405.1294.2928.186
ECoSim50.2404.2402.9838.029
MoLan (Ours)66.8273.2462.2374.824

Weakness 1:

  1. The main concern is regarding the lack of actual semantic correctness metric to verify "creating targeted scenarios and understanding the resulting agent interactions" as claimed in Contribution-1.
  2. Currently, the performance of language-conditioned motion generation is reported by distributional kinematics such as RMM and mADE, while the question-answering tasks are reported by text similarity such as BELU and BERTScore. These metrics do not seem to reflect semantic instruction satisfaction or relation realization, e.g.
    a. An erroneous rollout that does not semantically satisfy the input instruction of "stop"/"yield" (if theses notions are defined); b. An answer that is textually close but semantically incorrect ("The relation between A1 & A2 is x" and "The relation between A1 & A2 is y").
  3. While the current metrics are appropriate for assessing overall pipeline performance, it would be good if a semantic metric such as instruction satisfaction rate could be reported (e.g. on a subset of the test cases).

Question 1: Do the current metrics reflect semantic correctness as stated in the weakness section?

Evaluating QA correctness using LLM-as-a-Judge approach (GPTScore): To better assess the quality of the generated answers and whether the generated response really addresses the user question, we use the LLM-as-a-Judge approach. We use GPT-6.1 Luna, which is presented with a question, the predicted and reference answers, and tasked to determine a score on a scale of 0 to 10 (higher is better). We use the prompt used in WOMD-Reasoning to evaluate GPTScore. Due to the limitations of computational resources of LLM, we evaluate GPTScore on a subset of 2000 question-answer (QA) pairs, stratified by question type across four categories: road environment, ego agent’s motion status, surrounding agents’ states, and agent interactions and intentions.

The results show that MoLan is able to generate well-grounded responses. Specifically, MoLan achieves the highest GPTScore of 8.343, outperforming Motion-LLaVA (6.503) by 28.3%. Beyond the overall performance, MoLan consistently achieves the best performance across all four categories, demonstrating its versatility in understanding diverse driving scenarios. Most notably, MoLan achieves an 8.128 GPTScore on interactions and intentions, outperforming Motion-LLaVA (4.239).

Question typeRoad environmentEgo agent’s motion statusOther (surrounding) agents’ motion statusInteractions and intentionsOverall
Non-finetuned VLM w. BEV input3.2382.0211.3240.9911.894
Finetuned VLM w. BEV input6.3294.1223.2392.4714.040
Motion-LLaVA7.7247.8136.2374.2396.503
MoLan (Ours)9.1288.1297.2878.1288.343

Question 2: According to Tab-4, the M2T- and T2M-only variants already seem to demonstrate strong bidirectional performances without specific bidirectional training. Does this imply that the backbones already have the ability to generalize across modalities?

To better understand the performance differences between the training variants, we expand Table 4 to include the RMM breakdown (kinematic, interaction, and map-based), minADE, %Gain for instruction conditioning, and additional QA metrics: GPTScore, factual accuracy, and median error (m). Compared with T2M-only training, joint supervision improves the interaction score from 0.797 to 0.819 (+2.76%) and increases controllability %Gain from 17.943% to 21.22% (+18.26% relative; +3.28 percentage points).

Additionally, although T2M-only achieves a BERTScore of approximately 0.605, its GPTScore is only 3.981, compared with 8.343 for joint training, and its factual accuracy is 22.3%, compared with 68.1%. We attribute this discrepancy to BERTScore measuring contextual similarity to the reference answer, which can remain high despite incorrect scene details. Conversely, M2T-only performs well on QA but shows substantially weaker motion generation, with a minADE of 2.540 and a negative %Gain of -69.175%.

RMM ↑Kinematic ↑Interaction ↑Map-based ↑minADE ↓%Gain ↑GPTScore ↑Factual QAs: Accuracy ↑Factual QAs: Median Error (m) ↓
M2T-only0.7120.4700.6810.8902.540-69.175%8.2910.6091.251
T2M-only0.7750.4870.7970.9111.232+17.943%3.9810.2239.125
Joint training0.7860.4900.8190.9121.18321.22%8.3430.6810.513

Joint supervision improves generation RMM from 0.775 to 0.786 (+1.42%) over T2M-only training. Compared with M2T-only training, it increases GPTScore from 8.291 to 8.343 (+0.63%), improves factual accuracy from 60.9% to 68.1% (+7.2 percentage points), and reduces median error from 1.251 m to 0.513 m (−58.99%). These results support the benefit of joint training: supervising both tasks improves motion generation and factual traffic understanding beyond their respective single-task variants.

Clarification 1: Sec-4.2 adopts learned agent-ID embedding for cross-modality feature correspondence. Why is this chosen over a fixed embedding, and does it affect the overall performance?

We use learnable agent-ID embeddings to allow the representations establishing cross-modality feature correspondence to be optimized jointly with the model. To evaluate this choice, we compare learnable and fixed sinusoidal agent-ID embeddings:

Agent-ID embeddingRMM ↑Kinematic ↑Interaction ↑Map-based ↑minADE ↓%Gain ↑GPTScore ↑
Fixed sinusoidal0.7770.4890.8020.9091.20719.58%7.892
Learnable0.7860.4900.8190.9121.18321.22%8.343

Learnable embeddings consistently improve both motion generation and traffic understanding: RMM increases from 0.777 to 0.786, minADE decreases from 1.207 to 1.183, and the GPTScore increases from 7.892 to 8.343. These results support our choice of learnable embeddings. Importantly, even with fixed sinusoidal embeddings, MoLan outperforms ProSim, ECoSim, and LangTraj on RMM and minADE, and Motion-LLaVA on GPTScore.

Clarification 2: Does the bridge-layers require re-training when migrating to other backbones in Sec-5.2?

Yes, the bridge layers require training when migrating to other backbones. However, since the motion and language models remain frozen, only the bridge blocks, agent-ID embeddings, and text embeddings for agent tokens are trained. This makes the training process fast and computationally inexpensive.

Reviewer_r2BdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoLan connects frozen traffic and language models through bidirectional bridges for language-controlled multi-agent simulation and scene question answering. Joint training improves both tasks over single-task variants.

Critical strengths

Well motivatedl: traffic control and understanding through a shared language interface.

Preserves native backbone computation by construction. Tested with several motion and language backbones.

Strong reported results against the evaluated baselines, including ECoSim with the same motion backbone and Motion-LLaVA trained on the same QA data.

Critical weaknesses

limited methodological novelty. Language-controlled simulation, lightweight adaptation of frozen traffic models, motion-grounded QA, and gated cross-attention all have close precedents. The contribution seems primarily their integration. Unclear what this unification enables beyond supporting two existing tasks in one model.

Joint-training transfer is potentially the most interesting result, but improvement is not significant. RMM improves from 0.777 to 0.786; no uncertainty estimates or interaction-component ablation. The claim that understanding supervision particularly improves interactions is therefore not directly supported.

Language control evaluated indirectly through realism and proximity to the logged future. Neither establishes reliable instruction following, especially for counterfactual instructions. Would like direct maneuver-success rates and paired results under different instructions for the same scene.

QA improvements are promising, but aggregate text metrics offer limited evidence of interaction reasoning. Missing breakdown by question type and analysis of incorrect answers.

Questions for authors

What benefit requires a unified model, beyond maintaining separate control and QA systems? Any experiment demonstrating a combined capability?

For joint versus single-task training, are task-specific updates and data exposure matched? Can the authors report interaction scores and variability across runs?

What are the exact training/evaluation splits and sizes after expanding WOMD-Reasoning? How are generated answers checked?

6 comments

We thank the reviewer for the positive and constructive feedback. We appreciate the recognition of our motivation, of the design's generality across backbones, and of the strength of our results against controlled baselines. We address the reviewer's concerns below.

Weakness 1: limited methodological novelty. Language-controlled simulation, lightweight adaptation of frozen traffic models, motion-grounded QA, and gated cross-attention all have close precedents. The contribution seems primarily their integration. Unclear what this unification enables beyond supporting two existing tasks in one model.

Question 1: What benefit requires a unified model, beyond maintaining separate control and QA systems? Any experiment demonstrating a combined capability?

We agree that gated cross-attention and frozen-backbone adaptation are established mechanisms, and we do not claim them as contributions. Our contribution is the new tasks they enable. Concretely, MoLan differs from prior work (Tab. 1) in three ways:

  • Unified scene understanding and language-conditioned rollout generation: Prior traffic models are unidirectional. ProSim, ECoSim, and LangTraj only generate motion from text, whereas Motion-LLaVA only describes motion in text. MoLan supports both tasks using the same backbones. Beyond supporting two tasks, joint supervision improves motion-generation realism and factual-answer quality, as quantified below.
  • Multi-agent grounding: Dual-tower human-motion-language models such as LLaMo [1] and MotionGPT3 [2] target single-body human motion, where language does not face the same need to distinguish among multiple agents. Traffic scenes in WOMD are complex and can contain up to 128 agents, and instructions and answers are agent-specific. Our shared agent identity embeddings establish cross-modality feature correspondence.
  • Native-capability preservation: Prior methods fine-tune or modify the language model and/or motion model, which couples them to one architecture and alters its native behavior and capabilities. MoLan leaves both towers frozen. Unconditional simulation is exactly that of the underlying simulator (Table 2), and the same bridge design transfers across three motion and three language backbones with consistent gains (Table 5). MoLan can therefore adopt stronger simulators or LLMs as they appear, which prior language-conditioned simulators cannot do without redesign.

Additionally, to quantify the benefit of unification, we expanded Table 4 with additional generation and QA metrics. Compared with T2M-only training, joint supervision improves the interaction score from 0.797 to 0.819 (+2.76%) and increases controllability %Gain from 17.943% to 21.22% (+18.26% relative; +3.28 percentage points). Thus, understanding supervision benefits both the realism of agent interactions and the effectiveness of language conditioning. Overall generation RMM also improves from 0.775 to 0.786 (+1.42%).

The transfer also benefits scene understanding. Compared with M2T-only training, joint supervision improves factual accuracy from 60.9% to 68.1% (+7.2 percentage points) and reduces median error from 1.251 m to 0.513 m (−58.99%). These gains show that motion-generation supervision improves both the accuracy and precision of factual answers. Together, the results support a unified model through cross-task improvements, beyond simply supporting generation and understanding in one architecture. Moreover, they show that the joint training make the most of the two tasks by improving the metrics that are favoured by each of them, e.g. RMM for T2M and Factual QAs for M2T.

RMM ↑Kinematic ↑Interaction ↑Map-based ↑minADE ↓%Gain ↑GPTScore ↑Factual QAs: Accuracy ↑Factual QAs: Median Error (m) ↓
M2T-only0.7120.4700.6810.8902.540-69.175%8.2910.6091.251
T2M-only0.7750.4870.7970.9111.232+17.943%3.9810.2239.125
Joint training0.7860.4900.8190.9121.18321.22%8.3430.6810.513

References:

  1. Li, Zekun, et al. “LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens.” In CVPR, 2026.
  2. Zhu, Bingfan, et al. “MotionGPT3: Human Motion as a Second Modality.” In ICLR, 2026.

Weakness 2: Joint-training transfer is potentially the most interesting result, but improvement is not significant. RMM improves from 0.777 to 0.786; no uncertainty estimates or interaction-component ablation. The claim that understanding supervision particularly improves interactions is therefore not directly supported.

Question 2: For joint versus single-task training, are task-specific updates and data exposure matched? Can the authors report interaction scores and variability across runs?

Single-task (M2T-only, T2M-only) and joint training use the same training scenarios, the same number of training steps, and the same batch size and optimizer. They differ only in the training objective.

As discussed in our previous response, we expanded Table 4 to include the RMM breakdown (kinematic, interaction, and map-based), minADE, %Gain for instruction conditioning, and additional QA metrics: GPTScore, factual accuracy, and median error (m). The largest improvement among the RMM components is in interactions, where joint training increases the score from 0.797 to 0.819 (+2.76%) compared with T2M-only training. This directly supports our claim that understanding supervision particularly benefits interaction realism.

Variability across sampling seeds: We repeat evaluation with five different rollout-sampling seeds, keeping the trained checkpoints, evaluation scenarios, and sampling configuration (temperature 1.0) fixed. For each seed, we generate 32 joint rollouts per scenario. WOSAC uses these rollouts together to estimate feature distributions and compute scenario-level scores, which are then aggregated across the evaluation set. We report the mean and standard deviation across these evaluations. Joint training improves the interaction score from 0.797 to 0.819, with a standard deviation of 0.001 for both models.

RMM ↑Kinematic ↑Interaction ↑Map-based ↑
M2T-only0.712 (0.001)0.470 (0.000)0.681 (0.001)0.890 (0.001)
T2M-only0.775 (0.001)0.487 (0.003)0.797 (0.001)0.911 (0.002)
Joint training0.785 (0.001)0.489 (0.003)0.819 (0.001)0.912 (0.000)

Weakness 3: Language control is evaluated indirectly through realism and proximity to the logged future. Neither establishes reliable instruction following, especially for counterfactual instructions. Would like direct maneuver-success rates and paired results under different instructions for the same scene.

Instruction-following: We agree that RMM and minADE alone do not directly establish whether a rollout follows its instruction. However, we would like to clarify that we do report controllability, which measures how well the model follows the prompt by comparing the model’s realism gain against the unconditional model rollout. Following ProSim and ECoSim, we implement this metric by comparing relative improvement in rollout minADE with and without prompt conditioning. Tab. 2 shows that MoLan achieves the highest %Gain among language-conditioned methods (21.22% vs. 16.39% for ECoSim, both using CAT-k as the motion model).

To further strengthen the controllability evaluation, we additionally introduce Instruction-Success Rate (ISR), which measures whether a referenced agent performs the requested behavior. We first define 8 high-level action types: longitudinal (accelerate, decelerate, maintain speed, stopping, and parked) and lateral (left turn, right turn, and straight). We implement heuristic functions to detect each of the actions in a trajectory. We calculate ISR as the percentage of requested actions that are detected in the generated rollouts for the corresponding agents.

For each initial scene, we evaluate both original and counterfactual instructions. Original instructions describe actions extracted from the logged future. For the counterfactual setting, we randomly select 50% of simulated agents and change their longitudinal and/or lateral actions (e.g., replacing a left turn with a right turn, or accelerating with stopping), then convert the modified actions into text instructions using a prompt template. Because randomly modified instructions are not guaranteed to be feasible, we additionally report collision, off-road, and traffic-light violation rates to assess the safety of the resulting rollouts.

We compute ISR by first averaging action accuracy across agents within each scene, then averaging across all scenes.

The table below shows that MoLan achieves the highest ISR under both original and counterfactual instructions: 88.27% and 66.82%, respectively. Under counterfactual instructions, it exceeds ECoSim by 16.59 percentage points (66.82% versus 50.24%), while also achieving lower collision (3.25% versus 4.24%), off-road (2.24% versus 2.98%), and traffic-light violation rates (4.82% versus 8.03%).

Instruction-Success Rate (ISR) ↑Collision rate (%) ↓Off-road (%) ↓Traffic-light violation (%) ↓
Original instructions
ProSim69.2783.9283.4525.299
ECoSim76.2893.1392.3733.327
MoLan (Ours)88.2682.2721.9782.529
Counterfactual instructions (50% of agents)
ProSim10.8405.1294.2928.186
ECoSim50.2404.2402.9838.029
MoLan (Ours)66.8273.2462.2374.824

Quantitative results: In the updated manuscript, Figs. 3 and 7 show three different scenes, each paired with three distinct instructions that differ from the logged behavior. Across these prompts, we generate both unconditioned and text-conditioned traffic. MoLan demonstrates the ability to generate realistic behaviors while adhering to the provided instructions.

Reviewer_QxkbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoLan integrates pretrained motion and language models for language-conditioned multi-agent traffic simulation and motion-grounded question answering. The method connects frozen backbones through bidirectional cross-attention bridges, with shared agent embeddings for cross-modal correspondence and zero-initialized gates for stable adaptation. Experiments on WOMD, ProSim-Instruct, and an expanded WOMD-Reasoning dataset evaluate motion generation, question answering, and the effect of joint training.

Critical strengths

(+) The architecture supports motion generation and scene understanding within a single framework while retaining the native pathways of both pretrained models. The shared agent embeddings provide an explicit correspondence between language references and motion tokens.

(+) MoLan performs well on both tasks. It achieves 0.786 RMM and 1.183 minADE for language-conditioned simulation, and 68.1% factual-answer accuracy compared with 55.4% for the reproduced Motion-LLaVA baseline.

Critical weaknesses

(-) Instruction following is not directly quantified. The reported RMM and minADE assess traffic realism and agreement with the recorded future, but do not establish whether the model executes different feasible instructions for the same initial scene. The controllability claim would be stronger with instruction-success rates, including whether each requested behavior is assigned to the correct agent.

(-) Can the authors report instruction-success rates for multiple feasible prompts applied to the same initial scene, including prompts that request behavior different from the recorded future?

(-) What are the additional trainable parameter count, peak GPU memory, and rollout latency introduced by the bridges? How do these costs scale with the number of agents and bridge blocks?

Questions for authors

Please refer to Weaknesses.

3 comments

We thank the reviewer for the positive and constructive feedback. We appreciate the recognition of our unified framework for motion generation and scene understanding, and of its strong performance on both tasks. We address the reviewer's concerns below.

Weakness 1: Instruction following is not directly quantified. The reported RMM and minADE assess traffic realism and agreement with the recorded future, but do not establish whether the model executes different feasible instructions for the same initial scene. The controllability claim would be stronger with instruction-success rates, including whether each requested behavior is assigned to the correct agent.

We agree that RMM and minADE alone do not directly establish whether a rollout follows its instruction. However, we would like to clarify that we do report controllability, which measures how well the model follows the prompt by comparing the model’s realism gain against the unconditional model rollout. Following ProSim and ECoSim, we implement this metric by comparing relative improvement in rollout minADE with and without prompt conditioning. Tab. 2 shows that MoLan achieves the highest %Gain among language-conditioned methods (21.22% vs. 16.39% for ECoSim, both using CAT-k as the motion model).

To further strengthen the controllability evaluation, we additionally introduce Instruction-Success Rate (ISR), which measures whether a referenced agent performs the requested behavior. We first define 8 high-level action types: longitudinal (accelerate, decelerate, maintain speed, stopping, and parked) and lateral (left turn, right turn, and straight). We implement heuristic functions to detect each of the actions in a trajectory. We calculate ISR as the percentage of requested actions that are detected in the generated rollouts for the corresponding agents.

For each initial scene, we evaluate both original and counterfactual instructions. Original instructions describe actions extracted from the logged future. For the counterfactual setting, we randomly select 50% of simulated agents and change their longitudinal and/or lateral actions (e.g., replacing a left turn with a right turn, or accelerating with stopping), then convert the modified actions into text instructions using a prompt template. Because randomly modified instructions are not guaranteed to be feasible, we additionally report collision, off-road, and traffic-light violation rates to assess the safety of the resulting rollouts.

We compute ISR by first averaging action accuracy across agents within each scene, then averaging across all scenes.

The table below shows that MoLan achieves the highest ISR under both original and counterfactual instructions: 88.27% and 66.82%, respectively. Under counterfactual instructions, it exceeds ECoSim by 16.59 percentage points (66.82% versus 50.24%), while also achieving lower collision (3.25% versus 4.24%), off-road (2.24% versus 2.98%), and traffic-light violation rates (4.82% versus 8.03%).

Instruction-Success Rate (ISR) ↑Collision rate (%) ↓Off-road (%) ↓Traffic-light violation (%) ↓
Original instructions
ProSim69.2783.9283.4525.299
ECoSim76.2893.1392.3733.327
MoLan (Ours)88.2682.2721.9782.529
Counterfactual instructions (50% of agents)
ProSim10.8405.1294.2928.186
ECoSim50.2404.2402.9838.029
MoLan (Ours)66.8273.2462.2374.824

Weakness 2: Can the authors report instruction-success rates for multiple feasible prompts applied to the same initial scene, including prompts that request behavior different from the recorded future?

In the updated manuscript, Figs. 3 and 7 show three different scenes, each paired with three distinct instructions that differ from the logged behavior. Across these prompts, we generate both unconditioned and text-conditioned traffic. MoLan demonstrates the ability to generate realistic behaviors while adhering to the provided instructions. Computing the instruction-success rates at a large scale is challenging as it is not possible to know a priori what are the feasible prompts for a given scene.

Weakness 3: What are the additional trainable parameter count, peak GPU memory, and rollout latency introduced by the bridges? How do these costs scale with the number of agents and bridge blocks?

Trainable parameters: With the default configuration used throughout the paper, the bridges add 15M trainable parameters. This includes the bridge blocks (Q/K/V projections and gates), the agent identity embeddings, and the new agent tokens added to the language model's vocabulary. The motion and language models remain frozen.

Runtime analysis: We measure the runtime of MoLan using a single H100 GPU under two different settings: (1) the number of simulated agents per scenario, and (2) the rollout latency introduced by the bridges. We use SMART-7M as the motion backbone and LLaMA-3.2-1B as the language backbone. We measure the incurred latency and peak memory with and without conditioning.

  • Scaling with the number of simulated agents: Table below reports the average inference time of MoLan in milliseconds (ms) and peak memory in gigabytes (GB) for one scenario with batch size 1 with different numbers of simulated agents. The results show that MoLan supports up to 128 simulated agents with an additional inference cost over unconditional motion generation. With cross-attention bridges, language conditioning increases latency by 14.61-32.81% across different numbers of simulated agents. At 128 agents, this corresponds to an additional 56.89 ms, giving a total latency of 230.26 ms. Additionally, peak memory increases from 0.19 GB to 3.34 GB at this setting; this difference includes the language tower and conditioning-related computations. Compared with full attention, cross-attention reduces latency by 7.94-9.31% across different numbers of simulated agents, while maintaining similar peak memory usage.
# simulated agents163264128
MoLan (unconditional; Motion-to-Motion)
Latency (ms)117.88123.67137.05173.37
Peak memory (GB)0.060.070.090.19
MoLan with full-attention
Latency (ms)148.97159.53185.35250.12
Peak memory (GB)2.943.003.123.36
MoLan with cross-attention
Latency (ms)135.10146.74169.95230.26
Peak memory (GB)2.963.013.123.34
  • Scaling with the number of bridge blocks: Table below reports latency and peak memory as the number of bridge blocks increases from one to six, with SMART as the motion backbone and the number of simulated agents fixed at 64. Cross-attention latency increases from 160.81 ms to 169.95 ms (+5.68%), compared with 163.07 ms to 185.35 ms (+13.66%) for full attention. Peak memory increases from 3.03 GB to 3.12 GB (+2.97%) for both configurations.
# bridge blocks1246
MoLan with full-attention
Latency (ms)163.07167.03176.37185.35
Peak memory (GB)3.033.053.093.12
MoLan with cross-attention
Latency (ms)160.81162.95166.55169.95
Peak memory (GB)3.033.053.093.12

Overall, cross-attention provides a better trade-off compared with using full attention, with a smaller increase in latency as the number of bridge blocks grows and a similar increase in memory usage, while maintaining the performance benefits shown in Fig. 5.

Dear Reviewer Qxkb,

As the discussion deadline is approaching, we would like to kindly check whether our response has addressed all your concerns satisfactorily. Your insights are valuable to enhance our work, so please let us know if there are any remaining points that are still unclear.

If your concerns are addressed, we would be truly grateful if you would consider reflecting that in your scoring! Thank you for your time and constructive feedback!

Comments on the paper

2 comments
Area_Chair_jqkGOpenReview ↗

Discussion

Dear Reviewers, The authors have posted the rebuttal. Please review their responses and engage in the discussion.

Best, AC

AuthorsOpenReview ↗

Overview of Rebuttal Phase

We thank the reviewers for their thoughtful assessments and encouraging feedback. The reviewers recognized that MoLan is simple, straightforward, and well motivated (r2Bd, rsLT), and uses a simple bridge design that preserves the pretrained backbones’ native abilities (Qxkb, r2Bd, rsLT). They also noted that it works across several backbones (r2Bd), outperforms existing models on both language-controlled motion generation and QA (Qxkb, r2Bd), and includes thorough evaluations and ablations (rsLT, r2Bd).

The post-rebuttal average rating is 3.00, with an average confidence of 3.33. Reviewer r2Bd recognized that our additional experiments strengthen the paper and raised their score from 2 to 3 (weak accept). Reviewer rsLT stated that we had addressed their concerns and reaffirmed their positive assessment of the paper. Reviewer Qxkb did not engage during the rebuttal phase, but we addressed all their points comprehensively.

Below, we address each concern and summarize how it was addressed during the rebuttal.

  1. Instruction following (Qxkb, r2Bd, rsLT). We added a new Instruction-Success Rate (ISR) evaluation on original and counterfactual instructions. MoLan achieves 88.3% and 66.8%, respectively, with the lowest safety-violation rates, demonstrating its instruction-following capabilities compared to existing methods.

  1. QA correctness (r2Bd, rsLT). We added an LLM-as-a-Judge evaluation (GPTScore) with a per-category breakdown. MoLan scores 8.34 versus 6.50 for Motion-LLaVA overall, and 8.13 versus 4.24 on interaction questions.

  1. Novelty (r2Bd). We clarified how MoLan provides novel contributions to prior work (Table 1) in three ways: (i) unified scene understanding and language-conditioned rollout generation: prior traffic models either generate motion from text or describe it, whereas MoLan supports both, and joint supervision improves each (point 4); (ii) multi-agent grounding: unlike dual-tower models for single-body human motion (e.g., LLaMo, MotionGPT3), our shared agent identity embeddings establish internally a multi-agent cross-modality correspondence for up to 128 agents; (iii) native-capability preservation: both towers remain frozen, which inherently preserves the native simulation (Table 2) and language understanding of the base models, which guarantees no degradation in performance on the native tasks.

  1. Joint training (r2Bd, rsLT). Understanding supervision improves generated interactions (interaction RMM score: 0.797 to 0.819; +2.76%), while motion-generation supervision improves QA factual accuracy (60.9% to 68.1%; +7.2 percentage points). These cross-task gains demonstrate the benefits of joint training.

  1. Computational cost (Qxkb). The bridges add only 15M trainable parameters and modest latency (about 57 ms at 128 agents). Because both backbones remain frozen and the same bridge design transfers across motion and language backbones with consistent gains (Table 5), MoLan can directly benefit from future advances in motion models and LLMs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.