- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a straightforward mechanism that allows for joint modeling of language and motion inputs for multi-agent traffic simulation. The architecture adopts a cross-modality attention layer over a pre-trained language model and a motion simulator, allowing for joint learning of text and motion tasks. Experiments cover various modalities using large-scale datasets and standard metrics, where consistent improvements are shown to support the main claims.
Critical strengths
- The paper adopts a simple and straightforward bridge-layer design that connects pre-trained language models to pre-trained SMART motion simulator while preserving the native abilities of the backbones.
- Decent amount of evaluations and ablations cover various within- and cross-modality tasks against matching baselines, where improvements are reported over the standard metrics supported by the chosen datasets.
Critical weaknesses
The main concern is regarding the lack of actual semantic correctness metric to verify "creating targeted scenarios and understanding the resulting agent interactions" as claimed in Contribution-1.
Currently, the performance of language-conditioned motion generation is reported by distributional kinematics such as RMM and mADE, while the question-answering tasks are reported by text similarity such as BELU and BERTScore. These metrics do not seem to reflect semantic instruction satisfaction or relation realization, e.g.
- An erroneous rollout that does not semantically satisfy the input instruction of "stop"/"yield" (if theses notions are defined);
- An answer that is textually close but semantically incorrect ("The relation between A1 & A2 is x" and "The relation between A1 & A2 is y").
While the current metrics are appropriate for assessing overall pipeline performance, it would be good if a semantic metric such as instruction satisfaction rate could be reported (e.g. on a subset of the test cases).
Questions for authors
Main Questions:
- Do the current metrics reflect semantic correctness as stated in the weakness section?
- According to Tab-4, the M2T- and T2M-only variants already seem to demonstrate strong bidirectional performances without specific bidirectional training. Does this imply that the backbones already have the ability to generalize across modalities?
Other Clarifications:
- Sec-4.2 adopts learned agent-ID embedding for cross-modality feature correspondence. Why is this chosen over a fixed embedding, and does it affect the overall performance?
- Does the bridge-layers require re-training when migrating to other backbones in Sec-5.2?
5 comments
Weakness 1:
- The main concern is regarding the lack of actual semantic correctness metric to verify "creating targeted scenarios and understanding the resulting agent interactions" as claimed in Contribution-1.
- Currently, the performance of language-conditioned motion generation is reported by distributional kinematics such as RMM and mADE, while the question-answering tasks are reported by text similarity such as BELU and BERTScore. These metrics do not seem to reflect semantic instruction satisfaction or relation realization, e.g.
a. An erroneous rollout that does not semantically satisfy the input instruction of "stop"/"yield" (if theses notions are defined); b. An answer that is textually close but semantically incorrect ("The relation between A1 & A2 is x" and "The relation between A1 & A2 is y").- While the current metrics are appropriate for assessing overall pipeline performance, it would be good if a semantic metric such as instruction satisfaction rate could be reported (e.g. on a subset of the test cases).
Question 1: Do the current metrics reflect semantic correctness as stated in the weakness section?
Evaluating QA correctness using LLM-as-a-Judge approach (GPTScore): To better assess the quality of the generated answers and whether the generated response really addresses the user question, we use the LLM-as-a-Judge approach. We use GPT-6.1 Luna, which is presented with a question, the predicted and reference answers, and tasked to determine a score on a scale of 0 to 10 (higher is better). We use the prompt used in WOMD-Reasoning to evaluate GPTScore. Due to the limitations of computational resources of LLM, we evaluate GPTScore on a subset of 2000 question-answer (QA) pairs, stratified by question type across four categories: road environment, ego agent’s motion status, surrounding agents’ states, and agent interactions and intentions.
The results show that MoLan is able to generate well-grounded responses. Specifically, MoLan achieves the highest GPTScore of 8.343, outperforming Motion-LLaVA (6.503) by 28.3%. Beyond the overall performance, MoLan consistently achieves the best performance across all four categories, demonstrating its versatility in understanding diverse driving scenarios. Most notably, MoLan achieves an 8.128 GPTScore on interactions and intentions, outperforming Motion-LLaVA (4.239).
| Question type | Road environment | Ego agent’s motion status | Other (surrounding) agents’ motion status | Interactions and intentions | Overall |
|---|---|---|---|---|---|
| Non-finetuned VLM w. BEV input | 3.238 | 2.021 | 1.324 | 0.991 | 1.894 |
| Finetuned VLM w. BEV input | 6.329 | 4.122 | 3.239 | 2.471 | 4.040 |
| Motion-LLaVA | 7.724 | 7.813 | 6.237 | 4.239 | 6.503 |
| MoLan (Ours) | 9.128 | 8.129 | 7.287 | 8.128 | 8.343 |
Question 2: According to Tab-4, the M2T- and T2M-only variants already seem to demonstrate strong bidirectional performances without specific bidirectional training. Does this imply that the backbones already have the ability to generalize across modalities?
To better understand the performance differences between the training variants, we expand Table 4 to include the RMM breakdown (kinematic, interaction, and map-based), minADE, %Gain for instruction conditioning, and additional QA metrics: GPTScore, factual accuracy, and median error (m). Compared with T2M-only training, joint supervision improves the interaction score from 0.797 to 0.819 (+2.76%) and increases controllability %Gain from 17.943% to 21.22% (+18.26% relative; +3.28 percentage points).
Additionally, although T2M-only achieves a BERTScore of approximately 0.605, its GPTScore is only 3.981, compared with 8.343 for joint training, and its factual accuracy is 22.3%, compared with 68.1%. We attribute this discrepancy to BERTScore measuring contextual similarity to the reference answer, which can remain high despite incorrect scene details. Conversely, M2T-only performs well on QA but shows substantially weaker motion generation, with a minADE of 2.540 and a negative %Gain of -69.175%.
| RMM ↑ | Kinematic ↑ | Interaction ↑ | Map-based ↑ | minADE ↓ | %Gain ↑ | GPTScore ↑ | Factual QAs: Accuracy ↑ | Factual QAs: Median Error (m) ↓ | |
|---|---|---|---|---|---|---|---|---|---|
| M2T-only | 0.712 | 0.470 | 0.681 | 0.890 | 2.540 | -69.175% | 8.291 | 0.609 | 1.251 |
| T2M-only | 0.775 | 0.487 | 0.797 | 0.911 | 1.232 | +17.943% | 3.981 | 0.223 | 9.125 |
| Joint training | 0.786 | 0.490 | 0.819 | 0.912 | 1.183 | 21.22% | 8.343 | 0.681 | 0.513 |
Joint supervision improves generation RMM from 0.775 to 0.786 (+1.42%) over T2M-only training. Compared with M2T-only training, it increases GPTScore from 8.291 to 8.343 (+0.63%), improves factual accuracy from 60.9% to 68.1% (+7.2 percentage points), and reduces median error from 1.251 m to 0.513 m (−58.99%). These results support the benefit of joint training: supervising both tasks improves motion generation and factual traffic understanding beyond their respective single-task variants.
Clarification 1: Sec-4.2 adopts learned agent-ID embedding for cross-modality feature correspondence. Why is this chosen over a fixed embedding, and does it affect the overall performance?
We use learnable agent-ID embeddings to allow the representations establishing cross-modality feature correspondence to be optimized jointly with the model. To evaluate this choice, we compare learnable and fixed sinusoidal agent-ID embeddings:
| Agent-ID embedding | RMM ↑ | Kinematic ↑ | Interaction ↑ | Map-based ↑ | minADE ↓ | %Gain ↑ | GPTScore ↑ |
|---|---|---|---|---|---|---|---|
| Fixed sinusoidal | 0.777 | 0.489 | 0.802 | 0.909 | 1.207 | 19.58% | 7.892 |
| Learnable | 0.786 | 0.490 | 0.819 | 0.912 | 1.183 | 21.22% | 8.343 |
Learnable embeddings consistently improve both motion generation and traffic understanding: RMM increases from 0.777 to 0.786, minADE decreases from 1.207 to 1.183, and the GPTScore increases from 7.892 to 8.343. These results support our choice of learnable embeddings. Importantly, even with fixed sinusoidal embeddings, MoLan outperforms ProSim, ECoSim, and LangTraj on RMM and minADE, and Motion-LLaVA on GPTScore.
Clarification 2: Does the bridge-layers require re-training when migrating to other backbones in Sec-5.2?
Yes, the bridge layers require training when migrating to other backbones. However, since the motion and language models remain frozen, only the bridge blocks, agent-ID embeddings, and text embeddings for agent tokens are trained. This makes the training process fast and computationally inexpensive.
We thank the reviewer for the positive feedback and for recognizing our simple bridge-layer design. We also appreciate the acknowledgment of our extensive evaluations and ablations. We address the remaining concerns below.
Instruction-following: We agree that RMM and minADE alone do not directly establish whether a rollout follows its instruction. However, we would like to clarify that we do report controllability, which measures how well the model follows the prompt by comparing the model’s realism gain against the unconditional model rollout. Following ProSim and ECoSim, we implement this metric by comparing relative improvement in rollout minADE with and without prompt conditioning. Tab. 2 shows that MoLan achieves the highest %Gain among language-conditioned methods (21.22% vs. 16.39% for ECoSim, both using CAT-k as the motion model).
To further strengthen the controllability evaluation, we additionally introduce Instruction-Success Rate (ISR), which measures whether a referenced agent performs the requested behavior. We first define 8 high-level action types: longitudinal (accelerate, decelerate, maintain speed, stopping, and parked) and lateral (left turn, right turn, and straight). We implement heuristic functions to detect each of the actions in a trajectory. We calculate ISR as the percentage of requested actions that are detected in the generated rollouts for the corresponding agents.
For each initial scene, we evaluate both original and counterfactual instructions. Original instructions describe actions extracted from the logged future. For the counterfactual setting, we randomly select 50% of simulated agents and change their longitudinal and/or lateral actions (e.g., replacing a left turn with a right turn, or accelerating with stopping), then convert the modified actions into text instructions using a prompt template. Because randomly modified instructions are not guaranteed to be feasible, we additionally report collision, off-road, and traffic-light violation rates to assess the safety of the resulting rollouts.
We compute ISR by first averaging action accuracy across agents within each scene, then averaging across all scenes.
The table below shows that MoLan achieves the highest ISR under both original and counterfactual instructions: 88.27% and 66.82%, respectively. Under counterfactual instructions, it exceeds ECoSim by 16.59 percentage points (66.82% versus 50.24%), while also achieving lower collision (3.25% versus 4.24%), off-road (2.24% versus 2.98%), and traffic-light violation rates (4.82% versus 8.03%).