- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Trace4D, a vision-language framework for spatio-temporal reasoning that predicts sequences of 3D object and camera positions + answer questions based on their changes with temporal visual observations. A dataset/benchmark with perception examples and geometric annotations is also constructed particularly for this reasoning task. Experiments show that Trace4D outperforms existing baselines on spatio-temporal reasoning scenarios.
Critical strengths
- The problem is well-motivated.
- The empirical results are strong.
- At inference, only videos are needed as input, without needing external geometric models for spatio-temporal reasoning.
Critical weaknesses
- During data generation, the training states and answers largely come from automated annotation. Currently, the paper provides no validation or audit process/details regarding the accuracy and correctness of those annotations. If those annotations have errors, they can propagate into models through both supervision and evaluation and reduce their reliability.
- The current central novelty is not isolated experimentally. As discussed in the paper, MLLM-4D already uses predicted geometric states, visual analysis, and coordinate rewards for spatiotemporal reasoning. While the current performance gain over it is acknowledged, a more direct and controlled comparison between "start + end state" and "multi-timestep state" would strongly strengthen the core novelty of Trace4D.
- While the current ablation regarding CoT and state changes are acknowledged, corrupting, swapping, or replacing predicted states would provide stronger evidence that those intermediate states are faithfully used by the model and facilitates spatiotemporal reasoning.
Questions for authors
See weakness.
2 comments
W3. Faithfulness of predicted states.
This tests whether the predicted states are used more directly than our ablations could. Three results in the paper are consistent with the states being used. Accuracy falls as the error of the predicted trajectory grows, from 89.90 to 71.72 in the Relative setting and from 91.92 to 65.66 in the Absolute setting (Table 6). Training the model to reason over the states it has written adds 4.59 points over writing the same states and answering directly (Table 3). And moving reward weight from the answer to the states during RL raises answer accuracy. GRPO with an accuracy reward alone (w_acc = 0.9) reaches 78.33 on Trace4D-Bench, while GRPO with geometric rewards (w_acc = 0.5) reaches 79.25, with lower state error (Table 10).
None of these is an intervention at inference, and the interventions the reviewer suggests test the dependence directly. For each question, we take the model's own generated <4d> block, modify it, and let the model regenerate everything that follows, including the visual analysis, the reasoning, and the answer. We will use two of the suggested interventions.
Replacing. We replace the predicted states with the reference states. If the reasoning relies on the states, more accurate states should lead to more accurate answers, and the size of the gain shows how much state errors limit accuracy. VLM4D has no reference states, so the replacement applies to Trace4D-Bench only.
Corrupting. We replace the predicted coordinates with random values drawn within the range observed in the data, keeping the format and the number of timesteps unchanged. The corrupted states carry no information about the video. If the model ignored the states and answered from the video, its answers would not change.
We could not run the interventions during the rebuttal period. We will report accuracy and the fraction of answers that change under each condition in Section 4.4 of the final version.
Response to Reviewer Z16j
We thank the reviewer for the positive assessment and the concrete suggestions.
Note on new experiments. Our server became inaccessible during the rebuttal period because of a technical failure. Our data is stored on it as well, so we could not run the requested experiments or the audit, and we apologize for that. This response contains no new experimental results, and we answer from the evidence already in the paper and from prior work. We will run the experiments as soon as the server is restored and report them in the final version. Text added to the revised PDF in response to this review is marked in purple.
W1. Annotation validation.
This concern is shared by Reviewer 3mJc, and we will address it with a human audit. The pipeline already limits annotation errors in several ways, listed in full in our response to Reviewer 3mJc. Tracks are filtered by box size and by a multimodal verifier that rejects empty, background, or mislabeled regions. The answer rules use margins (a 20% change for distance and speed, a 0.3 component for direction, Table 9), so a small centroid error changes a label only near a threshold. For every training QA, a multimodal teacher sees the clip together with the reference states and is instructed to state a mismatch instead of forcing agreement (Fig. 6).
For the audit, annotators check 120 Trace4D-Bench questions (10 per type, sampled at random) and 120 training questions sampled the same way. For each question they watch the clip, judge whether the pipeline label is correct, select the correct option when it is not, and mark clips with occlusion or substantial camera motion. Since each label is derived from the reference trajectory, a trajectory error large enough to change the answer appears as a label judged incorrect.
The training-QA agreement rate bears on the supervision, through which, as the reviewer notes, errors could propagate. On that side, one result already does not depend on our labels. On VLM4D, whose labels are human-verified and independent of our pipeline, Trace4D-4B improves by 8.92 points over its base model (Table 2). For the evaluation, we will re-score the models on the benchmark questions where humans and the pipeline agree. The audit will be summarized in Section 4.2 and reported in full in Appendix D.
W2. Start+end vs. multi-timestep states.
This comparison would isolate the core design choice. MLLM-4D predicts states at the start and end frames as anchors and argues the answer from a visual description of the progression between them, and its reward scores the coordinates at those two anchors. Trace4D predicts a state at every annotated timestep, uses the numeric change over each interval as evidence alongside the matching visual observation (Fig. 1), and rewards the displacement between consecutive timesteps as well as positions.
The retrained start+end variant needs the teacher CoT to be regenerated, because the current traces cite intermediate states, and we will add it in the final version. Until then, three points bear on the question.
The first holds by construction. A change in speed is defined over consecutive displacements (Table 9) and therefore needs at least three states, so the 200 speed questions in Trace4D-Bench cannot be answered from start and end states. We have added this point to Section 2 of the revised PDF, where Trace4D is contrasted with MLLM-4D (marked in purple). Table 1 is in line with it. MLLM-4D, which writes start and end states, is near chance on the two speed types (35.0 and 34.0), where Trace4D reaches 79.0 and 73.0, although MLLM-4D was not trained on our question templates.
The second concerns the questions. Many Trace4D-Bench answers describe a change in two ordered phases, which two states cannot tell apart. In Fig. 1 the boxer–referee distance goes 0.142 → 0.024 → 0.183. The first and last states show only an overall increase, which fits several options, including the correct answer B ("closer, then farther") and option C ("farther, then about the same"). The distance examples in Fig. 9 (top) and Fig. 10 (bottom) have the same structure.
The third concerns the reward. Rewarding inter-timestep displacement in addition to positions improves over rewarding positions alone, with the total geometric weight held fixed, from 78.83 to 79.25 on Trace4D-Bench and from 63.11 to 64.32 on VLM4D, with lower centroid and trajectory error (Table 10). This reward is computed over each interval between consecutive states, which a start+end variant would reduce to a single interval, so the interval-wise signal requires a state sequence.