acceptodds
Under review as a conference paper at ICLR 2027

Trace4D: Grounded Spatiotemporal Reasoning via 4D State Sequences

Abstract

Understanding how objects and their spatial relationships evolve in 3D space over time remains challenging for vision-language models (VLMs), particularly under changing viewpoints. Recent approaches incorporate geometric information into reasoning, but explicit supervision for temporally ordered 4D state evolution and its use as intermediate evidence remains limited. We introduce Trace4D, which trains VLMs to predict 4D state sequences and reason over their inter-timestep changes alongside temporal visual observations. Its 4D-grounded chain-of-thought (CoT) explicitly represents 4D states, analyzes temporal visual context, and connects both as evidence for the answer. We construct Trace4D-Train from in-the-wild videos with 600K 4D perception examples and approximately 70K grounded QA examples, and train with joint supervised fine-tuning followed by geometry-aware GRPO. At inference, Trace4D predicts 4D states and reasoning directly from video without reference annotations or an external geometric model. Experiments on Trace4D-Bench and VLM4D show improved QA accuracy over the base VLMs and benefits from reasoning over 4D state changes and temporal visual observations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Model, 4D Understanding, Multimodal Large Language Model, Chain-of-Thoughts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z16jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Trace4D, a vision-language framework for spatio-temporal reasoning that predicts sequences of 3D object and camera positions + answer questions based on their changes with temporal visual observations. A dataset/benchmark with perception examples and geometric annotations is also constructed particularly for this reasoning task. Experiments show that Trace4D outperforms existing baselines on spatio-temporal reasoning scenarios.

Critical strengths
  • The problem is well-motivated.
  • The empirical results are strong.
  • At inference, only videos are needed as input, without needing external geometric models for spatio-temporal reasoning.
Critical weaknesses
  • During data generation, the training states and answers largely come from automated annotation. Currently, the paper provides no validation or audit process/details regarding the accuracy and correctness of those annotations. If those annotations have errors, they can propagate into models through both supervision and evaluation and reduce their reliability.
  • The current central novelty is not isolated experimentally. As discussed in the paper, MLLM-4D already uses predicted geometric states, visual analysis, and coordinate rewards for spatiotemporal reasoning. While the current performance gain over it is acknowledged, a more direct and controlled comparison between "start + end state" and "multi-timestep state" would strongly strengthen the core novelty of Trace4D.
  • While the current ablation regarding CoT and state changes are acknowledged, corrupting, swapping, or replacing predicted states would provide stronger evidence that those intermediate states are faithfully used by the model and facilitates spatiotemporal reasoning.
Questions for authors

See weakness.

2 comments

Response to Reviewer Z16j

We thank the reviewer for the positive assessment and the concrete suggestions.

Note on new experiments. Our server became inaccessible during the rebuttal period because of a technical failure. Our data is stored on it as well, so we could not run the requested experiments or the audit, and we apologize for that. This response contains no new experimental results, and we answer from the evidence already in the paper and from prior work. We will run the experiments as soon as the server is restored and report them in the final version. Text added to the revised PDF in response to this review is marked in purple.

W1. Annotation validation.

This concern is shared by Reviewer 3mJc, and we will address it with a human audit. The pipeline already limits annotation errors in several ways, listed in full in our response to Reviewer 3mJc. Tracks are filtered by box size and by a multimodal verifier that rejects empty, background, or mislabeled regions. The answer rules use margins (a 20% change for distance and speed, a 0.3 component for direction, Table 9), so a small centroid error changes a label only near a threshold. For every training QA, a multimodal teacher sees the clip together with the reference states and is instructed to state a mismatch instead of forcing agreement (Fig. 6).

For the audit, annotators check 120 Trace4D-Bench questions (10 per type, sampled at random) and 120 training questions sampled the same way. For each question they watch the clip, judge whether the pipeline label is correct, select the correct option when it is not, and mark clips with occlusion or substantial camera motion. Since each label is derived from the reference trajectory, a trajectory error large enough to change the answer appears as a label judged incorrect.

The training-QA agreement rate bears on the supervision, through which, as the reviewer notes, errors could propagate. On that side, one result already does not depend on our labels. On VLM4D, whose labels are human-verified and independent of our pipeline, Trace4D-4B improves by 8.92 points over its base model (Table 2). For the evaluation, we will re-score the models on the benchmark questions where humans and the pipeline agree. The audit will be summarized in Section 4.2 and reported in full in Appendix D.

W2. Start+end vs. multi-timestep states.

This comparison would isolate the core design choice. MLLM-4D predicts states at the start and end frames as anchors and argues the answer from a visual description of the progression between them, and its reward scores the coordinates at those two anchors. Trace4D predicts a state at every annotated timestep, uses the numeric change over each interval as evidence alongside the matching visual observation (Fig. 1), and rewards the displacement between consecutive timesteps as well as positions.

The retrained start+end variant needs the teacher CoT to be regenerated, because the current traces cite intermediate states, and we will add it in the final version. Until then, three points bear on the question.

The first holds by construction. A change in speed is defined over consecutive displacements (Table 9) and therefore needs at least three states, so the 200 speed questions in Trace4D-Bench cannot be answered from start and end states. We have added this point to Section 2 of the revised PDF, where Trace4D is contrasted with MLLM-4D (marked in purple). Table 1 is in line with it. MLLM-4D, which writes start and end states, is near chance on the two speed types (35.0 and 34.0), where Trace4D reaches 79.0 and 73.0, although MLLM-4D was not trained on our question templates.

The second concerns the questions. Many Trace4D-Bench answers describe a change in two ordered phases, which two states cannot tell apart. In Fig. 1 the boxer–referee distance goes 0.142 → 0.024 → 0.183. The first and last states show only an overall increase, which fits several options, including the correct answer B ("closer, then farther") and option C ("farther, then about the same"). The distance examples in Fig. 9 (top) and Fig. 10 (bottom) have the same structure.

The third concerns the reward. Rewarding inter-timestep displacement in addition to positions improves over rewarding positions alone, with the total geometric weight held fixed, from 78.83 to 79.25 on Trace4D-Bench and from 63.11 to 64.32 on VLM4D, with lower centroid and trajectory error (Table 10). This reward is computed over each interval between consecutive states, which a start+end variant would reduce to a single interval, so the interval-wise signal requires a state sequence.

W3. Faithfulness of predicted states.

This tests whether the predicted states are used more directly than our ablations could. Three results in the paper are consistent with the states being used. Accuracy falls as the error of the predicted trajectory grows, from 89.90 to 71.72 in the Relative setting and from 91.92 to 65.66 in the Absolute setting (Table 6). Training the model to reason over the states it has written adds 4.59 points over writing the same states and answering directly (Table 3). And moving reward weight from the answer to the states during RL raises answer accuracy. GRPO with an accuracy reward alone (w_acc = 0.9) reaches 78.33 on Trace4D-Bench, while GRPO with geometric rewards (w_acc = 0.5) reaches 79.25, with lower state error (Table 10).

None of these is an intervention at inference, and the interventions the reviewer suggests test the dependence directly. For each question, we take the model's own generated <4d> block, modify it, and let the model regenerate everything that follows, including the visual analysis, the reasoning, and the answer. We will use two of the suggested interventions.

Replacing. We replace the predicted states with the reference states. If the reasoning relies on the states, more accurate states should lead to more accurate answers, and the size of the gain shows how much state errors limit accuracy. VLM4D has no reference states, so the replacement applies to Trace4D-Bench only.

Corrupting. We replace the predicted coordinates with random values drawn within the range observed in the data, keeping the format and the number of timesteps unchanged. The corrupted states carry no information about the video. If the model ignored the states and answered from the video, its answers would not change.

We could not run the interventions during the rebuttal period. We will report accuracy and the fraction of answers that change under each condition in Section 4.4 of the final version.

Reviewer_3mJcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Trace4D, a framework for video-based spatiotemporal reasoning through explicit, temporally ordered 4D state sequences. The model predicts object bounding boxes, 3D centroids, and camera positions when required, then combines changes in these states with temporal visual observations to derive an answer. The authors construct approximately 600K perception examples and 70K grounded question-answer examples using pretrained perception models, rule-based question generation, and teacher-generated reasoning traces. Training combines joint supervised fine-tuning with geometry-aware GRPO, which rewards answer correctness, position accuracy, inter-timestep displacements, and bounding-box alignment. At inference, the VLM generates states and reasoning directly from video without an external geometry model. Experiments on the held-out Trace4D-Bench and external VLM4D benchmark report improvements over the base VLMs, while ablations examine the contributions of state-based reasoning, visual evidence, and geometric rewards.

Critical strengths
  1. Explicit and inspectable supervision for temporal geometry. Trace4D represents object and camera states across multiple timesteps and trains the model to connect their changes with visual observations. Position and displacement rewards provide concrete learning signals for both spatial accuracy and temporal evolution. This is consequential because it makes intermediate predictions directly evaluable, providing a clearer connection between perception and reasoning than final-answer accuracy alone.

  2. Supporting evidence from controlled ablations and external evaluation. The paper includes controlled comparisons of reasoning formats and GRPO rewards, alongside evaluation on an external benchmark. On VLM4D, the 4B model improves from 55.40 to 64.32, while geometry-aware GRPO outperforms accuracy-only GRPO by 1.65 points. The CoT ablations also show complementary benefits from combining state-based reasoning with visual observations. These results support the proposed components and provide evidence of benefits beyond the self-constructed benchmark.

Critical weaknesses
  1. Limited independent validation of the geometric references. The reference 3D states, training answers, and Trace4D-Bench labels are derived from the same automated perception and rule-based construction pipeline. Although tracking and semantic filters are described, the paper provides limited independent validation of the resulting 3D trajectories and geometry-derived answers. In particular, centroids obtained by averaging reconstructed points within visible masks may vary with occlusion and viewpoint changes. Consequently, agreement with these references may partly reflect consistency with the annotation pipeline rather than accurate physical motion understanding. An independently verified subset would strengthen confidence in both the supervision and evaluation.

  2. The large in-domain gains do not translate into consistent improvements on real videos. Trace4D-Bench holds out source videos but shares the training set’s question formulations and answer-generation rules. External VLM4D evaluation is therefore particularly important, yet its improvements are concentrated in synthetic videos. For the 4B model, synthetic accuracy rises from 48.76 to 77.30, whereas real-video accuracy increases only from 57.55 to 60.10. For the 8B model, real-video accuracy decreases from 60.25 to 59.08. These results limit the evidence that the method yields broadly transferable spatiotemporal understanding, rather than primarily improving performance on the constructed task distribution.

Questions for authors
  1. Could the authors clarify how the reference 3D trajectories and geometry-derived answer labels are validated independently of the automated construction pipeline? An independent audit of trajectory trends and derived answers on a representative subset, particularly under occlusion and viewpoint changes, would help quantify annotation errors and determine whether they affect the reported evaluation.

  2. Could the authors provide question-type breakdowns and uncertainty estimates for the improvements on VLM4D’s real-video subset, and explain the 8B model’s decrease from 60.25 to 59.08? This analysis would help establish which capabilities transfer to real videos and whether the observed gains and regressions are consistent or within evaluation variability.

2 comments

Response to Reviewer 3mJc

We thank the reviewer for the careful reading and the specific suggestions.

Note on new experiments. Our server became inaccessible during the rebuttal period because of a technical failure. The benchmark clips and saved model predictions are stored on it, so we could not carry out the audit or the breakdown requested here, and we apologize for that. The only new numbers in this response are confidence intervals computed from the accuracies already reported in the paper, and a count of question types in VLM4D's public question file. We will run both analyses as soon as the server is restored and report them in the final version. Text added to the revised PDF in response to this review is marked in blue.

W1/Q1. Independent validation of geometric references.

We agree that an independently verified subset is needed, and we will conduct the audit the reviewer describes as soon as the server is restored. Below we first summarize what the pipeline already does to limit annotation errors (Sec. 3.1, Sec. 3.2, App. A.2, App. A.3).

  • Source videos are selected from Koala-36M using its motion and training-suitability metadata, so that clips have observable dynamics, and clips that cannot be read at the required frame sampling are excluded.
  • A track is kept only if its box covers at least 1% of the image on average and never more than 50%. A multimodal verifier then rejects tracks whose region is empty, is background, or does not match its label.
  • A temporal window is kept only if every required target has a valid box and a valid 3D centroid. For two-target questions the two initial boxes must have IoU of at most 0.5.
  • The answer rules use margins. A distance or speed must change by more than 20% to count as increasing or decreasing, a direction component must exceed 0.3, and a path must exceed 0.04 to count as motion (Table 9). A small centroid error therefore changes a label only when the true change is near a threshold.
  • Duplicate and equivalent options are removed, and for motion questions any distractor that is also consistent with the reference trajectory is replaced, so that exactly one option is correct.
  • For every training QA, a multimodal teacher sees the clip together with the reference states and is instructed to state a mismatch between the two instead of forcing agreement, so the visual analysis in a trace is not invented to fit the states (Sec. 3.2, Fig. 6).

For the audit, annotators check 120 Trace4D-Bench questions (10 per type, sampled at random) and 120 training questions sampled the same way. For each question they watch the clip, judge whether the pipeline label is correct, select the correct option when it is not, and mark clips with occlusion or substantial camera motion. Since each label is derived from the reference trajectory, a trajectory error large enough to change the answer appears as a label judged incorrect. We will report agreement overall and separately for clips with and without occlusion and camera motion. This directly measures the concern the reviewer raises about mask-averaged centroids under occlusion and viewpoint change.

To check whether these errors affect the reported evaluation, we will re-score the models on the benchmark questions where humans and the pipeline agree. On VLM4D, whose labels are independent of our pipeline, Trace4D also improves over its base model (Table 2, see W2). We will include the audit in Appendix D.

W2/Q2. Real-video transfer.

Trace4D-Bench shares its question formulations with the training data, which makes VLM4D the more important test here. The overall real-video change is modest.

Uncertainty. Paired intervals need the per-question predictions stored on that server. What we can compute now, from the reported accuracies and question counts alone, are unpaired 95% intervals for each difference from the base model.

Subset (questions)ModelBaseTrace4DΔUnpaired 95% interval
Real (1,371)4B57.5560.10+2.55[−1.1, +6.2]
Real (1,371)8B60.2559.08−1.17[−4.8, +2.5]
Synthetic (445)4B48.7677.30+28.54[+22.5, +34.6]
Synthetic (445)8B55.7368.09+12.36[+6.0, +18.7]
All (1,816)4B55.4064.32+8.92[+5.7, +12.1]
All (1,816)8B59.1461.29+2.15[−1.0, +5.3]

These intervals treat the two models as independent samples, which overstates the variance of a comparison on the same questions, so they are conservative. The overall 4B gain and both synthetic gains exclude zero, and the real-video intervals include zero for both models. In the final version we will replace them with 95% paired bootstrap intervals, resampling videos rather than individual questions, since questions from the same video are not independent.

On the reviewer's broader question, the pattern follows what the predicted states describe. Trace4D's states are 2D boxes and 3D centroids, which directly describe where objects are and how they translate, and carry little information about rotation, counts, or actions. To check this against the benchmark itself, we classified the 1,371 real-video questions in VLM4D's public question file by what they ask about, using keyword rules on the question and its options.

Question asks aboutReal-video questions (1,371)
Translation of a named object, as seen by the camera569 (41.5%)
Other motion, turning, or spatial relation205 (15.0%)
Counting254 (18.5%)
Rotation141 (10.3%)
Motion from the object's own perspective90 (6.6%)
Hands, facing direction, actions, other112 (8.2%)

The first row is the kind of question our states describe, and it makes up 41.5% of the real subset. The last four rows, 597 questions or 43.5%, ask about something else, such as how many scooters move to the left, whether a bus spins clockwise or counter-clockwise, or whether a bear moves forwards or backwards from its own perspective. For these questions the states provide little evidence, and the model has to rely on its visual analysis. Section 4.3 of the revised PDF now notes that the real videos include such questions (marked in blue). The classification is by keyword and is approximate. The accuracy breakdown by question type, which we will add in the final version, will test this directly.

Trace4D-4B reaches 60.10 on the real subset, above its base model (57.55) and above every other 3D/4D-specialized model in Table 2 except MLLM-4D-8B (63.09), a model twice its size.

On the 8B result, the real-video change of −1.17 points has a confidence interval of [−4.8, +2.5], which includes zero, so it reflects evaluation variability rather than a consistent regression. What the 8B model does show is a smaller benefit from the same training than the 4B model, on Trace4D-Bench as well as on VLM4D. Section 4.3 suggests a capacity–data mismatch as one explanation, in which the same amount of supervision does not fully exploit the larger model.

Reviewer_ckBtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to address 4D vision-language spatial understanding tasks. The main ideas are: a) Generate a large training corpus (with CoT annotations) synthetically on in-the-wild videos b) Train a VLM model with SFT and GRPO on those annotations. The paper finds that training and evaluating their model in such a way helps improve its spatial reasoning capabilities on their introduced validation set as well as existing VLM-4D dataset

Critical strengths

Since the prompt only asks for critical strengths, I think their generated dataset could be potentially useful to the community, and the results achieved with their method on the VLM-4D dataset are strong.

Critical weaknesses

One of my main concerns is Novelty / What new we can learn from this paper: One of the main contributions of this paper is the Trace4D dataset. However, that dataset is very closely related to DSR-Suite (which the paper also mentions) and I am not very clear on what the difference is between the two. My understanding is that it is a more scaled-up version of DSR-Suite and also includes an LLM generated CoT annotations. Scaling up the data is not something new, and generating CoT annotations is very common occurence in the literature as well. Any help in clarifying the novelty of this work by the authors will be very helpful [and ideally should be included in the related work section clearly].

Next, I have some concerns on the experimental side.

  • Table 1: It shows that the Qwen3-VL model fine-tuned on the proposed Trace-4D dataset is significantly better than the base VLM. A known issue however, with DSR-Suite, is that even fine-tuning a qwen model in a blind setup (i.e. no visual input) on the training set gives competitive performance to the full model with visual inputs. Since this paper follows DSR_Suites data generation style, I think it is important to report blind baseline's performance to have more confidence on Trace-4D's validation benchmark's results

  • In table-1/table-2 or ablations, i think it is important to include a baseline without any CoT but just simply trained on the final answer. There are several ablations on CoT design choices and SFT/GRPO training but no ablation which shows results on the direct answer baseline.

  • In Table-3, we see that Visual-CoT only can already get to 59.25% accuracy, and additional 4D CoT adds less than 1% on VLM-4D benchmark. In my understanding, this means that 4D based CoT is not helping much, which brings into question part of the premise of this work that 4D CoT is helpful. Note that while I see it helping in Trace4D-Bench, I am currently leaning towards trusting the VLM-4D benchmark more as it has some real videos too (and we are missing some blind baselines on Trace4D-Bench).

Questions for authors
  1. Could the authors help clarify the novelty of their work compared to DSR_Suite and related methods? What new could a reader take away from this work?
  2. Blind baselines and no CoT baselines would help a lot
  3. Clarification whether indeed 4D CoT helps or just VisualCoT is enough; and its implications on takeaway messages from this paper
2 comments

Response to Reviewer ckBt

We thank the reviewer for the clear and specific questions.

Note on new experiments. Our server became inaccessible during the rebuttal period because of a technical failure, so we could not run the requested baselines, and we apologize for that. This response contains no new experimental results, and we answer from the evidence already in the paper and from prior work. We will run the experiments as soon as the server is restored and report them in the final version. Text added to the revised PDF in response to this review is marked in red.

Q1. Novelty relative to DSR-Suite, and what a reader can take away.

We agree the distinction was not clear and have updated the manuscript accordingly in Section 2 (marked in red). Our data pipeline reuses DSR-Suite's perception components and its answer-derivation rules (Sec. 3.1, Table 9). We do not claim any of this as a contribution.

The difference lies in what is supervised and what the model outputs. DSR-Train consists of multiple-choice question-answer pairs, and its model answers directly, with geometric features from the Pi3 encoder injected into the VLM at inference through the Geometry Selection Module. What we claim in Trace4D-Train is the supervision for grounded 4D reasoning built on the QA pairs. This consists of a 4D-grounded reasoning trace for each QA example, tied to the reference state sequence from which the answer was derived, and 600K 4D perception examples that supervise state-sequence prediction by itself. As a result the model writes out its geometric states and uses no geometric encoder at inference.

On the reviewer's point that generating CoT annotations is common, we agree, and we do not claim CoT generation itself. In grounded reasoning, methods are distinguished by what the reasoning is grounded in and how the trace is structured. LLaVA-CoT separates visual interpretation from reasoning, VisCoT and VoCoT ground the steps in image regions and objects, and VideoEspresso and Open-o3-Video ground them in frames, objects, and timestamps. Our 4D-grounded CoT grounds them in changes of a 4D state sequence and in the matching visual observations (Sec. 3.2). Each trace consists of the 4D state sequence, a separate temporal visual analysis, and 4D-grounded reasoning that connects each visual observation to the state change for the same target and interval and uses both as evidence for the answer. Table 3 ablates these parts, and we discuss what it shows below.

On what a reader can take away, we would point to two findings.

First, explicit geometric states and visual observations contribute in different ways. Table 3 compares four response formats trained on the same data. Reasoning over the state sequence is what drives accuracy on Trace4D-Bench, whose questions ask how a distance, direction, or speed changes over the clip, while the separate visual block is what drives accuracy on VLM4D, where adding it to state-based reasoning raises accuracy substantially. Combining the two gives the best VLM4D accuracy while keeping most of the in-domain gain, and learning the states still adds to transfer, since without the 4D perception supervision the same QA training is 2.98 points lower on VLM4D (Table 4). For work that adds explicit geometry to a VLM's reasoning, this supports recording the visual evidence separately and aligning it with the geometry interval by interval, as Trace4D does.

Second, the change between predicted states is a useful unit for both reasoning and training. Prior work writes out 3D locations at a single frame or at two anchor frames. In Trace4D, the answer is derived from how the states change over the clip, which identifies answers such as "closer, then farther" that two endpoints alone cannot (Fig. 1). Reasoning over these changes improves over writing the states and answering directly (Table 3), and rewarding the change between consecutive states during RL improves both accuracy and trajectory error beyond rewarding positions alone, with the total weight on geometric rewards held fixed (Table 10).

The conclusion of the revised PDF now states these two findings.

Q2. Blind and no-CoT baselines.

Both comparisons are useful. We could not run them during the rebuttal period and will add them to Section 4 in the final version.

Blind baseline. One analysis in the paper already spoke to this concern. Within each question type, accuracy decreases as the error of the trajectory the model predicts from the video increases (Table 6), which would not be expected if the model relied mainly on patterns in the question text. To test this directly, we will train a blind model from the base VLM on the same 70K QA data, using the question and options without video, as the reviewer describes for DSR-Suite.

A text bias of this kind may exist, as prior work on templated spatial QA has found [1]. Two points hold regardless of its size. First, the gains of our components are measured between models trained on the same questions, options, and answers, so a text prior is common to both sides and cannot produce the difference. These gains are +4.59 points from reasoning over state changes and +3.50 from adding the states to visual reasoning (Table 3), and +5.33 from perception supervision (Adaptation-only against Perception-first, averaged over the two splits in Table 4), whose 600K examples contain no questions or answers at all. Second, the gains appear on VLM4D, which shares no questions or templates with our training data, where Trace4D-4B improves by 8.92 points over its base model (Table 2).

No-CoT baseline. Our original ablations did not include it because the paper studies how a VLM's reasoning can be grounded in 4D evidence. In Trace4D, each answer comes with the states the model relied on and the visual observations it matched to them, so the reasoning can be traced and checked against references. A direct-answer model provides no such evidence, so our ablations compared different grounded formats with each other, the closest being State-only in Table 3, which writes the states and then answers directly. It reaches 74.83 on Trace4D-Bench and 52.86 on VLM4D, against 77.25 and 60.52 for 4D + Visual CoT trained on the same data. We agree that a direct-answer model is still a useful reference, and we will train one in the Table 3 setting, from the same perception-pretrained 4B checkpoint, with the video as input. As an external reference, DSR-Suite reports a limited gain on VLM4D from fine-tuning on its multiple-choice QA, under a different base model and protocol, while in our setting grounded-CoT training gains 5.12 points on VLM4D over the base model (Table 3).

Q3. Does 4D CoT help, or is Visual CoT enough?

Writing the 4D states into the reasoning improves accuracy on both benchmarks in Table 3, and it also allows geometric rewards during RL, which a Visual-only model cannot receive. The point of the paper, stated as the first finding in Q1, is that the states and the visual observations work together rather than one replacing the other.

Table 3 shows this from both directions. Adding the states to visual reasoning improves accuracy on both benchmarks, by 3.50 points on Trace4D-Bench and by a smaller 1.27 points on VLM4D, and adding the visual block to state-based reasoning improves VLM4D by 5.84 points. The combination therefore gives the best VLM4D accuracy while keeping most of the in-domain gain. All four variants, including Visual-only CoT, are trained with the same 4D perception examples, so the comparison isolates only whether the states are written into the reasoning.

Beyond supervised training, writing the states out also lets them be trained directly during RL. Geometric rewards on the written states raise VLM4D by 1.65 points beyond RL with an accuracy reward alone (Table 10), a signal that a Visual-only model cannot receive because it has no states to reward. MLLM-4D reports the same effect independently, with a coordinate reward on its written states improving VLM4D accuracy over GRPO without it.

We agree that VLM4D is the more informative test for transfer, since its labels are independent of our pipeline. For Trace4D-Bench, the blind baseline (Q2) and the human audit described in our response to Reviewer 3mJc address whether it can be trusted, and we will add both in the final version.

[1] Brown et al., Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts. arXiv 2511.04655, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.