- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work focuses on improving the spatiotemporal relational reasoning capabilities of LLMs, particularly for tracking dynamic object relationships, temporal events, and incomplete or ambiguous observations. The authors construct a programmatically generated benchmark with verifiable answers, and propose a teacher-student pipeline to generate large-scale training data, followed by SFT and RL to improve a smaller model's reasoning capabilities. The results show that the qwen3.5 0.8b model achieves 45% accuracy after training, compared with 38% after SFT alone, demonstrating the potential of this approach for improving complex relational reasoning.
Critical strengths
-
The proposed strategy is meaningful in practice. For tasks operating in real physical environments, the execution trajectories naturally involve various spatial memory and reasoning requirements, which are essential for developing AI systems that can operate reliably in the physical world. However, I think a more promising direction for this work may actually be laboratory environments rather than household scenarios. Many laboratory tasks involve potential safety risks and require small on-device models to make quick decisions, while the long reasoning time of large models could cause serious delays. In this regard, achieving such noticeable improvements with a relatively small 0.8B model is quite impressive (though still not enough).
-
Another interesting aspect of this work is the framework for programmatically generating verifiable datasets, which could theoretically produce sufficiently diverse training data at scale.
Critical weaknesses
If I have not missed anything, I think there are three main limitations:
-
The current ablation study is somewhat limited. For example, are both polishers really necessary, and how exactly do they contribute to the accuracy? It would be helpful to compare the performance using the original data directly, data generated by the teacher model without polishing, and data further refined by the polishers.
-
The generalization ability has not been sufficiently validated. The current training and testing seem to be conducted within similar types of environments, which may not be enough to demonstrate strong generalization. Future experiments could consider training in household environments and testing in laboratory environments, or introducing unseen systems, objects, and object states during testing.
-
The current accuracy is still not sufficient for real-world deployment, especially in scenarios where reliability is critical. Perhaps the authors could also explore slightly larger models in future work, since the inherent capability limitations of a 0.8B model may already be a major bottleneck.
Questions for authors
-
How do the temporal length and spatial complexity of events affect the model's performance? The paper covers 90 types of query algorithms, but it seems that the authors do not report the length of event sequences or the complexity of spatial dependencies. I think these statistics would be very helpful for understanding the model's performance and limitations. For example, in real-world applications, models may need to handle much longer event sequences and more complex spatial relationships. If the performance drops significantly under such conditions, it could become a major limitation for future practical applications.
-
If feasible, could the authors provide some additional experiments to evaluate the model's generalization ability? For example, as mentioned above, testing on environments that are fundamentally different in type from those in the training set, or using event sequences involving interactions with previously unseen objects. Such experiments would help better demonstrate how well the model generalizes beyond the training scenarios.
6 comments
To W2: Learning on unseen scenes and event histories
Training raises complete-answer accuracy from 1.39% for the original model to 29.91% after SFT and 31.76% after RL on 1,080 questions about 100 residential scenes absent from training and development. These gains demonstrate learned reasoning on new scenes and event histories, using the same public observations and queries for all three models.
To W3: Larger-model comparisons and batch inference cost
The trained 0.8B solver achieves 31.76% accuracy, exceeding original 4B/9B direct answering at 20.37%/27.78% on the same independent questions. Both polishers construct supervision offline; only the solver runs at inference.
Compared with single-pass 4B executable querying, the trained solver uses 97.5% fewer generated tokens and 98.2% less batch processing time. The trained 0.8B solver and the single-pass 4B query model both enable Thinking with the same 16,384-token budget per generation. The query model returns one program for execution; the trained solver returns the structured answer directly.
| System | Complete-answer accuracy | Batch time (seconds) | Generated tokens, including Thinking (millions) |
|---|---|---|---|
| Original 4B, single-pass executable query | 30.46% | 9,285 | 14.02 |
| Trained 0.8B, RL checkpoint answering directly | 31.76% | 171 | 0.35 |
Batch times cover generation, query execution where applicable, scoring, and output recording for the same 1,080 questions on an A800, after model loading, input preparation, and request admission.
To W1: How the two polishers affect solver accuracy and supervision quality
Polishing only the rationale helped the solver correctly answer 81 and 83 more of the 900 answerable questions than training on the original text in the two 1,200-example studies. The two polishers learn from teacher demonstrations to express executable queries as questions and explain program derivations as rationales. Within each study we kept the source examples and program-computed answers fixed. We trained each solver from the same Qwen3.5-0.8B weights for 500 SFT updates and evaluated it on the same 1,080 questions (Section 5.7 of the revised paper).
Adding question polishing to the polished rationales reduced false refusals on answerable questions from 212 to 92 in set 1 and from 64 to 19 in set 2. Rationale-only gave the most complete answers on answerable questions in both sets, and Raw scored higher than Dual overall.
We also trained each solver on the same 356 examples for which the teacher produced both texts. Dual yielded 161 correct answers out of 1,080, 21 more than question-only (140) and 61 more than rationale-only (100). The counts were 125 with Raw, 111 with teacher-direct text and 53 with text from a prompted original 4B model. Relative to Raw, Dual correctly answered 97 more insufficient-evidence questions (153 vs. 56 of 180) and 61 fewer answerable questions (8 vs. 69 of 900), for a net gain of 36.
Does polishing preserve the question and its derivation? From the 356 sources we selected two from each of the 15 query-family and evidence-state combinations before reading the texts. We first reviewed Raw, Dual and teacher-direct, then reviewed the three additional conditions on the same 30 sources. Each comparison used two review rounds with method labels hidden. A pair passed when the question preserved the executable query's meaning, the rationale followed from the public evidence, and the two agreed.
Both single-polisher conditions passed on all six sources where prompted 4B passed. In one example, the question polisher kept the restriction to chairs originally in Kitchen 1, which selects only Chair 4. Prompted 4B asked about both Chair 4 and Chair 3, then gave a rationale for Chair 4 alone.
On the 19 sources where Raw, Dual and teacher-direct all passed, Dual raised the number of questions rated natural from 10 to 16 and clear from 16 to 19 relative to Raw. Teacher-direct scored 16 and 18, respectively (Table 12 of the revised paper).