- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents STaR-VL, a staged post-training framework for visual-temporal memory. It trains three LoRA adapters on a frozen Qwen2.5-VL backbone. The stages cover perception, symbolic temporal memory, and visual adaptation. A recurrent module compresses observation histories into memory tokens. Each stage combines supervised fine-tuning and reinforcement learning. Experiments cover SpaMEM and four external benchmarks. The main improvements appear in object recall, event detection, and cumulative memory.
Critical strengths
1.It compares per-frame perception, visual-history reasoning, and symbolic-history reasoning. These experiments reveal a gap between object recognition and the effective use of past observations. Additional controls identify history coverage and accessible scene state as important factors in visual-temporal performance.
2.Experiments on the same backbone cover joint training, persistent-memory augmentation, and structured prompting. The results demonstrate the framework’s advantages over these alternative approaches to improving visual-temporal memory.
Critical weaknesses
1.The paper reports a 6.8-point gain over joint LoRA in Table 3. The authors do not establish matched memory architectures, optimization objectives, or total training budgets for this comparison.
2.The paper does not verify temporal answer correctness through its stated QA-dict reward. The paper defines the dictionary reward as key-set F1 in Section 3. The authors already supply all eight required keys in the QA-dict prompt. The paper therefore permits full task reward for answers with correct keys and incorrect counts, trajectories, or timestamps under the stated definition.
Questions for authors
1.The authors should compare staged and joint training with the same recurrent memory module, comparable adapter capacity, matched objectives, and matched total training budgets. The authors should train each configuration independently.
2.The authors should specify the implemented QA-dict reward. The authors should report the reward assigned to an answer with correct keys and incorrect field values, and clarify any discrepancy with the stated definition. And authors should compare key-set scoring with scoring that also checks field values.
3.The authors should keep the training settings fixed and report the corresponding temporal localization, tracking, and cumulative memory scores.
Second round reply
I thank the authors for the reply, as it has clarified some of my concerns. I address the items clarified and the items open After the revision, the results in the paper appear far more useful for understanding and validating the claims of StaR-VL. I am willing to raise my score if the authors provide sufficient empirical evidence regarding the full dataset distribution, model sizes and relevant results for the M-score
I must remind the authors to update the PDF. The discussion forum is informative, but what gets incorporated into the manuscript matters.
Comparison with the full dataset Since the authors train with a small portion of the dataset the alignment of this subset to the full data must be understood. I will request the author to provide the following statistics for the whole set so we can compare the distribution of the annotations used to calculate M-score and V-score
Action distribution ( percentage of spawn, place, remove, etc.) object category distribution (Apple, Pen, Egg, ...) distribution of source and target receptacles. I would like to see the distribution of train, validation and the excluded set directly compared In addition I want to understand if the trained network generalizes to the excluded data the same way it generalizes to the validation set. I want the authors to sample at least two more validation sets of the exact same size in houses as the current one. And perform Inference of at least 2 models. I fully understand the results are not going to be identical but the variability must be bounded. For this particular comparison I would like to see all the metrics (columns) presented in table 1
Dataset specification and reproducibility I don't see a clear specification of the dataset used in the PDF. It is clear to me that the images in the selected subset are identical, but the authors have already acknowledged that there are differences between the datasets. Clearly stating the version of the dataset is critical for the reproducibility of future work. The script to count the images was not provided. I had to write it myself, although I obtained the same result. I also had to write my own script to check the SHA hashes of the image files. I confirm they match.
V-score and model size The V-score results look reasonable for small models, but what about larger models? The original manuscript includes results in models up to 72B parameters. While the comparisons at 2B–4B look sound, providing only three results between 8B and 10B limits the evidence for the generality of the proposed method. Figure 2 should be updated to include the methods with in-domain pretraining.
M-score results Do the authors have results for the Symbolic-history regime (M-score)? To my understanding, the new results presented in the rebuttal cover only the V-score.