- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LOSCAR, which fuses visual features with LLM hidden states. Three spatial decoding branches produce BCD masks, semantic masks, and relevance masks alongside textual answers. The authors construct multi-task training data through automated question-answer generation and Gemini-based vocabulary expansion, and evaluate the model on multiple tasks, such as spatial and localization tasks and narrative tasks.
Critical strengths
-
The architecture covers a broad range of tasks. Separating the Change Decoder from the Relevance Decoder is a sensible design: the model can retain global change information while localizing the changes requested by the user.
-
Some results are competitive, such as on QAG-360K, LOSCAR achieves a CVG mIoU of 54.9, compared with 44.9 for VisTA. The ablations provide some evidence that multi-task supervision helps.
Critical weaknesses
-
The proposed design needs to be evaluated more clearly. The current ablations remove entire task branches, changing the architecture. They do not show whether the gains come from the proposed fusion mechanism. I would like to see comparisons that remove the LLM hidden states or replace cross-attention with a simpler fusion operation, while keeping the visual backbone, decoders, training data, and training budget the same. Open-vocabulary change detection is a central contribution, but it lacks quantitative evaluation. A vocabulary of ~ 4000 concepts that accepts arbitrary text queries does not demonstrate that the model can localize categories or change trajectories absent from its training supervision. The gains from vocabulary expansion are small. Also, removing the vocabulary expansion data (in the ablation study for now) and replacing the Embedding Decoder with a fixed-category classification head tests different things. The current experiments do not isolate the contribution of the open-vocabulary representation itself.
-
Some baselines are missing. The comparisons within each task are limited. There are no results from strong general-purpose VLMs, nor comparisons with modular systems that combine a VLM with grounding or segmentation tools. Closed-source VLMs that accept multiple images are also reasonable baselines for change captioning and VQA. Some baseline results differ from those reported in the original papers, such as MCI. If the authors reran these experiments, they should provide enough detail about the training and evaluation. The reported results need to be consistent across tables. For example, the CVG mIoU in Table 4 matches the result w/ pretraining in Table 6, while the textual answer OA matches the result w/o pretraining.
Questions for authors
See the questions in the weakness part.