- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a test-time method that augment frozen multimodal models based on an external memory. The external memory is constructed by extracting reasoning rules and problem-related exemplars and used to boost the performance at the inference time with a Shapley-based utility estimation. The experimental results support their claims in the main text.
Critical strengths
- They propose an interesting method, which considers general reasoning rules and specific exemplars to allow the multimodal models to leverage both general knowledge and concrete reasoning experiences.
- They point out that relying solely on similarity between the rules and problems are not enough, and they design a detailed method to solve this.
Critical weaknesses
-
The novelty is not enough. They design a method for test-time inference, where they introduce a constructed external memory and design a detailed utility estimation. From my end, the contribution is combining an external memory with a detailed utility estimation, which is not novel for ICLR conf level.
-
Considering the experiments are conducted only under testtime, maybe the more comprehensive experiments could be performed and needed. The current setup shows that the baselines are only three methods and they just used 3 datasets. It'd be better if they include more experiments.
Questions for authors
Could you explain why there are drops (red results in Table 2)?