- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why a demonstration could be useful or harmful in the multimodal in context learning setting. It quantifies the structure of demonstration utility across 13 models, three multiple-choice visual question answering datasets, and 27 conditions through one-shots evaluations, measuring changes in correctness.
It gets several findings through experiments:
- which demonstrations can help rarely generalize across the models and queries
- the query matters more than the demonstration itself on the final performance
- the content similarity between query and demonstration cannot used as a signal to predict where that demonstration will be useful or not.
It also develops a gate-vote method which can moderately improve the final answer accuracy in the multimodal ICL setting.
Critical strengths
The paper has the below strengths:
- Large scale of experiments. It did large scale of the matrix evaluations (over 250,000 runs per condition) which provides robust statistical foundation
- Metrics decouple. It decouple the performance metrics into rescue and breaks, which clearly explain why the standard retrievers often fail
- Clear control. The Option -Text Control successfully isolates and exposed superficial answer-latter copying biases.
Critical weaknesses
I have a few concerns about the experiments design and their results.
-
Data leak.
App. A says that A-OKVQA and AI2D are from their training splits. Among the evaluated models, some of them may have seen these dataset during the tuning (e.g LLaVA-1.5's training datasets include A-OKVQA). Could the authors state which models were exposed to which training splits? It raises a concern about the data contamination. Memorized queries would have high zero shot confidence and leave less room for any demonstration to help. Also the contamination would differ across models which pollutes the cross model comparison in Sec.3. It may also explain in sec 5 why the retrievers failing below the zero-shot. -
Unbalanced query letters may turn letter copying into apparent demonstration main effects.
App. A says demonstration candidates are balanced by the gold letter but the queries are not. Assuming letter A is the most common correct answers for the queries, based on the section 4.2's conclusion which a model tends to copy the demonstration's answer letter, then demonstration ending with B will rescue more queries then other letters. It will look like a good demonstration but the rescue has nothing to do with its actual content. -
Real content vs. Prompt noise.
While the paper says the aggregate variance share of pairing residuals is consistent across baseline and explicit wordings, this does not show the pairing effect is from prompt sensitivity or actual content. If the pairing residual is largely from the prompt sensitivity, then it won't be replicated across wordings. Since the paper has evaluated both instruction settings, it should also report the correlation between R_baseline and R_explicit to demonstrate if the pairing effect is from stable semantic interaction or unreplicable prompt sensitivity -
Multiple variables in the E2E evaluation The proposed Gate + Arbitration pipeline use 20 random demonstrations with a selective gate to outperform retrievers. However, this comparison changes three variables at a time including 1. when to use the demonstration (selective gating) 2. ensemble size (20-draw voting vs 1 draw) 3. selection strategy (random vs retrieved). To improve the retrieved is actually weaker, the authors need evaluate the Gate + Retriever Arbitration baseline (using the retriever's top-20 candidates for the vote)
Questions for authors
No specific questions apart from the above concerns. And for the above concerns, can the authors answer / address them?