- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper performs experiments on a number of VLM agent tasks to study the effect of ICL on performance. Specifically, the paper designs experiments to study this compared to other baselines such as having unrelated demos or having things in context that do not give the answer but provide additional domain context. The paper finds there is a narrowing of the gap due to other effects.
Critical strengths
None
Critical weaknesses
The elephant in the room is that this paper seems to be majority, if not fully AI generated. The AI disclosure acknowledges the use of AI to run experiments and write major sections of the paper. And having read it, the quality of the writing, analysis and choice of research problem reflects this. I think this alone should make this paper ineligible for acceptance.
If I were to completely ignore the AI part, the scope of the paper is extremely narrow and I do not believe has enough of a contribution to be accepted as an ICLR paper. The focus is very narrow: just on this set of questions about teasing out the effects of ICL from other effects by prompting different demonstrations into context and comparing numbers. There is no methodology, really, this is just doing really well-known prompting techniques and analyzing the results. This is the kind of experiments that I would expect as a subsection of an ICLR paper that also did something else: introduce a new method, create a new datasets. Even ICLR papers that are just analysis will introduce some interesting new method of analysis or tackle a really unique or unknown phenomenon and do a really good job of this. But there has been so much work at this point looking at in-context learning and prompting in LLMs, VLMs and agents, that even the choice of problem is not very unique.
Questions for authors
None