- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies many-shot multimodal in-context learning (ICL) in VLMs (Gemini 2, Gemma 3, Qwen 2.5 Omni). It shows pretrained models exhibit inconsistent, positionally-biased scaling ("lost-in-the-middle"), and that standard many-shot SFT (MS-SFT) yields little generalizable gain. In contrast, fine-tuning on a needle-in-a-haystack (NIH) proxy task substantially improves many-shot performance on held-out datasets after only ~200 gradient steps. Attention analysis suggests NIH training reduces positional bias and induces an approximately unbiased search over demonstrations.
Critical strengths
-
Many-shot multimodal ICL is important; the finding that a simpler proxy objective beats direct MS-SFT is counterintuitive and interesting.
-
Consistent across three model families, strengthening generality despite implementation differences.
-
Attention analysis coherently supports the hypothesis — entropy and primary-example attention metrics are interpretable proxies for positional bias.
-
Practical efficiency — gains after 200 steps is a valuable, well-emphasized result.
Critical weaknesses
-
Single seed per model, only 3 held-out datasets. Variance of headline claims is unknown; error bars on at least a subset would materially strengthen the central claim.
-
Authors concede attention analysis is "a useful heuristic rather than a mechanistic explanation"; Qwen GTSRB (lower entropy yet improved) and CLEVR Count (NIH-SFT degrades) are acknowledged but not reconciled. A taxonomy of when NIH helps vs. hurts is missing.
-
NIH's much lower loss (Fig. 4) is expected for a near-deterministic retrieval task and doesn't alone show MS-SFT is "wrong." A stronger MS-SFT baseline (more steps, matched compute) is needed to isolate the objective as the key variable.
-
No comparison to inference-time remedies (example selection/ordering, retrieval, curriculum) or to prior text-based NIH training — practical advantage over cheap fixes is unclear.
Questions for authors
-
Can you provide variance estimates (2–3 seeds) for at least the held-out average in Fig. 2?
-
How does NIH-SFT compare to a stronger MS-SFT baseline (e.g., 4× steps, matched compute)? Is the gain from objective or regime?
-
Can you ablate the NIH prompt's explicit textual mention of the needle (Appendix C.1)?
-
Why does NIH-SFT degrade on CLEVR Count? Can you characterize a task taxonomy where NIH is contraindicated?
-
What is the absolute Macro-F1 gain of NIH-SFT over base on the held-out average?
Response to Reviewer VDNA
"Can you provide variance estimates (2–3 seeds) for at least the held-out average in Fig. 2?"
Please see part 1 in the response to all reviewers.
"How does NIH-SFT compare to a stronger MS-SFT baseline (e.g., 4× steps, matched compute)? Is the gain from objective or regime?"
Please see part 2 in the response to all reviewers. Summary: Appendix Figure 6 shows that even when MS-SFT is given 4x the training compute, NIH-SFT still outperforms it.
"Can you ablate the NIH prompt's explicit textual mention of the needle (Appendix C.1)?"
Thanks for this observation! Held-out avg. Macro-F1 (Gemma 3 4B)
Removing the hint from the NIH training prompt on Gemma 3 4B actually improves held-out Macro-F1 across all shot counts (+0.024 on average, and +0.038 to +0.058 at 64–128 shots). This confirms NIH-SFT does not rely on a textual cue, and in fact, dropping the hint reduces prompt mismatch between NIH training and standard many-shot evaluation, pushing NIH-SFT further ahead in long contexts.
"Why does NIH-SFT degrade on CLEVR Count? Can you characterize a task taxonomy where NIH is contraindicated?"
As noted in Appendix B, NIH-SFT likely trains the model to rely on the most similar shots, which helps when similar-looking images share a label but may mislead when they don't. In CLEVR Count, visually similar scenes can contain different numbers of objects. Also MS-SFT relies less on one or a few shots and is therefore unaffected. We view tasks of this kind (counting and similar reasoning about the query itself) as outside the regime where NIH-SFT is intended to help, and will state this explicitly as a limitation.
"What is the absolute Macro-F1 gain of NIH-SFT over base on the held-out average?"
NIH-SFT's absolute held-out Macro-F1 gain over Base:
The gain grows with the number of shots because Base degrades as the context grows, while NIH-SFT keeps improving.
"NIH's much lower loss (Fig. 4) is expected for a near-deterministic retrieval task and doesn't alone show MS-SFT is "wrong." A stronger MS-SFT baseline (more steps, matched compute) is needed to isolate the objective as the key variable."
We agree that a lower loss in Figure 4 is expected for a near-deterministic retrieval task, and that the loss curve alone does not rule out MS-SFT simply needing more training steps. To investigate this, we ran the stronger 4x-steps MS-SFT baseline suggested by the reviewer (see part 2 in the response to all reviewers). We hypothesize that because NIH-SFT simplifies the input prompt to an unambiguous lookup task, it gives the model a clean signal to learn long-context visual attention, whereas MS-SFT requires learning task semantics and long-context lookup at the same time from noisier examples. This type of behavior is also supported by the literature on curriculum learning (e.g., Bengio et al., 2009), where simplifying the training inputs while keeping the learning objective similar helps guide optimization toward better solutions, leading not just to faster convergence, but also to stronger final generalization.
"No comparison to inference-time remedies (example selection/ordering, retrieval, curriculum) or to prior text-based NIH training — practical advantage over cheap fixes is unclear."
These strategies are largely complementary to NIH-SFT, though on their own they leave practical trade-offs because they work around positional bias rather than reducing it in the model: Ordering: Ordering requires deciding how to order the examples. Sorting by relevance relies on a separate embedding model capturing the right visual cues for the task at hand, which may be expensive or not always feasible. Retrieval: Our setup can already be considered a form of retrieval, where candidate examples have been "retrieved" into the prompt. Even with an external retriever, one still has to tune how many examples to include: too few leaves out useful context, while too many runs into the lost-in-the-middle problem. NIH-SFT removes the need to tune that count and pairs naturally with upstream retrieval. Curriculum: Curriculum is a training-time strategy, and as noted in Q6, NIH-SFT itself acts as a lightweight 200-step curriculum. Prior text-NIH: Prior text-NIH work focuses on literal long-context recall, whereas here we show that synthetic visual needle retrieval transfers zero-shot to class-level multimodal many-shot tasks and outperforms MS-SFT.