acceptodds
Under review as a conference paper at ICLR 2027

Improving Multimodal Many-Shot Learning with Needle-in-a-Haystack Training

Abstract

Vision-language models have shown significant promise in combining text and image modalities, yet their ability to effectively utilize multimodal many-shot context remains understudied. We study the performance of VLMs on multimodal many-shot classification problems and observe that their performance is inconsistent. On some datasets the classification performance improves linearly for every doubling of the number of shots, while for others, the performance outright degrades as the number of shots increases. Surprisingly, we find that finetuning these models on a set of multimodal many-shot classification tasks does not improve their performance on previously unseen many-shot classification tasks. Instead, we demonstrate that finetuning the models on a simpler but related needle-in-a-haystack problem, where the image to be classified and its label (needle) are hidden in the context (the haystack), significantly improves the performance and scaling with the number of shots. We analyze the attention patterns of these models to demonstrate evidence that needle-in-a-haystack training provides a strong learning signal to search the entire context, which changes the positional bias of the learned attention pattern and thus allows the model to generalize to many-shot settings. Overall, our results show that just 200 gradient steps of needle-in-a-haystack training may greatly improve performance on many-shot learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Many shot learning, Needle in a haystack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VDNAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies many-shot multimodal in-context learning (ICL) in VLMs (Gemini 2, Gemma 3, Qwen 2.5 Omni). It shows pretrained models exhibit inconsistent, positionally-biased scaling ("lost-in-the-middle"), and that standard many-shot SFT (MS-SFT) yields little generalizable gain. In contrast, fine-tuning on a needle-in-a-haystack (NIH) proxy task substantially improves many-shot performance on held-out datasets after only ~200 gradient steps. Attention analysis suggests NIH training reduces positional bias and induces an approximately unbiased search over demonstrations.

Critical strengths
  1. Many-shot multimodal ICL is important; the finding that a simpler proxy objective beats direct MS-SFT is counterintuitive and interesting.

  2. Consistent across three model families, strengthening generality despite implementation differences.

  3. Attention analysis coherently supports the hypothesis — entropy and primary-example attention metrics are interpretable proxies for positional bias.

  4. Practical efficiency — gains after 200 steps is a valuable, well-emphasized result.

Critical weaknesses
  1. Single seed per model, only 3 held-out datasets. Variance of headline claims is unknown; error bars on at least a subset would materially strengthen the central claim.

  2. Authors concede attention analysis is "a useful heuristic rather than a mechanistic explanation"; Qwen GTSRB (lower entropy yet improved) and CLEVR Count (NIH-SFT degrades) are acknowledged but not reconciled. A taxonomy of when NIH helps vs. hurts is missing.

  3. NIH's much lower loss (Fig. 4) is expected for a near-deterministic retrieval task and doesn't alone show MS-SFT is "wrong." A stronger MS-SFT baseline (more steps, matched compute) is needed to isolate the objective as the key variable.

  4. No comparison to inference-time remedies (example selection/ordering, retrieval, curriculum) or to prior text-based NIH training — practical advantage over cheap fixes is unclear.

Questions for authors
  1. Can you provide variance estimates (2–3 seeds) for at least the held-out average in Fig. 2?

  2. How does NIH-SFT compare to a stronger MS-SFT baseline (e.g., 4× steps, matched compute)? Is the gain from objective or regime?

  3. Can you ablate the NIH prompt's explicit textual mention of the needle (Appendix C.1)?

  4. Why does NIH-SFT degrade on CLEVR Count? Can you characterize a task taxonomy where NIH is contraindicated?

  5. What is the absolute Macro-F1 gain of NIH-SFT over base on the held-out average?

1 comment
AuthorsOpenReview ↗

Response to Reviewer VDNA

"Can you provide variance estimates (2–3 seeds) for at least the held-out average in Fig. 2?"

Please see part 1 in the response to all reviewers.

"How does NIH-SFT compare to a stronger MS-SFT baseline (e.g., 4× steps, matched compute)? Is the gain from objective or regime?"

Please see part 2 in the response to all reviewers. Summary: Appendix Figure 6 shows that even when MS-SFT is given 4x the training compute, NIH-SFT still outperforms it.

"Can you ablate the NIH prompt's explicit textual mention of the needle (Appendix C.1)?"

Thanks for this observation! Held-out avg. Macro-F1 (Gemma 3 4B)

1248163264128
NIH-SFT, with hint0.2100.2150.2210.2440.2570.2710.2760.305
NIH-SFT, no hint0.2300.2300.2490.2580.2760.3080.3340.343

Removing the hint from the NIH training prompt on Gemma 3 4B actually improves held-out Macro-F1 across all shot counts (+0.024 on average, and +0.038 to +0.058 at 64–128 shots). This confirms NIH-SFT does not rely on a textual cue, and in fact, dropping the hint reduces prompt mismatch between NIH training and standard many-shot evaluation, pushing NIH-SFT further ahead in long contexts.

"Why does NIH-SFT degrade on CLEVR Count? Can you characterize a task taxonomy where NIH is contraindicated?"

As noted in Appendix B, NIH-SFT likely trains the model to rely on the most similar shots, which helps when similar-looking images share a label but may mislead when they don't. In CLEVR Count, visually similar scenes can contain different numbers of objects. Also MS-SFT relies less on one or a few shots and is therefore unaffected. We view tasks of this kind (counting and similar reasoning about the query itself) as outside the regime where NIH-SFT is intended to help, and will state this explicitly as a limitation.

"What is the absolute Macro-F1 gain of NIH-SFT over base on the held-out average?"

NIH-SFT's absolute held-out Macro-F1 gain over Base:

Model1 shot2 shots4 shots8 shots16 shots32 shots64 shots128 shots
Gemma 3 4B−0.010+0.019+0.053+0.040+0.068+0.125+0.160+0.202
Qwen 2.5 Omni 7B+0.015+0.017+0.026+0.027+0.038+0.045+0.069*–

The gain grows with the number of shots because Base degrades as the context grows, while NIH-SFT keeps improving.

"NIH's much lower loss (Fig. 4) is expected for a near-deterministic retrieval task and doesn't alone show MS-SFT is "wrong." A stronger MS-SFT baseline (more steps, matched compute) is needed to isolate the objective as the key variable."

We agree that a lower loss in Figure 4 is expected for a near-deterministic retrieval task, and that the loss curve alone does not rule out MS-SFT simply needing more training steps. To investigate this, we ran the stronger 4x-steps MS-SFT baseline suggested by the reviewer (see part 2 in the response to all reviewers). We hypothesize that because NIH-SFT simplifies the input prompt to an unambiguous lookup task, it gives the model a clean signal to learn long-context visual attention, whereas MS-SFT requires learning task semantics and long-context lookup at the same time from noisier examples. This type of behavior is also supported by the literature on curriculum learning (e.g., Bengio et al., 2009), where simplifying the training inputs while keeping the learning objective similar helps guide optimization toward better solutions, leading not just to faster convergence, but also to stronger final generalization.

"No comparison to inference-time remedies (example selection/ordering, retrieval, curriculum) or to prior text-based NIH training — practical advantage over cheap fixes is unclear."

These strategies are largely complementary to NIH-SFT, though on their own they leave practical trade-offs because they work around positional bias rather than reducing it in the model: Ordering: Ordering requires deciding how to order the examples. Sorting by relevance relies on a separate embedding model capturing the right visual cues for the task at hand, which may be expensive or not always feasible. Retrieval: Our setup can already be considered a form of retrieval, where candidate examples have been "retrieved" into the prompt. Even with an external retriever, one still has to tune how many examples to include: too few leaves out useful context, while too many runs into the lost-in-the-middle problem. NIH-SFT removes the need to tune that count and pairs naturally with upstream retrieval. Curriculum: Curriculum is a training-time strategy, and as noted in Q6, NIH-SFT itself acts as a lightweight 200-step curriculum. Prior text-NIH: Prior text-NIH work focuses on literal long-context recall, whereas here we show that synthetic visual needle retrieval transfers zero-shot to class-level multimodal many-shot tasks and outperforms MS-SFT.

Reviewer_9omSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal many-shot classification and proposes needle-in-a-haystack (NIH) finetuning to improve context utilization. During training, the query image and its label are inserted among the demonstrations, allowing the model to learn image matching and label retrieval. Experiments across three model families and thirteen datasets show that 200 steps of NIH finetuning can improve classification on held-out datasets compared with the base models and direct many-shot finetuning. Attention analyses suggest that these improvements are associated with changes in positional bias and attention to demonstration labels.

Critical strengths
  1. The paper identifies a meaningful problem: adding more multimodal demonstrations does not consistently improve performance. Studying how training objectives affect this behavior is practically relevant.

  2. The proposed intervention is simple and requires no architectural changes. The comparison between NIH-SFT and MS-SFT uses the same training budget within each model family, providing useful evidence about their relative effectiveness under limited training.

  3. The evaluation covers diverse classification datasets and three model families, with separate held-out datasets for assessing transfer. The attention analyses also provide useful observations beyond the overall performance results.

Critical weaknesses
  1. The experiments do not sufficiently isolate improved use of visual demonstrations. Both training and evaluation retain natural-language class labels, so the gains may partly reflect label semantics or distributional cues. Controls that preserve the demonstration text while replacing images or shuffling image–label pairings would help establish whether NIH-SFT improves the use of visual demonstrations on held-out tasks.

  2. The attention analysis does not sufficiently establish that reduced positional bias explains the performance gains. The entropy of the most-attended example’s position measures positional preference across prompts, but does not establish whether relevant examples are selected. Table 5 also shows that Qwen’s entropy on GTSRB decreases from 2.3 to 1.9 after NIH-SFT despite improved classification performance. Evaluating fixed demonstration sets under controlled position changes would help clarify this relationship.

  3. The MS evaluation does not clearly exclude the query image from the demonstrations. Section 4 samples demonstrations from the same validation split as the query, and Appendix C.1 acknowledges possible overlap. For Hateful Memes, with 540 validation images, 256 shots would give approximately a 38%–47% inclusion probability if the query remains eligible for uniform sampling. Such overlap could favor NIH-SFT because it is trained on exact-match retrieval. Please clarify query exclusion and, if necessary, report results with strict query–demonstration separation.

Questions for authors

Please see the questions raised in the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9omS

"The experiments do not sufficiently isolate improved use of visual demonstrations. Both training and evaluation retain natural-language class labels, so the gains may partly reflect label semantics or distributional cues. Controls that preserve the demonstration text while replacing images or shuffling image–label pairings would help establish whether NIH-SFT improves the use of visual demonstrations on held-out tasks."

We conducted experiments where we replaced the labels with semantically meaningless words. The results are in Appendix E.2. The experiments were conducted on a dataset with 32 shots. For Qwen 2.5 Omni, removing label semantics results in a substantial performance degradation for both the base model and MS-SFT. Macro-F1 decreases from 0.421 to 0.162 for the base model and from 0.431 to 0.176 for MS-SFT. In contrast, NIH-SFT exhibits a much smaller decrease, from 0.512 to 0.443. Notably, NIH-SFT with semantically meaningless labels even outperforms MS-SFT with the original semantic labels (0.443 versus 0.431). For Gemma 3, performance is comparatively insensitive to label semantics across all three models. Nevertheless, NIH-SFT consistently achieves the highest Macro-F1, obtaining 0.293 with semantically meaningless labels compared to 0.182 for MS-SFT with the original labels. These findings suggest that NIH-SFT improves the model’s ability to leverage in-context examples without relying excessively on the pretrained semantic associations of class labels.

"The attention analysis does not sufficiently establish that reduced positional bias explains the performance gains. The entropy of the most-attended example’s position measures positional preference across prompts, but does not establish whether relevant examples are selected. Table 5 also shows that Qwen’s entropy on GTSRB decreases from 2.3 to 1.9 after NIH-SFT despite improved classification performance. Evaluating fixed demonstration sets under controlled position changes would help clarify this relationship."

We conducted experiments with random shuffling and included the results in Appendix E1. We observe that for both Gemma and Qwen the base model has a strong attention bias towards one position in the context for images and labels. The story with MS-SFT is mixed. For Gemma, the label attention follows the content moving to the same example after re-ordering more often but the image attention remains fixed at the image in the same location. For Qwen, MS SFT behaves very similarly to the base model. On the other hand, for both models, NIH-SFT's most-attended label stays with the same example in about 40% of prompts (random chance 3.1%) and with the same position in at most 8%. These results suggest that NIH-SFT substantially reduces positional bias, particularly in image attention, encouraging the model to select relevant examples based on their content rather than their location in the prompt.

"The MS evaluation does not clearly exclude the query image from the demonstrations. Section 4 samples demonstrations from the same validation split as the query, and Appendix C.1 acknowledges possible overlap. For Hateful Memes, with 540 validation images, 256 shots would give approximately a 38%–47% inclusion probability if the query remains eligible for uniform sampling. Such overlap could favor NIH-SFT because it is trained on exact-match retrieval. Please clarify query exclusion and, if necessary, report results with strict query–demonstration separation."

Thanks for this excellent observation. In our original pipeline, support examples were sampled uniformly from the split without explicitly filtering out the query, so small splits could occasionally include it. Therefore, we regenerated all evaluation prompts to enforce strict query–demonstration exclusion and updated Figure 2. Results are virtually unchanged, confirming that NIH-SFT's gains over Base and MS-SFT do not come from exact-duplicate leakage at test time, but from transfer to unseen class-level visual examples.

Reviewer_tE3AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies many-shot image classification with vision-language models. It proposes needle-in-a-haystack training, where the query image and its label appear in the training context. Experiments examine whether this retrieval task improves classification on held-out datasets. Attention analysis is also used to investigate changes in positional bias.

Critical strengths

The paper is generally well written and easy to follow.

Using a simple retrieval task to improve many-shot classification is an interesting idea.

The experiments cover three model families and include held-out datasets, providing useful evidence of transfer.

Critical weaknesses
  1. The stability of the reported improvements is unclear. Appendix C.1 reports only one fine-tuning seed per model. The standard deviations in the attention analysis describe variation across prompts, rather than independent training runs.

  2. Sensitivity to the training budget is not examined. Both NIH-SFT and MS-SFT are trained for 200 steps, so it is unclear whether the weaker transfer of MS-SFT would persist with a longer training run.

  3. The attention analysis does not directly establish why the method improves classification. A more uniform distribution of attended positions does not necessarily mean that the model uses more relevant examples. Controlled position changes and arbitrary label remapping would help distinguish the proposed explanation from retrieval effects or reliance on pretrained label meanings.

  4. The paper reports the training steps and hardware, but not the actual training time or computational cost. This makes the claim of improvement at minimal cost difficult to assess.

Questions for authors

Please see the weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer tE3A

Please find our responses to your comments:

"The stability of the reported improvements is unclear. Appendix C.1 reports only one fine-tuning seed per model. The standard deviations in the attention analysis describe variation across prompts, rather than independent training runs."

Please see part 1 in the response to all reviewers. Summary: We repeated our main experiments on 3 seeds for both MS SFT and NIH SFT on Gemma and Qwen. In these experiments we observe a clear separation between the performance of NIH SFT and MS SFT. This is more pronounces as the number of shots increases. Noise does not account for the improved performance of NIH-SFT. We will include the results with multiple seeds on Gemini as well in the final paper.

"Sensitivity to the training budget is not examined. Both NIH-SFT and MS-SFT are trained for 200 steps, so it is unclear whether the weaker transfer of MS-SFT would persist with a longer training run."

Please see part 2 in the response to all reviewers. Summary: Appendix Figure 6 shows that even when MS-SFT is given 4x the training compute, NIH-SFT still outperforms it, showing the gain comes from the NIH training signal rather than under-training MS-SFT.

"Controlled position changes and arbitrary label remapping would help distinguish the proposed explanation from retrieval effects or reliance on pretrained label meanings."

We conducted experiments with controlled position changes and included the results in Appendix E1. We observe that for both Gemma and Qwen the base model has a strong attention bias towards one position in the context for images and labels. The story with MS-SFT is mixed. For Gemma, the label attention follows the content moving to the same example after re-ordering more often but the image attention remains fixed at the image in the same location. For Qwen, MS SFT behaves very similarly to the base model. On the other hand, for both models, NIH-SFT's most-attended label stays with the same example in about 40% of prompts (random chance 3.1%) and with the same position in at most 8%. These results suggest that NIH-SFT substantially reduces positional bias, particularly in image attention, encouraging the model to select relevant examples based on their content rather than their location in the prompt.

We also conducted experiments where we replaced the labels with semantically meaningless words. The results are in Appendix E.2. The experiments were conducted on a dataset with 32 shots. For Qwen 2.5 Omni, removing label semantics results in a substantial performance degradation for both the base model and MS-SFT. Macro-F1 decreases from 0.421 to 0.162 for the base model and from 0.431 to 0.176 for MS-SFT. In contrast, NIH-SFT exhibits a much smaller decrease, from 0.512 to 0.443. Notably, NIH-SFT with semantically meaningless labels even outperforms MS-SFT with the original semantic labels (0.443 versus 0.431). For Gemma 3, performance is comparatively insensitive to label semantics across all three models. Nevertheless, NIH-SFT consistently achieves the highest Macro-F1, obtaining 0.293 with semantically meaningless labels compared to 0.182 for MS-SFT with the original labels. These findings suggest that NIH-SFT improves the model’s ability to leverage in-context examples without relying excessively on the pretrained semantic associations of class labels.

"The paper reports the training steps and hardware, but not the actual training time or computational cost. This makes the claim of improvement at minimal cost difficult to assess."

Wall-clock training times and hardware for all 200 NIH-SFT steps are: Gemma ~ 2.4 hours on 4 TPUs Qwen ~ 0.5 hours on 16 A100s Gemini ~ 2.7 hours on 512 TPUs

Every model finishes fine-tuning in under 3 hours (and just 30 minutes for Qwen). Compared to pre-training foundation models on thousands of chips for weeks, this is a negligible one-time cost for large many-shot gains (up to +20 Macro-F1 at 128 shots), and it uses a quarter of the compute of our 4x MS-SFT baseline while still outperforming it. We have added these training times to Appendix C.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall response to reviewer comments

We thank the reviewers for their thoughtful feedback. Your suggestions have helped us improve the paper. Several concerns were shared across reviews, so we ran a common set of new experiments and addressed them below.

  1. Run more seeds. (tE3A, VDNA) We ran three fine-tuning seeds for each model and updated Figure 2 with the mean and standard deviation bands. The results show that the standard deviation is quite small and noise does not explain the difference in performance between NIH SFT and MS SFT. Please see Figure 2 in the revised paper. We prioritized compute on the held-out datasets and on the Gemini and Qwen models for the rebuttal. We will include Gemini and training dataset results in the final paper.

  2. MS-SFT sensitivity to the training budget. (tE3A, VDNA) To test whether MS-SFT simply needs longer to converge on a harder training task, we trained MS-SFT for 4x as many gradient steps as NIH-SFT (giving MS-SFT 4x the total compute, since per-step compute is matched). Even with 4x the compute, MS-SFT plateaus below NIH-SFT and does not close the gap significantly (see Appendix Figure 6). This strengthens our central finding: NIH-SFT is not just faster to train, it learns a better-transferring many-shot capability on unseen tasks, even though MS-SFT trains directly on the many-shot format. As observed in curriculum learning, isolating the core cross-context lookup skill first appears to give cleaner supervision than training end-to-end on noisy many-shot prompts.

  3. Ablations. We extended the attention analysis to larger numbers of shots to see how the attention pattern changes. We updated Table 1 in the paper and added Appendix H.

Gemma:

ShotsModelTop image at position 1Top label at position 1Image-position entropyLabel-position entropyMax entropy
8Base100%90%0.000.683
8MS-SFT91 ±2%16 ±1%0.60 ±0.102.91 ±0.043
8NIH-SFT87 ±5%29 ±6%0.83 ±0.222.77 ±0.043
32Base100%83%0.001.005
32MS-SFT85 ±7%5 ±3%1.05 ±0.484.31 ±0.345
32NIH-SFT30 ±5%3 ±4%3.88 ±0.264.70 ±0.075
64Base100%99%0.000.086
64MS-SFT66 ±15%3 ±3%2.12 ±0.675.18 ±0.266
64NIH-SFT5 ±1%3 ±5%4.69 ±0.185.41 ±0.096

Qwen:

ShotsModelTop image @lastTop label @lastTop image @pos1Top label @pos1Image-position entropyLabel-position entropyMax entropy
8Base54%64%25%12%1.971.893
8MS-SFT56 ± 3%58 ± 3%25 ± 4%12 ± 2%1.84 ± 0.072.10 ± 0.083
8NIH-SFT38 ± 3%31 ± 2%33 ± 3%24 ± 3%2.31 ± 0.022.67 ± 0.013
32Base57%58%21%17%2.272.355
32MS-SFT55 ± 1%59 ± 2%20 ± 3%11 ± 2%2.39 ± 0.122.55 ± 0.055
32NIH-SFT33 ± 5%22 ± 1%21 ± 3%7 ± 1%3.53 ± 0.164.37 ± 0.045
64Base80%83%8%6%1.261.166
64MS-SFT63 ± 3%60 ± 1%20 ± 3%6 ± 1%1.87 ± 0.062.67 ± 0.016
64NIH-SFT46 ± 7%27 ± 4%20 ± 5%8 ± 2%3.07 ± 0.164.35 ± 0.166

For Gemma, we observe a strong bias toward the first example. The Base model attends most strongly to the first image in 100% of cases across all context lengths. MS-SFT partially mitigates this bias, while NIH-SFT reduces it substantially further, particularly as the number of shots increases. At 64 shots, the first image receives the most attention in 100% of cases for Base, 66% for MS-SFT, and only 5% for NIH-SFT. This trend is also reflected in image-position entropy, which increases from 0.00 for Base to 2.12 for MS-SFT and 4.69 for NIH-SFT. We observe similar improvements in label-position entropy, with NIH-SFT achieving 5.41 compared to 5.18 for MS-SFT and 0.08 for Base. Notably, the advantage of NIH-SFT over MS-SFT in image-position entropy grows with the number of shots, suggesting that NIH training is particularly effective at mitigating positional biases in longer contexts.

For Qwen, we instead observe a strong bias toward the final example, demonstrating that positional preferences can differ substantially across models. At 64 shots, the Base model attends most strongly to the final image and label in 80% and 83% of cases, respectively. MS-SFT reduces these percentages to 63% and 60%, while NIH-SFT further reduces them to 46% and 27%. Similarly, NIH-SFT achieves image- and label-position entropies of 3.07 and 4.35, compared to 1.87 and 2.67 for MS-SFT and 1.26 and 1.16 for Base. Although NIH-SFT does not completely eliminate the final-position bias, it consistently produces more diverse distributions of the most-attended positions. These results suggest that NIH-SFT mitigates a general positional bias in multimodal in-context learning.

We also provide individual responses to each reviewer's comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.