acceptodds
Under review as a conference paper at ICLR 2027

ICL-R: Reconstructing the Computational Effects of Demonstrations for Implicit Multimodal In-Context Learning

Abstract

Multimodal in-context learning (ICL) enables multimodal large language models (MLLMs) to adapt to downstream tasks through interleaved image and text demonstrations. However, retaining these demonstrations not only increases the sequence length and latency at inference time but also makes predictions sensitive to how demonstrations are selected and presented. Although existing implicit ICL methods approximate demonstration effects through representation shifts or attention routing, they still suffer from coarse or partial reconstruction of these effects. To address these limitations, we first decompose the change in attention output caused by demonstrations into the attention routing shift, the value shift, their interaction, and information contributed directly by demonstration tokens. Based on this decomposition, we propose In-Context Learning via Reconstruction (ICL-R), a lightweight framework for jointly reconstructing these computational effects. Aiming at incomplete reconstruction under demonstration removal, we propose mass-restoring joint attention-value calibration which jointly calibrates attention routing and value representations while rescaling the attention output. To further extend supervision beyond isolated state matching, we introduce a Riemannian state transition objective that aligns both incoming states and the transitions induced by attention. Extensive experiments across multiple MLLMs and benchmarks show that ICL-R improves performance while maintaining an inference cost close to that of standard zero-shot inference.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Implicit Multimodal In-Context Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QxAEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ICL-R, which jointly calibrates attention and value representations and uses a Riemannian state transition loss to transfer the behavior of a demonstration-conditioned teacher to a query-only student, reducing demonstration-related overhead at inference time.

Critical strengths
  1. Jointly calibrating attention and value representations is well motivated, and the ablation results support the effectiveness of value calibration.

  2. The experiments cover multiple models and six benchmarks. The appendix further provides cross-dataset transfer results, GPU memory usage, and error analyses, offering a fairly comprehensive evaluation.

Critical weaknesses
  1. Hyper-ICL already employs query-adaptive low-rank attention calibration and geometric distillation, making it closely related to this work. Although the paper cites this method, it does not include it in the experimental comparisons, making it difficult to assess the incremental benefits of the proposed value calibration and RST loss over this direct predecessor.

  2. In Table 2, the cross-entropy-only variant already achieves 61.64 and 73.20 on OK-VQA and VQAv2, respectively, while the full model improves these scores to 62.49 and 73.38. Does this suggest that most of the performance gains can be obtained without supervision from a demonstration-conditioned teacher?

  3. The current experiments do not directly compare against the simpler training objective of jointly aligning normalized pre-attention and post-attention states. Consequently, it remains unclear whether the gains come from additional state supervision or from spherical geometry and parallel transport themselves.

Questions for authors

Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/2)

Thank you for recognizing our motivation behind joint attention–value calibration and the breadth of our experimental evaluation. We sincerely apologize for the confusion caused by the insufficiently detailed explanations in the original manuscript. Here we provide additional experiments and clarifications.

1. Comparison with Hyper-ICL (W1)?

At the time of submission, we could not locate publicly released code for Hyper-ICL. We still could not find an available public implementation from the authors when preparing this response, which prevented us from directly evaluating their implementation under our experimental settings across all benchmarks. To provide this comparison, we independently reimplemented the query-adaptive logit adapter and hyperbolic anchor loss based on the paper. We report both a rank-4 configuration and a rank-16 configuration with a parameter count comparable to that of ICL-R. We also checked the paper and the authors’ webpages but found no link to a public implementation.

As shown in Table R3.1, both methods use a total rank of 16 (r=16r=16 for Hyper-ICL and rA=rV=8r_A=r_V=8 for ICL-R) and have approximately equal numbers of trainable parameters. In this comparison, ICL-R improves VQAv2, OK-VQA, MME, and SEED by 2.93, 2.67, 1.97, and 12.32 percentage points, respectively, and improves COCO CIDEr by 0.0796 and Flickr30k by 0.0178. These comparisons assess the complete method against prior work.

Table R3.1

MetricHyper-ICL (Reported)Hyper-ICL (Reproduced,1.181M Parameters)Hyper-ICL (Reproduced, 4.326M Parameters)ICL-R
OK-VQA62.2459.5959.8262.49 (+4.46%)
VQAv271.1770.3370.4573.38 (+4.16%)
COCO1.35661.24371.29611.3757 (+6.14%)
MMEN/A85.384.8686.83 (+2.32%)
SEEDN/A50.8751.6363.95 (+23.86%)
Flickr30kN/A0.90720.88040.8982 (+2.02%)

Percentages in parentheses are relative gains over the rank-16 Hyper-ICL reimplementation.

2. Contribution of Demonstration-Conditioned Teacher Supervision (W2)?

We truly understand your concern about our results in Table 2: CE-only already achieves strong performance on Idefics2 with 8-shot training, reflecting the effectiveness of our calibration modules. Under this configuration, the additional benefit from RST is relatively limited. However, this observation cannot be directly generalized to other models or the number of training shots. Below, we report results evaluated on a subset of 1,024 examples on VQAv2 and OK-VQA under 32-shot training with different models.

As shown in Table R3.2, CE + demonstration-conditioned RST outperforms CE-only in all six settings, with gains ranging from 0.41 to 12.85 percentage points. It also outperforms CE + query-only teacher RST in all six settings, supporting an additional contribution from demonstration-conditioned targets.

RST-only training exceeds CE-only in four of the six settings, showing that reconstructing teacher computations provides useful supervision even without the student CE term. Furthermore, demonstration-conditioned RST outperforms CE + normalized state matching in all six settings, indicating that the choice of alignment objective also matters. Together, we consider these comparisons to support the utility of both demonstration-conditioned distillation and our RST design.

Table R3.2

Model / TaskZero-shotCE OnlyCE + Query-Only Teacher RSTCE + Demonstration-Conditioned Teacher RST (ICL-R)RST OnlyCE + Normalized State MatchingΔ
Idefics-9B / VQAv244.101644.24852.685557.099655.898451.533212.8516
Idefics-9B / OK-VQA29.453143.984445.781350.62547.20742.56.6406
Qwen3-VL-2B / VQAv274.804774.150473.281378.574277.79372.93954.4238
Qwen3-VL-2B / OK-VQA42.851646.445346.484452.539152.070346.13286.0938
LLaVA-Interleave / VQAv235.410276.650476.865277.851675.839876.77731.2012
LLaVA-Interleave / OK-VQA18.906355.039155.019555.449250.781354.06250.4102

Δ is the gain of full ICL-R over CE only, in percentage points.

AuthorsOpenReview ↗

Response to Reviewer 3 (2/2)

3. Joint Alignment of Normalized Pre- and Post-Attention States (W3)?

Thank you for pointing out the comparison of joint alignment of pre-attention and post-attention states to better support the validity of our design. We have added this comparison together with four additional auxiliary losses to distinguish the contribution of supervising both sides of attention from the effects of the alignment geometry and parallel transport.

Following Sec. 3.3 of our paper, hiah_i^a denotes the pre-attention residual state, oiao_i^a the update after WOW_O, and z~ia=hia+oia\tilde{z}_i^a=h_i^a+o_i^a the unnormalized post-attention state, where a∈S,Ta\in\\{S,T\\} identifies the student or teacher. For the unit-sphere projection P(r)=r/∥r∥2P(r)=r/\lVert r\rVert_2, we define

xia=P(hia),zia=P(z~ia),via=Log⁡xia(zia).x_i^a = P(h_i^a),\qquad z_i^a = P(\tilde{z}_i^a),\qquad v_i^a = \operatorname{Log}_{x_i^a}(z_i^a).

For the hyperbolic loss, Φ=Exp⁡o∘LN⁡\Phi=\operatorname{Exp}_o\circ\operatorname{LN} and κ=−0.1\kappa=-0.1. The last row compares the logarithmic transitions directly in the ambient Euclidean space.

ObjectiveLoss
Hyperbolic endpointsℓhyp=12[dH2(Φ(hiS),Φ(hiT))+dH2(Φ(z~iS),Φ(z~iT))]\displaystyle \ell_{\mathrm{hyp}} = \frac12 \left[d_{\mathbb H}^2(\Phi(h_i^S),\Phi(h_i^T)) + d_{\mathbb H}^2(\Phi(\tilde{z}_i^S),\Phi(\tilde{z}_i^T))\right]
Raw endpointsℓL2-pair=12[∥hiS−hiT∥22+∥z~iS−z~iT∥22]\displaystyle \ell_{\mathrm{L2\text{-}pair}} = \frac12 \left[\lVert h_i^S-h_i^T\rVert_2^2 + \lVert\tilde{z}_i^S-\tilde{z}_i^T\rVert_2^2\right]
Unit-sphere endpointsℓsph=12[∥xiS−xiT∥22+∥ziS−ziT∥22]\displaystyle \ell_{\mathrm{sph}} = \frac12 \left[\lVert x_i^S-x_i^T\rVert_2^2 + \lVert z_i^S-z_i^T\rVert_2^2\right]
State and updateℓL2-update=12[∥hiS−hiT∥22+∥oiS−oiT∥22]\displaystyle \ell_{\mathrm{L2\text{-}update}} = \frac12 \left[\lVert h_i^S-h_i^T\rVert_2^2 + \lVert o_i^S-o_i^T\rVert_2^2\right]
Without parallel transportℓnoPT=dS2(xiS,xiT)+∥viS−viT∥22\displaystyle \ell_{\mathrm{noPT}} = d_{\mathbb S}^2(x_i^S,x_i^T) + \lVert v_i^S-v_i^T\rVert_2^2

We evaluate these objectives with Idefics2 on OK-VQA and COCO. Table R3.3 reports 62.49/1.3757 on OK-VQA/COCO for RST, while the requested normalized endpoint objective obtains 61.64/1.3707 and Euclidean state–update matching obtains 61.86/1.3742. Since these alternatives already constrain both sides of attention, the results support a contribution from RST’s loss formulation when paired-state supervision is already included.

The no-PT control obtains 62.03/1.3708. Full RST improves these scores by 0.46 percentage points and 0.0049 CIDEr without changing the starting-state or logarithmic-transition information being supervised. These comparisons indicate that parallel transport provides an additional gain while the supervised state information remains unchanged.

You can also refer to the results in our response to your second comment, where comparisons with CE-only training and normalized state matching provide further evidence for the contribution of our RST on different models and settings. We have added these comparisons to Table 3 and expanded the discussion of RST in the revised manuscript.

Table R3.3

DatasetHyperbolic endpointEndpoint L2Spherical endpoint L2State–update L2 lossWithout PTICL-R
OK-VQA59.62 (+2.87)61.91 (+0.58)61.64 (+0.85)61.86 (+0.63)62.03 (+0.46)62.49
COCO1.2377 (+0.1380)1.3578 (+0.0179)1.3707 (+0.0050)1.3742 (+0.0015)1.3708 (+0.0049)1.3757

Dear Reviewer QxAE,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Dear Reviewer QxAE,

Thank you for your constructive feedback. We summarize the key findings addressing your three concerns below.

  • W1. ICL-R outperforms our comparably sized Hyper-ICL reimplementation on all six benchmarks, including gains of 2.93 pp on VQAv2 and 12.32 pp on SEED (Table R3.1).
  • W2. ICL-R outperforms both CE-only and query-only teacher supervision in all six additional 32-shot settings, gaining 12.85 pp over CE-only on IDEFICS-9B/VQAv2(Table R3.2).
  • W3. RST achieves the best results among all tested loss variants on both OK-VQA and COCO under the same student architecture, including the normalized endpoint and no-parallel-transport controls (Table R3.3).

Thank you again for your time and helpful suggestions.

Best regards,
The Authors

Reviewer_konDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ICL-R, a framework for transferring demonstration-induced effects into a query-only multimodal model. Motivated by an exact decomposition of attention-output differences, the method jointly calibrates attention routing and value representations, introduces a missing-mass scaling factor, and uses a Riemannian State Transition objective to align teacher and student computations. Experiments across multiple backbones and benchmarks demonstrate promising performance with inference costs close to zero-shot inference.

Critical strengths
  1. The paper addresses a practically relevant problem, reducing the inference overhead of multimodal demonstrations while retaining useful task adaptation.
  2. The joint attention–value design is well motivated. The routing-only ablation under a matched total rank budget provides meaningful evidence for the benefit of value calibration.
  3. The experimental analysis is extensive, including cross-dataset transfer, image-level bootstrap confidence intervals, failure cases, and an efficient SDPA implementation.
Critical weaknesses
  1. The student is trained with ground-truth labels, and the CE-only variant already performs strongly. In Table 2, adding RST improves VQAv2 from 73.20 to 73.38 and OK-VQA from 61.64 to 62.49. These results support the effectiveness of the proposed architecture, but leave some uncertainty about how much additional benefit specifically comes from reconstructing demonstration-conditioned computations. A controlled comparison using the same student architecture and supervision, with demonstration-conditioned versus query-only teacher states, would help substantiate this central interpretation.
  2. Hyper-ICL shares several central elements with ICL-R, including low-rank attention-logit calibration, query-adaptive modulation, and geometric intermediate distillation. Although it is discussed throughout the paper, it is absent from the experimental comparisons. The routing-only ablation is informative, but does not fully replace a comparison with the complete prior method. Evaluating Hyper-ICL under the same data partitions and comparable parameter and optimization budgets would clarify the incremental contributions of joint value calibration, missing-mass scaling, and RST.
  3. RST changes both the geometry of the alignment objective and the information being supervised, aligning the incoming state together with its attention-induced transition. Consequently, its advantage over pointwise Euclidean matching may partly arise from supervising both sides of the attention operation. The controlled geometric analyses in Appendix A.7 are useful, but downstream comparisons with normalized pre-/post-attention state matching and Euclidean state-and-update matching would more directly establish the value of the spherical formulation and parallel transport.
  4. Under nonsingular conditions, the starting point x and the transition Logx(y) uniquely determine the endpoint y. It is therefore useful to distinguish two potential sources of improvement: jointly constraining the pre- and post-attention states, and adopting spherical distances and parallel transport. Useful additional comparisons would include paired matching of normalized pre- and post-attention states, Euclidean matching of states and attention updates, and downstream evaluation without parallel transport, with loss weights appropriately tuned for each objective.
  5. To what extent do the performance gains arise from reconstructing demonstration-induced computations versus supervised adaptation? The student is trained with ground-truth answers using cross-entropy, and each benchmark provides 1,000 training examples. The improvements may therefore also reflect the new modules learning task distributions, output formats, or answer priors.
Questions for authors
  1. With the student architecture, labels, and training budget held fixed, how does a demonstration-conditioned teacher compare with a query-only teacher in both downstream performance and computational recovery?
  2. Why the Hyper-ICL is not reported in the results? Hyper-ICL includes the low-rank attention-logit calibration, query-adaptive modulation and intermediate distillation.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (1/3)

Thank you for your careful reading and detailed feedback. We really appreciate the specific experiments you suggested to support our principal claim.

1. Contribution of teacher states and demonstration reconstruction?

Thank you for raising this important point. To address both questions, we conducted experiments on different models with 32 shot training under the same configuration. Due to time and computational constraints, evaluation uses 1,024 randomly sampled examples from each benchmark.

Contribution of demonstration-conditioned teacher states (W1/Q1): Following your suggestion, we compare CE-only training, CE + RST with query-only teacher states, and full ICL-R with demonstration-conditioned teacher states. As shown in Table R2.1.1, ICL-R outperforms the query-only teacher variant in all six model–dataset settings. On IDEFICS-9B, demonstration-conditioned targets improve VQAv2 from 52.69 to 57.10 and OK-VQA from 45.78 to 50.63. The same advantage is observed on Qwen3-VL-2B, where performance increases from 73.28 to 78.57 and from 46.48 to 52.54, respectively. On LLaVA-Interleave, ICL-R improves VQAv2 from 76.87 to 77.85 and OK-VQA from 55.02 to 55.45. We consider this controlled comparison supports an additional benefit from demonstration-conditioned targets beyond matching query-only teacher states.

Table R2.1.1

Model / TaskCE-onlyCE + RST (query-only teacher)ICL-R
Idefics-9B / VQAv244.25 (+12.85)52.69 (+4.41)57.10
Idefics-9B / OK-VQA43.98 (+6.64)45.78 (+4.85)50.63
Qwen3-VL-2B / VQAv274.15 (+4.42)73.28 (+5.29)78.57
Qwen3-VL-2B / OK-VQA46.45 (+6.09)46.48 (+6.06)52.54
LLaVA-Interleave / VQAv276.65 (+1.20)76.87 (+0.98)77.85
LLaVA-Interleave / OK-VQA55.04 (+0.41)55.02 (+0.43)55.45

Reconstruction versus direct supervised adaptation (W5/Q1): Extending your suggested comparison, we additionally evaluate an RST-only variant that removes the student CE term and retains only demonstration-conditioned state supervision. As shown in Table R2.1.2, RST-only outperforms CE-only on both benchmarks for both evaluated backbones. On IDEFICS-9B, it achieves 55.90 on VQAv2 and 47.21 on OK-VQA, exceeding CE-only by 11.65 and 3.22 percentage points, respectively. On Qwen3-VL-2B, the corresponding results are 77.79 and 52.07, with gains of 3.64 and 5.63 percentage points. These results show that reconstructing demonstration-conditioned computations provides useful supervision even without the student CE term. The gains therefore cannot be attributed solely to learning task distributions, output formats, or answer priors through direct ground-truth supervision. You can also refer to our reconstruction analyses in Section 4 (Figure 2) and Appendix A.5 (Figure 10), which directly measure directional alignment with demonstration-induced computational changes relative to the frozen query-only model. We have added these ablations to Table 2 and expanded the discussion of reconstruction versus supervised adaptation in our revised manuscript.

Table R2.1.2

Model / TaskCE-onlyRST-onlyΔ
Idefics-9B / VQAv244.2555.90+11.65
Idefics-9B / OK-VQA43.9847.21+3.22
Qwen3-VL-2B / VQAv274.1577.79+3.64
Qwen3-VL-2B / OK-VQA46.4552.07+5.63
AuthorsOpenReview ↗

Response to Reviewer 2 (2/3)

2. Direct comparison with Hyper-ICL (W2/Q2)?

Honestly, we ignored Hyper-ICL from the original experiments because we could not find the publicly released code at the time of submission (either until now). Without access to the standard implementation, we could not directly and fairly evaluate it under our experimental settings across all benchmarks.

Here we reproduced the Hyper-ICL method from its paper, including its query-adaptive logit adapter and hyperbolic anchor loss. We report two versions of our reimplementation, one using the published adapter settings and one adjusted to have approximately the same number of trainable parameters as ICL-R. As shown in Table R2.2, ICL-R outperforms the Hyper-ICL reimplementation on all six benchmarks when both methods use a total rank of 16 and have approximately equal numbers of trainable parameters. Specifically, ICL-R improves accuracy over Hyper-ICL by 2.93 percentage points on VQAv2, 2.67 points on OK-VQA, 1.97 points on MME and 12.32 points on SEED and improves CIDEr by 0.0796 on COCO and 0.0178 on flickr30k.

Table R2.2.

MethodOK-VQAVQAv2COCOMMESEEDFlickr30k
Hyper-ICL (reimplemented, rank=4)59.5970.331.243785.3050.870.9072
Hyper-ICL (reimplemented, rank=16)59.8270.451.296184.8651.630.8804
Hyper-ICL (reported originally)62.2471.171.3566N/AN/AN/A
ICL-R (Ours)62.49 (+4.46%)73.38 (+4.16%)1.3757 (+6.14%)86.83 (+2.32%)63.95 (+23.86%)0.8982 (+2.02%)
Δ vs. reimplemented Hyper-ICL (rank=16)+2.67+2.93+0.0796+1.97+12.32+0.0178

Percentages in parentheses are relative gains over the rank-16 Hyper-ICL reimplementation.

AuthorsOpenReview ↗

Response to Reviewer 2 (3/3)

3. Separating state-pair supervision from the geometry of RST (W3/W4)?

Thank you for highlighting this distinction. We apologize that our original ablations did not clearly separate the effect of supervising both attention endpoints from the effect of the geometry used in RST. As you said, wherever the spherical logarithm is well defined,

z=Exp⁡x(Log⁡x(z)).z=\operatorname{Exp}_x(\operatorname{Log}_x(z)).

Thus, (x,z)(x,z) and (x,Log⁡x(z))(x,\operatorname{Log}_x(z)) encode the same normalized endpoint information. RST changes how this information enters the loss by comparing transitions relative to their starting states and using parallel transport to align their tangent spaces.

To examine these effects, we compare five alternative auxiliary objectives while keeping the ICL-R architecture and training/evaluation protocol fixed. Following the notation in Sec. 3.3 in our paper, let hiah_i^a denote the pre-attention residual state, oiao_i^a the update after WoW_o, and z~ia=hia+oia\tilde{z}_i^a=h_i^a+o_i^a the unnormalized post-attention state, where a∈S,Ta\in\\{S,T\\} identifies the student or teacher. With P(r)=r/∥r∥2P(r)=r/\lVert r\rVert_2, the normalized endpoints and logarithmic transition are:

xia=P(hia),zia=P(z~ia),via=Log⁡xia(zia).x_i^a = P(h_i^a),\qquad z_i^a = P(\tilde{z}_i^a),\qquad v_i^a = \operatorname{Log}_{x_i^a}(z_i^a).

For the hyperbolic loss, Φ=Exp⁡o∘LN⁡\Phi=\operatorname{Exp}_o\circ\operatorname{LN} and κ=−0.1\kappa=-0.1. The last row compares the logarithmic transitions directly in the ambient Euclidean space.

ObjectiveLoss
Hyperbolic endpointsℓhyp=12[dH2(Φ(hiS),Φ(hiT))+dH2(Φ(z~iS),Φ(z~iT))]\displaystyle \ell_{\mathrm{hyp}} = \frac12 \left[d_{\mathbb H}^2(\Phi(h_i^S),\Phi(h_i^T)) + d_{\mathbb H}^2(\Phi(\tilde{z}_i^S),\Phi(\tilde{z}_i^T))\right]
Raw endpointsℓL2-pair=12[∥hiS−hiT∥22+∥z~iS−z~iT∥22]\displaystyle \ell_{\mathrm{L2\text{-}pair}} = \frac12 \left[\lVert h_i^S-h_i^T\rVert_2^2 + \lVert\tilde{z}_i^S-\tilde{z}_i^T\rVert_2^2\right]
Unit-sphere endpointsℓsph=12[∥xiS−xiT∥22+∥ziS−ziT∥22]\displaystyle \ell_{\mathrm{sph}} = \frac12 \left[\lVert x_i^S-x_i^T\rVert_2^2 + \lVert z_i^S-z_i^T\rVert_2^2\right]
State and updateℓL2-update=12[∥hiS−hiT∥22+∥oiS−oiT∥22]\displaystyle \ell_{\mathrm{L2\text{-}update}} = \frac12 \left[\lVert h_i^S-h_i^T\rVert_2^2 + \lVert o_i^S-o_i^T\rVert_2^2\right]
Without parallel transportℓnoPT=dS2(xiS,xiT)+∥viS−viT∥22\displaystyle \ell_{\mathrm{noPT}} = d_{\mathbb S}^2(x_i^S,x_i^T) + \lVert v_i^S-v_i^T\rVert_2^2

We evaluate these objectives with Idefics2 on OK-VQA and COCO. As shown in Table R2.3, RST achieves 62.49/1.3757 on OK-VQA/COCO, compared with 61.64/1.3707 for normalized endpoint matching and 61.86/1.3742 for Euclidean state-update matching. Since both controls already supervise both sides of attention, we consider these results support an additional benefit from RST’s formulation beyond paired-state supervision alone.

Furthermore, removing parallel transport gives 62.03/1.3708. Restoring it improves performance by 0.46 percentage points and 0.0049 CIDEr while retaining the same starting-state and logarithmic-transition information. This supports the contribution of comparing transitions in a common tangent space and clarifies that RST’s advantage concerns how the available information is compared rather than access to additional endpoint information.

We have added these loss comparisons to Table 3 and expanded the discussion of RST in our revised manuscript. Table R2.3.

DatasetHyperbolic endpointEndpoint L2Spherical endpoint L2State–update L2 lossWithout PTICL-R
OK-VQA59.62 (+2.87)61.91 (+0.58)61.64 (+0.85)61.86 (+0.63)62.03 (+0.46)62.49
COCO1.2377 (+0.1380)1.3578 (+0.0179)1.3707 (+0.0050)1.3742 (+0.0015)1.3708 (+0.0049)1.3757
Reviewer_4tjYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a new method named ICL-R, which is used to reproduce the effects of multimodal in-context demonstrations without input them at inference time. The authors study how demonstrations influence attention outputs. In order to complete the studies, they divide the research into four subparts: changes in attention weights, value representations, their interaction, and the direct contribution of demonstration tokens. Based on this analysis, ICL-R trains their own new model, during the training progress, they freeze the original model and leave several small modules to update parameters. Especially they use a Riemannian state transition loss to help the model follow the teacher’s hidden states and the variation of the states. Next, they test their model on two multimodal models and six benchmarks. The results show that their model outperforms the compared explicit and implicit ICL baselines in most settings. One thing deserve attention is that they keep inference costs close to zero-shot inference. Finally, they make some ablation studies to examine the exact contributions of the proposed components

Critical strengths

The paper provides solid evidence for the value calibration component. The authors compare two versions of the same total rank budget. One only changes attention weights and the other one changes both attention weights and values. Both these two methods bring the improvement about the accuracy by 5.62 percentage points on OK-VQA and 4.44 percentage points on VQAv2. This result strongly supports the design. It shows that the improvement does not come from the simple change of the capacity of the model. The author also make other informative component ablations, including comparisons of CE, L2, Point, Fiber, and the full RST objective, which help demonstrate the individual design choices.

Critical weaknesses
  1. The experimental comparison does not include Hyper-ICL, because it is a closely related method. This method already combines query-adaptive low-rank attention calibration with geometric teacher–student distillation. It seems that this paper use the similar workflow with the exited method. So the authors should include the similar method in the ablation experiments to clarify the difference to make their own contribution clearer.
  2. The evidence for missing-mass estimation and RST is less sufficient. Their reported gains are relatively small, so the authors should repeat the experiments with different random seeds to remove the influence coming from the noise. Table 2 tests several loss functions. But it does not compare RST with Hyper-ICL’s hyperbolic alignment loss. It also does not test a simpler option: directly matching the states before and after attention. These comparisons should use the same model architecture.
Questions for authors
  1. The ablation studies show a clear benefit from value calibration. However, without a direct comparison with Hyper-ICL, it is difficult to assess the gains over this closely related method. Could the authors provide some experiments on Hyper-ICL under the same configuration? Also, could they report results across multiple random seeds, with standard deviations or confidence intervals, to clarify that the smaller gains come from missing-mass estimation and RST but not caused by nosie?

  2. HiFICL already includes a query-dependent scaling factor for the attention mass assigned to query tokens. Could the authors explain how missing-mass estimator differs from the existing methods? The authors should also make it clear that what is the difference between the known mechanism in attention normalization and your method .

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/2)

Thank you for your careful review and constructive feedback. We have conducted additional experiments and clarified our explanations to address the concerns you raised.

1. Direct Comparison with Hyper-ICL (W1/Q1)?

Thank you for highlighting the close relationship with Hyper-ICL and the need for a direct comparison to clarify our contribution. At submission, we could not locate publicly released code for Hyper-ICL, and our renewed checks of the paper and the authors’ webpages while preparing this response still yielded no public implementation link. To address your concern, we have independently reproduced Hyper-ICL based on the paper, including both its query-adaptive logit adapter and hyperbolic anchor loss, and added a comparison with the complete method under our experimental settings. We evaluate a rank-4 configuration with 1.181M parameters and a rank-16 configuration with 4.326M parameters, with the latter having a parameter count comparable to that of ICL-R.

As shown in Table R1.1, ICL-R outperforms the Hyper-ICL reimplementation on all six benchmarks when both methods use a total rank of 16 and have approximately equal numbers of trainable parameters. Specifically, ICL-R improves accuracy over Hyper-ICL by 2.93 percentage points on VQAv2, 2.67 points on OK-VQA, 1.97 points on MME and 12.32 points on SEED and improves CIDEr by 0.0796 on COCO and 0.0178 on flickr30k.

Table R1.1

DatasetHyper-ICL (reported originally)Hyper-ICL (reproduced, rank=4)Hyper-ICL (reproduced, rank=16)ICL-R (ranks A and V = 8)Δ vs. reproduced Hyper-ICL (total rank=16)
OK-VQA62.2459.5959.8262.49+2.67 (+4.46%)
VQAv271.1770.3370.4573.38+2.93 (+4.16%)
COCO1.35661.24371.29611.3757+0.0796 (+6.14%)
MMEN/A85.3084.8686.83+1.97 (+2.32%)
SEEDN/A50.8751.6363.95+12.32 (+23.86%)
Flickr30kN/A0.90720.88040.8982+0.0178 (+2.02%)

2. Reliability of the Gains (W2/Q1)?

We evaluated four IDEFICS-9B variants with the same three random seeds. The CE-only baseline uses neither component, the two intermediate variants add missing-mass estimation or RST individually, and full ICL-R uses both. Each run uses 1,000 training examples and a randomly sampled subset of 1,024 evaluation examples.

Table R1.2. Accuracy (%) by seed and mean ± sample SD across seeds. Values in parentheses are the variant's score minus the full method's score for the same seed, in percentage points.

DatasetSeedCE OnlyCE + Missing MassCE + RSTFull ICL-R
OK-VQA340942.5781 (−1.7774)43.1055 (−1.2500)42.9492 (−1.4063)44.3555
OK-VQA341041.9922 (−1.0742)42.1289 (−0.9375)42.4609 (−0.6055)43.0664
OK-VQA341146.7383 (−3.2031)48.1641 (−1.7773)48.2617 (−1.6797)49.9414
OK-VQAMean ± SD43.77 ± 2.5944.47 ± 3.2444.56 ± 3.2245.79 ± 3.65
VQAv2340959.3652 (−1.1621)59.9609 (−0.5664)60.1855 (−0.3418)60.5273
VQAv2341057.8027 (−1.8262)58.5938 (−1.0351)59.4434 (−0.1855)59.6289
VQAv2341150.8594 (−5.0976)52.3828 (−3.5742)53.8672 (−2.0898)55.9570
VQAv2Mean ± SD56.01 ± 4.5356.98 ± 4.0457.83 ± 3.4558.70 ± 2.42

Full ICL-R scored above all three controls for every tested seed on both datasets. Across the three seeds, the mean gains over CE only, CE + Missing Mass, and CE + RST were 2.02, 1.32, and 1.23 percentage points on OK-VQA and 2.70, 1.73, and 0.87 on VQAv2.

AuthorsOpenReview ↗

Response to Reviewer 1 (2/2)

3. Comparison with Hyperbolic Alignment and Direct State Matching (W2)?

Thank you for suggesting these comparisons. We have added controlled experiments comparing RST with Hyper-ICL-style hyperbolic alignment and direct matching of pre- and post-attention states. All variants use the same Idefics2 backbone, ICL-R student architecture, demonstration-conditioned teacher, CE supervision, and training/evaluation protocol. We replace only the auxiliary alignment objective, allowing us to examine the loss formulation within the same architecture.

Following the notation in Sec. 3.3, let hiah_i^a denote the pre-attention residual state and z~ia=hia+oia\tilde{z}_i^a=h_i^a+o_i^a the post-attention state, where a∈S,Ta\in\\{S,T\\} identifies the student or teacher. We compare the following three objectives.

For the hyperbolic loss, Φ=Exp⁡o∘LN⁡\Phi=\operatorname{Exp}_o\circ\operatorname{LN} and κ=−0.1\kappa=-0.1. For the unit-sphere loss, P(r)=r/∥r∥2P(r)=r/\lVert r\rVert_2, xia=P(hia)x_i^a=P(h_i^a), and zia=P(z~ia)z_i^a=P(\tilde{z}_i^a).

ObjectiveLoss
Hyperbolic endpointsℓhyp=12[dH2(Φ(hiS),Φ(hiT))+dH2(Φ(z~iS),Φ(z~iT))]\displaystyle \ell_{\mathrm{hyp}} = \frac12 \left[d_{\mathbb H}^2(\Phi(h_i^S),\Phi(h_i^T)) + d_{\mathbb H}^2(\Phi(\tilde{z}_i^S),\Phi(\tilde{z}_i^T))\right]
Raw endpointsℓL2-pair=12[∥hiS−hiT∥22+∥z~iS−z~iT∥22]\displaystyle \ell_{\mathrm{L2\text{-}pair}} = \frac12 \left[\lVert h_i^S-h_i^T\rVert_2^2 + \lVert\tilde{z}_i^S-\tilde{z}_i^T\rVert_2^2\right]
Unit-sphere endpointsℓsph=12[∥xiS−xiT∥22+∥ziS−ziT∥22]\displaystyle \ell_{\mathrm{sph}} = \frac12 \left[\lVert x_i^S-x_i^T\rVert_2^2 + \lVert z_i^S-z_i^T\rVert_2^2\right]

As shown in Table R1.3, RST outperforms hyperbolic endpoint alignment on both benchmarks. It also improves upon direct endpoint matching, including when both endpoints are normalized. Since these alternatives already supervise the states on both sides of attention using the same student architecture and teacher targets, the results support an additional benefit from RST’s transition-based formulation beyond paired-state supervision alone. We have included these comparisons in Table 3 and expanded the corresponding discussion in the revised manuscript. Table R1.3

Auxiliary objectiveOK-VQACOCO
Hyperbolic endpoint alignment59.62 (+2.87)1.2377 (+0.1380)
Direct endpoint L2 matching61.91 (+0.58)1.3578 (+0.0179)
Normalized endpoint L2 matching61.64 (+0.85)1.3707 (+0.0050)
ICL-R62.491.3757

4. Relation to the Scaling Factor in HiFICL (Q2)?

We recognize HiFICL’s scaling factor as a way to account for the attention mass that goes to demonstrations. Its core parameterization consists of learnable virtual key–value pairs, with the scaling factor derived from the normalization terms associated with the original and virtual keys. Our ICL-R separately calibrates attention logits and value content without introducing virtual KV pairs, so HiFICL’s virtual-key-based scaling computation is not directly available in our architecture. We therefore introduce the missing-mass estimator to provide the corresponding mass correction within our calibration framework, serving as the same function through a different parameterization.

5. Difference from Standard Attention Normalization (Q2)?

This distinction is formalized in Section 3.2, Eqs. (5)–(6), and Appendix A.1, Step 4, Eqs. (18)–(21) in our paper. In our notation, mtTm_t^T denotes the teacher’s attention mass assigned to demonstrations, so the query-side weights sum to 1−mtT1-m_t^T. Specifically, Eq. (19) defines the normalized query-side distribution by dividing the teacher’s query-side weights by 1−mtT1-m_t^T. Equation (21) then shows that its contribution to the full teacher output includes this mass factor multiplying the normalized aggregation. Directly renormalizing the remaining weights after removing demonstrations produces a unit-sum distribution and discards this factor.

Our formulation retains standard softmax and adds a learned mass correction. Writing Eq. (6) using the notation of Appendix A.11:

A~t,:=softmax⁡(S0+ΔSA)t,:,∑sA~t,s=1.\tilde{A}_{t,:} = \operatorname{softmax}(S_0+\Delta S^A)_{t,:},\qquad \sum_s \tilde{A}_{t,s}=1.

A^t,s=(1−mt)A~t,s,∑sA^t,s=1−mt.\hat{A}_{t,s}=(1-m_t)\tilde{A}_{t,s},\qquad \sum_s\hat{A}_{t,s}=1-m_t.

Direct normalization alone uses A~\tilde{A}, assigning all mass to query-side tokens. The missing-mass estimator instead provides a learned coefficient 1−mt1-m_t to adjust their total contribution while preserving the relative weights established by softmax.

Dear Reviewer 4tjY,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Dear Reviewer 4tjY,

Thank you for your constructive feedback. We summarize the key evidence addressing your comments below.

  • W1. Against our comparably sized rank-16 Hyper-ICL reimplementation, ICL-R improves OK-VQA, VQAv2, and SEED by 2.67, 2.93, and 12.32 pp, respectively, with gains across all six benchmarks (Table R1.1).
  • W2. Under the same student architecture, RST achieves 62.49 on OK-VQA, compared with 59.62 for hyperbolic alignment, 61.91 for raw endpoint matching, and 61.64 for normalized endpoint matching (Table R1.3).
  • Q1. Across three seeds on 1,024-example evaluation subsets, full ICL-R exceeds both single-component controls by 1.23–1.32 pp on OK-VQA and 0.87–1.73 pp on VQAv2, with individual results and mean ± SD reported in Table R1.2.
  • Q2. Our estimator serves HiFICL’s mass-correction role through a separate learned coefficient without virtual KV pairs, preserving softmax while accounting for the mass factor discarded by renormalization.

Thank you again for your time and helpful suggestions.

Best regards,
The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Additional Evidence in Our Rebuttal

To make our rebuttal easier to navigate, we summarize the additional experiments and clarifications provided in our November 15 responses, with references to the corresponding tables.

1. Direct comparison with Hyper-ICL — Table R2.2

We report comparisons with our Hyper-ICL reimplementation across six benchmarks, including a rank-16 configuration with a parameter count comparable to ICL-R. ICL-R outperforms this configuration on all six benchmarks, with gains of 2.67 and 2.93 percentage points on OK-VQA and VQAv2, respectively.

2. Contribution of demonstration-conditioned supervision — Tables R2.1.1 and R3.2

With the student architecture and training protocol held fixed within each comparison, demonstration-conditioned teacher targets outperform query-only teacher targets across all six evaluated settings covering three backbones and two tasks. RST-only training, which removes the student CE term, also outperforms CE-only training in four of these six settings. These results support the utility of demonstration-conditioned supervision beyond direct student CE training. The additional experiments use 32-shot training and 1,024-example evaluation subsets.

3. Contribution of the RST formulation — Table R2.3

Using the same student architecture, we compare RST with hyperbolic endpoint alignment, raw and normalized pre-/post-attention state matching, Euclidean state–update matching, and a variant without parallel transport. RST achieves numerically higher scores on both OK-VQA and COCO than these controls. These comparisons help distinguish the contribution of the loss formulation from that of supervising both attention endpoints. We also clarify that RST changes how endpoint information is compared, rather than introducing additional endpoint information.

4. Three-seed component ablations — Table R1.2

We evaluate CE-only, CE + missing-mass estimation, CE + RST, and full ICL-R using three random seeds on IDEFICS-9B for OK-VQA and VQAv2. We report individual results and mean ± standard deviation. Full ICL-R scores above all three controls for every tested seed on both datasets. These runs use 1,000 training examples and 1,024-example evaluation subsets.

We hope this summary helps clarify how the additional evidence addresses the main concerns raised in the reviews. We welcome any remaining questions and would be happy to provide further clarification.

Dear Reviewers,

With approximately one hour remaining in the discussion period, we would sincerely appreciate any brief feedback on our rebuttal and the additional experiments.

Could you please let us know whether any of your main concerns remain unresolved? Even a short indication would be very helpful, as we would greatly value the opportunity to clarify these points before the discussion closes. We remain available to respond promptly.

We understand that this is a busy period and truly appreciate your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.