- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ICL-R, which jointly calibrates attention and value representations and uses a Riemannian state transition loss to transfer the behavior of a demonstration-conditioned teacher to a query-only student, reducing demonstration-related overhead at inference time.
Critical strengths
-
Jointly calibrating attention and value representations is well motivated, and the ablation results support the effectiveness of value calibration.
-
The experiments cover multiple models and six benchmarks. The appendix further provides cross-dataset transfer results, GPU memory usage, and error analyses, offering a fairly comprehensive evaluation.
Critical weaknesses
-
Hyper-ICL already employs query-adaptive low-rank attention calibration and geometric distillation, making it closely related to this work. Although the paper cites this method, it does not include it in the experimental comparisons, making it difficult to assess the incremental benefits of the proposed value calibration and RST loss over this direct predecessor.
-
In Table 2, the cross-entropy-only variant already achieves 61.64 and 73.20 on OK-VQA and VQAv2, respectively, while the full model improves these scores to 62.49 and 73.38. Does this suggest that most of the performance gains can be obtained without supervision from a demonstration-conditioned teacher?
-
The current experiments do not directly compare against the simpler training objective of jointly aligning normalized pre-attention and post-attention states. Consequently, it remains unclear whether the gains come from additional state supervision or from spherical geometry and parallel transport themselves.
Questions for authors
Weaknesses
4 comments
Response to Reviewer 3 (2/2)
3. Joint Alignment of Normalized Pre- and Post-Attention States (W3)?
Thank you for pointing out the comparison of joint alignment of pre-attention and post-attention states to better support the validity of our design. We have added this comparison together with four additional auxiliary losses to distinguish the contribution of supervising both sides of attention from the effects of the alignment geometry and parallel transport.
Following Sec. 3.3 of our paper, denotes the pre-attention residual state, the update after , and the unnormalized post-attention state, where identifies the student or teacher. For the unit-sphere projection , we define
For the hyperbolic loss, and . The last row compares the logarithmic transitions directly in the ambient Euclidean space.
| Objective | Loss |
|---|---|
| Hyperbolic endpoints | |
| Raw endpoints | |
| Unit-sphere endpoints | |
| State and update | |
| Without parallel transport |
We evaluate these objectives with Idefics2 on OK-VQA and COCO. Table R3.3 reports 62.49/1.3757 on OK-VQA/COCO for RST, while the requested normalized endpoint objective obtains 61.64/1.3707 and Euclidean state–update matching obtains 61.86/1.3742. Since these alternatives already constrain both sides of attention, the results support a contribution from RST’s loss formulation when paired-state supervision is already included.
The no-PT control obtains 62.03/1.3708. Full RST improves these scores by 0.46 percentage points and 0.0049 CIDEr without changing the starting-state or logarithmic-transition information being supervised. These comparisons indicate that parallel transport provides an additional gain while the supervised state information remains unchanged.
You can also refer to the results in our response to your second comment, where comparisons with CE-only training and normalized state matching provide further evidence for the contribution of our RST on different models and settings. We have added these comparisons to Table 3 and expanded the discussion of RST in the revised manuscript.
Table R3.3
| Dataset | Hyperbolic endpoint | Endpoint L2 | Spherical endpoint L2 | State–update L2 loss | Without PT | ICL-R |
|---|---|---|---|---|---|---|
| OK-VQA | 59.62 (+2.87) | 61.91 (+0.58) | 61.64 (+0.85) | 61.86 (+0.63) | 62.03 (+0.46) | 62.49 |
| COCO | 1.2377 (+0.1380) | 1.3578 (+0.0179) | 1.3707 (+0.0050) | 1.3742 (+0.0015) | 1.3708 (+0.0049) | 1.3757 |
Dear Reviewer QxAE,
Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
We sincerely look forward to your feedback.
Dear Reviewer QxAE,
Thank you for your constructive feedback. We summarize the key findings addressing your three concerns below.
- W1. ICL-R outperforms our comparably sized Hyper-ICL reimplementation on all six benchmarks, including gains of 2.93 pp on VQAv2 and 12.32 pp on SEED (Table R3.1).
- W2. ICL-R outperforms both CE-only and query-only teacher supervision in all six additional 32-shot settings, gaining 12.85 pp over CE-only on IDEFICS-9B/VQAv2(Table R3.2).
- W3. RST achieves the best results among all tested loss variants on both OK-VQA and COCO under the same student architecture, including the normalized endpoint and no-parallel-transport controls (Table R3.3).
Thank you again for your time and helpful suggestions.
Best regards,
The Authors
Response to Reviewer 3 (1/2)
Thank you for recognizing our motivation behind joint attention–value calibration and the breadth of our experimental evaluation. We sincerely apologize for the confusion caused by the insufficiently detailed explanations in the original manuscript. Here we provide additional experiments and clarifications.
1. Comparison with Hyper-ICL (W1)?
At the time of submission, we could not locate publicly released code for Hyper-ICL. We still could not find an available public implementation from the authors when preparing this response, which prevented us from directly evaluating their implementation under our experimental settings across all benchmarks. To provide this comparison, we independently reimplemented the query-adaptive logit adapter and hyperbolic anchor loss based on the paper. We report both a rank-4 configuration and a rank-16 configuration with a parameter count comparable to that of ICL-R. We also checked the paper and the authors’ webpages but found no link to a public implementation.
As shown in Table R3.1, both methods use a total rank of 16 (r=16 for Hyper-ICL and rA=rV=8 for ICL-R) and have approximately equal numbers of trainable parameters. In this comparison, ICL-R improves VQAv2, OK-VQA, MME, and SEED by 2.93, 2.67, 1.97, and 12.32 percentage points, respectively, and improves COCO CIDEr by 0.0796 and Flickr30k by 0.0178. These comparisons assess the complete method against prior work.
Table R3.1
Percentages in parentheses are relative gains over the rank-16 Hyper-ICL reimplementation.
2. Contribution of Demonstration-Conditioned Teacher Supervision (W2)?
We truly understand your concern about our results in Table 2: CE-only already achieves strong performance on Idefics2 with 8-shot training, reflecting the effectiveness of our calibration modules. Under this configuration, the additional benefit from RST is relatively limited. However, this observation cannot be directly generalized to other models or the number of training shots. Below, we report results evaluated on a subset of 1,024 examples on VQAv2 and OK-VQA under 32-shot training with different models.
As shown in Table R3.2, CE + demonstration-conditioned RST outperforms CE-only in all six settings, with gains ranging from 0.41 to 12.85 percentage points. It also outperforms CE + query-only teacher RST in all six settings, supporting an additional contribution from demonstration-conditioned targets.
RST-only training exceeds CE-only in four of the six settings, showing that reconstructing teacher computations provides useful supervision even without the student CE term. Furthermore, demonstration-conditioned RST outperforms CE + normalized state matching in all six settings, indicating that the choice of alignment objective also matters. Together, we consider these comparisons to support the utility of both demonstration-conditioned distillation and our RST design.
Table R3.2
Δ is the gain of full ICL-R over CE only, in percentage points.