acceptodds
Under review as a conference paper at ICLR 2027

In-Context Learning via Attention Patching

Abstract

In-context learning (ICL) enables language models to adapt to new tasks from demonstrations, but repeatedly including multiple examples increases context length and inference cost. Many existing implicit ICL methods encode demonstrations as fixed vectors, limiting adaptation of the intervention to individual queries and token positions. We propose Attention-Patching, an implicit ICL method that models contextual changes in self-attention outputs through task-specific updates to attention output projections. Using query-answer pairs processed with and without a fixed demonstration set, Attention-Patching fits layer-wise affine transformations from non-contextual attention representations to the demonstration-induced output changes. Across 12 classification benchmarks and four model families, Attention-Patching achieves higher average accuracy than the evaluated implicit ICL baselines while maintaining performance comparable to explicit ICL at near zero-shot inference cost. Further experiments on diverse generation tasks demonstrate its applicability to autoregressive multi-token generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-Context Learning, Implicit In-Context Learning, Efficient Inference, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fcbMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Attention Patching, which basically learns affine corrections to the attention output projections using paired prompts with and without demonstrations. These corrections depend on the current token representations. The corrections are then merged into the model weights, so the model does not need demonstration tokens during inference. The authors test the method on 12 classification benchmarks across different model families and also evaluate it on autoregressive generation. The ablation studies examine different calibration sizes and show that the weight correction is more important than the fixed bias.

Critical strengths
  • I think the merge operation in section 3.2 is a practical strength. The learned correction becomes part of an existing projection. This gives a clear explanation for the low online overhead without adding a separate intervention module.
  • Table 3 shows the weight correction recovering nearly all the full patch’s average accuracy. The bias correction alone performs worse. I find this comparison useful because it isolates the contribution of input dependent transformations.
  • I appreciate that Table 2 extends the evaluation to autoregressive generation. On GSM8K, the patch improves over inference without demonstrations and generally outperforms ICV. This provides evidence that the method remains useful beyond classification with a fixed label set.
Critical weaknesses
  • The fitting order in section 3.2 is unclear for me. The description does not specify whether later layers receive activations from previously patched layers. I think pseudocode and a comparison of fitting orders would clarify how calibration produces the deployed model.
  • Table 2 evaluates translation only through ROUGE L. Textual overlap does not fully establish whether the meaning is preserved. I think adding a translation quality measure alongside concrete error examples would make the generation claim more convincing.
  • I noticed that Figure 2 reports online costs without comparing against reuse of a cached demonstration prefix. Calibration cost is also omitted. I think including cached explicit ICL and preparation costs over repeated queries would clarify the practical efficiency benefit.
Questions for authors
  • Could you clarify the calibration procedure in section 3.2? Are later layer activations recomputed after earlier patches are installed, and are their regression targets updated too? I think more clear explanations or pseudocode showing these dependencies would make the method easier to reproduce.
  • Could you examine translation examples where the patch and explicit context produce different outputs in Table 2? Comparing whether each output preserves the source meaning would help identify information lost by the patch. I think this would complement the overlap metric.
  • When does patching become faster overall than explicit ICL with a cached demonstration prefix? Could you include calibration time in the comparison associated with Figure 2?
5 comments

We thank the reviewer for the constructive feedback and for recognizing the practical benefits of weight merging and the value of our ablation studies.

[W1/Q1. Calibration Procedure.]

Thank you for the feedback. AP adopts a sequential fitting procedure. Each fitted patch remains active when fitting subsequent layers. Thus, later layers receive activations that reflect the effects of previously patched layers. We added pseudocode in Appendix A.3 to clarify the layer-wise calibration process and improve reproducibility.

[W2/Q2. Quantitative and Qualitative Evaluation of Translation.]

We added COMET, a neural metric for translation quality, to Table 2 to complement lexical overlap. AP also generally outperforms the baselines on these additional metrics.

We further provide translation examples from the Korean and Spanish subsets of FLORES using Llama-3.2-3B-Instruct, selected based on ROUGE-L F1 scores to illustrate cases where AP outperforms or underperforms many-shot ICL.

Cases where many-shot ICL outperforms AP

FLORES(Ko)FLORES(Sp)
SourceThe accused are named as Baba Kanjar, Bhutha Kanjar, Rampro Kanjar, Gaza Kanjar and Vishnu Kanjar.The United States Strategic Command of the U.S. Department of Defense office is tracking the debris.
Reference피의자 이름은 바바 간자(Baba Kanjar), 부다 간자(Bhutha Kanjar), 람프로 간자(Rampro Kanjar), 가자 간자(Gaza Kanjar), 비쉬누 간자(Vishnu Kanjar)입니다.El Comando Estratégico de EE. UU., que depende de la oficina del Departamento de Defensa, realiza un rastreo de los restos.
Many-shot ICL기소된 사람들은 바바 카نج라(Baba Kanjar), 부타 카نج라(Bhutha Kanjar), 라mpr로 카نج라(Rampro Kanjar), 가자 카نج라(Gaza Kanjar), 비슈누 카نج라(Vishnu Kanjar)라고 불리며, 5명이 기소되었다. (90.9)El Comando Estratégico de Estados Unidos de la oficina del Departamento de Defensa de EE. UU. está rastreando el desecho. (60.5)
AP (Ours)기소된 사람들은 바바 카نج라, 부타 카نج라, 라mpr로 카نج라, 가자 카نج라, 비슈누 카نج라로 명명되었다. (0.0)La Comandancia Estratégica de los Estados Unidos del Departamento de Defensa de los Estados Unidos está siguiendo el desecho. (38.1)

Cases where AP outperforms many-shot ICL

FLORES(Ko)FLORES(Sp)
SourcePolice said they suspect an alleged Daesh (ISIL) militant of responsibility for the attack.Safaris are perhaps the greatest tourism draw in Africa and the highlight for many visitors.
Reference경찰은 공격에 대한 책임이 있다고 주장되는 다에시(ISIL) 무장세력에게 혐의를 두고 있다고 밝혔습니다.Es probable que los safaris sean el mayor atractivo turístico de África y la parte más memorable para muchos visitantes.
Many-shot ICL경찰은 이 공격의 책임자가 다시(다시스)라고 불리는 이슬람 국가의 이슬람 국가(다시스) militant이라고 의심하고 있다. (0.0)Los safaris son, sin duda, el gran atractivo del turismo en África y el destaque para muchos visitantes. (45.0)
AP (Ours)경찰은 이 공격에 대한 책임이 alleged Daesh (ISIL) militant인 것으로 추정하고 있다. (40.0)Los safaris son tal vez el mayor atractivo turístico de África y el destaque para muchos visitantes. (65.0)

[W3/Q3. Efficiency Comparison with Cached Explicit ICL.]

We updated Figure 2 (Section 5.1) to include explicit ICL with reusable prefix KV caches. AP maintains near-zero-shot TTFT across context lengths, whereas cached explicit ICL exhibits increasing TTFT, reaching approximately 5× the zero-shot latency at 16K.

We further conducted a break-even analysis accounting for AP's offline calibration cost (activation collection and ridge regression) and cached ICL's prefix preparation cost.

Context Budget1K2K4K8K16K
Break-even Queries1,42261338224.79.0

When activation collection is excluded, the ridge regression step alone incurs negligible cost, making AP advantageous from the first query across all evaluated context budgets. When accounting for the full offline calibration cost, AP becomes faster overall after approximately 25 queries at 8K and 9 queries at 16K.

These results highlight both AP's low parameter-fitting overhead and its amortized efficiency benefits for repeated inference with long demonstration contexts.

Thank you for your reply. Most of my concerns have been addressed. I have two remaining questions. How much does sequential fitting contribute to the reported gains compared with fitting each layer independently? I also share the concern about the labeled data budget. How does Attention Patching compare with explicit ICL when both methods have access to the same total number of labeled examples, including those used for calibration?

Thank you for these questions.

1. Sequential vs. Parallel Fitting

We compared AP's sequential fitting procedure with a parallel fitting variant in terms of average classification accuracy. For Llama, there was no difference between the two fitting procedures, while Qwen showed a minor difference. Sequential fitting was slightly more stable, but the performance difference was negligible.

MethodLlama-3.2-3B-InstructQwen3.5-0.8B
AP (Sequential)80.770.4
AP (Parallel)80.770.1

2. Matched Labeled-Data Budget ICL

We conducted additional explicit ICL experiments on Llama-3.2-3B-Instruct, matching the total number of labeled samples used by AP. We evaluated 10-shot per class (matching AP with 5 calibration samples per class in Appendix D.1, Figure 8) and 30-shot per class (matching AP with 25 calibration samples per class in the main paper). We report average classification accuracy.

MethodAvg. Accuracy (%)
AP (5)79.7
ICL (10-shot)80.2
AP (25)80.7
ICL (30-shot)81.2

The results show that the performance differences between AP and explicit ICL under matched labeled-data budgets are comparable.

Reviewer_12ChOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Attention-Patching (AP), which captures demonstration-induced changes in attention outputs through paired forward passes and learns input-dependent affine transformations via ridge regression. The learned patches are then merged into the attention output projection weights, enabling inference without demonstrations.

Critical strengths
  • Relatively novel method design. AP introduces input-dependent transformations to capture demonstration effects, improving flexibility over traditional fixed-vector interventions while maintaining inference efficiency.
Critical weaknesses

W1: Unclear applicability and limitations. Table 1 shows that AP is not the best-performing method on most individual tasks, with gains varying across models and tasks. The paper provides limited analysis of these variations, making it difficult to identify the conditions under which AP is most effective.

W2: Insufficient evaluation of adaptation efficiency. Although AP avoids backpropagation, it still requires paired forward passes, activation collection, and layer-wise ridge regression. The paper lacks comparisons with LoRA in terms of total adaptation time and memory usage, as well as inference efficiency against merged LoRA, leaving its practical efficiency advantages unclear.

W3: Incomplete comparisons with recent baselines. The paper lacks direct comparisons with recent methods such as DyVec, SITE, and ICR, making it difficult to assess AP's advantages over the latest related approaches.

W4: Insufficient control of supervision budgets. AP uses more labeled examples than explicit ICL. Without comparisons under matched total supervision budgets, it remains unclear whether the observed improvements stem from the proposed method or the additional labeled data.

Questions for authors

Please address the concerns raised in the Weaknesses.

1 comment

We thank the reviewer for the constructive feedback and for recognizing the novelty and inference efficiency of our method.

[W1. Unclear Applicability and Limitations.]

We have added a correlation analysis in Section 5.5 (Figure 5) to further investigate the performance variations across tasks and models. We compare the performance changes induced by explicit ICL and AP relative to zero-shot inference for each task and model.

We observe strong positive correlations across all five models (Pearson's r=0.850r = 0.850–0.9980.998), indicating that AP generally follows the performance trends of explicit ICL. Tasks where explicit ICL provides larger improvements tend to show larger gains with AP, while tasks where explicit ICL provides limited benefits tend to show smaller gains. These results suggest that AP's effectiveness is closely related to the task-specific benefits of explicit ICL.

[W2. Insufficient Evaluation of Adaptation Efficiency.]

Thank you for this suggestion. We conducted additional experiments comparing AP with LoRA, SITE, and LTV in terms of offline adaptation time, peak GPU memory usage, and average classification accuracy on Llama-3.2-3B-Instruct.

MetricAP (Ours)LoRASITELTV
Peak Memory (GiB)11.514.37.56.2
Calibration / Training (s)¹11.0 ± 6.72.6 ± 0.912.0 ± 6.86.5 ± 0.0
Ridge Solve (s)0.2 ± 0.1——0.1 ± 0.0
Coefficient Training (s)——5.0 ± 1.0—
Samples / Class²3025305
Avg. Accuracy (%)80.777.177.976.6

¹ Calibration/training time includes ridge regression for AP and LTV, and coefficient training for SITE.

Although one-epoch LoRA requires less adaptation time, it achieves lower average accuracy than AP (77.1 vs. 80.7). In additional experiments, increasing LoRA training to three epochs improves its accuracy to 80.2, approaching AP's 80.7, but also increases training cost. Thus, the adaptation-time gap narrows when comparing configurations with similar accuracy, although activation collection remains a non-negligible cost for AP.

Once activations are collected, AP requires only 0.2 ± 0.1 seconds for closed-form ridge regression. AP also uses less peak memory than LoRA (11.5 vs. 14.3 GiB), while SITE requires comparable calibration time but achieves lower accuracy.

For inference efficiency, AP and merged LoRA exhibit comparable FLOPs and TTFT, as shown in the updated Figure 2 (Section 5.1).

[W3. Incomplete Comparisons with Recent Baselines.]

Thank you for this suggestion. We have expanded our comparisons by adding SITE (2026) and LTV (2026), both of which have publicly available implementations, to the main classification results (Table 1) and generation experiments (Table 2), with additional results in Appendices C.1 and C.2 (Tables 5 and 6). Across all five models, AP improves average classification accuracy over SITE by 1.0–2.8 pp and over LTV by 1.4–6.1 pp.

We also considered DyVec and ICR, but their official implementations are not publicly available, making reliable reproduction difficult.

Notably, while SITE involves gradient-based optimization, AP achieves competitive performance through closed-form regression without backpropagation.

[W4. Insufficient Control of Supervision Budgets.]

We would like to clarify that the primary goal of implicit ICL is to reduce the inference overhead of explicit ICL by constructing reusable task representations from demonstrations. Our work follows this direction, aiming to improve the performance of existing implicit ICL methods while maintaining efficient, demonstration-free inference. Since these methods differ in how they construct task representations, we followed their original experimental settings and supervision budgets as closely as possible. For example, I2CL uses 5 demonstrations per class, whereas FV uses 75 10-shot prompts, resulting in 750 demonstration samples per task.

Importantly, Appendix D.1 already shows that AP outperforms the evaluated implicit ICL baselines even with only five calibration examples per class, in addition to five demonstrations per class. This suggests that AP's improvements are not solely attributable to using a larger calibration set.

Furthermore, our updated Figure 2 (Section 5.1) demonstrates that AP maintains near-zero-shot TTFT even compared with explicit ICL using reusable prefix KV caches. Our additional break-even analysis, which accounts for offline calibration and cache preparation costs, shows that AP becomes faster overall after approximately 25 queries at 8K and 9 queries at 16K.

Reviewer_31ZbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Attention-Patching, which uses closed-form ridge regression to learn attention patches and merges them into model weights to compress the effects of in-context demonstrations. Experiments cover classification and generation tasks, showing competitive classification performance with per-query inference costs close to zero-shot inference.

Critical strengths
  1. The method is simple and uses closed-form fitting. Merging the patches into model weights avoids additional matrix multiplications during inference.

  2. The evaluation covers four model families, including a 27B model, and several generation tasks. The component, patch-rank, and calibration-size ablations are reasonably comprehensive.

Critical weaknesses
  1. Insufficient comparisons with trainable methods. AP learns dense patches, while conventional fine-tuning is represented only by rank-8 LoRA trained for one epoch. More relevant trainable baselines with matched labeled-data and tuning budgets are needed.

  2. Limited comparisons with 2026 methods. Only TP is included among the baselines cited as 2026 work. Comparisons with relevant methods such as SITE (2026) and ICR (2026) should be added.

  3. Incomplete generation baselines in Table 2. ICV is the only implicit ICL baseline evaluated. Additional generation-capable methods, such as SITE, and key ablations on generation tasks are needed.

  4. Limited applicability to closed-source models. AP requires access to internal activations and modification of model weights, preventing direct application to commercial models available only through black-box APIs. This deployment limitation should be discussed.

  5. Incomplete efficiency analysis. The analysis focuses on per-query inference and omits offline fitting and tuning costs, as well as comparison with explicit ICL using reusable prefix KV caches.

  6. Degradation on some generation tasks. Table 6 reports GSM8K scores of 44.4–48.4 for AP on Gemma-2-2B, below the zero-shot score of 56.0. The causes of this degradation need further investigation.

  7. Insufficient evidence of stability. The main classification results report five-run averages without uncertainty estimates. This is particularly relevant to the 0.3-percentage-point improvement on Qwen.

Questions for authors

See weakness

4 comments

We thank the reviewer for the constructive feedback and for recognizing the simplicity and inference efficiency of our method, as well as the breadth of our evaluation.

[W1. Insufficient comparisons with trainable methods.]

Thank you for raising this concern. We would like to clarify that AP is primarily designed as an implicit ICL method, aiming to capture demonstration-induced effects while avoiding the repeated inference overhead of explicit ICL. Therefore, implicit ICL methods are the most directly relevant baselines for evaluating our primary research objective. We also provide an analysis under different calibration budgets in Appendix D.1, showing that AP remains effective with as few as five calibration examples per class.

Nevertheless, to strengthen the comparison with trainable methods, we conducted additional LoRA experiments on Llama-3.2-3B-Instruct using the same labeled-data budget as AP, varying the rank (8 and 16), target modules (O and Q/V projections), and training epochs (1–3).

RankTarget Modules1 Epoch2 Epochs3 Epochs
8O77.178.980.2
8Q, V76.378.979.5
16O76.878.780.1
16Q, V75.978.879.5

The best LoRA configuration achieves 80.2% average accuracy, compared with 80.7% for AP. This demonstrates that AP remains competitive with four LoRA variants trained for 3 epochs, while avoiding backpropagation.

Furthermore, we added SITE, a gradient-based implicit ICL method, to Table 1. AP outperforms SITE in average classification accuracy by 2.8 pp, 2.2 pp, and 1.0 pp on Llama-3.2-3B-Instruct, Gemma-2-27B-it, and Qwen3.5-0.8B, respectively.

These results support AP's effectiveness within the implicit ICL setting and its competitiveness with gradient-based approaches.

[W2/W3. Limited Baseline Comparisons.]

Thank you for this suggestion. We have expanded our comparisons by adding SITE (2026) and LTV (2026), both of which have publicly available implementations, to the main classification results (Table 1) and generation experiments (Table 2), with additional results in Appendices C.1 and C.2 (Tables 5 and 6). Across all five models, AP improves average classification accuracy over SITE by 1.0–2.8 pp and over LTV by 1.4–6.1 pp.

We also considered ICR, but its official implementation is not publicly available, preventing reliable reproduction.

Notably, while SITE involves gradient-based optimization, AP achieves competitive performance through closed-form regression without backpropagation.

[W4. Limited Applicability to Closed-Source Models]

We acknowledge that AP requires access to internal activations and model weights, limiting its applicability to closed-source models accessible only through black-box APIs. We have explicitly addressed this limitation in Section 7 of the revised paper.

Nevertheless, AP enables backpropagation-free adaptation by incorporating demonstration-induced effects directly into attention projections, without requiring additional inference-time modules.

[W5. Incomplete efficiency analysis.]

We updated Figure 2 to include explicit ICL with reusable prefix KV caches and conducted additional measurements of calibration time, peak GPU memory usage, and amortized inference efficiency.

1. Offline adaptation cost.

We measured the offline calibration/training costs of AP, LoRA, SITE, and LTV on Llama-3.2-3B-Instruct. We also report their average classification accuracy across 12 benchmarks for comparison. All measurements were conducted on a server with two AMD EPYC 7763 CPUs, 1 TiB of RAM, and three NVIDIA RTX 6000 Ada Generation GPUs.

MetricAP (Ours)LoRASITELTV
Peak Memory (GiB)11.514.37.56.2
Calibration / Training (s)¹11.0 ± 6.72.6 ± 0.912.0 ± 6.86.5 ± 0.0
Ridge Solve (s)0.2 ± 0.1——0.1 ± 0.0
Coefficient Training (s)——5.0 ± 1.0—
Samples / Class²3025305
Avg. Accuracy (%)80.777.177.976.6

¹ Calibration/training time includes ridge regression for AP and LTV, and coefficient training for SITE.

² AP and SITE use 5 labeled demonstrations and 25 additional labeled calibration samples per class. Since LoRA is not an implicit ICL method, we only use 25 training samples per class. LTV uses 5 labeled demonstrations and 25 additional unlabeled calibration samples per class.

Once activations are collected, AP requires only 0.2 ± 0.1 seconds for closed-form ridge regression. Although activation collection introduces a non-negligible offline cost, the subsequent fitting step is computationally inexpensive. Compared with SITE, AP achieves higher average accuracy (80.7 vs. 77.9) with comparable offline adaptation time. LTV requires less calibration time but yields lower accuracy (76.6).

2. Comparison with cached explicit ICL.

We additionally evaluated explicit ICL with reusable prefix KV caches, accounting for cache preparation and repeated query inference. We updated Figure 2 (Section 5.1) to include this baseline and conducted a break-even analysis.

Context Budget1K2K4K8K16K
Break-even Queries1,42261338224.79.0

The results indicate that AP's offline calibration cost can be amortized over repeated inference, particularly for longer demonstration contexts. At 8K and 16K, AP reaches break-even after approximately 25 and 9 queries, respectively.

[W6. Degradation on Some Generation Tasks.]

Thank you for the valuable feedback. We identified a metric mismatch in GSM8K: strict match was used for patch strength selection on the validation set, whereas flexible match was used on the test set. We corrected this inconsistency and repeated the generation experiments over three independent runs to improve the reliability of our results. We will also release the revised code before the rebuttal period ends.

We updated Table 6 with the corrected results. On Gemma-2-2B GSM8K, AP now achieves 57.6, 57.4, and 56.0 at 1K, 2K, and 4K, respectively, compared with 55.2 for zero-shot. Thus, the previously reported degradation is no longer observed.

We also added a correlation analysis in Section 5.5 (Figure 5) to further investigate performance variations across tasks and models. We observe strong positive correlations between the performance changes induced by AP and explicit ICL relative to zero-shot inference across all five models (Pearson's r=0.850r = 0.850–0.9980.998), suggesting that AP generally follows the performance trends of explicit ICL.

[W7. Insufficient Evidence of Stability.]

We have updated Table 1 (main paper) and Table 5 (Appendix C.1) to include standard deviations alongside the previously reported means over five independent runs.

I thank the authors for their detailed response. The rebuttal addressed some of my concerns; however, after considering the other reviewers' feedback as well, I believe the current score is consistent with my overall assessment of the paper, and I therefore maintain my score.

We thank the reviewer for the feedback.

We appreciate the reviewer's acknowledgment that our rebuttal has addressed some of the concerns. In the revised paper, we have strengthened comparisons with recent implicit ICL methods and trainable baselines, expanded generation evaluations, and incorporated offline adaptation costs, cached explicit ICL comparisons, and uncertainty estimates.

We would like to emphasize that AP's primary contribution is improving the performance of implicit ICL through input-dependent attention transformations while maintaining efficient, demonstration-free inference.

We hope these additional analyses and revisions help clarify the scope and contributions of our work. We would appreciate any clarification on which concerns remain unresolved, as this would help us address them more directly.

Comments on the paper

1 comment

General Response

We thank the reviewers for the constructive feedback. The main concerns are baseline comparisons, adaptation efficiency, and performance variations.

1. Baseline Comparisons (Reviewer 1: W2, W3; Reviewer 2: W3)

We have expanded our comparisons by adding SITE (2026) and LTV (2026) to the main results (Tables 1 and 2), with additional results in Appendices C.1 and C.2 (Tables 5 and 6). Across all five models, AP improves average classification accuracy over SITE by 1.0–2.8 pp and over LTV by 1.4–6.1 pp.

2. Adaptation Efficiency (Reviewer 1: W5; Reviewer 2: W2; Reviewer 3: W3, Q3)

We measured offline adaptation costs and compared AP with explicit ICL using reusable prefix KV caches. AP achieves higher average classification accuracy than LoRA, SITE, and LTV (80.7% vs. 77.1%, 77.9%, and 76.6%), with 11.0 ± 6.7 seconds of calibration time. Once activations are collected, ridge regression takes only 0.2 ± 0.1 seconds. We updated Figure 2 to include cached explicit ICL. Our break-even analysis (Appendix C.3, Figure 7) shows that AP amortizes its offline calibration cost after approximately 25 and 9 queries at 8K and 16K context budgets, respectively.

3. Performance Variations (Reviewer 1: W6; Reviewer 2: W1)

We identified a metric mismatch in GSM8K: strict match was used for patch strength selection on the validation set, whereas flexible match was used for the test set. After correction, the degradation on Gemma-2-2B GSM8K is no longer observed (Table 6). We will release the corrected code before the rebuttal period ends. We also added a correlation analysis (Section 5.5, Figure 5), showing strong correlations between AP and explicit ICL performance changes across five models (Pearson's r=0.850r = 0.850–0.9980.998).

Overall, AP is an implicit ICL method that achieves competitive performance with supervised fine-tuning under comparable labeled-data budgets, while enabling backpropagation-free adaptation and efficient, demonstration-free inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.