- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Instant Context Optimization (ICO), which compiles in-context demonstrations into weights offline so the model can be served zero-shot. Each linear projection is reparameterized as , so only per-output-channel gains are trained (~1.4M parameters on Llama-3-8B). The authors report +11.25 points over base on a 9-task development suite (Llama-3-8B), +12.67 points macro over 38 tasks × 4 models (8B–27B), and serving-efficiency gains over ICL.
Critical strengths
The problem is practical and well motivated.
Critical weaknesses
The reported numbers are internally inconsistent, and some cannot be produced by the described procedure.
- AR (Eq. 8) either accepts the candidate or reverts to , so each "Gold SFT (+AR)" entry in Table 2 should match either the Gold SFT entry or the Base entry. 8 of 9 match neither. For example, mbpp is 39.80 (SFT) and 40.80 (base), yet 30.80 with AR.
- In Table 8, different models share identical results. Base GSM8K is 41.09 for three models, and the teacher GSM8K score is 53.68 for all four. The headline 107.8% headroom is computed against that shared teacher value.
Because of these inconsistencies, I cannot assess the empirical claims.
Questions for authors
-
Please explain why several base and teacher numbers in Table 8 are identical across different models.
-
Please report LoRA, (IA), and all-linear gain SFT trained with the same LOO rollout-distillation objective and the same rollback gate as ICO.
3 comments
I think these inconsistencies show the manuscript is under-polished and I recommend the authors to resubmit to other late venues.
Thank you for your suggestions. We appreciate that you have identified a critical bug in our paper and experiment post-processing.
Thank you for actionable suggestions.
1. Question 1: Identical Base and Teacher Numbers Across Models in v1 Table 8
As documented in Row 1 above, we used unsloth/gemma-* and google/gemma-* interchangeably across two machines, which caused the scripts to fall back to Llama-3-8B defaults (
41.09% / 53.68%on GSM8K,11.33%on MGSM,31.34% / 63.21%on BBH,69.05% / 68.62%on LongBench; note that on MBPP 0-shot,Gemma-2-9Bgenuinely scores204/500 = 40.80%, matchingLlama-3-8B, whereas its 3-shot Teacher is53.80%vs.49.80%). DividingGemma-2-9B's authentic54.66%GSM8K score byLlama-3-8B's53.68%Teacher produced the erroneous107.8%headroom. Below are the corrected GSM8K and 38-task macro numbers across the 4 core models (Table 3, Figure 3b, full 5-scenario × 7-model breakdown in Appendix Table 16):Base / TeacherICO(Headroom)Base / TeacherCompiled / ServedMeta-Llama-3-8B41.09% / 53.68%50.64%(75.9%)31.66% / 61.89%38.98% / 40.31%Gemma-2-9B21.38% / 68.61%54.66%(70.5%)38.19% / 67.78%41.64% / 47.37%Gemma-3-12B-PT40.26% / 74.07%59.74%(57.6%)40.66% / 67.31%47.92% / 50.23%Gemma-3-27B-PT63.00% / 80.82%71.57%(48.1%)48.66% / 74.74%53.02% / 55.04%2. Question 2:
LoRA,(IA)³, and All-Linear Gain SFT Under Identical LOO Rollout Distillation and Rollback GateWe trained
LoRA(r=8),(IA)^3, unregularizedAll-Linear Gain + LOO(τ0=0, no soft thresholding or TIES), and recent PEFT methods onMeta-Llama-3-8Bwith the same LOO rollout-distillation objective (S=8,diff_bot, λgold=2.0) and rollback gate (+AR) (Table 2, Figure 3a, scenario and per-task scores in Appendix Tables 11, 13):Meta-Llama-3-8B, Same LOO + AR)Compiled / +ARCompiled / +AR039.43%(Base) /62.26%(ICL)31.66%(Base) /61.89%(ICL)1.38M(2.75 MB)41.99% / 42.32%LoRA(r=8) + LOO + AR20.97M(41.94 MB)29.07% / 43.22%37.23% / 41.80%(IA)^3+ LOO + AR0.52M(1.05 MB)43.01% / 48.05%38.07% / 39.43%1.38M(2.75 MB)46.16% / 51.54%ICO(τ0=0.10 + TIES, Ours)1.38M(2.75 MB)45.86% / 50.68%38.98% / 40.31%Core Findings (
§ 4.2):LoRAand(IA)^3: Under identical LOO supervision and rollback gating, both 1D channel-gain variants (51.54%and50.68%) beatLoRA(43.22%) and(IA)^3(48.05%) on the 9-task suite, avoidingLoRA's severe raw collapse on few-shot math/code (29.07%Raw). Across all 38 tasks, ICO achieves the highest rawCompiledaccuracy (38.98%), whileLoRA + ARreaches41.80%(via rank-8 capacity on symbolic BBH tasks once collapsed runs revert toBase) at 15.2× larger adapter memory.+8.36%to+9.22%overDirect SFT 1D(41.99% -> 42.32%). When S=8 rollouts provide dense supervision, unregularized τ0=0 slightly increases multi-round plasticity (51.54%vs.50.68%), whereas τ0=0.10 curbs raw drift on boolean/cross-lingual tasks, improves S=0 compilation (+3.19%, Appendix Table 12), and confines updates to ≤10% of channels. Both are compared side-by-side in Table 2 and Appendix Table 13.