- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the visual few-shot adaptation from labeled support examples in language-model in-context learning. The authors propose CoPA to compress activations via a Perceiver Resampler and use an episodically meta-trained hypernetwork to generate low-rank updates for query inference. Experiments are conducted across 14 datasets and 4 backbones. The results demonstrate that CoPA outperforms frozen ICL and one-step test-time training.
Critical strengths
-
The authors honestly state that CoPA is not training-free and the end-to-end single-query latency is not better than ICL.
-
The experiments cover 14 datasets, 3 task families, and 4 backbones.
-
Section 4.5 and Appendix G provide useful information about transfer and negative transfer.
Critical weaknesses
-
There is no comparison against methods that use the same labeled training data and the same adaptation. This is necessary to check whether the Perceiver resampler plus hypernetwork is needed.
-
One-step TTT matches frozen ICL to two decimal places across nearly all Table 1 entries, suggesting negligible update effects. Since these baselines are neither tuned nor multi-step, they are weak proxies for test-time optimization, undermining the claim of outperforming TTT.
-
Efficiency claims are overstated. Although query-phase cost is constant in K, support processing is not, and single-query latency exceeds ICL at both K = 16 and 64. The key scenario of multiple queries sharing one support set is unbenchmarked, and the absence of accuracy results at K = 64 leaves the long-context claim unverified.
-
Experimental reporting limits confidence in the results. The main table lacks variance estimates and multiple seeds, and regression MAEs across different scales are averaged together. Cars’ zero-shot accuracy of 0.00 suggests a possible output-format failure that exaggerates the apparent benefit of supervision. Moreover, the authors acknowledge that the FFN down-projection is suboptimal (Appendix H), yet use it for the main results.
If the authors can solve my concerns during the rebuttal, I am happy to raise my scores.
Questions for authors
-
What are the results of a LoRA/adapter baseline trained on the same labeled episodes and evaluated at the same shot counts?
-
How do the Perceiver and hypernetwork components compare to simpler aggregation?
-
Do the conclusions hold across seeds and across the other three backbones on more than one dataset?
-
What is the amortized latency when many queries share one support set?
5 comments
Thank you for the response. I appreciate your agreement that the current comparisons do not disentangle architectural impact from supervision, as well as your plan to tone down and qualify the claims regarding TTT and efficiency.
However, most of my primary concerns have been acknowledged rather than resolved, as the rebuttal does not present empirical results for Q1, Q2, or Q3. Consequently, the actual necessity of the Perceiver and hypernetwork remains unverified. Furthermore, the clarification that TTT baselines rely on a single AdamW update at a learning rate of 1e-4 suggests their performance would naturally mirror frozen ICL by design, severely limiting their informative value as baselines. Because the core methodological questions remain unaddressed empirically, I am maintaining my score at 2.
Two additional concerns:
-
Regarding Cars, Appendix F notes that candidate categories are explicitly provided in the prompt. This detail makes an exact zero-shot accuracy of 0.00% difficult to justify, especially given that Food-101 achieves 35.1% under the identical prompt template. Could you provide sample raw predictions and evaluate performance using constrained label matching?
-
The amortization analysis in Q4 is benchmarked solely against uncached ICL and omits the kNN setting reported in Table 1, where support sets vary dynamically across individual queries.
Thank you for requesting the Cars diagnostic. Preliminary results show that our earlier assurance about the evaluation implementation was too strong.
On the same 100 Cars queries using Gemma 3 4B-it, with all 196 candidate categories provided:
| Protocol | Accuracy |
|---|---|
| Original free generation, 15-token cap and scorer | 0/100 |
| Same outputs, separator-normalized candidate parsing | 0/100 |
| Candidate-constrained greedy decoding | 53/100 |
Of the unrestricted outputs, 98 reached the generation limit without a recognized ending token. Examples include “Based on the image, the vehicle is a Ford Edge SUV (” for Ford_Edge_SUV_2012, and “Based on the image, the car is a Ferrari 458 for Ferrari_458_Italia_Coupe_2012.
Thus, the reported zero is highly sensitive to the output protocol and cannot establish absent recognition capability. Separator normalization alone does not resolve it. Constrained decoding also allows complete category names, so this comparison does not isolate decoding constraints from generation length. We retract our earlier vocabulary-based explanation and will reassess the zero-shot comparison. The 53/100 result is a subset diagnostic, not a replacement for Table 1.
Additional evaluation-seed experiments are running; we will report their results separately. These do not resolve the missing supervision-matched or component baselines.
We also agree that shared-support amortization does not apply when kNN supports change per query. We withdraw the crossover estimate as evidence for Table 1’s setting and claim no advantage over prefix-cached ICL.
Thank you for the constructive review and for recognizing our training assumptions, evaluation breadth, and transfer analysis. We address your four weaknesses and questions below.
W1:
We agree that the comparisons do not isolate architecture from supervision. CoPA receives labeled episodic meta-training; frozen ICL does not. Our goal is support-to-weight adaptation without gradient-based training at test time, not a training-free method. We will qualify the results and acknowledge that superiority over supervision-matched static LoRA remains unestablished.
The Perceiver provides fixed-length support representations and the hypernetwork generates low-rank updates, but their necessity is not isolated. A preliminary fast-weight implementation inspired by “Linear Transformers Are Secretly Fast Weight Programmers” performed poorly, motivating hypernetwork exploration inspired by Text-to-LoRA. This is design context, not a controlled ablation; attention visualizations likewise do not establish component superiority.
W2:
These baselines use one AdamW update with learning rate 10−4, matching CoPA’s adapter rank and site but not its supervision or optimization budget.
We will consistently say “the evaluated one-step TTT baselines,” document their settings, and remove broader superiority claims. Their near-equality with frozen ICL does not establish that tuned or multi-step TTT is ineffective.
W3:
Query-phase independence from support count does not imply constant end-to-end cost. At 16 supports, CoPA totals 1596 ms versus 972 ms for ICL; at 64 supports, 6882 versus 4542 ms, also with higher overall peak phase memory.
The demonstrated property is approximately support-count-independent query-phase cost after adaptation, not lower single-query latency or overall peak memory. Multi-query amortization is unbenchmarked; Q4 provides a conditional estimate. Predictive evaluation extends to 16 supports, while profiling extends to 64; we will not claim accuracy scaling beyond the evaluated range.
W4:
Evaluation uses seed 42 and up to 1,000 queries per dataset, reusing the query subset across shot counts. For each query and shot count, one random support set is sampled without replacement from the training split, without repeated draws. We will report valid-prediction counts, acknowledge missing across-seed uncertainty, and remove the cross-scale regression “Avg.” column.
We double-checked the Cars code and found no implementation error. Our interpretation is that producing highly specific category names without examples is difficult under exact matching; demonstrations clarify the label vocabulary. We will not interpret 0.00 as absent recognition capability or attribute the entire gain to improved recognition.
Down-projection is a fixed default, not the best tested site. Up-projection improves Age MAE from 5.21 to 5.04 and EmoSet accuracy from 76.4% to 78.1%. We will acknowledge this without assuming generalization to other tasks.
Q1:
We do not have this baseline. It would help separate supervised task specialization from support-conditioned generation; one-step TTT is not a substitute. We will acknowledge this unresolved comparison.
Q2:
We lack controlled comparisons against pooling or another aggregator with a comparable weight generator. The preliminary alternative under W1 does not replace these controls. Figure 5 will remain descriptive, not evidence that the Perceiver causes the gains.
Q3:
Single-seed evaluation does not establish across-seed robustness. We will report evaluation and training-seed counts and avoid treating small differences as statistically established.
The other three backbones are evaluated on VQAv2 only; the 14-dataset study primarily uses Gemma 3 4B-it. We will clarify that this is not a complete 14-dataset-by-four-backbone evaluation.
Q4:
For Q serial queries sharing reusable supports, modeled average CoPA cost is
T(Q,K)=SK/Q+CK,
using reported support- and query-phase times. At 16 supports this is 1480/Q+116 ms; at 64, 6765/Q+117 ms. Against repeated uncached ICL at 972 and 4542 ms per query, respectively, both yield a modeled crossover at two queries.
These are estimates, not throughput measurements. They assume adapter reuse, exclude batching and scheduling, and do not establish an advantage over prefix-cached ICL or apply directly when retrieval changes supports.
Thank you again for the helpful feedback. We hope these clarifications and revisions address your concerns and would appreciate reconsideration of the score if you find them sufficient.