- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies ICL beyond pretraining support, focusing on unsupported instructions and unsupported target tasks. It develops a latent-task analysis for fixed versus varying instruction shifts and for executing unseen tasks through supported subtasks, supported by controlled experiments and evaluations with pretrained LLMs.
Critical strengths
- The paper makes a clear distinction between instruction-level support and task-level support.
- The taxonomy of prompt structures is useful.
- The theoretical presentation is careful, especially in keeping the reference predictor and the actual pretrained model separate. (However, some of the repeated caveats and defensive clarifications slightly dilute the presentation and make the main contributions less direct.)
- The experiments cover both controlled settings and pretrained LLMs, and several experiments are designed to directly probe assumptions used in the theory.
Critical weaknesses
- I am not fully convinced by the strength of the novelty beyond the ICL-specific formulation. Several core mechanisms seem closely related to classical posterior concentration, distribution shift, and compositional learning. Is the main contribution therefore better viewed as a unification and reformulation of these ideas in the ICL setting, rather than a fundamentally new generalization principle?
- The strongest guarantees are established for a constructed reference predictor, while the actual-model bound includes a model–reference transport term. Since analyticity only guarantees that this term is finite, but not necessarily small, how informative is the guarantee for real pretrained LLMs? (I would consider increasing my score if the authors can directly quantify the model–reference gap on a real pretrained LLM, ideally along the continuous instruction paths already used in the Qwen3 experiments. In particular, showing that this gap remains small and does not grow with the number of demonstrations would substantially strengthen the connection between the reference-model guarantee and actual LLM behavior.)
- The literal zero-support assumption cannot be verified for real pretrained models. The experiments therefore test proxies such as consistent versus varying mismatch rather than the exact theoretical regime. How much empirical evidence do these proxy settings provide for the stronger ‘out-of-support’ claim?
- The task-composition analysis assumes that a valid decomposition into supported subtasks is already available, while plan discovery is explicitly left outside the scope. Besides, the empirical evaluation is largely arithmetic. How far should we generalize these results to realistic long-horizon agent tasks?
Questions for authors
1.For the shared unsupported-instruction case, the paper notes that exponential convergence already follows if the off-support endpoint contributes a finite -independent perturbation. What essential result is gained from the analytic-continuation construction beyond this simpler assumption?
2. Can the authors provide stronger quantitative evidence that the model–reference transport error remains small for actual pretrained LLMs?
3. Since literal zero support is generally unobservable in pretrained LLMs, would the practical claim be more appropriately framed in terms of ‘effectively unsupported’ or extremely rare instruction–task pairings?
4. Can the authors clarify how sensitive the conclusions are to the shared-template factorization, where repeated copies of the same instruction are treated as a single task-conditional draw?
5. If the valid decomposition must itself be discovered by the model, what parts of the current guarantee would still remain applicable, and what additional assumptions would be required?