- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates context-authority discrimination in LLMs. The authors observe that conventional ICL fine-tuning can increase models’ tendency to follow misleading contextual rules. It introduces FakeContext-Bench, containing 3,500 instances across seven domains. Meanwhile, it also proposes Jurisdiction In-Context Learning (J-ICL), a post-training framework that jointly optimizes rule induction and contextual applicability judgment. The authors evaluate J-ICL across four model settings using ICLEval and FakeContext-Bench.
Critical strengths
[S1] The paper highlights an important issue: improving ICL ability can make models more susceptible to misleading demonstrations.
[S2] The paper introduces FakeContext-Bench, a benchmark for evaluating whether LLMs can resist coherent but misleading in-context demonstrations.
Critical weaknesses
[W1] J-ICL primarily applies standard SFT to the Jurisdiction-Contrastive Data, offering limited algorithmic novelty.
[W2] The baseline comparison includes only MetaICL (2022) and Symbol Tuning (2023).
Questions for authors
[Q1] The paper should clarify that its primary contribution is a benchmark rather than the training methodology. J-ICL largely relies on standard SFT with positive and negative pairs.
[Q2] The baseline comparison includes only MetaICL (2022) and Symbol Tuning (2023). Please include more recent and stronger baselines, such as methods for demonstration selection and reranking.
[Q3] The decomposition of ICL into task recognition and task learning [1] could help clarify the proposed task. Please discuss how context-authority discrimination relates to these two components.
[Q4] The paper contains some errors in its mathematical notation, such as the indicator function in Equation (3).
[Q5] Can the proposed approach be applied when model fine-tuning is unavailable, such as when using proprietary models through APIs?
[Q6] The text and number in the figures are too small.
[Q7] If the authors do not release their data, it will be difficult to verify and reproduce the reported results.
[1] What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning.
4 comments
The core contributions are the benchmark and the models. Please release them as supplementary material; otherwise, I cannot maintain my score. Also, the baselines you compared are outdated.
Thank you for your helpful suggestions. To facilitate inspection and reproducibility, we have included FakeContext-Bench and its scoring script in the supplementary materials. As the model checkpoints and training data exceed the supplementary submission size limit, we provide them through an anonymous repository, with its link included in the supplementary materials.
For newer baselines, we evaluated within-case adaptations of PDDS, from Performance-Driven Demonstration Selection for In-Context Learning (Wang et al., IJCAI 2026), and DisJEV, from Can Jev Improve Demonstration Retrieval for In-Context Learning? (Song et al., 2026). Both methods select two demonstrations on FakeContext-Bench. On ICLEval, they retain three quarters of the available demonstrations, rounded up to the nearest integer, with copying tasks unchanged.Cluster-based Search, from Rapid Selection and Ordering of In-Context Demonstrations via Prompt Embedding Clustering (Pham et al., ICLR 2025), reorders all demonstrations using next-token entropy.
| Method | Evaluation | Reality ↑ | Context ↓ | Reality Rate ↑ | ICLEval ↑ |
|---|---|---|---|---|---|
| Qwen3-8B (Base) | demos in original order | 23.06% | 60.74% | 27.51% | 69.66% |
| J-ICL | demos in original order | 28.86% | 56.14% | 33.95% | 76.03% |
| PDDS | Within-case selection and reranking | 29.00% | 57.17% | 33.65% | 62.70% |
| DisJEV | Within-case selection and reranking | 28.29% | 57.17% | 33.10% | 63.58% |
| Cluster-based Search | Within-case reranking | 25.34% | 60.57% | 29.50% | 66.08% |
All three baselines improve Reality Rate over Qwen3-8B Base, but their ICLEval accuracy drops. PDDS achieves slightly higher Reality accuracy than J-ICL (29.00% vs. 28.86%), at the cost of a substantial 6.96-percentage-point drop in ICLEval relative to Base (69.66% to 62.70%). J-ICL is the only method in this comparison that improves both Reality Rate and ICLEval over Base.
J-ICL fine-tunes the LLM on downstream data, so it is unfair to compare it with these training-free baselines. Moreover, training the LLM defeats the very purpose of ICL, whose core advantage is being training-free. Besides, as the authors themselves state, the core contribution is only a dataset, and the algorithm directly uses standard SFT, so I see no algorithmic contribution. I therefore keep my score.
Thank you for recognizing the importance of misleading demonstrations and the value of FakeContext-Bench. We respond to each question below.
Q1. Contribution and algorithmic novelty.
Our main contribution is to systematically benchmark an underexplored failure mode in ICL: applying coherent contextual rules beyond their legitimate scope. J-ICL complements this benchmark with a simple, targeted data-construction and sampling recipe that mitigates this failure while improving conventional ICL performance. We will clarify this emphasis and better articulate the contributions of both the benchmark and the training approach.
Q2. More recent selection and reranking baselines.
We have added within-case adaptations of ByCS (Bayesian Example Selection Improves In-Context Learning for Speech, Text and Visual Modalities, EMNLP 2024) and ConE (Revisiting Demonstration Selection Strategies in In-Context Learning, ACL 2024). Both use the same unadapted Qwen3-8B model and select only from each case's supplied demonstrations, rather than an external retrieval pool.
For select & rerank, we select two demonstrations on FakeContext-Bench, retaining the first two for cases without explicit demonstration answers. On ICLEval, we retain k = ⌈3N/4⌉ of the N demonstrations and leave copying tasks unchanged. Rerank only preserves all demonstrations and changes only their order. Generation settings and scoring remain fixed within each benchmark.
Under our within-case adaptations, both ByCS and ConE improve Reality Rate over the reported Base result, but neither yields higher ICLEval accuracy, even when reranking retains all demonstrations. One possible explanation is the restricted candidate pool: these methods can only select or reorder the demonstrations already supplied in each case, rather than retrieve more suitable examples from a large external pool. J-ICL tuning and demonstration selection/reranking are also compatible: the former trains the model to assess contextual applicability, while the latter organizes the context presented at inference time. Their combination is a promising direction for future work.
Q3. Relation to task recognition and task learning.
Thank you for pointing us to What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning. We will discuss this distinction in Related Work. Context-authority discrimination concerns whether a recognized or newly learned rule applies to the current query. It complements both components: a model can identify or learn the demonstrated mapping correctly yet apply it outside its scope.
Q4. Mathematical notation.
We will correct the indicator notation in Equation (3) and check all equations for consistency.
Q5. API-only models.
J-ICL requires fine-tuning and cannot be directly applied when an API provides no fine-tuning access. FakeContext-Bench can still evaluate such models through their generated responses. We will clarify this distinction.
Q6. Figure readability.
We will enlarge text, legends, and numerical annotations, adjust crowded layouts, and verify that all figures are readable at normal viewing size.
Q7. Data release.
We will publicly release data used in this work to support verification and reproducibility, and make this commitment explicit in the paper.