acceptodds
Under review as a conference paper at ICLR 2027

You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinement

33 views

Abstract

In-context learning (ICL) is crucial for boosting the inference performance of large language models (LLMs). However, the effectiveness of ICL in LLMs is greatly influenced by the choice of demonstration sets. Exhaustive searches over these sets are combinatorial, and existing selectors often rely on relevance or likelihood proxies to implicitly assess ICL quality. Making repeated queries to the target LLM with these strategies can incur substantial costs. This work simplifies selection by framing it as a constrained local search problem and presents local demonstration editing (LDE). Starting with an initially retrieved set of demonstrations, LDE employs a single structured edit to explore its surrounding neighborhood while balancing performance gains with search costs. Technically, LDE is reduced to a policy search problem, for which we train a small LLM, referred to as Jev-LDE. This model as the System-1 modifies the retrieved demonstration set by performing actions such as Keep, Delete, or Replace elements, all within a framework of reinforcement learning with verifiable rewards. At test time, Jev-LDE executes a single edit of the retrieved demonstration set, followed by one inference from the target LLM, avoiding the need for iterative context scoring or subset searches. Across standard classification benchmarks, various target LLMs with Jev-LDE as the plug-and-play module consistently improve ICL performance, and Jev-LDE shows transferability to held-out benchmarks and models without retraining. These findings indicate that the LDE approach offers an efficient and adaptable method for harnessing the ICL capabilities of target LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
In-Context Learning, Demonstration Selection, Local Demonstration Editing

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Jf7ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Local Demonstration Editing (LDE), a framework that refines an initially retrieved in-context demonstration set using a single structured edit. The proposed editor, Jev-LDE, is initialized from Qwen3-1.7B and trained with GRPO using binary correctness feedback from a frozen Meta-Llama-3-8B target model. Given a query, selected demonstrations, and replacement candidates, Jev-LDE chooses to keep the set, delete one demonstration, or replace one demonstration. At inference time, the editor makes one deterministic decision before a single target-model call, avoiding iterative subset evaluation. Experiments show strong improvements over Semantic TopK and several demonstration-selection baselines on classification tasks, with transfer to held-out datasets, target models, and pre-selectors. However, the gains are less consistent on mathematical reasoning tasks.

Critical strengths

1.The paper presents a simple and practically motivated formulation that converts combinatorial demonstration selection into a constrained one-edit refinement problem. 2.Jev-LDE requires only one editor call and one target-model inference at test time, avoiding the repeated target-model evaluations used by model-aware subset-search methods. 3.The classification results are strong: Jev-LDE improves its Semantic TopK initialization in 46 of 48 main configurations and achieves the best or tied-best result in 44.

Critical weaknesses

1.The main gains are concentrated on classification tasks. Improvements on GSM8K are small, while performance decreases on the held-out AQuA-RAT benchmark, limiting evidence that the method generalizes to reasoning-intensive ICL. 2.The theoretical analysis relies on simplified assumptions, including binary context-invariant judgments, majority voting, and a finite policy class. Its connection to the actual neural editor, multiclass tasks, and GRPO optimization is limited. 3.The comparison mixes training-free selectors with a learned editor that has absorbed feedback from multiple labeled benchmarks. The results therefore do not isolate whether the gains come from the one-edit formulation, additional supervised feedback, or the capacity of the trained 1.7B editor.

Questions for authors

Why does Jev-LDE provide little improvement on GSM8K and reduce accuracy on AQuA-RAT? Is this caused by weak candidate retrieval, inappropriate editing actions, or limited reasoning ability of the editor?

Reviewer_C3DnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LDE (Local Demonstration Editing), a method for improving in-context learning by editing demonstration sets. The authors frame demonstration selection as a constrained local search problem and train a small "System-1" model, Jev-LDE, with reinforcement learning and verifiable rewards. Starting from an initially retrieved set, Jev-LDE performs a single structured edit, Keep, Delete, or Replace, followed by one inference from the target LLM, avoiding iterative context scoring or subset search.

Across standard classification benchmarks, Jev-LDE consistently improves ICL performance for various target LLMs and transfers to held-out benchmarks and models without retraining.

Critical strengths

The method is effective. Combined with reinforcement learning, LDE consistently surpasses common existing demonstration-selection methods across multiple datasets.

The learned selector, Jev-LDE, transfers across different tasks without retraining, demonstrating practical potential as a plug-and-play module.

Critical weaknesses

Computational cost for large candidate sets. Although LDE reduces selection to a single pass over the candidate set, it still requires iterating over the entire set, which is more expensive than retrieval-based alternatives, especially when the candidate set is large. The training cost on large candidate sets may also be substantial. The authors should discuss this efficiency tradeoff and clarify when LDE remains preferable to simpler retrieval-based baselines.

Limitations of the candidate-size experiments. The reported results on candidate number have two weaknesses. First, the TREC 1-shot results may not represent model behavior on other dataset and under other settings. Second, performance degrades when the candidate set grows from 16 to 32, suggesting that LDE may be unstable on larger sets. The authors should analyze the cause of this decline and provide additional evidence that the method generalizes reliably as the candidate set grows.

Justification of the single-edit design. The choice of performing only one structured edit is central to the claimed efficiency, but it is not clear how much performance is sacrificed by this constraint. It would strengthen the paper to compare against a few additional edits or to report the impact of the number of allowed edits.

Questions for authors

Please see the weaknesses sections above.

Reviewer_amM8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates few-shot in-context learning (ICL) from the lens of Local Demonstration Editing (LDE), framing sample selection as a constrained local search problem around an initially retrieved set.

Rather than evaluating all combinatorial subsets or executing multiple test-time target LLM queries, LDE executes a single atomic membership edit: Keep, Delete, or Replace within a local candidate neighborhood. Technically, the authors train a lightweight policy model (Qwen3-1.7B, termed Jev-LDE) via group relative policy optimization (GRPO) using binary correctness rewards obtained from a frozen target LLM (Meta-Llama-3-8B-Instruct). At test time, Jev-LDE takes the query, initial demonstration set, and a small candidate pool (e.g., top-16), outputs a single JSON edit action deterministically, and passes the refined prompt to the target LLM for a single downstream forward pass.

Empirically, the method is evaluated on classification benchmarks (TREC, DBPedia, Banking77, SST-2, TACRED, TREC-50) and reasoning benchmarks (GSM8K, AQUA-RAT) across target models (Qwen2.5-7B, Qwen3-8B, Qwen3.5-9B, Meta-Llama-3-8B, DeepSeek-V4-Flash). The results demonstrate consistent gains on classification (e.g., 1-shot accuracy rising from 81.2% to 88.1% over Semantic TopK) with low wall-clock overhead (~11%), transferring zero-shot across unseen target models and task domains, while showing limited efficacy on multi-step reasoning tasks.

Critical strengths
  1. Practical, accurate and efficient
  2. Technique generalizes to diverse model families
Critical weaknesses
  1. Limited efficacy on complex (multi-step) reasoning tasks
Questions for authors
  1. Retrieved Sample Ordering: Assuming retrieved samples are ordered by retrieval score, did you examine whether joint local editing of permutation/ordering alongside edits/changes provide further improvements?
Reviewer_QVTYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Local Demonstration Editing (LDE), a lightweight approach to optimize In-Context Learning (ICL) prompts. Instead of searching for the best demonstration subset globally, authors train a small "System-1" model (Jev-LDE, 1.7B parameters) to make a single atomic edit (Keep, Delete, or Replace) to an initially retrieved demonstration set. Jev-LDE is trained using GRPO based on binary correctness feedback from a frozen target model (Llama-3-8B). At inference time, it requires zero search overhead. The approach yields strong empirical gains on classification benchmarks (+6.9 points in average 1-shot accuracy) and demonstrates zero-shot transferability across unseen target models (e.g., DeepSeek-V4-Flash) and datasets.

Critical strengths
  • The paper constraints the action space to a one-edit neighborhood. This approach bypasses the combinatorial explosion of traditional subset selection while still capturing the vast majority of performance gains. The wall-time comparison (Figure 8) validates this efficiency.

  • Jev-LDE improves ICL performance on unseen target models (including frontier models like DeepSeek-V4-Flash) and unseen tasks (TACRED, SST-2) without retraining which is impressive. It suggests the editor learns generalized heuristics for semantic relevance and label boundaries rather than simply overfitting to the training target's quirks.

  • Propositions 1 and 2 justify the method well. Proposition 1 supports the empirical observation that a single replacement is often sufficient to repair a flawed retrieval, while Proposition 2 links the restricted action space to sample efficiency during RL training.

  • The ablation showing that Jev-LDE improves upon BM25, CASE, and TopK+ConE (Figure 6) proves it can function as a versatile, plug-and-play module in existing pipelines.

Critical weaknesses
  • Jev-LDE is bounded by the quality of the initial retriever as it only edits within the top-16 candidates provided by the pre-selector. The paper touches on this by expanding to n=32 (which degraded performance due to a larger action space), but it does not deeply analyze how retrieval quality ceilings limit LDE.

  • The paper reports that Jev-LDE provides negligible or negative benefits on complex reasoning tasks (GSM8K, AQuA-RAT). While the authors hypothesize that compute scaling is more important than context shots for reasoning, this significantly limits the scope of the paper's claims, confining the primary benefits to classification tasks.

  • The verifiable reward mechanism relies on exact-match or simple parsing (e.g., extracting a final label or math answer). It is unclear how this framework would adapt to open-ended generation tasks such as summarization, translation, code generation where binary correctness feedback is unavailable.

Questions for authors
  • On AQuA-RAT, Jev-LDE actively degrades performance (Table 5). Does the 1.7B editor delete longer CoT examples or replace them with shorter ones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.