- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Local Demonstration Editing (LDE), a framework that refines an initially retrieved in-context demonstration set using a single structured edit. The proposed editor, Jev-LDE, is initialized from Qwen3-1.7B and trained with GRPO using binary correctness feedback from a frozen Meta-Llama-3-8B target model. Given a query, selected demonstrations, and replacement candidates, Jev-LDE chooses to keep the set, delete one demonstration, or replace one demonstration. At inference time, the editor makes one deterministic decision before a single target-model call, avoiding iterative subset evaluation. Experiments show strong improvements over Semantic TopK and several demonstration-selection baselines on classification tasks, with transfer to held-out datasets, target models, and pre-selectors. However, the gains are less consistent on mathematical reasoning tasks.
Critical strengths
1.The paper presents a simple and practically motivated formulation that converts combinatorial demonstration selection into a constrained one-edit refinement problem. 2.Jev-LDE requires only one editor call and one target-model inference at test time, avoiding the repeated target-model evaluations used by model-aware subset-search methods. 3.The classification results are strong: Jev-LDE improves its Semantic TopK initialization in 46 of 48 main configurations and achieves the best or tied-best result in 44.
Critical weaknesses
1.The main gains are concentrated on classification tasks. Improvements on GSM8K are small, while performance decreases on the held-out AQuA-RAT benchmark, limiting evidence that the method generalizes to reasoning-intensive ICL. 2.The theoretical analysis relies on simplified assumptions, including binary context-invariant judgments, majority voting, and a finite policy class. Its connection to the actual neural editor, multiclass tasks, and GRPO optimization is limited. 3.The comparison mixes training-free selectors with a learned editor that has absorbed feedback from multiple labeled benchmarks. The results therefore do not isolate whether the gains come from the one-edit formulation, additional supervised feedback, or the capacity of the trained 1.7B editor.
Questions for authors
Why does Jev-LDE provide little improvement on GSM8K and reduce accuracy on AQuA-RAT? Is this caused by weak candidate retrieval, inappropriate editing actions, or limited reasoning ability of the editor?