acceptodds
Under review as a conference paper at ICLR 2027

Does the Label Space Need to Evolve? Bounding the Gains of Self-Improving Context Engineering for In-Context Classification

Abstract

Can a classifier improve by rewriting its label space during inference? We study this question with EvoLabel, which separates a prediction architecture from an optional evolution layer. The predictor compares two retrieval views over a fixed labeled pool, returns immediately when their canonical predictions agree, and invokes a configured arbiter otherwise. A coverage gate avoids the second view when leave-one-out retrieval is unreliable. The evolution layer aggregates prediction-pair feedback and may propose structured Merge Describe updates to the presented label state, without changing model weights or adding labeled examples. In a 24-method, 12-dataset, three-seed comparison, the adaptive retrieval–arbitration configuration reaches mean accuracy, versus for the strongest external baseline; on four many-class intent datasets the corresponding result is versus . The same configuration exceeds the frozen dual-arm baseline on three additional models by – percentage points. On the intent tasks, agreement covers of queries and identifies a substantially more accurate subset ( versus on disagreements), although this contrast is descriptive rather than a calibrated risk guarantee. Crucially, the evolution-enabled runs in the replication commit no label-space edits: the observed gains support the prediction architecture, not successful online evolution. A conditional decomposition expresses rewrite gain as affected-query mass times conditional accuracy change under an explicit outside-subset invariance assumption. Together with a frequency-matched operator audit, this explains what the null does and does not establish. The study provides an audited, trace-backed comparison and a bounded claim: complementary views and arbitration help on this suite, while unseen-dataset validation and more effective rewrite operators remain open.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, context engineering, label space, self-improving systems, demonstration selection, selective prediction, test-time adaptation, prompt optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qW2BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoLabel combines two retrieval views with agrement checks and arbitration for in-context classification. It also includes optional label-space edits, but no edits are committed in the reported runs. Experiments across 12 datasets show prediction gains, while the analysis separates these gains from label-space evolution.

Critical strengths

(1) The paper clearly reports that label edits never occur and avoids attributing prediction gains to an inactive component. This is a useful diagnostic result. (2) Comparisions across datasets and backbones support the prediction pipeline. Paired analyses provide evidence that its gains are not solely due to evaluation noise.

Critical weaknesses

(1) Routing and expert choices are selected using the evaluation dataset suite. Additional backbones reuse that suite, so performance on unseen datasets remains uncertain. (2) Zero committed edits demonstrate an activation failure, but provide little evidence about whether executed label changes help or hurt. The conditional analysis does not resolve this central empirical question..

Questions for authors

(1) Do the gains hold when routing choices are selected without access to the target dataset, such as through leave-one-dataset-out evaluation? (2) Can valid label edits be activated and compared with a matched frozen-label system, including errors introduced outside the targeted examples?

1 comment

Thank you for the detailed rebuttal and additional results. I will keep my current score.

Reviewer_97W4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether online label-space evolution improves in-context classification. It proposes EvoLabel, combining complementary retrieval views, disagreement-based arbitration, and optional label-space rewriting. Experiments on 12 datasets and four LLM backbones show improved classification accuracy. However, no label-space edits are committed during the reported evolution-enabled runs, so the observed improvements primarily support the retrieval and arbitration architecture.

Critical strengths
  1. The paper investigates an interesting and underexplored question concerning online label-space adaptation.
  2. The evaluation covers 12 datasets, multiple LLM backbones, and a broad set of comparison methods.
  3. The disagreement analysis provides useful insights into identifying difficult classification instances.
  4. The authors transparently report negative results and carefully distinguish architectural improvements from label-space evolution.
Critical weaknesses
  1. The central label-space evolution mechanism commits no edits, leaving its effectiveness unvalidated.
  2. The adaptive routing strategy is calibrated on the evaluation datasets, raising concerns about generalization to unseen tasks.
  3. The main performance gains appear to arise from combining existing retrieval and arbitration techniques, limiting methodological novelty.
  4. The evaluation lacks a controlled demonstration of successful label-space rewriting, and the theoretical gain decomposition relies on a restrictive invariance assumption.
Questions for authors
  1. Why did the evolution mechanism fail to commit any edits, and which conditions prevented its activation?
  2. How does the adaptive routing strategy perform when selected using separate validation datasets?
  3. Can the authors provide a controlled experiment with executed label-space edits to assess their actual benefits and risks?
Reviewer_dhWvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether an in-context classifier benefits from rewriting its label space during inference. To test this the authors build EvoLabel, which combines a two-view retrieval classifier with an optional label-rewriting layer. The two-view classifier improves accuracy by about 1.5 to 2 points over strong selection baselines on 12 tasks and four backbones. However, the rewriting layer never commits an edit. However, the evolution layer commits no edits in the audited runs. The positive results therefore support retrieval and arbitration but not label-space evolution.

Critical strengths
  • The 24-method grid covers a range of retrieval, diversity, uncertainty, and demonstration-selection baselines across 12 datasets.
  • Testing three additional models provides some evidence that the overall configuration is not unique to Qwen3-8B.
  • The authors correctly distinguish gains from the retrieval-arbitration architecture from gains due to label updates. This prevents the main accuracy improvements from being incorrectly attributed to evolution.
Critical weaknesses
  • Since no edit is committed, so there is no comparison between an edited state and its frozen counterfactual. There is no treatment arm from which to estimate whether executed label evolutions help or hurt accuracy. Due to this the paper’s main question remains open.
  • The experimental results are weak. Gain over the strongest external baseline is 1.6 pp on paired subset.
  • The routing decision depends on label cardinality. The configuration is selected using the same dataset suite used for reporting results. So the reported results cannot establish cardinality as the reason to choose one configuration over another.
  • The gain decomposition follows from the outside-subset invariance assumption. But it does not establish that label edits will preserve predictions outside the targeted subset or provide a practical bound without that assumption.
  • The paper is hard to follow. The main text contains many qualifications about protocols. The appendix mixes current results with archived tables. There are inconsistencies in naming. A shorter paper focused on the prediction architecture with archived studies removed would be clearer.
Questions for authors
  • What prevents edits from being committed: insufficient pair support, proposal generation or validation? Can you report counts at each stage?
  • Can you isolate the effects of the second retrieval view, coverage gate, disagreement rule, and arbiter through controlled ablations?
  • Does the routing rule generalise to held-out datasets, including tasks with intermediate label counts?
  • What happens when label descriptions are genuinely ambiguous? Can the evolution mechanism be evaluated on overlapping taxonomies where merges or descriptions have measurable effects?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.