acceptodds
Under review as a conference paper at ICLR 2027

AdaptEvo: Learning under Imperfect Supervision through Confidence-Adaptive Training with Knowledge Evolution

Abstract

Rule-governed contextual decision tasks require models to apply specified rules to case-specific context and evidence. Written rules can leave gaps in decision guidance and process evaluation, while reference judgments vary in their support from the rules and evidence. To address these challenges, we introduce AdaptEvo, a framework for learning under imperfect supervision that couples confidence-adaptive policy optimization with evolving decision knowledge and evaluation rubrics. Its Training module uses Confidence-Adaptive GRPO (CA-GRPO) to balance outcome and process rewards according to reference confidence. Its Evolution module synthesizes reusable decision knowledge from recurring failures across training cases and refines process rubrics to detect overlooked errors. To support empirical evaluation, we construct and release an industrial multimodal content moderation dataset comprising a training set and In-Period and Out-of-Period test sets, with the latter collected under changed rules. Using Qwen3.6-35B-A3B, AdaptEvo achieves 61.9% exact-label accuracy and 72.2% binary decision accuracy on In-Period, exceeding GRPO by 7.5 and 3.7 percentage points, respectively. On Out-of-Period, the policy trained with CA-GRPO retains exact-label accuracy gains over the base model across evaluated checkpoints without injected decision knowledge, while GRPO declines with continued training. CA-GRPO also outperforms the tested fixed reward mixtures on both Out-of-Period metrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Content Moderation, Confidence-Adaptive Rewards, Knowledge Evolution, Imperfect Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Si3sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AdaptEvo, a framework for rule-governed decision tasks with imperfect supervision. Its main contributions are Confidence-Adaptive GRPO (CA-GRPO), which adjusts the balance between outcome and process rewards based on reference confidence, and an Evolution stage that updates decision knowledge and process rubrics from recurring training failures.

The method is validated on an industrial multimodal content-moderation dataset with both In-Period and rule-shifted Out-of-Period test sets. Using Qwen3.6-35B-A3B, AdaptEvo reaches 61.9% ELA / 72.2% BDA In-Period and 59.5% / 69.5% Out-of-Period, outperforming standard GRPO. Additional comparisons examine fixed reward mixtures, checkpoint generalization, and evolved knowledge.

Critical strengths

The Out-of-Period evaluation and checkpoint analysis are informative. Rather than evaluating only on an IID test split, the paper considers a later-period setting with rule changes and tracks performance throughout training. The results show that standard GRPO degrades at later checkpoints while CA-GRPO maintains substantially better Out-of-Period performance

Critical weaknesses
  1. The proposed training methods are evaluated with only a single trainable backbone, Qwen3.6-35B-A3B, and almost entirely on a new benchmark introduced by the paper itself. The paper does not show that CA-GRPO or AdaptEvo transfers across model families or to an established external benchmark. In addition, the confidence mechanism requires multiple human annotations for each example, which may limit its applicability to datasets where such annotations are unavailable.
  2. The final AdaptEvo model simultaneously introduces evolved knowledge, an updated rubric, and additional policy training. Therefore, its improvement over CA-GRPO cannot be clearly attributed to knowledge evolution, rubric evolution, or simply extra optimization.
Questions for authors
  1. Do the authors have results for AdaptEvo or CA-GRPO on any additional trainable backbone beyond Qwen3.6-35B-A3B?
  2. How do the authors assess the generalizability of AdaptEvo and CA-GRPO to other datasets?
  3. Can the authors provide matched-compute results that isolate the contribution of the Evolution stage, in particular continued CA-GRPO with the original knowledge and rubric versus continuation with evolved knowledge and/or the evolved rubric?
Reviewer_hm8cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the problem of rule-governed contextual decision tasks, following a agentic multimodal content moderation approach. The authors identify two major problems in existing flows. Reference labels may have uneven support from the rules and evidence, and written rules and process rubrics may leave gaps. AdaptEvo alternates between two modules: training and evolution. Training is done using GRPO with a mix of outcome and process rewards. Evolution is using an external model to synthesize knowledge from recurring failures and improve rubrics. Experiments using Qwen3.5+ family models show that AdapatEvo can improve over standard GRPO.

Critical strengths
  • Problem is well-motivated as human or AI-generated rubrics can be weak and may need to be adjusted during training.

-- Out-of-period results show the effectiveness of the proposed approach compared to GRPO.

  • Method simple and a lightweight addition on top of GRPO.

  • Appendix is very detailed. I didn't have time to read it but it will be helpful for readers.

Critical weaknesses
  • Experimental results are validated only one dataset (domain) and no variance is reported. More grounded empirical evidence would be more supportive for the paper's claims.

  • Evolution can be costly and reduce training efficiency significantly.

  • Most of the information seems to come from the knowledge extracted using a very strong external model.

Questions for authors
  • Is the compute budget set same across compared techniques?

  • How does the base model and GRPO perform with only adding external knowledge?

Reviewer_Tp79OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AdaptEvo addresses learning under imperfect supervision in rule-governed contextual decision tasks—such as content moderation—where models must apply written rules to case-specific evidence. The authors identify two supervision failures: written rules leave gaps that fixed process evaluators cannot catch, and reference labels vary in the support they receive from rules and evidence. The framework alternates between two modules. The Training module uses Confidence-Adaptive GRPO (CA-GRPO), which balances outcome and process rewards according to a per-case confidence score derived from human annotation agreement and an LLM's assessment of how well each reference label is supported by rules and evidence. The Evolution module synthesizes reusable decision knowledge from recurring failure patterns across training cases and refines the process rubric to detect previously overlooked errors.

Critical strengths
  1. Well-chosen problem with real motivation. Rule underspecification plus heterogeneous reference support is a real and under-served problem, and the industrial moderation setting with a rules-changed test period is a better testbed than the usual random split.
  2. Detailed methodological description. The confidence mapping, the three-level support gate, the graded outcome reward, the protocol penalty, the five process dimensions and their weights, missing-score handling, and the rubric-screening sample sizes are all specified.
Critical weaknesses
  1. Neither named mechanism is proved to be responsible for the reported gain. The headline increment over knowledge-augmented CA-GRPO (+2.8 ELA / +0.9 BDA) bundles continued training with the new rubric; no arm isolates either, and on Out-of-Period the increment is +0.9 / −0.2, leaving rubric evolution unattributed. Separately, per-sample reward weighting beats the best fixed mixture by only 0.2–1.1 points and loses In-Period ELA to the fixed 0.2:0.8 mixture, while every result is a single run and the test-set sizes are never reported.

  2. The largest effect admits a simpler explanation that is not controlled for. The +3.7 / +3.2 gain comes entirely from injecting strong-model-written guidance at inference with no parameter updates, yet there is no control in which the same model rewrites the rules without access to failure cases, and no length-matched uninformative guidance; "the rule document was simply rewritten more clearly" is therefore not excluded, and it is exactly the explanation the paper's own motivating figure invites. No verbatim knowledge entry appears anywhere, although that text carries the effect.

Questions for authors
  1. How much of the gap remains if GRPO receives the same injected knowledge, and if the guidance is produced from the rules alone without failure cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.