acceptodds
Under review as a conference paper at ICLR 2027

Evidence, Adaptation, and Tail Risk in Verifiable-Reward Taxonomy Alignment

Abstract

We audit verifiable-reward and supervised post-training for institutional taxonomy alignment: mapping national occupational records into ISCO-08 across 878 country/entity–system–source cells. On a development-exposed benchmark, five row-only post-SFT variants leave worst-cell reliability within three points of SFT, and five checkpoints share a 73-cell worst-decile core. Continued SFT on evidence-augmented prompts with gold targets only (three seeds) changes this pattern: worst-cell conditional value-at-risk increases by 0.072–0.081 relative to plain-prompt SFT, while overall strict accuracy falls by 4.7 points to 74.2–74.4%. The change is heterogeneous: cells defined as difficult from development records improve from 14.3% to 24.1–24.2%, while easy cells fall from 99.3% to 91.3–91.6%. At the coverage of the explicit ABSTAIN-target control, confidence-selected gold-only SFT reaches 96.1–96.2% selective accuracy versus 84.0–84.3%; at the strongest selective RL policy's 70.8% coverage, it reaches 92.6–92.8% versus 63.3%. On a prospectively frozen, single-source transfer of 976 queries, five labeled references raise gold-only accuracy from 9.1–9.9% to 28.9–29.6% at fixed weights; below a post-hoc nearest-code lookup (35.8%). These results identify recipe-specific tradeoffs between institutional-tail accuracy, aggregate error and deferral. They do not establish a general information-theoretic limit of verifiable rewards, or isolate training evidence from the additional optimization in continued SFT.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reinforcement learning with verifiable rewards, selective prediction, abstention, group-relative policy optimization, worst-case reliability, conditional value-at-risk, taxonomy alignment, occupational coding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_y7nPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tried to study how language models can map occupation codes from different institutions into a common classification system. The authors investigate whether reinforcement learning can improve performance on difficult groups of data and whether providing additional information can help. The experiments compare different training methods to evaluate their effectiveness in improving model accuracy and reliability across different data groups.

Critical strengths

The problem the authors are trying to address is important and worth studying. Improving fairness across different groups is a meaningful research direction, and I agree with the motivation behind this work.

Critical weaknesses
  • This paper is very difficult to follow. The abstract is particularly hard to understand, with too many technical terms, abbreviations, and numbers introduced without clearly explaining the main idea. The introduction does not help much either. I think the writing and overall structure need serious improvement to meet the expectations of this venue. Some parts also feel repetitive and unnecessarily complicated.

  • I found the experimental sections difficult to follow as well. The text contains too many numbers and often repeats information that is already available in the tables. I think the purpose of the text should be to explain what these results mean, what trends we can observe, and where the proposed methods perform well or poorly. The experiments would be much more informative if the authors focused on explaining the results rather than simply reporting them.

  • As I understand, in Section 3.1 authors used the test results to identify poorly performing groups and this information influenced some of the later experimental choices. The authors acknowledge this limitation, but I am concerned that the same test data was also used to evaluate the later experiments. This makes it difficult to know whether the reported improvements would hold on completely new data.

  • The paper shows that additional supervised training with supporting information can improve performance on difficult groups. However, I could not find a direct comparison where the model receives the same amount of additional training without this information. Without such a comparison, it is difficult to understand whether the improvement comes from the additional information or simply from further training.

  • I am not convinced about the practical value of the more complex RL approach. The results show that simpler supervised methods with confidence-based filtering often perform better. I think the paper should better explain what the proposed approach actually offers over alternatives.

Questions for authors

Please refer to the Weaknesses.

Reviewer_LCCWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies institutional tail risk in occupational taxonomy alignment, mapping national occupational records to four-digit ISCO-08 codes across 878 country/entity–system–source cells. It first audits SFT, DPO, GRPO, cell-CVaR variants, and tail-cell SFT, finding that their worst-cell performance remains close to the SFT baseline and that five checkpoints share a persistent 73-cell worst-decile core. The paper then analyzes why centered group-relative estimators provide no gradient on zero-variance groups and why full-weight within-group CVaR does not provide a positive learning signal for an unsampled abstention action. To study selective behavior, the authors introduce CellTilt+X, which combines cell-level reward tilting with a forced ABSTAIN rollout. This improves selective accuracy modestly over the no-exploration ablation but does not improve worst-cell reliability. In contrast, continued gold-only SFT on evidence-augmented prompts raises worst-cell CVaR by 0.072–0.081 across three seeds, although overall strict accuracy decreases by 4.7 percentage points. Confidence thresholding of this gold-only model substantially outperforms the tested explicit-abstention and RL policies at matched coverage. Finally, a prospectively frozen single-source NOL transfer shows that five labeled references improve gold-only accuracy from 9.1–9.9% to 28.9–29.6%, while a post-hoc nearest-code rule reaches 35.8%.

Critical strengths

The analysis goes well beyond aggregate accuracy: it examines worst-cell CVaR, checkpoint overlap, cell-size sensitivity, fully deferred cells, confidence-ranked prediction, and dependence-aware bootstrap variants. This makes the negative and tradeoff results substantially more informative than a conventional post-training benchmark comparison.

Critical weaknesses

The test split was used to identify the persistent 73-cell tail and subsequently informed method development and analysis. The paper also states that its pre-specified predictions were fixed after the supervised results were known. The prospective NOL evaluation is valuable, but it contains only one source/version cell and therefore cannot independently validate the central cross-cell CVaR conclusions. This substantially limits how confidently the observed tail improvement and checkpoint overlap can be expected to replicate.

Gold-only continued SFT changes both the input evidence and the amount of supervised optimization relative to plain-prompt SFT, while the matched-budget continued-SFT control without evidence has not been completed. Most selective RL runs start from the original SFT model rather than an evidence-adapted SFT model, and the RL-from-gold-only experiment has only one seed. Consequently, the results convincingly show that the tested RL recipes are unnecessary for obtaining the reported gains, but they do not yet establish which part of those gains is specifically caused by training-time evidence or how a fully matched RL comparison would behave.

Questions for authors

Could the authors provide an untouched multi-cell evaluation, or a comparably strong held-out confirmation, on which the target cells, evidence construction, operating points, and model-selection decisions were fixed before any outcomes were observed? If this is not currently available, could the paper more explicitly enumerate which headline results are exploratory consequences of development exposure and which are genuinely prospective?

Can the authors report a matched-budget continued-SFT control that uses the same initialization, records, number of updates, optimizer settings, and seeds as gold-only augmented SFT but omits the neighbor and correspondence evidence? Does the 0.072–0.081 worst-cell CVaR improvement remain after separating evidence exposure from additional supervised optimization?

How should the deployment conclusion be interpreted given that confidence-ranked plain-prompt SFT produces substantially fewer wrong automatic answers at 70% coverage, whereas gold-only augmented SFT completely defers far fewer institutional cells? Would the authors’ preferred method remain unchanged under an explicit utility that jointly penalizes wrong answers and fully deferred cells, and under what range of relative costs?

Reviewer_fpXUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors audit verifiable-reward and supervised post-training for institutional alignment for the specific application of mapping national occupation records (e.g., from the EU) into an international labor statistic standard. In this diagnostic and mechanism study, the authors find issues with tail risks.

Critical strengths
  • Understanding tail risks in institutional alignment through RLVR is an ongoing research question and domain specifically applications/evaluations like this work can provide crucial insights to improve methodologies.
Critical weaknesses

The submission suffers from severe writing and clarity issues. It is too densely written and I found it extremely hard if not impossible to follow. I recommend rejection solely on that basis. Specifically:

  • It reads like a dense list of purely technical details and individual numerical results without connecting elements and motivations. Even the abstract does not motivate our outline research questions and consists of contextual, specific numerical results.

  • The paper uses non-standard terminology for the RLVR setting, making it harder to parse.

  • It is unclear how the paper’s findings relate to other research in the field and how it could advance the field, e.g., as the authors do not motivate their work (This includes the introduction, but also other parts of the paper). Even the related work section is sparsely populated, making it difficult to position the paper within the larger field.

  • Parts of the text read more like concatenated bullet points than a paper (the limitations section is a particularly bad offender).

  • I do not see how the findings generalize beyond the very specific tested application and the authors also propose no new algorithmic solutions. While purely diagnostic studies and negative results can be crucial to advance the field, I struggle to see the imapct of this work.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.