acceptodds
Under review as a conference paper at ICLR 2027

Latent Rubrics: Identifiability and Semantic Transfer in Preference Models

Abstract

Can reusable evaluation criteria be learned without updating a language-model backbone? We study parameter-efficient semantic grounding on frozen representations: our Qwen3-32B main study trains only a 15,363-parameter rubric head, with no backbone updates. The central challenge is identification, not parameter scale. Preference likelihood is invariant to changes of latent basis; exact named interventions identify directions up to positive scales under a restricted linear model. A synthetic positive control raises axis recovery from 0.730 to 0.996 with one intervention per criterion, while preference accuracy changes little. On real text, a human-audited filter passes 19/20 fresh blind pairs, but locally valid edits alone do not ensure transfer: Silver-200 obtains 61.40% TEST rubric composition versus 72.57% for DEV-aligned preference-only training. A subsequent 10,240-parameter semantic-ranking head, using additional human labels and retaining the frozen backbone, reaches 72.67% DEV composition: 3.81 points above the natural-contrast intervention variant, but not a confirmed improvement over aligned BT. This demonstrates a lightweight route to improving development-set semantic metrics, not established TEST superiority. Our framework combines restricted identification theory, auditable supervision, and controlled diagnostics to distinguish parameter-efficient learning from the stronger requirement of reusable semantic axes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
reward modeling, preference learning, identifiability, rubric learning, weak supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XnizOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper trains only a small scoring head on a frozen language model, learning correctness and coherence coordinates from preference pairs and named rewrites. Precise one-factor edits in synthetic data can align the axes with the generating factors. On HelpSteer2, even when the rewrite directions are fitted perfectly, these coordinates do not become better evaluation criteria than those from preference-only training.

Critical strengths
  1. The problem is clearly formulated, and the theoretical claims stay within the stated assumptions. The paper shows that the Bradley–Terry preference likelihood is invariant under invertible coordinate transformations, and it gives a complete linear-algebra proof under explicit conditions, including a shared invertible mixing matrix and exact one-factor edits.
  2. The synthetic experiments separate fitting the total score from aligning the coordinates. The failure on real data is also supported by corresponding evidence.
Critical weaknesses
  1. The paper does not show that its rubric is useful. The experiments never use the learned coordinates to train a language model or replace human ratings, and the paper does not optimize a policy. Beyond the reported diagnostic comparisons, there is no evidence for what this rubric can actually be used for, which makes the contribution less compelling.
  2. The conditions required by the theory do not match the real training setup, and the HelpSteer2 results do not show that the method yields better criteria. The theory proves uniqueness under exact one-factor edits, which requires a shared invertible mixing matrix and coordinate changes confined to the named axis. Silver-200 fits its training-edit directions at 100%, but its rubric composition is 61.40%, below the 72.57% of preference-only training with development-set alignment. Real training therefore does not establish a more accurate correctness or coherence coordinate.
  3. I suggest that the authors tighten the main narrative to reduce obstacles to reading. The exposition is cumbersome. The core comparison is separated from the reader by protocol details and by metrics that should not be compared directly, including rubric composition, purity, and natural target effect.
Questions for authors

Refer to Weakness.

Reviewer_9f6xOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the identifiability of interpretable latent evaluation criteria in preference-based reward modeling. The authors argue that preference supervision alone cannot uniquely identify semantically meaningful latent coordinates, since different latent representations can produce identical scalar rewards and preference predictions. To address this limitation, they propose Intervention-grounded Latent Rubrics (ILR), which uses named, directional text interventions to encourage individual latent dimensions to capture specific evaluation criteria.

The paper establishes a restricted theoretical result showing that exact one-factor interventions can identify semantic axes up to positive scaling under a shared linear mixing model. The proposed approach learns a small linear reward head on top of frozen LLM representations, requiring only 15,363 trainable parameters in the main Qwen3-32B experiments.

The authors evaluate the approach through synthetic identification experiments, human-audited intervention construction, and real-world experiments on HelpSteer2. Synthetic results show substantial improvements in axis recovery with minimal intervention supervision. However, the main real-text experiments reveal that intervention supervision does not translate into improved semantic generalization: Silver-200 achieves 61.40% TEST rubric composition accuracy, compared with 72.57% for a DEV-aligned preference-only baseline. Follow-up experiments using additional human attribute supervision and a separate semantic-ranking head improve DEV performance, but do not establish superiority over the baseline on an independent test set.

Overall, the paper provides a theoretical and empirical investigation of the gap between preference fitting, local intervention consistency, and transferable semantic representations.

Critical strengths
  1. Clear problem formulation and theoretically motivated semantic grounding.

The paper identifies an important distinction between preference prediction and semantic identifiability in structured reward modeling. Proposition 1 formally characterizes the invariance of preference likelihood under invertible transformations of the latent representation, while Proposition 2 provides a sufficient condition for identifying named semantic axes through exact directional interventions. Although the theoretical setting is restrictive, this formulation clearly explains why preference supervision alone is insufficient and what additional information is needed for semantic grounding.

The synthetic experiments provide supporting evidence for this mechanism. In particular, axis recovery improves from 0.730 to 0.996 with one exact intervention per criterion, while preference accuracy changes very little. This demonstrates that semantic recovery and preference prediction measure fundamentally different properties.

  1. Careful empirical diagnostics and transparent analysis of negative results.

A strength of this paper is its effort to validate the intervention supervision itself and distinguish local intervention success from semantic generalization. The authors conduct human audits of generated edits, evaluate transfer through weighted rubric composition, and investigate the failure of the proposed intervention method.

The analysis showing that 100 independent intervention feature differences leave a large nullspace in the 5,120-dimensional representation is particularly valuable. It provides a concrete explanation for why perfectly fitting the observed intervention directions does not guarantee correct semantic behavior on unseen responses.

Critical weaknesses
  1. Limited empirical evidence for effective semantic grounding.

The proposed intervention method does not improve semantic generalization on real-world data. In Table 2, Silver-200 achieves only 61.40% TEST rubric composition accuracy, compared with 72.57% for the aligned BT baseline, despite perfectly fitting the training intervention directions. The subsequent semantic-ranking head improves DEV performance but uses additional human supervision and lacks independent TEST validation. Moreover, its semantic coordinates do not contribute to the preference reward. Thus, the paper provides limited positive evidence that the proposed approach learns reusable semantic reward decompositions.

  1. Gap between theoretical identifiability and practical validation.

Proposition 2 relies on exact one-factor interventions and a globally shared linear mixing model, assumptions that are difficult to justify for natural-language representations. The main experiments use only 100 independent intervention families in a 5,120-dimensional feature space, leaving substantial ambiguity in the learned coordinates. Although the human audit supports local intervention validity, it does not establish sufficient semantic coverage or generalization. Furthermore, the strong composition performance of BT despite near-zero axis purity raises questions about whether the evaluation reliably measures semantic disentanglement.

Questions for authors

No.

Reviewer_s5zDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether reusable semantically identifiable reward dimensions can be learned from human preferences. It proposes a framework that trains a linear head on frozen LLM representations using preference supervision and directional text interventions. While the method achieves strong axis recovery on synthetic data, it underperforms a DEV-aligned preference-only baseline on real-text TEST composition. Subsequent semantic-ranking experiments show exploratory improvements on DEV but no confirmed advantage over the baseline.

Critical strengths
  • The paper provides useful empirical analysis showing that fitting preference labels and intervention constraints does not necessarily lead to generalizable semantic representations.
Critical weaknesses
  • The paper could benefit from a clearer organization and a more coherent narrative.

  • The strongest positive results come from a custom synthetic setting. The proposed intervention-based method underperforms the baseline on real-text TEST evaluation. The current finding do not sufficiently establish novel insifghts beyond existing literature.

Questions for authors
  • Could the authors clarify the main novel insights of this work beyond existing research on identifiability and intervention-based representation learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.