acceptodds
Under review as a conference paper at ICLR 2027

Detecting Structured Effects in Preference Data: A Fisher Information Perspective

Abstract

Preference comparisons selected to improve reward-model fit need not be informative for detecting interactions among evaluation dimensions. We study this selection problem through nuisance-adjusted Fisher information, which combines preference variance with the interaction signal remaining after accounting for main effects. This criterion links labeling budget to a local detection boundary and motivates Cond-Opt, a frozen-pilot selector accompanied by checks of selected-set identifiability and information retention. In fixed HelpSteer2+3 and UltraFeedback pools, the top 1% of comparisons account for 92.6% and 89.8% of estimated interaction information. In matched-budget simulations with split-null calibration, Cond-Opt achieves higher power than unresidualized interaction selection and one-shot full-parameter Fisher leverage across all six evaluated budget–noise settings at γ* = 0.5. Against uniform sampling, its median gain is 58.1 percentage points within the prespecified uniform-power window of 20–80% in a separate grid. On HelpSteer2+3 features with simulated labels, it reduces the 50%-power detection boundary by 36%. Tests on observed human labels do not confirm the specified interaction at the evaluated scale. Together, these results identify the value of targeting residual interaction information and provide a framework for assessing which comparisons can inform a specified preference question.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
preference data, reward models, active learning, optimal experimental design, Fisher information, statistical power, LLM alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7LFUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cond-Opt, a data selection method based on nuisance-adjusted Fisher information, to improve the detection of structured interaction effects in preference data. Experiments demonstrate improved detection power over baseline methods.

Critical strengths

1, The paper addresses a meaningful problem by focusing on preference data selection for detecting structured interaction effects, rather than solely improving reward-model fitting.

2, The paper includes theoretical analysis and extensive experiments across different labeling budgets, noise levels, and data settings.

Critical weaknesses

1, The paper is generally well-written and follows a clear overall structure. However, the connections between some theoretical analyses and the proposed method could be better explained, see questions below.

2, The authors should better clarify the purpose of each experiment and the connections between them to strengthen the overall narrative and highlight the paper's main contributions.

3, The paper offers an interesting application of classical statistical principles to preference data selection. However, the core methodology largely builds on established Fisher information and optimal experimental design techniques, which limits the novelty of the contribution.

Questions for authors

1, In line 55, the authors use γ⋆\gamma^\star without first defining or explaining it.

2, The interaction between verbosity and correctness used to motivate the problem in the Introduction differs from the aggregate conflict feature (line158) studied in the paper. The authors should better explain the motivation for choosing this particular interaction feature.

3, The main improvements in detection power are demonstrated using simulated preference labels generated from the logistic model assumed by the proposed method. It would be valuable to evaluate the robustness of Cond-Opt under model misspecification, particularly when the true preference-generating process differs from the assumed logistic interaction model.

Reviewer_B1KFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the selection problem where the informative preference comparisons are selected to improve interaction detection. It proposes the framework called Cond-Opt that ranks the preferences according the nuisance-adjusted Fisher information metric. From the empirical studies, the effectiveness of Cond-Opt is justified via fewer sample size required for interaction detection.

Critical strengths
  1. The statistical result from the nuisance-adjusted Fisher information is solid. The overall results look rigorous and make sense to me.
  2. Important problem with practical use case/potential. This is a problem that is faced when using the right data more smartly.
Critical weaknesses
  1. Need to compare with existing similar literature to justify novelty. (see the questions)
  2. The paper considers simulated labels and a favorable, frozen pilot, which still have gap with the practical usage. How powerful is the method toward having misspecified preference models or independently estimated pilots?
Questions for authors
  • Can you distinguish better between the following literature? To me, the idea looks quite similar. Compare Yujing Shao, Lei Wang*, Heng Lian. Optimal Decorrelated Score Subsampling for High-Dimensional Generalized Linear Models Under Measurement Constraints

  • Does the APO refer to a different paper than Das et al.? Because Das et al. use uncertainty based on feature differences and an inverse information matrix instead of selecting based on the largest Bernoulli variance.

Will positively consider raising scores by addressing the questions and weaknesses.

Reviewer_xxsSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a Fisher-information-based method for preference data selection, improving the statistical power of testing interaction effects among evaluation dimensions (e.g., verbosity and correctness). However, the improvements are primarily observed on simulated preference labels rather than real human labels.

Critical strengths

-This paper highlights a methodological distinction between data selection for predictive fit and for statistical inference, thus emphasizing aligning methodological choices with intended use. -The proposed method is grounded in Fisher information and connects preference uncertainty to residualized interaction signal, which provides a principled approach to preference data selection for interaction detection. -The experiments systematically compare Cond-Opt with multiple baselines across different simulation conditions and demonstrate improved statistical power.

Critical weaknesses

-The motivation could be introduced in the beginning of the paper with a clearer context. First, the abstract could have stated why the interactions among evaluation dimensions are important. Second, although these interactions are instantiated in the first paragraph of the introduction, more terminologies such as ‘preference selection for interaction detection’, ‘selection criteria’, and ‘frozen null pilot’ need further explanation. Third, I am convinced that data valuable for preference prediction is not necessarily valuable for interaction detection, but I would like to see more discussions of what such interactions enable beyond improving predictive fit.

-The specification of target interactions needs further clarification. In L157-163, the aggregate effect is identified by cross-dimensional conflicts rather than specific evaluation dimensions. This seems inconsistent with your example of verbosity & correctness, as the effects detected by Cond-Opt could also be from complexity & coherence if there are large conflicts between these two dimensions. The attribution and interpretability of your findings would therefore be impaired.

-Another concern is that the main power gains are based on simulated preference labels rather than real human labels. Both the model for generating simulated labels and Cond-Opt assume a logistic preference model with linear main effects and a given interaction feature, then what if these assumptions are violated? This warrants additional robustness analysis and discussion of model misspecification.

Questions for authors

-The first sentence appears a little difficult to grasp for readers who are not very familiar with reward model training (including me). In other words, readers may not tell whether & why interactions among evaluation dimensions are important from reading this. -Is it fair to consider this study as a simulation study? If the gains based on simulated labels represent best-case improvements, could you expand a little on how they might change on real human labels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.