- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies active preference acquisition with systematically biased LLM judges. It proposes Nuisance-Adjusted Optimal Design (NAOD), which selects comparisons using effective information about human preferences after accounting for judge-specific nuisance parameters, weighted by downstream policy relevance. The paper provides a conditional local asymptotic minimax characterization, analyzes the additional risk caused by learned nuisance representations, and develops a finite-pool selection algorithm. Synthetic experiments examine the theoretical mechanisms, while a Chatbot Arena evaluation reports improvements in proxy policy regret and held-out human-preference prediction.
Critical strengths
-
The paper explicitly models human preferences and judge-specific deviations as separate components. This formulation allows the acquisition criterion to account for uncertainty in the nuisance parameters and the sensitivity of the downstream policy to reward estimation errors.
-
The representation-learning analysis identifies an additional risk term and conditions under which representation error can reverse the ranking of acquisition methods. The synthetic experiments include examples of this reversal, documenting a limitation of the proposed approach.
Critical weaknesses
-
My main concern is the choice of evaluation reference. The primary Arena metric is , where is trained on upstream human labels and is already used to construct the nuisance representation. Simply retaining this model would therefore give zero proxy regret, yet its held-out human cross-entropy and accuracy are not included in the comparison. Without this baseline, the reported 29.1% reduction tells us how NAOD compares with Target-info, but leaves a more basic question unanswered: does collecting additional judge labels improve on the human preference model already available? This matters particularly because the upstream stage uses 8,575 human-labeled clusters (Appendix F.2; supplementary experiment.py and representation.py).
-
There is a gap between the estimator analyzed in the theory and the one used in Arena. Appendix B.2 constructs preliminary estimates from the current observations, with an error that decreases as the sample size grows. In the released code, the preliminary estimates are fitted once on Init128 and reused across budgets, followed by a single Newton update. A fixed initialization sample does not generally provide the shrinking error required by the proof. The paper therefore needs to explain how the attainment result relates to this implementation and establish whether the empirical advantage survives a change in the final estimator (Appendices B.2 and F.1; supplementary estimator.py).
-
The paper's own analysis shows that representation error can reverse the ranking of acquisition methods, but the Arena experiments do not examine this dependence. Section 5 derives a risk of the form , while NAOD optimizes only the fitted information criterion. The synthetic experiments illustrate the resulting limitation clearly. On real data, however, the upstream sample size is fixed, so there is little guidance on how much historical data is needed for NAOD to help, or when representation error makes its selections worse than those of simpler methods.
-
The role of the learned target representation needs closer examination. Section 5 studies errors in with fixed, whereas the Arena pipeline learns both from upstream data. Moreover, is constructed separately for each judge using human and judge preference heads. It is unclear how the learning-cost analysis accounts for this additional source of uncertainty. The experiments also fix both target and nuisance dimensions at two. Results with a shared target space and larger dimensions would help establish whether the gains depend on this particular construction. A shared target space would also make the cross-judge relationship between coupling and performance easier to interpret (Section 5; Appendix F.1).
Questions for authors
-
How does NAOD compare with simply using ? Please report this model's held-out human cross-entropy and accuracy, together with a human-only model fitted on the upstream and current trusted labels. Using the same target representation and test split, does additional judge acquisition improve either metric beyond these baselines?
-
Do the acquisition rankings change with the final estimator? Keeping the selected subsets fixed, please compare the Init128-based one-step update with the estimator constructed in Appendix B.2 and with a joint likelihood fit run to convergence. Paired comparisons of NAOD, Target-info, and D-opt, with uncertainty intervals, would help determine whether the reported gains depend on the update rule. Please also clarify how the current implementation satisfies the assumptions used for attainment.
-
What happens when less historical data is available to learn ? Please vary this sample size on Arena while keeping , the evaluation reference, current budgets, and test set fixed. Report both proxy regret and human-label metrics. Does the ranking reversal predicted in Section 5.2 occur, and over what range of sample sizes does NAOD retain an advantage?
-
Are the results robust to the representation choices? Please test a modest range of target and nuisance dimensions, and include a target space shared across judges and learned without judge-specific labels. Do the performance gains and the reported relationship between coupling and improvement persist?
-
What exactly does Theorem 5.1 cover when the target representation is also learned? Is the result conditional on the realized , and does account for any error from learning ? Please clarify how the theorem should be interpreted for the Arena pipeline, where both and are learned from the historical sample.
3 comments
Thanks for clarifying the assumptions in Appendix B.2. I’ll raise Soundness to 2. I’m keeping my overall rating at 1 because the gains at small human-label budgets still depend on the estimator. I still find the paper hard to follow.
Dear Reviewer,
Thank you for acknowledging our clarification of Appendix B.2 and raising the Soundness rating to 2. We would like to further clarify the two remaining concerns.
1. Estimator dependence at small human-label budgets.
We acknowledge the observed finite-sample dependence, but its interpretation is important. This behavior is consistent with the conditions underlying Appendix B.2, rather than evidence against our theoretical results or acquisition principle.
The asymptotic efficiency of the Newton one-step estimator requires sufficiently accurate preliminary estimates. With only 32 or 64 current human labels, a current-only preliminary fit can be unstable, and a single Newton update is not guaranteed to achieve asymptotic efficiency.
Our additional experiments exhibit behavior consistent with this mechanism. The B.2-style estimator's absolute proxy regret decreases substantially as the human-label budget increases. In the exploratory subgroup with H = 128, 256, and 512, NAOD achieves an 18.39% mean regret reduction over Target-info using this estimator.
Our original Arena protocol explicitly specifies the shared historical Init128 initialization, providing a more stable starting point under limited current-label budgets. Importantly, all acquisition methods use the same estimator within each comparison, isolating the effect of the acquisition strategy.
Furthermore, NAOD retains a 14.40% mean regret reduction over Target-info under converged joint MLE across the full budget grid, with a positive paired confidence interval. Thus, its gains are not exclusive to Init128.
We do not claim estimator-independent improvements or asymptotic attainment for every practical initialization. The observed small-budget behavior highlights precisely why preliminary estimation accuracy matters in Appendix B.2. As the budget increases, the B.2-style estimator improves markedly, consistent with the theoretical mechanism. These findings clarify the finite-sample conditions for effective implementation rather than contradicting our conditional guarantees or invalidating the proposed acquisition criterion.
2. Readability and presentation.
Our methodology builds on the established statistical optimal experimental design framework, extended to accommodate judge-specific nuisance parameters. The central derivation follows a standard statistical progression from joint target–nuisance Fisher information, through Schur-complement effective information, to a policy-weighted acquisition criterion. We then develop a practical finite-pool selection algorithm.
We have further clarified the assumptions, estimator construction, and experimental protocol throughout our rebuttal. We would appreciate clarification of any specific definitions, derivations, or proof steps that remain difficult to follow, so that we can address them directly.
Thank you again for your time and consideration.
Dear Reviewer,
Thank you again for your valuable feedback. As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our rebuttal and additional experiments have addressed your concerns, or if any further clarification is needed.
Thank you for your time and consideration.