- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies online learning with numerical regret. At each round, the learner draws two actions, accrues their average utility, gets at most m binary comparisons of that pair, and can buy exact numerical scores for at most of the rounds. The authors characterize when comparisons alone achieve sub-linear regret. For a cubic link family, they show that known calibration gives regret of order without audits, while unknown calibration within radius adds a linear term. They extend this to a quintic family and to smooth changes of link shape. On recorded HelpSteer2 judgments, a baseline that follows the critic judge outperforms all audited methods, and audits do not improve over comparison-only learning.
Critical strengths
-
The main contribution is Theorem 3.1, which gives a necessary and sufficient condition for comparisons alone to achieve sublinear regret under a fixed label cap. The condition requires one polynomial of degree at most m that recovers every numerical gap up to a positive scale. This separates three failure modes: a non-polynomial inverse, inconsistent shapes across the family, and vanishing signal strength. A notable consequence is that under the standard logistic link, any fixed cap gives linear regret even when the temperature is known.
-
The audit lower bound is technically careful and general. It bounds score information conditional on every pre-audit transcript, including same-type audits, rather than only matching the marginal comparison laws. Proposition J.2 shows that the cost arises for any regular change of link shape, by fixing the latent comparison distribution and varying only its numerical interpretation. The phenomenon is therefore not an artifact of the cubic construction, although the paper gives this argument only one paragraph in Section 4.3.
Critical weaknesses
- The writing is disorganized to the point that the paper is very difficult to follow. Some examples include:
- Several key terms and quantities are used before they are defined. For instance, half-gap, admission, and transcript appear early without definition, and θ and D_θ are used in Figure 1 and Section 2, while the link family they refer to is only given in Section 3.2.
- The cubic family in (3.2) is introduced without motivation, and the reason for its form appears only later in the lower-bound construction.
- Equation (4.3) relies on quantities that are not defined in the main text.
- Section 2 is too brief. It should be expanded to define the protocol, the link family, and the minimax values before they are used.
- The experiments do not test the theoretical model, and the empirical conclusions are fragile. Some examples include:
- The theory assumes m conditionally independent comparisons of the same pair from a fixed link. The replay uses a single recorded judgment per pair, so this setting never occurs.
- Algorithm 1 is never evaluated. Its nominal route requires a declared radius r within the cubic family, which has no meaning for real judges. The evaluated Audit+comparison method is a heuristic advice mixture without a numerical-regret guarantee for its comparison component.
- The best learning method is Audited-mean, a greedy baseline with no adversarial guarantee. This suggests that the worst-case analysis does not explain the observed behavior.
- The strong direct-critic result depends heavily on which critic is followed. Following GPT-4o gives training regret of 0.92, while following Gemini or Nemotron gives 12.81 and 17.31, and the ranking changes on validation (Table 3).
Questions for authors
-
How can the experiments be modified to better reflect the theory?
-
The matching upper bound selects either the nominal route or the audit-only route before the run. Can comparisons and audits be combined to improve on both? In Theorem 4.1, the lower-bound constant decreases with m, while the upper bound does not depend on m, which suggests such a combination could help when m is large.
-
Why does adding audits to the comparison advice increase regret in the replay (96.69 for Audit+comparison versus 90.45 for comparison-only in Table 2)? Is this caused by the learning rate or exploration choices dictated by the worst-case analysis?
We are sorry that the original presentation did not guide readers through the model and reasoning. We have revised the paper around the specific problems you identified.
Section 2 now defines the feedback protocol, half-gap, transcript, link family and minimax benchmark before the results. We removed “admission.” Section 4.1 explains the cubic choice before giving its formula: three independent judgments can estimate a cubic inverse without bias, allowing us to isolate the cost of an unknown link. The endpoint and interpolation conditions then determine its coefficients. Section 4.2 defines the conditional audit distribution before the KL bound. Figure 1 no longer introduces θ or Dθ before their definitions.
We now evaluate Algorithm 1 under its stated model: utilities from the cubic lower-bound construction, three independent judgments of the same pair from its link, a declared radius r and a prescribed audit budget. The algorithm selects its route from the regret bounds before each run (§6.1; Appendix H.1). Averaged over the two environments at K=2, T=131,072 and r=1/16, known-link regret is 1787.95, below the 1997.29 lower bound for unknown-link learners with at most three judgments per round and no audits. Algorithm 1 with B=0 gives 2073.47.
The human panels in §6.2 separately test gap estimation with three judgments per pair. The recorded-answer replay has one verdict per pair and does not instantiate the repeated-comparison model. It evaluates Audit+comparison, whose guarantee against the best fixed rule comes from updates using audited ratings and holds for any predictable comparison adviser (Appendix D.6). This gives no numerical-regret guarantee for the adviser alone and requires no cubic-family radius for real judges.
On the GPT-4o training replay, Audited-mean remains the strongest learning baseline, with regret 67.38. The worst-case bound does not explain this empirical ranking. We also report all three direct critics on both splits. GPT-4o leads on training, whereas Nemotron leads on validation for helpfulness (Tables 8–9). The empirical conclusions are therefore stated for the corresponding critic and split.
Your observation about the m-dependent lower-bound constant is correct. It leaves room for better constants and for improvement as m grows. Corollary F.3 shows only that, for each fixed K and m≥3, arbitrary joint use of comparisons and audits has minimax rate Θ_{K,m}(√T+T min{r,(B+1)^(−1/2)}). The constants may depend on m. The optimal dependence on a growing judgment cap remains open.
In the GPT-4o training setting you cite, that comparison changes both feedback and initial weights. Audit+comparison initially gives 0.9 to the comparison adviser and 0.1 to fixed rules. Freezing these mixture weights, while letting the adviser continue learning, adds 9.00±1.30 regret relative to comparison-only. Updating the same mixture from ratings recovers 2.76±0.89, leaving the observed 6.24 excess.
The initial mixture thus accounts for most of the excess. The 0.9/0.1 split is a design choice; the guarantee allows other positive initial rule weights, with a corresponding change in its bound. Separately, dividing the denominator offset by four lowers regret by 1.76±0.60; dividing only the learning rate raises it by 1.66±0.81. Appendix I.5 reports all 126 settings; these tuning effects are not distinguishable from zero on validation. Intervals are paired 95% Monte Carlo intervals.
We would be grateful for the opportunity to have the revised paper reconsidered.