acceptodds
Under review as a conference paper at ICLR 2027

Categorical Regression in REBEL: Rate Preservation, Boundary Bias, and No General Robustness

Abstract

REBEL reduces policy improvement to squared-loss regression of reward gaps onto policy log-ratio differences and asks whether cross-entropy does better. We answer for the tied HL-Gauss loss, which encodes the scalar as a cross-entropy-trained histogram. Inside the support the loss is two-sidedly calibrated to the squared loss with constants free of support width and bin count, so REBEL's bound carries over, with its square-root rate sharp under the stated assumption class. At the boundary the encoding winsorizes noisy targets, a bias absent from squared regression, bounded by how far and how often targets cross the boundary. Preregistered bandit experiments confirm this picture: under symmetric contamination HL trails MSE in all 20 seeds, by 3.4% of clean regret on average. One-sided, feature-correlated spikes give a replicated benefit of 0.7% of clean regret near the interpolation threshold, below the preregistered smallest effect of interest and, in the registered runs, not separated from curvature-matched Huber. Widening the support attenuates both effects; at LLM scale HL and Huber are equivalent within the registered margin of ±0.035 clean-reward units. The replacement keeps REBEL's rate but buys no general robustness: what matters is where a loss clamps, not whether it is categorical; curvature-matched Huber remains the conservative default.

Primary area:
reinforcement learning
Keywords:
REBEL, RLHF, policy optimization, HL-Gauss, cross-entropy regression, regression as classification, calibration, minimax lower bounds, robustness, preregistered experiments

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BXNEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether replacing the standard squared-error loss in REBEL with a classification-based loss makes RL more robust to noisy rewards. The authors show that the new loss has the same theoretical guarantees but offers no practical benefit. Any improvements mainly come from limiting the influence of extreme reward values, rather than from using classification instead of regression.

Critical strengths

The theoretical analysis and experiments are presumably solid and comprehensive but the paper is quite incredibly dense and hard to digest so many details have not be audited carefully.

Critical weaknesses
  1. What is the actual novelty and practical significance of the work? The finding seems to be more a null result, and even then, it is unclear what fundamental new insight is to be gained?
  2. The theory and experiment gap needs to be better addressed? The theory guarantees require predictions to be within the histogram's interior but that is not necessarily true in experiment?
  3. Not sure how generalizable the conclusions are given the experiments use a relatively simple synthetic contextual bandit with fixed sampling rather than on-policy REBEL procedure.
  4. The paper is unnecessarily dense and filled with jargons that are ill-defined and hard to read...
Questions for authors

What is actually the main takeaways besides stating the obvious in the practitioner's takeaway section, any deeper insights?

1 comment

Dear Reviewer BXNE,

The paper and our first reply were both dense, so here is a short version.

Deeper insight (your question). The natural expectation is that a classification loss is more robust to heavy-tailed reward noise. We preregistered that prediction, and it failed in 20 of 20 seeds: HL was 3.4% worse than MSE, while Huber matched MSE. The reason is that HL limits targets at the edge of its histogram, which biases the fit even when the noise has zero mean. Huber limits residuals, so under symmetric noise it still aims at the mean. This explanation predicts that a wider support shrinks the deficit. It does: from 3.4% to under 0.1% in the bandit, and from 1.7% to 0.2% in a preregistered on-policy MLP test.

Theory and experiments (Weakness 2). You are right that the regret guarantee does not cover our experiments, and the revision says so at the end of Section 4. The results that do apply there are Proposition 2 and Theorem 6. They describe the bias of the fit when targets cross the support edge.

Generality (Weakness 3). The deficit was also tested on-policy with an MLP (400 seeds). It kept its sign at about half the size.

For practice the result is negative, as you say. Our case for the paper is that REBEL's authors posed this question, the answer is now supported by preregistered tests, and the main difference between the losses has an explanation.

We are glad to clarify anything before Nov 18.

Reviewer_KZKjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper explores using a categorical parameterization for regression with the REBEL algorithm. Some theory is derived to compare to the categorical loss (HL-Gauss) to MSE, finding that they only differ by a constant factor for certain theoretical error bounds. Some experiments compare the two.

Critical strengths

None

Critical weaknesses

The paper is difficult to follow and the writing is poor; It's unclear what the main insights are. The novelty is fairly limited, combining categorical losses with REBEL, without any particularly interesting findings. For one, there are many theoretical results, but none with any impact. At best, it seems like there's a negative result about the value of categorical losses over MSE when used with REBEL. Then, there's a lengthy discussion about clipping values that does not yield much.

The experiments are similarly disorganized and not well motivated. For example:

  • Why is L-BFGS being run in particular?
  • Fig.1 is disorganized and it is unclear what is trying to be shown.
Questions for authors

None

Reviewer_x2R2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The core contribution of this work is examining the effect, both theoretically and empirically, of changing the loss used by the REBEL reduction. The paper focuses on the HL-Gauss map, which maps an input to a categorical Gaussian distribution. By mapping to this distribution, rather than using MSE to compute the loss between the REBEL input and target, cross entropy between the mapped input and target distributions can be used instead.

The main insight provided in this paper is that the primary effect of the HL-Gauss CE loss is found at the boundary of the support, termed the compression zone. Theorem 1 shows that, inside the support of the distribution, the HL-Gauss is equivalent to scaled squared loss. However, at the boundary of the support, the distribution is clamped inwards, resulting in distinct behaviour compared to MSE, or Huber loss.

Empirically, the paper examines how these different losses behave under clean data, or under noisy rewards, considering both symmetrical and one-sided losses.

Critical strengths

This paper represents a thorough and detailed examination of the effects of HL-Gauss CE loss in the context of REBEL. The paper presents detailed Theorems to analyse these effects. The work also includes a clear set of experiments that empircally explore the conditions under which the REBEL loss formulation has an effect.

I think the main strength of the paper is in the insight provided on the relationship between MSE and the HL-Gauss loss. Theorem 1 shows a global upper bound, and a lower bound within the support interval, which indicates the importance of the boundary of the support. Figure 1b) clearly illustrates how HL-Gauss loss behaves differently at its support boundary compared to MSE.

Critical weaknesses

This paper addresses the question posed in the original REBEL paper about whether alternative loss functions could result in better performance in practice, and so I think this paper represents only relatively incremental work. Whilst this is not necessarily a critical weakness of the work, given this incremental nature of the work, I think that the empirical results need to be stronger.

The current results appear show that, under specific conditions (e.g. symmetrical loss, in the interpolation regime), HL-Gauss does outperform MSE and Huber. However, outside of these conditions, it performs comparably or worse than MSE. Based on the results presented, it appears that outside of the interpolation regime (when n > 32), HL Gauss is outperformed by Huber loss. I don't think showing performance improvements only when there is a very small amount of data represents a strong endorsement of the method. The authors are not claiming that HL Gauss is a superior loss, but I don't think relatively incremental work, combined with what amounts mostly to a negative result, represents a sufficiently significant contribution.

The additional MLP/LLM testing also seems somewhat insignificant, as the results read as inconclusive at best. Perhaps I have misinterpretted this section, but I didn't find these results to be particularly insightful. I would appreciate if the authors could clarify exactly what the main takeaway of these additional experimental results, beyond showing minimal performance difference between the losses.

Questions for authors
  1. Can the authors outline a clear, pratical implication of the presented result? The benefits of HL-Gauss seem minimal in this setting, and only under certain conditions.

  2. Do you have an insight into why HL-Gauss appears to perform better in the interpolation regime compared to the averaging regime? What do you think causes the perfomance change when n=64n=64

  3. Continuing from the last question, what happens when you increase nn beyond 64? Does the performance gap increase?

2 comments

Dear Reviewer x2R2,

Our first reply was long, so here is a short version in case it is useful.

Your Q3. We ran the sweep you asked about. Beyond n = 64 the benefit does not come back. At n = 512 HL is 0.8% behind MSE (0 of 40 seeds favor HL), and Huber is ahead of HL from n = 64 on (new Table 19).

What the paper claims. We do not propose HL-Gauss for REBEL. REBEL's authors posed the switch to cross-entropy as their first open question, and classification losses are reported to help value-based RL, so the switch is a natural thing to try. We find no general gain. REBEL's square-root rate is sharp under its assumptions (Theorem 3), so changing the loss cannot improve that rate. The clearest difference is a deficit under the symmetric noise we preregistered: HL is worse than MSE in all 20 seeds. We can explain it: HL clips targets at its support edge, and this biases the fit. The explanation predicts that a wider support shrinks the deficit, and a preregistered on-policy test confirmed that.

In practice. Keep MSE, or use Huber when the reward scale is uncertain. If you use HL, widen its support.

You had no substantive concerns about soundness. Your concern was whether a mostly negative result is significant enough. We ask that the paper be weighed as a negative result. The question was open in the literature, and the answer now rests on preregistered tests, a check in a second (nonlinear, on-policy) setting, and an explanation. If breadth is what is missing, we would like to know which additional setting you would find most convincing.

We are glad to answer anything further before Nov 18.

Thank you for your thorough and detailed response to my review, I very much appreciate the time taken to answer my questions about the paper and the inclusion of the additional results for when nn is increased beyond 64. I also apologise for my slow response, especially given you were so quick to produce your rebuttal.

However, my original concerns do ultimately remain. As the authors admit, this is incremental work, it is directly answering a question that was posed in the original REBEL paper. There is not a huge amount of novelty in this work as the idea had previously been proposed by the authors of the original paper.

Whilst this is not necessarily an issue on its own, I personally think the combination of an incremental piece of work, whilst only showing a negative result, does not represent a sufficiently interesting or significant contribution for an ICLR paper. I don't have any issues with the soundness of the work, but I don't think that alone is sufficient in a paper that, at its core, is only able to show that this specific HL-Gauss loss for REBEL effectively does not show any performance gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.