acceptodds
Under review as a conference paper at ICLR 2027

Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

Abstract

Classical reinforcement learning (RL) typically seeks a deterministic policy that maximizes the expected sum of a scalar reward. Yet, modern applications such as language model fine-tuning or scientific discovery demand diversity. Existing remedies such as entropy regularization or diversity bonuses often require fragile trade-offs that sacrifice performance for stochasticity or rely on heuristic metrics that can misalign policy rankings. We argue that diversity is more naturally understood as the rational response to _uncertainty_ in the reward. When the reward function is not perfectly known—as is the case with ambiguous preferences or imperfect reward models—committing to a single action can be sub-optimal. Building on this, we propose a fundamental reformulation of the RL objective by replacing the scalar reward with a distribution over reward functions, and applying a non-linear objective over _sets_ of actions. The result is a framework in which _calibrated_ behavioural diversity emerges naturally, remains controllable through the reward function distribution, and is obtained without sacrificing expected reward. Focusing on the contextual bandit setting as commonly used in large language model (LLM) post-training, we derive a principled gradient estimator for this objective and prove that our formulation naturally generalizes both vanilla policy gradient and more recently developed action-set approaches. We provide didactic experiments which complement our theoretical results, and our large-scale empirical results in LLM reasoning further demonstrate that this framework offers a robust and theoretically grounded alternative for complex RL tasks where the traditional formulation of the problem fails to induce the desired breadth of agent behaviour.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, LLM, post-training, diversity, reward uncertainty, policy gradient

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3gQ4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ROSA, which trains a policy to generate a set of responses that performs well across a distribution of reward functions. Its main objective averages the best reward within the sampled set over possible reward functions. The paper derives policy-gradient estimators, characterizes optimal policies in a binary-reward setting, and also evaluates the approach using toy problems and LLM fine-tuning experiments involving response-length preferences and noisy rewards.

Critical strengths

The paper provides an intuitive connection between reward uncertainty and diverse behavior: evaluating a group of responses can encourage the model to produce several useful options. The theoretical analysis characterizes optimal sampling probabilities under simplified reward models. The proposed objective also admits an unbiased policy-gradient estimator, making it straightforward to optimize. Experiments with competing preferences, response lengths, and noisy judges illustrate several potential applications to language-model training.

Critical weaknesses

First, I have one concern about the prior work. domain randomization [1] already considers training across different environments or reward functions, and subsequent works have studied policy gradients under domain randomization. I believe the authors should clarify how their work differs from this literature and discuss these connections in the related work section if relevant.

Second, the theoretical guarantees cover a narrow class of reward structures. The principal optimal-policy results assume binary reward functions, each rewarding a distinct action. While this setting allows for a clean analysis, it does not establish comparable guarantees for overlapping preferences, continuous rewards, or a misspecified reward distribution.

Additionally, the experiments provide limited evidence of practical usefulness. They mainly use manually designed reward functions and artificial judges, which may not reflect the errors made by real reward models. Experiments with realistic reward models and a closer examination of answer quality would help establish whether the method produces meaningful diversity.

Chen, Xiaoyu, et al. "Understanding domain randomization for sim-to-real transfer." ICLR (2022).

Questions for authors

Could you clarify the distinction with domain randomization literature?

3 comments

We thank the reviewers for their comments, and are encouraged to hear that the connection between reward uncertainty and policy diversity is intuitive, and that the proposed objective accounting for the reward function uncertainty is straightforward to optimize. We respond below.

Concern about prior work: domain randomization

We are happy to add discussion of the relationship between domain randomization and our work! However, we believe it is less closely related than the areas we already discuss in the paper (multi-objective RL, risk-sensitive RL, distributional RL, etc.)

Specifically, a principal aim of domain randomization is to bridge the sim-to-real gap and improve policy robustness. Typically, these approaches (i) randomize simulator dynamics, (ii) keep a fixed reward, and (iii) optimize the expectation (i.e. linear) over sampled dynamics. This is in contrast to ROSA, which aims to learn diverse policy by (ii) randomizing over rewards, and (iii) crucially, use a modified objective that depends non-linearly on the set of rewards.

We are happy to discuss further and include in the text if the reviewer deems it helpful. Currently, we’ve also re-organized our main text related works section to more clearly distinguish between existing related literatures (Section 3.3).

Second, the theoretical guarantees cover a narrow class of reward structures.

We agree with the reviewer’s assessment of the limitations of the current assumptions. Indeed, the theoretical analysis in this work focuses on binary, non-overlapping rewards (as we explicitly stated in Prop 3.1, Theorem 4.1 and Prop 4.2). A few notes of clarification:

  • The objective and gradient estimators do not rely on any of these assumptions, and empirically works for continuous reward functions with overlapping optimas (see Figure 7)
  • On the theoretical side, binary reward is a standard assumption for “RL from verifiable rewards” works in current LLM literature. Indeed, many algorithms have analysis focus exclusively on the binary reward setting, such as [1, 2]

Thus, our assumptions are fairly standard. The simple assumptions have also led to insights about the algorithm. Nevertheless, we agree generalizing the theory to continuous, overlapping reward functions has value and is exciting future work. To this end, we’ve included a standalone paragraph in Section 6 of the updated pdf outlining limitations to facilitate productively extending the existing results. We hope the reviewer finds this suitable and we are happy to discuss any other limitations the reviewer may have in mind!

[1] Tajwar, Fahim, et al. "Maximum likelihood reinforcement learning." arXiv preprint arXiv:2602.02710 (2026).

[2] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

Additionally, the experiments provide limited evidence of practical usefulness. … Experiments with realistic reward models and a closer examination of answer quality would help establish whether the method produces meaningful diversity.

We acknowledge that the existing results are focused primarily on single-turn, verifiable MATH problems. This was done as a didactic demonstration of our claims that ROSA results in policy which is diverse along the dimensions where the reward function distribution disagrees.

We’ve added three new experiments to better demonstrate ROSA’s general usefulness. To summarize:

  1. Semantic diversity with non-verifiable LLM-as-judge setting: we train ROSA on open-ended chat settings with LLM-as-judge (“reaslitic reward function”). It preserves the similar quality improvement as standard PG while improving on diversity (Table 3, Appendix D.5)
  2. Greater mathematical answer and reasoning strategy diversity: we re-analyzed the improved OOD results from Figure 8b and find that ROSA trained models generates more unique final answers, and use more diverse problem solving strategies (Table 2, Appendix D.7)
  3. Multi-turn task: we show ROSA achieves higher pass@k and solution diversity in a multi-turn Sokoban setting (new Figure 9, Appendix D.4)

The detailed results are in the mentioned section in the updated pdf; we provide a summarized highlight in the response below as well.

All in all, we believe the additional results, taken together, re-affirms our central thesis that ROSA is a generally useful, widely applicable tool for inducing controllable diversity along the dimension we can specify. We thank the reviewer for the recommendations which have led to a better understanding and exciting future directions.

New Experiments

Unless otherwise stated, we bold the best & ones statistically insignificantly tied with the best.

Semantic diversity and non-verifiable rewards (Table 3 & Appendix D.5): We train models on WildChat conversational chats with an LLM-as-judge. ROSA is trained by having another LLM cluster the outputs into groups; we treat each group as being preferred by a different reward function under the ROSA formalism. We evaluate the trained checkpoints under the standard NoveltyBench evaluation and report the results below (details in Appendix D.5). As we can see, ROSA tends to perform at least as well as standard policy gradient in terms of quality, and always perform better in terms of diversity as measured by the Distinct@32 metric.

MethodQuality@1Quality@32Distinct@32
WildChat-1k (n=1000n=1000 prompts)
Base model5.85 [5.66, 6.04]8.16 [8.00, 8.32]8.65 [8.14, 9.18]
PG6.43 [6.24, 6.61]8.46 [8.31, 8.61]8.14 [7.61, 8.70]
ROSA+Max@86.27 [6.08, 6.45]8.47 [8.32, 8.62]8.64 [8.12, 9.19]
ROSA+Max@46.60 [6.41, 6.78]8.69 [8.55, 8.84]8.74 [8.20, 9.30]
Curated (n=100n=100 prompts)
Base model6.04 [5.46, 6.60]8.84 [8.37, 9.23]12.92 [11.30, 14.58]
PG6.36 [5.81, 6.90]8.82 [8.39, 9.20]10.43 [9.02, 11.92]
ROSA+Max@86.30 [5.74, 6.86]8.86 [8.45, 9.21]12.88 [11.26, 14.52]
ROSA+Max@46.30 [5.75, 6.84]8.82 [8.38, 9.20]12.42 [10.89, 13.99]

Closer examination of diverse mathematical reasoning strategy (Table 2 & Appendix D.7): We further analyzed a sub-sample of the LLM outputs from our mathematical reasoning tasks (Figure 6 and 8b). Specifically, we summarize the rollouts via another LLM (to avoid length bias), embedded the summary, and evaluated diversity of the rollouts in terms of both unique \boxed{} answers as well as Vendi score (Friedman & Dieng, 2022) of the embedded reasoning strategy.

For the ROSA reward distribution defined by output length (Figure 6), we found that while ROSA successfully achieved its intended objective of generating diverse lengths, with similarly diverse reasoning strategies as PG (Appendix D.3). For the ROSA reward distribution defined by unknown true rewards (Figure 8b), we found that ROSA generates significantly more unique final answers (almost 2x that of PG) and also significantly more diverse reasoning strategies as characterized by the embedding Vendi score (full results in Appendix D.6 of the updated pdf, and an example question in Figure 20).

MethodAccuracyChar. length [min, max]Distinct boxedStrategy Vendi
Base model0.003 [.000, .007][264, 4731]8.5 [5.2, 12.5]2.07 [1.98, 2.16]
PG (mean)0.000 [.000, .001][278, 15720]65.6 [51.9, 80.1]1.97 [1.89, 2.05]
ROSA+Max0.001 [.000, .002][263, 24722]113.3 [94.1, 132.9]2.25 [2.12, 2.39]
ROSA+Softmax0.001 [.000, .002][82, 5030]80.6 [64.9, 97.6]2.01 [1.91, 2.11]

Multi-turn task (Figure 9 & Appendix D.4): We set up ROSA under the multi-turn conversation Sokoban task as used by RAGEN (Wang et al 2025). In this task, the LLM needs to act over multiple turns (via a multi-turn conversation format) to push a box into a goal location on the map. This involves taking many actions and reasoning ahead, and there are often more than one route. We detail our experimental set-up in Appendix D.4 of the updated pdf, and highlight the main result below. As we can see, ROSA learns both higher route coverage and higher pass@k, and this generally demonstrates ROSA’s applicability to multi-turn settings.

Methodpass@1pass@16Route coverage
Base model0.029 [.02, .04]0.161 [.11, .22]0.076 [.05, .10]
RLOO0.173 [.11, .24]0.179 [.11, .24]0.086 [.06, .12]
DARLING0.180 [.12, .24]0.221 [.16, .29]0.105 [.07, .14]
ROSA+Max@40.136 [.10, .18]0.393 [.31, .47]0.180 [.14, .22]
ROSA+Max@160.104 [.08, .13]0.464 [.39, .55]0.219 [.18, .26]

I thank the authors for the responses. They addressed my concerns and questions. I will raise the score to 3 accordingly.

Reviewer_6p8NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors argue that RL training often leads to diversity collapse because traditional RL optimizes a single scalar reward. Even when there are multiple correct solutions, the model tends to converge to one high-reward behavior, since there is no incentive to preserve different ways of solving the same problem. To address this, the paper proposes using multiple reward functions and jointly optimizing a group of rollouts. The goal is to encourage the model to generate diverse correct answers while maintaining accuracy. The key insight is that diversity does not necessarily need to be maintained through an explicit entropy bonus. Instead, when there is uncertainty in the reward, maintaining multiple valid solutions can naturally become an optimal strategy.

Critical strengths
  1. I think the paper is well motivated, especially the connection between diversity collapse and reward uncertainty. The theoretical analysis is also a strength, as it explains why maintaining diverse behaviors can be optimal under certain conditions, rather than something that needs to be encouraged by an entropy bonus.

  2. The experimental evaluation is relatively comprehensive and covers different aspects of the paper's main claims. Figure 5 shows that ROSA can learn different behaviors when reward functions conflict, while Figure 6 shows that it can maintain correct mathematical answers with different lengths. Figure 8 further demonstrates improved robustness to inaccurate reward judges. These experiments provide useful evidence supporting the paper's motivation and theoretical analysis, beyond improvements in benchmark scores.

Critical weaknesses

The paper proposes ROSA as a general RL objective that can improve behavioral diversity while maintaining reward performance. However, the LLM experiments are limited to mathematical reasoning tasks (MATH and AIME), where diversity is mainly demonstrated through different response lengths. It remains unclear whether ROSA can encourage more meaningful behavioral diversity in other settings, such as code generation, open-ended instruction following, or multi-turn agents. I think this limits how strongly the current experiments can support the general applicability of the proposed method.

Questions for authors

Q1. Figure 8 evaluates ROSA using 8 judges, but what happens if these judges share similar systematic biases, or if the reward distribution does not accurately reflect the true preferences? How sensitive is ROSA to the choice of reward distribution?

Q2. Figure 6 mainly measures diversity through response length, but different lengths do not necessarily imply different reasoning strategies. Could the authors evaluate whether ROSA improves semantic or reasoning diversity, rather than just length diversity?

Q3. Since ROSA optimizes the best reward among a group of rollouts, I wonder whether the improvement in diversity or Pass@k comes at the cost of single-sample performance. Could the authors provide a more systematic comparison of Pass@1 and Pass@k under different sampling budgets?

Q4. ROSA is mainly studied in the contextual bandit setting, where an entire response is treated as one action. Could the authors discuss whether the method can be extended to multi-step agent RL, where actions affect future states? For example, could ROSA maintain different successful tool-use strategies?

3 comments

We thank the reviewer for the informative review. We are encouraged to find our presentation, informativeness, and soundness to be all of high quality. We are also glad to find that the diversity and uncertainty angle is well motivated, with strong theoretical analysis, along with relatively comprehensive evaluations.

We address the reviewer’s comments on weaknesses below.

LLM experiments are limited to mathematical reasoning tasks (MATH and AIME), where diversity is mainly demonstrated through different response lengths

Q2. Figure 6 mainly measures diversity through response length, but different lengths do not necessarily imply different reasoning strategies. Could the authors evaluate whether ROSA improves semantic or reasoning diversity, rather than just length diversity?

Q4. Could the authors discuss whether the method can be extended to multi-step agent RL, where actions affect future states?

Many thanks for the suggestion. We have added three new experiments to better demonstrate ROSA’s general usefulness. We’ve added them to the new pdf, summarize them below, and also include the core new results in the subsequent response.

  1. Greater mathematical answer and reasoning strategy diversity: we re-analyzed the OOD results from Figure 8b and find that ROSA trained models generates more unique final answers, and use more diverse problem solving strategies (Table 2, Appendix D.7)
  2. Semantic diversity with non-verifiable LLM-as-judge setting: we train ROSA on open-ended chat settings with LLM-as-judge (“reaslitic reward function”). It preserves the same quality improvement as standard PG which improving over diversity (Table 3, Appendix D.5)
  3. Multi-step task: we show ROSA achieves higher pass@k and solution diversity in a multi-step Sokoban setting (new Figure 9, Appendix D.4)

All in all, the results support our thesis that ROSA is a generally applicable tool for inducing controllable diversity along the dimension we can specify with ρ\rho. We thank the reviewer for the recommendations which have led to a better understanding.

Q1. Figure 8 evaluates ROSA using 8 judges, but what happens if these judges share similar systematic biases, or if the reward distribution does not accurately reflect the true preferences? How sensitive is ROSA to the choice of reward distribution?

ROSA can be thought of as promoting coverage over optimal solutions which different judges / reward functions may disagree on. This way, ROSA is more robust to systematic biases and less sensitive to mis-specification as compared to standard policy gradient (PG) as it hedges across all preferences.

Theoretically, we have characterized how even with less probability over reward functions, ROSA can still put mass there (Prop 4.2, Figure 4). Empirically, Figure 8(a) shows this: even when only 1/8 judge (12.5% correct) prefers the true correct answer, ROSA is able to put mass over that preference, and therefore achieve higher pass@k as compared to PG.

If no reward functions reflect the true preference(s), then the optimal ROSA policy will not put mass on it, but neither will the optimal PG policy.

Q3. Since ROSA optimizes the best reward among a group of rollouts, I wonder whether the improvement in diversity or Pass@k comes at the cost of single-sample performance. Could the authors provide a more systematic comparison of Pass@1 and Pass@k under different sampling budgets?

Thank you for this interesting question. Please see existing figures 5b, 6, and 8b for figures of the pass@1 to k performance of the trained models. We’ve additionally included Tables 8-10 in the appendix detailing pass@1 and @k.

To summarize: in theory, ROSA’s diversity does not require sacrificing pass@1, as ROSA’s objective optimum is reward-maximal (for binary rewards). In practice with finite compute budgets:

  • In smaller settings when optimization can be done to near completion, ROSA achieves optimal pass@1 and @k and deals well with competing reward functions (Fig 5b)
  • With length diversity, ROSA and PG have similar correctness pass@1 and @k, with ROSA being more length-diverse (Fig 6)
  • With noisy, incorrect judges, ROSA+Softmax achieves higher Pass@1 and Pass@8 over both PG and ensemble baselines (Fig 8a, Table 10).
  • For multi-turn Sokoban, ROSA achieves higher pass@k and route coverage, albeit the Max@16 ROSA variant has statistically lower pass@1 compared to PG, while Max@4 does not. Note that with large n, the ROSA+Max learning signal shrinks once the policy already has reasonable pass@1 (due to flatter objective landscape, see Appendix C.1), which may explain this result.

All in all, across majority of experiments, ROSA tends to match (or sometimes out-perform) the quality of standard PG training, and on all experiments it improving upon generation diversity.

New Experiments

Unless otherwise stated, we bold the best & ones statistically insignificantly tied with the best.

Closer examination of diverse mathematical reasoning strategy (Table 2 & Appendix D.7): We further analyzed a sub-sample of the LLM outputs from our mathematical reasoning tasks (Figure 6 and 8b). Specifically, we summarize the rollouts via another LLM (to avoid length bias), embedded the summary, and evaluated diversity of the rollouts in terms of both unique \boxed{} answers as well as Vendi score (Friedman & Dieng, 2022) of the embedded reasoning strategy.

For the ROSA reward distribution defined by output length (Figure 6), we found that while ROSA successfully achieved its intended objective of generating diverse lengths, with similarly diverse reasoning strategies as PG (Appendix D.3). For the ROSA reward distribution defined by unknown true rewards (Figure 8b), we found that ROSA generates significantly more unique final answers (almost 2x that of PG) and also significantly more diverse reasoning strategies as characterized by the embedding Vendi score (full results in Appendix D.7 of the updated pdf, and an example question in Figure 21).

MethodAccuracyChar. length [min, max]Distinct boxedStrategy Vendi
Base model0.003 [.000, .007][264, 4731]8.5 [5.2, 12.5]2.07 [1.98, 2.16]
PG (mean)0.000 [.000, .001][278, 15720]65.6 [51.9, 80.1]1.97 [1.89, 2.05]
ROSA+Max0.001 [.000, .002][263, 24722]113.3 [94.1, 132.9]2.25 [2.12, 2.39]
ROSA+Softmax0.001 [.000, .002][82, 5030]80.6 [64.9, 97.6]2.01 [1.91, 2.11]

Semantic diversity and non-verifiable rewards (Table 3 & Appendix D.5): We train models on WildChat conversational chats with an LLM-as-judge. ROSA is trained by having another LLM cluster the outputs into groups; we treat each group as being preferred by a different reward function under the ROSA formalism. We evaluate the trained checkpoints under the standard NoveltyBench evaluation and report the results below (details in Appendix D.5). As we can see, ROSA tends to perform at least as well as standard policy gradient in terms of quality, and always perform better in terms of diversity as measured by the Distinct@32 metric.

MethodQuality@1Quality@32Distinct@32
WildChat-1k (n=1000n=1000 prompts)
Base model5.85 [5.66, 6.04]8.16 [8.00, 8.32]8.65 [8.14, 9.18]
PG6.43 [6.24, 6.61]8.46 [8.31, 8.61]8.14 [7.61, 8.70]
ROSA+Max@86.27 [6.08, 6.45]8.47 [8.32, 8.62]8.64 [8.12, 9.19]
ROSA+Max@46.60 [6.41, 6.78]8.69 [8.55, 8.84]8.74 [8.20, 9.30]
Curated (n=100n=100 prompts)
Base model6.04 [5.46, 6.60]8.84 [8.37, 9.23]12.92 [11.30, 14.58]
PG6.36 [5.81, 6.90]8.82 [8.39, 9.20]10.43 [9.02, 11.92]
ROSA+Max@86.30 [5.74, 6.86]8.86 [8.45, 9.21]12.88 [11.26, 14.52]
ROSA+Max@46.30 [5.75, 6.84]8.82 [8.38, 9.20]12.42 [10.89, 13.99]

Multi-turn task (Figure 9 & Appendix D.4): We set up ROSA under the multi-turn conversation Sokoban task as used by RAGEN (Wang et al 2025). In this task, the LLM needs to act over multiple turns (via a multi-turn conversation format) to push a box into a goal location on the map. This involves taking many actions and reasoning ahead, and there are often more than one route. We detail our experimental set-up in Appendix D.4 of the updated pdf, and highlight the main result below. As we can see, ROSA learns both higher route coverage and higher pass@k, and this generally demonstrates ROSA’s applicability to multi-turn settings.

Methodpass@1pass@16Route coverage
Base model0.029 [.02, .04]0.161 [.11, .22]0.076 [.05, .10]
RLOO0.173 [.11, .24]0.179 [.11, .24]0.086 [.06, .12]
DARLING0.180 [.12, .24]0.221 [.16, .29]0.105 [.07, .14]
ROSA+Max@40.136 [.10, .18]0.393 [.31, .47]0.180 [.14, .22]
ROSA+Max@160.104 [.08, .13]0.464 [.39, .55]0.219 [.18, .26]

Thank you for your thoughtful response. The clarifications have addressed my concerns, and I will maintain my positive rating.

Reviewer_6PB5OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper tackles the problem of learning diverse behaviors via reinforcement learning, and using the uncertainty in reward functions as the basis for this diversity. It clearly motivates the need for diversity in generative models, and their formulation i.e. diversity is proportional to reward uncertainty or underspecification. The paper introduces ROSA an objective that maximizes the maximum reward obtained by the policy distribution, when marginalized over the entire reward distribution. The authors claim that this learns the optimal stochastic policy with diversity enforced by the reward distribution, and not a result of arbitrary diversity / entropy based bonuses. As a result of this improved objective, it leads to better performance under multiple objectives. The method is tested across small-scale experiments and LLMs on math problems. The authors include detailed theoretical proofs to solidify their claims of the optimality of the objective under the reward distribution, and an unbiased gradient estimator for the given objective. Overall, this is a very interesting paper -- but I request the authors to move the summary table to the main paper. This will help readers understand the novelty and importance of the contribution very easily.

Critical strengths
  • The paper is well written and motivates the problem setup clearly. The introduced objective is very simple and scales to improving LLMs for generating diverse behavior on math problems.
  • The resulting policy learned under this objective has some interesting properties: it smoothly transitions between different action distributions based on the reward uncertainty, show that in a 3-simplex setup the convergence point is the optimal stochastic policy under the reward and preservers diversity conditioned on the reward function variance and not arbitary notion of diversity.
  • The authors introduce theory showing that ROSA learns an optimal diverse policy and has an unbiased gradient estimate.
  • The experiments span simple didactic experiments exposing the benefits of the contribution clearly, and scaled up LLM experiments with interesting behavior on Math showing that there's is the only objective that preserves correctness and learns a policy able to follow all different underlying reward functions.
Critical weaknesses
  • The presentation is slightly weak when making comparisons to prior work. The connection and differentiation to MORL and other related works is spread out over the paper. While there are both empirical results and discussions, it is hard to parse and stitch it together.
  • L166 it reads as a broader claim : that the policy generates an action set that is optimal each reward function. But it is optimal under the expected reward function right?
  • The current method maximizes the average under the reward function. How would a min-max objective behave in such settings? Would that be a better objective? Some additional discussion around the choice of the objective would be interesting.
  • Is preserving diversity under epistemic uncertainty of the reward function not harmful for safety of the models? Generally, if the model has an underspecified reward at a certain state, is the ideal behavior not refusing to generate a response / return something generic. How do the authors think about the safety implication of this?
  • Initially it seems that the method has a strong assumption: needing access to the explicit reward distribution and probabilities. But in Section 5.3 the authors have experiments with access to samples only. I would recommend the authors to discuss and mention this earlier to improve clarity and scope.
Questions for authors

see weaknesses.

2 comments

We thank the reviewer for the encouraging review. We are glad to hear the motivation is clear, the objective is simple and scalable, and that it leads to a resulting policy with interesting and desirable properties. We are also encouraged that the reviewer finds our theoretical analysis to be helpful, along with the experiments being helpful for exposing the benefits.

I request the authors to move the summary table to the main paper. This will help readers understand the novelty and importance of the contribution very easily. Better presentation re: prior work: the connection and differentiation to MORL and other related works is spread out over the paper. While there are both empirical results and discussions, it is hard to parse and stitch it together.

Thank you for the recommendation. Our initial structure split up the related works in a way such that key discussions are covered as they become relevant, and we have made significant effort to cover all related works and perspectives in Appendix A. That said, we take the reviewer’s point that a single unified presentation of related work could benefit the readers.

We’ve re-organized the main text to include a related works section (Section 3.3) that distinguishes the core ROSA objective from alternatives (such as MORL), and also moved up Table 1, so we have a unified conceptual comparison section. We hope the reviewer finds this better and we are open to other stylistic suggestions.

L166 it reads as a broader claim : that the policy generates an action set that is optimal each reward function. But it is optimal under the expected reward function right?

If we understand the question correctly: the objective is not the expected reward function. For binary rewards and uniform distribution over reward functions, the optimal ROSA policy puts probability mass over the optimal action of each reward functions (though a particular sampled set might not include them all). We have improved the phrasing of this sentence to make the assumption and conclusion clearer.

Is preserving diversity under epistemic uncertainty of the reward function not harmful for safety of the models? Generally, if the model has an underspecified reward at a certain state, is the ideal behavior not refusing to generate a response / return something generic. How do the authors think about the safety implication of this?

This is an interesting and important point. ROSA is designed to target underspecified solutions: it is able to hedge its action probabilities over multiple potentially good solutions as specified by high-reward regions of the multiple reward functions. When epistemic uncertainty reflects risk rather than diversity in preferences, training to model to deterministically refuse, rather than be stochastically diverse (as ROSA does), may be more desirable.

We believe a way forward could be to have a distribution of benign reward functions encoding diversity, but each one should also additionally pass a safety gate/filter for the reward to be given at all. This way can help preserve diversity only along “safe” dimensions.

Initially it seems that the method has a strong assumption: needing access to the explicit reward distribution and probabilities. But in Section 5.3 the authors have experiments with access to samples only. I would recommend the authors to discuss and mention this earlier to improve clarity and scope.

Thank you for this comment, we have added clarification in Section 3, below Equation (4), that one can estimate ER∼ρ[.]E_{R \sim \rho}[.] either via a Monte-Carlo sample-based estimator, or by summing/integrating over reward functions if they are available.

The current method maximizes the average under the reward function. How would a min-max objective behave in such settings? Would that be a better objective? Some additional discussion around the choice of the objective would be interesting.

Thank you for pointing out this interesting potential comparison, we discuss them here and will be glad to include it in the Appendix.

For reference, the ROSA+Max objective is EY1:nER∼ρ[max⁡(R(X,Y1),…,R(X,Yn))]E_{Y_{1:n}} E_{R \sim \rho} [\max (R(X, Y_1), …, R(X, Y_n))]

The fundamental contrast is that a min-max objective optimizes the policy against the worst-case reward function, whereas ROSA aims for behaviour that reflects the whole distribution ρ\rho over reward functions. Concretely, suppose we have a finite set of reward functions R1,…,RmR_1, …, R_m.

If the minimization is performed per response, we have EY∼π[min⁡kRk(X,Y)]E_{Y \sim \pi} [\min_k R_k(X, Y)] This corresponds to robust MDPs with "rectangular" uncertainty sets, which are tractable to optimize but have deterministic optimal policies (Wiesemann et al., 2013), so it does not induce diversity. This is the "Ensemble min" baseline in Figure 8, which under-performs ROSA.

If instead the minimization is performed for the whole policy, we have min⁡kEY∼π[Rk(X,Y)]\min_k E_{Y \sim \pi} [R_k(X, Y)] Here we believe the optimal policy can be stochastic, but policy optimization becomes a saddle-point problem (analogous to computing a Nash equilibrium of a two-player zero-sum game), which typically requires more complex approaches such as multiple optimization timescales (Cheng et al., 2022).

Finally, one could combine the worst case with our set-level max, EY1:n[min⁡kmax⁡(Rk(X,Y1),…,Rk(X,Yn))]E_{Y_{1:n}} [\min_k \max (R_k(X, Y_1), …, R_k(X, Y_n))] We can briefly analyze this by considering the setting of our theoretical results (binary reward functions, each with a distinct optimal action). It would seem the term inside the expectation is always zero if the number of sampled action nn is smaller than the number of reward functions mm, as one out of the m>nm > n reward functions is never satisfied and give 0 reward to all actions. This is in contrast to ROSA+Max which provides a signal for any n≥2n \geq 2 and is responsive to the reward distribution ρ\rho (Prop. 4.2). We are happy to discuss this further and believe a more involved analysis is an interesting direction for future work.

We are curious if the reviewer has any additional thoughts and we are happy to add a combined version to the Appendix if the reviewer wishes.


References:

Wiesemann, W., Kuhn, D. and Rustem, B., 2013. Robust Markov decision processes. Mathematics of Operations Research, 38(1), pp.153-183.

Cheng, C.A., Xie, T., Jiang, N. and Agarwal, A., 2022, June. Adversarially trained actor critic for offline reinforcement learning. In Proceedings of the International Conference on Machine Learning.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of discussion and revision

We are grateful to the reviewers and the area chair for their time and for a constructive discussion.

We are glad that the reviewers found:

  • The motivation from reward uncertainty to diverse behaviour clear and intuitive (all three reviewers)
  • The objective simple, scalable and straightforward to optimize, with an unbiased policy-gradient estimator (6PB5, 3gQ4)
  • The theoretical analysis is strong and explains why diversity can be reward-optimal rather than ad hoc (6PB5, 6p8N)
  • The experiments supportive of the main claims, including didactic examples and LLM fine-tuning (6PB5, 6p8N).

We are also encouraged that all three reviewers recommended acceptance.

The revised PDF contains the following changes:

  • Three new experiments (per 6p8N, 3gQ4): deeper analysis of reasoning-strategy diversity on AIME (Table 2, Appendix D.7), open-ended chat with an LLM judge (Table 3, Appendix D.5), and multi-turn Sokoban (Figure 9, Appendix D.4).
  • Main Text related work (per 6PB5, 3gQ4): a new Section 3.3 with the comparison table (Table 1) moved into the main text.
  • Aggregated limitations section (per 3gQ4): a new paragraph in Section 6 outlining limitations and future directions
  • More detailed reporting of Pass@1 and Pass@k (per 6p8N)
  • General clarity improvements based on reviewer feedback

We thank the reviewers again for a productive discussion session which have made this work better.

~Paavo_Parmas1OpenReview ↗

Discussion of ReMax

Dear Authors,

Thank you for citing our earlier ReMax work (published at ICML2026 https://arxiv.org/abs/2606.00151, first appeared online in 2022 https://openreview.net/forum?id=GKsNIC_mQRG) that is identical to ROSA+Max. It seems that several readers did not catch the connection, so I would like to give some suggestions on how to improve the discussion of our work, and also point out some of our other related concurrent works to yours.

For reference, you define ROSA+Max in Equation 3 as

JROSA+Max(πθ)=EX∼μ,,Y1,…,Yn∼i.i.d.πθ(⋅∣X)[ER∼ρ[max⁡1≤i≤nR(X,Yi)]]\mathcal{J}_{\text{ROSA+Max}}(\pi_\theta) = \mathbb{E}_{X \sim \mu,\\, Y_1,\ldots,Y_n \overset{\mathrm{i.i.d.}}{\sim} \pi_\theta(\cdot \mid X)}\left[\mathbb{E}_{R \sim \rho}\left[\max_{1 \le i \le n} R(X,Y_i)\right]\right]

Our ReMax objective in the 2026 paper is defined as

ReMax, Eq. (1) of Nishimori et al.

JReMaxM(π):=Eμ∼Π[EA[M]∼π[max⁡m∈[M]μAm,|,μ]]J^{M}_{\mathrm{ReMax}}(\pi) := \mathbb{E}_{\mu \sim \Pi}\left[\mathbb{E}_{A_{[M]} \sim \pi}\left[\max_{m \in [M]} \mu_{A_m} \\,\middle|\\, \mu\right]\right]

ROSA+Max and ReMax are identical. To see the correspondence, fix a context xx, set M=nM=n, and take the ReMax policy to be π=πθ(⋅∣x)\pi=\pi_\theta(\cdot\mid x). Drawing a reward function R∼ρR\sim\rho in ROSA induces a ReMax value vector with entries μa=R(x,a)\mu_a=R(x,a). The distribution of this vector is Π\Pi. Both objectives therefore evaluate the same quantity: the largest value among the sampled actions, with every action evaluated under the same reward draw. The equations write the expectations in opposite orders, but at fixed xx the action samples are independent of the reward draw, so the order does not affect the result.

Specifically, we proposed this objective for exploration in RL, and argued that this objective causes the optimal policy to become stochastic when there is a distribution over the rewards. The work discussed different properties of the ReMax objective function and how it induces diversity and exploration. The equations explicitly included reward uncertainty, and the bandit experiments use explicit posteriors over the reward functions so that policies are optimized over a distribution of rewards. However, the RL experiments argued that there is “implicit” uncertainty and did not maintain explicit reward distributions. Your work correctly points out that our initial focus was RL rather than generative modeling. However, in our concurrent work on multi-axis max@k (https://arxiv.org/abs/2607.14962) we leverage this approach in GRPO-style RL post-training of diffusion models for generative modeling using a distribution over reward vectors to induce diverse generations (this work was not cited). The current writing in the paper mentions that we focused on RL, but it omits mentioning that we also used the same objective for generative modeling in concurrent work to you.

Another concurrent work to yours is our work on MaxPO (also not cited) a gradient estimator for the max@k objective (https://arxiv.org/abs/2606.06080) that we applied in LLM post-training. Specifically, your estimator is that of Tang et al. (2025) that uses a baseline for max@k using the other k-1 samples. There is an improvement on this estimator by Walder et al. who average Tang’s estimator over subsets of size k when the batch size is N, and improves over Tang. Our MaxPO further considerably improves over Walder’s PKPO.

Finally we also have concurrent work on regret guarantees in bandit algorithms with this objective (https://arxiv.org/abs/2605.20854), and we also extend the max@k objective to arbitrary order-statistics (https://arxiv.org/abs/2606.06096).

While it is understandable that you do not have to compare with all of these approaches, the policy strongly encourages discussing concurrent work.

I want to emphasize that there are many interesting results in your work, and you did also acknowledge our earlier ReMax work clearly for example on the footnote on page 3. However, judging from the reviews, it did not appear conveyed to the readers that ROSA+Max is identical to our earlier proposed ReMax, so I would appreciate it if the clarity regarding this could be improved. Moreover, discussing our broader body of work would make it clearer to the reader that we did not only consider it for exploration in RL but were also actively using it for promoting diversity in generative modeling.

Specifically, I think it would be easier for the reader to understand that ROSA+Max is identical if you just called the objective “ReMax” in the paper without changing the name compared to our prior work.

~Paavo_Parmas1OpenReview ↗

Continuation of comment

I also think that some passages could be clearer. For example, in the introduction, you write:

“In response, we propose policy gradients that depend non-linearly on sets of actions (Koyamada et al., 2022; Tang et al., 2025; Hamid et al., 2026) and sampled reward functions; while set-based updates have previously been explored to promote exploration in Markov decision processes (Koyamada et al., 2022), we provide an in-depth analysis as applied to generative modelling.”

I find that this passage remains unclear about what our Koyamada et al. actually did. It seems related to sets of actions, but the “sampled reward functions” bit is not clearly connected to Koyamada et al. You could write more clearly that updates with a max over a set with sampled reward functions have been explored previously. There are many ways how the connection could be made clearer.

Additionally, it would be nice if you add ReMax into table 1. Currently, the table caption notes:

“ROSA is the only approach that satisfies all desiderata”

However, because ReMax is identical, it also satisfies all desiderata.

I would appreciate it if you consider a bit how you could make it easier for the reader to understand the relationship to our prior ReMax, and also introduce some of our concurrent work on generative modeling with this objective so that the reader gets a more full picture of our work.

Finally, I apologize for the last minute comment; I was busy with my own rebuttals, and rushed this out at the last minute.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.