- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ROSA, which trains a policy to generate a set of responses that performs well across a distribution of reward functions. Its main objective averages the best reward within the sampled set over possible reward functions. The paper derives policy-gradient estimators, characterizes optimal policies in a binary-reward setting, and also evaluates the approach using toy problems and LLM fine-tuning experiments involving response-length preferences and noisy rewards.
Critical strengths
The paper provides an intuitive connection between reward uncertainty and diverse behavior: evaluating a group of responses can encourage the model to produce several useful options. The theoretical analysis characterizes optimal sampling probabilities under simplified reward models. The proposed objective also admits an unbiased policy-gradient estimator, making it straightforward to optimize. Experiments with competing preferences, response lengths, and noisy judges illustrate several potential applications to language-model training.
Critical weaknesses
First, I have one concern about the prior work. domain randomization [1] already considers training across different environments or reward functions, and subsequent works have studied policy gradients under domain randomization. I believe the authors should clarify how their work differs from this literature and discuss these connections in the related work section if relevant.
Second, the theoretical guarantees cover a narrow class of reward structures. The principal optimal-policy results assume binary reward functions, each rewarding a distinct action. While this setting allows for a clean analysis, it does not establish comparable guarantees for overlapping preferences, continuous rewards, or a misspecified reward distribution.
Additionally, the experiments provide limited evidence of practical usefulness. They mainly use manually designed reward functions and artificial judges, which may not reflect the errors made by real reward models. Experiments with realistic reward models and a closer examination of answer quality would help establish whether the method produces meaningful diversity.
Chen, Xiaoyu, et al. "Understanding domain randomization for sim-to-real transfer." ICLR (2022).
Questions for authors
Could you clarify the distinction with domain randomization literature?
3 comments
New Experiments
Unless otherwise stated, we bold the best & ones statistically insignificantly tied with the best.
Semantic diversity and non-verifiable rewards (Table 3 & Appendix D.5): We train models on WildChat conversational chats with an LLM-as-judge. ROSA is trained by having another LLM cluster the outputs into groups; we treat each group as being preferred by a different reward function under the ROSA formalism. We evaluate the trained checkpoints under the standard NoveltyBench evaluation and report the results below (details in Appendix D.5). As we can see, ROSA tends to perform at least as well as standard policy gradient in terms of quality, and always perform better in terms of diversity as measured by the Distinct@32 metric.
| Method | Quality@1 | Quality@32 | Distinct@32 |
|---|---|---|---|
| WildChat-1k ( prompts) | |||
| Base model | 5.85 [5.66, 6.04] | 8.16 [8.00, 8.32] | 8.65 [8.14, 9.18] |
| PG | 6.43 [6.24, 6.61] | 8.46 [8.31, 8.61] | 8.14 [7.61, 8.70] |
| ROSA+Max@8 | 6.27 [6.08, 6.45] | 8.47 [8.32, 8.62] | 8.64 [8.12, 9.19] |
| ROSA+Max@4 | 6.60 [6.41, 6.78] | 8.69 [8.55, 8.84] | 8.74 [8.20, 9.30] |
| Curated ( prompts) | |||
| Base model | 6.04 [5.46, 6.60] | 8.84 [8.37, 9.23] | 12.92 [11.30, 14.58] |
| PG | 6.36 [5.81, 6.90] | 8.82 [8.39, 9.20] | 10.43 [9.02, 11.92] |
| ROSA+Max@8 | 6.30 [5.74, 6.86] | 8.86 [8.45, 9.21] | 12.88 [11.26, 14.52] |
| ROSA+Max@4 | 6.30 [5.75, 6.84] | 8.82 [8.38, 9.20] | 12.42 [10.89, 13.99] |
Closer examination of diverse mathematical reasoning strategy (Table 2 & Appendix D.7): We further analyzed a sub-sample of the LLM outputs from our mathematical reasoning tasks (Figure 6 and 8b). Specifically, we summarize the rollouts via another LLM (to avoid length bias), embedded the summary, and evaluated diversity of the rollouts in terms of both unique \boxed{} answers as well as Vendi score (Friedman & Dieng, 2022) of the embedded reasoning strategy.
For the ROSA reward distribution defined by output length (Figure 6), we found that while ROSA successfully achieved its intended objective of generating diverse lengths, with similarly diverse reasoning strategies as PG (Appendix D.3). For the ROSA reward distribution defined by unknown true rewards (Figure 8b), we found that ROSA generates significantly more unique final answers (almost 2x that of PG) and also significantly more diverse reasoning strategies as characterized by the embedding Vendi score (full results in Appendix D.6 of the updated pdf, and an example question in Figure 20).
| Method | Accuracy | Char. length [min, max] | Distinct boxed | Strategy Vendi |
|---|---|---|---|---|
| Base model | 0.003 [.000, .007] | [264, 4731] | 8.5 [5.2, 12.5] | 2.07 [1.98, 2.16] |
| PG (mean) | 0.000 [.000, .001] | [278, 15720] | 65.6 [51.9, 80.1] | 1.97 [1.89, 2.05] |
| ROSA+Max | 0.001 [.000, .002] | [263, 24722] | 113.3 [94.1, 132.9] | 2.25 [2.12, 2.39] |
| ROSA+Softmax | 0.001 [.000, .002] | [82, 5030] | 80.6 [64.9, 97.6] | 2.01 [1.91, 2.11] |
Multi-turn task (Figure 9 & Appendix D.4): We set up ROSA under the multi-turn conversation Sokoban task as used by RAGEN (Wang et al 2025). In this task, the LLM needs to act over multiple turns (via a multi-turn conversation format) to push a box into a goal location on the map. This involves taking many actions and reasoning ahead, and there are often more than one route. We detail our experimental set-up in Appendix D.4 of the updated pdf, and highlight the main result below. As we can see, ROSA learns both higher route coverage and higher pass@k, and this generally demonstrates ROSA’s applicability to multi-turn settings.
| Method | pass@1 | pass@16 | Route coverage |
|---|---|---|---|
| Base model | 0.029 [.02, .04] | 0.161 [.11, .22] | 0.076 [.05, .10] |
| RLOO | 0.173 [.11, .24] | 0.179 [.11, .24] | 0.086 [.06, .12] |
| DARLING | 0.180 [.12, .24] | 0.221 [.16, .29] | 0.105 [.07, .14] |
| ROSA+Max@4 | 0.136 [.10, .18] | 0.393 [.31, .47] | 0.180 [.14, .22] |
| ROSA+Max@16 | 0.104 [.08, .13] | 0.464 [.39, .55] | 0.219 [.18, .26] |
I thank the authors for the responses. They addressed my concerns and questions. I will raise the score to 3 accordingly.
We thank the reviewers for their comments, and are encouraged to hear that the connection between reward uncertainty and policy diversity is intuitive, and that the proposed objective accounting for the reward function uncertainty is straightforward to optimize. We respond below.
We are happy to add discussion of the relationship between domain randomization and our work! However, we believe it is less closely related than the areas we already discuss in the paper (multi-objective RL, risk-sensitive RL, distributional RL, etc.)
Specifically, a principal aim of domain randomization is to bridge the sim-to-real gap and improve policy robustness. Typically, these approaches (i) randomize simulator dynamics, (ii) keep a fixed reward, and (iii) optimize the expectation (i.e. linear) over sampled dynamics. This is in contrast to ROSA, which aims to learn diverse policy by (ii) randomizing over rewards, and (iii) crucially, use a modified objective that depends non-linearly on the set of rewards.
We are happy to discuss further and include in the text if the reviewer deems it helpful. Currently, we’ve also re-organized our main text related works section to more clearly distinguish between existing related literatures (Section 3.3).
We agree with the reviewer’s assessment of the limitations of the current assumptions. Indeed, the theoretical analysis in this work focuses on binary, non-overlapping rewards (as we explicitly stated in Prop 3.1, Theorem 4.1 and Prop 4.2). A few notes of clarification:
Thus, our assumptions are fairly standard. The simple assumptions have also led to insights about the algorithm. Nevertheless, we agree generalizing the theory to continuous, overlapping reward functions has value and is exciting future work. To this end, we’ve included a standalone paragraph in Section 6 of the updated pdf outlining limitations to facilitate productively extending the existing results. We hope the reviewer finds this suitable and we are happy to discuss any other limitations the reviewer may have in mind!
[1] Tajwar, Fahim, et al. "Maximum likelihood reinforcement learning." arXiv preprint arXiv:2602.02710 (2026).
[2] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.
We acknowledge that the existing results are focused primarily on single-turn, verifiable MATH problems. This was done as a didactic demonstration of our claims that ROSA results in policy which is diverse along the dimensions where the reward function distribution disagrees.
We’ve added three new experiments to better demonstrate ROSA’s general usefulness. To summarize:
The detailed results are in the mentioned section in the updated pdf; we provide a summarized highlight in the response below as well.
All in all, we believe the additional results, taken together, re-affirms our central thesis that ROSA is a generally useful, widely applicable tool for inducing controllable diversity along the dimension we can specify. We thank the reviewer for the recommendations which have led to a better understanding and exciting future directions.