- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes ODR, which optimizes the worst-performing objective for each context, measured as relative improvement over the logging policy, and trains by using that objective as the policy-gradient reward. Experiments on a 10-item synthetic environment and KuaiRec report the best worst-objective improvement across intent mixtures.
Critical strengths
The objective is simple, clearly described, and requires no change at serving time.
Critical weaknesses
W1. The KuaiRec experiment seems to favor ODR by design. For each user, ODR selects the candidate ranking that maximizes the worst-objective score and is then evaluated on the same metric. Its advantage over the baselines is therefore largely expected and does not necessarily demonstrate its effectiveness in real-world recommendation.
W2. The core idea is to identify the worst-performing objective during training and update the model accordingly. However, the KuaiRec experiment simply selects from pre-generated rankings, while the synthetic experiment directly optimizes a differentiable surrogate rather than using the proposed policy-gradient procedure. Thus, the proposed training algorithm is not directly evaluated, leaving its practical effectiveness unclear.
W3. Relevance is typically the primary objective in recommendation systems. It remains unclear how much relevance ODR sacrifices to improve weaker objectives and whether this trade-off is worthwhile.
Questions for authors
Q1. Is ODR's superior performance due to effective learning, or simply because it directly selects rankings that maximize the evaluation metric?
Q2. What happens if a recommendation model is actually trained using the proposed ODR algorithm and evaluated on unseen users and items, rather than selecting from 45 pre-generated rankings?
4 comments
We would like to kindly remind the reviewer that we have submitted a revised manuscript, with all changes highlighted in red, addressing the key concerns raised in the review. In particular, we have clarified the neural re-ranking policy learning setup, the rationale for evaluating worst-objective robustness, and the relevance–robustness trade-offs already examined in our experiments.
We would appreciate any remaining questions or substantive objections before the discussion deadline, so that we can address them directly. If no further concerns are raised, we believe the major issues identified in the review have been satisfactorily resolved and hope that these revisions will be reflected in a more positive final assessment.
Thank you for your response. Since my main concerns remain unaddressed, I am keeping my score.
W1. I do not object to evaluating a method on its target criterion. However, on KuaiRec, ODR selects the ranking that maximizes the worst-objective score directly on the same users it is evaluated on. This advantage is therefore guaranteed by construction and says little about whether the underlying learning principle actually generalizes.
W2. The response states that a neural re-ranking policy is trained, whereas the paper describes the KuaiRec experiment as selecting the best pre-generated ranking per user—please clarify which is accurate. If a policy is indeed trained, performance on unseen users (Q2) would be the key evidence needed. Additionally, the synthetic experiment aspect of W2 remains unaddressed: because it optimizes a differentiable surrogate, the proposed policy-gradient procedure is still untested.
W3. The figures show that a trade-off exists, not that it is worthwhile, and the drop in relevance appears substantial. The -weighted variant mentioned in the response has not been evaluated; plotting a relevance vs. worst-objective trade-off curve for this variant would directly address this concern.
We sincerely thank the reviewer for the follow-up comments. However, we believe that the remaining concerns largely stem from misunderstandings of our experimental protocol and the central research objective of ODR. We clarify these points below.
W1. Generalization in the KuaiRec experiment
We respectfully disagree with the reviewer's characterization of our experimental setup. In the KuaiRec experiments, users are explicitly divided into training and test sets. The respective recommendation policies are learned using the training users and subsequently evaluated on held-out test users. Therefore, ODR does not optimize rankings using the evaluation feedback of the same users on which its performance is reported. Contrary to the reviewer's assertion, the observed improvement on test users is not guaranteed by construction, but reflects the performance of a learned policy on unseen users. Our experimental protocol thus directly evaluates generalization, and we do not believe the concern raised in W1 applies to our actual experiments.
W2. Learning a parameterized recommendation policy
As we have already clarified in the revised manuscript, ODR learns a parameterized neural re-ranking policy from training data, rather than merely selecting the highest-scoring pre-generated ranking for each evaluation user. Its performance is then measured on held-out users, directly addressing the reviewer's request for evidence of generalization. We therefore respectfully disagree that our experiments fail to evaluate a learned recommendation policy.
Regarding the synthetic experiments, we have explicitly explained that they use differentiable soft-ranking surrogates to implement the same active-worst-objective optimization principle. Although we do not separately benchmark the score-function estimator presented as an alternative gradient implementation, our contribution is the robust optimization formulation and its context-dependent learning principle, not a new score-function estimator. The synthetic experiments already provide empirical evidence for this proposed learning principle, and the absence of a separate score-function benchmark does not invalidate that evidence.
W3. Relevance–robustness trade-off
We believe this concern reflects a misunderstanding of our central research objective. ODR is designed to improve the worst-performing objective under unknown and potentially varying user intent distributions, rather than to optimize the trade-off between relevance and other recommendation metrics. Relevance itself can be included among the plausible objectives, and our formulation protects against poor performance on whichever objective is weakest for a given user.
We previously discussed a relevance-weighted extension in response to the reviewer's interest in controlling this trade-off. However, this extension is not the motivation or primary contribution of our paper, and evaluating its trade-off curve would address a different research question. Our existing experiments already report individual metric performance to make the consequences of worst-objective optimization transparent. We therefore do not believe that the absence of experiments on an optional relevance-weighted variant constitutes a limitation of the proposed method.
Overall, we believe that W1 and the principal concern in W2 are addressed by clarifying our actual learning and held-out evaluation protocol, while W3 requests an analysis beyond the explicitly stated objective of our work. We respectfully ask the reviewer to reconsider these concerns in light of the clarified experimental setup and the intended scope of our contribution.
We sincerely thank the reviewer for the constructive feedback and for recognizing the practical motivation and potential value of our approach. We appreciate the opportunity to clarify our experimental design and the intended scope of our empirical evaluation.
1. Does optimizing the evaluation metric favor ODR?
We thank the reviewer for raising this question. However, we respectfully disagree that alignment between the optimization objective and the evaluation criterion constitutes a methodological weakness.
Our primary research question is whether explicitly protecting the weakest plausible objective provides a useful alternative to optimizing a single objective or a fixed scalarization. Accordingly, evaluating worst-objective performance is the direct and appropriate way to assess whether this robustness principle achieves its intended goal.
The same reasoning applies to conventional recommendation methods. Relevance-oriented methods are evaluated using relevance metrics, while diversity-oriented methods are evaluated using diversity metrics. Evaluating an algorithm against the objective it is designed to optimize is not inherently problematic, provided that the resulting claims are interpreted appropriately.
Importantly, we do not rely exclusively on worst-objective performance. Figures 2(b) and 4(b) additionally report individual metric improvements, allowing readers to examine the trade-offs between relevance and discovery-oriented objectives.
We therefore believe that the experiment provides a valid comparison of alternative optimization objectives within the specified candidate class. It is not intended to establish causal superiority in real-world user satisfaction, and such a claim is not necessary to support our central contribution.
2. Training an actual recommendation model
We thank the reviewer for this question. We believe the concern stems from a misunderstanding of the role of candidate generation and re-ranking in our experiments. Our method is not intended to learn the candidate generation process itself. Rather, ODR optimizes a parameterized re-ranking policy after candidate generation, which is a standard and practically important stage in recommendation systems.
In the KuaiRec experiments, we construct a common candidate-ranking pool and optimize a neural re-ranking policy according to the proposed worst-objective. Thus, the candidate generation process is fixed, but the final re-ranking decisions are optimized using our proposed objective, rather than simply adopting a predetermined ranking. This design intentionally isolates the effectiveness of the proposed optimization principle from differences in candidate generation.
We therefore respectfully disagree that our experiments merely evaluate pre-generated rankings without optimizing a recommendation policy. Nevertheless, we recognize that the distinction between candidate generation and re-ranking may not have been sufficiently clear in the current manuscript, and we will clarify this experimental setup to avoid misunderstanding.
3. Relevance–robustness trade-off
We thank the reviewer for emphasizing the importance of relevance. However, we would like to point out that the trade-offs between relevance and other objectives are already explicitly examined in our experiments.
Figures 2(b) and 4(b) report individual metric improvements, including relevance-oriented metrics. These results allow readers to assess how ODR balances relevance against diversity, novelty, and serendipity, rather than evaluating the worst-objective metric in isolation.
Moreover, the possibility of sacrificing some relevance to protect weaker objectives is not an unintended limitation, but an inherent and deliberate property of the robust optimization formulation. Our objective is designed for settings in which severe failures on less frequent user priorities are considered costly.
The relative importance of relevance and robustness ultimately depends on the application's utility structure. Importantly, our proposed objective can be easily extended to accommodate different application-specific preferences, for example, by imposing a minimum relevance constraint or optimizing a weighted combination of relevance and worst-objective performance. Such extensions allow practitioners to flexibly control the trade-off between relevance and robustness. However, no specific threshold, weighting scheme, or alternative utility criterion is suggested in the review.
Without an explicit criterion for determining an acceptable trade-off, it is difficult to conclude that the observed relevance sacrifice constitutes a methodological weakness. Our experiments already expose the relevant trade-offs, while the proposed formulation provides a principled alternative for applications prioritizing protection against severe objective-level failures.