- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on mapping natural-language preferences to numerical weights for multi-objective reinforcement learning across different tasks and objective sets. Language2Utility combines frozen preference–objective embeddings, set attention, and ordinal and admissible-set supervision. It outperforms the baselines on four MO-Gymnasium tasks, with semantic transfer further evaluated using 75 human-written preferences. The paper also introduces a controlled benchmark and a theoretical bound relating weight-prediction error to downstream utility regret.
Critical strengths
-
This paper focuses on mapping natural-language preferences to numerical weights for multi-objective reinforcement learning across different tasks and objective sets.
-
The paper is clearly organized, and the appendix provides detailed descriptions of the architecture, data construction, and evaluation metrics.
Critical weaknesses
-
Table 3 includes only 8B direct LLM baselines. Larger open-weight models, such as 70B model, should be evaluated under the same transfer protocol to check whether they already provide comparable zero-shot transfer. Testing additional embedding backbones would also clarify whether the reported gains generalize beyond the chosen encoder.
-
Training accepts any weight vector in the admissible set, while Centroid-L1 measures distance to its centroid and the utility metrics average uniformly over the set. A valid prediction can therefore still receive a worse score. The paper should justify why this uniform weighting represents the uncertainty in users’ preferences.
-
Table 11 contains only 38 test specifications, including one for DST and four for Minecart. The 174 test sentences are different expressions of these specifications. Results across five training seeds do not address the uncertainty caused by such a small set of test preferences.
-
In Table 16, DeepSets w/ Joint has better mean results on all six Minecart metrics, including Set-L1 of 0.1539 vs. 0.1930 for the proposed method. This weakens the claim of consistent transfer gains.
-
Proposition 1 applies to any weight predictor and does not depend on the relational embeddings, set attention, or training objective. It connects prediction error to utility regret, but does not show why the proposed design should reduce prediction error on unseen tasks or objective sets.
Questions for authors
Please see the weakness part.
6 comments
Response to W2: Semantic Validity and the Uniform Utility Reference
W2. Training accepts any weight vector in the admissible set, while Centroid-L1 measures distance to its centroid and the utility metrics average uniformly over the set. A valid prediction can therefore still receive a worse score. The paper should justify why this uniform weighting represents the uncertainty in users’ preferences.
Response to W2. We appreciate the reviewer’s careful observation about set-valued validity and centroid-based evaluation.
Uniform weighting is not intended to model user-specific uncertainty. Semantic validity is defined by compatibility with the admissible set, while uniform averaging provides a symmetric aggregate reference without introducing additional user-specific probabilities over admissible weights.
Semantic validity and complementary metrics. The admissible set W(c) represents numerical preferences compatible with the language, rather than a probability distribution over users’ latent preferences. This accommodates expressions that specify qualitative priorities without uniquely determining numerical trade-offs. For example, prioritizing passenger comfort over travel efficiency in autonomous driving establishes an ordering but leaves their numerical gap unspecified. Training therefore uses the nearest-set loss
min over w ∈ W(c) of ‖ŵ(x) − w‖₁,
which treats every member of W(c) as a valid target. Any admissible prediction receives zero Set-L1.
The metrics evaluate complementary aspects of a semantic specification. Set-L1 directly measures compatibility with the full admissible set, while the other metrics capture ordinal consistency, central tendency, and behavioral relevance.
| Evaluation metrics | Reference | Interpretation |
|---|---|---|
| Set-L1 ↓ | Full admissible set W(c) | Semantic compatibility. Every admissible weight receives zero Set-L1 |
| Ordinal Accuracy ↑ | Stated ordinal constraints | Ordinal consistency. Agreement with the qualitative priority relations |
| Pair Accuracy ↑ | Return pairs unanimously ordered by W(c) | Preference-invariant behavior. Ordering shared by all admissible weights, independent of within-set averaging |
| Centroid-L1 ↓ | Admissible-set centroid w̄(c) | Central tendency. Not a semantic-validity criterion |
| Language utility ↑ | Uniform average over W(c), equivalently w̄(c) | Realized behavioral utility. Utility of the controller return under a symmetric aggregate reference, not an estimated user-specific distribution |
| Language regret ↓ | Best empirical utility under the same aggregate reference | Behavioral shortfall. Gap to the best return available in the fixed controller library |
An admissible prediction away from the centroid can therefore obtain zero Set-L1 but nonzero Centroid-L1. This is not a contradiction: the former evaluates semantic validity with respect to the full set, whereas the latter summarizes distance from its center. Pair Accuracy further evaluates only preference-invariant behavioral comparisons.
Uniform utility reference and its scope. Uniform averaging provides a symmetric card-level reference across compatible realizations without introducing additional user-specific probabilities over admissible weights. It serves as an aggregate evaluation reference rather than an estimate of a user-specific uncertainty distribution. Under linear utility,
mean over w ∈ W(c) of (w · R) = w̄(c) · R,
so the uniformly averaged utility is exactly the utility evaluated at the admissible-set centroid. We now state this equivalence in Appendices A.3 and C.3 and clarify that it defines the aggregate utility reference, not semantic validity.
Controlled comparison with centroid supervision. Table 2 and Appendix D.3 compare the full model with Ordinal + Centroid, which retains the same architecture and ordinal loss but replaces admissible-set supervision with regression to w̄(c). In-domain, set-valued supervision achieves lower Set-L1, .1136 versus .1272, with similar Ordinal Accuracy and Pair Accuracy. Centroid regression obtains lower Centroid-L1 and utility regret because its target directly matches the centroid-based evaluation reference. Under leave-one-domain-out (LODO) transfer, the two variants have similar Set-L1, while set-valued supervision yields slightly better average Ordinal Accuracy, Pair Accuracy, realized utility, and utility regret. These results make the expected metric alignment explicit rather than treating the centroid as the unique semantically correct target.
Overall, uniform weighting provides a symmetric aggregate utility reference, while Set-L1, Ordinal Accuracy, and Pair Accuracy evaluate the full admissible set and its ordinal implications.
Revision. The revised Sections 3.1 and 4.1 and Appendices A.3, C.3, and D.3 make these distinctions explicit. Table 2 reports the controlled comparison.
Response to W3: Expanded Semantic-Card Coverage and Statistical Uncertainty
W3. Table 11 contains only 38 test specifications, including one for DST and four for Minecart. The 174 test sentences are different expressions of these specifications. Results across five training seeds do not address the uncertainty caused by such a small set of test preferences.
Response to W3. We thank the reviewer for highlighting the importance of distinguishing linguistic variation, semantic-card coverage, and training-seed variability.
We expanded the leave-one-domain-out (LODO) evaluation from 38 held-out semantic cards (174 language realizations) to all 386 benchmark cards (1,759 language realizations), and added stratified paired card-level bootstrap intervals to quantify variation across semantic specifications.
Aggregation unit and sources of variation. In both the original and expanded evaluations, the semantic card is the aggregation unit. Metric values are first averaged over the language realizations of each card, after which all cards receive equal weight. The original 174 language realizations therefore evaluate robustness to different phrasings of 38 distinct semantic preferences. The five training seeds quantify training variability, while the expanded card-level bootstrap analysis additionally quantifies variation across semantic specifications.
Expanded coverage and paired analysis. Under LODO evaluation, the entire target domain is excluded from training, so every evaluated target-domain card is unseen by the grounding model. The expanded evaluation covers all 386 semantic cards retained in the benchmark. This includes all 7 Deep Sea Treasure (DST) cards and all 43 Minecart cards induced by the importance-role rules, together with 144 Lunar Lander and 192 Fruit Tree cards. For each card, we average over its language realizations and compute the paired difference between Language2Utility and DeepSets w/ Joint, the closest set-based alternative, across five training seeds. We estimate pointwise 95% percentile intervals using 100,000 paired card-level bootstrap replicates stratified by domain.
Expanded LODO results from Table 18 over all 386 semantic cards (1,759 language realizations). Paired gain is defined in the favorable direction, so positive values favor Language2Utility. Intervals are pointwise 95% percentile intervals from the stratified card-level bootstrap. Bold indicates the better mean.
Semantic metrics
| Method | Set-L1 ↓ | Centroid-L1 ↓ | Ordinal Accuracy ↑ |
|---|---|---|---|
| Language2Utility | .3237 | .3458 | .7763 |
| DeepSets w/ Joint | .3418 | .3672 | .7576 |
| Paired gain Δ | .0181 | .0215 | .0187 |
| 95% card-level interval | [.0071, .0289] | [.0102, .0325] | [.0088, .0290] |
Behavioral metrics
| Method | Pair Accuracy ↑ | J(lang) ↑ | Reg(lang) ↓ |
|---|---|---|---|
| Language2Utility | .9191 | .6439 | .0328 |
| DeepSets w/ Joint | .8942 | .6350 | .0417 |
| Paired gain Δ | .0249 | .0089 | .0089 |
| 95% card-level interval | [.0187, .0314] | [.0058, .0121] | [.0058, .0121] |
Language2Utility obtains favorable paired gains on all six metrics, and every card-level 95% interval lies entirely above zero. It reduces Set-L1 and Centroid-L1 by .0181 and .0215 and improves Ordinal Accuracy and Pair Accuracy by 1.87 and 2.49 percentage points. This more than tenfold expansion in semantic-card coverage supports the aggregate transfer advantage under broader preferences and directly quantifies specification-level uncertainty beyond training-seed variation.
Revision. We added this analysis in Appendix E.2 with Table 18 and clarified the card-level aggregation procedure in Appendix C.1.
Response to W1: Larger Open-Weight LLMs and Embedding Backbones
Response to W1. We thank the reviewer for suggesting larger open-weight models and alternative embedding backbones.
We added a 72B open-weight direct large language model (LLM) and two additional frozen embedding backbones under the same transfer protocol.
Larger open-weight direct LLM comparison. Revised Appendix D.1 and Table 12 include Qwen2.5-72B-Instruct alongside the two 8B-scale direct LLMs. All direct LLMs use the same test examples, prompting protocol, and averaging procedure as the leave-one-domain-out (LODO) comparison.
LODO results on representative semantic, ordering, and regret metrics. Table 12 reports all six metrics.
Qwen2.5-72B-Instruct improves over both 8B models on all six metrics, confirming the benefit of greater open-weight model scale. Language2Utility nevertheless obtains the better result on every metric in Table 12, including lower Set-L1, .2996 versus .3358, and lower language-conditioned regret, .0343 versus .0371. The transfer advantage therefore persists at the 72B scale.
Consistent gains across embedding backbones. Newly added Appendix F.3 and Table 20 repeat both in-domain and four-fold LODO evaluations using text-embedding-3-large and gemini-embedding-2 in addition to Qwen3-Embedding-8B. For each frozen encoder, we compare Language2Utility with its parameter-matched DeepSets w/ Joint counterpart over five seeds.
Set-L1 comparison across frozen encoders. Table 20 reports standard deviations and all six metrics.
Language2Utility achieves the better mean on all six metrics for every encoder and evaluation setting. Its Set-L1 reduction ranges from 10.2% to 23.6% in-domain and from 8.4% to 8.6% under LODO. Although absolute performance varies across encoders, the consistent within-backbone gains show that the architectural advantage extends beyond the default language encoder.
Overall, the expanded comparisons show that the transfer advantage over open-weight direct LLMs persists from 8B to 72B and that the architectural advantage persists across three frozen embedding backbones. Language2Utility also produces simplex-valued outputs without autoregressive decoding and achieves approximately 20× lower median latency than the locally evaluated 8B direct LLMs.
Revision. We updated Sections 4.1, 4.2, and 4.3.1 and Appendix D.1, and added Appendix F.3 with Table 20.