- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Uses factored skills where the skill affects particular factors, and then applies preference scores to prioritize more consequential executions. The preferences between different skills are determined by the preference measure. Preferences are computed using the Bradley-Terry model and modulate the mutual information lower bound by multiplying it by the sum of skill rewards for a trajectory.
Critical strengths
The paper is generally straightforward to follow and applies a straightforward application of preferences to unsupervised skill discovery.
Critical weaknesses
Language agents have the capability of creating scripted skills directly, which raises the question: if we are using the language model for the preference, why are we using an unsupervised skill discovery model for the skills and not also using the language model for the skills? The best reasoning would be that the model is unable to do skill learning but is good as a discriminator, which may be true, but is not evaluated in this work.
The figures (especially Figure 3) do a poor job of demonstrating what is being added in the context of the experiment by the preferences given by the language model. In particular, it looks like in the longer sequential multiparticle is where the best learning happens, but it is not as clear what kinds of preferences would be necessary in those tasks, nor is there any visualization or figure to represent this in the main paper.
It is not clear how to interpret the downstream reward, since the numbers appear to be quite small. The main paper also appears to be missing the performance graph for iGibson wipe, which is the only "real" domain, at least insofar as it is a realistic simulator.
Questions for authors
See Weaknesses
3 comments
Response to Authors
I thank the authors for their lengthy response, and I believe that the additions will improve the paper, though none of them significantly change my view of the core weaknesses. In particular, the logic, while sound, is ungrounded in why not just use an LM to script behaviors, and would be stronger with a direct comparison. The Figure 3 weaknesses appear to still be present, and the description of the Gibson wipe is still vague. I will maintain my score for the original reason.
Thank you very much for reading our response, for noting that the additions improve the paper, and for maintaining your positive assessment!
Following your helpful feedback, we have also updated the manuscript with two targeted clarifications:
- Visualizing preferred Multi-Particle behaviors (
Section 5.3&Appendix F.1, Figure 7): We added Figure 7 to directly show concrete 5-keyframe factor-wise preference pairs during task-agnostic pretraining (preferring coherent landmark settling, controlled interior retreat, and persistent directional transit over transient pass-by drift, boundary wall-pinning, and erratic reversals) alongside how a downstream controller composes them. - Clarifying
iGibson Wipe(Appendix E): We expanded theiGibson Wipedescription to detail the episode structure and the three physical components of its downstream reward (base workspace positioning, wiping-hand proximity/posture guidance, and dirt-mark removal bonus).
We also appreciate your suggestion regarding an empirical comparison with LLM-scripted control as a valuable direction for future work. Thank you again for your time and constructive feedback!
We thank the reviewer for finding our work relevant and impressive and for giving valuable feedback. We updated the paper with expanded related-work discussion (Section 2), paired Multi-Particle skill visualizations (Appendix F.1, Figure 7), explicit downstream reward-scale definitions (Section 5.2, Appendix E), and 7-method results including
iGibson Wipe(Figure 3(f), Table 1).1. Why Use USD + LLM Preferences Instead of Scripting Skills Directly?
Your intuition captures the core reason accurately, which we make explicit in Section 2 (
p. 3):2. What Preferences Matter in Multi-Particle and Why Gains Grow at L=8,9
Appendix G.3): During pretraining, the LLM receives no information about sequential orders, food, or poison. Given 5 sparse keyframes of an agent's motion relative to its paired landmark, the prompt favors coherent, purposeful behaviors—smooth approach, decisive landmark entry, sustained holding, or clean traversal—over aimless jitter, loops, or drifting into empty arena boundaries.Appendix F.1, Figure 7): We added a paired comparison between DUSDi and PCSD across all 5 discrete factor skills from identical resets, plotting 2D paths and landmark distance di(t). Without preferences, DUSDi treats every separable motion equally and drives multiple skills outward to slide along arena boundaries, leaving sparse coverage near the landmark. PCSD suppresses boundary wandering and allocates multiple skills to landmark entry and sustained holding (di<0.6), alongside controlled standby and retreat (Figures 7–8).SequentialL=8,9 (Table 1, Figure 3(a–c)): Sequential tasks require L agents to successively reach and hold inside their landmarks while others avoid premature entry. Because DUSDi learns fewer reliable hold/standby skills per factor, chaining L accurate skills degrades rapidly as L grows: even under zero-shot random skill sampling (Table 1), PCSD more than doubles DUSDi's positive-return access at L=8 (6.20% vs. 3.10%) and L=9 (2.50% vs. 1.00%), directly driving the larger downstream gains in Figure 3(b, c).3. Downstream Reward Scales and
iGibson WipePerformanceSection 5.2, Appendix E, Eq. 17): Following standard hierarchical evaluation (H=50 steps per skill), downstream wrappers average each segment's step rewards by 1/H, rhigh,t=H1∑h=1Hrenv,t+h (Eq. 17), keeping PPO value scales consistent. Episodic returns in Figure 3 sum these 1/H-averaged segment rewards over T/H skill selections (raw cumulative environment rewards are 50× larger). For instance, inMoMa2D(20 selections, raw step reward ≤+2), the return ceiling is 40.0 (Figure 3(g)), which PCSD nearly saturates; inMulti-Particle Sequential, completing all stages grants +10 per remaining segment, yielding returns above 100 (Figure 3(a–c)). Full definitions for all 12 tasks are in Appendix E.iGibson Wipeperformance (Figure 3(f), Table 1): Figure 3(f) in the main text reports full learning curves oniGibson Wipeacross all 7 methods (including CDP and HaSD), where PCSD achieves the highest return by a clear margin, corroborated by a 16.00% zero-shot mark-removal rate in Table 1 (vs. 2.50% for DUSDi).