- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents goal-conditioned supervised learning (GCSL) for offline fine-tuning of LLMs. It improves over previous methods like SFT, DPO and RL-based alignment in that it is offline, it does not require paired data (only feedback on individual responses), and it aims not to suffer from "average-seeking" responses. Their method conditions the LLM on natural-language goals that state a minimum level of quality, with each response used as a demonstration for every threshold it exceeds, and goals below the median are discarded. During inference, the highest level is used so that the model can extrapolate to higher-quality responses. They demonstrate performance on non-toxic generation, code generation, and recommendation.
Critical strengths
- Simple extension of SFT that can be easily integrated and provides out-of-the-box improvements while keeping SFT's benefits (offline, single responses, no reward model).
- Broad experiments on different tasks (toxicity, code generation, recommendation).
Critical weaknesses
-
Limited methodological novelty relative to reward-conditioned SFT. If I understood correctly, GCSL amounts to providing the feedback in the prompt when doing SFT with the main changes being a natural-language prompt for the goal and the "at least" relabeling. Reward-conditioned SFT already exists (e.g., Rewards-in-Context, Quark, Decision Transformers), thus the main novelties are the ones above. This makes the paper primarily empirical, so I would expect the experiments setup to be strong.
-
The experiments confound several factors, so it is hard to tell where the gains come from. From SFT to GCSL-bey-NL several things change at once: (a) data filtering (GCSL-bey discards the worst 50% of samples using the ground-truth score, with the cutoff and K apparently chosen on the evaluation split); (b) the amount of information (GCSL sees the score, SFT a fixed 25% cutoff, DPO only the largest-gap pair, etc, with many baselines missing ablations to support these choices); (c) the methodology itself (relabeling + NL goals). Already SFT to classic GCSL changes both the filtering and the score in the prompt. Without the filter, GCSL-bey-NL is worse than all the baselines on toxicity (for example DPO, Table 4: 0.156 vs 0.142 Avg.max). Baselines are run with fixed data choices (25% split, largest-gap pairs), and it is unclear whether their settings are tuned on these datasets, while the method's own filter and K are tuned on the evaluation dataset. Together with the central "average-seeking" claim being argued but not measured, the evidence is not strong enough for an empirical paper.
Questions for authors
- Were the number of bins K and the filtering ratio selected on the same split used for evaluation? If so, could you report results with these chosen on a held-out validation split?
- How do the baselines (and GCSL-bey-NL) perform when trained on the full dataset, and on the same filtered data GCSL-bey uses (top 50%)? How do the other baselines behave across data-splitting ratios, as done for SFT and GCSL?
- Was the DPO pair construction ablated? Using the best and worse responses can be seen as an easy/weak negative, while another construction, such as the two best answers, would provide a harder and more informative negative. Does this affect the DPO results?
- How does GCSL compare with offline methods that improve over SFT without paired data or a reward model, e.g., SPIN (https://arxiv.org/abs/2401.01335) or DFT (https://arxiv.org/abs/2502.18679), which build negatives from the base model's own generations? They share the practical advantages claimed here while needing less supervision (no graded feedback), so I would expect GCSL to beat these baselines.
- Since the highest goal in GCSL-bey is still trained only on the highest bin, and App. D.9 concedes the averaging bound also applies for GCSL-bey at a fixed threshold, what evidence shows that learning from the lower bins breaks the average-seeking bias (e.g., this could be partially supported by, on the training data prompts, showing the quality distribution of generated outputs vs. that of the top bin's training data)?
Minor Questions 6. What is the hit rate on the recommendation task? It is a standard metric in recommendation. NDCG/ERR with graded relevance rewards exactly what GCSL optimizes, which is fine, but it is important to know whether there is any hit-rate tradeoff. 7. Are GCSL goals complementary to preference optimization, e.g., could they be combined with DPO? 8. Online methods perform better (Table 5), and GCSL could be iterated like Quark; are the improvements constrained to the single-round offline scenario?