- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a novel self-training method for improving the ability of LLMs to follow instructions that adapt and change over the course of long conversations, using both rejection-sampled SFT with self-generated data (4.2) and DPO preference tuning which contrast well-formed answers with those which ‘relapse’ or incorrectly follow a revoked constraint (4.3).
As noted elsewhere: the font and format are wrong, but the article is well within the page limit and otherwise appropriate for the venue, so I completed a full review.
Critical strengths
This paper introduces an interesting approach to modeling and evaluating instruction following over long conversations, addressing a key problem in LLMs which tend to lose the ability to follow an active set of instructions in longer conversations.
Critical weaknesses
- From the problem definition, it is not clear why relapse is undesirable/poor instruction following except in cases where a revoked constraint contradicts an active one. For example, if I ask for a bulleted-list where all sentences start with the letter ‘A’, then revoke the bulleted-list constraint, I have not supplied a requested alternative format: it's reasonable for the next answer to still be a bulleted list even if I do not require one. To my understanding, the problem setup treats this as unsuccessful, scoring 1/1 on persistence and 1/1 on relapse. Does this explain why relapse scores are generally high for models without an oracle ledger or preference training (<~0.4)? My thought is that the improvement from preference training comes from penalizing the following of constraints in when I think a more natural interpretation of instruction following would be indifferent to .
- A significant part of the main paper (Table 3 + 6.3) discusses a replication of the main experiments with a second random seed. I think this should go into the appendix and the main paper should instead include more analysis, which seems relatively light currently. For example, what kind of constraints are followed well, exhibit the most relapse, etc? At what turn indices?
- The dataset is fairly small, but if I understand right, the main evaluation datasets are synthetic and new. Pre-registration addresses statistical significance to some degree but I think a simpler approach would be to generate more data for evaluation, or just show that the current results are statistically significant, rather than pre-registering. Overall, the evaluation data for Table 1, etc, should be explained more clearly since it is introduced in this paper.
- Only models up to 8B are evaluated. While I think this is reasonable for the training experiments, an evaluation of larger models for inference only should be included to see if the overall instruction-following problem is resolved at larger model sizes.
Questions for authors
- The paper mentions pre-registration of hypotheses and includes timestamps and hypotheses in the Appendix. Are these also pre-registered with some verifiable registry? I understand these might be hard to share anonymously, but think this would be important for actually making claims about pre-registration in a camera-ready version. Though in my opinion, a better approach would be to just generate more data to ensure the findings are statistically robust.