- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper identifies and attacks an interesting, quite niche but application-general problem in LLMS, whereby they take decisive action without gathering all the useful information first. The authors basically claim that this is inefficient to address as a side-effect of standard post-training and can benefit from targeted intervention, and this is the subject of the rest of the paper. They start by formalising the problem, and proposing a “true” way of measuring the usefulness of gathering information about a particular precondition: by rolling out multiple times with and without the precondition having been checked. Then they propose a cheaper “proxy” method for this: just insert a statement that the condition has/hasn’t been checked into the LLM’s context and see how much this influences its next action. Finally, armed with this cheap method, they propose an augmentation of GRPO that additionally rewards taking actions that are identified as useful by this cheap method. This method is quite similar to a type of self-distillation policy optimisation. The authors show that their method helps somewhat in various settings with a small model, and provide various supporting empirical evidence for all of the methods they propose.
Critical strengths
- The problem is relevant and real, and an instance of a particular behaviour that i may be desirable to correct for in post-training.
- The authors have correctly identified that some kind of process reward / denser feedback signal would be helpful here (as is generally the case across all sorts of post-training problems today).
- The experimental sequence is generally very strong and the authors have provided substantial empirical evidence for the validity of most of their claims (albeit at a toy scale).
- The method proposed, although there are some aspects that feel brittle (e.g. the extraction and identification of particular candidate prerequisites), is generally quite compatible with successful post-training techniques recently (most notably SDPO). The authors have nicely framed their training procedure using an additional KL-divergence loss term.
- The results are quite promising, though it is hard to tell how significant they are.
Critical weaknesses
- The presentation of the paper is quite laden with terminology, and therefore dense and difficult to read. It would be an easier read if the authors leant into “simple English” and using the least possible additional buzzwords to describe their methods.
- I am not convinced about the usefulness of the formalisation. Probably this is not useful as a standalone contribution and is mainly an instrument for conveying the problem and methods to the reader, and I generally find plain English plus some minimal notation is more effective at this.
- The experiments are somewhat toy, though this may be due to resource constraints.
- The related work section is currently very brief and does not, in my view, adequately address the relation of these techniques to broader self-distillation literature.
- The measurement of the action distribution probabilities (length-normalised log-probs) seems a bit iffy to me – I would be interested in more discussion or evidence of the reliability of this method.
- There is not much discussion, or indeed a limitations section, and I believe this is lacking and the work could be made stronger by being placed into context a bit better.
Questions for authors
- Do you think the presentation of the paper could be made more effective with more context and discussion and less dense, coined terminology?
- What are your thoughts on the robustness of measuring action probabilities using length-normalised log-probs in the way you do? Did you try any alternatives / do you have any empirical evidence here?