- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper points at a problem that mentions that for RLVR, pass@K metrics measure whether a model is able to preserve correctness, but not whether it is capable of preserving diverse solution strategies. The authors experimentally demonstrate that even though GRPO is able to preserve pass@K in their settings, the diversity of solutions reduces significantly.
For this, they provide a method STRATA which preserves such rare solution strategies via a weighted SFT-like objective on a teacher's rollouts during the online RL phase. The authors demonstrate improvements using this technique over other training objectives such as vanilla GRPO and variants of On policy distillation (OPD).
Critical strengths
The papers strength is that the idea is simple enough to use in practice, and can potentially be useful in preserving coverage measured by in solution strategy diversity. The experimental design is also clear and evaluations are done on multiple benchmarks.
Critical weaknesses
There are a couple of weaknesses in the paper:
- It is very difficult to parse the paper owing to many undefined terms such as 'survival', 'dose-response decoupling', 'cadence', 'audit', and 'anchors'.
- The experiments use Qwen3-1.7B which I'm assuming is the thinking version of the model and not the base model. However, this points to a clear flaw in the experimental design which is the choice of number of generated tokens during inference which is 1024. Thinking model generates far longer CoT of the order of 32k tokens, and this is required to match the official numbers of the Qwen3 technical report. Such a short context is only going to train the model to compress its CoT and not learn anything meaningful. If the authors wish to show genuine improvement, it would be a better choice to train the base model which has much shorter CoT and can learn meaningfully.
- The experiments use a teacher model, which is Qwen3-8B in their experimental setup. I wonder if this choice changes the interpretation of the results. For instance, can the teacher model be an earlier checkpoint of the student model itself which is more diverse in terms of solution strategies. It would be impressive if the authors can demonstrate gains in settings where there is no need for extra models to generate diverse traces.
My concerns are written in the same order of their priority, 1 > 2 > 3. 3 is not a critical weakness but I would appreciate if the authors could shed more light on this question. Overall I like the idea but the execution and especially the writing need more consideration.
Questions for authors
My primary question would be point 3 in the weaknesses. How would the results look like if there is no separate teacher model?