- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors want RL to learn behaviors the base model never produces, using the textual feedback a judge already gives. In practice they add a self-distillation KL term to GRPO, where the teacher is the same model with a hint placed before the failing turn. They test this on one internal coding setup, and the behaviors it learns are output formats such as markdown tables and status updates written as poems.
Critical strengths
Using the judge's textual feedback instead of only a scalar reward is a reasonable idea. The finding that distilling only on failing turns is more stable than always distilling is a useful practical note.
Critical weaknesses
-
I don't think the experiments show what the paper claims. The main claim is that TGRL learns the new behaviors without hurting coding performance. But nothing in the paper indicates more than one run per method, and the "error bars" in Table 1 are the standard deviation over the last 32 steps of training. The numbers are training metrics on a task set the paper never names, with no held-out evaluation. So when TGRL gets 0.33±.05 and GRPO gets 0.33±.07, that tells us nothing about whether coding was hurt. Figure 5 is worse. The TGRL run stops around step 420 while every other run goes to 500, and for most of that range its pass rate is slightly below the baseline. Why was it cut short?
-
I can't see what is new here, and I can't check the experiments either. The method is GRPO plus on-policy self-distillation (Zhao et al., 2026), and the paper credits gating distillation to failing rollouts to Li et al., Ding, and Lu et al. (2026). What is left is moving the hint to the failing turn and applying it to several criteria. If the method isn't new, the paper has to stand on its experiments. Yet the dataset and the judge model are never named, only one feedback string is shown (Figure 3), and the rubric items appear only as short column names in Table 1. There is no appendix and no code. Nobody outside the authors' group could rerun any of this.
-
I'm not convinced the baselines were run properly. LaTRO receives extra style signal on top of the coding reward, yet it does no better than the Baseline that ignores style on six of the seven criteria, and is clearly worse on Bolded (0.43 to 0.21) and Sol. Desc (0.70 to 0.57). Its pass rate also falls from 0.33 to 0.14, and Online SFT and Verifree lose a lot of pass rate too (0.26 and 0.20). A method that makes things worse on the very thing it is supposed to improve looks more like a broken implementation or poor tuning than a fair comparison. Beyond a one-sentence description of each, the paper gives no implementation or tuning details. Without that, Table 1 can't tell us whether TGRL is better or the alternatives were just set up badly.
Questions for authors
Did the GRPO baseline see the style requirements in its prompt? Markdown stays at 0.00 for both Baseline and GRPO, which looks like the model was simply never asked.
How many seeds were run, and is there any held-out evaluation?
Why does the TGRL run in Figure 5 end early, and what does pass rate do after that?
Which judge model was used, and what are the feedback strings for the style and poem criteria?
The authors did not provide any response during the rebuttal phase, so I am maintaining my score.