- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reasoning distillation from Qwen3 teachers (1.7B–8B) to small students (0.6B/1.7B). The idea is that a CoT step mixes "what to do" and "how to do it", so they prompt the teacher to write each step as an abstract action followed by a concrete execution. ACTED trains the student with SFT on verified teacher trajectories plus an on-policy part, where action lines get cross-entropy on the teacher's argmax and execution lines get forward KL. They evaluate on StrategyQA, CSQA, ARC-C, GSM8K and AQuA, and give analysis experiments.
Critical strengths
- The motivation is easy to follow, and separating the decision from the execution is a reasonable idea for small students.
- The robustness analysis is comprehensive and detailed.
Critical weaknesses
- In abstract, "Across five reasoning benchmarks, ACTED achieves the highest average Pass@1 and Pass@5 among the compared methods in every tested configuration". But in Table 5, it's not the truth.
- Pass@5 is not a very meaningful metric on these datasets. StrategyQA is yes/no, so random guessing already gives 97% Pass@5, and all methods are around 54–55% P@1 there, close to chance. So the headline 80.4->89.3 may mostly reflect more diverse sampling rather than better reasoning. For the multiple-choice datasets, the same. Also some baselines looks a little bit strange (SGFT 0.7% and Think-SFT 8.3% on StrategyQA with the 8B teacher, is this correct?).
- There are many differences between ACTED and the baselines at the same time, so I can't tell where the gain comes from: for example, the action lines, different teacher prompt, a different evaluation prompt, and different training budget. In this case, the comparison is not fair, in my view.
Questions for authors
- Could you explain the claim in the abstract given Table 5? CoT-SFT has a higher average Pass@1 than ACTED for both the 4B and 8B teachers.
- Since Pass@5 is close to saturated on yes/no and multiple-choice questions, could you report a metric like majority voting?
- Could you add a controlled comparison, e.g., CoT format with the same teacher filtering, evaluation prompt and training budget as ACTED.
2 comments
Response to authors
Thanks for the reply, but a few things still aren’t resolved. For W2, the new M@5 results actually show the gain mostly disappears: on StrategyQA voting doesn’t beat P@1, and on average you’re only 0.4 ahead of GKD. The below-chance baselines look like a broken setup rather than weak methods. And for W3, DAER still uses the action format, so the CoT control is still missing. So overall, I’ll keep my score.
We appreciate your thoughtful review and your acknowledgment that our idea and motivation are reasonable and our analysis is comprehensive! Your questions about our claim, metrics and baselines, and ablation studies helped us better demonstrate the effectiveness and interpretation of our method!
W1&Q1: In the abstract, "Across five reasoning benchmarks, ACTED achieves the highest average Pass@1 and Pass@5 among the compared methods in every tested configuration". But in Table 5, it's not true.
R1: We apologize for the confusion. The main body shows the results for the 0.6B student, where our claim is true. Table 5 in Appendix D shows results for 1.7B students, where our method is not always best but still close to the optimal one. We will revise our abstract accordingly to avoid such overclaim.
W2&Q2: Pass@5 is not a very meaningful metric on these datasets. Could you report a metric like majority voting? Also, some baselines look a little bit strange.
R2: Thanks for pointing these out!
W3&Q3: There are many differences between ACTED and the baselines at the same time, so I can't tell where the gain comes from. Could you add a controlled comparison, e.g., CoT format with the same teacher filtering, evaluation prompt and training budget as ACTED.
R3: Thanks for the suggestion!
[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.
[2] Goyal, Sachin, David Lopez-Paz, and Kartik Ahuja. "Distilled pretraining: A modern lens of data, in-context learning and test-time scaling." International Conference on Learning Representations. Vol. 2026. 2026.
[3] Yang, Jialin, et al. "Structural Rationale Distillation via Reasoning Space Compression." arXiv preprint arXiv:2605.07139 (2026).
We would be more than happy to provide further clarification or address any remaining concerns!