acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Do and How to Do It: Distilling the Decisions Behind Chain-of-Thought Makes Reasoning Robust

Abstract

Distilling multi-step reasoning from large language models into smaller students commonly relies on chain-of-thought (CoT) traces. Yet a CoT step shows how an operation was carried out while leaving the choice of operation implicit, requiring the student to infer what to do from a problem-specific execution. We propose action-execution reasoning, which factorizes each step into a latent *action* that specifies what to do and an *execution* that specifies how to do it. Building on this representation, **ACT**ion–**E**xecution **D**istillation (ACTED) combines supervised learning from verified teacher trajectories with teacher supervision on student rollouts, coupling reliable demonstrations of correct reasoning with corrective guidance at states encountered by the student. Across five reasoning benchmarks, ACTED achieves the highest average Pass@1 and Pass@5 among the compared methods in every tested configuration, distilling Qwen3 teachers of 1.7B–8B parameters into 0.6B-1.7B students. With a 1.7B teacher and 0.6B base student, ACTED improves StrategyQA Pass@5 from 80.4% for the strongest baseline to 89.3%. Diagnostic and controlled perturbation experiments further indicate reduced execution uncertainty and improved robustness to noisy reasoning history.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
reasoning distillation, reasoning trajectory, robust reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z9edOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reasoning distillation from Qwen3 teachers (1.7B–8B) to small students (0.6B/1.7B). The idea is that a CoT step mixes "what to do" and "how to do it", so they prompt the teacher to write each step as an abstract action followed by a concrete execution. ACTED trains the student with SFT on verified teacher trajectories plus an on-policy part, where action lines get cross-entropy on the teacher's argmax and execution lines get forward KL. They evaluate on StrategyQA, CSQA, ARC-C, GSM8K and AQuA, and give analysis experiments.

Critical strengths
  • The motivation is easy to follow, and separating the decision from the execution is a reasonable idea for small students.
  • The robustness analysis is comprehensive and detailed.
Critical weaknesses
  • In abstract, "Across five reasoning benchmarks, ACTED achieves the highest average Pass@1 and Pass@5 among the compared methods in every tested configuration". But in Table 5, it's not the truth.
  • Pass@5 is not a very meaningful metric on these datasets. StrategyQA is yes/no, so random guessing already gives 97% Pass@5, and all methods are around 54–55% P@1 there, close to chance. So the headline 80.4->89.3 may mostly reflect more diverse sampling rather than better reasoning. For the multiple-choice datasets, the same. Also some baselines looks a little bit strange (SGFT 0.7% and Think-SFT 8.3% on StrategyQA with the 8B teacher, is this correct?).
  • There are many differences between ACTED and the baselines at the same time, so I can't tell where the gain comes from: for example, the action lines, different teacher prompt, a different evaluation prompt, and different training budget. In this case, the comparison is not fair, in my view.
Questions for authors
  • Could you explain the claim in the abstract given Table 5? CoT-SFT has a higher average Pass@1 than ACTED for both the 4B and 8B teachers.
  • Since Pass@5 is close to saturated on yes/no and multiple-choice questions, could you report a metric like majority voting?
  • Could you add a controlled comparison, e.g., CoT format with the same teacher filtering, evaluation prompt and training budget as ACTED.
2 comments

We appreciate your thoughtful review and your acknowledgment that our idea and motivation are reasonable and our analysis is comprehensive! Your questions about our claim, metrics and baselines, and ablation studies helped us better demonstrate the effectiveness and interpretation of our method!

W1&Q1: In the abstract, "Across five reasoning benchmarks, ACTED achieves the highest average Pass@1 and Pass@5 among the compared methods in every tested configuration". But in Table 5, it's not true.

R1: We apologize for the confusion. The main body shows the results for the 0.6B student, where our claim is true. Table 5 in Appendix D shows results for 1.7B students, where our method is not always best but still close to the optimal one. We will revise our abstract accordingly to avoid such overclaim.

W2&Q2: Pass@5 is not a very meaningful metric on these datasets. Could you report a metric like majority voting? Also, some baselines look a little bit strange.

R2: Thanks for pointing these out!

  1. Rationale for using Pass@5: As discussed in Sec.5.1, P@1 and P@5 measure different aspects. Besides the diversity of sampling, P@5 also represents the reasoning capacity boundary, while P@1 measures the sampling efficiency [1]. Since the datasets require multi-step reasoning, P@5 measures the transfer of the teacher’s solution modes [2].
  2. Majority voting results: The table below shows the results of majority voting from five sampled answers (M@5). Our method still outperforms others in most baselines and on average. | Method | StrategyQA | CSQA | ARC-C | GSM8K | AQuA | Avg | |---|---|---|---|---|---|---| | ACTED | 54.7 ± 0.8 | 65.7 ± 0.3 | 71.0 ± 0.7 | 62.9 ± 0.5 | 54.3 ± 0.8 | 61.72 | | Freeform | 53.7 ± 1.4 | 61.3 ± 0.4 | 66.4 ± 0.1 | 62.8 ± 0.4 | 51.0 ± 2.8 | 59.04 | | GKD | 54.2 ± 1.3 | 65.1 ± 0.4 | 68.3 ± 1.1 | 61.4 ± 0.7 | 57.5 ± 4.3 | 61.30 | | CoT-SFT | 52.5 ± 2.7 | 65.8 ± 0.9 | 69.1 ± 1.8 | 59.7 ± 1.2 | 55.5 ± 4.9 | 60.52 | | D-RPC | 51.5 ± 0.4 | 61.8 ± 1.1 | 66.4 ± 0.8 | 59.9 ± 0.2 | 47.4 ± 3.0 | 57.4 |
  3. Baselines results: Our experiments follow D-RPC [3] to use 2,048 as the token length limit. Three weak baselines are due to different reasons. Students trained with Think-SFT and Scaffold usually generated repeated sentences, consuming the token budget and blocking the reasoning, leading to no answer being generated. SGFT requires the teacher to write detailed guidance for the student, which makes it hard for the student to understand and is prone to consuming tokens and making mistakes. Please see R1 from Reviewer nKYC for more results.

W3&Q3: There are many differences between ACTED and the baselines at the same time, so I can't tell where the gain comes from. Could you add a controlled comparison, e.g., CoT format with the same teacher filtering, evaluation prompt and training budget as ACTED.

R3: Thanks for the suggestion!

  1. In Table 4 in the paper, we conduct an ablation study on different components of ACTED, where DAER is the offline version of ACTED, e.g., the CoT format with the ACTED training recipe. It shows that the online branch improves up to 3% accuracy on multi-hop reasoning and question-answering tasks, with lower variance. We will rename DAER as ACTED-offline to avoid confusion.
  2. Besides, the benefit of ACTED is not only the performance improvement, but also robustness to the noisy reasoning history. As shown in Sec.5.3, compared to offline SFT, ACTED with an on-policy branch can significantly improve robustness to the noisy reasoning history.

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

[2] Goyal, Sachin, David Lopez-Paz, and Kartik Ahuja. "Distilled pretraining: A modern lens of data, in-context learning and test-time scaling." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Yang, Jialin, et al. "Structural Rationale Distillation via Reasoning Space Compression." arXiv preprint arXiv:2605.07139 (2026).

We would be more than happy to provide further clarification or address any remaining concerns!

Reviewer_z9edOpenReview ↗

Response to authors

Thanks for the reply, but a few things still aren’t resolved. For W2, the new M@5 results actually show the gain mostly disappears: on StrategyQA voting doesn’t beat P@1, and on average you’re only 0.4 ahead of GKD. The below-chance baselines look like a broken setup rather than weak methods. And for W3, DAER still uses the action format, so the CoT control is still missing. So overall, I’ll keep my score.

Reviewer_TCUNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

A CoT trace shows the steps used to answer a problem, but information about why each step (action) was chosen is kept implicit. ACTED aims to improve strong-to-weak distillation by helping a student explicitly learn the underlying action distribution via a two-armed GKD distillation approach. Across a suite of qwen student/teacher pairs, ACTED yields strong performance gains.

Critical strengths

The main strengths of this paper are 1) the intuitive motivation behind why we should learn the action distribution, and 2) empirical performance gains. The paper lays out the clear explanation that a vanilla CoT trace hides the critical information about why each action was taken to solve a problem + provides a clear means to learn the underlying action distribution via GKD. The empirical gains from their approach are consistent, and the author's inclusion of 3 seed run stds is appreciated.

Critical weaknesses

As it stands, we don't have a measure on if ACTED generalizes to non-qwen models. I would raise score upon seeing experiments showing ACTED's gains remain when applied to an alternate model family.

Questions for authors

Is there any idea on how performance compares with a compute matched GRPO run on the student?

1 comment

Thank you for your careful review, constructive feedback, and for acknowledging the motivation and performance gains of our method! Your questions about model families and other baselines helped us better explain the effectiveness and practicality of this paper!

W1: As it stands, we don't have a measure of whether ACTED generalizes to non-Qwen models. I would raise the score upon seeing experiments showing ACTED's gains remain when applied to an alternate model family.

Q1: Is there any idea on how performance compares with a compute-matched GRPO run on the student?

R1: Thanks for the suggestion! We have conducted experiments that distill Llama3.2-1B student from Llama3.2-3B-instruct teacher. Based on the suggestion from Reviewer z9ed (W2), we use Pass@1 and MajorityVote@5 as the metrics. The results below show that ACTED consistently outperforms other main baselines. We will revise our paper later to include more Llama models and GRPO results.

MethodMetricStrategyQACSQAARC-CGSM8KAQuA
Pass@1
ACTED (Ours)P@155.057.848.522.517.3
GKDP@146.155.645.819.912.6
CoT-SFTP@123.454.239.518.114.2
MajorityVoting@5
ACTED (Ours)M@560.762.352.125.025.2
GKDM@556.358.147.923.825.6
CoT-SFTM@554.159.549.421.222.8

Please let us know if there are any remaining concerns that we are glad to further clarify!

Reviewer_nKYCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a distillation approach that distills the action-execution reasoning rather than the raw trajectory from a teacher model, which splits each chain-of-thought step into two parts. The action is a short, free-form, abstract description of the next operation. The execution carries that operation out on the specific problem. The proposed method, ACTED, is evaluated using Qwen3 teachers (1.7B/4B/8B), 0.6B and 1.7B students on five benchmarks: StrategyQA, CSQA, ARC-C, GSM8K, and AQuA.

Critical strengths
  1. The idea of separating "what to do" from "how to do it" at every step is straight-forward.
  2. Broad evaluation: The paper covers three teacher sizes, base and instruction-tuned students, and five benchmarks from three domains.
Critical weaknesses
  1. Weak baselines: Several baselines (Think-SFT, SGFT, Scaffold) score extremely low overall and even far below chance on StrategyQA and AQuA (e.g., SGFT 0.7% P@1 on binary StrategyQA with the 8B teacher), and their accuracy falls as the teacher grows. This suggests their outputs often fail to reach a parseable final answer, possibly because generation is cut off. This might be due to the token limit at evaluation time, the teacher model's outputs are truncated by the token limit, or something else. Without further analysis and explanation, comparisons against these baselines are hard to interpret.
  2. Unclear what contributes to the improvements from the proposed method. Based on the ablation study, the variant without the proposed on-policy distillation performs similarly to the full method, so SFT rather than the on-policy distillation is the main contribution. Also, there is a potential difference in data curation between ACTED and baseline methods like CoT-SFT. ACTED filters out teacher trajectories based on correctness, while CoT-SFT is trained on a single teacher rollout per question with no correctness filtering described for it. Given these differences, it's unclear which parts contribute to the gap between the baselines and the proposed method.
Questions for authors

See the weaknesses.

2 comments

We appreciate your thoughtful review and your acknowledgment that our idea is straightforward and our evaluation is broad. Your questions about baselines and ablation studies helped us strengthen the paper!

W1: Weak baselines. Several baselines score low overall. This suggests their outputs often fail to reach a parseable final answer, possibly because generation is cut off.

R1: Thanks for the comments! Our experiments follow D-RPC [1] to use 2,048 as the token length limit.

  1. First, we clarify that Think-SFT requires the thinking mode of Qwen models, so it is an unfair comparison, as it needs more budget.
  2. Second, the reason for the failure of Think-SFT and Scaffold is that the student learned to output lots of repeated sentences, such as:

I think I might have confused the birth year. Let me think. Oh right, Chuck Norris was born in 1950. Wait, no,I think he was born in 1950, but maybe I'm mixing up the years. Wait, no, I think he was born in 1950, but maybe he was born in a different year […] But wait, 2024 is a leap year, so the year is 2024. So, in 2024, he would be 74 years old. But wait, 2024 is a leap year, so the year is 2024. So, in 2024, he would be 74 years old [...]

Furthermore, the table below shows the ratios of repetition and failure caused by repetition in these two methods:

MethodStrategyQACSQAARC-CGSM8KAQuAAll
Repitition
Think-SFT40.2%53.3%48.0%28.8%60.4%43.5%
Scaffold53.7%53.3%41.0%25.4%69.7%43.2%
Failure caused by repetition
Think-SFT32.5%37.2%65.5%60.5%87.4%52.9%
Scaffold64.1%70.3%53.3%41.9%88.6%60.1%
  1. For SGFT, it requires the teacher to write detailed guidance before the solution, which is hard for the student to learn. Thus, in generation, students consume more tokens, are harder to understand, and are more prone to make mistakes.

W2: Unclear what contributes to the improvements from the proposed method. SFT rather than on-policy distillation is the main contribution. Also, there is a potential difference in data curation between ACTED and baseline methods like CoT-SFT.

R2: Thanks for pointing this out!

  1. In Table 4, we conduct an ablation study on different components of ACTED, where DAER is the offline version of ACTED, e.g., the CoT format with the ACTED training recipe. It shows that the online branch improves up to 3% accuracy on multi-hop reasoning and question-answering tasks, with lower variance. We will rename DAER as ACTED-offline to avoid confusion.
  2. Besides, the benefit of ACTED is not only the performance improvement, but also robustness to the noisy reasoning history. As shown in Sec.5.3, compared to offline SFT, ACTED with an on-policy branch can significantly improve robustness to the noisy reasoning history.

[1] Yang, Jialin, et al. "Structural Rationale Distillation via Reasoning Space Compression." arXiv preprint arXiv:2605.07139 (2026).

We welcome any further questions or feedback and would be happy to provide additional clarification!

Thank you for the response. My concern for the low baseline scores remains, since the reported scores differ too much from the performance reported in the original papers, and the provided explanation does not explain why the repetition problem appears in your setting but not in the previous papers. It's possible that the repetition issue also exists in the teacher models (which has been reported openly for Qwen3 models), but the proposed approach somehow gets around the problem when distilling teacher trajectories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.