acceptodds
Under review as a conference paper at ICLR 2027

Cross-Format Transfer for Distilling Long-Horizon Policies

Abstract

Reasoning models become long-horizon agents through reinforcement learning or, more cheaply, through fine-tuning on a stronger teacher's trajectories, which contain the teacher's chain-of-thought as well as its actions. The standard fine-tuning approach imitates both, yet the strongest teachers are closed models whose chain-of-thought is hidden behind an API. We show that the chain-of-thought can be left out altogether and that the long-horizon policy can be distilled from the teacher's actions alone. The student is trained on these trajectories with its reasoning block left empty, while a KL loss on a replay buffer of its own generations limits drift from its initial model. Surprisingly, the policy learned from these trajectories with an empty reasoning block carries over to inference with thinking enabled, where the student reasons in its own style while acting like the teacher, an effect we call cross-format transfer. Distilling the actions of Qwen3.6-27B raises the task success of Qwen3-4B on AppWorld from 7% to 18% and that of Qwen3-32B from 12% to 25%. We further introduce CoT-proxies, short summaries of the teacher's reasoning placed after the empty reasoning block, which raise Qwen3-4B further, to 35%. Since neither actions nor CoT-proxies need the hidden chain-of-thought, the recipe applies unchanged to closed teachers. Distilling GPT-5.6 Luna from its visible outputs and the reasoning summaries its API returns brings Qwen3-4B to 48% and Qwen3-32B to 59%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
distillation, agents, long-horizon, chain-of-thought, supervised fine-tuning, reasoning models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dvzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether an agent policy can be learned from a stronger teacher without imitating its chain-of-thought, by training on the teacher's actions with an empty reasoning block. The student is held near its initial model with a KL penalty on offline replay data, and may also be given short summaries of the teacher's reasoning. Students are evaluated with thinking enabled, where they reason in their own way while acting like the teacher.

Critical strengths

1.The method transfers agent capabilities using only actions and optional reasoning summaries, making it applicable to closed teachers. The improvements on AppWorld and SWE-bench are substantial. 2.LoRA fine-tuning on 270 fixed teacher trajectories, no RL, no environment interaction during training, and a replay buffer built once offline rather than sampled on-policy. This is a low-cost recipe that others can run, and the offline replay corpus also makes the setup easy to reproduce.

Critical weaknesses

1.The recipe combines action imitation, summary supervision and standard replay-based KL regularization. Cross-format transfer is an interesting empirical finding, but the mechanism remains unclear 2.CoT-proxies significantly outperform full-CoT in only one of four open-teacher settings. Capability retention is tested mainly on AIME25 and replay-domain continuations, which does not establish broad preservation of the student’s original abilities.

Questions for authors

1.Why is action-only clearly worse than full-CoT for Qwen3-4B on AppWorld (17.9 vs 29.2) but roughly tied for Qwen3-32B on SWE-bench (14.3 vs 21.1 — well, closer)? What determines when the teacher's CoT is actually needed? 2.With thinking disabled, CoT-proxy drops from 21.7% to 5.1% on SWE-bench. Since the gain seems to live in the student's own reasoning, would you call this policy distillation, or is it mostly teaching the student to reason better on the task?

3 comments

Thank you for your review. We address each of your points below.

The recipe combines action imitation, summary supervision and standard replay-based KL regularization. Cross-format transfer is an interesting empirical finding, but the mechanism remains unclear.

The final recipe combines three complementary components, and we ablate each of them, showing that: 1) action-only training (action imitation) does transfer long-horizon behaviour of the teacher to some degree; 2) reasoning supervision via proxy (in form of summaries) provides additional signal and matches performance of full CoT supervision; 3) replay-based KL regularization helps to retain model's existing capabilities, which in case of action-only and CoT-proxy training is important, as we do not want the model to lose its initial reasoning capability.

We do not have a complete mechanistic explanation of cross-format transfer, but our ablations narrow it down: the gain from CoT-proxy largely disappears when thinking is disabled at inference, when the summaries are placed inside the reasoning block, and when the summaries are shuffled across turns. The transfer itself was also shown independently in AI safety work by Chua et al. (2025).

To strengthen the empirical evidence for cross-format transfer, we trained Qwen3-4B on SWE-bench trajectories distilled from the closed teacher (GPT Luna):

TargetSWE-bench Verified (%)
Initial2.7
Action-only18.0
CoT-proxy27.8

These results are the best we've got on SWE, surpassing even 32B student trained on open teacher's trajectories. The pattern of CoT-proxy outperforming action-only holds here as well.

CoT-proxies significantly outperform full-CoT in only one of four open-teacher settings.

We agree, and we do not aim to show that CoT-proxy is always superior to full-CoT. Our claim is that it is at least as good in the long-horizon distillation setting (it has the highest mean in all settings and is never significantly worse) which makes it an alternative to standard full-CoT distillation that doesn't need the teacher's verbatim reasoning. This is what enables distillation from closed teachers, where full-CoT training is not possible.

Capability retention is tested mainly on AIME25 and replay-domain continuations, which does not establish broad preservation of the student's original abilities.

We have additionally evaluated the checkpoints on IFEval and a stratified 1,000-question MMLU-Pro subset. Every cell is the mean over the same three seeds as Table 3:

TargetReplayTGCAIME25IFEvalMMLU-Pro (1k)KL (×10⁻³)
Initial–7.374.787.973.00
Full-CoT0%28.268.987.172.442.1
Full-CoT15%29.274.287.972.62.7
Action-only0%14.371.187.972.112.0
Action-only15%17.971.788.172.41.3
CoT-proxy0%29.471.486.873.427.4
CoT-proxy15%34.573.387.473.82.2

The pattern here is the same -- anchoring helps with retaining the original capabilities outside of target domain.

Why is action-only clearly worse than full-CoT for Qwen3-4B on AppWorld (17.9 vs 29.2) but roughly tied for Qwen3-32B on SWE-bench (14.3 vs 21.1 — well, closer)? What determines when the teacher's CoT is actually needed?

We would like to point out that in both of these cases full-CoT is significantly better than action-only (+11.3 and +6.7 points, p = 0.002 in both, as shown in Appendix D.4).

We do not see a single factor that predicts the size of the gap across the settings, and we do not claim one. What holds is that CoT-proxy matches or exceeds full-CoT, which means that the teacher's verbatim CoT is not needed -- a short summary of its reasoning is enough.

With thinking disabled, CoT-proxy drops from 21.7% to 5.1% on SWE-bench. Since the gain seems to live in the student's own reasoning, would you call this policy distillation, or is it mostly teaching the student to reason better on the task?

We would call it policy distillation that is executed through the student's own reasoning. This ablation shows that the best performance on these tasks requires reasoning at inference, and this is what motivates cross-format transfer -- we supervise only actions and summaries, and the student applies what it learned through its own reasoning, which is never supervised directly.


If you have any further questions, please let us know and we will try to address them before the end of the rebuttal.

Thank you again for your review. We would be grateful to hear if anything in our response is still unclear, or if there are other points you would like us to address.

As the discussion period ends soon, we would appreciate it if you could take our response and the new results into consideration in your final score.

Reviewer_eT9ZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a reasoning student can learn an agent policy without a teacher’s hidden chain of thought. It converts teacher trajectories into action-only targets with an empty reasoning block, adds KL replay against the initial student, and tests a CoT-proxy target with a teacher summary after the empty block. Thinking is restored at inference. Qwen3-4B and Qwen3-32B students are evaluated on AppWorld and SWE-bench Verified with an open teacher, plus a closed teacher on AppWorld. The core contribution is evidence that action supervision can improve long-horizon success across this format boundary.

Critical strengths
  1. A practically useful demonstration of long-horizon adaptation without hidden teacher CoT. Action-only training improves AppWorld success from 7.3% to 17.9% for the 4B student and from 11.5% to 25.4% for the 32B student, while evaluation retains student-generated reasoning. Using the closed teacher’s visible outputs and summaries raises performance further to 48.2% and 59.1%. This addresses a consequential practical constraint: useful agent behavior can be distilled even when the teacher’s original reasoning traces are unavailable.

  2. Substantive evidence that the additional supervision matters. The shuffled-summary control reduces CoT-proxy performance from 34.5% to 23.0% on AppWorld and from 15.1% to 3.7% on SWE-bench, supporting the importance of relevant summary content. The thinking ablation also shows that the tested trained model depends strongly on inference-time reasoning. Together with three independent training seeds and paired statistical comparisons, these controls make the empirical contribution substantially more convincing than a comparison of aggregate scores alone.

Critical weaknesses

The central formatting explanation is not cleanly isolated from changes in supervision. In Table 5, moving the summary inside the think block also changes whether the closing think token is supervised. More generally, averaging cross-entropy over different target token sets changes the relative contribution of action supervision. The results establish that the complete training configurations behave differently, but do not isolate summary location as the cause. This limits confidence in the paper’s central methodological explanation and the design principle readers should generalize from it.

Questions for authors
  1. Under gradient accumulation, are teacher and replay losses normalized over all corresponding tokens in the complete update, as Equation (1) specifies, or are per-sample means averaged? What effective KL coefficient results from the implementation?

  2. Were unsuccessful, invalid, or incomplete teacher trajectories included? Please provide the retained trajectory and turn counts after filtering, parsing, and truncation.

  3. What are the length distribution and empty-summary frequency of the unshortened API summaries? Were identical turns retained for the closed-teacher action-only and CoT-proxy comparisons?

  4. Which exact replay dataset versions and subsets were used, and were their prompts checked for overlap with AIME 2025? This determines how independently the reported retention result can be interpreted.

6 comments

Thank you for the review. Below you can find answers to your points.

The central formatting explanation is not cleanly isolated from changes in supervision. In Table 5, moving the summary inside the think block also changes whether the closing think token is supervised.

We reran the summary-as-CoT ablation (Qwen3-4B, AppWorld, 3 seeds) with the loss on the closing think token masked to isolate the effect:

TeacherTargetTGC (%)
OpenCoT-proxy34.5
OpenSummary as CoT, closing token supervised (paper)23.4
OpenSummary as CoT, closing token masked6.2
ClosedCoT-proxy48.2
ClosedSummary as CoT, closing token supervised (paper)33.9
ClosedSummary as CoT, closing token masked0.0

Without supervision of the closing token, the summary-as-CoT student does not learn to end its think block after reasoning, so its turns produce no executable action and the performance collapses. The in-think-block placement therefore depends on supervising the closing token, similarly as one should supervise this token in full-CoT training. CoT-proxy never supervises it and is the best target, so the closing-token difference cannot explain its advantage.

More generally, averaging cross-entropy over different target token sets changes the relative contribution of action supervision.

True, but the summary-as-CoT ablation already shows it couldn't be the sole contribution to the gains. The reasoning part (think block or summary block) has the same number of tokens in both configurations, hence the relative contribution of action supervision is the same for "summary as CoT" and "summary as proxy".

Under gradient accumulation, are teacher and replay losses normalized over all corresponding tokens in the complete update, as Equation (1) specifies, or are per-sample means averaged? What effective KL coefficient results from the implementation?

Yes, the losses are normalized over the whole update. Before the first micro-batch of each update we count tokens of CE and KL parts, and then after each micro-batch we add summed CE or KL divided by the corresponding token count. This approach makes the gradient equal to the gradient of Equation 1.

Were unsuccessful, invalid, or incomplete teacher trajectories included? Please provide the retained trajectory and turn counts after filtering, parsing, and truncation.

All trajectories are used, so that the student also sees the teacher's behavior on more difficult tasks it fails.

Here are the counts of retained trajectories and turns:

Teacher / EnvironmentTrajectoriesSuccessfulCut at the context (Qwen3-4B)Turns kept
Open, AppWorld2701481897.6%
Closed, AppWorld270252299.8%
Open, SWE-bench2501256493.0%
Closed, SWE-bench2501836491.1%

What are the length distribution and empty-summary frequency of the unshortened API summaries?

Here are the numbers for API summaries from GPT Luna model compared with open-model summaries on AppWorld:

Closed teacher (API summaries)Open teacher (generated)
Turns4,2183,704
Empty summary2,219 (52.6%)5 (0.1%)
Nonempty summaries, mean tokens10126

Were identical turns retained for the closed-teacher action-only and CoT-proxy comparisons?

Yes, we retained the same turns for both configurations.

Which exact replay dataset versions and subsets were used, and were their prompts checked for overlap with AIME 2025? This determines how independently the reported retention result can be interpreted.

Replay prompts come from two public datasets: OpenR1-Math-220k for math and OASST1 for chat and instruction prompts. We checked all AIME 2025 problems against the replay prompts and found no exact or near-duplicate match.

As AIME is still in the same broad domain as the math replay prompts, we additionally evaluated on IFEval and MMLU-Pro (a stratified subset of 1000 samples):

TargetReplayTGCAIME25IFEvalMMLU-Pro (1k)KL (×10⁻³)
Initial–7.374.787.973.00
Full-CoT0%28.268.987.172.442.1
Full-CoT15%29.274.287.972.62.7
Action-only0%14.371.187.972.112.0
Action-only15%17.971.788.172.41.3
CoT-proxy0%29.471.486.873.427.4
CoT-proxy15%34.573.387.473.82.2

The pattern is the same as for AIME -- anchoring helps with retaining the original capabilities outside of target domain.


If you have any further questions or comments, please let us know.

Regarding your first experiment, I believe the issues observed stem primarily from the failure of the chat template during training, rather than revealing any inherent impact of the summary's position on the proposed method. I am curious about the specific proportions of cases involving unclosed "thought blocks," unparseable actions, or instances where the maximum generation length was reached. While the addition of IFEval and MMLU-Pro results is valuable, the improvements are marginal (generally less than 1%)—a margin that falls within the range of statistical fluctuation—which leads me to question the effectiveness of the method.

Furthermore, given that 52.6% of the turns in the data from the closed-source "teacher" model contain empty summaries, I am unclear as to the source of the overall performance gains. Do these gains primarily originate from the turns that actually include summaries?

Thank you for the follow-up.

Regarding your first experiment, I believe the issues observed stem primarily from the failure of the chat template during training, rather than revealing any inherent impact of the summary's position on the proposed method. I am curious about the specific proportions of cases involving unclosed "thought blocks," unparseable actions, or instances where the maximum generation length was reached.

Yes, the issue of that additional configuration with masked closing think token comes from the broken output format; that's why the original version in the paper had the token supervised.

The original ablation tests whether we need that cross-format transfer approach at all, or whether we can simply use summaries in place of model's CoT. The results show that summaries used as CoT do not lead to the same performance gains as CoT-proxy (23.4% vs 34.5% and 33.9% vs 48.2%), demonstrating the effect of cross-format transfer -- where the supervised signal from summaries transfers to student's reasoning, but doesn't force it to mimic the summary verbatim.

Here are the requested proportions:

TeacherTargetUnclosed think block (%)No parseable action (%)Stopped at max generation length (%)AppWorld (%)
OpenFull-CoT4.75.40.329.2
OpenCoT-proxy19.219.20.034.5
OpenSummary as CoT, closing token supervised (paper)0.00.40.023.4
OpenSummary as CoT, closing token masked62.364.14.26.2
ClosedCoT-proxy0.00.20.048.2
ClosedSummary as CoT, closing token supervised (paper)0.00.30.033.9
ClosedSummary as CoT, closing token masked99.499.48.70.0

Summary-as-CoT with supervised closing token has no unclosed think blocks at all and is still clearly worse than CoT-proxy, which has 19.2% unclosed think blocks in open-teacher setting.

While the addition of IFEval and MMLU-Pro results is valuable, the improvements are marginal (generally less than 1%)—a margin that falls within the range of statistical fluctuation—which leads me to question the effectiveness of the method.

We would like to clarify that these evaluations on out-of-domain tasks are not meant to improve. The aim of it is to quantify catastrophic forgetting, which is a common concern of supervised finetuning of post-trained models. The results show that finetuning based on cross-format transfer (action-only and CoT-proxy) does not lead to noticable degradation, and is comparable to the baseline full-CoT finetuning.

The benefit of the replay on OOD tasks is indeed small, but consistent across configurations. One does not need to apply replay for cross-format transfer methods, as it's not the core of the approach, but we show the benefit of it in retaining OOD capabilities and higher in-domain performance.

The experiments demonstrating effectiveness of the cross-format methods have results in Tables 1 and 2 in the paper. There we can see that training on actions alone, or together with summaries, leads to substantial improvements over initial post-trained models, and CoT-proxy being at least on par with baseline full-CoT finetuning that requires access to verbatim CoT of the teacher.

Furthermore, given that 52.6% of the turns in the data from the closed-source "teacher" model contain empty summaries, I am unclear as to the source of the overall performance gains. Do these gains primarily originate from the turns that actually include summaries?

Yes. Action-only and CoT-proxy are trained on the same turns and the same actions, and the only difference between them is the content of the summaries, so the gain (35.1% to 48.2%) comes from the 47.4% of turns that include a summary.

The frontier models available via API operate in 'adaptive' thinking mode; they reason where it's necessary and do not on simple turns, so the API doesn't return summaries for all of the turns. In our data, every trajectory has summaries and they are common on turns with longer code. The results therefore show that it's enough to provide summaries on the turns where reasoning is valuable, and these turns are determined by the closed teacher we are distilling.

We will include this analysis with the number of empty summaries in the closed-teacher section of the paper.

Reviewer_b38nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes a simple cross-format method for distilling long-horizon agents from open or closed teachers using actions and brief reasoning summaries, and evaluates it on AppWorld and SWE-bench.

Critical strengths
  • The method is simple and practically relevant: it can distill a closed teacher’s policy without access to its full CoT reasoning trace, using actions and API-provided reasoning summaries.

  • CoT-proxy yields substantial gains over action-only training for both student sizes in the closed-teacher AppWorld experiment, and for the open teacher on both AppWorld and SWE-bench.

Critical weaknesses
  • The cost and quality requirements of CoT-proxy summaries are unclear. The paper does not quantify summary-generation time, tokens, or retries, or test whether a smaller, cheaper model could provide equally useful summaries.

  • The most practically relevant closed-teacher experiment is limited to AppWorld. Key ablations are also limited to the 4B student on AppWorld, leaving their applicability to SWE-bench uncertain.

  • The long-horizon claim lacks direct analysis by task or trajectory length. Roughly half of CoT-proxy runs on SWE-bench exhaust the context budget, and the paper does not report how often long training trajectories are truncated. More comprehensive experiments are needed to validate the paper's core claim.

  • The paper lacks a strong, low-cost baseline for the information supplied by CoT-proxy. It removes the teacher’s visible explanatory text from all training targets, but does not test whether retaining that text, or generating a short action-aligned explanation without access to the teacher’s reasoning, would provide similar gains. The shuffled-summary control shows that relevant content matters, but does not establish that reasoning summaries are necessary.

Questions for authors
  • What is the end-to-end cost of obtaining CoT-proxy summaries, including generation tokens, latency, and retries? Can a smaller or cheaper model produce summaries with comparable downstream performance?

  • Does the closed-teacher CoT-proxy gain extend to SWE-bench? Do the key summary-placement and replay ablations show similar behavior there?

  • How does success vary with task or teacher-trajectory length? What fraction of training trajectories are truncated, and how does that affect performance on later steps?

  • Can the authors compare CoT-proxy with retaining the teacher’s visible explanatory text or generating a short explanation from the task and action alone, under matched training-token budgets? Would reasoning summaries still provide a meaningful gain?

4 comments

Thank you for your review. We address each point below, and split the response into two parts due to length limit. If you have any further questions, comments, or requests -- please let us know.

The cost and quality requirements of CoT-proxy summaries are unclear. The paper does not quantify summary-generation time, tokens, or retries, or test whether a smaller, cheaper model could provide equally useful summaries.

What is the end-to-end cost of obtaining CoT-proxy summaries, including generation tokens, latency, and retries?

The cost metrics of generating summaries for open-teacher are below; we will add it to appendix. Summaries of closed-teacher reasoning come at no additional cost as part of API response.

AppWorldSWE-bench
Turns3,7048,993
Retries (extra requests after a rejected draft)78 (2.1%)341 (3.8%)
Requests incl. retries3,7829,334
Input tokens per request (mean)313264
Output tokens per request, annotator reasoning + summary (mean)1,500 + 281,368 + 22
Total input / output tokens1.2M / 5.8M2.5M / 13.0M
GPU-hours7.517.6

The majority of the generation cost is due to annotator's reasoning. This suggests an experiment with an instruct model as the annotator, and we present the results in the next point.

Can a smaller or cheaper model produce summaries with comparable downstream performance?

Yes. We regenerated the AppWorld summaries with Qwen3-4B-Instruct-2507, the same size as the student and without a reasoning mode. We also tested a smaller and less capable 1.7B model. The results for Qwen3-4B following the same training recipe, with average over 3 seeds:

AnnotatorAnnotation GPU-hours (job wall-clock)TGC (%)SGC (%)
Qwen3.6-27B, thinking (paper)7.534.514.3
Qwen3-4B-Instruct-2507, no thinking0.0836.915.5
Qwen3-1.7B, no thinking0.0927.68.3

The 4B annotator's summaries are at least as useful at a fraction of initial generation's cost, while a 1.7B annotator underperforms. This shows that summaries can be generated by smaller models as well, as long as they're capable enough.

The most practically relevant closed-teacher experiment is limited to AppWorld. [...] Does the closed-teacher CoT-proxy gain extend to SWE-bench?

Yes. We trained Qwen3-4B on 2970 SWE trajectories distilled from GPT Luna for 5 epochs:

TargetSWE-bench Verified (%)
Initial2.7
Action-only18.0
CoT-proxy27.8

These results are the best we've got on SWE, surpassing even 32B student trained on open teacher's trajectories. The pattern of CoT-proxy outperforming action-only holds here as well.

Key ablations are also limited to the 4B student on AppWorld, leaving their applicability to SWE-bench uncertain. [...] Do the key summary-placement and replay ablations show similar behavior there?

The summary-content ablation is on SWE-bench already (Appendix D.2). Additionally, we have run placement and replay ablations for Qwen3-4B on SWE-bench:

TargetSWE-bench Verified (%)
CoT-proxy, 15% replay (paper)15.1
CoT-proxy, 0% replay7.5
Summary as CoT, 15% replay1.6

Both ablations behave as on AppWorld -- the same summaries inside the think block collapse performance, while outside the block they give 15.1%; removing replay halves CoT-proxy's success.

The long-horizon claim lacks direct analysis by task or trajectory length. Roughly half of CoT-proxy runs on SWE-bench exhaust the context budget, and the paper does not report how often long training trajectories are truncated.

How does success vary with task or teacher-trajectory length? What fraction of training trajectories are truncated, and how does that affect performance on later steps?

We will add both analyses to appendix. Below is success by length of the teacher's trajectory, three seeds. For each benchmark the test tasks are split into three equal-sized groups by the teacher's mean number of turns -- SWE-bench: under 34 / 34–53 / over 53 turns; AppWorld: under 11 / 11–18 / over 18 turns:

ModelSWE-bench short / mid / long (%)AppWorld short / mid / long (%)
Teacher88.0 / 70.7 / 49.267.3 / 65.5 / 49.4
Qwen3-4B initial6.4 / 1.4 / 0.420.8 / 1.2 / 0.0
Qwen3-4B full-CoT10.4 / 1.4 / 0.045.2 / 26.8 / 15.5
Qwen3-4B action-only13.6 / 2.6 / 0.439.3 / 9.5 / 4.8
Qwen3-4B CoT-proxy31.9 / 11.4 / 2.061.3 / 27.4 / 14.9
Qwen3-32B initial11.6 / 4.6 / 1.225.6 / 6.5 / 2.4
Qwen3-32B full-CoT44.9 / 13.8 / 4.444.0 / 24.4 / 14.9
Qwen3-32B action-only31.5 / 9.4 / 2.046.4 / 20.2 / 9.5
Qwen3-32B CoT-proxy43.5 / 16.6 / 4.846.4 / 25.6 / 17.3

Training truncation:

StudentCorpusContextTrajectories cutKept
Qwen3-4BAppWorld32,76818 / 27097.6% of turns
Qwen3-4BSWE-bench32,76864 / 25093.0% of turns
Qwen3-32BAppWorld20,48023 / 27092.7% of turns
Qwen3-32BSWE-bench20,480147 / 25071.7% of turns

Truncation is strongest for Qwen3-32B on SWE-bench, where the longest trained trajectory has 56 turns. At test time the student still runs past that point (for CoT-proxy these account for 17% of its solved tasks) so the student is not limited to the trained length.

The paper lacks a strong, low-cost baseline for the information supplied by CoT-proxy. It removes the teacher's visible explanatory text from all training targets, but does not test whether retaining that text, or generating a short action-aligned explanation without access to the teacher's reasoning, would provide similar gains.

Can the authors compare CoT-proxy with retaining the teacher's visible explanatory text or generating a short explanation from the task and action alone, under matched training-token budgets?

Here are results for Qwen3-4B and AppWorld (3 seeds), but with visible text retained. Keeping the teacher's visible text in the targets hurts rather than helps:

TeacherTargetNo visible textVisible text kept
OpenFull-CoT29.228.2
OpenAction-only17.94.6
OpenCoT-proxy34.524.6
ClosedAction-only35.15.6
ClosedCoT-proxy48.236.1

We hypothesize that the drop in performance might be caused by the additional text "diluting" the useful signal coming from actions and reasoning part.

As for explanations generated without the teacher's reasoning, we have run the same annotator and length limit as used during summary generation, but providing the task, the last observation and taken action instead of the thought (3 seeds):

Summary source (Qwen3-4B, AppWorld, open teacher)TGC (%)SGC (%)
Summary of the teacher's reasoning (CoT-proxy)34.514.3
Post-hoc explanation, no access to the reasoning29.010.1
Shuffled summaries (Appendix D.2)23.07.1
No summary (action-only)17.96.5

Generated explanations do improve over shuffled summaries baseline, but underperform when compared with summaries generated from CoT. It shows that the best performance will be obtained with summaries, but when the CoT or API summaries are unavailable, one can use post-hoc explanations. We will add these results to the paper, as they provide an additional use case for the method.

As the discussion period is coming to an end, we wanted to follow up on our response and the new experiments. We would appreciate your feedback, and we are happy to answer any further questions before the deadline.

Thank the authors for the additional experiments. My concerns are mostly addressed by the cheaper-summary-model results, closed-teacher SWE-bench evaluation, and alternative explanation baselines. Moreover, I find the simple, low-cost approach to distilling closed-source agents practically valuable, a significant technical challenge. I will raise my score. Also, I hope for comprehensive new evidence on real long-horizon tasks to strengthen the contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.