acceptodds
Under review as a conference paper at ICLR 2027

TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery

Abstract

Emerging long-horizon agentic tasks require repeated model calls, worsening the inference cost of already-costly language models. While narrow agentic tasks suggest potential for aggressive model pruning without performance drop, empirical results show existing methods proposed for question answering tasks severely degrade task performance when applied to agentic models. We trace this failure to two decisions: what to prune and how to recover. For pruning, one-shot importance estimates fail to track how the pruned model adapts. For recovery, offline distillation covers only teacher prefixes, while full-trajectory on-policy distillation causes student errors to compound across turns. In this work, we propose Trajectory-Anchored Pruning (TAP), the first structural pruning framework for reinforcement learning (RL)-trained agents. TAP couples structural pruning with efficient on-policy recovery, anchoring interactions to teacher trajectories while allowing the student to generate each reasoning-action response. A frozen dense teacher supervises the student's response prefixes, addressing within-response training-inference mismatch while preventing student-induced deviations from propagating across training turns. Instead of one-shot pruning, TAP re-scores channels using gradients of the recovery objective on the recovered student, connecting iterative channel selection to the evolving policy. With 60% of FFN channels removed, TAP retains 99.2% and 88.0% of the dense 7B agents' task success rates on ALFWorld and WebShop, respectively, while reducing GPU time per successful task by approximately 22% and 17%. These results demonstrate effective structural compression of long-horizon agents under a limited recovery budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Agents, Structural Pruning, On-Policy Distillation, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A9RJOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TAP for structurally pruning reinforcement-learning-trained agents on long-horizon interactive tasks. TAP alternates FFN channel pruning with trajectory-anchored distillation and uses gradients of the recovery objective to rescore channels after adaptation. Experiments with Qwen2.5 agents on ALFWorld and WebShop demonstrate improved task-success retention at high pruning ratios and reduced deployment costs in selected configurations.

Critical strengths
  1. Practically relevant problem: The paper evaluates task completion after compression, demonstrates strong performance at high pruning ratios, and reports deployment cost per successful task (Tables 1 and 4).

  2. Convincing pruning ablations: Table 2 compares pruning strategies under a fixed recovery objective and budget, providing clear evidence for rescoring channel importance after recovery.

  3. Recovery requires no additional environment interaction. Once teacher trajectories are collected, TAP can recover the pruned model without further environment rollouts, making it practical when online interaction is costly.

Critical weaknesses
  1. The incremental contribution needs clearer delineation: Iterative structured pruning has precedents, including LoRAPrune (Findings of ACL 2024). TAP should clarify the additional contribution of coupling anchored recovery with channel rescoring, particularly since Table 7 evaluates its specific scoring objective in only one configuration.

  2. The recovery ablation contains confounding factors: The KD–anchored OPD comparison in Table 3 changes response-prefix source, KL direction, and warm-up together, limiting attribution of the gains specifically to student-generated prefixes.

  3. The baseline comparison has a restricted scope: CFSP’s and SlimGPT’s native allocation strategies are replaced with uniform per-layer pruning ratios. The results therefore primarily establish advantages over these adapted variants (Appendix C.4).

  4. The proposed failure mechanism lacks direct validation. Full-trajectory OPD deteriorates substantially on ALFWorld at 80% removal, but the explanation involving cross-turn drift and reduced teacher reliability remains indirectly supported.

Questions for authors
  1. Could the KL-versus-adaptive-NLL scoring comparison in Table 7 be extended to one 80%-removal setting to test the specific scoring objective’s additional value?
  2. How does a strong baseline retaining its native nonuniform allocation perform at a matched total parameter budget?
  3. Could teacher continuations from student-visited and teacher-anchor states be compared directly to test the teacher-reliability explanation?
  4. Could the authors add same-hardware deployment results for the dense 1.5B agent and report recovery GPU-hours for one representative configuration?
  5. Please check the week points. I will consider raising my score if they are satisfactorily resolved.
4 comments

We thank the reviewer for the detailed and constructive feedback. In the following comments, we address each concern and report additional experimental results.

W1 + Q1. Contribution over LoRAPrune; scoring objective at 80%

TAP aims to preserve the task performance of RL-trained long-horizon agents under structural pruning by jointly addressing what to prune and how to recover through channel re-scoring and anchored recovery.

(a) Contribution over prior iterative pruning. LoRAPrune updates importance throughout fine-tuning using LoRA weights and gradients of the NLL objective on dataset-provided text. TAP differs in both what channel importance measures and where it is evaluated. First, TAP uses teacher–student distribution matching to explicitly measure policy discrepancy, rather than the likelihood of dataset-provided tokens. Second, it evaluates this objective on the current student’s own responses in teacher-anchored contexts, so channel selection accounts for the response prefixes encountered by the student.

These same contexts and student-generated responses support anchored OPD recovery. After recovery, TAP re-scores the adapted student under the same objective before the next pruning stage, coupling policy recovery with channel selection.

(b) Scoring objective at 80% removal. To directly address Q1, we extended the KL-versus-adaptive-NLL comparison in Table 7 to 80% FFN channel removal on the 7B agent. The two variants differ only in the scoring loss, with the pruning schedule, recovery procedure, and total recovery budget held fixed.

Scoring lossALFWorld InALFWorld OutWebShop
Teacher–student KL (TAP)76.2 ± 1.872.1 ± 3.162.3 ± 1.2
Adaptive NLL73.3 ± 1.565.4 ± 3.159.1 ± 1.1

Scoring with teacher–student KL yields higher mean success rates on all three evaluations, including a 6.7-percentage-point gain on ALFWorld Out. These results support the additional value of explicitly measuring teacher–student distribution mismatch when selecting channels under aggressive pruning.

W2. Separating the effects of prefix source and KL direction

We first clarify that the logits-KD and OPD variants in Table 3 share an initial forward-KL KD phase, accounting for 20% of the recovery budget at each pruning stage. The standard KD variant continues with the same objective afterward, so warm-up does not introduce an additional difference in the comparison in Table 3.

To separate the effects of prefix source and KL direction, we added two recovery variants at 80% FFN channel removal. H uses the same forward-KL KD warm-up as TAP, followed by anchored OPD with forward KL instead of reverse KL. G uses offline KD with reverse KL. All four variants use TAP’s fixed pruning masks and follow the setting in Table 3.

KL directionTeacher prefixes (offline KD)Student prefixes (anchored OPD)
Forward KLD: 76.0 / 70.9 / 59.8H: 74.3 / 72.9 / 63.1
Reverse KLG: 68.1 / 64.9 / 55.5C: 76.2 / 72.1 / 62.3

Each cell reports ALFWorld In / ALFWorld Out / WebShop success rates (%), averaged over three seeds.

Effect of prefix source (D vs. H). With KL direction held fixed at forward KL, student-generated prefixes improve mean success rates on ALFWorld Out and WebShop by 2.0 and 3.3 percentage points, respectively. These results indicate that the improvements on ALFWorld Out and WebShop do not require switching to reverse KL.

Effect of KL direction (H vs. C). With student-generated prefixes held fixed, changing the KL direction yields comparable mean success rates across the three evaluations.

W3 + Q2. Native allocation of the baselines

As described in Appendix C.4, we use the same per-layer FFN removal ratio across all methods, to isolate differences in channel selection and recovery from differences in layer-wise allocation.

We agree that evaluating the native allocation provides an important complementary comparison. We therefore additionally evaluated CFSP, with its native nonuniform allocation on the 7B agent, matching the total parameter count of the uniform version at 80% FFN channel removal.

MethodAllocationALFWorld InALFWorld OutWebShop
CFSP + KDUniform66.2 ± 2.263.2 ± 3.156.0 ± 1.4
CFSP + KDNative62.1 ± 1.464.4 ± 1.948.7 ± 0.9
TAPUniform76.2 ± 1.872.1 ± 3.162.3 ± 1.2

Native allocation gives mixed results relative to uniform allocation: mean success decreases on ALFWorld In and WebShop and increases slightly on ALFWorld Out. TAP remains higher on all three evaluations.

We attribute this result in part to how native allocation concentrates pruning: the most heavily pruned layer retains 15% of its original FFN channels, leaving it with 25% fewer channels than under uniform allocation. This likely makes recovery harder at 80% removal. Moreover, the angular distance between a block’s input and output activations may not adequately reflect its importance for long-horizon agent performance.

W4 + Q3. Failure mechanism of full-trajectory OPD

We thank the reviewer for suggesting this direct comparison. We conducted teacher-continuation experiments on ALFWorld with the 7B agent at 80% FFN channel removal. After replaying the first k turns of either a teacher trajectory or a full-trajectory OPD student rollout, we let the dense teacher complete the episode. Both conditions use the same remaining step budget. For each k, we pair games in which both trajectories are still ongoing and average success over three continuation seeds per game.

kGamesFrom teacher-visited states (%)From student-visited states (%)Difference
521390.684.8−5.8
106477.662.0−15.6
153763.151.4−11.7

On the paired games, the teacher succeeds less often from student-visited states at every tested k. This directly shows that the student’s resulting states make successful teacher continuation more difficult.

Teacher predictions on training prefixes. We also evaluated the frozen teacher on 600 sampled training context–response pairs per method. Metrics are computed over response tokens, using student-generated responses for both OPD methods and teacher-generated responses for the reference row.

Training context–response pairsTeacher entropyTeacher NLL of responseTeacher top-1 probability
Teacher trajectories and responses (reference)0.210.140.93
Anchored OPD (TAP), stage 1 → 30.32 → 0.280.61 → 0.500.89 → 0.91
Full-trajectory OPD, stage 1 → 30.49 → 0.480.90 → 0.980.84 → 0.84

The teacher exhibits higher predictive entropy on full-trajectory OPD’s context–response pairs than on TAP’s. Across pruning stages, teacher NLL on sampled student responses decreases under anchoring but increases under full-trajectory OPD, indicating greater disagreement with the teacher on the latter’s later-stage samples.

Together, these results support our explanation of cross-turn drift: the student’s own rollouts steer supervision toward states unlike those in teacher trajectories, where teacher supervision becomes less reliable.

Q4. Dense 1.5B deployment and recovery cost

Deployment. We evaluated the dense 1.5B agent on the same A6000 using the protocol in Table 4: 512 input tokens, 128 generated tokens, and batch size 1.

ModelParamsPrefill (ms)Decode per token (ms)
Dense 7B7.62B82.624.1
TAP, 80% FFN removal3.05B42.510.6
Dense 1.5B1.54B22.310.9

The dense 1.5B agent has faster prefill than the pruned 7B agent, while their per-token decoding latencies are comparable. TAP reduces both prefill and decoding latency relative to its dense 7B starting point.

Recovery cost. For the 7B agent at 80% FFN channel removal on ALFWorld, recovery takes approximately 1.19 GPU-hours on a single A6000, including KD warm-up and anchored OPD. The full pruning-and-recovery pipeline takes 1.25 GPU-hours.

ComponentGPU-hours
Channel scoring, slicing, and final merging0.05
KD warm-up0.18
Anchored OPD1.01
Pruning and recovery1.25
Teacher trajectory collection (500 episodes)0.44
Total1.69

Totals are calculated before rounding.

Q5

We have addressed W1–W4 in the preceding comments with clarifications and additional experiments. We thank the reviewer for the constructive suggestions and hope these responses resolve the concerns.

Reviewer_A9RJOpenReview ↗

Reply to TAP authors

Thank you for the clarifications and additional experiments. The new scoring ablation, controlled recovery comparisons, and native CFSP results largely address my main concerns. Although the contribution remains incremental and some limitations persist in the mechanism analysis and deployment evaluation, the added evidence strengthens the empirical support for TAP. Therefore I raising score from 2 to 3. Please incorporate the new results and analyze the remaining limitations into the revised version.

Thank you for the careful reading of our response and for raising the rating. We appreciate your constructive feedback throughout the review.

Reviewer_U1cmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TAP structurally prunes feed-forward channels from reinforcement-learning-trained language agents. It alternates pruning with recovery, then re-scores the remaining channels using gradients of the recovery objective. During recovery, the student generates its own responses within contexts taken from fixed teacher trajectories; a frozen dense teacher supervises the student-generated token prefixes. Experiments on ALFWorld and WebShop use Qwen2.5 agents at 1.5B and 7B scales. The paper compares pruning methods, isolates channel re-scoring and recovery choices, and reports task success and deployment cost per successful task.

Critical strengths
  1. Strong task-level results at aggressive pruning ratios. With 60% of FFN channels removed from the 7B agents, TAP retains 99.24% of dense ALFWorld success and 87.96% of dense WebShop success. At 80% removal it remains substantially ahead of the strongest listed pruning baseline on both tasks. Reporting success alongside parameter reduction is consequential for agent compression.

  2. The ablations support both design decisions. Holding the recovery objective fixed, re-scoring channels after recovery improves on fixed rankings, especially at 80% removal. Holding TAP’s pruning masks fixed, anchored recovery avoids the ALFWorld collapse of full-trajectory OPD. These comparisons make the proposed combination more convincing than the main baseline table alone.

Critical weaknesses
  1. The main pruning baselines use modified allocation rules. LLM-Pruner, CFSP, and SlimGPT are adapted to uniform layer-wise FFN removal, replacing parts of their native allocation strategies. This is reasonable for isolating channel selection, but it leaves open how much of TAP’s headline advantage remains against each method’s best native configuration. Equal supervised-response token budgets also do not establish equal total training compute.

  2. The tested scope is narrow for a long-horizon deployment claim. Results cover two environments and task-specific checkpoints from one model family. Anchored recovery never executes student actions during training, so its coverage depends on the teacher trajectories. The paper acknowledges this limitation, but does not measure how performance changes as the deployed student visits states increasingly unlike those anchors.

Questions for authors
  1. How does TAP compare with the native, tuned layer-allocation strategies of the pruning baselines at the same final parameter count? What are the total GPU-hours for trajectory collection, pruning, and recovery under these comparisons?

  2. How sensitive is TAP to the size and diversity of the teacher trajectory pool, particularly for tasks or starting conditions that induce student states absent from that pool? This would materially clarify when anchoring remains effective.

2 comments

We thank the reviewer for the positive assessment and constructive feedback. In the following comments, we address the questions on native allocation, computational cost, and anchor coverage, and report additional experimental results on both environments.

W1 + Q1. Native allocation; total compute

Native allocation. We additionally evaluated CFSP, the strongest pruning baseline in Table 1, with its native nonuniform allocation on the 7B agent, matching the final parameter count of the uniform version at 80% FFN channel removal.

MethodAllocationALFWorld InALFWorld OutWebShop
CFSP + KDUniform66.2 ± 2.263.2 ± 3.156.0 ± 1.4
CFSP + KDNative62.1 ± 1.464.4 ± 1.948.7 ± 0.9
TAPUniform76.2 ± 1.872.1 ± 3.162.3 ± 1.2

Native allocation gives mixed results relative to uniform allocation: mean success decreases on ALFWorld In and WebShop and increases slightly on ALFWorld Out. TAP remains higher on all three evaluations.

We attribute this result in part to how native allocation concentrates pruning: the most heavily pruned layer retains 15% of its original FFN channels, leaving it with 25% fewer channels than under uniform allocation. This likely makes recovery harder at 80% removal. Moreover, the angular distance between a block’s input and output activations may not adequately reflect its importance for long-horizon agent performance.

Total compute. TAP aims to improve the inference efficiency of RL agents while preserving their task performance, with pruning and recovery incurred as a one-time cost. Our experiments use a matched recovery data budget, with deployment efficiency reported in Table 4. We additionally report the total GPU-hours for teacher trajectory collection, pruning, and recovery below, measured on a single RTX A6000 for the 7B ALFWorld agent at 80% FFN channel removal.

MethodTotal GPU-hours
CFSP + KD1.14
TAP1.69

Compared with CFSP + KD, TAP uses 0.55 additional GPU-hours and improves success rates by 10.0 and 8.9 percentage points on ALFWorld In and Out, respectively.

W2 + Q2. Scope; pool size and coverage

Our additional experiments show that TAP retains most of its performance when the anchor pool is halved. While effective recovery depends on covering task-specific behaviors, this coverage can be achieved with a moderately sized pool in our setting.

Coverage of deployed states. Anchoring provides teacher interaction contexts for policy recovery, while the recovered agent must act in new evaluation tasks. We therefore examine whether the anchor pool covers similar interaction situations encountered during deployment. We characterize each state by its task type, turn range, and task-progress stage, together with the types of the current observation and preceding action. A state is counted as covered when all five attributes match those of an anchor state.

We measure coverage by comparing states visited in evaluation trajectories with those in the teacher anchor pool.

Turns0–45–910–1920–2930+
TAP states covered by the anchor pool98%91%91%91%75%

The anchor pool covers most interaction situations encountered by TAP under this state description.

We then examine how task success varies with the frequency of encountering uncovered state categories. Among TAP episodes still ongoing at turn T, we group them by the proportion of states up to T that are uncovered: 0%, more than 0% but no more than 20%, and above 20%. The table reports eventual success rates.

Turn TAll covered>0% to 20% uncovered>20% uncovered
568%75%56%
1052%56%29%
2027%38%32%
3018%18%17%

Episodes with occasional uncovered states have comparable or higher observed success rates than fully covered episodes. Episodes with more than 20% uncovered states show lower eventual success at T = 5 and T = 10, but account for only 1.3% and 3.7% of ongoing episodes, respectively. And this pattern does not persist at later turns.

Unseen starting conditions. All evaluation tasks are separate from those used to construct the anchor pool. TAP retains a similar proportion of the dense agent’s success on the ALFWorld Out and In splits: 80.6% versus 80.8% at 80% removal.

Q2 continued. Anchor-pool size and diversity

We further examine sensitivity to the size and behavioral coverage of the anchor pool.

Pool size. We halved the teacher trajectory pool while keeping the recovery budget fixed.

PoolALFWorld InALFWorld OutWebShop
Full76.272.162.3
Half74.066.961.0

The half-pool ALFWorld results are averaged over three random subsets. Halving the pool reduces success by 2.2, 5.2, and 1.3 percentage points on ALFWorld In, ALFWorld Out, and WebShop, respectively. Although ALFWorld Out is more affected, TAP retains most of its full-pool performance across all three evaluations.

Task types absent from the pool. As a stress test of anchor-pool diversity, we removed all trajectories of one ALFWorld task type at a time and evaluated recovery on that excluded type. This requires recovery to rely entirely on anchors from other task types. We repeat this test for all six task types and report success rates as ALFWorld In / Out.

Held-out task typeGames (In / Out)Success with type held outSuccess with full pool
pick_and_place_simple35 / 2494.3 / 81.994.3 / 76.4
look_at_obj_in_light13 / 1853.8 / 70.466.7 / 79.6
pick_two_obj_and_place24 / 1748.6 / 29.476.4 / 64.7
pick_heat_then_place16 / 2339.6 / 33.372.9 / 66.7
pick_clean_then_place27 / 3121.0 / 18.364.2 / 72.0
pick_cool_then_place25 / 2112.0 / 30.268.0 / 74.6

Removing a task type has markedly different effects across tasks. Simple pick-and-place retains its performance, since its behaviors are shared with the other task types, whereas tasks that involve heating, cleaning, cooling show substantial drops.

These results suggest that effective anchored recovery depends on the anchor pool covering the behaviors specific to the target tasks. This coverage can be achieved with a reasonably sized anchor pool, enabling TAP to retain strong task performance with a modest recovery data budget.

Scope. Our evaluation covers two environments, two model sizes, and two RL recipes (GiGPO and SPEAR) within the Qwen2.5 family. Evaluation on additional environments and model families remains an important direction for future work.

Reviewer_KF5rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an optimizations to do FFN pruning for models doing Agentic tasks. Their main contributions are:

  1. The authors show that iterative rounds of pruning with interleaved recovery phases with OPD (with a cubic schedule) outperforms one-shot pruning + OPD recovery
  2. The authors show that at high pruning percentages, when the student (pruned model) is very weak, the full agentic trajectories might drift a lot compared to the teacher, which can make OPD unstable. They propose anchored context: which is basically generating previous turns / actions from the teacher model and only generating the final turn response/action from the student model for OPD. This reduces the drift with the teacher and avoids the failure mode.

The paper tests their methods on 2 model scales: 1.5B vs 7B, 2 agentic ends: ALFWorld and Webshop and do detailed ablations isolating the effect of iterative pruning vs. one shot pruning as well as effect of off policy logit distillation, full on-policy distillation as well as their proposed hybrid anchored approach (past turns using teacher as context and on policy distillation for the final turn)

Critical strengths
  • Their proposed approach significantly outperforms the baselines that they consider, especially at more aggressive ratios
  • They provide detailed ablation experiments to explain the contribution of different components of their proposed approach: iterative pruning vs. one-shot pruning, and different levels of on-policy vs. off policy distillation (full off policy, full on-policy and their proposed anchored context)
  • In addition to task performance, the paper also does the end to end cost analysis on agentic tasks to show pruning can decrease end to end cost
Critical weaknesses
  • It would have been better if the paper addresses how pruning compares to simply training a smaller sized agent. In Table 1, I see that the 1.5B parameter dense model beats Tap-7B @ 80% compression (3.05B parameters) by a significant margin. This indicates that having a good RL recipe for training smaller models might be better than training a bigger model and then applying pruning? The 1.5B and 7B models are trained using different recipes, so the comparison is not fair, but this still raises a concern for practitioners that pruning might be inefficient compared to training a smaller model from scratch

  • In Table 2, I see that U (uniform recalibrated) performs much worse than A (one-shot, fixed), which shows that both "iterative pruning" and "recalibration" performance gains might be brittle and heavily sensitive to the schedule (uniform vs. cubic)?

Questions for authors
  1. How does TAP compare at matched size against (a) a GiGPO 1.5B/3B agent and (b) a small dense student distilled from the same teacher with the same budget?
  2. Do you have any intuition on why on shot pruning without recalibration beats one-shot fixed?
  3. In the baselines section, I see "LLM-Pruner and CFSP select channels once and recover with logits KD", could you explain comprehensively all the differences between TAP / LLM-Pruner and CFSP?
4 comments

We thank the reviewer for recognizing TAP’s gains under aggressive pruning, our detailed ablations, and the end-to-end cost analysis. In the following comments, we address each concern and report additional experimental results.

AuthorsOpenReview ↗

W1 + Q1. Pruning vs. a small agent at matched size

We agree that directly training or distilling a smaller agent provides important alternatives to pruning. Our focus is on efficient structural pruning of existing RL agents while retaining their task performance. To address the reviewer’s concern, we clarify the scope and limitations of our comparison with smaller RL-trained agents in (a), and provide a new matched-size distillation comparison using the same teacher and recovery budget in (b).

(a) Comparison with smaller RL-trained agents. The dense 1.5B and 7B agents in Table 1 were trained with different RL recipes (SPEAR and GiGPO, respectively), so their comparison does not isolate the effect of pruning. We agree that a matched-size GiGPO comparison would be valuable, but we have not been able to complete it due to time constraints.

We agree with the reviewer that the RL training recipe can substantially affect an agent’s performance. Improvements in these recipes provide stronger starting points for pruning and recovery; our goal is to retain the capabilities of the resulting agent while reducing its size. To ensure reproducibility and evaluate TAP across different RL recipes, we use publicly released checkpoints from GiGPO (Feng et al., NeurIPS 2025) and SPEAR (Qin et al., ICLR 2026). With this different RL recipe, TAP is also effective on the SPEAR-trained 1.5B agent: at 40% FFN channel removal, it retains 93.0% of the dense agent’s success on ALFWorld and 81.3% on WebShop, the highest retention among the evaluated pruning methods (Table 1).

TAP incurs only a small additional training cost for an existing RL agent: its full pipeline for the 7B agent at 80% FFN channel removal takes about 1.7 GPU-hours on a single A6000, including teacher trajectory collection, with no further environment interaction. For context, GiGPO’s reported iteration time and configuration imply approximately 30 H100 GPU-hours for a single 1.5B ALFWorld training run (362.83 s × 150 iterations on two H100 GPUs; Feng et al., 2025).

(b) Comparison with a distilled dense student. We additionally distilled a dense Qwen2.5-3B-Instruct student (3.09B parameters), closely matching TAP’s 3.05B parameters at 80% FFN channel removal. We used logits KD with the same 7B teacher, recovery data, and token budget as the 80% runs in Table 1. The experimental protocol and reporting conventions follow those in the paper.

MethodParamsALFWorld InALFWorld OutWebShop
Dense 3B + KD3.09B71.4 ± 3.769.7 ± 1.753.7 ± 1.7
CFSP + KD (80%)3.05B66.2 ± 2.263.2 ± 3.156.0 ± 1.4
TAP (80%)3.05B76.2 ± 1.872.1 ± 3.162.3 ± 1.2

At matched size, TAP outperforms the distilled dense student on both benchmarks, whereas the strongest pruning baseline falls behind it on ALFWorld. Pruning a larger agent and recovering it with distillation is therefore not inherently better than directly distilling a smaller dense model. These results reveal TAP’s advantage in jointly designing what to prune and how to recover.

AuthorsOpenReview ↗

W2 + Q2. Uniform vs. one-shot; schedule sensitivity

We agree that the pruning schedule can substantially affect performance at aggressive removal ratios. Below, we explain our schedule choice and examine the contribution of recalibration under a fixed schedule.

(a) Schedule sensitivity. For the pruning schedule, we adopt an established cubic schedule (Zhu & Gupta, 2018), which removes more weights early and progressively reduces the pruning increments. At 80% removal, the final pruning stage removes 57% of the remaining channels under the uniform schedule, compared with 13% under the cubic schedule. Both allocate only one third of the total recovery budget to the final stage. In contrast, one-shot pruning allocates the entire recovery budget to the model at its target sparsity, contributing to its advantage over uniform iterative pruning.

To examine this rationale, we added a geometric schedule that removes approximately 41.5% of the remaining channels at each stage, reaching the same 80% cumulative removal after three stages.

ScheduleCumulative FFN removal by stageALFWorld InALFWorld OutWebShop
Cubic, recalibrated56.3% → 77.0% → 80.0%76.2 ± 1.872.1 ± 3.162.3 ± 1.2
Geometric, recalibrated41.5% → 65.8% → 80.0%72.4 ± 3.263.4 ± 2.757.3 ± 2.4
Uniform, recalibrated26.7% → 53.3% → 80.0%50.0 ± 1.943.0 ± 0.951.9 ± 1.9

Geometric pruning gives intermediate performance on all three evaluations, consistent with this rationale. The effect is also less pronounced at 60% removal, where the cubic–uniform gaps are only 1.9, 0.25, and 2.9 percentage points on ALFWorld In, ALFWorld Out, and WebShop, respectively (Table 2).

(b) Iteration and recalibration. Table 2 in the paper separates two questions: whether interleaving recovery between cuts helps by itself, and whether re-scoring channels on the recovered student provides further benefits. All variants share TAP’s initial scores, recovery procedure, and total recovery budget.

Staging alone (A vs. B). A removes the selected channels in a single cut followed by recovery; B removes exactly the same final set of channels in three cubic-scheduled cuts, with recovery after each and no re-scoring. With the ranking fixed, staging changes when recovery occurs but cannot adapt channel selection to the recovered student. Its benefits are mixed: it improves performance at 80% removal but offers no consistent benefit at 60%.

Re-scoring (B vs. C). C follows the same schedule as B but re-scores the remaining channels on the recovered student before each subsequent cut. This yields more consistent improvements, with gains of up to 8.2 percentage points on ALFWorld Out at 80% removal. These results support coupling iterative pruning with channel selection adapted to the recovered student.

AuthorsOpenReview ↗

Q3. Differences between TAP, LLM-Pruner and CFSP

The table below summarizes the key differences between TAP and the baselines in our experiments.

LLM-Pruner + KDCFSP + KDTAP
What to prune
Scoring signalGradients of the LM lossActivation × relative weightGradients of the recovery objective
Scored onTeacher trajectoriesTeacher trajectoriesStudent's responses
Channel selectionOne-shotOne-shotIterative, recalibrated
How to recover
RecoveryLogits KD on teacher trajectoriesLogits KD on teacher trajectoriesAnchored OPD on the student's responses

For calibration, both baselines use teacher trajectories from the target agent tasks, adapting their calibration data to our setting while retaining their channel-scoring criteria.

Although the table presents what to prune and how to recover separately, TAP couples the two through a shared objective: the recovery objective also guides channel scoring, and the recovered student is re-scored before each subsequent pruning stage. Recovery therefore informs which channels are removed next.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.