- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents TAP for structurally pruning reinforcement-learning-trained agents on long-horizon interactive tasks. TAP alternates FFN channel pruning with trajectory-anchored distillation and uses gradients of the recovery objective to rescore channels after adaptation. Experiments with Qwen2.5 agents on ALFWorld and WebShop demonstrate improved task-success retention at high pruning ratios and reduced deployment costs in selected configurations.
Critical strengths
-
Practically relevant problem: The paper evaluates task completion after compression, demonstrates strong performance at high pruning ratios, and reports deployment cost per successful task (Tables 1 and 4).
-
Convincing pruning ablations: Table 2 compares pruning strategies under a fixed recovery objective and budget, providing clear evidence for rescoring channel importance after recovery.
-
Recovery requires no additional environment interaction. Once teacher trajectories are collected, TAP can recover the pruned model without further environment rollouts, making it practical when online interaction is costly.
Critical weaknesses
-
The incremental contribution needs clearer delineation: Iterative structured pruning has precedents, including LoRAPrune (Findings of ACL 2024). TAP should clarify the additional contribution of coupling anchored recovery with channel rescoring, particularly since Table 7 evaluates its specific scoring objective in only one configuration.
-
The recovery ablation contains confounding factors: The KD–anchored OPD comparison in Table 3 changes response-prefix source, KL direction, and warm-up together, limiting attribution of the gains specifically to student-generated prefixes.
-
The baseline comparison has a restricted scope: CFSP’s and SlimGPT’s native allocation strategies are replaced with uniform per-layer pruning ratios. The results therefore primarily establish advantages over these adapted variants (Appendix C.4).
-
The proposed failure mechanism lacks direct validation. Full-trajectory OPD deteriorates substantially on ALFWorld at 80% removal, but the explanation involving cross-turn drift and reduced teacher reliability remains indirectly supported.
Questions for authors
- Could the KL-versus-adaptive-NLL scoring comparison in Table 7 be extended to one 80%-removal setting to test the specific scoring objective’s additional value?
- How does a strong baseline retaining its native nonuniform allocation perform at a matched total parameter budget?
- Could teacher continuations from student-visited and teacher-anchor states be compared directly to test the teacher-reliability explanation?
- Could the authors add same-hardware deployment results for the dense 1.5B agent and report recovery GPU-hours for one representative configuration?
- Please check the week points. I will consider raising my score if they are satisfactorily resolved.
4 comments
W3 + Q2. Native allocation of the baselines
As described in Appendix C.4, we use the same per-layer FFN removal ratio across all methods, to isolate differences in channel selection and recovery from differences in layer-wise allocation.
We agree that evaluating the native allocation provides an important complementary comparison. We therefore additionally evaluated CFSP, with its native nonuniform allocation on the 7B agent, matching the total parameter count of the uniform version at 80% FFN channel removal.
| Method | Allocation | ALFWorld In | ALFWorld Out | WebShop |
|---|---|---|---|---|
| CFSP + KD | Uniform | 66.2 ± 2.2 | 63.2 ± 3.1 | 56.0 ± 1.4 |
| CFSP + KD | Native | 62.1 ± 1.4 | 64.4 ± 1.9 | 48.7 ± 0.9 |
| TAP | Uniform | 76.2 ± 1.8 | 72.1 ± 3.1 | 62.3 ± 1.2 |
Native allocation gives mixed results relative to uniform allocation: mean success decreases on ALFWorld In and WebShop and increases slightly on ALFWorld Out. TAP remains higher on all three evaluations.
We attribute this result in part to how native allocation concentrates pruning: the most heavily pruned layer retains 15% of its original FFN channels, leaving it with 25% fewer channels than under uniform allocation. This likely makes recovery harder at 80% removal. Moreover, the angular distance between a block’s input and output activations may not adequately reflect its importance for long-horizon agent performance.
W4 + Q3. Failure mechanism of full-trajectory OPD
We thank the reviewer for suggesting this direct comparison. We conducted teacher-continuation experiments on ALFWorld with the 7B agent at 80% FFN channel removal. After replaying the first k turns of either a teacher trajectory or a full-trajectory OPD student rollout, we let the dense teacher complete the episode. Both conditions use the same remaining step budget. For each k, we pair games in which both trajectories are still ongoing and average success over three continuation seeds per game.
| k | Games | From teacher-visited states (%) | From student-visited states (%) | Difference |
|---|---|---|---|---|
| 5 | 213 | 90.6 | 84.8 | −5.8 |
| 10 | 64 | 77.6 | 62.0 | −15.6 |
| 15 | 37 | 63.1 | 51.4 | −11.7 |
On the paired games, the teacher succeeds less often from student-visited states at every tested k. This directly shows that the student’s resulting states make successful teacher continuation more difficult.
Teacher predictions on training prefixes. We also evaluated the frozen teacher on 600 sampled training context–response pairs per method. Metrics are computed over response tokens, using student-generated responses for both OPD methods and teacher-generated responses for the reference row.
| Training context–response pairs | Teacher entropy | Teacher NLL of response | Teacher top-1 probability |
|---|---|---|---|
| Teacher trajectories and responses (reference) | 0.21 | 0.14 | 0.93 |
| Anchored OPD (TAP), stage 1 → 3 | 0.32 → 0.28 | 0.61 → 0.50 | 0.89 → 0.91 |
| Full-trajectory OPD, stage 1 → 3 | 0.49 → 0.48 | 0.90 → 0.98 | 0.84 → 0.84 |
The teacher exhibits higher predictive entropy on full-trajectory OPD’s context–response pairs than on TAP’s. Across pruning stages, teacher NLL on sampled student responses decreases under anchoring but increases under full-trajectory OPD, indicating greater disagreement with the teacher on the latter’s later-stage samples.
Together, these results support our explanation of cross-turn drift: the student’s own rollouts steer supervision toward states unlike those in teacher trajectories, where teacher supervision becomes less reliable.
Q4. Dense 1.5B deployment and recovery cost
Deployment. We evaluated the dense 1.5B agent on the same A6000 using the protocol in Table 4: 512 input tokens, 128 generated tokens, and batch size 1.
| Model | Params | Prefill (ms) | Decode per token (ms) |
|---|---|---|---|
| Dense 7B | 7.62B | 82.6 | 24.1 |
| TAP, 80% FFN removal | 3.05B | 42.5 | 10.6 |
| Dense 1.5B | 1.54B | 22.3 | 10.9 |
The dense 1.5B agent has faster prefill than the pruned 7B agent, while their per-token decoding latencies are comparable. TAP reduces both prefill and decoding latency relative to its dense 7B starting point.
Recovery cost. For the 7B agent at 80% FFN channel removal on ALFWorld, recovery takes approximately 1.19 GPU-hours on a single A6000, including KD warm-up and anchored OPD. The full pruning-and-recovery pipeline takes 1.25 GPU-hours.
| Component | GPU-hours |
|---|---|
| Channel scoring, slicing, and final merging | 0.05 |
| KD warm-up | 0.18 |
| Anchored OPD | 1.01 |
| Pruning and recovery | 1.25 |
| Teacher trajectory collection (500 episodes) | 0.44 |
| Total | 1.69 |
Totals are calculated before rounding.
Q5
We have addressed W1–W4 in the preceding comments with clarifications and additional experiments. We thank the reviewer for the constructive suggestions and hope these responses resolve the concerns.
Reply to TAP authors
Thank you for the clarifications and additional experiments. The new scoring ablation, controlled recovery comparisons, and native CFSP results largely address my main concerns. Although the contribution remains incremental and some limitations persist in the mechanism analysis and deployment evaluation, the added evidence strengthens the empirical support for TAP. Therefore I raising score from 2 to 3. Please incorporate the new results and analyze the remaining limitations into the revised version.
Thank you for the careful reading of our response and for raising the rating. We appreciate your constructive feedback throughout the review.
We thank the reviewer for the detailed and constructive feedback. In the following comments, we address each concern and report additional experimental results.
W1 + Q1. Contribution over LoRAPrune; scoring objective at 80%
TAP aims to preserve the task performance of RL-trained long-horizon agents under structural pruning by jointly addressing what to prune and how to recover through channel re-scoring and anchored recovery.
(a) Contribution over prior iterative pruning. LoRAPrune updates importance throughout fine-tuning using LoRA weights and gradients of the NLL objective on dataset-provided text. TAP differs in both what channel importance measures and where it is evaluated. First, TAP uses teacher–student distribution matching to explicitly measure policy discrepancy, rather than the likelihood of dataset-provided tokens. Second, it evaluates this objective on the current student’s own responses in teacher-anchored contexts, so channel selection accounts for the response prefixes encountered by the student.
These same contexts and student-generated responses support anchored OPD recovery. After recovery, TAP re-scores the adapted student under the same objective before the next pruning stage, coupling policy recovery with channel selection.
(b) Scoring objective at 80% removal. To directly address Q1, we extended the KL-versus-adaptive-NLL comparison in Table 7 to 80% FFN channel removal on the 7B agent. The two variants differ only in the scoring loss, with the pruning schedule, recovery procedure, and total recovery budget held fixed.
Scoring with teacher–student KL yields higher mean success rates on all three evaluations, including a 6.7-percentage-point gain on ALFWorld Out. These results support the additional value of explicitly measuring teacher–student distribution mismatch when selecting channels under aggressive pruning.
W2. Separating the effects of prefix source and KL direction
We first clarify that the logits-KD and OPD variants in Table 3 share an initial forward-KL KD phase, accounting for 20% of the recovery budget at each pruning stage. The standard KD variant continues with the same objective afterward, so warm-up does not introduce an additional difference in the comparison in Table 3.
To separate the effects of prefix source and KL direction, we added two recovery variants at 80% FFN channel removal. H uses the same forward-KL KD warm-up as TAP, followed by anchored OPD with forward KL instead of reverse KL. G uses offline KD with reverse KL. All four variants use TAP’s fixed pruning masks and follow the setting in Table 3.
Each cell reports ALFWorld In / ALFWorld Out / WebShop success rates (%), averaged over three seeds.
Effect of prefix source (D vs. H). With KL direction held fixed at forward KL, student-generated prefixes improve mean success rates on ALFWorld Out and WebShop by 2.0 and 3.3 percentage points, respectively. These results indicate that the improvements on ALFWorld Out and WebShop do not require switching to reverse KL.
Effect of KL direction (H vs. C). With student-generated prefixes held fixed, changing the KL direction yields comparable mean success rates across the three evaluations.