- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The proposed CATS is a method for selecting to keep which multi-turn trajectory for agentic SFT. It use a small probing model to obtain a quality score for each trajectory. Then each trajectory is transformed a labeled sequence. A similarity graph is built based on the edit distance. The selection is conducted based on the calculated scores. It demonstrates good performance across different benchmarks.
Critical strengths
- It provides a method that is scalable and could be applied to real problems.
- The performance gains across different base models and budgets are consistent.
Critical weaknesses
- The ablation experiments don't reveal that the core components matter.
- The comparison with some backbones should be shown in the paper.
Questions for authors
- Could you compare the performance with D2 pruning itself?
- Why do all variants with single component difference fall below Random? But why quality-only and graph-only ones can match it?
- How about the consumed token per selected subset?
2 comments
Official Comment by Authors (Part2)
Q1. Direct comparison with D² Pruning
Question: Can CATS be compared directly with D² Pruning?
We adapt D² using conventional probe NLL, an embedding graph, and the same message passing selector, while fixing the candidate pool, backbone, and supervision budget. The D² adaptation scores 43.4 on τ³, 22.0 on Vita, and 32.7 on μAGENT. CATS scores 45.6, 23.4, and 34.5, respectively, improving by 2.2, 1.4, and 1.8 percentage points.
This comparison supports the added value of the proposed score and structural representation within the inherited selection framework. We attribute the selector to D² and distinguish it from our contributions. The comparison applies to the adaptation described here.
Selection takes 16.0 wall hours for D² and 17.8 for CATS; SFT costs are 1,400.0 and 1,333.3 GPU hours. These quantities are reported separately. Converting selection wall time into GPU hours requires the corresponding hardware allocation, and neither cost component alone establishes total efficiency.
Q2. Why some partial variants perform worse than Random
Question: Why do some partial variants underperform Random, while graph selection alone and quality scoring alone perform similarly to Random?
Removing a signal entirely differs from retaining an incomplete signal. A partial score can bias selection toward particular trajectories, while a coarse graph can group trajectories with different behavior. We therefore separate the effects of supervision volume, data distribution, and component design.
| Configuration | Original setting | Equal supervision | Also matching length and source |
|---|---|---|---|
| AComp only + full graph | 28.9 | 29.5 | 30.3 |
| CDep only + full graph | 30.5 | 30.8 | 31.4 |
| Full score + graph without order | 27.0 | 28.2 | 29.4 |
| Full score + graph using tools only | 29.1 | 29.8 | 30.7 |
| Random | 31.7 | 32.0 | 32.2 |
| CATS | 34.7 | 34.5 | 34.2 |
The partial variants recover some performance after matching supervision, length, and source, but remain below Random and full CATS. These factors explain part, but not all, of the degradation.
Among 100 neighboring trajectory pairs from each graph, 76 are behaviorally equivalent for the full graph, compared with 57 for the graph without order and 54 for the graph using tools only. With the graph fixed, permuting the trajectory scores reduces performance from 34.5 to 32.4. These results support the structural representation's ability to distinguish behavior and the importance of assigning scores to the correct trajectories.
Q3. Tokens consumed by the selected subsets
Question: How many tokens does training actually consume for each selected subset?
The full 4B table appears in our response to Reviewer 87xq, Q1. Over five complete passes, the supervised/input totals for Random, LONGEST, and CATS are 2.00/5.00B, 4.50/11.00B, and 2.40/6.00B, respectively. These are conversions from the per-pass counts; actual batch counts take precedence when truncation, resampling, or budget stopping changes consumption.
| Backbone | Supervised tokens, B | Input tokens, B | SFT GPU hours |
|---|---|---|---|
| Qwen3.5-4B | 2.400 | 6.000 | 1,333.3 |
| Gemma-4-12B | 2.592 | 6.480 | 3,200.0 |
| Qwen3.5-35B-A3B | 2.400 | 6.000 | 2,051.3 |
We report supervised tokens, input tokens, and GPU hours separately. The AComp scoring mask is distinct from the SFT supervision mask. Counts across backbones use each model's tokenization and training configuration rather than extrapolating from another model's throughput.
We assess total efficiency through two separate comparisons. At equal total cost, normalized to 100 for both methods, the strongest baseline scores 33.8 and CATS scores 34.7. At equal performance, the normalized costs are 100 for the baseline and 93 for CATS, a 7% reduction. These results support higher performance at the same cost and lower cost at the same performance, respectively.
Total cost includes scoring, selection, and training, with reference generation and environment execution additionally counted for RL. Each comparison is normalized to its own baseline. The efficiency claim applies to the reported budgets and target performance, rather than every possible budget.
Official Comment by Authors (Part1)
W1. Effectiveness of the core components
Concern: The existing ablations do not adequately establish the contributions of the core components.
Random selection, graph selection alone, quality scoring alone, and full CATS achieve aggregate scores of 31.7, 32.1, 32.0, and 34.7. We further isolate the score and graph through a crossed comparison.
Probe NLL with an embedding graph scores 32.7; CATS scoring with the same graph scores 33.5. With the structural graph, probe NLL scores 33.1 and CATS scores 34.5. ToolRM with the structural graph scores 33.6. Comparisons with the graph held fixed support the scoring contribution, while comparisons with the score held fixed support the structural representation.
The estimated average marginal effects of the score and graph are +1.1 and +0.7 percentage points, respectively. The interaction estimate is +0.6 points. These estimates support component contributions without establishing a statistically significant interaction.
W2. Comparisons across backbones
Concern: The paper should compare CATS with baselines on multiple backbones.
CATS exceeds the best evaluated baseline on all three backbones. On Gemma, the advantage also persists under budget controls: CATS and ATLAS score 24.0 and 22.0 under equal supervision, and 23.9 and 22.2 under equal training cost.
Initial checkpoint performance and valid tool call rates can help explain differences between models, but do not replace task success rates. Model comparisons also require explicit checkpoint identifiers, interface templates, and call validation rules. Fewer baselines were evaluated on the MoE model, so our claim is limited to the models and baselines tested.