- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes X-Tree, which aims to improve how efficiently multi-step agents use limited interaction experience. The method normalizes actions and, by combining occurrence frequency, fragment length, and success information, recursively constructs a reusable hierarchical skill library without relying on LLMs to generate skills. The authors further apply this approach to training examples in offline RL, additional rewards in online RLVR, and teacher context in self-distillation. Experiments on WebArena, ScienceWorld, and WebShop demonstrate that this approach improves agent performance across various settings and shows potential for replacing LLM-based skill generation with deterministic skill mining.
Critical strengths
- The author proposes the X-tree method as a plug-in, which has broad application prospects in modern machine learning problems.
- The experiments span multiple environments, model sizes, and another family of models, and include analyses of additional SFT training, full-trajectory RL, random fragments, random trees, and the number of rollouts, providing multifaceted evidence of the method’s effectiveness under various configurations.
Critical weaknesses
- In the WebShop settings, , causing the gate in Eq. 4 to remain constant at . When the teacher is fixed and the derivative is computed directly on the sampled trajectories, the gradient of the auxiliary loss no longer depends on the teacher’s probability; therefore, it cannot explain the performance differences resulting from changing the skill library, as shown in Table 23. The authors need to clarify the actual loss function and how gradients are handled, and explain their relationship with Eq. 4.
- Offline training uses only two rollouts per group. Under standard deviation normalization within the standard group, the advantage depends almost entirely on the reward ranking, and the completion and depth rewards in Eq. 2 do not alter this ranking. Therefore, the improvement ranging from 20.7 to 22.9 in Table 2 is difficult to explain directly by the described reward mechanism. It is necessary to clarify how the actual advantage is calculated and to verify whether this ablation experiment altered only the reward terms.
- On ScienceWorld G2, random trees with matched length and depth achieved 26.7, higher than X-Tree’s 25.6; in WebShop, shuffled skill texts still reached 76.5, close to the 77.1 achieved with complete texts. Furthermore, the teacher prompt included manually written task rules. These results suggest that gains may also stem from general fragment reuse, lexical prompts, or task knowledge.
Questions for authors
- Do WebArena’s three evaluations involve independent training seeds, or are they repeated runs of the same model in the same environment?
- Based on Eq. 3 and the parameters from ScienceWorld, does in 5.2.1 actually refer to ? Additionally, is the reward multiplier of 10 in Table 7 of WebShop used for all relevant experiments? Could you note this consistently in the main text?
5 comments
Response to Reviewer Spe9 – Part II
Clarification on WebArena Ablations (W2)
Thank you for pointing this out. We will clarify the ablation settings and ease your concern about the depth term. We will add these points in the revision.
Ablation settings. Only the Plain mixing experiment replaces the completion bonus with a fixed reward with other settings fixed. We acknowledge that the difference on this setting is only from training run variance. The other WebArena ablations (Table 2) change only the training data used by our offline RL method, or replace offline RL with SFT. They do not change the reward terms.
Depth term clarification. You are correct. We acknowledge that the WebArena Plain mixing–Full comparison does not establish the effect of the depth-scaled term. With , two unequal rewards yield advantages of and . The completion bonus preserves the step-matching ranking, yielding the same advantages. Therefore, we cannot attribute the reported gap to the depth term in this comparison. Since WebArena experiments are slow and costly, we will add an experiment with on WebArena in the revision.
Depth term ablation. To ease your concern in the rebuttal with complementary evidence, we ablate the depth term at 1.5B with rollouts on ScienceWorld: instead of scaling the bonus by node depth, we assign a fixed bonus across depths. The RLVR bonus from X-Tree is calculated the same way as the depth term in WebArena. Results shows that removing the depth term degrades the results by 0.8–1.0 points, while still improving the outcome-only RLVR.
Table W2: ScienceWorld depth-term ablation.
| Method | G0 | G1 | G2 |
|---|---|---|---|
| Outcome-only | 38.6 ± 2.5 (−1.2) | 36.2 ± 0.3 (−3.3) | 22.8 ± 1.3 (−2.8) |
| w/o depth term | 39.0 ± 1.2 (−0.8) | 38.5 ± 0.4 (−1.0) | 24.8 ± 1.7 (-0.8) |
| X-Tree | 39.8 ± 1.0 | 39.5 ± 1.7 | 25.6 ± 1.3 |
Clarification on the G2 gap in ScienceWorld (W3)
The G2 setting tests generalization to held-out task types, where patterns mined from the trajectory data naturally transfer less well. We therefore do not expect X-Tree to outperform a randomly merged tree in every held-out setting. However, X-Tree exceeds the matched random tree by +1.7 on G0 and +2.7 on G1. On G2, X-Tree still improves over outcome-only RLVR by +2.8.
Ablation study on WebShop OPSD (W3)
Scrambled-bank clarification. We clarify that the scrambled-bank is distinct from the randomly merged tree. Instead, it tests the sensitivity to how the X-Tree bank is rendered: we shuffle the actions within the X-Tree chains while preserving X-Tree vocabulary. For example, click<option-size>, click<option-color>, click<buy> could be shuffled into click<option-color>, click<option-size>, click<buy> which still presents the information as a prompt. The teacher prompt also includes a fixed task description. We conduct detailed ablations of the teacher context under the corrected OPSD implementation. We will clarify this clearly in the revision.
Ablation Study. To address your concern about the source of the WebShop OPSD gains, we compare the X-Tree bank with a randomly merged tree, X-Score merging over raw actions without canonicalization, a task-description-only context and a scrambled X-Tree bank.
At 1.5B, the task-description-only control is 2.2 points below the X-Tree bank, close to outcome-only RLVR. The randomly merged tree and raw-action control score 1.8 and 0.9 points below X-Tree, respectively, while the scrambled bank is 0.1 points lower. These results suggest that the gain is from the rendered X-Tree bank composed of the canonicalization, the merging and the rendering method, and that performance is relatively insensitive to the rendering method. We will add these results into the revision.
It’s worth noting that in other integrations, X-Tree exceeds the random tree ablation by 5.0 points (27.9% relative) on WebArena and by up to 2.7 points (7.3% relative) on ScienceWorld (Section 5).
Table W3: WebShop OPSD teacher-context ablations.
| Method | Success (%) |
|---|---|
| Outcome-only RLVR | 73.4 ± 1.3 (−2.8) |
| OPSD: Task-description-only | 74.0 ± 1.1 (−2.2) |
| OPSD: Randomly merged tree | 74.4 ± 1.3 (-1.8) |
| OPSD: X-Score on raw actions, w/o canonicalization | 75.3 ± 0.6 (−0.9) |
| OPSD: Scrambled X-Tree | 76.1 ± 1.3 (-0.1) |
| OPSD: X-Tree bank | 76.2 ± 2.2 |
Kind request for feedback on our response
Dear Reviewer Spe9,
Thanks again for your insightful and helpful comments! In our response, we clarified the WebArena evaluation protocol, the WebArena ablations, the notation and bonus in WebShop and the random tree results on ScienceWorld and WebShop. We also reported multiple training seeds result of WebArena and corrected the WebShop OPSD experiments. We hope these had addressed all your concerns.
As the discussion period ends soon, we sincerely look forward to your feedback on our responses. We are happy to clarify anything further to improve our paper.
Authors of Submission 38615
Response to Reviewer Spe9 – Part I
Thank you for your constructive feedback. We clarify the WebArena evaluation protocol, report multiple training seeds result, correct the WebShop OPSD results and clarify the WebArena ablations. We also clarify the notation, the WebShop bonus multiplier, the random tree results on ScienceWorld and WebShop. All results reported here are by 3 training x 3 evaluation seeds. We will include these results and clarifications in the revision.
WebArena multiple training seeds result (Q1)
Common evaluation protocol in literature. In the submitted paper, the three WebArena evaluations are repeated passes of a single trained checkpoint per configuration (Appendix E.2). Go-Browse, our main baseline, reports one run per model with one greedy evaluation, as do WebAgent-R1 (Wei et al., 2025) and AgentGym (Xi et al., 2024).
Independent training runs. To address your concern about training-seed variance, we now report three independent training runs per configuration, each evaluated three times. X-Tree Full exceeds SFT-Full in all three runs, with a 3.8-point difference in mean normalized success rate. We will add multi-seed raining results of other settings in the revision.
Table Q1: WebArena results across independent training runs.
Win-rate notation and the WebShop multiplier (Q2)
Win-rate notation. Yes, you are correct: 0,21,1 are the possible values of the group win rate wg, not the bonus weight λg, when n=2. With ScienceWorld's λ0=0.75 and wref=0.4, Eq. 3 gives λg(wg=0)=0.75 and λg(wg=21)=λg(wg=1)=0. We will clarify this point in Section 5.2.1.
WebShop multiplier. The factor of 10 in Table 7 multiplies the skill bonus used in all of the WebShop RLVR integration experiments. We will state the implemented coefficient explicitly alongside Eq. 3.
The OPSD objective and rerun on WebShop (W1)
Thank you for pointing this out. We acknowledge the mistake. With β=0, the WebShop update is teacher-independent. We rerun the experiments with β=5, the default gate sharpness in the released SDAR recipes (Lu et al., 2026), the same as in our ScienceWorld experiments.
The corrected runs use the same objective as Eq. 4, changing only the gate sharpness: L=LGRPO(Rout)+∣y∣c∑tgtδt, where δt=logπθˉ(yt∣stea)−logπθ(yt∣sstu), gt=σ(βδt), and c=0.01. With this correction, our conclusion stays unchanged. Our X-Tree integration with OPSD, without any LLM calls, consistently exceeds outcome-only at all three scales and is comparable with LLM-written skill bank.
Table W1: Corrected WebShop OPSD results across model scales.
Stop-gradients. In both ScienceWorld and WebShop, we follow SDAR: the teacher log-probabilities and gate are detached, so gradients flow only through the student log-probabilities. With β>0, the teacher still influences the update through the gate’s value.
References