acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical planning with cross-level credit assignment for long-horizon tool use

Abstract

Long-horizon tasks require large language model (LLM) agents to plan hierarchically, in which a global plan fixes task-wide objectives and local subgoals adapt to observations revealed only during execution. When to place these local subgoals depends on observations that emerge during execution, so that we formulate it as a learning problem which however poses a credit assignment challenge. Prior hierarchical planning methods supply this credit through a learned value model or a prescribed process reward, which provides neither the after-execution evidence the global plan needs nor the task-wide reference a local subgoal lacks. We introduce HiCLA (Hierarchical Planning with Cross-Level credit Assignment), which supplies both through cross-level on-policy self-distillation: after a trajectory terminates, the policy (1) generates terminal evidence that rescores the global plan (i.e., execution-to-planning) and (2) produces a revised plan that in turn rescores each recorded state and decision (i.e., planning-to-execution), supplementing GRPO and used only during training. Across three Qwen backbones, HiCLA achieves the highest τ²-bench mean success over four evaluation seeds among the compared methods and exceeds the strongest baseline on held-out BFCL-V3 with the 7B and 4B models by up to 5.25%.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Hierarchical Planning, Reinforcement Learning, Long-Horizon Tool Use

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ciHYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiCLA combines a persistent global plan with adaptive local subgoals in a shared LLM policy. After execution, successful and failed trajectories under the same initial plan provide terminal evidence for two auxiliary paths: rescoring the initial plan, and generating a revised plan that rescores recorded local states and decisions. A success/failure likelihood contrast trains revised-plan generation and gates local supervision. These signals supplement hierarchical GRPO and are absent at inference. Across three Qwen backbones, HiCLA has the highest reported four-evaluation-seed mean on tau2-bench among the compared methods; held-out BFCL-V3 improves over the strongest baseline at 7B and 4B but not 8B. Matched-rollout 4B ablations support both planning levels and the auxiliary objectives.

Critical strengths
  1. The method makes a careful separation between privileged training feedback and deployable behaviour. Plan and execution updates retain their original generation contexts, while revised plans only supply detached scores. Separate state and decision feedback can express differences that a shared terminal advantage cannot. The revised-plan-loss ablation, which retains generation and gating, is particularly informative about learning the feedback source.

  2. The evaluation includes relevant training baselines, matched environment-rollout budgets and transfer to BFCL-V3 without using it in training or checkpoint selection. The 4B task-credit-only comparison improves from 37.50% to 41.67% in the ablation protocol. The method also leads the main four-evaluation-seed tau2-bench mean on all three backbones, supporting utility beyond one isolated configuration.

Critical weaknesses
  1. The central auxiliary credit signal lacks independent validation. A revised plan is rewarded for increasing relative likelihood on the same successful and failed traces used to construct it, and is not executed. The theorem includes a within-outcome KL residual, while the implemented score additionally normalises by blocks and steps; neither establishes correct local credit signs. The paper acknowledges these limits, but its diagnostics measure disagreement and association rather than whether a changed signal corrects behaviour. With only 16.04% of sampled plans eligible and 6.93% positively gated in the 4B archive, evidence from this selected subset is especially important for explaining the claimed credit-assignment benefit.

  2. The robustness of the incremental gains is difficult to assess. All configurations use one training run; avg@4 repeats evaluation of a fixed checkpoint, and the mechanism ablations use a separate original evaluation without uncertainty. Several ablation gaps are small, while some comparison runs show tool-use collapse. These facts do not negate the results, but weaken the evidence that the proposed components reliably improve over a stable, tuned alternative rather than a particular training trajectory.

Questions for authors
  1. Targeted validation: can existing held-out traces or a small same-state replay audit test whether positive revised-plan/local signals correspond to actual corrections? Please distinguish independent outcome evidence from likelihood and model-generated-edit diagnostics. A large new experiment is not requested.

  2. Clarification: can you provide paired per-task outcomes and evaluation-seed uncertainty for the main 4B comparison and its key ablations under one protocol? Were HiPER/SEED collapse cases checked against their intended implementations and reasonable validation-selected settings? If training repeats are unavailable, please explicitly bound the reliability claim rather than treating evaluation repeats as training replication.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ciHY (1/4)

We sincerely thank the reviewer for the careful and detailed review. We also appreciate the recognition of HiCLA's separation of training and inference, separate feedback for states and decisions, ablation design, and cross-task evaluation. We address the questions about the auxiliary credit signal and experimental reliability in four parts, drawing on the existing validation and additional results:

  • W1 & Q1: Independent validation of local credit signals. We added correctness annotations and matched decision comparisons to evaluate local signal quality.
  • W1: Theory and evidence for local credit signs. We clarified what Proposition 1, normalization, and gating establish, and how local validation and ablations support the use of these signals.
  • W2 & Q2: Reliability of performance gains. We added independent 4B training runs and repeated ablation evaluations to assess the reliability of the gains.
  • W2 & Q2: Fairness of baseline comparisons. We documented baseline settings and checkpoint selection, and added reproduction and collapse analyses to assess comparison fairness.

Please kindly let us know whether our response addresses the issues you raised, and whether any concerns remain. Please note that all references to the paper's sections, figures, and tables are to the revised manuscript; their relative positions may change in the revision. Tables R-prefixed tables belong to this response.


(W1 & Q1) Independent validation of local credit signals

Targeted validation: can existing held-out traces or a small same-state replay audit test whether positive revised-plan/local signals correspond to actual corrections? Please distinguish independent outcome evidence from likelihood and model-generated-edit diagnostics. A large new experiment is not requested.

We thank the reviewer for suggesting targeted validation. We use two checks on existing traces: whether positive local signals identify correct content and actions, and whether a higher signal favors a successful continuation when the decision context is the same. The first uses task-based correctness labels; the second uses outcomes recorded by the environment.

HiCLA achieves higher positive-signal precision at both token and step levels

  • HiCLA's positive likelihood changes more often identify correct tokens and actions. GPT-6-Astra labels content and tool actions against task facts, domain policies, and observed interaction feedback. For a tool action, this includes checking the tool, its arguments, authorization, and prerequisites. Agreement with a model-generated edit is not a correctness criterion.
  • Token-level precision is 92.7% for HiCLA versus 61.9% for SEED; step-level precision is 75.0% for HiCLA versus 41.1% for HiPER. Precision is the fraction of positive predictions judged correct, excluding cases whose correctness could not be determined.

Table R1. Positive-signal precision on τ²-bench (16 annotated rollouts per method; §5.4, Table 3 and Appendix D.5, Table 6).

GranularityMethodPositive-signal precision (%)
Token-levelSEED61.9
Token-levelHiCLA92.7
Step-levelHiPER41.1
Step-levelHiCLA75.0
  • HiCLA's positive predictions use likelihood increases before gating: token scores precede block aggregation, and step scores combine state and decision scores. SEED uses token likelihood increases before sigmoid weighting. HiPER uses a low-level critic threshold calibrated on the other annotated trajectories.

Matched-context signals favor successful continuations

  • The decision context is matched; the outcome label comes from the environment. We compared 131 pairs of next decisions from τ²-bench training rollouts across 10 training steps. Each pair came from the same task with the same interaction history and current environment feedback. One recorded continuation succeeded and the other failed; their signals were compared at the next decision.
  • The successful side received the higher signal more often, including after gating. Before gating, 59 pairs favored the successful side and 42 favored the failed side. After gating, fewer pairs showed a preference, but the ratio among those pairs rose from 1.40× to 1.57×.

Table R2. Next-decision preferences for matched continuations (Appendix D.5, Table 7). The ratio divides the successful-side count by the failed-side count; ties contribute to neither.

OPSD signalSuccessful side higherFailed side higherPreference ratio
Before gating59421.40×
After gating33211.57×

AuthorsOpenReview ↗

Response to Reviewer ciHY (2/4)

(W1) Theory and evidence for local credit signs

The theorem includes a within-outcome KL residual, while the implemented score additionally normalises by blocks and steps; neither establishes correct local credit signs.

We thank the reviewer for raising this distinction. Proposition 1 explains the trajectory-level basis for the success–failure contrast. The local validation shows that the signals favor correct content and successful continuations, while ablations show that using them improves task performance.

(1) Theory motivates the success–failure contrast

  • The population contrast is linked to the change in success log-odds.
    • For two fixed plans under the stated assumptions, this change equals CΛ+K1−K0C_\Lambda+K_1-K_0. The within-outcome KL residual can affect its direction (Appendices B.2–B.3).
  • Normalization controls length effects in the contrast.
    • Averaging within blocks and across steps prevents longer blocks or trajectories from receiving greater weight solely because of their length.
    • Averaging within a fixed block preserves the sign of its summed likelihood change. A positive gate preserves that sign through η(Q)tanh⁡(δi,jb)\eta(Q)\tanh(\delta^b_{i,j}).
    • The gate uses a normalized contrast on the recorded pair used to construct QQ. This differs from the population contrast CΛC_\Lambda (Appendix B.4).

(2) Correctness and outcomes support local signal quality

  • Positive likelihood changes identify correct content and tool actions.
    • Table R1 measures positive-signal precision at token and step levels before gating, using correctness labels based on task facts, domain policies, and observed feedback.
  • Matched-context signals favor successful continuations, including after gating.
    • Table R2 complements this correctness check with environment outcomes. For recorded training continuations from matched decision contexts, the successful side more often receives the higher signal at the next decision.

(3) Revised-plan supervision improves task performance

  • Planning-to-execution OPSD contributes to the gain.
    • Removing this local supervision lowers avg@4 under the matched rollout budget while retaining revised-plan training (Table R4).
  • Training the revised plan and using it as the teacher both help.
    • Full HiCLA scores higher than both the revised-plan-loss ablation and the terminal-evidence teacher variant (Table R4).

Together, these results support the revised-plan signals as useful auxiliary supervision alongside task-level credit.


(W2 & Q2) Reliability of performance gains

Clarification: can you provide paired per-task outcomes and evaluation-seed uncertainty for the main 4B comparison and its key ablations under one protocol?

If training repeats are unavailable, please explicitly bound the reliability claim rather than treating evaluation repeats as training replication.

HiCLA’s 4B advantage holds across training seeds, and full HiCLA scores higher than every ablation. We trained each method with three seeds for the main comparison and each ablation configuration once. Both tables use the same τ²-bench avg@4 protocol: four evaluation seeds per checkpoint. We report aggregate scores; paired per-task outcomes and separate estimates of evaluation-seed uncertainty are unavailable.

HiCLA leads both benchmarks at all three 4B training seeds

  • HiCLA also outperforms GRPO and GiGPO across all three seeds.
    • τ²-bench: HiCLA beats both baselines at seeds 20, 42, and 50. Its mean avg@4 is 38.40%, compared with 33.58% for GRPO, the strongest baseline on this metric.
    • BFCL-V3: HiCLA also beats both at all three seeds. Its mean overall pass@1 is 29.58%, compared with 25.33% for GiGPO, the strongest baseline here.
  • Table R3 reports how much scores vary when we retrain each method.
    • For example, HiCLA’s τ²-bench avg@4 was 40.00%, 37.50%, and 37.71% for training seeds 20, 42, and 50, respectively. These three scores give the reported 38.40 ± 1.39% (mean ± sample standard deviation).

Table R3. Main 4B results (Qwen3-4B; selected checkpoints; mean ± sample standard deviation across three training seeds, %; Table 1).

Methodτ²-bench Overall pass@1τ²-bench avg@4BFCL-V3 Overall
GRPO33.54 ± 0.7533.58 ± 1.6820.75 ± 1.02
GiGPO33.12 ± 0.7233.04 ± 0.7625.33 ± 1.33
HiPER34.44 ± 0.9629.74 ± 2.0422.79 ± 1.71
SEED30.00 ± 2.2128.06 ± 1.597.00 ± 0.54
HiCLA41.67 ± 0.8438.40 ± 1.3929.58 ± 0.47

Independent training runs currently cover 4B. We will complete and report the 7B and 8B repeats after the rebuttal.

AuthorsOpenReview ↗

Response to Reviewer ciHY (3/4)

(W2 & Q2) Reliability of performance gains

Clarification: can you provide paired per-task outcomes and evaluation-seed uncertainty for the main 4B comparison and its key ablations under one protocol?

If training repeats are unavailable, please explicitly bound the reliability claim rather than treating evaluation repeats as training replication.

HiCLA’s 4B advantage holds across training seeds, and full HiCLA scores higher than every ablation. We trained each method with three seeds for the main comparison and each ablation configuration once. Both tables use the same τ²-bench avg@4 protocol: four evaluation seeds per checkpoint. We report aggregate scores; paired per-task outcomes and separate estimates of evaluation-seed uncertainty are unavailable.

HiCLA leads both benchmarks at all three 4B training seeds

  • HiCLA also outperforms GRPO and GiGPO across all three seeds.
    • τ²-bench: HiCLA beats both baselines at seeds 20, 42, and 50. Its mean avg@4 is 38.40%, compared with 33.58% for GRPO, the strongest baseline on this metric.
    • BFCL-V3: HiCLA also beats both at all three seeds. Its mean overall pass@1 is 29.58%, compared with 25.33% for GiGPO, the strongest baseline here.
  • Table R3 reports how much scores vary when we retrain each method.
    • For example, HiCLA’s τ²-bench avg@4 was 40.00%, 37.50%, and 37.71% for training seeds 20, 42, and 50, respectively. These three scores give the reported 38.40 ± 1.39% (mean ± sample standard deviation).

Table R3. Main 4B results (Qwen3-4B-instruct; mean ± sample standard deviation across three training seeds, %; Table 1).

Methodτ²-bench Overall pass@1τ²-bench avg@4BFCL-V3 Overall
GRPO33.54 ± 0.7533.58 ± 1.6820.75 ± 1.02
GiGPO33.12 ± 0.7233.04 ± 0.7625.33 ± 1.33
HiPER34.44 ± 0.9629.74 ± 2.0422.79 ± 1.71
SEED30.00 ± 2.2128.06 ± 1.597.00 ± 0.54
HiCLA41.67 ± 0.8438.40 ± 1.3929.58 ± 0.47

Independent training runs currently cover 4B. We will complete and report the 7B and 8B repeats after the rebuttal.

Full HiCLA outperforms each ablation on average over four evaluation seeds

  • Both OPSD directions contribute to the gain.
    • Without execution-to-planning OPSD, avg@4 falls from 37.50% to 35.42%.
    • Without planning-to-execution OPSD, it falls to 36.25%. Overall pass@1 also favors full HiCLA in both comparisons (Table R4).
  • Training the revised plan and using it as the OPSD teacher both help.
    • Removing only the revised-plan loss lowers avg@4 from 37.50% to 34.58%, even though the model still generates, scores, and gates the plan.
    • Using terminal evidence directly as the teacher gives 35.83%, compared with 37.50% for the revised plan. At the same rollout budget, the revised plan provides a better OPSD signal. Overall pass@1 follows the same pattern.

Table R4. τ²-bench ablations (Qwen3-4B; avg@4, %; Table 2).

ConfigurationOverall pass@1avg@4
HiCLA (full)41.6737.50
Without execution-to-planning OPSD39.1735.42
Without planning-to-execution OPSD40.0036.25
Without revised-plan loss36.6734.58
Terminal evidence as teacher39.1735.83
Task-level GRPO only37.5036.04

(W2 & Q2) Fairness of baseline comparisons

Were HiPER/SEED collapse cases checked against their intended implementations and reasonable validation-selected settings?

We checked HiPER and SEED against their original papers. Both implementations reach WebShop scores close to the published results on the same backbone. For the τ²-bench comparison, we tuned both baselines and report each run’s best validation checkpoint, including when the run collapsed later. We then examine how critic estimates and single-rollout self-reinforcement can sustain collapse.

Both implementations follow the original training procedures

  • HiPER uses its hierarchical critic to assign credit. We initialize the policy and critic from the same backbone and train them jointly. As in [1, §4.3], we use the critic’s value estimates to compute advantages for subgoal selection, switching, and action execution.
    • Our HiPER reproduction comes close to the published WebShop result on both metrics. At step 120, our Qwen2.5-7B-Instruct run reaches a task score of 92.17/100 and a success rate of 82.81%. For the same backbone, the original paper reports a task score of 92.2 ± 0.3 and a success rate of 83.3 ± 0.9% across three training seeds [1, Table 1].

Table R5a. HiPER WebShop validation task scores (Qwen2.5-7B-Instruct; one training run).

Training stepsTask score (0–100)
5081.90
10083.32
12092.17
Reviewer_9QwmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes HiCLA, a hierarchical planning framework for long-horizon LLM tool use that combines global planning with adaptive local subgoals. It introduces cross-level on-policy self-distillation to improve credit assignment. After execution, terminal evidence is used to rescore global plans, while revised plans provide likelihood-based feedback for local states and decisions. These signals supplement GRPO during training. The authors evaluate HiCLA on τ²-bench and BFCL-V3 using three Qwen backbones. HiCLA achieves the highest mean success across four evaluation seeds on τ²-bench among the compared methods and outperforms the strongest BFCL-V3 baseline on the 7B and 4B models. Ablation studies examine the contributions of hierarchical planning and cross-level supervision.

Critical strengths

None.

Critical weaknesses
  1. The paper argues that existing hierarchical planning methods cannot adequately use terminal evidence to evaluate global plans or provide task-wide guidance for local decisions. However, it does not clearly explain why existing value models or fine-grained credit assignment methods are insufficient for these purposes. The need for the proposed cross-level hindsight supervision therefore remains unclear.

  2. HiCLA uses likelihood changes under terminal evidence and revised plans as fine-grained credit signals. Although the method contrasts successful and failed executions, this does not establish whether the resulting signals correctly assign credit to individual states and decisions. The theoretical analysis does not directly validate the normalized scores used in training, and revised plans are not executed. It therefore remains unclear how reliably these signals reflect the contributions of individual decisions to task success.

  3. The paper contains numerous writing and editing errors, including:

  • Page 1: "p blacketermined".

  • Page 1: "c blackit assignment".

  • Page 1: "conqure this".

  • Page 2: "c blackit Assignment".

  • Page 2: "Inspi black".

  • Page 2: "sha black policy".

  • Page 2: "compa black methods".

  • Page 2: "HiCLR" instead of "HiCLA", incorrectly naming the proposed method.

  • Page 7: "exeperimental settings".

Questions for authors
  1. What specific limitations of existing value models and fine-grained credit assignment methods does cross-level hindsight supervision address?

  2. What evidence shows that likelihood changes under revised plans reflect the contributions of individual states and decisions to task success, rather than simply their compatibility with the revised plan?

  3. How does Proposition 1 support the normalized block-level credit signals used in training, given that it concerns a population-level likelihood contrast?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 9Qwm (1/6)

We sincerely thank the reviewer for the careful review and detailed feedback. We have revised the manuscript and added analyses and experiments to address these concerns. We address the following points:

  • W1 & Q1: Research motivation. We explain the limitations of HiPER’s value estimates and SEED’s local credit signals, then show how cross-level hindsight supervision uses terminal evidence and revised plans to address them.
  • W2 & Q2: Local signal effectiveness. We evaluate revised-plan credit through matched decision comparisons, independent correctness labels, and ablations, showing its task relevance and training benefits.
  • W2 & Q3: Theoretical analysis. We explain Proposition 1’s trajectory-level identity and connect it to normalized local signals through aggregation and gating. We then present empirical evidence for their effectiveness.
  • W3: Writing and editing errors. We corrected the reported writing errors, checked for similar issues, and verified consistent use of the method name.

Please kindly let us know whether our response addresses the issues you raised, and whether any concerns remain. Please note that all references to the paper's sections, figures, and tables are to the revised manuscript; their relative positions may change in the revision. Tables R-prefixed tables belong to this response.

AuthorsOpenReview ↗

Response to Reviewer 9Qwm (2/6)

(W1 & Q1) Motivation for cross-level hindsight supervision

What specific limitations of existing value models and fine-grained credit assignment methods does cross-level hindsight supervision address?

We sincerely thank the reviewer for raising this question; clarifying the limitations of existing methods and how HiCLA addresses them further substantiates the motivation of our work. Below, we explain why existing value models (e.g., HiPER [1]) and fine-grained credit assignment methods (e.g., SEED [2]) are insufficient to use terminal evidence to evaluate global plans and to provide task-wide guidance for local decisions. We then explain how cross-level hindsight supervision addresses these limitations.

Why existing value models (e.g., HiPER [1]) are insufficient

  • The value model is only trained on the scalar outcome of a trajectory, not the terminal evidence of why it succeeded or failed. As a result, it fails to fully judge whether a global plan is adequate.
    • The value model itself is biased. HiPER’s high-level values are its critic's predictions of the scalar outcome, while these predictions contain bootstrapping bias and critic approximation bias ([1], §4.4).
    • A low critic loss does not mean the critic can distinguish effective planning choices. When failed trajectories dominate training, nearly all outcomes are close to zero, so the critic can fit them well simply by predicting low returns.
      • In the same HiPER run, validation success drops rapidly after peaking at step 30 (Table R1). Meanwhile, the critic fits low-return targets better in later stages, while the mean training task success declines (Table R2).
  • The value model bootstraps low-level advantages from high-level values. As a result, it passes critic errors on to local decisions and cannot provide task-wide guidance for them.
    • The bootstrapping and critic approximation biases in HiPER’s high-level values carry over to local advantages, which bootstrap from high-level values at subgoal boundaries ([1], §4.3, Eqs. (6)–(8) and (13)–(15)).
    • Many incorrect local decisions in HiPER are however reinforced. We measure this by step-level positive-signal precision, the fraction of local decisions with a positive signal that GPT-6-Astra labels as correct. GPT-6-Astra labels correctness using task facts, domain policies, and interaction feedback, independently of agreement with the revised plan; decisions it cannot label are excluded.
      • Only 41.1% of the local decisions that HiPER reinforces are correct, compared with 75.0% for the proposed HiCLA (Table R3).
    • As a consequence of incorrectly reinforced local decisions, HiPER’s subgoal control and tool use deteriorate over training.
      • As shown in Table R4, the SWITCH rate rises from 67.79% to 98.97% from early to late training, yet the subgoal text mostly remains unchanged (17.31% →\rightarrow 20.37).
      • Both the tool-call rate and the mean number of tool calls per episode fall to zero, and validation success drops from 14.73% to 11.88% (Table R4; §5.4).
  • Related analyses on the difficulty of value estimation and noisy advantages in multi-turn training appear in RAGEN [3] and SORL [4].

Table R1. HiPER validation success (Qwen3-4B-Instruct-2507, τ²-bench, seed 20; Figure 9(a)).

Training stepsValidation success (%)
1029.17
2023.33
3033.33
4028.33
500.83
603.33
700.00
800.00
900.00
1000.00

Table R2. HiPER critic loss and training task success (Qwen3-4B-Instruct, τ²-bench, seed 20; Figures 9(b) and 10(d)). Critic loss sums the three value-head losses.

MetricEarly training (steps 1–20)Late training (steps 81–100)
Mean critic loss14.920.93
Mean training task success (%)24.532.50

Table R3. Local signal correctness (τ²-Bench; 16 annotated rollouts per method). A signal is positive when HiPER’s low-level critic value reaches a dynamic threshold set from the other annotated trajectories, or when HiCLA’s ungated likelihood increases under the revised plan. For SEED, a positive signal is a raw token likelihood increase after adding the hindsight skill, before sigmoid weighting (Appendix D.5).

GranularityMethodPositive-signal precision (%)
StepHiPER41.1
StepHiCLA75.0
TokenSEED61.9
TokenHiCLA92.7

Table R4. HiPER subgoal control and validation success (Qwen2.5-7B-Instruct, τ²-bench; Figures 3(a) and 8).

ModelTraining stageSWITCH rate (%)Subgoal change rate (%)Validation success (%)
Qwen2.5-7B-InstructStep 1–2067.7917.3114.73
Qwen2.5-7B-InstructStep 81–10098.9720.3711.88
AuthorsOpenReview ↗

Response to Reviewer 9Qwm (3/6)

(W1 & Q1, continued)

Why existing fine-grained credit assignment methods (e.g., SEED [2]) are insufficient

  • SEED assigns fine-grained credit by rescoring each token of a local decision under a skill that the current model summarizes from the rollout [2, §3.2–3.3]. Since each skill comes from from a single rollout, without contrast between successful and failed executions, the skill lacks a task-wide reference for recognizing errors in local decisions. Even more, as SEED weights local decisions always positively, the errors the skill fails to recognize are still reinforced.
    • The skill often fails to recognize local decision errors. We measure this by token-level positive-signal precision again.
      • Only 61.9% content tokens with a positive signal from SEED are labeled correct, compared with 92.7% for HiCLA (92.7%) (Table R3; §5.4, Table 3; Appendix D.5, Table 6).
    • Worse, the errors the skill fails to recognize are still reinforced. SEED converts each likelihood change into a sigmoid weight, which remains positive even when the likelihood change is negative (Appendix D.6, Equation (21)).
      • As shown in Table Table R5 (and Appendix D.6, Table 8), incorrect responses from Qwen2.5-7B-Instruct, e.g., "Your exchanges are on their way!" with a mean log-probability change of −0.0077, receive OPSD weights close to 0.5.
    • As a result, SEED reinforces the response bias present after cold-start SFT, making it difficult for the model to escape the fixed output pattern.
      • For Qwen2.5-7B-Instruct, the tool-call rate falls from 0.40% to 0.00% from early to late training, and the share of episodes without tool use rises from 96.99% to 100.00% (Table R6; Appendix D.6, Figure 11).

Table R5. SEED signals for incorrect responses (Qwen2.5-7B-Instruct, τ²-bench, training step 1; SFT-initialized policy; Table 8).

Incorrect responseMean log-probability changeMean OPSD weight
1. “I have retrieved the order details for W4316152.”−0.00030.4996
2. “You will not need to send the old kettles back.”+0.01260.5126
3. “Your exchanges are on their way!”−0.00770.4904

Table R6. SEED output behavior and training task success (Qwen2.5-7B-Instruct, τ²-bench; Figure 11).

ModelTraining stageTool-call rate (%)Response rate (%)Episodes without tool use (%)Training task success (%)
Qwen2.5-7B-Instructsteps 1–200.4097.9196.9912.73
Qwen2.5-7B-InstructLate (training steps 81–100)0.0096.19100.0015.55

How cross-level hindsight supervision in HiCLA addresses the above limitations

  • Execution-to-planning OPSD: After a trajectory terminates, HiCLA generates terminal evidence, a summary and reflection conditioned on the observed outcome. Terminal evidence from successful and failed executions of the same plan rescores the sampled global plan tokens (§4.2, Equation (3)).
  • Planning-to-execution OPSD: HiCLA revises the global plan using terminal evidence from successful and failed executions of the same plan. The revised plan, serves as a task-wide reference: it rescores each recorded state and decision separately, and each score can be positive or negative (§4.3, Equations (4)–(7)).

References

[1] Jiangweizhi Peng et al. HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents. arXiv:2602.16165v2, 2026.

[2] Jinyang Wu et al. SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning. arXiv:2607.14777v1, 2026.

[3] Zihan Wang et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073v2, 2025.

[4] Chenliang Li et al. Stabilizing Off-Policy Training for Long-Horizon LLM Agents via Turn-Level Importance Sampling and Clipping-Triggered Normalization. COLM, 2026.


Reviewer_yVVHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HiCLA, a hierarchical agent-training method that combines an initial global plan with adaptive local subgoals. It supplements GRPO with two forms of hindsight-based self-distillation: terminal evidence supervises the initial plan, and an evidence-conditioned revised plan provides separate feedback for recorded states and decisions. These additional contexts are used only during training. Experiments cover three Qwen backbones on τ²-bench and held-out BFCL-V3, with component ablations and analyses of the auxiliary signals.

Critical strengths
  1. A coherent way to connect planning and execution supervision. Using terminal evidence to revisit the initial plan, then using a revised plan to score local behavior, gives the two planning levels complementary feedback. Keeping updates under the original generation contexts is an important design choice: the deployed policy does not require hindsight information.
  2. The empirical evaluation supports more than the full system alone. HiCLA leads τ²-bench avg@4 across all three backbones and improves held-out BFCL-V3 performance on two of them. The matched-rollout ablations separately examine planning structure, the two supervision directions, and revised-plan training, providing useful evidence for the design.
Critical weaknesses
  1. The robustness of the key component gains remains unclear. Each configuration uses one training run, while the ablations retain a separate evaluation rather than the main table’s four-seed protocol. Some differences between the original evaluation and avg@4 are large, so the smaller ablation gains are difficult to interpret confidently. The reported tool-use collapse of some baselines also warrants checks that the comparisons are not overly sensitive to implementation or tuning choices.
  2. The revised-plan scores are not independently validated as useful local credit. A revised plan is generated and rewarded using the same successful/failed trajectory pair. Improved likelihood separation may therefore reflect fitting those recorded trajectories rather than identifying which local decisions should be reinforced. The paper acknowledges this limitation, but an independent signal-quality check and a simpler direct-feedback distillation control would make the cross-level mechanism more convincing.
Questions for authors
  1. What differs between the original evaluation and avg@4? In particular, what explains the 4B backbone’s change from 23.54% to 7.08%?
  2. How does HiCLA compare with directly conditioning executor self-distillation on terminal evidence, without generating a revised plan, while keeping the hierarchy and rollout budget fixed?
  3. Can the signs of the local feedback be checked against independently identified planning or action errors, rather than against model-generated edits alone?
6 comments
AuthorsOpenReview ↗

Response to Reviewer yVVH (1/5)

We sincerely thank the reviewer for the constructive feedback and for recognizing HiCLA's algorithmic design, originality, and experimental results.

We address the questions about experimental robustness, evaluation protocols, and local signal validity with additional experiments and analyses:

  • W1: Robustness of the component gains. We report ablation avg@4, results from multiple training seeds, and variability across runs.
  • Q1: Evaluation protocols and the 4B discrepancy. We clarify pass@1 and avg@4, and explain and correct the transcription errors in the 4B backbone results.
  • W1: Fairness of baseline comparisons. We check both HiPER and SEED through WebShop reproduction, describe tuning and checkpoint selection, and analyze their training instability.
  • W2 & Q3: Independent validation of local credit signals. We assess local behavior using task facts, domain policies, and observed interaction feedback to test whether the signals support correct behavior and decisions associated with task success.
  • W2 & Q2: Revised plans versus terminal evidence. We compare revised plans with terminal evidence for local supervision and separately test the contribution of revised-plan training.

Please kindly let us know whether our response addresses the issues you raised, and whether any concerns remain. Please note that all references to the paper's sections, figures, and tables are to the revised manuscript; their relative positions may change in the revision. Tables R-prefixed tables belong to this response.


(W1) Robustness of the component gains

The robustness of the key component gains remains unclear. Each configuration uses one training run, while the ablations retain a separate evaluation rather than the main table’s four-seed protocol.

We thank the reviewer for distinguishing repeated evaluation of a trained policy from replication across independently trained policies. We evaluated each ablation with four evaluation seeds. For the 4B comparison, we trained HiCLA and each baseline with three independent seeds.

Repeated evaluation of the component ablations

  • Full HiCLA outperforms every ablation on both pass@1 (41.67%) and avg@4 (37.50%). We trained one 4B policy per configuration. For avg@4, we evaluated the same selected checkpoint across four evaluation seeds (Table R1).
    • Removing execution-to-planning OPSD or planning-to-execution OPSD lowers avg@4 to 35.42% or 36.25%, respectively. Both directions contribute under the repeated-evaluation protocol.
    • Using terminal evidence directly, removing the revised-plan loss, or removing explicit roles also lowers avg@4. Among the task-level GRPO variants, removing the local state produces the largest drop, from 36.04% to 28.33%.

Table R1. Ablations on τ²-bench (Qwen3-4B-Instruct-2507; seed 42 for training; avg@4 over four evaluation seeds; Table 2).

ConfigurationPass@1 (%)Avg@4 (%)
HiCLA (full)41.6737.50
Without execution-to-planning OPSD39.1735.42
Without planning-to-execution OPSD40.0036.25
Without revised-plan loss36.6734.58
Terminal evidence as the source of supervision39.1735.83
Without explicit roles38.3336.58
Task-level GRPO only37.5036.04
Task-level GRPO without global plan36.6735.71
Task-level GRPO without local state32.5028.33

Main results across independent training seeds

  • Across three independent 4B training runs (seeds 20, 42, and 50), HiCLA has the highest mean performance on both benchmarks.
    • τ²-bench: HiCLA achieves 41.67 ± 0.84% Overall pass@1 and 38.40 ± 1.39% avg@4, ahead of all baselines on both metrics (Table R2). For avg@4, we first averaged four evaluations of each policy, then computed the mean and sample standard deviation across the three policy averages.
    • Held-out BFCL-V3: HiCLA achieves 29.58 ± 0.47%, compared with 25.33 ± 1.33% for GiGPO, the strongest baseline (Table 1).

Table R2. τ²-bench results (Qwen3-4B-Instruct-2507; mean ± sample SD). Trained methods use seeds 20, 42, and 50.

MethodOverall pass@1 (%)Avg@4 (%)
Backbone24.59 ± 2.5624.63 ± 1.28
GRPO33.54 ± 0.7533.58 ± 1.68
GiGPO33.12 ± 0.7233.04 ± 0.76
HiPER34.44 ± 0.9629.74 ± 2.04
SEED30.00 ± 2.2128.06 ± 1.59
HiCLA41.67 ± 0.8438.40 ± 1.39

Limited GPU resources have prevented us from completing training repeats for all backbones during the rebuttal period. We appreciate the reviewer’s emphasis on robustness. We will complete training and evaluation across multiple seeds for each backbone and report variability in the final version.


AuthorsOpenReview ↗

Response to Reviewer yVVH (2/5)

(Q1) Evaluation protocols and the 4B discrepancy

What differs between the original evaluation and avg@4? In particular, what explains the 4B backbone’s change from 23.54% to 7.08%?

We thank the reviewer for flagging the apparent drop in the 4B backbone score. The original evaluation reports pass@1; avg@4 averages success rates across four evaluation seeds at the same checkpoint. We traced the apparent drop to transcription errors in the original table.

Avg@4 reduces dependence on a single evaluation.

  • Pass@1 measures task success in one attempt; avg@4 averages success rates across four evaluation seeds. Both use the selected checkpoint, and Overall equally weights the three domains.
  • Means and sample standard deviations across training seeds describe variability between independently trained policies.

The 4B discrepancy arose from transcription errors.

  • We corrected the reported 23.54% and 7.08%. The corrected original results are 21.67% Overall pass@1 and 23.54% avg@4. The Backbone row in Table R2 summarizes the expanded repeated evaluations of the same pretrained model.
  • We have checked the experiment records and the other table entries.

(W1) Fairness of baseline comparisons

The reported tool-use collapse of some baselines also warrants checks that the comparisons are not overly sensitive to implementation or tuning choices.

To address the reviewer’s concern about implementation or tuning, we checked both baselines against their original training procedures and reproduced their WebShop results. HiCLA retains its 4B advantage at each method’s best validation checkpoint. Below, we report these checks and examine the collapse observed in the HiPER and SEED runs.

Both baselines approach their published WebShop scores

  • HiPER uses its hierarchical critic to assign credit. Following [2, §4.3], we initialized the policy and critic from the same backbone and trained them jointly. We used the critic’s value estimates to compute advantages for subgoal selection, SWITCH/KEEP decisions, and action execution.
    • The reproduction is close on both WebShop metrics. At step 120, our run scored 92.17/100 (Table R3a) with 82.81% task success. The original paper reports 92.2/100 and 83.3% on the same backbone [2, Table 1].

Table R3a. Task scores on WebShop (Qwen2.5-7B-Instruct; Hiper).

Training stepsTask score (%)
5081.90
10083.32
12092.17
  • SEED generates hindsight skills before using them for distillation. Following [1, §§3.2–3.3], we used supervised fine-tuning on trajectories annotated with hindsight skills. During RL, the current policy generated a skill for each rollout; we trained it with outcome rewards and skill-conditioned OPD.
    • The WebShop task score also approaches the published result. On the same backbone, SEED’s task score rose from 77.21% at step 50 to 86.22% at step 150 (Table R3b), compared with 89.7% in the original paper [1, Table 1].

Table R3b. Task scores on WebShop (Qwen2.5-7B; SEED).

Training stepsTask score (%)
5077.21
10085.11
15086.22

Baseline tuning and consistent evaluation

  • We used the same rule to select checkpoints for all methods. We tuned HiPER and SEED over multiple rounds to nsure a fair comparison. On τ²-bench, we evaluated checkpoints every 10 training steps. We selected the checkpoint with the highest average validation success across Airline, Retail, and Telecom, giving each domain equal weight. We report both pass@1 and avg@4 for that checkpoint.
  • All 4B methods use the same training seeds and evaluation protocol. We trained each method with seeds 20, 42, and 50. For each run, we computed avg@4 over four evaluation seeds at the selected checkpoint. Table R2 reports the mean and sample standard deviation across the three training runs: 38.40 ± 1.39% for HiCLA, 29.74 ± 2.04% for HiPER, and 28.06 ± 1.59% for SEED.
AuthorsOpenReview ↗

Response to Reviewer ciHY (3/5)

(W1, continued)

HiPER: declining task success and ineffective subgoal switching

  • The 4B critic’s loss fell even as task success declined. In the seed-20 run, the policy called tools more often late in training, yet training success fell from 24.53% to 2.50% as critic loss fell from 14.92 to 0.93 (Table R4).
    • Fitting failed trajectories can explain why the policy did not recover. When most trajectories fail, predicting low returns can reduce critic loss without distinguishing useful subgoals from poor ones. HiPER also uses subgoal values to compute action advantages at subgoal boundaries [2, §§4.3–4.4]. An error in a subgoal value can therefore affect the feedback given to its actions.

Table R4. HiPER critic loss, task success, and tool use (Qwen3-4B-Instruct-2507, τ²-bench, seed 20; Figures 9–10). Critic loss sums the three value-head losses before averaging over each window.

MetricTraining steps 1–20Training steps 81–100
Mean critic loss14.920.93
Mean training task success (%)24.532.50
Mean tool-call rate (%)44.5477.05
  • In the 7B run, SWITCH decisions dominated but subgoal text rarely changed. Late in training, HiPER chose SWITCH in 98.97% of valid switching decisions, but only 20.37% of adjacent subgoal texts differed. Tool use stopped and validation success declined (Table R5).

Table R5. HiPER subgoal control and validation success (Qwen2.5-7B-Instruct, τ²-bench; Figures 3 and 8).

ModelTraining stageSWITCH rate (%)Subgoal change rate (%)Validation success (%)
Qwen2.5-7B-InstructStep 1–2067.7917.3114.73
Qwen2.5-7B-InstructStep 81–10098.9720.3711.88

SEED: distillation can reinforce replies that omit tool calls

  • An incorrect reply can still receive positive feedback. SEED derives a hindsight skill from a rollout and uses it to rescore that same rollout. The sigmoid weighting gives sampled tokens a positive imitation weight even when the skill lowers their likelihood [1, §3.3].
    • For example, the policy claimed that a replacement had shipped without executing the exchange. Conditioning on the skill lowered the reply’s mean log-probability, yet its mean imitation weight was still 0.4904 (Table R6).

Table R6. SEED signals for incorrect responses (Qwen2.5-7B-Instruct, τ²-bench; SFT-initialized policy; means over scored response tokens; Table 8).

Incorrect responseMean log-probability changeMean OPSD weight
Claims to have looked up an order without performing the lookup−0.00030.4996
Claims that the old item need not be returned, contradicting the return policy+0.01260.5126
Claims that a replacement has been shipped without executing the exchange−0.00770.4904
  • This feedback can sustain an early preference for text. Tool calls accounted for just 0.40% of actions in the first 20 steps of the 7B τ²-bench run. By the last 20 steps, no episode contained a tool call, although the policy continued to reply (Table R7).

Table R7. SEED output behavior and training task success (Qwen2.5-7B-Instruct, τ²-bench; Figure 11).

Training stepsTool-call rate (%)Response rate (%)Episodes without tool calls (%)Training task success (%)
1–200.4097.9196.9912.73
81–1000.0096.19100.0015.55
  • The BFCL-V3 traces distinguish omitted calls from parsing failures. The 4B SEED policy returned only text on 2,046 of 2,924 turns requiring a tool. All 1,223 calls it generated were parsed and executed. Under the same output protocol, the fraction of tasks with no calls was 68.25% for SEED, 2.25% for GRPO, and 0.63% for GiGPO (Table R8).

Table R8. BFCL-V3 tool use (Qwen3-4B-Instruct-2507; 800 tasks per method; identical output protocol).

MethodMean calls/taskTasks without calls (%)Tool-required turns without calls (%)
SEED1.5368.2569.97
GRPO6.292.2514.84
GiGPO7.030.639.76

References

[1] Jinyang Wu et al. SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning. arXiv:2607.14777v1, 2026.

[2] Jiangweizhi Peng et al. HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents. arXiv:2602.16165v2, 2026.


Comments on the paper

7 comments
AuthorsOpenReview ↗

Response to Reviewer FEMB (1/5)

We sincerely thank the reviewer for carefully reading the paper and recognizing its algorithm design, methodological novelty, and experimental results. We especially appreciate the recognition of bidirectional credit assignment between global task planning and adaptive local execution, as well as the comparisons across three backbones and the component ablations.

We address the reviewer's concerns about credit faithfulness, empirical robustness, and feedback sensitivity in six parts:

  • W1 & Q1: Faithfulness of revised-plan credit. We use a verifiable step-level proxy to assess action correctness and matched decision pairs to relate local credit to task outcomes.
  • W2 & Q2: Robustness across training runs. We report results from three independent 4B training runs per method and distinguish training-run variability from repeated evaluation of a fixed policy.
  • W2: BFCL-V3 generalization across backbones. We limit the transfer claim to the evaluated 4B and 7B Instruct models and show how format errors help explain the weaker 8B result.
  • W2 & Q3: Simulator/evaluator sensitivity. We show that HiCLA remains ahead after replacing the simulator/evaluator in both training and evaluation.
  • Q3: Feedback quality. We find that token-level OPSD precision stays high with either HiCLA or DeepSeek generating the revised plan, and that training with revised plans improves task success over using terminal evidence directly.
  • Q3: Additional computation. We show that revised plans improve task success at the same rollout budget and that HiCLA outperforms SEED with less training time per step.

Please kindly let us know whether our response addresses the issues you raised, and whether any concerns remain. Please note that all references to the paper's sections, figures, and tables are to the revised manuscript; their relative positions may change in the revision. Tables R-prefixed tables belong to this response.


(W1 & Q1) Faithfulness of revised-plan credit

Such likelihood changes can reflect textual compatibility with a hindsight plan rather than a causal improvement in the environment.

Can the authors test the faithfulness of revised-plan credit by actually resuming matched environment states under the revised plan and examining whether high-scored versus low-scored decisions produce systematically better outcomes? Alternatively, is there a verifiable step-level proxy correlated with the proposed likelihood-ratio signal?

We thank the reviewer for suggesting a verifiable step-level proxy. We check whether positive revised-plan credit identifies correct tool actions and favors decisions from successful continuations under matched contexts. A complementary ablation tests its contribution to task success during training.

Positive revised-plan credit helps identify correct local decisions

  • Positive-signal precision was 75.0% for HiCLA's tool actions, compared with 41.1% for HiPER's positive critic predictions [1]. GPT-6-Astra checks tool choice, arguments, authorization, and prerequisites against task facts, domain policies, and observed interaction feedback. The label therefore measures action correctness independently of agreement with the revised plan.
  • The same pattern holds for content tokens: positive-signal precision was 92.7% for HiCLA and 61.9% for SEED [2] (§5.4, Table 3; Appendix D.5, Table 6).

Table R1. Positive-signal precision on τ²-Bench (16 annotated rollouts per method). Precision is the fraction judged correct among positive predictions labeled correct or incorrect.

GranularityMethodPositive-signal precision (%)
StepHiPER41.1
StepHiCLA75.0
TokenSEED61.9
TokenHiCLA92.7
  • At step level, HiCLA combines state and decision likelihood changes before gating. HiPER predicts a positive step when its low-level critic value meets or exceeds a threshold calibrated on the other annotated trajectories.

  • At token level, we use raw likelihood increases before HiCLA's block aggregation or SEED's sigmoid weighting. We focus on positive signals because a decrease can also reflect a different valid action or wording.

AuthorsOpenReview ↗

Response to Reviewer FEMB (2/5)

(W1 & Q1, continued)

Revised-plan credit prefers decisions from successful rollouts

  • Given the same interaction history and current environment feedback, the signal more often favored the next decision from the successful rollout. We matched 131 pairs of recorded continuations across 10 training steps. Each pair came from the same task, with one continuation ending in success and the other in failure. The comparison uses the observed environment outcomes; we did not resume environment states under the revised plan.
  • Among pairs with unequal scores, the successful side was favored 1.40× as often as the failed side before gating and 1.57× as often after gating (Appendix D.5, Table 7). These ratios describe a preference between next decisions, rather than a change in task success rate.

Table R2. OPSD preferences (τ²-bench; 131 pairs across 10 training steps; matched histories and environment feedback; labels from final task outcomes).

OPSD signalSuccess preferredFailure preferredPreference ratio
Before gating59421.40×
After gating33211.57×

Using revised-plan credit for local supervision improves task success

  • Applying planning-to-execution OPSD improved task success in the 4B ablation. With revised-plan training retained, adding this local supervision raised avg@4 from 36.25% to 37.50% and pass@1 from 40.00% to 41.67% (Table R4; §5.3, Table 2). This ablation measures the training benefit; the two experiments above test the local signal itself.

References

[1] Jiangweizhi Peng et al. HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents. arXiv:2602.16165v2, 2026.

[2] Jinyang Wu et al. SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning. arXiv:2607.14777v1, 2026.


(W2 & Q2) Robustness across training runs

Appendix C.1 states that there is one training run per configuration; τ²-Bench avg@4 reflects evaluation seeds, not independently trained policies.

Can the authors report results over independent training seeds, or at least uncertainty intervals that do not conflate evaluation stochasticity with training-run variability?

We thank the reviewer for distinguishing training-run variability from evaluation stochasticity. We added independent training runs for the 4B main comparison and repeated evaluations of a fixed checkpoint for each 4B ablation. The two analyses answer different parts of the reviewer's concern.

HiCLA achieves the highest average scores across independent training seeds

  • We trained each method with seeds 20, 42, and 50. For τ²-Bench avg@4, we first average four evaluations of each policy at its selected checkpoint. Table R3 then reports the mean and sample standard deviation of the three policy averages. The standard deviation describes how much the scores varied across training runs.
  • HiCLA leads the strongest baseline on each benchmark: 38.40% versus 33.58% for GRPO on τ²-Bench avg@4, and 29.58% versus 25.33% for GiGPO on BFCL-V3 pass@1 (§5.2, Table 1).

Table R3. Main results for Qwen3-4B-Instruct-2507 (mean ± sample standard deviation across three independent training runs).

Methodτ²-Bench Overall pass@1 (%)τ²-Bench avg@4 (%)BFCL-V3 Overall (%)
GRPO33.54 ± 0.7533.58 ± 1.6820.75 ± 1.02
GiGPO33.12 ± 0.7233.04 ± 0.7625.33 ± 1.33
HiPER34.44 ± 0.9629.74 ± 2.0422.79 ± 1.71
SEED30.00 ± 2.2128.06 ± 1.597.00 ± 0.54
HiCLA41.67 ± 0.8438.40 ± 1.3929.58 ± 0.47

Full HiCLA outperforms its ablations on average across evaluation seeds

  • With training settings and environment rollout budgets matched, we trained each ablation configuration once and evaluated its selected checkpoint with four seeds. Its avg@4 measures average performance across those evaluations.
  • Full HiCLA reaches 37.50% avg@4 in the ablation comparison. Removing execution-to-planning OPSD gives 35.42%; removing planning-to-execution OPSD gives 36.25%. Full HiCLA also has the highest pass@1 among the configurations in Table R4 (§5.3, Table 2).

The 37.50% is the score of one checkpoint averaged over four evaluation seeds. The 38.40 ± 1.39% in Table R3 summarizes three independently trained policies.

Table R4. Component ablations on τ²-Bench with Qwen3-4B-Instruct-2507 (one training run per configuration; four evaluation seeds for avg@4).

ConfigurationOverall pass@1 (%)avg@4 (%)
HiCLA (full)41.6737.50
Without execution-to-planning OPSD39.1735.42
Without planning-to-execution OPSD40.0036.25
Without revised-plan loss36.6734.58
Terminal evidence as teacher39.1735.83

AuthorsOpenReview ↗

Response to Reviewer FEMB (3/5)

(W2) BFCL-V3 generalization across backbones

Moreover, BFCL performance is not uniformly best: on the 8B backbone HiCLA is below GiGPO and GRPO in BFCL overall (Table 1). The paper should clearly bound its claims and provide stronger uncertainty estimates.

We thank the reviewer for pointing out the 8B result. HiCLA leads on the evaluated 4B and 7B Instruct models, but falls below GRPO and GiGPO on 8B. Difficulties following the required output format help explain the weaker 8B performance. We have limited the BFCL-V3 generalization claim in §5.2 and §6 to the two evaluated Instruct models.

Analysis of HiCLA’s generalization on the 4B and 7B backbones

  • Qwen3-4B-Instruct-2507: Across three independent training seeds, HiCLA achieves a mean BFCL-V3 Overall score of 29.58%, compared with 25.33% for GiGPO, the strongest baseline. Table R3 reports the sample standard deviations across training runs.
  • Qwen2.5-7B-Instruct: On BFCL-V3, HiCLA achieves an Overall score of 21.88%, which is higher than GiGPO’s 20.50%. These results are based on one training run per method (§5.2, Table 1).

Analysis of HiCLA’s generalization on the 8B backbone

  • Qwen3-8B non-thinking: On BFCL-V3, HiCLA achieves an Overall score of 35.25%, which is lower than GRPO’s 36.12% and GiGPO’s 36.75%. These results are based on one training run per method.
  • Format violations were more frequent in failed 8B tasks. HiCLA requires the same output format on both backbones: a state followed by an action or response, each with its corresponding tags. Across 800 tasks per backbone, 224 failed 8B tasks contained a format violation, compared with 57 failed 4B tasks (Appendix D.2, Table 5).
  • A format violation can prevent the intended tool call from executing. GRPO and GiGPO use native function calls without these additional tags. The 8B model’s difficulty following HiCLA’s format instructions therefore helps explain why HiCLA loses its advantage over these baselines.

Table R5. BFCL-V3 Overall scores and HiCLA format errors. The 4B scores are means across three training seeds. Error counts use one HiCLA checkpoint per backbone; each failed task with a format violation is counted once.

BackboneHiCLA Overall (%)GRPO Overall (%)GiGPO Overall (%)HiCLA failed tasks with format errors
Qwen3-4B-Instruct-250729.5820.7525.3357
Qwen3-8B non-thinking35.2536.1236.75224

(W2 & Q3) Sensitivity to simulator/evaluator choice

The comparison also uses a specific user simulator/evaluator (DeepSeek-v4-flash).

How sensitive are the benefits to the quality of terminal summaries/revised plans and to the simulator/evaluator choice?

We thank the reviewer for raising this question. HiCLA retains its lead and shows the smallest performance decline after the simulator/evaluator replacement. On τ²-Bench, we replaced DeepSeek-v4-flash with Qwen3-8B non-thinking as both the user simulator and evaluator during training and evaluation. We kept the agent backbone (Qwen3-4B-Instruct-2507) and training seed (20) fixed.

HiCLA retains its lead when the simulator and evaluator are replaced

  • When DeepSeek-v4-flash was used as the simulator and evaluator, HiCLA achieved an avg@4 score of 40.00%, compared with 31.50% for HiPER. After replacing DeepSeek-v4-flash with Qwen3-8B non-thinking, HiCLA still achieved the highest score, at 32.80%, compared with 22.71% for HiPER (Table R6).
  • The same ranking holds for macro pass@1 under the replacement: 34.17% for HiCLA, 25.00% for HiPER, and 19.17% for SEED (Appendix D.9, Table 9).

Table R6. Simulator/evaluator sensitivity on τ²-Bench (one training run per method and environment; avg@4 averages four evaluation seeds).

MethodDeepSeek-v4-flash avg@4 (%)Qwen3-8B non-thinking avg@4 (%)Qwen − DeepSeek (%)
Backbone24.3215.21−9.11
SEED29.7918.24−11.55
HiPER31.5022.71−8.79
HiCLA40.0032.80−7.20

HiCLA shows the smallest performance decline

  • avg@4: The score changes (Qwen − DeepSeek) were −7.20% for HiCLA, −8.79% for HiPER, −11.55% for SEED, and −9.11% for the backbone. Macro pass@1 showed the same pattern: HiCLA went from 40.83% to 34.17% (−6.66%), compared with −8.33% for HiPER and −12.50% for SEED.
  • The scores of all methods decreased, but the decrease was smaller for HiCLA than for SEED or HiPER. HiCLA therefore continued to outperform both methods on τ²-Bench after the simulator and evaluator were changed in both training and evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.