- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HiCLA combines a persistent global plan with adaptive local subgoals in a shared LLM policy. After execution, successful and failed trajectories under the same initial plan provide terminal evidence for two auxiliary paths: rescoring the initial plan, and generating a revised plan that rescores recorded local states and decisions. A success/failure likelihood contrast trains revised-plan generation and gates local supervision. These signals supplement hierarchical GRPO and are absent at inference. Across three Qwen backbones, HiCLA has the highest reported four-evaluation-seed mean on tau2-bench among the compared methods; held-out BFCL-V3 improves over the strongest baseline at 7B and 4B but not 8B. Matched-rollout 4B ablations support both planning levels and the auxiliary objectives.
Critical strengths
-
The method makes a careful separation between privileged training feedback and deployable behaviour. Plan and execution updates retain their original generation contexts, while revised plans only supply detached scores. Separate state and decision feedback can express differences that a shared terminal advantage cannot. The revised-plan-loss ablation, which retains generation and gating, is particularly informative about learning the feedback source.
-
The evaluation includes relevant training baselines, matched environment-rollout budgets and transfer to BFCL-V3 without using it in training or checkpoint selection. The 4B task-credit-only comparison improves from 37.50% to 41.67% in the ablation protocol. The method also leads the main four-evaluation-seed tau2-bench mean on all three backbones, supporting utility beyond one isolated configuration.
Critical weaknesses
-
The central auxiliary credit signal lacks independent validation. A revised plan is rewarded for increasing relative likelihood on the same successful and failed traces used to construct it, and is not executed. The theorem includes a within-outcome KL residual, while the implemented score additionally normalises by blocks and steps; neither establishes correct local credit signs. The paper acknowledges these limits, but its diagnostics measure disagreement and association rather than whether a changed signal corrects behaviour. With only 16.04% of sampled plans eligible and 6.93% positively gated in the 4B archive, evidence from this selected subset is especially important for explaining the claimed credit-assignment benefit.
-
The robustness of the incremental gains is difficult to assess. All configurations use one training run; avg@4 repeats evaluation of a fixed checkpoint, and the mechanism ablations use a separate original evaluation without uncertainty. Several ablation gaps are small, while some comparison runs show tool-use collapse. These facts do not negate the results, but weaken the evidence that the proposed components reliably improve over a stable, tuned alternative rather than a particular training trajectory.
Questions for authors
-
Targeted validation: can existing held-out traces or a small same-state replay audit test whether positive revised-plan/local signals correspond to actual corrections? Please distinguish independent outcome evidence from likelihood and model-generated-edit diagnostics. A large new experiment is not requested.
-
Clarification: can you provide paired per-task outcomes and evaluation-seed uncertainty for the main 4B comparison and its key ablations under one protocol? Were HiPER/SEED collapse cases checked against their intended implementations and reasonable validation-selected settings? If training repeats are unavailable, please explicitly bound the reliability claim rather than treating evaluation repeats as training replication.
7 comments
Response to Reviewer ciHY (2/4)
(W1) Theory and evidence for local credit signs
The theorem includes a within-outcome KL residual, while the implemented score additionally normalises by blocks and steps; neither establishes correct local credit signs.
We thank the reviewer for raising this distinction. Proposition 1 explains the trajectory-level basis for the success–failure contrast. The local validation shows that the signals favor correct content and successful continuations, while ablations show that using them improves task performance.
(1) Theory motivates the success–failure contrast
- The population contrast is linked to the change in success log-odds.
- For two fixed plans under the stated assumptions, this change equals . The within-outcome KL residual can affect its direction (Appendices B.2–B.3).
- Normalization controls length effects in the contrast.
- Averaging within blocks and across steps prevents longer blocks or trajectories from receiving greater weight solely because of their length.
- Averaging within a fixed block preserves the sign of its summed likelihood change. A positive gate preserves that sign through .
- The gate uses a normalized contrast on the recorded pair used to construct . This differs from the population contrast (Appendix B.4).
(2) Correctness and outcomes support local signal quality
- Positive likelihood changes identify correct content and tool actions.
- Table R1 measures positive-signal precision at token and step levels before gating, using correctness labels based on task facts, domain policies, and observed feedback.
- Matched-context signals favor successful continuations, including after gating.
- Table R2 complements this correctness check with environment outcomes. For recorded training continuations from matched decision contexts, the successful side more often receives the higher signal at the next decision.
(3) Revised-plan supervision improves task performance
- Planning-to-execution OPSD contributes to the gain.
- Removing this local supervision lowers avg@4 under the matched rollout budget while retaining revised-plan training (Table R4).
- Training the revised plan and using it as the teacher both help.
- Full HiCLA scores higher than both the revised-plan-loss ablation and the terminal-evidence teacher variant (Table R4).
Together, these results support the revised-plan signals as useful auxiliary supervision alongside task-level credit.
(W2 & Q2) Reliability of performance gains
Clarification: can you provide paired per-task outcomes and evaluation-seed uncertainty for the main 4B comparison and its key ablations under one protocol?
If training repeats are unavailable, please explicitly bound the reliability claim rather than treating evaluation repeats as training replication.
HiCLA’s 4B advantage holds across training seeds, and full HiCLA scores higher than every ablation. We trained each method with three seeds for the main comparison and each ablation configuration once. Both tables use the same τ²-bench avg@4 protocol: four evaluation seeds per checkpoint. We report aggregate scores; paired per-task outcomes and separate estimates of evaluation-seed uncertainty are unavailable.
HiCLA leads both benchmarks at all three 4B training seeds
- HiCLA also outperforms GRPO and GiGPO across all three seeds.
- τ²-bench: HiCLA beats both baselines at seeds 20, 42, and 50. Its mean avg@4 is 38.40%, compared with 33.58% for GRPO, the strongest baseline on this metric.
- BFCL-V3: HiCLA also beats both at all three seeds. Its mean overall pass@1 is 29.58%, compared with 25.33% for GiGPO, the strongest baseline here.
- Table R3 reports how much scores vary when we retrain each method.
- For example, HiCLA’s τ²-bench avg@4 was 40.00%, 37.50%, and 37.71% for training seeds 20, 42, and 50, respectively. These three scores give the reported 38.40 ± 1.39% (mean ± sample standard deviation).
Table R3. Main 4B results (Qwen3-4B; selected checkpoints; mean ± sample standard deviation across three training seeds, %; Table 1).
Method τ²-bench Overall pass@1 τ²-bench avg@4 BFCL-V3 Overall GRPO 33.54 ± 0.75 33.58 ± 1.68 20.75 ± 1.02 GiGPO 33.12 ± 0.72 33.04 ± 0.76 25.33 ± 1.33 HiPER 34.44 ± 0.96 29.74 ± 2.04 22.79 ± 1.71 SEED 30.00 ± 2.21 28.06 ± 1.59 7.00 ± 0.54 HiCLA 41.67 ± 0.84 38.40 ± 1.39 29.58 ± 0.47 Independent training runs currently cover 4B. We will complete and report the 7B and 8B repeats after the rebuttal.
Response to Reviewer ciHY (3/4)
(W2 & Q2) Reliability of performance gains
Clarification: can you provide paired per-task outcomes and evaluation-seed uncertainty for the main 4B comparison and its key ablations under one protocol?
If training repeats are unavailable, please explicitly bound the reliability claim rather than treating evaluation repeats as training replication.
HiCLA’s 4B advantage holds across training seeds, and full HiCLA scores higher than every ablation. We trained each method with three seeds for the main comparison and each ablation configuration once. Both tables use the same τ²-bench avg@4 protocol: four evaluation seeds per checkpoint. We report aggregate scores; paired per-task outcomes and separate estimates of evaluation-seed uncertainty are unavailable.
HiCLA leads both benchmarks at all three 4B training seeds
- HiCLA also outperforms GRPO and GiGPO across all three seeds.
- τ²-bench: HiCLA beats both baselines at seeds 20, 42, and 50. Its mean avg@4 is 38.40%, compared with 33.58% for GRPO, the strongest baseline on this metric.
- BFCL-V3: HiCLA also beats both at all three seeds. Its mean overall pass@1 is 29.58%, compared with 25.33% for GiGPO, the strongest baseline here.
- Table R3 reports how much scores vary when we retrain each method.
- For example, HiCLA’s τ²-bench avg@4 was 40.00%, 37.50%, and 37.71% for training seeds 20, 42, and 50, respectively. These three scores give the reported 38.40 ± 1.39% (mean ± sample standard deviation).
Table R3. Main 4B results (Qwen3-4B-instruct; mean ± sample standard deviation across three training seeds, %; Table 1).
Method τ²-bench Overall pass@1 τ²-bench avg@4 BFCL-V3 Overall GRPO 33.54 ± 0.75 33.58 ± 1.68 20.75 ± 1.02 GiGPO 33.12 ± 0.72 33.04 ± 0.76 25.33 ± 1.33 HiPER 34.44 ± 0.96 29.74 ± 2.04 22.79 ± 1.71 SEED 30.00 ± 2.21 28.06 ± 1.59 7.00 ± 0.54 HiCLA 41.67 ± 0.84 38.40 ± 1.39 29.58 ± 0.47 Independent training runs currently cover 4B. We will complete and report the 7B and 8B repeats after the rebuttal.
Full HiCLA outperforms each ablation on average over four evaluation seeds
- Both OPSD directions contribute to the gain.
- Without execution-to-planning OPSD, avg@4 falls from 37.50% to 35.42%.
- Without planning-to-execution OPSD, it falls to 36.25%. Overall pass@1 also favors full HiCLA in both comparisons (Table R4).
- Training the revised plan and using it as the OPSD teacher both help.
- Removing only the revised-plan loss lowers avg@4 from 37.50% to 34.58%, even though the model still generates, scores, and gates the plan.
- Using terminal evidence directly as the teacher gives 35.83%, compared with 37.50% for the revised plan. At the same rollout budget, the revised plan provides a better OPSD signal. Overall pass@1 follows the same pattern.
Table R4. τ²-bench ablations (Qwen3-4B; avg@4, %; Table 2).
Configuration Overall pass@1 avg@4 HiCLA (full) 41.67 37.50 Without execution-to-planning OPSD 39.17 35.42 Without planning-to-execution OPSD 40.00 36.25 Without revised-plan loss 36.67 34.58 Terminal evidence as teacher 39.17 35.83 Task-level GRPO only 37.50 36.04
(W2 & Q2) Fairness of baseline comparisons
Were HiPER/SEED collapse cases checked against their intended implementations and reasonable validation-selected settings?
We checked HiPER and SEED against their original papers. Both implementations reach WebShop scores close to the published results on the same backbone. For the τ²-bench comparison, we tuned both baselines and report each run’s best validation checkpoint, including when the run collapsed later. We then examine how critic estimates and single-rollout self-reinforcement can sustain collapse.
Both implementations follow the original training procedures
- HiPER uses its hierarchical critic to assign credit. We initialize the policy and critic from the same backbone and train them jointly. As in [1, §4.3], we use the critic’s value estimates to compute advantages for subgoal selection, switching, and action execution.
- Our HiPER reproduction comes close to the published WebShop result on both metrics. At step 120, our Qwen2.5-7B-Instruct run reaches a task score of 92.17/100 and a success rate of 82.81%. For the same backbone, the original paper reports a task score of 92.2 ± 0.3 and a success rate of 83.3 ± 0.9% across three training seeds [1, Table 1].
Table R5a. HiPER WebShop validation task scores (Qwen2.5-7B-Instruct; one training run).
Training steps Task score (0–100) 50 81.90 100 83.32 120 92.17
Response to Reviewer ciHY (1/4)
We sincerely thank the reviewer for the careful and detailed review. We also appreciate the recognition of HiCLA's separation of training and inference, separate feedback for states and decisions, ablation design, and cross-task evaluation. We address the questions about the auxiliary credit signal and experimental reliability in four parts, drawing on the existing validation and additional results:
Please kindly let us know whether our response addresses the issues you raised, and whether any concerns remain. Please note that all references to the paper's sections, figures, and tables are to the revised manuscript; their relative positions may change in the revision. Tables R-prefixed tables belong to this response.
(W1 & Q1) Independent validation of local credit signals
We thank the reviewer for suggesting targeted validation. We use two checks on existing traces: whether positive local signals identify correct content and actions, and whether a higher signal favors a successful continuation when the decision context is the same. The first uses task-based correctness labels; the second uses outcomes recorded by the environment.
HiCLA achieves higher positive-signal precision at both token and step levels
Matched-context signals favor successful continuations