- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1(−1)
Read review
Paper summary
The authors present H-CBLM, an extension to CB-LLM that adds a hierarchical gating mechanism to enable coarse- and fine-grained control of text generation. Experiments show substantial improvements in joint steering success across six datasets.
Critical strengths
- The paper tackles a clear problem, and adding coarse-grained control on top of fine-grained concepts is a sound idea.
- It covers a wide variety of datasets, and the metrics (especially joint success) are well chosen.
- The paper is generally easy to read and follow.
Critical weaknesses
-
In the main manuscript results (Tables 2 and 3), the authors compare H-CBLM against a baseline that has neither the gating nor the steering (concept replacement) loss. Since the method is presented as a hierarchical framework, it is unclear how much of the improvement is due to the hierarchy versus the loss objective. I think a flat model trained with the steering loss would be needed to isolate the contribution of the hierarchy.
-
Section 4.4 does not compare zeroing a coarse concept against a flat model where the corresponding fine-grained concepts are zeroed directly, so it is unclear what the hierarchy adds for suppression.
-
(Minor) Some appendices are not referenced in the main text, so readers must go through them one by one to find what they contain. It would be good to have pointers in the main text to the relevant sections.
Questions for authors
-
Do the gains in Tables 2 and 3 come from the loss differences with respect to CB-LLM, or do they stem from the hierarchical addition?
-
How do the authors measure the advantage of zeroing a coarse-grained concept against directly setting its associated fine-grained concepts to zero in a flat model? In which circumstances do the authors envision coarse-grained suppression being preferable, and could they provide an experiment that shows it?
-
How sensitive is H-CBLM to the value tau set for the steering loss? Does the best inference value track the training value, or one should use a different one?
4 comments
Rebuttal (2/2)
Q2: Coarse-grained suppression vs. zeroing the children in a flat model
We ran this comparison in the tool-use setting of Section 4.4, where suppression matters because the tool-call concept is active at the decision points. We trained a flat bottleneck with the same concepts (2 coarse, 15 fine tool families) and the same objectives, and intervened at the same natural tool-call positions on all 2,700 When2Tool tasks. Mean ± sd over 3 seeds; changes relative to natural decoding.
| Model | Intervention | Δ Accuracy (pts) | Δ Tool calls / task |
|---|---|---|---|
| Flat | all 15 fine concepts → 0 | 0.00±0.00 | 0.00±0.00 |
| Flat | coarse: non-tool-call → 100, tool-call → 0 | −2.57±0.73 | −0.41±0.04 |
| H-CBLM | coarse: non-tool-call → 100, tool-call → 0 (gate closes the tool children) | −1.28±1.15 | −0.45±0.08 |
Natural accuracy is 62.3% (flat) and 62.6% (H-CBLM).
Zeroing the children does not suppress the category. In the flat model, the fine activations were positive at 83% of the 665 intervened positions, yet zeroing them changed none of the next-token decisions. The decision is carried by the coarse coordinate, which also feeds the vocabulary head directly. Suppression therefore has to act on the category. H-CBLM does this with one coordinate and guarantees that every child is closed along with it. It reduces tool calls as much as the flat coarse edit, at a comparable or lower accuracy cost.
Zeroing the parent also overrides fine-grained edits. The hierarchy matters most when a coarse switch must override fine-grained activations that are active, for example because they were set explicitly. We set a fine concept to τ (pre-gate) while zeroing its parent, or while activating a contradicting parent instead (food → service in CEBaB; positive → negative in GoEmotions), and compared with the flat model trained with the same objectives (DeepSeek-V4.1-Flash, 3 seeds; %, lower is better):
| CEBaB Flat + rec. | CEBaB H-CBLM | GoEmotions Flat + rec. | GoEmotions H-CBLM | |
|---|---|---|---|---|
| Parent zeroed: fine concept still expressed | 48.0 | 21.2 | 33.0 | 9.6 |
| Contradicting parent: fine concept and that parent both expressed | 30.8 | 10.8 | 9.6 | 3.4 |
In H-CBLM, the zeroed parent removes the fine concept to close to its natural rate, whereas the flat model still expresses it in about half (CEBaB) or a third (GoEmotions) of outputs. A GPT-5-mini re-scoring of a 40-pair subset agrees (20.8% vs. 53.3% on CEBaB; 14.2% vs. 37.5% on GoEmotions).
When coarse suppression is preferable: when the target is a category with many members, or members not known in advance (e.g., "no tool calls" rather than enumerating 15 tool families); when the category is actually active at the point of intervention; and when fine-grained activations may contradict the coarse decision, as above. By contrast, on CEBaB and GoEmotions with neutral prompts, the target concepts are rarely active. Zeroing the parent, the children, or both changes concept presence by less than 2 points for every model, so zeroing has little to suppress there.
Q3: Sensitivity to the training τ, and whether the best inference value tracks it
We retrained H-CBLM on CEBaB with training τ ∈ {50, 100, 200} (3 seeds each, identical pipeline otherwise) and swept the inference value over {10, 50, 100, 150, 200, 300, 500} with the protocol of Table C.2. Fine-grained JS (%), mean ± sd over 3 seeds; the best inference value per row is in bold.
| Training τ \ Inference | 10 | 50 | 100 | 150 | 200 | 300 | 500 |
|---|---|---|---|---|---|---|---|
| 50 | 16.7±5.1 | 42.9±5.1 | 43.3±2.6 | 13.8±4.3 | 2.1±0.7 | 0.0±0.0 | 0.0±0.0 |
| 100 | 15.0±2.2 | 28.3±2.9 | 38.8±2.2 | 36.7±6.2 | 16.2±3.3 | 1.2±1.2 | 0.0±0.0 |
| 200 | 19.6±4.7 | 23.8±4.5 | 32.1±4.7 | 42.1±3.8 | 37.2±3.2 | 21.7±10.5 | 0.8±0.7 |
The best inference value tracks the training τ. The JS peak moves from 50–100 (τ = 50) to 100 (τ = 100) to 150–200 (τ = 200). Hit and fluency shift in the same way: Hit peaks at 150, 200 and 300, and fluency falls below 3 at 150, 200 and 300 for τ = 50, 100 and 200, respectively. The peak JS itself is insensitive to τ (38.8–43.3%). The best inference value is at or slightly above the training value; with the 100-token cap of Tables C.2–C.4, JS for τ = 100 models also peaks at 150–200. We nevertheless use the training value (τ = 100) as the intervention value in all main results, fixed in advance without any selection on test outputs.
To control cost, these runs were scored with DeepSeek-V4.1-Flash instead of GPT-5-mini. Absolute scores may differ between evaluators, but the trends we report are robust to this choice.
Minor: Appendix references
Thank you for pointing this out. In the revision, we will add pointers in the main text to every appendix section where the related content is discussed, so that readers can locate the supporting details directly.
Thank you for running the additional experiments and addressing my questions.
-
I appreciate the new experiments comparing coarse-grained suppression with directly zeroing the fine-grained concepts in a flat model. The experiments involving conflicting parent and child interventions help demonstrate the practical utility of hierarchical gating.
-
The sensitivity analysis on is also useful. I think it would be useful to translate this into a practical recommendation for users in the paper.
-
My main remaining concern is that the new ablations suggest that the improvements reported in the main results come primarily from the reconstruction objective rather than the hierarchical gating mechanism. I acknowledge that the hierarchy provides additional value, as demonstrated by the new experiments in Q2. However, I believe the current manuscript does not sufficiently disentangle these contributions or adequately reflect their respective roles in the reported improvements.
I appreciate the additional work, but I believe that the findings would require a broader revision of the manuscript, particularly in how the contributions are framed and how the main results are presented and interpreted. I therefore maintain my original score.
Thank you for your valuable feedback and suggestions. We agree that the manuscript did not sufficiently disentangle the contributions of the reconstruction objective and the hierarchical gating. We have revised the framing of the contributions in the abstract, introduction, Section 4.3, and Appendix C.1. The revised version now states that the improvements in the main results come primarily from the reconstruction objective, while the hierarchical gating provides complementary benefits for controllability, as shown by the coarse-grained suppression and conflicting parent–child intervention experiments. However, the analysis of the hierarchical design still needs more evidence to support it. We fully respect your assessment. Thank you again for your time and constructive comments.
Rebuttal (1/2)
Q1: Do the gains come from the loss or from the hierarchy?
Thanks for raising this question. To complete the 2×2 design, we trained a flat CB-LLM with the same reconstruction objective (τ=100, same data, budget, seeds and protocol). Mean ± sd over 3 seeds, judged by GPT-5-mini as in the paper.
Fine-grained control (Table 2 protocol)
Coarse-grained control (Table 3 protocol)
The steering gains come mainly from the reconstruction objective. It raises fine-grained JS from 0.2–11.7% to 41–56% with or without gating. With reconstruction, gating changes fine- and coarse-grained JS by at most 3 points, within seed variation.
The hierarchy instead provides category-level switching: a child is exactly zero at every token where its parent is inactive, so a single coarse coordinate switches a whole category on or off, which a flat model does not guarantee (see our response to Q2).