- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
GRASP clusters calibration gradients into steering prototypes and trains an activation-based router to choose a prototype for each input. Gradients are needed only during calibration. On seven binary-choice behaviors, the paper reports substantial accuracy gains over two base models and compares against several steering methods.
Critical strengths
-
The method is computationally simple at inference. Algorithm 1 and Appendix B.1 clearly describe the prototype construction and router training, with separate fitting, tuning, and test examples.
-
The reported gains span two backbones. Random and shuffled routing, prototype-count ablations, and oracle diagnostics provide useful evidence that selecting a direction for each input matters.
Critical weaknesses
-
The claimed behavioral geometry has an unresolved answer-label explanation. Equation 1 uses a signed A/B logit difference, so reversing the desired answer letter reverses the gradient sign. Opposing prototypes and balanced clusters could therefore reflect answer letters rather than distinct behavioral directions. The paper does not report cluster association with the desired letter or compare against a direct answer-label probe with a fixed steering axis. Figure 2 already recovers much of the gain with two rank-one prototypes, which are opposite signs of one axis. Since all evaluations use binary-choice scoring, this ambiguity affects the central interpretation of the method.
-
The comparative evidence is difficult to assess. Several baselines are described as adapted without corresponding implementation details or tuning budgets. Tables B1 and B3 give different unsteered Corrigibility and Hallucination accuracies, 74/60 versus 72/58, without explaining a change in evaluation inputs. Calibration randomness cannot explain that discrepancy on identical inputs. Each task also uses only 50 test examples, and variation across calibration seeds does not quantify uncertainty over test examples. These issues leave both the comparisons and the broader behavioral-steering claim insufficiently supported.
Questions for authors
-
How strongly do prototype assignments track the desired A/B letter? Does GRASP outperform a direct answer-label probe plus a fixed direction under the same data budget, and does the result persist when answer order is controlled?
-
What adaptations and tuning budgets were used for each baseline? Which split or preprocessing differences explain Tables B1 and B3, and what uncertainty remains when test examples are resampled?