- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies inductive reasoning using differences between hidden states after successive examples, termed representation velocities. It investigates their relationship with rules through similarity analysis and activation steering, then proposes velocity-based contrastive fine-tuning. Experiments cover two models, eight main benchmarks, and several generalization settings.
Critical strengths
- Training on changes between representations is a useful contribution: the gains over static contrastive learning suggest that these changes contain information worth optimizing. This remains valuable even if the proposed mechanistic explanation is narrowed.
- Two models, eight benchmarks, and several generalization settings provide substantial supporting evidence. The same-rule donor interventions and incorrect/random-direction controls also connect the training idea to observable model behavior.
Critical weaknesses
- The novelty discussion should distinguish velocity-based training from prior Task Vectors and Function Vectors. This is a positioning issue; the existing static-CL comparison already provides useful evidence for the training contribution.
- The main concern is the evaluation protocol: rule definitions, split sizes, and scoring of programs or natural-language rules are insufficiently specified. These details determine what the central claim of unseen-rule generalization actually measures.
- The implementation needs clarification: Section 4 defines a velocity sequence, while Section 5 uses a d-dimensional vector. The step alignment or pooling, and the rule identities used to form training pairs, should be explicit.
- The interventions support behaviorally relevant directions, but Tables 1 and 6 omit unmodified-model Pass@10. They therefore establish the claimed improvement over normal inference less clearly than the ranking among interventions.
- Minor corrections: Table 10's largest gain is 19.06 points, not 19.9; capitalize “particularly” on page 8; change “a (incomplete) sequence” to “an incomplete sequence” on page 15.
Questions for authors
- Can you provide the rule-level split definitions, sample counts, representative constructed tasks, and scoring procedures needed to verify the generalization results?
- What exact velocity representation enters each loss and intervention, and how are positive/negative rule identities obtained?
- What is unmodified-model Pass@10 under the steering experiments' decoding settings, and are the intervention magnitudes comparable?
- What training examples, CoT supervision, and optimization budgets are used for COCONUT/CODI, and what varies across the five independent runs?
3 comments
Thank you for the revisions. Several concerns have been addressed, but the remaining inconsistency in the dataset statistics leaves me uncertain about the construction of the rule-generalization evaluation. I am therefore revising my rating from 3 (Weak acceptance) to 2 (Weak rejection), pending clarification.
Clarification regarding the number of sequences in Table 6
We thank the reviewer for pointing out this ambiguity and would like to provide a clarification. In the new Table 6, each benchmark contains a certain number of rules for training & held-out test sets (two sets are non-overlapped). For each rule, we generate a number of sequences governed by that rule.
For most benchmarks, we generate approximately 5 sequences per rule. For Crypto with a small number of rules, we generate more sequences per rule (approximately 28-29 sequences per rule for Crypto). The previous sentence "3-5 sequences per rule" was not accurate to describe Table 6 as a whole. We have made the correction accordingly in the latest revision.
As explained in Appendix A3, from each benchmark, the set of rules is partitioned into 2 disjoint sets, one for training and one for held-out evaluation. The training set of rules is used to generate sequences for training, and the held-out evaluation set of rules is used to generate sequences for held-out evaluation (Tables 2 & 8). These 2 sets are employed consistently among all methods and across 5 independent runs.
We hope this clarification addresses the reviewer's concern.
Novelty, Implementation Details and Additional Steering Results
We thank the reviewer for the constructive feedback that helps us improve our paper. W1: Novelty Discussion We have added a paragraph at the end of Section 2.3. This paragraph clarifies the distinction between our approach and Task Vectors and Function Vectors, which demonstrate how internal task representations can guide model behavior. In contrast, our work investigates how these representations evolve across successive observations and exploits their dynamics through velocity-based contrastive fine-tuning to improve inductive reasoning. Furthermore, our static-CL baseline provides empirical evidence for the additional benefit of optimizing representation velocities rather than static representations.
W2 & Q1: Evaluation Protocol
W3 & Q2: Implementation Velocity clarification: thank you for pointing out the notation mismatch in Section 5. We have corrected it to be consistent with Section 4. Rule identities:
W4 & Q3: Intervention We have added the results for unmodified model (``unsteered” in our paper). Please find Tables 1 & 7. According to our findings, steering in the right direction leads to the highest number of accurate predictions, surpassing unsteered (no intervention) and steering along incorrect directions.
W5: Writing Correction Thank you for pointing it out. We have made the corrections accordingly.
Q4: Clarification on competitors