acceptodds
Under review as a conference paper at ICLR 2027

On the Inductive Reasoning Capacity of Large Language Models

Abstract

The rapid advancement of Large Language Models (LLMs) has sparked attention to their reasoning capacities. In this work, we investigate the underlying mechanics of inductive reasoning in LLMs and discover that internal hidden embedding trajectories capture the respective rule acquisition dynamics. We show that hidden-layer velocity functions serve not only as a strongly correlated signal of the underlying logic, but also as a causal lever capable of steering model behavior during reasoning. Building on this discovery, we introduce a contrastive fine-tuning framework that directly optimizes these representation trajectories to enhance rule inference. Comprehensive evaluation across 8 benchmarks demonstrates the effectiveness of our approach, improving LLM inductive reasoning accuracy by up to 23 percentage points in in-domain generalization and up to 8.6 percentage points in cross-domain generalization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Inductive Reasoning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_E5qWOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies inductive reasoning using differences between hidden states after successive examples, termed representation velocities. It investigates their relationship with rules through similarity analysis and activation steering, then proposes velocity-based contrastive fine-tuning. Experiments cover two models, eight main benchmarks, and several generalization settings.

Critical strengths
  1. Training on changes between representations is a useful contribution: the gains over static contrastive learning suggest that these changes contain information worth optimizing. This remains valuable even if the proposed mechanistic explanation is narrowed.
  2. Two models, eight benchmarks, and several generalization settings provide substantial supporting evidence. The same-rule donor interventions and incorrect/random-direction controls also connect the training idea to observable model behavior.
Critical weaknesses
  1. The novelty discussion should distinguish velocity-based training from prior Task Vectors and Function Vectors. This is a positioning issue; the existing static-CL comparison already provides useful evidence for the training contribution.
  2. The main concern is the evaluation protocol: rule definitions, split sizes, and scoring of programs or natural-language rules are insufficiently specified. These details determine what the central claim of unseen-rule generalization actually measures.
  3. The implementation needs clarification: Section 4 defines a velocity sequence, while Section 5 uses a d-dimensional vector. The step alignment or pooling, and the rule identities used to form training pairs, should be explicit.
  4. The interventions support behaviorally relevant directions, but Tables 1 and 6 omit unmodified-model Pass@10. They therefore establish the claimed improvement over normal inference less clearly than the ranking among interventions.
  5. Minor corrections: Table 10's largest gain is 19.06 points, not 19.9; capitalize “particularly” on page 8; change “a (incomplete) sequence” to “an incomplete sequence” on page 15.
Questions for authors
  1. Can you provide the rule-level split definitions, sample counts, representative constructed tasks, and scoring procedures needed to verify the generalization results?
  2. What exact velocity representation enters each loss and intervention, and how are positive/negative rule identities obtained?
  3. What is unmodified-model Pass@10 under the steering experiments' decoding settings, and are the intervention magnitudes comparable?
  4. What training examples, CoT supervision, and optimization budgets are used for COCONUT/CODI, and what varies across the five independent runs?
3 comments
AuthorsOpenReview ↗

Novelty, Implementation Details and Additional Steering Results

We thank the reviewer for the constructive feedback that helps us improve our paper. W1: Novelty Discussion We have added a paragraph at the end of Section 2.3. This paragraph clarifies the distinction between our approach and Task Vectors and Function Vectors, which demonstrate how internal task representations can guide model behavior. In contrast, our work investigates how these representations evolve across successive observations and exploits their dynamics through velocity-based contrastive fine-tuning to improve inductive reasoning. Furthermore, our static-CL baseline provides empirical evidence for the additional benefit of optimizing representation velocities rather than static representations.

W2 & Q1: Evaluation Protocol

  • Rule definition and split sizes: please find the details in Appendix A3 where we provide the statistics and obtain the train-test split.
  • To illustrate the rules and instances in each benchmark, we have provided some examples in Appendix A4.
  • Scoring: the model is tasked with predicting the next value of the sequence. The evaluation for each sequence is binary: accurate (if the model predicts correctly the next value) or inaccurate. The results reported in the Tables are the percentage of test sequences for which the model correctly predicts the next element.

W3 & Q2: Implementation Velocity clarification: thank you for pointing out the notation mismatch in Section 5. We have corrected it to be consistent with Section 4. Rule identities:

  • For each transformation rule, we generate several sequences.
  • Given a particular sequence (anchor), we select another sequence generated by the same rule as "positive” and a sequence generated by a different rule as "negative”.
  • As the model progresses through each sequence, we compute the velocity vector. As the result, for each triplet (anchor, positive, negative) of sequences, we obtain a corresponding triplet (anchor, positive, negative) of velocity vectors. The triplet of velocity vectors goes into the training loss in Equation 1.

W4 & Q3: Intervention We have added the results for unmodified model (``unsteered” in our paper). Please find Tables 1 & 7. According to our findings, steering in the right direction leads to the highest number of accurate predictions, surpassing unsteered (no intervention) and steering along incorrect directions.

W5: Writing Correction Thank you for pointing it out. We have made the corrections accordingly.

Q4: Clarification on competitors

  • We employ the same training examples: the sequences bearing the transformation rules in the training set. In our set-up, given a sequence of m steps, the last step as the output and all previous steps are treated as the chain of reasoning. We run the methods natively, without constraining the running time or GPU resources.
  • Across five independent runs:
  • The train/test split remains fixed and the same among the methods.
  • Five independent runs reflect 5 random seeds, affecting initialization and training data orders.

Thank you for the revisions. Several concerns have been addressed, but the remaining inconsistency in the dataset statistics leaves me uncertain about the construction of the rule-generalization evaluation. I am therefore revising my rating from 3 (Weak acceptance) to 2 (Weak rejection), pending clarification.

AuthorsOpenReview ↗

Clarification regarding the number of sequences in Table 6

We thank the reviewer for pointing out this ambiguity and would like to provide a clarification. In the new Table 6, each benchmark contains a certain number of rules for training & held-out test sets (two sets are non-overlapped). For each rule, we generate a number of sequences governed by that rule.

For most benchmarks, we generate approximately 5 sequences per rule. For Crypto with a small number of rules, we generate more sequences per rule (approximately 28-29 sequences per rule for Crypto). The previous sentence "3-5 sequences per rule" was not accurate to describe Table 6 as a whole. We have made the correction accordingly in the latest revision.

As explained in Appendix A3, from each benchmark, the set of rules is partitioned into 2 disjoint sets, one for training and one for held-out evaluation. The training set of rules is used to generate sequences for training, and the held-out evaluation set of rules is used to generate sequences for held-out evaluation (Tables 2 & 8). These 2 sets are employed consistently among all methods and across 5 independent runs.

We hope this clarification addresses the reviewer's concern.

Reviewer_VoeXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observed high similarity among velocities of hidden features of strings (collected by "\n") during the induction/inference of LLM for number series, and proposed contrastive learning to improve similarities of feature velocities to improve LLM performance on induction tasks of number series. The method is benchmarked on a limited domain of tasks -- guessing next number, without showing its impact to a larger scope of reasoning capability of LLMs. The capability of induction in this paper is also limited to guessing next number instead of figuring out the underlying rule.

Critical strengths

None

Critical weaknesses

The method is benchmarked on a limited domain of tasks -- guessing next number, without showing its impact to a larger scope of reasoning capability of LLMs. The capability of induction in this paper is also limited to guessing next number instead of figuring out the underlying rule.

Questions for authors
  • what is its impact to general and reasoning benchmarks after training?
2 comments
AuthorsOpenReview ↗

Clarification on the Evaluation scope of our paper

We thank the reviewer for the comment and question. While we acknowledge that our evaluation may not fully characterize the general inductive reasoning capabilities of LLMs, our experiments extend beyond next-number prediction. We evaluate eight benchmarks covering diverse inductive reasoning tasks, including program synthesis (SyGuS), explicit rule induction (InductionBench), and instruction induction (Instruction), which require models to identify or generate underlying rules rather than merely predict the next element.

Furthermore, Tables 4 and 9 demonstrate improvements on broader reasoning benchmarks after fine-tuning solely on FACT, with gains of 5.43/6.51 percentage points on BBH/BBEH for Qwen-3 8B and 8.45/7.80 percentage points for Llama 3.1 8B. Importantly, these improvements are achieved without fine-tuning on the target benchmarks, providing evidence that the benefits of velocity-based training transfer beyond numerical sequence prediction to more diverse and challenging reasoning tasks across different model architectures. While these findings are encouraging, we acknowledge that further evaluation is needed to establish their broader generality.

Thanks for the feedback. I would keep the original rating as my concerns haven't been fully resolved.

Reviewer_bmdcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inductive reasoning in LLMs. The authors measure how hidden states change between examples in a sequence and call this change “velocity.” They find that sequences following the same rule have similar velocities. They also use a velocity from another sequence with the same rule to steer the model. Based on these observations, they train Qwen-3 8B and Llama 3.1 8B with a contrastive loss that brings same-rule velocities closer.

The method improves accuracy on many benchmarks. For example, on Qwen-3, the largest gain over the base model in the held-out rule setting is about 23 percentage points on SyGuS.

Critical strengths
  1. The paper studies a useful question: whether internal representations contain information about an induced rule.

  2. It combines representation analysis, an intervention experiment, and a training method. This gives a more complete story than accuracy results alone.

  3. The evaluation includes two model families and several settings, including held-out rules, composed rules, and changed surface forms. The gains over the static-representation contrastive baseline are encouraging.

Critical weaknesses
  1. The causal claim is too strong for the current steering experiment. The “correct velocity” comes from another sequence that follows the target rule. This gives the intervention information about the correct rule. Table 1 compares it with incorrect and random directions, but does not show an unsteered model. The experiment shows that this direction can influence the output; it does not yet establish that the model naturally uses velocity as its reasoning mechanism. Please add the unsteered baseline and test directions estimated without access to the target rule.

  2. Orthogonal alignment needs a held-out test. Appendix A.1 fits a rotation to bring two velocity matrices closer, and Figures 1–2 then measure similarity after that alignment. Since the rotation is fitted from the compared data, the stronger post-alignment separation is difficult to interpret. The unaligned plots are useful evidence, but the aligned result should be checked by fitting the rotation on one set of sequences and evaluating it on different sequences.

  3. The training signal is described as self-supervised, but positive pairs require knowing which instances share the same rule. These rule identities are meaningful supervision. Please clarify where the rule labels come from and compare with a baseline that uses the same labels and training budget in a simpler way.

  4. Several implementation details needed to judge the result are missing or hard to find. The paper should clearly report the selected hidden layer, how velocity sequences of different lengths are compared, steering magnitude, numbers of rules and examples in each split, and training hyperparameters. The exact held-out rule and composition splits are especially important for the generalization claim.

  5. The out-of-domain conclusion should be more precise. In Table 4, the approximately 8.6-point Qwen gain is on NumSeq, which the paper calls close-domain. The gains on BBH and BBEH are smaller. I suggest reporting close-domain and out-of-domain results separately in the abstract and conclusions.

Questions for authors

See weaknesses.

  • How sensitive are the results to the hidden layer and steering strength?
  • Would a rule-label supervised contrastive baseline, without the velocity objective, obtain similar gains?
1 comment
AuthorsOpenReview ↗

Additional Observations & Implementation Details

We sincerely thank the reviewer for the valuable suggestions. To improve our paper, we have addressed the comments of the reviewer as follows:

W1: Steering experiments We have added the results for unsteered & steered in the velocity of the very sequence. In the later, we assume no knowledge of the target rule as we only compute the velocity vector of the sequence in question. Please refer to Tables 1 & 7. According to our findings, steering in the right direction leads to the highest number of accurate predictions, surpassing unsteered (no intervention) and steering along incorrect directions.

W2: Orthogonal alignment. We agree that fitting the orthogonal transformation on the evaluated data may introduce optimistic bias. To address this, we conducted a held-out evaluation, fitting the rotation on training sequences and applying it to disjoint test sequences without refitting. This ensures that the post-alignment similarity is evaluated independently of the data used to estimate the transformation.

W3 & Q2: Rule identities and Simple baseline Supervision: While we did not explicitly encode the rule or the outcome by following the rule to the training, we agree that rule identities (same transformation rule vs. different transformation rules) are employed in supervision.
Rule labels: During the training, for each transformation rule (in the training data), we generate several sequences bearing that rule. The velocity of hidden layer embedding of these sequences are employed for training. For contrastive learning in Section 5, for each sequence (anchor), we select those sequences bearing the same rule as positives and those bearing different rules as negatives. Baseline: Please pay attention to baseline STATIC, by which we also employ a contrastive learning scheme on the same rule labels. However, instead of velocity, we compute and train on the similarity of the hidden layer vector, extracted when LLM processes the last sample of the training sequence. Refer to Appendix A5 for a detailed comparison between STATIC and our proposed method. According to Tables 2-5, 8-11, training contrastive learning on static embedding proves sub-optimal, as STATIC only offers minor improvement over the base model in a handful of datasets while significantly lags behind Qwen-3 CL and Llama CL.

W4: Implementation Details

  • Hidden layer to extract velocity: heuristically, we extract velocity vector at layer 24 for Qwen 3 and layer 21 for Llama . We will supplement the experimental results to demonstrate the sensitivity with regards to the hidden layer.
  • Steering magnitude: the steering magnitude for the experiments in Section 4.2 is 1.
  • Train & test split sizes: please see Table 6 in Appendix A3.
  • For illustration of held-out and composition, please find the examples in Appendix A4.

W5: Writing correction Thank you for pointing out the issues. We have made the corrections accordingly.

Q1: Sensitivity to hidden layer and steering strength We appreciate this helpful suggestion. Please find Tables 8, 9, 10 in Appendix A6 where we examine the sensitivity to the hidden layer and steering magnitude. We repeat the experiment in Section 4.2 to employ the velocity as an intervention and compare the prediction accuracy versus the unsteered variant. The results suggest that while the performance is largely insensitive to the designated layer, it is lightly sensitive as the steering magnitude varies. Nonetheless, these results still reinforce that velocity serves as a helpful causal lever for the reasoning performance of LLMs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.