On the Inductive Reasoning Capacity of Large Language Models

The rapid advancement of Large Language Models (LLMs) has sparked attention to their reasoning capacities. In this work, we investigate the underlying mechanics of inductive reasoning in LLMs and discover that internal hidden embedding trajectories capture the respective rule acquisition dynamics. We show that hidden-layer velocity functions serve not only as a strongly correlated signal of the underlying logic, but also as a causal lever capable of steering model behavior during reasoning. Building on this discovery, we introduce a contrastive fine-tuning framework that directly optimizes these representation trajectories to enhance rule inference. Comprehensive evaluation across 8 benchmarks demonstrates the effectiveness of our approach, improving LLM inductive reasoning accuracy by up to 23 percentage points in in-domain generalization and up to 8.6 percentage points in cross-domain generalization.