Beyond Uniform Ranks: Hierarchical Rank Allocation for LLM Fine-Tuning in Sentiment Analysis

Financial sentiment analysis has become crucial for market forecasting and investment decision-making, with Low-Rank Adaptation (LoRA) enabling efficient large language model (LLM) fine-tuning by introducing trainable low-rank matrices. However, LoRA typically assigns the same rank to every transformer layer, although layers respond differently to rank: across four depth groups, the rank with the lowest measured MSE ranges from 128 to 384. To address this limitation, we propose Hierarchical Rank Allocation (HiRank), a performance-based exploration framework that treats per-layer rank as a decision variable under a global parameter budget and scores each candidate allocation by the validation error of a complete fine-tuning run. HiRank searches in two phases: (1) coarse-grained joint exploration scores candidate layer groupings and narrows a rank interval for each group, and (2) fine-grained rank optimization assigns one rank per group within these intervals while keeping the total rank within the budget. The allocation is fixed before training, so the fine-tuned model keeps the inference cost of uniform LoRA. On three financial sentiment datasets and two parameter budgets, HiRank lowers MSE relative to uniform-rank LoRA in all 24 grouping, budget, and dataset settings, by 6.70% on average with eight layer groups. At the 512-rank budget and with the same number of fine-tuning evaluations, it also lowers MSE by 4.38% relative to random search over per-layer rank vectors.