acceptodds
Desk rejected at ICLR 2027

Beyond Uniform Ranks: Hierarchical Rank Allocation for LLM Fine-Tuning in Sentiment Analysis

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Financial sentiment analysis has become crucial for market forecasting and investment decision-making, with Low-Rank Adaptation (LoRA) enabling efficient large language model (LLM) fine-tuning by introducing trainable low-rank matrices. However, LoRA typically assigns the same rank to every transformer layer, although layers respond differently to rank: across four depth groups, the rank with the lowest measured MSE ranges from 128 to 384. To address this limitation, we propose Hierarchical Rank Allocation (HiRank), a performance-based exploration framework that treats per-layer rank as a decision variable under a global parameter budget and scores each candidate allocation by the validation error of a complete fine-tuning run. HiRank searches in two phases: (1) coarse-grained joint exploration scores candidate layer groupings and narrows a rank interval for each group, and (2) fine-grained rank optimization assigns one rank per group within these intervals while keeping the total rank within the budget. The allocation is fixed before training, so the fine-tuned model keeps the inference cost of uniform LoRA. On three financial sentiment datasets and two parameter budgets, HiRank lowers MSE relative to uniform-rank LoRA in all 24 grouping, budget, and dataset settings, by 6.70% on average with eight layer groups. At the 512-rank budget and with the same number of fine-tuning evaluations, it also lowers MSE by 4.38% relative to random search over per-layer rank vectors.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
low-rank adaptation, layer-wise rank allocation, parameter-efficient fine-tuning, budget-constrained search, hierarchical layer grouping, financial sentiment regression

Related papers

Discussion (0)

Sign in to comment.