BLOC: Blockwise Composition for Adaptive-Rank Fine-Tuning in LLMs

Parameter-efficient fine-tuning (PEFT) critically depends on how weight updates are parameterized with a small number of trainable parameters. A common PEFT approach assumes that the trainable matrices are formulated as a transformation of the product of two or more low-dimensional matrices. Such parameterizations often impose structural assumptions on the resulting update, such as favoring low- or high-rank solutions, rather than allowing the realized rank to emerge from optimization and vary adaptively across layers. To address this limitation, we propose BLOC, a PEFT method based on BLOckwise Composition. Specifically, BLOC partitions each weight update matrix into smaller blocks and constructs each block as a sparse linear combination of trainable block components defined directly in the block space. To control the number of trainable parameters, BLOC shares these components across compatible Transformer layers and uses a one-time gradient-guided calibration for task-informed initialization. Across three backbones and diverse reasoning tasks, BLOC achieves strong performance, while producing adaptive-rank updates and effectively reusing shared block components across layers.