Quan-CEFT: Quantum-Inspired Computational Efficient Fine-tuning via Tensor Ring Variational Generators

Advancements in Generative Artificial Intelligence have led to rapid progress in Large Language Models (LLMs). However, training and fine-tuning LLMs presents its own set of challenges. The immense number of parameters in these models requires vast computational resources, making conventional Parameter Efficient Fine-Tuning approaches and adapters computationally expensive and impractical for many use cases. Recently, Quantum-inspired Parameter Adaptation (QPA) methods have demonstrated notable parameter reduction in Small-scale Language Models (SLMs). However, their practical application is hindered by challenges in trainability and scalability of Variational Quantum Circuits (VQC). To address these issues, we proposed a novel and scalable Quantum-inspired Computational Efficient Fine-Tuning (Quan-CEFT)The Pytorch implementation of our Quan-CEFT is available in https://anonymous.4open.science/r/QUAN-CEFT/ framework introducing a Tensor Ring simulation of Hamming-weight preserving variational Quantum Generator (TR-HQG) circuit, fully delegating the generation of VQC parameters to the TR network. As all trainable parameters reside in the classical TR cores and a Tensor Train mapping network, the circuit acts only as a fixed forward-pass evaluator, which is designed to mitigate barren plateaus and to improve training convergence, thereby enabling computationally efficient fine-tuning for SLMs. On WikiText-2 causal language modeling, the proposed Quan-CEFT framework uses $8.1\times$ to $60.5\times$ fewer trainable parameters than rank-$4$ LoRA and $2\text{mbox}-6\times$ fewer than the most compact quantum baseline across GPT-2, OPT-125M, Pythia-160M, and TinyLlama-1.1B, with $5$ to $29\%$ lower per-step training latency than QPA in CPU simulation, while retaining competitive perplexity and ROUGE-L under a short, five-step CPU adaptation budget. The reduced parameter footprint and faster CPU training of Quan-CEFT nevertheless show promise towards on-device fine-tuning on resource-constrained devices.