How to escape the alignment resistance dilemma of LLM? An extended elasticity theory from data dimension perspective.

Alignment in large language models can be rapidly undermined by subsequent fine-tuning, a phenomenon known as alignment resistance. Existing elasticity theory explains, from the perspective of data compression, why models tend to drift toward the pretraining distribution, and establishes the relationship between the scale gap separating pretraining data and alignment data and the rate of drift. However, even when the sizes of the pretraining data and the alignment dataset are held constant, the strength of resistance to rebound still varies across. To investigate this problem, we propose Alleviation Elasticity Theory. This theory inherits the “prediction is compression” perspective from the original elasticity theory, and further employs mathematical expressions to characterize the amount of abstract information contained in datasets as well as the amount of data information absorbed by the model after alignment, which we term the alignment data dimension and the internal model representation dimension, respectively. Subsequently, using entropy as a bridge, we incorporate both into the weights of the joint data distribution, thereby obtaining a more precise characterization of the rate at which the model drifts toward the pretraining data distribution after being perturbed by fine-tuning. This theory provides a theoretical basis for identifying methods to mitigate alignment elasticity, namely that reducing the dimension of the alignment data or that of the internal model representation can improve alignment stability. While holding the sizes of the pretraining data and the alignment dataset constant, our experiments compare datasets of different dimensions and under different alignment methods, not only validating the theory but also demonstrating the effectiveness, under the guidance of this theory, of enhancing large models' resistance to rebound. Code at https://anonymous.4open.science/r/Alleviation-Elasticity-Theory-278F.