acceptodds
Under review as a conference paper at ICLR 2027

How to escape the alignment resistance dilemma of LLM? An extended elasticity theory from data dimension perspective.

Abstract

Alignment in large language models can be rapidly undermined by subsequent fine-tuning, a phenomenon known as alignment resistance. Existing elasticity theory explains, from the perspective of data compression, why models tend to drift toward the pretraining distribution, and establishes the relationship between the scale gap separating pretraining data and alignment data and the rate of drift. However, even when the sizes of the pretraining data and the alignment dataset are held constant, the strength of resistance to rebound still varies across. To investigate this problem, we propose Alleviation Elasticity Theory. This theory inherits the “prediction is compression” perspective from the original elasticity theory, and further employs mathematical expressions to characterize the amount of abstract information contained in datasets as well as the amount of data information absorbed by the model after alignment, which we term the alignment data dimension and the internal model representation dimension, respectively. Subsequently, using entropy as a bridge, we incorporate both into the weights of the joint data distribution, thereby obtaining a more precise characterization of the rate at which the model drifts toward the pretraining data distribution after being perturbed by fine-tuning. This theory provides a theoretical basis for identifying methods to mitigate alignment elasticity, namely that reducing the dimension of the alignment data or that of the internal model representation can improve alignment stability. While holding the sizes of the pretraining data and the alignment dataset constant, our experiments compare datasets of different dimensions and under different alignment methods, not only validating the theory but also demonstrating the effectiveness, under the guidance of this theory, of enhancing large models' resistance to rebound. Code at https://anonymous.4open.science/r/Alleviation-Elasticity-Theory-278F.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model alignment, alignment rebound, elasticity theory, dataset intrinsic dimensionality, model-perceived dimensionality, data compression.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GKg8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends an existing compression-based elasticity model by incorporating dataset entropy and internal representation dimension into the relative weights of pretraining and alignment data. It recommends lower-dimensional alignment datasets or representations to improve robustness to subsequent fine-tuning. Three DistilGPT2/IMDb experiments compare initial positive-sentiment alignment and its retention under negative-review fine-tuning, using different data subsets and alignment methods.

Critical strengths
  1. The paper addresses a meaningful question: why equally sized alignment corpora or different alignment methods can produce different retention under subsequent fine-tuning. Distinguishing dataset structure from internal representation structure provides a useful perspective.
  2. The paper extends elasticity theory to investigate how data and representation structure may affect alignment rebound. The accompanying experiments provide preliminary evidence of retention differences across datasets and alignment methods beyond sample count.
Critical weaknesses
  1. It is unclear why lower dimension should reduce κ\kappa. Section 3.4 says lower-dimensional alignment data decrease κ\kappa. If lower dimension reduces alignment entropy HaH_a, as the leading term in Eqs. (7) and (9) suggests, Eq. (3) instead gives a larger κ\kappa, with other terms fixed. Lowering internal dimension does the same for G(μ)=μG(\mu)=\mu. The paper needs to explain how these relations support using lower-dimensional data or representations to reduce rebound.
  2. The experiments do not yet support the broader claims. All experiments use DistilGPT2 on IMDb sentiment continuation, so it is unclear whether the findings generalize to other datasets or alignment tasks. More importantly, the paper defines rebound as movement toward the pretraining distribution, while the evaluation relies on positive-output rates. These capture sentiment retention but do not directly measure that return. A metric more closely tied to the pretraining distribution would provide stronger support for the paper's interpretation.
Questions for authors
  1. Why should lower dimension reduce κ\kappa in Section 3.4, when reducing HaH_a or G(μa)G(\mu_a) in Eq. (3) increases it with other terms fixed?
  2. Can the authors assess rebound with a metric more directly tied to the pretraining distribution, and check whether the findings hold beyond IMDb sentiment continuation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer GKg8

  • W2-1 : The experiments do not yet support the broader claims. All experiments use DistilGPT2 on IMDb sentiment continuation, so it is unclear whether the findings generalize to other datasets or alignment tasks.

We supplement our study with additional experiments across varying model scales, tasks, and random seeds. The results validate the conclusion that lower dimension alleviates alignment elasticity. Please refer to Q2a and Q2b of the General response for further details.

  • W2-2: More importantly, the paper defines rebound as movement toward the pretraining distribution, while the evaluation relies on positive-output rates. These capture sentiment retention but do not directly measure that return. A metric more closely tied to the pretraining distribution would provide stronger support for the paper's interpretation.

We conduct supplementary experiments to measure the KL divergence between the current model and the pretrained model during negative sentiment fine-tuning. The results show that the KL divergence between the two exhibits a decreasing trend, whereas it shows an increasing trend under direct fine-tuning without prior alignment. This indicates that the model exhibits a tendency to return toward the pretraining distribution.

Table 1

Number of fine-tuning samplesseed17seed42direct fine-tuning without prior alignment
00.3430.3490.000
1280.3360.3400.075
5120.3290.3310.134
10240.3180.3220.176
15360.3100.3140.194
20000.3050.3070.211
  • Q1/W1: It is unclear why lower dimension should reduce κ\kappa. Section 3.4 says lower-dimensional alignment data decrease κ\kappa. If lower dimension reduces alignment entropy HaH_a, as the leading term in Eqs. (7) and (9) suggests, Eq. (3) instead gives a larger κ\kappa, with other terms fixed. Lowering internal dimension does the same for G(μ)=μG(\mu)=\mu. The paper needs to explain how these relations support using lower-dimensional data or representations to reduce rebound.

We sincerely apologize for the confusion this caused. This is a typographical error in our writing(Section 3.4). For detailed explanation and why dimension reduction weakens alignment elasticity, please refer to Q1 of the General response.

  • Q2: Can the authors assess rebound with a metric more directly tied to the pretraining distribution, and check whether the findings hold beyond IMDb sentiment continuation?

By measuring the KL divergence between the fine-tuned model and the pretrained model, the experiments (W2-2) confirm that the model exhibits a tendency to return toward the pretraining distribution.

We supplement our study with additional experiments across varying model scales, tasks, and random seeds. The results validate the conclusion that lower dimension alleviates alignment elasticity. Please refer to Q2 of the General response for further details.

We hope this clarifies our contribution and would appreciate your reconsideration of the score.

We kindly remind you to read our rebuttal. We sincerely apologize for a typo in Line 303: it is an easily fixable typo by replacing decreases with increase, and all the other parts are based on the correct statement. So we hope you read our rebuttal and reconsider your score. Thank you very much!

We kindly remind you that the rebuttal period is drawing to a close. We sincerely hope to clarify the misunderstandings in our paper and address your concerns, and we truly appreciate your taking the time to read our response.

Reviewer_bRQXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends compression-based alignment elasticity theory by incorporating dataset entropy and internal representation dimension into an effective data-volume ratio. It argues that lower-dimensional alignment data or representations improve robustness to subsequent fine-tuning. The paper also proves, under geometric and partition-correspondence assumptions, that leaf-node entropy scales with dataset intrinsic dimension via covering and quantization entropy. Experiments use DistilGPT2 on IMDb, comparing equal-sized data subsets and three alignment methods: full SFT, LoRA, and BitFit.

Critical strengths

Relevant research question. Investigating robustness differences beyond alignment dataset size is worthwhile. The distinction between dataset geometry and internal representation geometry offers a potentially useful perspective on data selection and alignment methods.

Critical weaknesses
  1. The central theoretical conclusion is unsupported. In Eqs. (1) and (3), the alignment entropy and representation factors HaG(μa)H_aG(\mu_a) appear in the denominator of the elasticity coefficient κ\kappa. Lower-dimensional alignment data would reduce HaH_a and/or G(μa)G(\mu_a), thereby increasing κ\kappa, not decreasing it as Section 3.4 claims. Moreover, the effective volume niHiG(μi)n_iH_iG(\mu_i) is introduced as a factorization assumption rather than derived for the training procedure. The conditional-entropy results in Section 3.3 only link leaf-node entropy to intrinsic dimension; they do not justify the effective-volume factorization or resolve the sign contradiction.

  2. The experiments do not establish the claimed dimension effect or its relevance to LLM alignment. All results use DistilGPT2 on a single sentiment task, without multi-seed validation. Selecting different corpora also changes their content, while switching SFT/LoRA/BitFit changes optimization beyond representation dimension. No direct test of the predicted compression-rate relation is provided. Experiment 1 reports Pearson/Spearman p-values of 0.144/0.109; Experiment 2 is a single run; Experiment 3 is acknowledged by the authors to be inconsistent across multiple rounds. These experiments offer preliminary correlations, but do not establish that reducing dimension improves safety alignment or causes slower rebound toward the pretraining distribution.

Questions for authors
  1. Clarification: How can the low-dimension mitigation claim be derived consistently from Eqs. (1) and (3), and what justifies the proposed effective-volume weighting for actual training?

  2. Clarification: Do the authors have existing multi-seed or controlled results that isolate dataset dimension from corpus composition and optimization?

Suggestion for future work: Controlled repeated experiments and validation on a safety-alignment task with larger models would strengthen the claim, but I do not require them for this assessment.

6 comments
AuthorsOpenReview ↗

Response to Reviewer bRQX

  • W2-1: All results use DistilGPT2 on a single sentiment task, without multi-seed validation.

We supplement our study with additional experiments across varying model scales, tasks, and random seeds. The results validate the conclusion that lower dimension alleviates alignment elasticity. Please refer to Q2 of the General response for further details.

  • W2-2: Experiment 1 reports Pearson/Spearman p-values of 0.144/0.109.

These two values indicate that there is no statistically significant correlation between dataset dimensionality and initial alignment performance, which is exactly consistent with our experimental expectation. The main claim of this work is that lower dimensionality alleviates alignment elasticity. If lower dimensionality led to worse initial alignment performance, mitigating alignment elasticity would be meaningless. This experiment demonstrates that mitigating alignment elasticity via lower dimensionality is meaningful.

  • W2-3: Experiment 2 is a single run.

We supplement multi-seed repeated trials, and the experimental results are consistent with our theory. Please refer to Q2d of the General response for details.

  • W2-4: Experiment 3 is acknowledged by the authors to be inconsistent across multiple rounds.

The instability observed in Experiment 3 only manifests in one experimental group: the theoretical pattern is less pronounced for this group. we perform repeated experiments across multiple random seeds, and the results validate our conclusion. Detailed experimental results are provided in Q2 of the General response.

  • W2-5: No direct test of the predicted compression-rate relation is provided.These experiments offer preliminary correlations, but do not establish that reducing dimension improves safety alignment or causes slower rebound toward the pretraining distribution.

We conduct supplementary experiments to measure the KL divergence between the fine-tuned model and the pretraining model during negative sentiment fine-tuning. The results show a decreasing trend in this KL divergence, indicating that the model tends to move toward the pretraining distribution.

Table 1

Number of fine-tuning samplesseed17seed42direct fine-tuning without prior alignment
00.3430.3490.000
1280.3360.3400.075
5120.3290.3310.134
10240.3180.3220.176
15360.3100.3140.194
20000.3050.3070.211
  • Q1-1/W1-1: Clarification: The inconsistency regarding the decrease of κ\kappa in Section 3.4. How can the low-dimension mitigation claim be derived consistently from Eqs. (1) and (3).

We sincerely apologize for the confusion this caused. This is a typographical error in our writing(Section 3.4). For detailed explanation and why dimension reduction weakens alignment elasticity, please refer to Q1 of the General response.

  • Q1-2/W1-2: The effective volume niHiG(μi)n_iH_iG(\mu_i) is introduced as a factorization assumption rather than derived for the training procedure.

Admittedly, we have not yet derived the most appropriate explicit expression. Our core finding is that both the intrinsic dataset dimension and the model-perceived dimension, like the size of the alignment dataset, make positive contributions to the effective volume V, which in turn affects the elasticity coefficient κ\kappa. In the revised manuscript, we will replace the simple product form of the effective volume with a general function f(ni,Hi,G(μi))f(n_i,H_i,G(\mu_i)), where f is monotonically increasing with respect to each of its arguments nin_i, HiH_i, and G(μi)G(\mu_i), without specifying its exact functional form.

  • Q2: Clarification: Do the authors have existing multi-seed or controlled results that isolate dataset dimension from corpus composition and optimization?

To isolate the dataset dimension effect from that of the corpus, datasets with different dimensions in the original manuscript are all sampled from the same dataset. Besides corpus, we further isolate dimension from alignments. We add supplementary experiments using a single alignment method LoRA with varying ranks to control for differences in model-perceived dimension. The values in the table show the relative drop in model alignment performance over the fine-tuning process and smaller values correspond to weaker alignment rebound. The ID denotes the model’s perceived dimensionality. The experimental results show that groups with lower dimension, exhibit weaker alignment elasticity.

Table 2

Fine-tuning processrank=4(ID=18.29)rank=8(ID=16.77)rank=16(ID=14.66)rank=32(ID=12.84)
0$\rightarrow$48059.8%56.3%51.6%46.8%
480$\rightarrow$160069.7%68.3%66.2%61.7%

We hope this clarifies our contribution and would appreciate your reconsideration of the score.

We kindly remind you to read our rebuttal. We sincerely apologize for a typo in Line 303: it is an easily fixable typo by replacing decreases with increase, and all the other parts are based on the correct statement. So we hope you read our rebuttal and reconsider your score. Thank you very much!

We kindly remind you that the rebuttal period is drawing to a close. We sincerely hope to clarify the misunderstandings in our paper and address your concerns, and we truly appreciate your taking the time to read our response.

Reviewer_XWjkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates alignment rebound in large language models during subsequent fine-tuning and proposes Alleviation Elasticity Theory, an extension of existing compression-based elasticity theory. The proposed framework incorporates dataset intrinsic dimension and model-perceived dimension into the characterization of alignment elasticity, establishing a theoretical connection between data dimensionality, information entropy, and alignment stability. Based on this framework, the authors suggest that reducing alignment dataset dimension or internal representation dimension can improve resistance to alignment degradation. Experiments using DistilGPT2 on IMDb sentiment data examine the effects of dataset dimension and different alignment methods (full-parameter SFT, LoRA, and BitFit) on alignment retention. The reported results suggest that lower-dimensional alignment data may exhibit greater robustness to subsequent fine-tuning, providing preliminary empirical support for the proposed perspective.

Critical strengths
  1. The paper addresses alignment rebound during subsequent fine-tuning, an important challenge for maintaining the stability of aligned LLMs. Extending existing elasticity theory beyond dataset size to account for data characteristics is a meaningful research direction.

  2. The paper introduces dataset intrinsic dimension and model-perceived dimension into the analysis of alignment elasticity, attempting to connect geometric properties of data representations with information-theoretic compression. This offers a potentially useful perspective for understanding alignment stability.

  3. The experiments examine the relationships between dataset dimension, initial alignment performance, alignment retention, and different alignment methods. In particular, comparing equal-sized datasets with different estimated intrinsic dimensions provides preliminary evidence that dataset structure may influence alignment robustness.

Critical weaknesses
  1. The proposed effective data volume relies on a factorization assumption involving entropy and model-perceived dimension, while the function G(μ) lacks sufficient theoretical justification or empirical calibration. More importantly, Eq. (3) appears inconsistent with the conclusion in Section 3.4: reducing alignment-data entropy or its representation factor would increase κ, rather than decrease it, when other factors remain fixed. Furthermore, the asymptotic relationship between intrinsic dimension and entropy does not directly guarantee the claimed ordering across finite-resolution datasets. These issues weaken the theoretical foundation of the central claim.

  2. The experiments primarily measure changes in the proportion of positive-sentiment outputs following negative-sentiment fine-tuning. However, declining alignment scores do not necessarily indicate that the model is reverting toward its pretraining distribution; they may instead reflect ordinary task interference or adaptation to the new training objective. Without directly measuring distributional drift or compression behavior, the experiments do not fully establish that the observed phenomenon corresponds to the alignment elasticity characterized by the theory.

  3. Although the experiments suggest an association between intrinsic dimension and alignment retention, they do not isolate dimension as the determining factor. Datasets with different dimensions may also differ in semantic diversity, complexity, or training difficulty. Similarly, different alignment methods introduce changes beyond model-perceived dimension, making their effects difficult to disentangle. Consequently, the experiments provide limited causal evidence for the proposed mechanism.

  4. All experiments are conducted using DistilGPT2 on an IMDb sentiment task, which differs substantially from realistic instruction or safety alignment settings. The experiments also lack systematic evaluation across random seeds, model scales, and tasks. The initial dimension-performance correlation is not statistically significant under the reported across-group tests, and the method-comparison results are not consistently aligned with the theoretical hypothesis. These limitations weaken the generality of the conclusions.

Questions for authors
  1. Could the authors clarify how reducing alignment dataset dimension or model-perceived dimension leads to weaker elasticity under Eq. (3)? In particular, how can the conclusion in Section 3.4 be reconciled with the definition of κ, and what theoretical or empirical evidence supports the chosen form of G(μ)?

  2. Can the authors directly measure whether subsequent fine-tuning moves the model distribution toward its pretraining distribution, rather than merely reducing the alignment score? It would be valuable to compare the observed distributional changes with the predictions of the proposed elasticity theory.

5 comments
AuthorsOpenReview ↗

Response to Reviewer XWjk

  • W1: The asymptotic relationship between intrinsic dimension and entropy does not directly guarantee the claimed ordering across finite-resolution datasets.

We admit that the leaf-node entropy–intrinsic dimension relation in our theory is asymptotic and holds as resolution approaches zero, which real datasets cannot strictly satisfy. Residual terms do affect this relation, yet they are subdominant to the leading-order term. Thus, the asymptotic condition violation does not reverse the monotonic trend, and our core conclusions remain valid.

  • W3-1: Although the experiments suggest an association between intrinsic dimension and alignment retention, they do not isolate dimension as the determining factor. Datasets with different dimensions may also differ in semantic diversity, complexity, or training difficulty.

To isolate the dataset dimension effect from that of the corpus, datasets with different dimensions in the original manuscript are all sampled from the same dataset. Our goal is to show that dataset dimensionality affects alignment elasticity and explore ways to alleviate it.

  • W3-2: Similarly, different alignment methods introduce changes beyond model-perceived dimension, making their effects difficult to disentangle.

We add supplementary experiments using a single alignment method LoRA with varying ranks to control for differences in model-perceived dimension. Table values report relative alignment performance drops during fine-tuning: smaller values mean weaker rebound. ID denotes model-perceived dimension. Results show lower-ID groups exhibit weaker alignment elasticity.

Table 1

Fine-tuning processrank=4(ID=18.29)rank=8(ID=16.77)rank=16(ID=14.66)rank=32(ID=12.84)
0$\rightarrow$48059.8%56.3%51.6%46.8%
480$\rightarrow$160069.7%68.3%66.2%61.7%
  • W4-1: All experiments are conducted using DistilGPT2 on an IMDb sentiment task, which differs substantially from realistic instruction or safety alignment settings. The experiments also lack systematic evaluation across random seeds, model scales, and tasks.

We add experiments across diverse model scales, tasks and random seeds. The results confirm that lower dimensionality alleviates alignment elasticity. Please refer to Q2 of the General response for further details.

  • W4-2: The initial dimension-performance correlation is not statistically significant under the reported across-group tests.

These two values indicate that there is no statistically significant correlation between dataset dimensionality and initial alignment performance, which is exactly consistent with our experimental expectation. The main claim of this work is that lower dimensionality alleviates alignment elasticity. If lower dimensionality led to worse initial alignment performance, mitigating alignment elasticity would be meaningless. This experiment demonstrates that mitigating alignment elasticity via lower dimensionality is meaningful.

  • Q1-1: Could the authors clarify how reducing alignment dataset dimension or model-perceived dimension leads to weaker elasticity under Eq.(3)? In particular, how can the conclusion in Section 3.4 be reconciled with the definition of κ\kappa.

We sincerely apologize for the confusion this caused. This is a typographical error in our writing. For detailed explanation and why dimension reduction weakens alignment elasticity, please refer to Q1 of the General response.

  • Q1-2: What theoretical or empirical evidence supports the chosen form of G(μ)G(\mu)?

Admittedly, no direct explicit expression can be derived for G(μ)G(\mu) at present. However, it can be confirmed that G(μ)G(\mu) is positively correlated with μ\mu (where μ\mu denotes the model-perceived dimension). In other words, G(μ)G(\mu) is a monotonically increasing function of μ\mu, and we do not specify its exact functional form.

  • Q2/W2: Can the authors directly measure whether subsequent fine-tuning moves the model distribution toward its pretraining distribution, rather than merely reducing the alignment score? It would be valuable to compare the observed distributional changes with the predictions of the proposed elasticity theory.

We measure KL divergence between the current and pretrained models during negative sentiment fine-tuning. Its decreasing trend indicates a tendency to return toward the pretraining distribution.

Table 2

Number of fine-tuning samplesseed17seed42direct fine-tuning without prior alignment
00.3430.3490.000
1280.3360.3400.075
5120.3290.3310.134
10240.3180.3220.176
15360.3100.3140.194
20000.3050.3070.211

We hope this clarifies our contribution and would appreciate your reconsideration of the score.

We kindly remind you to read our rebuttal. We sincerely apologize for a typo in Line 303: it is an easily fixable typo by replacing decreases with increase, and all the other parts are based on the correct statement. So we hope you read our rebuttal and reconsider your score. Thank you very much!

We kindly remind you that the rebuttal period is drawing to a close. We sincerely hope to clarify the misunderstandings in our paper and address your concerns, and we truly appreciate your taking the time to read our response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their valuable comments. We list the concerns raised by the three reviewers as follows: Q1. Clarification of a critical typographical error; Q2. Demonstration of the consistent performance of our theory across more models and datasets.

  • Q1a.why lower dimension should reduce κ\kappa.

We sincerely apologize for the confusion this caused. The statement in Section 3.4 (lines303) claiming that lower-dimensional alignment datasets reduce κ\kappa is incorrect. Lower-dimensional alignment datasets should instead increase κ\kappa. This is a typographical error in our writing, which can be straightforwardly corrected by revising “decrease of κ\kappa” to “increase of κ\kappa”. The preceding context and all subsequent derivations (lines303-lines307) are consistent and correct, as they are in fact developed under the assumption that κ\kappa increases.

  • Q1b.How can the low-dimension mitigation claim be derived.
\frac{d\gamma_{p_\theta}^{D_a/D}}{d\lambda} &= \Theta\\left( -\kappa\frac{d\gamma_{p_\theta}^{D_p/D}}{d\lambda} \right),\\ \frac{d\gamma_{p_\theta}^{D_p/D}}{d\lambda}< 0,\quad \frac{d\gamma_{p_\theta}^{D_a/D}}{d\lambda}>0,\\ \lambda= \frac{|D_t|H_tG(\mu_t)}{|D_a|H_aG(\mu_a)} \ll 1,\\ \kappa= \frac{|D_p|H_pG(\mu_p)}{|D_a|H_aG(\mu_a)} \gg 1. \end{aligned}$$ This is the central equation of our paper. When the intrinsic dimension of the alignment dataset decreases, the entropy of the alignment data $H_a$ decreases and then $\kappa$ increases. The left-hand side of the equation remains approximately unchanged. The value of $\frac{d\gamma_{p_\theta}^{D_p/D}}{d\lambda}$ on the right-hand side of the equation increases, indicating a weaker tendency to return toward the pretraining distribution. This corresponds to weaker elasticity . Similarly, a decrease in the model-perceived dimension after alignment increases $\kappa$, thereby weakening alignment rebound. A more detailed explanation is provided in Appendix A.1. **Q2**. To more fully verify our core claim that reducing dimensionality alleviates alignment rebound, we add experiments on the **new GPT-2 and SmolLM2-135M-Instruct models, a new JSON output preference retention task, a new safety alignment task, and multi-seed repeated experiments.** The ID in the table denotes the dataset dimensionality and the model’s perceived dimensionality. The values in the table are relative measures of model alignment performance degradation over the fine-tuning process: **the lower the value is, the weaker the elasticity it has, and the better the alignment effect is preserved.** The ± symbol is the standard deviation across different random seeds. The experimental results show that our claim holds under all settings. - **Q2a.Cross-scale experiments**. We add experiments on the larger GPT-2 model. **Table A** | Fine-tuning process | Low-ID | Medium-ID | High-ID | |:---:|:---:|:---:|:---:| | 0\$\rightarrow\$512 | 27.3±0.9% | 28.5±1.1% | 33.4±1.7% | | 512\$\rightarrow\$2000 | 43.4±1.3% | 46.6±0.6% | 47.3±0.8% | - **Q2b.New alignment task**. We add a JSON output preference retention task constructed from the AG News classification dataset. **Table B1** | Fine-tuning process | Low-ID | Medium-ID | High-ID | |:---:|:---:|:---:|:---:| | 0\$\rightarrow\$512 | 15.4±1.6% | 17.5±0.5% | 18.4±1.8% | | 512\$\rightarrow\$2000 | 66.8±0.8% | 70.6±2.1% | 73.3±1.1% | **Table B2** | Fine-tuning process | SFT(ID=14.98±0.18)| LoRA(ID=16.10±0.32) | BitFit(ID=16.87±1.00) | |:---:|:---:|:---:|:---:| | 0\$\rightarrow\$512 | 21.4±0.9% | 24.7±1.4% | 26.2±1.2% | | 512\$\rightarrow\$2000 | 71.5±1.5% | 75.6±0.8% | 77.3±0.9% | - **Q2c.Cross-model-scale experiments on safety alignment task**. We conduct safety alignment experiments on a new model, SmolLM2-135M-Instruct, using the PKU-SafeRLHF dataset. **Table C** | Fine-tuning process | Low-ID | Medium-ID| High-ID | |:---:|:---:|:---:|:---:| | 0\$\rightarrow\$512 | 59.2±0.7% | 61.8±2.2% | 65.8±1.4% | | 512\$\rightarrow\$2000 | 86.2±1.4% | 89.2±1.6% | 90.2±1.2% | - **Q2d.Multi-seed repeated experiments**. We replicate the original experiments(table2) on the DistilGPT2 model with the IMDb sentiment task, and perform repeated trials using cross combinations of random data sampling seeds and random training seeds. The three groups in the table correspond to different random seed combinations. **Table D** | Fine-tuning process | Low-ID 1 | Medium-ID 1 | High-ID 1 | Low-ID 2| Medium-ID 2 | High-ID 2 | Low-ID 3| Medium-ID 3 | High-ID 3 | |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | 0\$\rightarrow\$512 | 21.4% |25.4% | 29.4% | 22.4% |22.8% | 28.7% | 22.9% | 24.9% | 30.3% | | 512\$\rightarrow\$2000 | 38.7% |40.8% |44.6% | 40.2% | 41.8% | 44.1% | 38.5% | 40.7% | 47.8% |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.