- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether fine-tuned language models can report unintended safety-related behavioral changes relative to their base models. The authors introduce a practical introspection setting in which the behavior being explained is not deliberately implanted during fine-tuning but instead emerges as a side effect. The paper further proposes the Delta-Aware Introspection Adapter (DAIA), which separately processes base-model activations and fine-tuning-induced activation differences to generate self-reports. Experiments on safety-shift classification and free-form reporting show that introspection learning generalizes to unseen models and safety categories, while DAIA generally outperforms a standard LoRA-based introspection adapter.
Critical strengths
Compared with the existing work, the questions raised by the author are closer to the real model deployment scenarios and have research significance. The design concept of DAIA is straightforward, the motivation is clear, and the mechanism analysis clearly demonstrates how DAIA operates. The experimental setup is comprehensive, demonstrating that DAIA has good generalization ability. The ablation experiments are relatively sufficient, verifying the necessity of the two branches. The mechanism analysis details how DAIA forms the judgment of safe changes.
Critical weaknesses
- The presentation of the proposed method should be improved. Although DAIA is mathematically defined, the paper lacks an overall framework diagram or pseudocode illustrating the complete workflow, including dataset construction, adapter training, and inference.
- The paper motivates introspection as a potentially inexpensive first-pass safety diagnostic, but it does not provide an end-to-end cost analysis of DAIA. Constructing the training data requires evaluating 213 models on 1,523 safety prompts, performing LLM-based scoring and reference-report generation, and subsequently conducting SFT and DPO. The authors should compare both the one-time training cost and the per-model inference cost of DAIA with those of direct safety evaluation, and clarify under what deployment scale DAIA becomes cost-effective.
- The performance improvement of DAIA is not stable. Some indicators are lower than the baselines, especially in the OOD experiments, and the superiority of the algorithm is not demonstrated.
- The baseline comparison is insufficient. Although the authors cite two recent introspection methods in the Introduction, they do not clearly identify which method the “LoRA” baseline corresponds to or provide separate comparisons with these methods.
- The same LLM is responsible for generating the reference report and evaluating the generated report, which makes the credibility of the report lower. The author did not prove that the internal audit report output by DAIA truly reflects the safety changes, or if it merely outputs text that is more in line with the LLM's preference.
Questions for authors
- Which prior method does the LoRA-based IA baseline correspond to? The authors cite the introspection methods of Shenoy et al. and Goel et al., but it is unclear whether the reported baseline faithfully reproduces either method or combines elements from both. The authors should clarify its provenance and explain why the two methods are not evaluated separately.
- The authors should provide a detailed analysis of the cost of constructing the dataset and training DAIA. Collecting 213 models, evaluating them on 1,523 safety prompts, and using LLM-based scoring and reference-report generation appear computationally expensive.
- How reliable are the reference reports and their associated evaluation scores? Were the ten LLM-generated reference reports manually validated? Did the authors assess agreement across different judge models? Moreover, can DAIA’s self-reports be corroborated using independent behavioral evidence to ensure that they reflect genuine safety changes rather than the preferences of a particular judge model?
5 comments
Response to reviewer 5T1c
W5&Q3 (LLM judge reliability)
We agree that the reviewer has raised an important concern. To address this issue, we re-evaluated the free-form reporting task on Qwen3-14B using Gemini 3.8 Flash and Claude Haiku 5.5, following the same evaluation protocol as in the original experiments. The results are presented in Table 11 and Figure 11 of Appendix D.5. For convenience, we reproduce Table 11 below.
・OOD-A
| Method | GPT | Gemini | Claude |
|---|---|---|---|
| Raw | 1.73 ± 0.02 | 1.26 ± 0.01 | 1.43 ± 0.02 |
| LoRA | 3.41 ± 0.12 | 3.56 ± 0.09 | 3.01 ± 0.08 |
| DAIA | 3.57 ± 0.10 | 3.70 ± 0.09 | 3.15 ± 0.07 |
・OOD-B
| Method | GPT | Gemini | Claude |
|---|---|---|---|
| Raw | 1.53 ± 0.02 | 1.26 ± 0.00 | 1.38 ± 0.01 |
| LoRA | 2.56 ± 0.08 | 2.24 ± 0.05 | 2.12 ± 0.04 |
| DAIA | 2.74 ± 0.01 | 2.47 ± 0.02 | 2.22 ± 0.02 |
・OOD-C
| Method | GPT | Gemini | Claude |
|---|---|---|---|
| Raw | 1.41 ± 0.04 | 1.19 ± 0.00 | 1.26 ± 0.02 |
| LoRA | 2.49 ± 0.10 | 2.18 ± 0.08 | 2.06 ± 0.05 |
| DAIA | 2.75 ± 0.01 | 2.47 ± 0.08 | 2.21 ± 0.06 |
The results show that the ranking Raw < LoRA < DAIA is preserved across all OOD settings and all three LLM judges, consistently supporting our main findings. Furthermore, Figure 11 presents the pairwise correlations of evaluation scores among the three LLM judges. Both Pearson and Spearman correlation coefficients are consistently high, ranging approximately from 0.8 to 0.9 across all settings.
These results suggest that our main findings are robust to the choice of LLM judge and are not solely attributable to the preferences of the model used to generate the reference reports. In particular, the improvements of introspection-trained IAs over raw fine-tuned models, as well as the performance advantage of DAIA over the conventional LoRA-based IA, are consistently observed across different LLM judges.
Response to reviewer 5T1c
We hope that the above explanations address your concerns. If any concerns remain, please let us know.
Otherwise, we would greatly appreciate it if you could reconsider your score in light of our responses and revisions. Thank you for your time and consideration.
Response to reviewer 5T1c
We thank reviewer 5T1c for taking the time to provide insightful feedback. Below are our responses to each of the concerns.
W1 (The presentation of the proposed method)
We agree that the original manuscript lacked a visual overview of the proposed framework. To address this, we have added Figure 1 to the revised manuscript, illustrating the complete workflow from collecting fine-tuned models and constructing supervision data to training DAIA and performing inference.
W2&Q2 (Cost analysis)
We conducted an end-to-end cost analysis for the Qwen3-14B free-form reporting task, including dataset construction, reference report generation, SFT, and DPO.
All computational costs were estimated using an NVIDIA A100-SXM4 (80 GB) GPU at RunPod's rate of $1.79/hour.
DAIA reduces per-model evaluation costs by approximately 94.2%, requiring only 5.8% of the cost of direct benchmark evaluation at inference time.
To account for the one-time cost, we derive the break-even point:
N∗=K(b−d)F
where:
Assuming a fixed one-time cost and constant per-category costs, the break-even point decreases inversely with the number of evaluated categories.
For example, with 30 safety categories:
N∗=30(0.0642−0.0037)197≈109
Thus, DAIA becomes more cost-effective after evaluating approximately 109 additional fine-tuned models under these assumptions.
Importantly, large-scale collections of over 1,000 fine-tuned models have been observed on model hubs such as Hugging Face and in real-world production environments[1][2].
These findings suggest that DAIA could offer substantial cost savings in large-scale, repeated safety auditing by amortizing its one-time training cost across many fine-tuned models.
We have added this analysis to Appendix D.6 of the revised manuscript.
[1] Liu, Hongyi, et al. "LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystem.", EMNLP (Findings), 2025.
[2] Jaiswal, Shashwat, et al. "Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems.", arXiv, 2025.
W3 (The performance improvement of DAIA)
We acknowledge that DAIA does not consistently outperform the LoRA-based IA across all classification metrics. However, DAIA achieves higher accuracy in 11 out of 12 cases, although improvements in macro-F1 are less consistent. In particular, DAIA achieves a higher average score in OOD-C, where both fine-tuned models and safety categories are unseen during training. More importantly, DAIA consistently achieves higher mean scores than the LoRA-based IA in the free-form reporting task across all cases.
One possible explanation for this difference is the granularity of the output task. Classification requires predicting only a coarse safety-shift label, potentially limiting the extent to which detailed information about fine-tuning-induced differences can be reflected in the output. In contrast, free-form reporting requires generating detailed behavioral descriptions over multiple tokens, allowing DAIA's explicit processing of fine-tuning-induced activation differences to influence a broader range of generated content.
W4&Q1 (The baseline comparison)
Both prior methods correspond to the "LoRA" baseline in our setting. Both studies adopt a standard LoRA architecture for their introspection adapters, and we therefore denote the baseline representing both methods as "LoRA". To clarify this point, we have added an explanation to the Baselines paragraph in Section 4.1 of the revised manuscript.