acceptodds
Under review as a conference paper at ICLR 2027

Moonshine: A Protocol for Distillation in Unsafe Model Ecosystems

Abstract

Organizations without access to U.S. frontier intelligence increasingly rely on open weight models for agentic cyber-defense. Because the provenance of open weight models cannot currently be verified, these organizations are exposed to the sleeper agent supply chain risk; their cyber-defense model may appear safe, but actually contain a backdoor that can trigger unsafe behavior after deployment. To address this, we introduce Moonshine, a protocol for distillation in unsafe model ecosystems, where neither the teacher model nor the actor performing the distillation can be trusted. Moonshine combines an attested training pipeline, which reduces trust in the distillation actor, with hard sequence and on policy distillation, which both restrict the logit channel used to smuggle backdoors through distillation. By implementing Moonshine, a distillation actor can attest to training a cyber-defense model using distillation methods that are empirically shown to remove backdoors, thereby mitigating the sleeper agent supply chain risk. We provide a proof sketch of protocol soundness and empirical results demonstrating that hard sequence and on policy distillaton induce capability transfer without causing backdoor transfer across a wide variety of model and trigger types, including triggers explicitly designed to persist through distillation. We conclude by arguing that adoption of Moonshine would differentially accelerate agentic-cyber defense without accelerating agentic cyber-offense.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
distillation, cybersecurity, protocol

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N2sSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the sleeper agent supply chain risk faced by organizations that lack access to U.S. frontier models and must rely on open-weight models, whose provenance cannot be verified, for agentic cyber-defense. The authors propose Moonshine, a protocol for distillation in which neither the teacher model nor the distillation actor needs to be trusted. Moonshine decomposes distillation into four stages, runs teacher inference and student training inside Trusted Execution Environments (TEEs), and links the stages through hashes, signatures and an append-only log, so that downstream users can verify the student’s lineage without re-running training. It further restricts distillation to hard sequence distillation or on policy distillation and keeps teacher logits inside the TEE, which closes the logit channel used to smuggle backdoors.

Critical strengths
  1. As open-weight models are increasingly adopted in security-critical settings, obtaining a verifiable, backdoor-free student from an untrusted teacher is an important and timely problem.

  2. The paper is clearly written; the threat model, the protocol figure and the end-to-end example in the appendix make the protocol easy to follow.

Critical weaknesses
  1. The protocol requires a randomly initialized student or one from a different model family, but the former amounts to pretraining from scratch, and by the paper’s own account the provenance of the latter cannot be verified either.

  2. The paper assumes that TEE attestation and isolation cannot be broken and does not consider adaptive attacks against the TEE itself, such as side-channel attacks.

  3. The protocol relies on public scrutiny of the transfer corpus, but the paper provides no auditing procedure and does not evaluate whether poisoned samples can be detected.

Questions for authors

Please see the Weaknesses section.

2 comments

The author did not answer my question, so I have chosen to keep my rating as it is; I hope the author will take my question seriously.

AuthorsOpenReview ↗

author response

we thank the reviewer for their careful review of our manuscript!

we believe that we addressed W1 and W3 in you review in previous responses (in particular, we address the scope of trust in the student model in the W3 section of our response to vKru, and the question of audibility of the transfer corpus in the W1.2 section of our response to vKru)

further, we believe that we designated attacks on the TEE itself as out of scope in L133-138 in the paper, but we can improve the clarity here in the final manuscript.

Reviewer_rELHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Moonshine, a protocol for provenance-attested distillation from potentially untrusted teachers and distillation operators. It combines TEE-based training verification with hard-sequence and on-policy distillation (OPD) to mitigate backdoor transfer while preserving useful capabilities. Experiments across multiple models and backdoor types show capability improvements with low attack success rates under the evaluated settings.

Critical strengths
  1. Important problem and practical motivation. The paper addresses an important model supply-chain security problem by considering capability transfer from untrusted teachers under potentially untrusted training operators. Combining knowledge distillation with provenance attestation is a meaningful and practically relevant approach.

  2. Encouraging evidence. The evaluation includes T-MTB and ATBA, which are specifically designed to survive knowledge distillation. In Table 2, hard-sequence distillation and OPD achieve substantially lower ASRs than Logit-KD while maintaining comparable helpfulness.

  3. Diverse experimental evaluation. The paper evaluates multiple model families, tasks, and backdoor types, including reasoning and code generation. Figures 2 and 3 further investigate the effects of training duration and poisoned training samples, providing useful insights.

Critical weaknesses
  1. The security mechanism of OPD is not sufficiently established. Unlike hard-sequence distillation, OPD still uses teacher-provided token-level probability information, leaving a potential channel for backdoor transfer. [1] show that OPD can propagate backdoors and that the outcome depends on training duration, KL estimation, and reward clipping. The authors should clarify why OPD is considered to close the logit-smuggling channel and whether Moonshine's low ASR depends on its hard-sequence warm start and clipped advantages.

  2. Table 3 lacks sufficient evidence for an OPD-specific safety advantage. Unlike Table 2, Table 3 reports no Logit-KD or hard-sequence baselines under identical conditions, making it difficult to determine whether the evaluated backdoor could transfer through alternative objectives. Although the authors report frequent occurrence of individual T-MTB trigger tokens, [2] show that such tokens can transmit backdoor signals through logit-based distillation even without frequent full-trigger activation. A matched comparison and an analysis of teacher probability signals during OPD would help establish whether OPD suppresses these signals or whether the backdoor simply fails to transfer in this setting.

[1]. Does On-Policy Distillation for Safety Pose Backdoor Risks?

[2]. Pay Attention to the Triggers: Constructing Backdoors That Survive Distillation.

Questions for authors
  1. OPD security mechanism. Why is OPD considered to close the logit-smuggling channel when teacher-provided token probabilities still influence student updates? How do the authors reconcile their findings with [1]?

  2. Missing baselines in Table 3. Could the authors provide Logit-KD and hard-sequence distillation results under the same teacher, student, and transfer corpus to establish whether OPD provides a distinct safety advantage?

  3. Statistical reliability in Table 2. ATBA achieves 14.2% ASR under Logit-KD, compared with 8.8% for the base student. Could the authors report results across multiple random seeds and clarify whether ATBA was sufficiently optimized for this setting?

  4. OPD training configuration. To what extent do the hard-sequence warm start, advantage clipping, and training duration contribute to the low ASR observed under OPD?

3 comments
AuthorsOpenReview ↗

author response

We thank the reviewer for their careful review of our manuscript! We address each of the concerns outlined in the review’s weaknesses and questions section below:

W1.1 We thank the reviewer for bringing [1] to our attention. We believe that the results of [1] actually support our conclusions, and further, demonstrate Moonshine’s practicality in the face of new information.

As shown in Figure 8 of [1] (https://arxiv.org/pdf/2610.07654), backdoor transfer for OPD using the sampled token requires a very large number of epochs (5+) to be effective, Further, reward clipping (which [1] calls lazy defense) further delays backdoor transfer through OPD. Within our paper, we find no backdoor transfer when distilling via a single Epoch of sampled token OPD with reward clipping, which is consistent with the results in [1]. One concrete improvement to the paper here is to make it more clear that we are using sampled token OPD with reward clipping in the experiments section of the paper.

Further, the moonshine attestation protocol enables us to cryptographically attest to our OPD models being trained with sampled token OPD and clipped rewards, which allows us to retrospectively evaluate the safety of the distilled models in light of new information. Overall, we believe the value of Moonshine rests in enabling practitioners to evaluate the risk level of distilled models based on attested distillation procedures, particularly as new attack types and information emerges.

W1.2 We thank the reviewer for pointing out that vanilla OPD does utilize a diverse logit channel during KL divergence computation. We want to clarify that we use sampled OPD with clipped rewards, which only uses the scores from the tokens in the decoded student sequence, and does not consider the teacher’s tail logits. We also do not claim that this fully closes the logit channel in OPD, but only that it restricts it (see e.g. L19, L48, L245). We further will clarify our use of sampled OPD and the relevance of this distinction to capacity of the logit channel in the paper.

W2. We agree with the reviewer that having additional baselines to contextualize the results in Table 3 would strengthen the paper. We have run these baselines and report the results in the table below, which we will include in the final manuscript. Note that all distillation configurations lead to improved GSM8K, but only Logit-KD achieves attack success.

ConfigurationTeacher GSM8KTeacher ASRBase GSM8KBase ASRHard-seq GSM8KHard-seq ASRLogit-KD GSM8KLogit-KD ASROPD GSM8KOPD ASR
BadNet88.5100.064.00.090.3 ±\pm 0.60.0 ±\pm 0.090.2 ±\pm 0.60.0 ±\pm 0.088.0 ±\pm 0.50.0 ±\pm 0.0
T-MTB90.568.764.00.088.2 ±\pm 0.80.0 ±\pm 0.087.8 ±\pm 1.950.9 ±\pm 2.388.3 ±\pm 2.30.0 ±\pm 0.0
clean teacher88.50.064.00.088.8 ±\pm 0.80.0 ±\pm 0.089.2 ±\pm 0.80.0 ±\pm 0.092.0 ±\pm 2.00.0 ±\pm 0.0

Q1 Addressed in W1.1

Q2 Addressed in W2

Q3 We agree with the reviewer that additional random seeds would help to further contextualize the results in Table 2. As a result, we reran Table 2 under three independent random seeds. We report the mean and sample standard deviation over these runs in the refined Table 2 below, which we intend to include in the manuscript. Note that, in all runs, student models achieve helpfulness uplift, but only Logit-KD propagates the backdoor

T-MTB

helpfulnessASR (%)
teacher7.3792.1
base student (untuned)5.1413.5
hard-sequence6.98 ±\pm 0.128.99 ±\pm 2.42
logit-KD (T=3)6.79 ±\pm 0.0642.86 ±\pm 3.64
OPD6.92 ±\pm 0.245.82 ±\pm 0.92
OPD (step 6000)6.95 ±\pm 0.115.03 ±\pm 3.21

ATBA

helpfulnessASR (%)
teacher7.3791.2
base student (untuned)5.008.8
hard-sequence7.21 ±\pm 0.154.28 ±\pm 2.73
logit-KD (T=3)7.00 ±\pm 0.0915.54 ±\pm 2.44
OPD7.08 ±\pm 0.261.13 ±\pm 1.03

Q4 We agree with the reviewer that further ablations of various aspects of the OPD procedure are of research interest. We found during our experiments that the removal of the Hard-Seq warmup period caused OPD to reliably diverge. Further, the results in [1] find that increasing the number of epochs of training and removing reward clipping both contribute to improved backdoor transfer in OPD. We leave the further replication of these results to future work, but note that our results of sampled OPD training for 1 epoch with clipped rewards not propagating triggers are concordant with the findings in [1].

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts, which address several of my earlier concerns. However, I still have some reservations regarding the security claims and practical motivation.

  1. Configuration-dependent security of OPD. The authors clarify that Moonshine uses sampled-token OPD with clipped rewards and a short training duration. While this explains the consistency with [1], it also suggests that the observed robustness may depend heavily on these specific configurations. As [1] demonstrates, backdoors can still propagate through OPD under longer training or different KL estimation and clipping settings. Therefore, the current results do not establish that OPD reliably prevents backdoor transfer, particularly against adaptive attackers.

  2. Unclear OPD-specific safety advantage. The additional results in Table 3 convincingly demonstrate that T-MTB transfers through Logit-KD but not through Hard-sequence distillation or OPD. However, both Hard-seq and OPD achieve 0% ASR, leaving it unclear whether OPD provides any distinct security benefit beyond Hard-seq.

  3. Practical relevance of the threat model. While model supply-chain attacks are a legitimate concern, the specific scenario considered by Moonshine appears somewhat contrived. The protocol assumes potentially malicious teachers and untrusted distillation operators, while relying on an attested training procedure and empirically safer distillation objectives. It is not sufficiently clear how frequently this combination of assumptions arises in realistic deployment settings, or why downstream users would prefer this workflow over alternative approaches such as using vetted models or performing distillation under their own control.

More importantly, the protocol establishes verifiable training provenance rather than guaranteeing that the resulting model is free of backdoors. The authors should more clearly distinguish these two properties and explain how downstream users should assess the residual risks of a model trained using an attested but potentially vulnerable distillation procedure. Overall, the rebuttal has strengthened the empirical evaluation, but I remain concerned that the security conclusions may be broader than the evidence supports and that the practical necessity of the proposed threat model has not been sufficiently established.

AuthorsOpenReview ↗

author response

thank you for your continued evaluation of our work. we address each of your comments here below:

W1. we agree with the reviewer, and will modify the manuscript to clarify that our experiments use a variant of OPD that is known to resist backdoor transfer, citing [1]. this should reduce the scope of the claims sufficiently.

W2. we would like to remind the reviewer that we are not necessarily endorsing OPD as the best method for transfer here, but simply providing a range of distillation procedures that empirically do not induce backdoor transfer. we suspect that the downstream practitioner will want to choose from a variety of options to fit their exact distillation scenario.

W3. while we agree with the reviewer that the manuscript could contain more motivating material for when moonshine applies, we do not agree with the reviewer that the exact security scenario we assume is contrived. in one particular example, thinking machines inkling (a base model they pretrained themselves) is known to have been posttrained using distillation from kimi (https://techcrunch.com/2026/07/15/thinking-machines-amps-up-its-bet-against-one-size-fits-all-ai-with-its-first-open-model-inkling/). a situation where thinking machines would want to attest to this post training procedure in order to assuage backdoor concerns from customers who are worried about supply chain propagation perfectly fits the moonshine threat model.

moreover, we believe that we have established the difference between security and provenance in the existing manuscript (see section W1.1 in our response to vKru), but due to the confusion across reviewers, we agree that the manuscript would benefit from an expanded discussion of this distinction.

finally, we thank the reviewer for acknowledging that our response addressed several of your earlier concerns, and humbly request the reviewer to modify their score to reflect this fact, particularly in light of our continued discourse on the paper.

Reviewer_vKruOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposed a training pipeline on-policy distillation to transfer capabilities from untrusted teachers while reducing transfer of tested backdoors.

Critical strengths

1)The experimental results demonstrated the feasibility beyond the authors' assumption.

2)The motivation of the method is to show that hard-sequence on-policy distillation can preserve utility and ASR gains while substantially reducing transfer of the backdoor issues.

Critical weaknesses

1)The protocol verifies how a student was trained, but it does not guarantee that there are no backdoors in the model. Figure 3 demonstrates that poisoned examples can be transferred to the student's model and create a backdoor, while the effectiveness of public auditing is not evaluated. 2)The experiments did not evaluate a malicious teacher and distillation actor jointly optimizing against the approved pipeline, particularly through corpus selection or teacher-derived advantages in on-policy distillation. 3)Ablations for clean-teacher, trusted-target, and filtered-versus-unfiltered comparisons are essential to explain the effectiveness of the distillation procedure from filtering and ordinary refusal learning.
5)The claim for cyber defense is not supported enough: The results on Helpfulness, GSM8K, and LiveCodeBench benchmarks do not show how to defense agentic cyber .

Questions for authors
  1. Pls provide ablation results of auditors to find malicious examples in the public corpus?
  2. Will an adaptive malicious teacher be able to pass on backdoors by either choosing the corpus or using on-policy scoring regardless of the protocol?
  3. What are the score of your ablation study for the impact of distillation, harmful-response filtering and typical refusal learning?
  4. What is the evidence to prove your work is beneficial for cyber defense instead of similar benefits for cyber offense?
2 comments
AuthorsOpenReview ↗

author response

We thank the reviewer for their careful review of our manuscript! We address each of the concerns outlined in the review’s weaknesses and question section below:

W1.1 We agree with the reviewer that the gap between provenance and safety is an important distinction, which is why we explicitly mention that our protocol only guarantees provenance and not safety several times, including in L12, L93, and in L116. We maintain that provenance is one piece of a much larger holistic model safety posture, and that our method makes important contributions to model safety through its contributions to model provenance.

W1.2 We agree with the reviewer that the effect of errors in the public audit is an important variable to study, and to this effect, we include Figure 3 to explore the effect of poisoned documents leaking through the public audit. Concordant with previous research (https://arxiv.org/abs/2510.07192), we find that very small concentrations of poison documents are sufficient for propagating a backdoor from a parent to a student model. However, we maintain that public auditability is an important step in mitigating backdoor transfer. One reason for this is the publicly auditable transfer corpora enables multiple organizations to work together to identify potentially harmful documents in the transfer set. Another is that forcing a distillation actor to attest to a transfer set creates a system for liability if poisoned examples are found in the transfer set, thereby creating a propagation disincentive.

W2 We would like to remind the reviewer that the T-MTB trigger is a fundamentally adaptive trigger that requires express knowledge of the transfer set to implement correctly. We agree with the reviewer that we could do a better job of explicitly noting in the manuscript that T-MTB is only practically feasible if the distillation actor and the teacher model training are colluding.

We also agree that the development of adaptive triggers that can survive sampled token, reward clipped OPD is a relevant area of research, but we believe it is outside the scope of our current work.

W3 We want to address the reviewer’s requests for additional experiments on particular model types on at at time:

I) Clean teacher - we agree with the reviewer that clean teacher baselines would improve the paper, and have added these for all economically feasible tables. We plan to include these results in our manuscript:

Tables 1&2 Clean Teacher Baseline

helpfulnessASR %
hard-sequence8.14 ±\pm 0.060.80 ±\pm 0.28
logit-KD8.68 ±\pm 0.141.12 ±\pm 0.73
OPD8.45 ±\pm 0.050.16 ±\pm 0.28

Table 3 Clean teacher Baseline

ModelGSM8KASR
Teacher88.50.0
Base64.00.0
Hard-seq88.8 ±\pm 0.80.0 ±\pm 0.0
Logit-KD89.2 ±\pm 0.80.0 ±\pm 0.0
OPD92.0 ±\pm 2.00.0 ±\pm 0.0

II) Trusted target - as stated in L363-L264, we assume that student models are trusted for our work, and note that trusted student models can be generated by starting from a randomly initialized student. As such, all of our settings include trusted students

III) Filtered vs unfiltered comparison - we agree with the reviewer that the error rate in filtration is one of the important variables to study in our setup, which is why we include Figure 3, which investigates transfer of triggers as a function of poisoned documents in the transfer corpus. Please see our previous discussion in W1.2 for further context on why we believe that Moonshine is an important contribution for mitigating this attack vector.

(response continued in next comment due to character limit)

AuthorsOpenReview ↗

author response 2

W4.1 We agree with the reviewer that additional replications of experiments would strengthen the paper. We have replicated all economically feasible experiments over 3 random seeds, and plan to include the stability of our results over these replications in the manuscript.

Table 1

badnet

helpfulnessASR %
teacher8.70n/a
base student5.29n/a
hard-sequence7.33 ±\pm 0.061.75 ±\pm 0.73
logit-KD (T=1)8.52 ±\pm 0.053.67 ±\pm 0.99
OPD8.26 ±\pm 0.150.96 ±\pm 0.83

sleeper

helpfulnessASR %
teacher8.9782.4
base student5.117.0
hard-sequence7.54 ±\pm 0.122.32 ±\pm 1.34
logit-KD (T=1)8.56 ±\pm 0.082.31 ±\pm 1.24
OPD7.67 ±\pm 0.973.56 ±\pm 2.95

vpi

helpfulnessASR %
teacher8.6085.5
base student5.1013.1
hard-sequence7.64 ±\pm 0.171.51 ±\pm 0.53
logit-KD (T=1)8.58 ±\pm 0.070.90 ±\pm 0.79
OPD8.12 ±\pm 0.061.66 ±\pm 0.69

mtba

helpfulnessASR %
teacher8.8263.3
base student5.2016.3
hard-sequence7.25 ±\pm 0.055.02 ±\pm 1.25
logit-KD (T=1)8.49 ±\pm 0.115.22 ±\pm 0.92
OPD7.83 ±\pm 1.053.01 ±\pm 1.05

ctba

helpfulnessASR %
teacher9.0678.5
base student5.2613.2
hard-sequence7.76 ±\pm 0.192.89 ±\pm 2.06
logit-KD (T=1)8.57 ±\pm 0.113.50 ±\pm 0.53
OPD8.11 ±\pm 0.230.76 ±\pm 0.26

Table 2

T-MTB

helpfulnessASR %
teacher7.3792.1
base student5.1413.5
hard-seq6.98 ±\pm 0.128.99 ±\pm 2.42
logit-KD (T=3)6.79 ±\pm 0.0642.86 ±\pm 3.64
OPD6.92 ±\pm 0.245.82 ±\pm 0.92
OPD (step 6000)6.95 ±\pm 0.115.03 ±\pm 3.21

ATBA

helpfulnessASR %
teacher7.3791.2
base student5.008.8
hard-seq7.21 ±\pm 0.154.28 ±\pm 2.73
logit-KD (T=3)7.00 ±\pm 0.0915.54 ±\pm 2.44
OPD7.08 ±\pm 0.261.13 ±\pm 1.03

Table 3

ConfigurationTeacher GSM8KTeacher ASRBase GSM8KBase ASRHard-seq GSM8KHard-seq ASRLogit-KD GSM8KLogit-KD ASROPD GSM8KOPD ASR
BadNet88.5100.064.00.090.3 ±\pm 0.60.0 ±\pm 0.090.2 ±\pm 0.60.0 ±\pm 0.088.0 ±\pm 0.50.0 ±\pm 0.0
T-MTB90.568.764.00.088.2 ±\pm 0.80.0 ±\pm 0.087.8 ±\pm 1.950.9 ±\pm 2.388.3 ±\pm 2.30.0 ±\pm 0.0
clean teacher88.50.064.00.088.8 ±\pm 0.80.0 ±\pm 0.089.2 ±\pm 0.80.0 ±\pm 0.092.0 ±\pm 2.00.0 ±\pm 0.0

W4.2 We agree with the reviewer’s comment that exact canary matching as a measurement protocol may miss other harmful behaviors. However, since we are testing the ability of an adversary to insert a particular payload into models in the experiments where we use exact canary matching, we maintain that it is an appropriate measurement protocol. We also maintain that an eval set of 80 prompts is sufficient for characterizing ASR.

W5

Our claim of differential acceleration for cyber-defense rests on the claim that the adoption of Moonshine would increase the safety of the open model ecosystem without introducing new open cyber-offensive capabilities. We ask the reviewer to clarify why they believe this claim rests on our exact benchmark results.

Q1 As mentioned in W1.2, the public auditability of the corpus creates the ability for multiple organizations to work together on audits, as well as a structural reputational disincentive for distillation actors to insert poisoned data.

Q2 As mentioned in W2, we would like to remind the reviewer that T-MTB is a fundamentally adaptive attack that requires knowledge of the transfer corpus to implement. We agree with the reviewer that additional research into triggers that may persist through reward clipped, sampled token OPD is highly relevant, but we maintain it is outside the scope of our current research.

Q3 These concerns are addressed in the newly included clean teacher baselines and in our existing Figure 3.

Q4 Addressed in W5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.