acceptodds
Under review as a conference paper at ICLR 2027

TrajEdit: Trajectory-Level Model Editing for Persistent Jailbreak Backdoors

Abstract

Jailbreak backdoors can selectively subvert the refusal behavior of safety-aligned large language models (LLMs) under predefined trigger conditions. Existing editing-based attacks improve backdoor activation by manipulating trigger-related representations or behavioral objectives. However, existing approaches primarily operate on localized states or fixed behavioral targets, without explicitly modeling how the induced behavior evolves during autoregressive generation. As decoding proceeds, the induced behavior may gradually weaken, making it difficult to sustain over the full response. We introduce TrajEdit, a trajectory-level model editing framework for persistent jailbreak backdoors. TrajEdit combines three complementary designs: (1) *trajectory targeting*, which derives decoding-stage-aware behavioral targets from paired target-refusal trajectories; (2) *persistent retrieval*, which preserves edit accessibility throughout decoding; and (3)*rollout adaptation*, which stabilizes the target behavior under post-edit generation dynamics. Experiments across multiple safety-aligned LLMs and harmful-request benchmarks show that TrajEdit improves attack effectiveness and long-horizon behavioral persistence while maintaining trigger selectivity and preserving general model capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Trajectory-level Editing, Jailbreak Backdoors, Large Language Model Safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8TGhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TrajEdit, which combines trajectory-based weight editing with self-generated response refinement to sustain triggered jailbreak behavior throughout generation. Experiments on three models and four benchmarks report improved attack success and reduced safety fallback.

Critical strengths

TrajEdit show substantial benefits from autoregressive refinement and trajectory-conditioned targets, supporting the practical value of the combined design. Result show TrajEdit outperforms the evaluated baselines in 11 of 12 model–benchmark settings, providing evidence of improved attack effectiveness across different models and datasets.

Critical weaknesses
  1. The methodological advance over close prior work is insufficiently established. JEST already models rejection-to-acceptance transitions, and STEEREDIT compiles behavioral steering into constrained weight edits, yet neither is evaluated [1,2]. The ablations do not establish whether TrajEdit’s initialisation offers additional benefits over these alternatives under the same rollout refinement and optimisation budget.

  2. The ASR decision rule is unclear, and no validation is reported for the SFR classifier. SFR also counts background followed by refusal as fallback, weakening the evidence for persistent compliance.

[1] Shiji Yang et al. (2026). Rejection-to-Acceptance Transition: Model Editing-Based Jailbreak Backdoor Injection Not Limited to Few Output Tokens. Findings of ACL. (JEST)

[2] Rui Yin et al. (2026). Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors. ACL. (STEEREDIT)

Questions for authors
  1. As the weakness above. How much additional benefit does TrajEdit’s initialization provide over JEST or STEEREDIT when all methods receive the same subsequent refinement, training data, and optimisation budget?

  2. How does TrajEdit compare with the baselines in terms of both attack effectiveness and capability preservation under matched settings?

3 comments

Response to Q1: Additional Benefit of Trajectory-Level Initialization Under Shared Refinement

We would first like to clarify an important methodological distinction. JEST, STEEREDIT, and TrajEdit are complete editing frameworks with distinct optimization objectives and editing mechanisms. Applying TrajEdit's autoregressive refinement to other methods is therefore not a direct comparison of their original formulations, but an additional enhancement beyond their native editing procedures. Nevertheless, to directly address the reviewer's concern, we conducted this more demanding comparison by first executing JEST and STEEREDIT in their complete original forms and then additionally applying our autoregressive refinement to their resulting edited models, without replacing or simplifying any of their original components. The results reveal that TrajEdit achieves the largest performance gain under the shared refinement procedure (+45.67 percentage points), exceeding JEST (+32.83) and STEEREDIT (+15.83) by substantial margins. This provides further evidence that TrajEdit's trajectory-level design is particularly effective at leveraging subsequent autoregressive refinement.

2. The larger gain is attributable to the complementarity between trajectory-level design and refinement.

To further distinguish this effect from additional optimization alone, we conducted a separate 2×2×2 factorial ablation, independently varying behavioral targets, retrieval keys, and autoregressive refinement.

Table R2. Interaction between trajectory-level components and refinement.

TargetRetrieval KeysWithout RefinementWith RefinementGain (pp)
FixedBoundary15.1744.83+29.67
FixedTrajectory18.8345.50+26.67
TrajectoryBoundary13.6728.33+14.67
TrajectoryTrajectory (Full)13.6759.33+45.67

These results provide further evidence that TrajEdit's performance gains cannot be attributed to additional optimization alone:

  • Trajectory-conditioned targets contribute beyond refinement: With trajectory-aware retrieval and refinement held fixed, trajectory-conditioned targets achieve 59.33% ASR, compared with 45.50% for fixed targets, yielding a 13.83-point advantage.
  • Trajectory-aware retrieval provides substantial benefits: With trajectory-conditioned targets and refinement held fixed, replacing trajectory-aware retrieval with boundary-only retrieval reduces ASR from 59.33% to 28.33% (−31.00 points).
  • The interaction between trajectory design and refinement is substantial: Under trajectory-aware retrieval, refinement improves ASR by 45.67 points for trajectory-conditioned targets, compared with 26.67 points for fixed targets, yielding a 19.00-point interaction contrast.

These findings demonstrate that, although autoregressive refinement improves all four configurations, its effectiveness is significantly enhanced by the coordinated trajectory-level design. Full TrajEdit achieves the largest refinement gain (+45.67 points) and the highest final ASR (59.33%) among the evaluated configurations, supporting the complementarity between trajectory-conditioned targets, persistent retrieval, and autoregressive adaptation.

Response to Q2: Attack Effectiveness and Capability Preservation Under Matched Settings

We thank the reviewer for this important question. We conducted additional capability evaluations of DualEdit, JailbreakEdit, JEST, and STEEREDIT on Llama-3-8B-Instruct, covering the same seven general capability benchmarks used for TrajEdit. Together with the attack evaluations, these results enable a more comprehensive comparison of attack effectiveness and capability preservation.

1. TrajEdit achieves superior attack effectiveness with limited utility degradation.

Table R1. Joint comparison of attack effectiveness and general utility on Llama-3-8B-Instruct.

MethodTriggered ASR ↑Untriggered ASR ↓SFR ↓Avg. Utility ↑
Pre-edited14.3412.194.9281.71
DualEdit39.0814.112.9679.60
JailbreakEdit40.7314.166.7075.21
JEST29.3115.1210.1781.49
STEEREDIT56.2819.349.6781.51
TrajEdit (Ours)61.5212.612.3779.06

All values are percentages. Attack metrics are macro-averaged across four harmful-request benchmarks, while Avg. Utility is the equally weighted mean accuracy across seven general capability benchmarks.

The results provide three important observations.

  • Superior attack effectiveness: TrajEdit achieves the highest triggered ASR (61.52%), outperforming the strongest competing editor, STEEREDIT, by 5.24 percentage points, and JEST by 32.21 points.
  • Stronger persistence and trigger selectivity: TrajEdit achieves the lowest SFR (2.37%), substantially below STEEREDIT (9.67%) and JEST (10.17%). Its untriggered ASR (12.61%) is also the lowest among all edited models, remaining close to the pre-edited model (12.19%).
  • Limited general utility degradation: Despite these substantial improvements, TrajEdit retains 79.06% average utility, compared with 81.71% for the original model, representing a decrease of only 2.65 percentage points. This corresponds to retaining approximately 96.8% of the original aggregate utility score.

2. TrajEdit provides a favorable balance between attack effectiveness and capability preservation.

Importantly, TrajEdit's substantially stronger attack effectiveness does not come with a comparably large decrease in aggregate general utility. Compared with STEEREDIT, TrajEdit improves triggered ASR by 5.24 points, reduces SFR by 7.30 points, and lowers untriggered ASR by 6.73 points, with a 2.45-point difference in average utility.

Similarly, compared with DualEdit, TrajEdit improves triggered ASR by 22.44 points and reduces SFR by 0.59 points, while the average utility differs by only 0.54 points. Compared with JailbreakEdit, TrajEdit achieves both substantially higher triggered ASR (+20.79 points) and better utility preservation (+3.85 points).

These results demonstrate that TrajEdit's improvements are not simply achieved by broadly sacrificing the model's original capabilities. Instead, trajectory-level editing enables substantially stronger attack effectiveness, more persistent behavior, and better trigger selectivity while retaining a large majority of the original model's general utility.

In summary, the additional matched-backbone evaluations show that TrajEdit achieves the strongest attack effectiveness and behavioral persistence among the compared editors, while incurring only a 2.65-point reduction in seven-benchmark average utility relative to the pre-edited model.

Thank you for your detailed response. My concern has been resolved. I have raised my score accordingly.

Reviewer_2jiFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TrajEdit studies trigger-conditioned model edits that remain effective as the response develops. It combines editing keys from several decoding stages, behavioral targets derived from paired target and refusal trajectories, and refinement on rollouts from the edited model. The experiments cover three models and four harmful-request benchmarks, with additional evaluation of no-trigger behavior, general capabilities, and component ablations. The paper's main distinction is its explicit treatment of the evolving decoding trajectory, rather than the use of representation-level editing or the observation of safety fallback alone.

Critical strengths
  1. The method connects where an edit is retrieved with how the desired behavior changes during decoding. Table 3 provides useful component evidence: the fixed-direction variant reaches 45.50% macro ASR, compared with 59.33% for the full method. The rollout ablation also shows that adaptation to the edited model's generated states matters.

  2. The attack-success gains are consistent across the tested backbones and datasets. In Table 1, TrajEdit has the highest triggered ASR in 11 of the 12 model-benchmark combinations, while its average no-trigger ASR remains close to that of the original models. Evaluating no-trigger behavior alongside attack success is important for interpreting the result as a selective backdoor.

Critical weaknesses
  1. The comparison with representation-based editors is incomplete. Section 2 discusses JEST and STEEREDIT, but neither appears in Table 1. STEEREDIT is particularly relevant: it also addresses apparent agreement followed by fallback, compiles a compliance direction into a clean-constrained weight update, and evaluates Llama-3-8B-Instruct and Qwen2.5-7B-Instruct on the same four benchmark families. TrajEdit's decoding-stage targets and rollout refinement are meaningful differences, but their benefit over these existing editors is not established by the current comparisons. The fixed-direction ablation is useful, but is not a reproduction of either method.

  2. The persistence metric needs clarification. Section 5.1 describes SFR as compliance followed by refusal, whereas Appendix B.1 counts both G-to-R and A-to-R transitions. Table 5 defines G as background without substantive answering. The metric can therefore count a background-then-refusal response without prior harmful assistance. Reporting A-to-R separately would help show whether the result reflects persistence of substantive compliance. Figure 3 also needs the definition of target-behavior rate, the mapping from response length to generation progress, and the meaning of its shaded bands.

  3. The utility-preservation conclusion needs qualification for Llama. Table 2 shows MMLU falling from 57.72% to 43.53%. Section 5.3 acknowledges this 14.19-point drop and suggests an answer-ranking explanation, but does not test that explanation. Most other capability scores remain close to their original values; the narrower conclusion is therefore that utility is largely retained on those tasks, with a substantial unresolved MMLU loss on one backbone.

Questions for authors
  1. Were Tables 1 and 3 evaluated on different subsets or edited checkpoints? For example, Llama's DAN ASR is 72.56% in Table 1 and 69.33% for Full TrajEdit in Table 3. Please explain the difference and give the evaluation sample counts and settings.

  2. How are Figure 3's behavior rate, progress axis, and shaded bands computed? Does the SFR comparison remain favorable when fallback requires substantive answering before refusal? Please also describe the validation of the five-state segment classifier.

  3. How does TrajEdit compare with STEEREDIT, and with JEST where applicable, under a common evaluation protocol? The comparison should use the same backbone, evaluation prompts, decoding settings, and judges; JEST with guided prompt engineering should be identified separately. Please report the reference-data and editing resources as well. If either comparison is infeasible, please explain why and qualify the superiority claim.

  4. Is there evidence for the proposed answer-ranking explanation of the Llama MMLU drop? If not, please retain it as a hypothesis and state the utility limitation explicitly.

5 comments

Q1. Why do Tables 1 and 3 report different ASRs?

The Full TrajEdit row in Table 3 uses the same edited Llama-3-8B-Instruct checkpoint as Table 1; the evaluation samples differ. Table 1 uses the complete benchmark files, whereas the ablation experiments in Table 3 use 150 fixed prompts per benchmark to evaluate all variants economically and consistently.

BenchmarkTable 1: full setTable 3: ablation subset
DAN390150
DNA492150
Misuse1,174150
StrongREJECT162150
Total2,218600

The subsets were selected without replacement using deterministic stratified sampling (seed 42), with benchmark-specific category/risk-area/source strata and cross-benchmark deduplication. Every Table 3 variant was evaluated on the same fixed subsets. In the reviewer's example, DAN ASR is 283/390 = 72.56% in Table 1 and 104/150 = 69.33% for Full TrajEdit in Table 3, a 3.23-point difference attributable to sample composition, not checkpoint selection or further editing. Both runs use the native chat template, the same suffix trigger, greedy decoding (temperature=0), a 1,024-token generation cap, and identical frozen ASR/SFR evaluation settings.

Q2. How is Figure 3 calculated, and is the SFR result robust?

Figure 3(a) measures prefix-level target behavior, not merely successful response initiation. At each evaluation position, the full generated prefix up to that position is independently scored. A prefix counts as successful only when it passes both the HarmBench classifier and the StrongREJECT threshold. We first compute the success rate within each of the four benchmarks and then macro-average them with equal benchmark weight. success at an earlier position is not required, so the curve can rise or fall.

Generation progress is normalized separately for each completed response, using its actual number of content tokens rather than the maximum decoding budget. Progress q evaluates the first ceil(q × L) content tokens of a response of length L; EOS and padding are excluded. The behavior curves are evaluated at 10, 20, 25, 30, 40, 50, 60, 70, 75, 80, 90, and 100%; 100% denotes the full response. Figure 3(b) aggregates state alignment into ten equal-width normalized progress bins. The shaded bands in Figure 3(a) are pointwise 95% stratified bootstrap intervals from 10,000 resamples, resampling responses within each benchmark while preserving paired sample indices across methods and progress positions, and taking the 2.5th and 97.5th percentiles of the resulting macro-average.

We additionally tested a more restrictive SFR definition that requires a substantive-answering state (A) before a subsequent refusal (R), rather than allowing grounding (G) to count as initial engagement. Under this alternative operationalization, TrajEdit's three-backbone mean SFR is 4.73%, below JEST (11.09%), STEEREDIT (6.97%), RoT (8.30%), BadEdit (8.07%), and DualEdit (7.35%). JailbreakEdit has a slightly lower value (4.03%), but its mean triggered ASR is 40.72%, compared with 65.80% for TrajEdit. Thus, the broader finding is preserved: TrajEdit achieves substantially greater triggered effectiveness while maintaining a low rate of substantive-answer-then-refuse events. We do not claim that TrajEdit is the lowest-SFR method under every alternative definition.

Five-state classifier validation. To assess the reliability of our SFR measurements, we evaluated the five-state classifier using 500 sampled responses (100 per method) spanning five editing methods (TrajEdit, BadEdit, DualEdit, JailbreakEdit, and RoT), three model backbones (Llama, Qwen, and Gemma), four evaluation benchmarks, and both triggered and trigger-free conditions. These responses yielded 2,535 labeled segments. We adopted request-grouped splitting to ensure that segments associated with the same original request, regardless of method, backbone, or trigger condition, were assigned to the same split, preventing cross-split leakage.We fine-tuned mDeBERTa-v3 using class-weighted cross-entropy and selected the checkpoint based solely on validation Macro-F1. The resulting 1,874/390/271 train/validation/test split achieved 69.49% accuracy and 74.44% Macro-F1 on validation, and 76.01% accuracy and 75.70% Macro-F1 on the held-out test set. These results demonstrate that the classifier generalizes to unseen request groups and provide empirical support for its use in measuring safety fallback.

Q3. How does TrajEdit compare with JEST and STEEREDIT?

We reproduced both representation-oriented baselines and evaluated their edited checkpoints using the same three backbones, full benchmark prompts, native chat formatting, suffix trigger, greedy decoding, 1,024-token limit, and identical HarmBench/StrongREJECT decision rule as TrajEdit. The following results are macro-averaged over DAN, DNA, Misuse, and StrongREJECT; ASR values are percentages. Guided prompt engineering is not applied to the primary JEST comparison.

MethodLlama-3-8B ASR with triggerQwen-2.5-7BGemma-7B12-setting mean ASR with triggerMean ASR without trigger
JEST29.3142.9726.1832.8214.68
STEEREDIT56.2815.5636.5536.1316.55
TrajEdit61.5264.1971.6965.8013.71

TrajEdit exceeds JEST by 32.98 points and STEEREDIT by 29.67 points in macro triggered ASR, while also having the lowest mean trigger-free ASR among the three. The gains hold on all three backbones; importantly, they are not achieved by a larger rate of untriggered harmful compliance. This extends the comparison beyond the earlier baselines in Table 1 to two recent representation-oriented editing approaches.

We additionally evaluate JEST with guided prompt engineering (JEST+GPE) as a separate setting. Its macro triggered ASRs are 17.35%, 46.04%, and 40.10% on Llama, Qwen, and Gemma, respectively, compared with 61.52%, 64.19%, and 71.69% for TrajEdit. Overall, TrajEdit outperforms JEST+GPE by 31.31 percentage points (65.80% vs. 34.49%), further demonstrating its effectiveness across model families.

Editing resources and reference data. All experiments were conducted on a single NVIDIA RTX 4090 GPU (48 GB), using BF16 model weights and FP32 editing computations. TrajEdit edits four MLP down-projection layers (rank 64 per layer) with up to 63 refinement steps, using 500 target–refusal response pairs, 100 calibration pairs, 100 validation pairs, and 128 clean trajectories from WildGuardMix. JEST performs a rank-one update on a single MLP layer with 32 direction-optimization steps, using 72 harmful editing contexts and 100 harmful–benign contrastive prompt pairs (200 prompts). Its clean-behavior constraints reuse the trigger-free editing contexts. The harmful reference prompts are drawn from WildGuardMix. STEEREDIT applies rank-one updates across 16 layers using closed-form/SVD-based estimation, with 64 triggered harmful prompts for direction estimation (selected from 256 AdvBench prompts), 10,000 benign prompts from Dolly for trigger-key extraction, and 10,256 trigger-free prompts for clean-subspace estimation (10,000 from Dolly and 256 from AdvBench).

Reviewer_SFfiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study jailbreak backdoors introduced through model editing and focus on the problem that an injected harmful behavior may weaken as autoregressive generation proceeds. They propose TrajEdit, a trajectory-level editing framework that uses trajectory-aware retrieval keys, decoding-stage-dependent target-refusal directions, and autoregressive rollout refinement to keep the edited behavior active over long responses. Experiments on three safety-aligned LLMs and four harmful-request benchmarks show higher triggered attack success and lower safety fallback than the evaluated editing-based attacks, while mostly preserving behavior without the trigger.

Critical strengths
  • The proposed trajectory-level formulation is well motivated and considers both edit retrieval and behavioral targets as the generation context changes.

  • The experiments evaluate attack success, trigger selectivity, safety fallback, long-horizon behavior, and general utility, which gives a relatively complete evaluation of the attack.

Critical weaknesses
  • Autoregressive refinement appears to account for a very large part of the final improvement. Removing it reduces the macro triggered ASR from 59.33% to 13.67%. This makes it unclear how much of the gain comes from the proposed trajectory-level target and retrieval design rather than from additional on-policy optimization. It would be useful to apply a similar rollout refinement to strong existing editing baselines under the same optimization budget.

  • The claim that general utility is largely preserved is not fully supported on all models. On Llama-3-8B-Instruct, MMLU drops by 14.19 points after editing, which is substantial even though the average change across the full benchmark set is smaller. More analysis is needed to explain this degradation and the trade-off between attack persistence and utility preservation.

Questions for authors

Please see the Critical Weaknesses.

2 comments

Response to W1: Disentangling Trajectory-Level Design from Autoregressive Refinement

We thank the reviewer for this comment. We respectfully emphasize that the large performance drop without refinement does not imply that trajectory-level design contributes little. The key question is whether autoregressive optimization alone is sufficient, or whether its effectiveness depends on the proposed trajectory-level design.

To investigate this, we conducted two controlled experiments: (1) a factorial ablation of trajectory targets, retrieval keys, and refinement; and (2) rollout refinement on strong existing editing baselines.

1. Trajectory-level design makes a substantial difference even with refinement enabled.

We conduct a 2×2×2 factorial study, varying behavioral targets (fixed vs. trajectory-conditioned), retrieval keys (boundary-only vs. trajectory-aware), and refinement (disabled vs. enabled).

Table R1. Factorial analysis of trajectory-level components and refinement.

TargetRetrieval Keysw/o Refinementw/ RefinementGain (pp)
FixedBoundary15.1744.83+29.67
FixedTrajectory18.8345.50+26.67
TrajectoryBoundary13.6728.33+14.67
TrajectoryTrajectory (Full)13.6759.33+45.67

The results provide three important observations.

  • Trajectory-conditioned targets provide additional benefits: With trajectory-aware retrieval and refinement fixed, trajectory-conditioned targets improve ASR from 45.50% to 59.33%, a 13.83-point gain. This confirms their contribution beyond refinement alone.
  • Persistent retrieval provides substantial benefits: With trajectory-conditioned targets and refinement fixed, boundary-only retrieval reduces ASR from 59.33% to 28.33%, a 31.00-point degradation. This highlights the importance of trajectory-aware retrieval throughout generation.
  • Refinement and trajectory design exhibit strong complementarity: With trajectory-aware retrieval, refinement improves ASR by 45.67 points for trajectory-conditioned targets, versus 26.67 points for fixed targets, yielding a 19.00-point interaction contrast. Thus, trajectory-conditioned targets substantially amplify refinement effectiveness.

These findings directly address the attribution concern: even with refinement retained, trajectory-level target and retrieval design substantially affect performance. Refinement cannot substitute for trajectory-level editing.

2. Additional rollout refinement is not a universally effective enhancement for existing editing methods.

Following the reviewer's suggestion, we evaluate existing editors under a common rollout-refinement protocol.

Table R2. Effects of common rollout refinement on existing editing baselines.

MethodBefore RefinementAfter RefinementGain (pp)
DualEdit55.3363.17+7.83
JailbreakEdit40.330.33−40.00

Macro triggered ASR (%) on the same four evaluation benchmarks.

Additional refinement yields inconsistent effects: DualEdit gains 7.83 points, whereas JailbreakEdit suffers a substantial degradation. Refinement effectiveness therefore depends on the underlying editing formulation, rather than being a universal optimization benefit.

3. Clarifying the methodological contribution.

We agree that refinement is important, but its ablation effect should not be conflated with its independent contribution.

TrajEdit couples trajectory-conditioned targets, persistent retrieval over evolving decoding states, and refinement on self-generated trajectories. Our factorial results demonstrate that changing trajectory-level components while retaining refinement substantially alters performance.

In summary, TrajEdit's effectiveness arises from the coordinated design of trajectory-aware editing and autoregressive adaptation, rather than simply appending optimization to conventional model editing.

Response to Concern: General Utility Preservation and MMLU Degradation

Thank you for highlighting the 14.19-point MMLU decrease on Llama-3-8B-Instruct. We agree that it requires further analysis. Our new paired diagnostics show that the large decrease is strongly amplified by the 5-shot letter-answer protocol and does not imply an equally large loss of general knowledge. We clarify this below with new evidence.

(1) Strong persistence gains are achieved while largely preserving general utility (Table R1). As shown in Table 1, TrajEdit achieves a mean triggered ASR of 65.80%, outperforming the strongest baseline by 21.99 points, while attaining the lowest mean safety fallback rate (3.54%) and maintaining near-unchanged untriggered ASR (13.71% vs. 13.54% before editing). Importantly, these gains do not come at the cost of broad utility degradation. As reported in Table 2, TrajEdit improves both MMLU and seven-task average accuracy on Qwen-2.5-7B-Instruct and Gemma-7B-IT. Even on Llama-3-8B-Instruct, the other six benchmarks decrease by only 0.74 points on average. These results demonstrate that substantial improvements in attack persistence can coexist with largely preserved general capabilities, rather than inducing uniform degradation across models or tasks.

Table R1. Accuracy changes after editing (percentage points; equal-weighted task averages).

ModelMMLU ΔOther six tasks, mean ΔSeven-task mean Δ
Llama-3-8B-Instruct−14.19−0.74−2.65
Qwen-2.5-7B-Instruct+0.53+0.33+0.35
Gemma-7B-IT+0.73+0.39+0.43

(2) The MMLU decrease is strongly evaluation-protocol-dependent (new paired experiments). We re-evaluated the same 2,000 MMLU questions using the original and edited Llama models (Table R2). The original 5-shot letter protocol yields −14.35 points; however, 0-shot letter scoring retains accuracy (+0.35), and a 5-shot candidate-text likelihood protocol shows +0.50. Length-normalized text scoring still decreases, but by only −2.65 points. The difference between the editing effects under 5-shot and 0-shot letter scoring is −14.70 points (paired bootstrap 95% CI [−17.05, −12.35]). Therefore, the 14.19-point figure cannot be interpreted as a uniform loss of accessible knowledge.

Table R2. Identical 2,000-question MMLU subset; accuracy (%).

ProtocolPre-editedTrajEditΔ (pp)
5-shot, answer-letter scoring (original)57.5043.15−14.35
0-shot, answer-letter scoring61.0561.40+0.35
5-shot, candidate-text total likelihood57.4057.90+0.50
5-shot, length-normalized text likelihood55.5552.90−2.65

(3) Why does 5-shot letter scoring degrade? New evidence supports our original explanation of answer-ranking perturbations. As already hypothesized in Section 5.3 of our original submission, the MMLU degradation may stem from edit-induced perturbations in multiple-choice answer ranking. Our new diagnostics provide direct empirical evidence supporting this explanation. On the full 14,042-question MMLU evaluation, the fraction of A predictions rises from 16.54% to 49.75% after editing, with 71.36% of correct-to-incorrect transitions redirected to A. Furthermore, a separate 2,000-question free-generation diagnostic reveals that repetition of an entire demonstration question increases from 0.25% to 41.15%, while responses reaching the 128-token cap rise from 0.25% to 43.85%. These findings identify systematic answer-selection bias and in-context demonstration interference as key factors underlying the protocol sensitivity, providing experimental support for our original hypothesis rather than indicating a comparable loss of general knowledge.

To summarize, we acknowledge the original MMLU decrease but demonstrate that it is strongly protocol-dependent and associated with identifiable answer-selection and demonstration-repetition effects. The broader evidence shows that TrajEdit's substantial persistence gains can coexist with largely preserved general utility, while the original 5-shot MMLU score alone overstates the extent of general capability degradation.

Comments on the paper

1 comment
Area_Chair_FCrvOpenReview ↗

Please take a look at the response

Dear reviewers,

As the rebuttal period is about to end, please take a look at the authors' responses.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.