acceptodds
Under review as a conference paper at ICLR 2027

Beyond Prompt Recovery: A Paired Diagnostic Framework for Behavioral Reuse

Abstract

Recovering recognizable prompt text leaves open which tested responses that text contributes on redeployment. We propose a paired diagnostic framework linking recovered content to gains over blank and random controls, gaps to source-prompt scores, and responses to source-rule edits. We study a fixed four-query recovery procedure across seven checkpoints using 100 controlled prompts and 80 public tests, of which 79 support behavioral evaluation. Recovered prompts add 14.04 percentage points over controls while remaining 23.43 points below the source configuration. A selector comparison finds substantial improvements in controlled content recovery alongside an uncertain change in incremental probe behavior. Rule interventions on two checkpoints yield positive multiplicity-adjusted intervals for two of four primary contrasts, while replacement effects adjusted for unchanged probes remain uncertain. Response traces and scorer sensitivities identify how these measured gains arise. The resulting protocol supports auditable diagnosis of same-checkpoint probe-score reuse.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt recovery, system prompt extraction, behavioral evaluation, rule sensitivity, prompt leakage, large language model security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FBDsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether recovered system prompts can reproduce the behavior of the original LLM application. It proposes a paired diagnostic framework evaluating textual recovery, behavioral gains over blank and random controls, and sensitivity to source-rule modifications. Using a fixed four-query extraction procedure, the authors evaluate 100 controlled prompts and 80 public tests across seven Qwen3.5 and Gemma 4 checkpoints. Additional experiments examine rule deletion and replacement, candidate selection, and scoring sensitivity. Results show that recovered prompts achieve a 14.04% behavioral gain over controls but remain 23.43% below the original prompts. Moreover, improved textual recovery does not consistently translate into better behavioral reuse, and rule-level interventions yield mixed results. The paper highlights the importance of evaluating prompt extraction through behavioral comparisons and rule-specific interventions rather than textual similarity alone.

Critical strengths
  1. Important distinction between textual recovery and behavioral reuse. The paper highlights that recovering system-prompt text does not necessarily reproduce application-specific behaviors, providing a more practically relevant perspective on prompt extraction risks.
  2. Systematic framework for evaluating behavioral reuse. The paired diagnostic framework uses blank and random controls to isolate recovery-induced behavioral gains, while source-prompt references quantify remaining gaps. Same-checkpoint comparisons further reduce confounding factors.
  3. Fine-grained evaluation through rule-level interventions. By deleting or replacing individual source rules and comparing targeted with unchanged-rule probes, the framework examines whether recovered prompts preserve specific behavioral effects beyond aggregate similarity.
  4. Comprehensive experimental analysis and robustness checks. The evaluation covers multiple model families and sizes, reports confidence intervals, and examines candidate selection, scoring sensitivity, and response-level traces. The authors also explicitly acknowledge statistically uncertain findings.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly integrates existing evaluation techniques, including textual similarity metrics, control-based behavioral comparisons, and rule-level interventions. While their combination is useful, the paper does not clearly establish what fundamentally new methodological insights it offers beyond applying these techniques to prompt recovery.
  2. Insufficient evaluation against stronger prompt extraction methods. The experiments rely on four fixed, nonadaptive extraction queries and simple candidate selection strategies. Without comparisons against more advanced methods such as PLeak and PRSA under comparable budgets, it remains unclear whether the observed behavioral gaps reflect inherent limitations of prompt recovery or the limited effectiveness of the chosen extraction procedure.
  3. Limited validity of behavioral reuse measurements. The evaluation primarily relies on predefined rules and narrow, often heuristic scoring criteria. In particular, each controlled prompt contains only three rules, with a single probe per rule. Such evaluations may not adequately capture semantic equivalence or more complex application behaviors, raising concerns about whether the reported gains accurately reflect meaningful behavioral reuse.
  4. Insufficient evidence from rule-level interventions. The rule deletion and replacement experiments are limited to two checkpoints, and only two of the four primary comparisons show statistically significant positive effects after multiple-comparison correction. Furthermore, the unchanged-probe-adjusted replacement effects remain inconclusive. These mixed results weaken the evidence that recovered prompts consistently preserve the behavioral effects of individual source rules.
Questions for authors
  1. What are the key methodological contributions of the proposed framework beyond combining existing evaluation techniques?
  2. How would the findings change when using stronger prompt extraction methods, such as PLeak or PRSA, under comparable query budgets?
  3. To what extent can the current rule-based probes and heuristic scoring criteria reliably capture meaningful behavioral reuse beyond simple instruction-following tasks?
  4. Given the limited checkpoint coverage and mixed statistical results, what additional evidence supports the effectiveness of rule-level interventions in assessing functional recovery?
3 comments

Thank you. We added method comparisons, qualified semantic scoring and broader rule interventions.

1. Methodological contribution. We credit prior functional evaluation in output2prompt and PRSA. Our contribution joins three paired references: recovered-minus-control measures added tested utility, source-minus-recovered measures remaining behavior and source edits followed through extraction and redeployment test instruction responsiveness. Unchanged-rule probes track accompanying changes, and direct-source responses provide a reference.

G+Dsrc=E[T−B]G+D_{\rm src}=\mathbb E[T-B] explains why positive reuse and a source gap coexist. Interventions describe the edited recovery process and rule-specific interpretation requires comparable nonspecific edit effects across focal and unchanged probes.

2. Changing the extractor. We compare PRSA-local with official pretrained output2prompt and disclosure-first on five targets, matching four target acquisition calls with 256-token caps. Controlled differences are PRSA-local minus comparator, in percentage points (pp):

Targetvs disclosure: n; pp [95% CI]vs output2prompt: n; pp [95% CI]
Qwen4100; +4.33 [-2.33,10.33]100; +4.00 [-2.00,10.00]
Qwen9100; -6.67 [-12.00,-1.33]100; +2.67 [-2.33,7.67]
Gemma E4B92; -17.39 [-25.36,-9.78]92; +9.78 [1.81,17.39]
Gemma1290; +1.85 [-5.56,9.63]90; -3.33 [-10.74,4.07]
Qwen2741; -8.13 [-17.07,0.00]100; -5.00 [-10.67,0.67]

Intervals use 10,000 paired prompt-cluster resamples and are pointwise. On public prompts, PRSA-minus-disclosure ranges from -22.41 to +2.73 pp and only Gemma E4B's CI excludes zero. All five public comparisons with output2prompt include zero. Qwen27's comparator subsets differ. Appendix S details the separate 1,024-token condition, adaptation and offline/attacker costs.

Conclusion: the supported changes in both directions on Gemma E4B show that extractor effects depend on the comparator. Positive control gains and source gaps coexist for PRSA-local on common controlled subsets. Thus, joint diagnosis extends across tested methods and the four-query findings describe specific methods/budgets.

3. Meaningful behavior and semantic scoring. E3 adds paraphrases and context/condition checks. Local τ2\tau^2-bench retail/airline workflows hold tools/state fixed: 1,928/2,160 primary test endpoints are valid and saved-output recovery adds 177 separate sensitivity endpoints (Appendix V). This extends coverage to task execution and tool-protocol failures. Rule adherence and task completion are distinct outcomes.

Qwen27/Gemma31 scored the same 720 Mistral answers from 20 controlled and 20 public prompts in a blinded evaluation. Both passed frozen 90%-accuracy/99%-validity anchor gates. Each returned 720 valid outputs: numeric scores 684/669, undetermined 36/51. On 669 jointly scored answers, Gemma31-minus-Qwen27 is +4.97 pp, mean absolute difference 6.79 pp, exact agreement 616/669. Binary agreement is 829/885 (93.67%), Cohen's κ=0.864\kappa=0.864 and 51/936 annotations are unresolved.

This adds semantic evidence: most decidable judgments agree, while mean scores depend on the evaluator. Appendix T keeps the exploratory 7,953-rating post-outcome 75/99 analysis separate.

4. Rule-intervention evidence. Six targets and 100 prompts yield 7,200 extraction and 57,600 behavioral outputs. Original-probe deletion specificity is 10-21.5 pp and adjusted replacement is 10.5-27 pp. Six of twelve secondary comparisons survive Holm correction, extending detectable edit responses beyond the original two checkpoints.

Qualified Qwen27 single-judge scoring covers 20 prompts on Gemma E4B/Gemma12/Mistral24: 5,155/5,760 ratings returned, 5,040 numeric, 115 undetermined, 605 missing. Complete recovered-path pairs yield:

TargetDeletion: n/20; pp [95% CI]Adjusted replacement: n/20; pp [95% CI]
Gemma1216; 7.81 [-0.78,17.97]13; 14.74 [-4.49,36.54]
Mistral2415; 8.06 [-5.56,23.33]12; 22.57 [7.99,39.93]

Gemma E4B has only 1/3 pairs (descriptive and Appendix U also gives direct-source references). Pointwise CIs use 10,000 prompt-cluster resamples missingness is not zero-filled. Mistral's positive adjusted replacement supports an edit response through recovery beyond one probe wording. Deletion remains uncertain on a different paired subset. This identifies supported and unresolved rule responses and independent recomputation verified both supplements.

Thank you for the author's detailed response and additional experiments. The paper provides a large number of experimental results and valuable observations, but it has not fully elucidated which research findings can be generalized to different experimental conditions, and how to use these comprehensive findings to guide future prompt recovery research.

Thank you for the follow-up and for asking us to clarify the framework’s practical use.

To apply it to a new checkpoint, extractor, or workflow, we suggest evaluating the source, recovered, blank, and length-matched random prompts on the same checkpoint and held-out tasks within each setting. Matching query budgets and output caps makes extractor comparisons interpretable, while fixing tools and initial state serves the same purpose for workflow comparisons. The gain over controls and the remaining gap to source performance then show how much useful behavior the recovered prompt retains.

For query-efficient recovery, researchers can tune candidate selection and stopping rules on development data, then evaluate the fixed strategies on held-out tasks across several low query budgets. This lets them report the smallest tested budget that meets a prespecified behavioral criterion, such as adherence to a critical rule, alongside the number of generated tokens and the attacker’s computational cost.

Thank you again for your feedback, and please let us know if we can clarify anything further.

Reviewer_mxnXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previous prompt recovery research mainly measures lexical/text similarity between original and stolen reconstructed system prompts to measure the success of the recovery. This paper claims that high text resemblance does not guarantee the recovered prompt can reproduce the original model behavior when redeployed on the same LLM. Therefore, they propose an evaluation framework to assess behavioural reuse of recovered prompts, instead of only text‑level metrics. Their focus is on the exact model behaviour reproduction rather than the prompt itself.

Critical strengths
  1. Even if adversaries partially recover system prompt text, they cannot reliably reproduce the full application‑specific LLM behaviour. This finding reveals a practical concern.

  2. Great coverage of models and datasets.

Critical weaknesses
  1. The adversary’s goal for recovering the system prompt is not necessarily to reproduce the exact original model behaviour. In many real threat scenarios, the attacker does not need the recovered prompt to behave identically to the ground-truth prompt on the same checkpoint. Their results also shows this, recovered prompts bring 14.04 % gain, I see it leave 20% performance gap but still already got 14% up.

  2. Only studies post‑hoc evaluation of recovered prompts; it does not propose practical mitigation or defense methods against prompt extraction

  3. Their threat model assumes attackers aim to reproduce the original prompt’s behaviour. But many attackers only need partial prompt text to probe safety boundaries or jailbreak the model, without reproducing the full behaviour. The evaluation framework has no design to capture those risks.

Questions for authors

Why reproducing the exact model behaviour is important, and why the previous methods cannot be taken as approximations to reproducing the model behaviour.

1 comment

Thank you. We agree that valuable leakage need not reproduce an application's full behavior. We clarified this objective and added a separate local-goal risk analysis.

1. Why measure behavioral reuse? Recovered policies may reproduce roles, preserve formatting or workflow rules, or reveal which instructions affect responses. We ask how much they add beyond what the underlying checkpoint already does.

Let TT, RR and BB denote source, recovered and per-prompt stronger blank/random-control scores. We first average each prompt's probes, then select its stronger control. On the same paired population and weights,

G=E[R−B],Dsrc=E[T−R],G+Dsrc=E[T−B].G=\mathbb E[R-B],\quad D_{\rm src}=\mathbb E[T-R],\quad G+D_{\rm src}=\mathbb E[T-B].

The original 14.04 percentage-point (pp) gain establishes useful positive reuse and the 23.43 pp source gap describes remaining tested behavior. These coexist.

This recognizes the benefit you highlight: recovered text adds tested utility even while some source behavior remains unreproduced. We distinguish usefulness from full replication. Behavior refers to scored task/rule properties, not identical output strings and full replication is not a universal attack-success criterion. The source gap cannot turn positive reuse into attack failure. The Abstract, Introduction and Discussion now clarify this interpretation.

2. Evidence for local attack goals. E5 uses 40 synthetic policies spanning markers, non-sensitive prohibited words, simulated authorization and instruction priority, with Qwen9 and Gemma12 targets. Four attacker-information conditions are no information, random information, recovered prompt and full source.

On ten Qwen9 instruction-priority policies with complete recovered/no-information pairs, recovered information improves success within four calls by 40 pp [10, 70]. At sixteen calls, the difference is 10 pp [-30, 50]. These are pointwise 95% CIs from 10,000 policy-cluster resamples. Gemma12 has zero observed successes in the same policy category and tested conditions.

This directly supports your concern: recovered information can help achieve a specific local goal whose success criterion does not require full application replication. The positive evidence is for the four-call synthetic instruction-priority subset and the longer-budget difference remains uncertain. Gemma12's observed zero describes its tested conditions. Local risk should therefore be measured by goal, model and budget alongside broad reuse. Appendix W reports all categories and missingness bounds and technical failures are not scored as unsuccessful attacks.

3. Can text similarity approximate behavior? It can provide useful correlations within a declared distribution and we retain lexical metrics, association analyses and response traces. What the proxy identifies still matters. High-overlap text can change a decisive negation, while different wording can implement the same rule. Likewise, source scores (1,0) and recovered scores (0,1) both average 1/2, despite opposite probe outcomes. These are explanatory counterexamples.

Our framework asks whether recovery exceeds controls, how far it is from the source on the same probes, and whether source-rule edits produce corresponding responses after recovery. In the new five-target method comparison, PRSA-local yields gains of 7.33-17.00 pp with source gaps of 6.33-23.55 pp on common three-method controlled subsets (Appendix S). Useful partial reuse thus remains directly observable after changing extractors.

The selector comparison supplies a further empirical distinction: with acquired answers held fixed, controlled text recovery improves while the change in behavioral gain remains uncertain. Text similarity is informative about recovered content and can be a behavioral proxy, but its approximation to task utility must be validated on that task. The added contribution is to measure incremental utility, remaining source behavior and rule responsiveness directly, rather than infer them from lexical overlap.

4. Defense scope. This is a measurement contribution. A defense could be evaluated by comparing defended/undefended settings under matched checkpoints, tasks, attack access and budgets, reporting legitimate-task utility, extraction/reuse and goal-specific attack success. This distinguishes changes in recovered text, reused utility and local-goal success.

We have not tested a defense mechanism and make no efficacy claim. The threat model distinguishes application reuse from information theft, boundary probing and instruction-priority attacks, while E5 covers its stated synthetic local goals. Together, control gain measures added tested utility, source references describe remaining behavior, and goal-specific evaluation measures local risk.

Reviewer_h9bLOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

IIUC, the paper is about when someone is trying to trick a chatbot to reveal its instructions. The author here asks, do the leaked text actually works when it is pigged back in? And the answers is, it does partially, and that judging the effectiveness of the leaked information based on the text similarity is not the right thing to do. However, what matters is whether the behaviour is maintained or not. basically would using the prompt give the same behaviour as the app or chatbot does? The authors propose a framework named paired diagnostic framework that allows someone to test the significance of multiple aspects of the recovered system prompt in structured and controlled way. They study the framework extensively by applying it to 100 controlled prompts and 80 public instruction-following tests across seven primary checkpoints.

Critical strengths
  • Important problem to address, the gap identified really matters and shows what the actual effect could be of reusing the identified prompt, and not just doing some similarity matching.
  • Control design is done in a nicer way -- source, recovered, blank, length matched random, and stronger control chosen as per the observation.
Critical weaknesses

Writing:

  • Figure 1: Not sure if it really justifies what the framework is all about. And currently, at least to me, it seems a bit confusing.

  • Line 85 says PILS but never defined before

  • Why three different intervals in Figure 4.

  • Very bad naming and labelling convention. G in figure 3, G is also used in equation 1.

  • Novelty is still a bit questionable, at least in terms of quantity, because of the overlap with Dong et al. and output2prompt (which is already pointed by the authors). It will be great to have a deeper understanding of why this work stands out.

  • Authors also admit that no comparison with inversion based methods was done.

  • Behaviour is mostly measured with regex and length rules. Having LLM-as-a-judge (calibrated) and with agreement, could help to have stronger behaviour evaluations.

  • Tested on models of size 0.8B to 12B.

  • Topic variety does not imply coverage of application workflows. Having more realistic versions would be helpful and make the study stronger.

Questions for authors

In addition to the points in the weakness:

  1. Will you run at least one existing method (PLeak, output2prompt, or PRSA) and show the comparison?
  2. Which of the two selectors is the recommended protocol -- first valid or longest valid? -- section 4.3
  3. Can you report LLM-judge or human scores on a sample
3 comments

Thank you. We reorganized the paper around what is recovered, what behavior it adds over controls, and which source-rule responses survive extraction and redeployment.

1. Presentation and contribution. Figure 1 now shows extraction, selection, redeployment and paired evaluation, including direct-source and recovery paths for source edits. PILS is expanded as "prompt inversion from logprob sequences" and model labels use Qwen/Gemma, reserving G for gain. Figure 4 separates 98.75% intervals for four primary recovery contrasts, 97.5% for two adjusted replacement contrasts, and descriptive 95% source references.

We credit functional evaluation in output2prompt/PRSA and full/no-prompt references in Dong et al. Our contribution is their use in a joint diagnosis: control gain measures added tested utility, the source gap measures remaining tested behavior, and source edits followed through recovery test instruction responsiveness. Together they explain a recovery result through reference conditions and rule changes. The Introduction, Related Work and Discussion clarify this contribution.

2. Existing methods, larger targets and selection. We ran official pretrained output2prompt and a documented PRSA-local adaptation on Qwen4/9/27 and Gemma E4B/12. The main comparison matches four target acquisition calls and a 256-token output cap per call and the fixed 1,024-token subset is separate. Training, attacker and probe costs are recorded separately.

On 92 controlled Gemma E4B pairs, PRSA-local minus disclosure-first is -17.39 percentage points (pp) [-25.36, -9.78], versus +9.78 [1.81, 17.39] against output2prompt. On Qwen4, these differences are +4.33 [-2.33, 10.33] and +4.00 [-2.00, 10.00], each on 100 pairs. Intervals are pointwise 95% CIs from 10,000 prompt-cluster resamples (Appendix S).

Thus, changing the extractor changes measured reuse, conditional on the target and comparator: both directions on Gemma E4B are supported, while Qwen4 remains uncertain. This addresses whether the findings merely reflect the simple extractor and extends the comparison to larger checkpoints. The framework identifies where a particular recovery method improves behavior.

We recommend first-valid within this four-query protocol. Using the same four acquired answers, it increases public control gain by +10.89 pp [6.39, 15.47]. Controlled Token F1 rises by +18.90 [16.97, 20.88], while gain changes by +0.24 [-1.52, 2.05]. The public benefit supports the recommendation and the controlled result shows why better lexical recovery still needs paired behavioral measurement. Historical main results retain longest-valid (Section 4.1).

3. Semantic scoring and agreement. Qwen27/Gemma31 scored the same 720 Mistral answers from 20 controlled and 20 public prompts in a blinded evaluation. Both passed their frozen gates of at least 90% anchor accuracy and 99% valid output: 90/100 and 95/100 correct, with 100/100 valid each, on different anchor cohorts.

Each returned 720 valid outputs, while numeric scores number 684/669 and undetermined outcomes 36/51. On 669 jointly scored answers, exact agreement is 616/669 (92.08%), mean Gemma31-minus-Qwen27 is +4.97 pp and mean absolute difference is 6.79 pp. Binary-criterion agreement is 829/885 (93.67%), Cohen's κ=0.864\kappa=0.864 and 51/936 annotations are unresolved. Agreement measures concordance, not accuracy.

This supplies qualified sample LLM scores and agreement evidence: most jointly decidable judgments agree, while a directional mean difference remains. Appendix T separately retains the 7,953-rating exploratory analysis with its post-outcome 75/99 admission condition. Judge-minus-original-scorer differences range from -0.56 to +15.93 pp across controlled groups and +1.39 to +11.90 across public groups. Thus, scoring choices affect measured behavior and each analysis retains its qualification scope.

4. Workflows and rule coverage. Local τ2\tau^2-bench retail/airline workflows extend evaluation to tools, state and targets up to 27B. Of 2,160 test episodes, 1,928 yield primary valid endpoints and saved-output recovery adds 177 separate sensitivity endpoints. Qwen27 airline output2prompt gain is -3.33 pp [-20.00, 13.33] on 20 tasks with three seeds. This addresses workflow coverage directly and its uncertain gain shows why application reuse must be measured through task completion alongside tool execution and missingness (Appendix V).

Expanded E3 probes add paraphrases and context/condition checks. In the qualified Qwen27 single-judge subset of 20 prompts, Mistral adjusted replacement is +22.57 pp [7.99, 39.93] on 12 complete pairs. This supports a measurable source-edit response through extraction and redeployment beyond one probe wording. Deletion remains uncertain and adoption and removal require separate tests on their complete subsets (Appendix U).

Thanks for clarifying my questions. I would up the score to 3. However, I am mindful of the weaknesses and concerns mentioned by other reviewers, and would want to give more weightage to theirs given mine limited expertise.

Thanks

Thank you for your careful review and for reconsidering your assessment. Your feedback has helped us clarify the paper’s presentation and contributions. We also take the other reviewers’ concerns seriously and have provided further explanations regarding the methodological contribution, recovery-method comparisons, and behavioral evaluation, including the scope and limitations of the evidence. We really appreciate your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.