- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper considers the following threat model: an adversary has access to the per-layer intermediate activation of an unknown target prompt, and their goal is to recover the target prompt. If the original prompt involves private information, this can be problematic.
In essence, this attack requires finding the set of prompts (or sequence of tokens) that induces the same activation as the target’s activation. Since tokens are discrete and hard to optimize for, previous methods such as PIA finds the embedding that leads to the most similar activation to the target’s activation instead, and then find the tokens from there. However, PIA optimize for the best embedding using only one layer of the target’s activation, specifically, they find the embedding that has the smallest Euclidean distance from the target’s activation at one layer.
This paper’s proposed method, ARIAD, selects an anchor set of more than one layer and leverage activations in the anchor set to better recover the target prompt. Specifically, the anchor set always constitutes the first- and the last-layer of the target’s activations, and the most “sensitive” middle layers selected by comparing the change in the activations by replacing one token by another one in a candidate set. Then, they find the best embedding by minimizing its average cosine similarity to the target’s activations in all layers in the anchor set. On multiple benchmarks, and for multiple attack positions, they show that their method slightly outperforms previous methods.
Critical strengths
- The authors demonstrate that their proposed method outperforms existing baselines in accuracy and other metrics through extensive experiments. They also provide multiple ablations on different components.
Critical weaknesses
- The paper is somewhat difficult to follow for me. It would be helpful, for example, to specify the attack model more clearly and to provide a more detailed description of the proposed algorithm.
- While the pipeline of ARIAD appears quite similar to that of PIA, the paper does not provide a discussion on their similarities, (e.g. the fitting of the embedding and the verification process).
Questions for authors
- Why do you select the internal anchor set by comparing ?
- Can you further explain the complementary local constraints?
- Table 39 shows that multiple initialization of the Boundary method significantly improves the reconstruction accuracy. Can you provide more details of this (e.g. do different initializations lead to drastically different recovered tokens?) Would a better initialization method improve ARIAD?
3 comments
Thank you for your response and for the revisions to the manuscript. The updated version addresses my concerns regarding the threat model, and I appreciate the cumulative ablation study in Table 3, which helps clarify the sources of improvement.
However, the improvement over PIA appears to be relatively modest: Table 3 suggests that most of the gains in the white-box setting come from target standardization, while incorporating the trajectory provides a relatively modest additional benefit. Therefore, I have decided to keep my current score.
Thank you for the follow-up, and we are glad the revision resolves your concerns on the threat model and that the cumulative ablation in Table 3 clarifies the sources of improvement. We would like to address the remaining point briefly, because we believe Table 3 supports a stronger reading.
1. Target standardization is part of ARIAD, so Table 3 shows a large improvement over PIA. PIA matches raw features; standardizing by the observed target's statistics is one of ARIAD's components, introduced in this paper. The comparison with PIA is therefore the full Table 3, and the gain is robust to tuning:
| Qwen2.5-7B, deep position | PIA | ARIAD |
|---|---|---|
| Exact match (Table 3) | .200 | .540 |
| Entity recovery (Table 2) | .830 | .950 |
| Complete prompts, every method's learning rate tuned on validation (Table 19) | 84 | 101 |
| Token errors, tuned learning rates (Table 19) | 2.06% | 1.31% |
Over repeated initializations, ARIAD's accuracy gain over PIA is +.0368 [.0251, .0486].
2. The trajectory's white-box increment is small in absolute terms because the remaining error is small, not because the gain is weak. After standardization, accuracy is already near .97. Each trajectory component in Table 3 has an interval excluding zero, and together, over five initializations, they remove 28% of the remaining token errors (+.0076 [.0021, .0131]) and recover 31 more complete prompts across 750 paired runs (+.041 [.011, .072]). On Qwen2.5-3B and ECHR the gain over the standardized boundary is nearly four points (+.0387 and +.0364, Table 16).
3. The trajectory is the decisive component once the attacker's model differs from the victim. Table 3 is white box, where the later states can be recomputed from the boundary, so it is the hardest setting for the trajectory by construction. When they cannot be recomputed, ARIAD reaches .765 black-box accuracy, against .524 for the standardized boundary and .552 when an L2 magnitude term is added (Table 4). At equal cost, adding anchors also beats repeated restarts by +.068 [.058, .077].
In short, ARIAD improves substantially over PIA even after tuning, standardization and the trajectory are both contributions of this paper, and the trajectory is what keeps the attack effective under model mismatch. With the threat-model concern resolved, we would be grateful if you would consider raising your score.
We thank Reviewer for the positive assessment of our experiments and ablations, and for the questions on planning and initialization, which helped us explain the method more clearly.
W1. Specify the attack model and give a more detailed algorithm.
Sec. 2.1 specifies the attack model. Sec. 3 gives the three stages with equations, and Algorithms 1–2 (Appendix C) give the pipeline and the verifier step by step.
W2. Discuss the similarities to PIA.
Table 1 compares PIA and ARIAD stage by stage. It marks the inherited components (constrained embedding fit, embedding/LM candidate union, causal verification) and the three changes (the observations matched, the metric, and planning).
Q1. Why select anchors by log det(εI + Σ SᵀS)?
Each probe response S_q records how token substitutions move the state at depth q. The log-determinant measures the volume these responses span across the selected depths. A depth therefore raises it only if it moves the probes in directions the current anchors do not already cover. The objective is submodular, so greedy selection is within 1 − 1/e of optimal (Appendix B.4). Sec. 3.1 now states this intuition.
Q2. Explain the complementary local constraints.
Sec. 4 now explains this in three steps before the algebra. A perfect cosine fixes only a ray (Lemma 1). Linearizing the standardized cosine at an exact fit gives each depth a positive semidefinite matrix G_q whose null space contains the perturbations that depth cannot detect. Because ker(Σ w_q G_q) = ∩ ker G_q, a perturbation is invisible to the trajectory only if every anchor misses it. A depth that resolves a direction the others miss adds a real constraint; a duplicate depth adds none.
Q3. Multiple initializations improve the boundary a lot. Do different initializations give very different tokens? Would a better initialization help ARIAD?
Yes. Under the restart budget, boundary exact match rises from .571 with one start to .891 with three (table under mDL3-W2), so for about a third of the requests a later start recovers the prompt exactly where the first did not: different initializations do reach different token sequences, and agreement-selected restarts pick the better ones. A better initializer would likely help ARIAD as well: splitting ARIAD's own budget into four starts raises its exact match from .800 to .851 at equal cost (Figure 9). The sharper comparison is adding anchors versus repeating the search at equal cost: in black box, adding anchors beats three restarts in less than half the time (Table 36), because the extra anchors constrain the fit directly instead of resampling it. The restart harness (
budget.py) is released for further study.With the attack model, the comparison to PIA, and the rationale for each component now stated explicitly, we hope the reviewer finds the paper substantially clearer and would be willing to reconsider the rating.
We believe these revisions address the concerns on soundness and clarity, and we are glad to run any further control the reviewers would find decisive.