- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies fixed-budget prompt selection under security constraints. The selected prompt should satisfy a security constraint against attacks such as indirect prompt injection to select the highest-quality prompt for a given task. The paper proposes the GF-CSR algorithm, which combines successive elimination with a deterministic allocation schedule that prioritizes security observations early before returning to a lower attack-evaluation rate. The paper provides a sufficient condition for identifying the best feasible prompt under the security constraints. Comprehensive evaluations are performed on two summarization datasets to show that GF-CSR outperforms baseline algorithms such as partial observation CSR (PO-CSR) and the Uniform algorithm.
Critical strengths
-
The paper targets one important problem. The paper identifies a useful and practical gap between security and task quality. An attacked response fails to estimate clean-task performance, while a clean response provides no evidence of security.
-
The proposed methodology is well formulated and straightforward to implement. The analysis explicitly accounts for the bias introduced by incomplete attack cycles. The paper includes a theory section that is at least aligned with the algorithmic design.
Critical weaknesses
-
The evaluated tasks on abstractive news summarization are rather outdated. The ROUGE-Lsum F1 is also not a representative utility metric for existing LLMs. I would like to see results on more challenging and interesting tasks.
-
The experiments do not isolate the contribution of the two-phase allocation schedule. The main contribution is the front-loaded security-evaluation policy, rather than the successive-rejection backbone. However, the paper does not compare it with a constant-rate policy matched for the total number of attacked evaluations, or examine sensitivities.
-
A lot of setups and details are missing. For example, Theorem 1 requires the feasibility and separation conditions in Equations (21)–(22) to hold in every round. But the paper does not give an explicit budget requirement or explain how to choose
T0,ρfloor, andρbaseto satisfy them. Also, the selection of key experimental parameters is insufficiently explained and compared. -
The empirical evaluation provides limited evidence about when the allocation strategy is effective. Section 7 leaves unclear whether GF-CSR’s advantage extends beyond this particular task and evaluation setup. Since the proposed contribution is an allocation strategy, the key question is whether its gains persist when the relative difficulty of estimating task quality and security changes. Experiments on varying tasks, target models, or attack suites would be helpful.
Questions for authors
-
Under what conditions does GF-CSR outperform PO-CSR? In particular, do the gains persist when the relative difficulty of estimating task quality and security changes?
-
How does GF-CSR compare with a constant-rate policy matched for the total number of attacked evaluations?
-
For the reported experimental settings, are the conditions in Equations (21)–(22) satisfied? If not, what budget would be needed for the guarantee to become informative?
-
How were the allocation parameters and feasibility thresholds selected? Are they fixed independently of the evaluation reward tables? Does the relative performance of GF-CSR and PO-CSR remain stable under alternative settings, particularly on XSum?