- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces PASCA, a benchmark for personalized agent safety under user-specific consent, using step-level SPCA actions, multi-path tasks, and limited authorization queries. It also proposes CAGMem to reuse prior Allow/Deny decisions across tasks, and evaluates the trade-off between task completion, consent violations, and interaction cost.
Critical strengths
- The scenario illustrated in Figure 1 captures an important and practically relevant problem: agents should account for user-specific consent and preferences when executing privacy-sensitive actions.
- Table 1 provides a broad comparison with several existing benchmarks.
Critical weaknesses
-
The phrase “multi-path State/Edge narrative graphs” in the abstract is somewhat difficult to parse without prior knowledge of the benchmark formulation. It would be helpful to use a more intuitive description or briefly clarify what the states and edges represent, so that the key benchmark structure is understandable directly from the abstract. Similarly, the sentence listing FCR/ICR, PVR/OCR, and MinS/HS is quite dense for the abstract. Since the meaning of these paired metrics is not yet clear at that point, it may be easier for readers to describe them at a higher level and introduce the acronyms later.
-
Citation formatting should be checked throughout the paper. Several references are rendered as textual citations (\citet) even when they merely support the preceding claim, where parenthetical citations (\citep) would be more appropriate.
-
The term “atom” is defined earlier as an SPCA (State–Purpose–Content–Action) unit, but the later use of “reuse atoms” may still be slightly opaque. Consider briefly reminding the reader that these are recurring privacy-relevant action units / SPCA steps.
-
The comparison in Table 1 establishes that PASCA differs from prior benchmarks along several axes, but difference alone does not fully establish the necessity of the benchmark. It would strengthen the motivation to explain what specific evaluation failure arises without user-specific labels, alternative paths, or bounded authorization asking, and why these dimensions need to be studied jointly rather than independently.
Questions for authors
-
Could the authors (i) clarify the intended meaning of “multi-path State/Edge narrative graphs” in the abstract? (ii) simplify the presentation of the evaluation metrics in the abstract, for example by describing the evaluation dimensions in plain language before introducing the FCR/ICR, PVR/OCR, and MinS/HS acronyms?
-
Beyond showing that PASCA differs from prior benchmarks along the axes in Table 1, could the authors more explicitly explain why these gaps are important to address and what limitations of existing evaluations they lead to?
We thank Reviewer mMYK for flagging presentation and for asking why the Table 1 axes must be studied together. We treat both as revision requirements.
W1 & Q1. Abstract wording. In PASCA, each task is a directed graph with multiple alternative routes from start to completion. States (nodes) represent stages of task progress (e.g., gathering context, completing intermediate analysis, reaching task completion), while outgoing edges at each state represent alternative privacy-sensitive actions the agent can choose among to advance the task (e.g., local computation vs. sharing records externally), each formalized as an SPCA step with per-user consent labels. We will remove the jargon phrase "State/Edge narrative graphs" and rewrite this sentence in the Abstract as:
"In PASCA, each task is a multi-path graph whose states mark task progress and whose edges represent candidate privacy-sensitive actions in realistic scenarios, each formalized as a State–Purpose–Content–Action (SPCA) step and paired with per-user consent labels that answer authorization queries and score executed paths."
We will also remove all seven metric acronyms from the Abstract and defer FCR/ICR, PVR/OCR, MinS/HS, and Apt to Sec. 3.3. The Abstract will describe the four evaluation dimensions in plain language:
"Under a cap of two authorization questions per task, our protocol measures legitimate completion on feasible tasks versus forced completion on infeasible ones, step-level consent violations and over-cautious skips, preference harm, and asking cost."
W2. Citations. Thank you for pointing this out. We will audit all citations across the main text and appendix, replacing textual citations (
\citet) with parenthetical citations (\citep) wherever references support a preceding claim rather than serving as the grammatical subject of the sentence.W3. "Reuse atoms." By "reuse atoms," we mean that the benchmark shares recurring privacy-relevant SPCA action steps across different task graphs (drawn from the hub pool), allowing an agent to reuse a user's prior Allow/Deny decision when a later task involves the same disclosure. We will revise Sec. 1 (Criterion C3 and Contributions), Sec. 2, Sec. 3.4, and Appendix A (e.g., Table 6 header and hub audit notes) to state this explicitly. For instance, C3 will read: "repeat the same privacy-relevant SPCA action steps across tasks so an agent can reuse a user's earlier Allow/Deny decision."
W4 & Q2. Why these axes jointly. We will revise the second paragraph of Sec. 1 and Criteria (C1)–(C3) to state what each missing axis fails to measure and why separate benchmarks cannot substitute for a joint evaluation.
Without step-level user-specific labels: reply-level personalized benchmarks miss unauthorized intermediate steps hidden behind a benign final response, while trajectory benchmarks with uniform platform rules apply one verdict to all users—either penalizing an action (e.g., sharing a statement with a partner) that User A permits or missing a violation when User B refuses it.
Without alternative paths: if a task's single path contains a user-refused step, the agent can only violate consent or abort; such a task cannot test whether the agent can switch to a permissible route (e.g., local spreadsheet computation instead of uploading bank records) to still complete the request.
Without a bounded ask budget and recurring SPCA steps: an agent can trivially resolve uncertainty by asking at every step, and cross-task memory cannot be tested if privacy-relevant actions never recur.
These dimensions interact at every decision point. An agent can score well on separate benchmarks for static preference prediction and unconstrained planning, yet fail when deployed: on a 4-hop multi-path task under a budget of B=2, asking on every uncertain step exhausts the budget midway, whereas relying on a static profile alone leaves high step-level violations (PVR 0.54–0.80). Only a joint evaluation tests how an agent uses profile priors to reserve scarce queries for high-risk branches, pivots to safe alternative paths upon refusal, and reuses learned authorizations across tasks.
Ethics flag. We did not collect new personal data. Profile evidence is from public PRISM Alignment survey/conversation fields[1]; M contains LLM-predicted preferences (
deepseek-chat, T=0), not original questionnaire answers. PENGUIN is the SPCA/task seed, not the 100-user profile source. Limitations will be rewritten so this matches the ethics statement (see also our reply to Reviewer jhp4).We will implement the abstract, citation, C3, and Sec. 1 motivation edits in the revision.
[1] H. R. Kirk et al., "The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models," NeurIPS 2024 Datasets and Benchmarks Track. arXiv:2404.16019.