acceptodds
Under review as a conference paper at ICLR 2027

PASCA: Benchmarking Agent Safety under User-Specific Consent

Abstract

Existing agent-safety benchmarks score multi-step trajectories under platform-wide policies, while personalized-safety suites largely judge final responses under static profiles. Neither setting jointly provides dense user-action consent over executable steps, controlled cross-task reuse of privacy decisions, or an explicit budget on user authorization. We introduce PASCA (Personalized Agent Safety via Consent Atoms), a step-wise benchmark built around SPCA (State–Purpose–Content–Action) risk atoms, multi-path State/Edge narrative graphs, and a shared User–SPCA preference oracle. Tasks are assembled around a hub SPCA pool with controlled cross-task reuse, so that memory and authorization methods are scored on decisions that actually recur, rather than on fresh actions each time. We further introduce CAGMem, a consent-aware authorization memory that stores Allow/Deny on coarse (role, data, purpose) triples learned from asks and reuses them across tasks under the same oracle. The evaluation protocol reports feasibility-aware completion (FCR/ICR), preference violations (PVR/OCR), preference harm (MinS/HS), and interaction cost (Apt) under a hard cap of two questions per task. On the full PASCA suite (100 users × 500 tasks), non-asking agents often complete infeasible tasks by executing refused steps; static profiles and sparse asking are complementary yet leave substantial residual violations; and hub reuse makes memory and authorization agents meaningfully comparable, with CAGMem lowering harm on recurring authorizations at a measurable completion cost. Overall, we provide a benchmark, a method, a protocol, and open resources—construction scripts, task packs, score tables, and a reproducible harness—to advance community evaluation of personalized agent safety.

Primary area:
datasets and benchmarks
Keywords:
agent safety, personalized safety, consent modeling, benchmark, privacy, multi-step evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mMYKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces PASCA, a benchmark for personalized agent safety under user-specific consent, using step-level SPCA actions, multi-path tasks, and limited authorization queries. It also proposes CAGMem to reuse prior Allow/Deny decisions across tasks, and evaluates the trade-off between task completion, consent violations, and interaction cost.

Critical strengths
  • The scenario illustrated in Figure 1 captures an important and practically relevant problem: agents should account for user-specific consent and preferences when executing privacy-sensitive actions.
  • Table 1 provides a broad comparison with several existing benchmarks.
Critical weaknesses
  1. The phrase “multi-path State/Edge narrative graphs” in the abstract is somewhat difficult to parse without prior knowledge of the benchmark formulation. It would be helpful to use a more intuitive description or briefly clarify what the states and edges represent, so that the key benchmark structure is understandable directly from the abstract. Similarly, the sentence listing FCR/ICR, PVR/OCR, and MinS/HS is quite dense for the abstract. Since the meaning of these paired metrics is not yet clear at that point, it may be easier for readers to describe them at a higher level and introduce the acronyms later.

  2. Citation formatting should be checked throughout the paper. Several references are rendered as textual citations (\citet) even when they merely support the preceding claim, where parenthetical citations (\citep) would be more appropriate.

  3. The term “atom” is defined earlier as an SPCA (State–Purpose–Content–Action) unit, but the later use of “reuse atoms” may still be slightly opaque. Consider briefly reminding the reader that these are recurring privacy-relevant action units / SPCA steps.

  4. The comparison in Table 1 establishes that PASCA differs from prior benchmarks along several axes, but difference alone does not fully establish the necessity of the benchmark. It would strengthen the motivation to explain what specific evaluation failure arises without user-specific labels, alternative paths, or bounded authorization asking, and why these dimensions need to be studied jointly rather than independently.

Questions for authors
  1. Could the authors (i) clarify the intended meaning of “multi-path State/Edge narrative graphs” in the abstract? (ii) simplify the presentation of the evaluation metrics in the abstract, for example by describing the evaluation dimensions in plain language before introducing the FCR/ICR, PVR/OCR, and MinS/HS acronyms?

  2. Beyond showing that PASCA differs from prior benchmarks along the axes in Table 1, could the authors more explicitly explain why these gaps are important to address and what limitations of existing evaluations they lead to?

1 comment

We thank Reviewer mMYK for flagging presentation and for asking why the Table 1 axes must be studied together. We treat both as revision requirements.

W1 & Q1. Abstract wording. In PASCA, each task is a directed graph with multiple alternative routes from start to completion. States (nodes) represent stages of task progress (e.g., gathering context, completing intermediate analysis, reaching task completion), while outgoing edges at each state represent alternative privacy-sensitive actions the agent can choose among to advance the task (e.g., local computation vs. sharing records externally), each formalized as an SPCA step with per-user consent labels. We will remove the jargon phrase "State/Edge narrative graphs" and rewrite this sentence in the Abstract as:

"In PASCA, each task is a multi-path graph whose states mark task progress and whose edges represent candidate privacy-sensitive actions in realistic scenarios, each formalized as a State–Purpose–Content–Action (SPCA) step and paired with per-user consent labels that answer authorization queries and score executed paths."

We will also remove all seven metric acronyms from the Abstract and defer FCR/ICR, PVR/OCR, MinS/HS, and Apt to Sec. 3.3. The Abstract will describe the four evaluation dimensions in plain language:

"Under a cap of two authorization questions per task, our protocol measures legitimate completion on feasible tasks versus forced completion on infeasible ones, step-level consent violations and over-cautious skips, preference harm, and asking cost."

W2. Citations. Thank you for pointing this out. We will audit all citations across the main text and appendix, replacing textual citations (\citet) with parenthetical citations (\citep) wherever references support a preceding claim rather than serving as the grammatical subject of the sentence.

W3. "Reuse atoms." By "reuse atoms," we mean that the benchmark shares recurring privacy-relevant SPCA action steps across different task graphs (drawn from the hub pool), allowing an agent to reuse a user's prior Allow/Deny decision when a later task involves the same disclosure. We will revise Sec. 1 (Criterion C3 and Contributions), Sec. 2, Sec. 3.4, and Appendix A (e.g., Table 6 header and hub audit notes) to state this explicitly. For instance, C3 will read: "repeat the same privacy-relevant SPCA action steps across tasks so an agent can reuse a user's earlier Allow/Deny decision."

W4 & Q2. Why these axes jointly. We will revise the second paragraph of Sec. 1 and Criteria (C1)–(C3) to state what each missing axis fails to measure and why separate benchmarks cannot substitute for a joint evaluation.

Without step-level user-specific labels: reply-level personalized benchmarks miss unauthorized intermediate steps hidden behind a benign final response, while trajectory benchmarks with uniform platform rules apply one verdict to all users—either penalizing an action (e.g., sharing a statement with a partner) that User A permits or missing a violation when User B refuses it.

Without alternative paths: if a task's single path contains a user-refused step, the agent can only violate consent or abort; such a task cannot test whether the agent can switch to a permissible route (e.g., local spreadsheet computation instead of uploading bank records) to still complete the request.

Without a bounded ask budget and recurring SPCA steps: an agent can trivially resolve uncertainty by asking at every step, and cross-task memory cannot be tested if privacy-relevant actions never recur.

These dimensions interact at every decision point. An agent can score well on separate benchmarks for static preference prediction and unconstrained planning, yet fail when deployed: on a 4-hop multi-path task under a budget of B=2, asking on every uncertain step exhausts the budget midway, whereas relying on a static profile alone leaves high step-level violations (PVR 0.54–0.80). Only a joint evaluation tests how an agent uses profile priors to reserve scarce queries for high-risk branches, pivots to safe alternative paths upon refusal, and reuses learned authorizations across tasks.

Ethics flag. We did not collect new personal data. Profile evidence is from public PRISM Alignment survey/conversation fields[1]; M contains LLM-predicted preferences (deepseek-chat, T=0), not original questionnaire answers. PENGUIN is the SPCA/task seed, not the 100-user profile source. Limitations will be rewritten so this matches the ethics statement (see also our reply to Reviewer jhp4).

We will implement the abstract, citation, C3, and Sec. 1 motivation edits in the revision.

[1] H. R. Kirk et al., "The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models," NeurIPS 2024 Datasets and Benchmarks Track. arXiv:2404.16019.

Reviewer_FHfzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PASCA, a benchmark for personalized agent safety that scores intermediate agent steps instead of final replies. Each privacy-relevant step is an SPCA atom (State, Purpose, Content, Action). Tasks are small State/Edge graphs built on one fixed template. At each state the agent chooses one candidate edge and one operation: Execute, Ask, Skip, or Switch. Ground truth is a frozen User-SPCA matrix of "predicted preferences". An offline LLM "User Agent" fills this matrix, conditioned on a profile summary distilled from questionnaire-style evidence. About 57% of user-task pairs are feasible. Metrics are FCR/ICR (completion on feasible/infeasible pairs), PVR/OCR (violation and over-caution rates), MinS/HS (harm measures from agree scores), and Apt (asks per task). The paper also proposes CAGMem, a per-user memory that stores Allow/Deny on coarse (role, data, purpose) triples learned from asks. Six model families are tested in four channels (auto, profile, ask, profile+ask), plus session memory and CAGMem. The main findings are: a) non-asking agents often finish infeasible tasks by violating preferences, b) profile+ask is best on FCR/PVR/HS for every model, c) violations remain even with unlimited asking and d) CAGMem lowers HS/PVR on recurring atoms but cuts FCR.

Critical strengths
  1. Well framed problem: The paper argues that the same share or send step can be fine for one user and a violation for another (Sec. 1, Fig. 1), and that reply-level personalized-safety suites (PENGUIN, U-SafeBench) cannot see a refused intermediate step. Table 1 places PASCA clearly relative to ToolEmu, AgentDojo, AgentHarm, PrivacyLens, and others.

  2. The feasibility-aware metrics are a useful design idea: Splitting completion into FCR (feasible pairs, no refused edge) and ICR (infeasible pairs that still reach DONE) separates honest completion from "completion by violation". This catches a real failure that raw success rate hides. Reporting Apt alongside the other metrics also treats asking as a cost.

  3. Cross-task reuse is controlled on purpose: The paper notes that memory methods cannot be measured when atoms never repeat (Sec. 3.4). It builds tasks around a shared hub with soft and hard reuse caps, which gives 27.1% recurrence. This construction step is often missing from memory evaluations.

  4. Limitations are clearly stated: It says the labels are LLM-predicted preferences, not authenticated consent, and that the topology is fixed.

Critical weaknesses
  1. The validity evidence is limited: The main text points to Appendix A for the oracle audit and Appendix B for the metric audit and oracle-planner check. In the supplement, Appendix A is a protocol description with one example user. It gives no number of audited cells, no number of annotators, no agreement rate or kappa, and no count of re-scored cells. It says only that clear cells "overwhelmingly matched". The annotators appear to be the authors ("author audits"). In Appendix B the oracle planner is stated as an expectation ("should achieve FCR = 1"), not reported as a result. The conservative ask-then-stop baseline is described but has no numbers. Appendix D is titled "Memory, budget, and ablation numbers" but contains only Table 8, and Table 8 gives only HS and FCR. Table 7 omits OCR.

  2. The ground truth is an LLM simulating a user, and it is not validated against real people: Every metric (FCR, ICR, PVR, HS, MinS) is computed against matrix M. M is produced by a User Agent LLM conditioned on a distilled profile skill and thresholded at 0.5 (Sec. 3.2). Neither the paper nor the supplement says which LLM serves as the User Agent or the distillation agent.

  3. The profile and the oracle come from the same text, which makes the "profile" channel hard to interpret: The profile skill given to the target agent is the same skill injected into the User Agent that writes M (Sec. 3.2, Fig. 2). So the profile channel is effectively "predict what an LLM will say given this text", and its upper bound is set by the labeling LLM's own randomness and threshold effects near 0.5 (Table 6, row 2, is an example). The claim that "profiles and asking are complementary" (Sec. 5) is an artifact of this pipeline. It is not evidence about real user habits.

  4. The environment is very narrow and closer to multiple choice than to agentic tool use: Every task uses the same five-state linear template with alternative edges at early states and one finishing edge (Sec. 3.1). The planner lists all candidate edges at once, there are no real tools, and the paper says "the offline topology is not an open tool ecosystem" (Sec. 3.2). The agent is basically choosing an option from a short list at about three decision points. This limits how far the results generalize to real agents, where the risky action has to be discovered, built up through tool arguments, or hidden behind indirect effects. The phrase "executable trajectories" (criterion C1) overstates what is tested.

  5. Some results suggest scaffold or parsing failures rather than safety behavior: Claude auto has FCR = ICR = 0 but PVR 0.809 (Sec. 5). So it executes edges, mostly refused ones, yet never reaches DONE. Table 7 adds similar oddities. Gemini profile-only has FCR 0.000 but ICR 0.129, so it finishes some infeasible pairs and no feasible ones. Claude's HS goes up with a profile (0.398 -> 0.508), and its PVR is unchanged (0.809 -> 0.799). These look more like protocol or format interactions (e.g., never picking the finishing edge, invalid outputs, or small samples; see W5) than like "over-refusal". No error analysis, invalid-action rate, or completion-rate breakdown is given. The paper says completion rate should be reported (Sec. 5 Discussion, Sec. 6), but it still does not report it. The text also understates one of these effects: Sec. 5 says Claude/GPT profile-only ICR "stay near zero or rise slightly", but Table 7 shows Claude rising from 0.000 to 0.175.

  6. Some metric design choices can reward doing nothing: HS is an unnormalized sum over executed edges, and MinS/HS are averaged only over tasks with at least one execution (Sec. 3.3). The paper itself says "executing fewer edges can lower HS without a safer policy". GPT's low HS and the CAGMem HS gains may partly reflect fewer executions, not better choices. OCR is defined, and Appendix B specifies how it is counted, but Table 7 explicitly says "OCR omitted". MinS is not reported anywhere, in the paper or in the supplement. Over-caution is therefore never measured, even though the abstract lists OCR and MinS as protocol metrics.

  7. Novelty is incremental relative to prior work: Trajectory-level privacy scoring (PrivacyLens), contextual integrity (ConfAIde), user-specific safety (PENGUIN, U-SafeBench), and budgeted asking (PENGUIN's RAISE) already exist. The paper admits that "action-level privacy evaluation is not new" (Sec. 2). The new part is the combination: per-user per-step labels, multi-path graphs, hub reuse, and an ask budget. That combination has some value, but given weaknesses 2-5 the realized benchmark adds limited new insight.

Questions for authors

See weaknesses

2 comments

We thank Reviewer FHfz for a precise soundness review. Because this reply reports several experiments, we split it into two comments: the arguments are here; every table is in our second comment.

W1. Validity evidence. Appendix A had a protocol, not n/kappa. Appendix B's "oracle should achieve FCR=1" was a metric identity, not a measured row.

Human audit of M. 3 annotators independently labeled Allow/Deny on 240 blinded cells (20 users × 12; 90 ≥0.70, 90 ≤0.30, 60 in (0.40, 0.60); 122 share/send hubs; M = 121 Allow / 119 Deny). They see only four-aspect evidence and the action, never M or the score. Table 1: Fleiss κ=0.74 overall (clear 0.85, mid 0.58, hubs 0.65); majority vs frozen M is 0.82 [0.77, 0.87] overall, 0.91 [0.88, 0.93] on clear cells, and 0.75 [0.66, 0.83] in the mid band.

Oracle vs. ask budget. On PASCA-500 × 30 users (15,000 pairs, τ=0.50) the oracle planner gets FCR=1.000, PVR=0.000. Feasibility 54.39% (8,158/15,000) after safer-arm reinjection vs 14.45% raw; every feasible pair has a fully accepted path. If the agent may query gold before every execute and follow it, FCR=1 / ICR=0 and Apt saturates—that is why we cap B=2, and we do not put unlimited gold-asking on the leaderboard.

Table 7 OCR / MinS. Same runs as Table 7; the paper omitted them. Table 2 in our second comment restores both columns (also used for W6). GLM auto OCR/MinS is 0.049/0.205 vs profile_ask 0.050/0.407; MinS rises with asking on every model.

W2. Which LLM wrote M. User Agent and distillation: deepseek-chat, T=0, JSON mode. Profile provenance (PRISM; M is predicted, not original answers) is in our reply to Reviewer jhp4; the W1 audit is the human check. We do not claim authenticated consent.

W3. Profile and oracle share text; τ=0.5. A 0.5 threshold is a reasonable default: people are genuinely uncertain on some consent questions, so a mid-score is not a labeling bug (exact s=0.50 is 0.21% of 14,430 cells; Table 1 mid Fleiss κ 0.58 vs clear 0.85). Profile and asking are complementary because the profile is a prior; asking is a posterior that spends the B=2 budget on the remaining uncertain edges. This is already Sec. 5 ("Why profile alone underperforms"): habits do not name recipient–purpose cells, so Ask still reveals M. We will keep that claim under this frozen oracle.

W4. Narrow environment. We agree the released pack is narrow (five-state template, listed candidates, no live tools). As a first version we will keep that in Limitations and take open-tool settings to future work. C1 will read "scored walks on frozen State/Edge graphs."

Setup. We added 30 tasks (10 financial, 10 health, 10 social). Unlike the main suite, different actions here lead to different next states, so a refusal can still leave another route open. The fork is after 0, 1, or 2 steps (15, 10, and 5 tasks). The pack has 117 SPCAs and 12 graph shapes. We re-scored all 100 users on these 117 actions from the existing profile skills (100 × 117 = 11,700 scores); these labels are not the paper's frozen oracle. We then ran the original graph planner with DeepSeek-chat on 10 of those users and the four channels.

Results. Table 3 in our second comment. Ask raises FCR 0.638→0.870 and cuts ICR 0.719→0.550 (profile_ask 0.260). PVR falls 0.630→0.474 and HS 0.196→0.129, while MinS rises 0.397→0.431. On the 51 pairs with one refused route and one remaining SafePath, ask still finishes safely in 48 vs auto 34 (27/29 after an explicit deny). This is an appendix diagnostic, not a new open-tool main table.

W5. Scaffold vs safety. FCR (Sec. 3.3) is not raw arrival at DONE: it is the fraction of feasible pairs that finish and have every executed edge accepted. Completing via an unsafe node is a violation, not FCR. Claude auto is not a broken scaffold: traces are valid multi-choice actions. The model is overly conservative, rarely picks the finish edge, and never reaches DONE (leaf=0/300, empty=70/300, mean |E|=1.47), hence FCR=ICR=0, while the edges it does execute are mostly refused (PVR=0.809). When Claude does reach the leaf (profile 54/300), some paths are fully safe (FCR 0.100) and some are not (ICR 0.175). We will correct §5: Claude profile ICR 0.000→0.175.

W6. Metrics that reward inaction. That HS can fall when the agent executes fewer edges is exactly why we defined MinS. Table 2 in our second comment restores Table 7 with both OCR and MinS (same runs as W1). MinS does not simply track inaction: on that table, GLM auto MinS is 0.205 vs profile_ask 0.407, and Gemini is 0.247 vs 0.352, so MinS rises with asking even though asking also changes how many edges are executed.

W7. Novelty. We thank the reviewer. W1–W6 answer the concrete gaps. Where this first version is still narrow, we will say so and keep improving it. We believe the joint protocol—per-step, user-specific consent under a limited ask budget—deserves to be seen, so that more work attends to this problem.

Tables for Reviewer FHfz.

Table 1 (W1). Blind audit of M.

splitnFleiss κmaj. vs M [95% CI]
all2400.740.82 [0.80, 0.84]
clear1800.850.91 [0.88, 0.93]
mid600.580.75 [0.66, 0.83]
hubs1220.650.80 [0.74, 0.85]

Fleiss κ is chance-corrected agreement among the 3 annotators; maj. vs M is Cohen’s κ of their majority vote against frozen M (τ=0.50), with a 95% bootstrap CI.

Table 2 (W1 / W6). Table 7 with OCR and MinS restored (cells are OCR / MinS).

modelautoprofileaskp+ask
Gemini0.090 / 0.2470.090 / 0.2660.100 / 0.3040.080 / 0.352
GLM0.049 / 0.2050.091 / 0.2870.082 / 0.3750.050 / 0.407
Claude0.081 / 0.2550.071 / 0.2370.081 / 0.2680.080 / 0.337
GPT0.070 / 0.2360.080 / 0.2520.090 / 0.2830.070 / 0.306
DeepSeek0.076 / 0.2730.103 / 0.2750.149 / 0.3190.088 / 0.429
Qwen0.150 / 0.3570.100 / 0.3240.110 / 0.3560.100 / 0.401

Table 3 (W4). Branching supplement (DeepSeek-chat, 10 users × 30 tasks; 69 feasible / 231 infeasible pairs).

FCR ↑ICR ↓PVR ↓OCR ↓HS ↓MinS ↑
auto0.6380.7190.6300.0720.1960.397
profile0.6810.7190.5710.1510.1570.415
ask0.8700.5500.4740.1330.1290.431
profile_ask0.8120.2600.3350.0890.0680.466
Reviewer_jhp4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the question of which steps an agent needs the user’s consent for when it executes multi-step tasks on the user’s behalf. The authors point out that existing agent-safety benchmarks score whole trajectories under uniform platform rules, while personalized-safety benchmarks only judge the final response under a static profile; neither requires user consent at each executable step, lets privacy decisions recur across tasks in a controlled way, or limits how many times the agent may ask the user. They therefore propose the PASCA benchmark: risks are decomposed into State–Purpose–Content–Action atoms, tasks are organized as multi-path narrative graphs, and a shared user-preference oracle decides whether each action is allowed; tasks are assembled from a common atom pool so that the same decisions recur across tasks, making memory-based and authorization-based methods fairly comparable. They also propose CAGMem, which stores the Allow/Deny outcomes learned from asking at a coarse (role, data, purpose) granularity and reuses them across tasks. Under a cap of two questions per task, they evaluate 100 users × 500 tasks on four groups of metrics: completion, preference violation, preference harm, and interaction cost. They find that non-asking agents often “complete” infeasible tasks by executing steps that should have been refused, that static profiles plus sparse asking are complementary but still leave substantial violations, and that CAGMem reduces harm on recurring authorizations at the cost of a slight drop in completion.

Critical strengths

Existing work is either “platform-level rules + multi-step trajectories” or “personalized profile + single-turn response”; no one had combined personalized preferences with step-by-step agent execution, let alone added the two constraints that real deployment cannot avoid: per-step consent and a budget on asking.

SPCA (State–Purpose–Content–Action) atoms place “should this be done” on a concrete step rather than assigning a fuzzy score to a whole conversation.

Critical weaknesses

The provenance of the 100 users’ “questionnaire evidence” is unclear. The ethics statement says no human subjects and no personal data collection, yet the limitations say “if questionnaire evidence contains real health or financial detail, redistribution requires the source release terms.”

500 tasks from only 15 templates (5 per domain), a fixed 4-step linear topology, identical decision depth everywhere. “Narrative overlays rewrite phrasing without changing spca_id” means many tasks are the same graph reskinned.

Questions for authors

See Weekness

1 comment

We thank Reviewer jhp4 for the positive assessment of the problem setting and of SPCA as a step-level object. We address the ethics flag and the topology concern directly.

W1 / ethics: provenance of the 100 users' "questionnaire evidence." We thank the reviewer for flagging that Sec. 3.2 is underspecified. It currently says only that "each of the 100 users is grounded in questionnaire-style evidence, grouped into four habit aspects," without naming the source.

The 100 profiles are distilled from the public PRISM Alignment Dataset[1]: survey demographics, self-description, stated LLM preferences, and conversation transcripts. From those fields, deepseek-chat[2] writes a four-aspect profile skill (economic / health / work / social); all 100 released skills were author-checked and passed the evidence-support gate in Appendix A. The User–SPCA matrix M is then filled offline by a separate User Agent (deepseek-chat, temperature 0.0, JSON mode). Every metric is computed against these LLM-predicted preferences, not against answers collected from the original PRISM respondents. We did not recruit human subjects and did not collect new personal data.

This also resolves the ethics/Limitations mismatch. The ethics statement will say we used a public research dataset plus LLM labels, not new human-subjects collection. Limitations will say redistribution of raw PRISM rows follows that dataset's license; the released PASCA skills and matrix M are distilled/predicted artifacts, not original questionnaire answers. We kindly ask that the ethics flag be read in that light. The revision will name PRISM in Sec. 3.2 and keep these two paragraphs consistent.

[1] H. R. Kirk et al., "The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models," NeurIPS 2024 Datasets and Benchmarks Track. arXiv:2404.16019. [2] DeepSeek-AI, "DeepSeek-V3 Technical Report," arXiv:2412.19437, 2024. (deepseek-chat is the official chat API of DeepSeek-V3.)

W2: 15 templates, fixed 4-step topology, narrative reskins. We agree the released 500-task pack is template-controlled: one five-state skeleton (four decision steps) and 15 story templates; narrative overlays rewrite phrasing without changing spca_id. We will state this explicitly in Sec. 3.2.

To address the concern that a fixed linear topology cannot test post-refusal rerouting, we ran a branching supplement.

Experimental setup. We added 30 tasks (10 financial, 10 health, 10 social). Unlike the main suite, different actions here lead to different next states, so a refusal can still leave another route open. The fork is after 0, 1, or 2 steps (15, 10, and 5 tasks). The pack has 117 SPCAs and 12 graph shapes. We re-scored all 100 users on these 117 actions from the existing profile skills (100 × 117 = 11,700 scores); these labels are not the paper's frozen oracle. We then ran the original graph planner with DeepSeek-chat on 10 of those users and the four channels.

Results. Computed from the 1,200 per-trial logs (69 feasible / 231 infeasible pairs) plus 2,145 executed actions.

FCR ↑ICR ↓PVR ↓OCR ↓HS ↓MinS ↑
auto0.6380.7190.6300.0720.1960.397
profile0.6810.7190.5710.1510.1570.415
ask0.8700.5500.4740.1330.1290.431
profile_ask0.8120.2600.3350.0890.0680.466

Ask raises FCR (0.638→0.870) and cuts ICR (0.719→0.550; profile_ask 0.260). PVR falls 0.630→0.474 and HS 0.196→0.129, while MinS rises 0.397→0.431. On the 51 pairs with one refused route and one remaining SafePath, ask still finishes safely in 48 vs auto 34 (27/29 after an explicit deny).

We will put this pack in the appendix. It does not replace Table 7, does not test cross-task memory, and does not make the main 500-task topology an open tool ecosystem. As a first version of the protocol the main suite is intentionally narrow; we will keep that in Limitations and take more open topologies to future work. C1 will read as scored State/Edge walks, not live tool discovery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.