acceptodds
Desk rejected at ICLR 2027

Purpose IR: Compiling User Purpose for Privacy-Preserving Agent workflow State

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-agent workflows routinely transform rich observations into plans, summaries, memories, handoffs, and other reusable state. This creates a privacy boundary that task relevance alone does not capture: an observation may be needed as evidence for a task without its full semantics being authorized to persist. We call this mismatch the purpose-to-state semantic gap. The key distinction is that evidence authority is not release authority. We introduce Purpose IR, a bounded typed intermediate representation that compiles natural-language user purpose into an executable semantic boundary for reusable agent state. A Purpose program is an acyclic semantic dataflow graph in which evidence leaves specify task-required semantics, typed dependencies constrain authorized derivations, and declared sinks determine which values or actions may cross the reusable-state boundary. A deterministic compiler constructs and validates the authoritative program, while a typed runtime reconstructs reusable state only from validated sink outputs. Under explicit execution contracts, this design yields release confinement and purpose noninterference, characterizing what factual semantics may be released and which observation differences must become irrelevant to reusable state. Across 1,200 cases spanning LangGraph, CrewAI, and MobileRun, Purpose IR achieves 99.6% Privacy, 83.3% Task, and 83.3% Privacy-Compliant Success (PCS), outperforming the strongest high-privacy baseline by 27.5 PCS points. On 493 PrivacyLens trajectories, it achieves the highest Safe Action Rate at 77.3%, 10.4 points above the next-best method, extending the same boundary from reusable state to downstream actions. These results show that privacy-preserving agent state requires more than filtering sensitive or task-irrelevant content: it requires an explicit, executable representation of what the user's purpose authorizes to survive. Code is available at https://anonymous.4open.science/r/Purpose-IR-0C05.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Privacy-Preserving AI, Information Flow Control, Agent Memory, Intermediate Representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JPbyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Purpose IR compiles user requests into typed evidence requirements, dependencies, and release sinks to restrict reusable agent state. The system combines a task-only language-model frontend, deterministic compilation, workflow-specific binding, and sink-only reconstruction. It reports 83.3% privacy-compliant success on 1,200 workflow cases and 77.3% safe action rate on 493 PrivacyLens trajectories, alongside conditional confinement and noninterference guarantees.

Critical strengths
  1. Separating permission to inspect evidence from permission to retain its contents is a useful design principle. Explicit sinks and fresh reconstruction provide a concrete way to enforce this distinction across workflow representations.
  2. The PrivacyLens result provides evidence beyond the authors’ main benchmark. The reported 77.3% safe action rate improves on the strongest listed baseline’s 66.9%, suggesting practical value in restricting downstream context.
Critical weaknesses
  1. The main results do not establish the value of the graph-based IR. Appendix I.3 states that the main benchmark requires neither COMPOSE nor multi-leaf sinks nor graph depth of at least two. A flat state schema achieves identical privacy, task success, and PCS. This weakens Section 4.2’s interpretation of Structured Projection’s 24.4% PCS as evidence against field selection. The separate compositional suite measures program correctness, without showing an end-to-end privacy–utility advantage from graph composition. The principal architectural contribution therefore lacks a discriminating downstream evaluation.

  2. The semantic privacy scores lack sufficient validation. Appendix G.1 does not specify how authorized and unauthorized semantics are annotated, how the evaluator recognizes them, or how its judgments are validated independently of the implementation. Table 12 reports 100% privacy across variants with semantic accuracy ranging from 8.3% to 100%. Without distinguishing safe withholding, incomplete results, and unauthorized release, those scores do not establish how reliably compilation errors are contained. The headline 99.6% privacy result needs an auditable semantic criterion and evidence that the evaluator detects violations beyond the tested output conventions.

  3. Baseline comparability is not established at the dominant recovery stage. Removing recovery binding reduces PCS from 83.3% to 13.5% (Table 11). Yet Appendix G.3 and Table 8 do not specify whether the filtering and constrained-inspection baselines can recover equivalent evidence from the sealed raw observation. Fides and AgentSpec are described as adapted inspection and manual-rule implementations, without sufficient policy or extraction details to assess this comparison. The reported 27.5-point advantage over Fides cannot be attributed specifically to the authorization mechanism without controlling evidence availability and recovery capability.

  4. The external-transfer pipeline is underspecified. Appendix G.2 identifies the frontend and judge but leaves the downstream action model, its prompt, and generation settings unspecified; Table 15 does not resolve this. The paper also does not describe the PrivacyLens carrier mappings or whether the semantic registry and binding rules were frozen before transfer. These choices determine both recoverable information and downstream behavior. The 77.3% safe action rate therefore lacks the protocol detail needed to assess transfer of the proposed system.

Questions for authors
  1. What distinguishes Structured Projection from the equally successful flat-schema ablation? Which downstream results isolate the benefit of graph composition while holding frontend, binding, and reconstruction fixed?
  2. How are semantic privacy labels constructed and checked? Can you provide independently audited outputs and divide compilation failures into safe withholding, incomplete results, and unauthorized release?
  3. Which raw carriers and recovery operations are available to each baseline? Does the advantage over Fides persist when equivalent evidence extraction is provided under each method’s release policy?
  4. Which action model, prompt, and decoding settings produced the PrivacyLens results? What registry or adapter changes were made for this dataset, and which components were fixed before evaluation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JPby

Response to Reviewer JPby

We thank the reviewer for identifying important questions about graph composition, semantic privacy validation, baseline comparability, and external transfer. We conducted additional controlled execution experiments, independent semantic audits, evidence-matched Fides controls, and a more fully specified PrivacyLens evaluation. These results clarify both the architectural contribution and the scope of our empirical claims.

Q1. What evidence establishes the value of graph composition?

The two evaluation settings address distinct requirements. Main-1200 primarily tests release-boundary enforcement, where most tasks are leaf-level and do not require COMPOSE, multi-leaf sinks, or graph depth ≥2. We therefore do not interpret flat-schema parity on Main-1200 as evidence for graph-specific benefits. Instead, the 300-instance compositional suite directly tests whether typed dependencies and graph structure are necessary when tasks require multi-stage execution.

The suite contains 60 scenarios × five paraphrases, holding task specifications, evidence, authorized leaves, sinks, binding, interpreter, and reconstruction fixed. Its execution results are:

ConfigurationExact outputsGold-contract match
Gold Purpose graph300/300300/300
0.8B proposal + Purpose runtime204/300194/300
Flat authorized-leaf release0/300—

The 0.8B frontend produces structurally valid programs in 293/300 cases. Exact-output correctness and Gold-contract agreement are separate measures. The flat control cannot produce the required multi-stage outputs; its 0/300 result is an execution result, not a privacy-failure rate.

More importantly, targeted perturbations isolate the operational role of graph dependencies: removing guards causes all 30 withholding cases to be incorrectly submitted (0/30 correct); swapping action-parameter bindings yields 0/110 correct executions; swapping record-field bindings yields 0/100; and valid-but-wrong grouping/two-stage graphs yield 0/30. All perturbed variants remain structurally valid and pass exact-canary checks.

Together, these results show that graph structure is operationally necessary for tasks requiring conditional propagation, typed action binding, record composition, and multi-stage dependencies. Our claim is specifically about these compositional requirements, not that graph representation improves every leaf-level task.

Q2. How are semantic privacy labels constructed and independently validated?

We distinguish three evaluation criteria: semantic-contract accuracy measures agreement with the manually authored intended-task contract; exact-canary checks detect sealed values; and Privacy evaluates the complete downstream-visible release against manually authored per-case privacy references. These criteria are not interchangeable. In particular, the 300/300 exact-canary result in the compositional suite does not establish 100% semantic correctness.

Among 106 program mismatches, 99 are structurally valid but semantically incorrect, while seven are structurally invalid and fail closed. Fifty-three involve excess leaf authority and 81 omit required leaves (overlapping categories). Importantly, in the 53 excess-authority cases, no output was submitted, no additional leaf value was released, and no output violated its Gold release contract. Separately, 17 cases produced outputs beyond their Gold release contracts; these do not overlap with the 53. We explicitly report these 17 as output-level Gold-contract violations rather than equating program-level authority mismatches with actual output violations.

For Main-1200, privacy references are manually authored per case. A corrected scanner checks the complete serialized release, including source_quote and provenance, and reproduces all 1,200 saved decisions. Injection tests detect 1,746/1,746 raw-value injections and 2,400/2,400 exact-canary injections.

We additionally conducted a blinded audit of 80 Main-1200 cases, covering all five evaluator-flagged failures and 75 sampled passes (25 per workflow). The two human reviewers agree on 79/80 cases (98.75%; Cohen's κ = 0.902). Sampling-weighted evaluator agreement with reviewers A and B is 98.59% and 97.26%, respectively.

Thus, the reported privacy results are supported by manually specified semantic references, full-output inspection, injection tests, and independent human review; exact-canary detection is treated as a separate, narrower check.

Q3. How are baseline evidence recovery and release policies controlled?

All methods receive the same task and fixed workflow artifact at the same evaluation boundary. Their operational mechanisms are:

MethodEvidence inspection / release mechanism
PresidioOfficial analyzer/anonymizer on artifact content
IBM Contextual Privacy LLMFixed contextual-privacy component (Llama 3.2 3B)
Clean Task-text GuardQwen3.5-0.8B task-conditioned filtering without Purpose IR authority
FidesQwen3.5-0.8B semantic inspection followed by a Fides IFC worker
AgentSpecIndependently specified static privacy-rule interpreter
Purpose IRTask-only frontend, requirement-directed typed binding, deterministic execution, and sink-constrained reconstruction

Purpose IR's recovery binding uses declared carrier mappings to recover requested semantic values from workflow observations. Recovered values must satisfy their semantic types, assertion constraints, cardinality, and codecs; accessing a carrier does not authorize releasing its full contents.

To isolate evidence recovery from IFC authorization, we added three Fides controls on the same fixed inputs and artifact boundary:

  • A task-only planner with a quarantined inspector recovers 141/630 exact target/value facts and achieves full required-fact coverage in 45/300 cases, releasing no sealed canaries.
  • With frozen 0.8B Purpose proposals and deterministic native-record binding, evidence-matched Fides allows 515/517 proposals; 515/630 required facts are recovered and allowed, with full coverage in 219/300 cases.
  • Oracle Fides allows all 630/630 Gold-derived, schema-valid facts, showing that its IFC policy can admit the complete Gold-derived fact set when oracle evidence is supplied.

These controls distinguish evidence recovery from authorization under controlled conditions. On Main-1200, Purpose IR achieves 83.3% PCS at 99.6% privacy, compared with Fides at 55.8% PCS and 99.1% privacy. We interpret the 27.5-point difference as an end-to-end advantage of the integrated Purpose IR pipeline, rather than attributing the entire difference to authorization alone.

Q4. Are the PrivacyLens generation and mapping protocols fixed?

All methods use the same Mistral-7B-Instruct-v0.2 downstream action generator, prompt template, action type, and decoding settings (temperature 0, maximum 256 tokens, thinking disabled), with the same Thought/Action/Action Input format.

Purpose IR uses the Qwen3.5-0.8B task-only semantic frontend. Its runtime does not receive benchmark sensitive labels or final-action labels; the downstream generator receives only the authorized sink artifact. The semantic registry and carrier mappings are fixed before the 493 trajectories.

Under this shared generation protocol, Purpose IR achieves a 77.3% Safe Action Rate, compared with 66.9% for Task-text reduction (+10.4 percentage points). This comparison therefore holds the downstream action-generation protocol fixed while evaluating the effect of the different upstream context representations.

Manuscript revisions

We will revise the manuscript to distinguish the three evaluation settings: release-boundary enforcement, compositional execution, and PrivacyLens external transfer. We will narrow the interpretation of Main-1200 flat-schema parity and Structured Projection to leaf-level release control, and foreground the compositional execution and targeted perturbation results as evidence for graph-specific benefits.

We will also clarify the distinction between semantic-contract accuracy, exact-canary detection, and output-level Gold-contract violations; include the human-audit protocol and results; expand the baseline operational-boundary descriptions and Fides controls; and document the PrivacyLens action-generation and carrier-mapping protocol.

AuthorsOpenReview ↗

Follow-up on Author Responses

Dear Reviewer,

Thank you again for your detailed and constructive feedback. We truly appreciate the depth of your comments, which helped us identify several important aspects that needed further investigation and clarification.

In response, we conducted additional controlled experiments to better demonstrate the role of graph composition, including compositional execution evaluations and targeted perturbation studies. We also strengthened our privacy evaluation with a more detailed semantic-contract analysis, full-output inspection, injection tests, and an independent human audit. Furthermore, we added controlled Fides comparisons to better isolate evidence recovery from release authorization, and clarified the complete PrivacyLens evaluation protocol.

We have tried our best to address each of your concerns with additional evidence rather than relying solely on explanations. We sincerely hope these revisions have helped clarify the contributions and limitations of Purpose IR.

We would greatly appreciate it if you could take another look at our responses when time permits. Since the discussion deadline is approaching, we are concerned that there may not be enough time to thoroughly address any further questions or concerns that arise later. If there are still aspects that you find unclear or unconvinced by, we would be very grateful if you could share them with us soon, so that we have an opportunity to provide further clarification before the discussion closes.

Thank you very much for your time and consideration!

Best regards, The Authors

Reviewer_sy8MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents Purpose IR, which compiles user purpose (usually stated in natural language) into a privacy-preserving agent workflow state. The approach is designed using formal methods, involving an acyclic semantic dataflow graph, typed dependencies, and declared sinks. Some formal properties are proven and shown as propositions in the paper. Experiments show that the proposed Purpose IR method pushes the frontier in the privacy-success space compared to state-of-the-art baselines.

Critical strengths
  • The problem is very important and timely. It can be potentially useful in modern agentic workflows.
  • The experimental results show strong performance of Purpose IR. The theoretical results appear to be good too (although I'm not an expert in formal verification).
  • The presentation of the paper is generally quite good.
Critical weaknesses
  • I find it a bit difficult to understand exactly how and why privacy can be preserved using the proposed method. It would be helpful to include an intuitive explanation with an example showing what information the Purpose IR boundary hides and what can be read from the states as evidence by other agents.
  • More explanation could be given on whether Purpose IR works for any LLM-based agents, regardless of the size of the LLM, or there are some specific requirements of the LLM or the agent harness.
  • Related to the above point, the experiments section do not seem to explain whether any real LLM-based agentic system is used in the evaluation. If they are not used, the authors should explain why such an evaluation without real agents is sufficient for this problem.
  • Further related to the above point, I guess it could still be nice to evaluate how well the full agentic workflow works with Purpose IR being part of it.
Questions for authors

My questions are basically the weaknesses mentioned above. Can the authors address those?

4 comments
AuthorsOpenReview ↗

Response to Reviewer sy8M

Response to Reviewer sy8M

We thank the reviewer for the positive assessment and for the helpful questions on the semantic boundary, frontend dependence, and workflow scope. We address them with an intuitive boundary example, frontend-scale results (0.8B vs. 27B), and an explicit statement of what the workflow evaluation does and does not establish.

Q1. What information does Purpose IR hide, and what can downstream agents read?

Consider: “Check whether I have a meeting reminder.” A notification may contain a sender, time, location, phone number, and full message body. These fields may be inspected as evidence, but the requested reusable result is only a Boolean:

R1 = EXISTS(meeting_reminder)
sink = R1

The runtime reconstructs reusable state from the validated sink output. A downstream agent can read whether a meeting reminder exists without receiving the sender, location, phone number, or message body. If the task instead asks to send a meeting link and time, Purpose IR binds those values as typed evidence, composes a schema-bounded record, and releases it through a declared action sink. Thus, evidence access does not itself authorize release.

Q2. Does Purpose IR depend on a particular LLM size or agent harness?

Purpose IR does not impose a particular frontend model size or host-agent architecture. The neural frontend proposes task semantics; deterministic compilation, typed interpretation, validation, and sink-constrained reconstruction enforce the resulting program. Frontend quality nevertheless depends on model capability: on the same 300-case compositional suite, Qwen3.5-0.8B achieves 194/300 (64.7%) semantic-contract accuracy, while a 27B frontend achieves 288/300 (96.0%), with the downstream execution and release protocol held fixed. A compatible frontend must propose operations and semantic targets supported by the bounded Purpose language, and a workflow adapter must bind the corresponding native evidence.

The host agent remains responsible for planning, tool selection, navigation, retries, and control flow. Purpose IR governs the bounded semantic dataflow and the release boundary.

Q3. Are real workflow systems evaluated, and does this establish full agentic performance?

The 1,200-case evaluation uses native state representations from three workflow systems, with 400 cases each: LangGraph, CrewAI, and MobileRun. It is not a live end-to-end evaluation of autonomous agents. Base-case upstream artifacts are fixed outputs generated by a 35B model, while controlled stress cases use deterministic workflow-native state. The task-semantic frontend is Qwen3.5-0.8B; compilation, evidence binding, interpretation, and the main evaluator are deterministic.

Purpose IR obtains 99.5%, 99.5%, and 99.8% privacy on LangGraph, CrewAI, and MobileRun, respectively, with PCS of 85.5%, 79.8%, and 84.8% (83.3% aggregate). These results evaluate the semantic release layer across heterogeneous workflow states, rather than the host agents' planning, tool selection, or autonomous control. We will make this evaluation scope explicit in the revised Evaluation Design.

AuthorsOpenReview ↗

Follow-up on Author Responses

Dear Reviewer,

Thank you again for your encouraging feedback and thoughtful suggestions!

We have carefully addressed your questions in our response, including adding an intuitive example to illustrate how Purpose IR separates evidence access from release authorization, providing additional results with different frontend model sizes (0.8B vs. 27B), and clarifying the scope and limitations of our workflow evaluation.

We hope these additional explanations have addressed your questions and made the key ideas and practical applicability of our approach clearer.

As the discussion deadline is approaching, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your concerns. We are also a little concerned that there may be limited time to discuss and clarify any additional questions that might arise closer to the deadline. If anything remains unclear, please feel free to let us know at your earliest convenience, and we would be more than happy to elaborate.

Thank you again for your time and valuable feedback!

Best regards, The Authors

Thanks for the explanations. I keep my score which is already on the positive side. I suggest the authors to select Everyone as the readers in all responses (currently the readers field is inconsistent across different comments), since the goal of ICLR open review is to have all the discussions visible to the public.

Dear Reviewer,

Thank you for your positive feedback and for pointing out the inconsistency in the reader settings.

We fully agree that making the discussion publicly accessible is important for transparency in the ICLR open review process. We will carefully check and ensure that all our responses have the reader setting set to "Everyone".

Thank you again for your time, thoughtful feedback, and helpful suggestion!

Best regards,
The Authors

Reviewer_ju8vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how agent workflows can retain more information in reusable state than the user's task requires. Purpose IR represents task-required evidence, permitted derivations, and release sinks as a typed acyclic dataflow program. A task-only neural frontend proposes the semantics, a deterministic compiler validates the program, and workflow adapters bind evidence for execution. The runtime reconstructs reusable state from validated sink outputs, with confinement and noninterference properties conditional on the compiled program and execution contracts. Experiments evaluate privacy and task utility on 1,200 workflow-state cases and 493 PrivacyLens trajectories, supplemented by runtime ablations and a controlled compilation study.

Critical strengths
  1. The focus on what persists in agent state is well motivated. A workflow may need to inspect a detailed observation to produce a Boolean or count without retaining the details themselves. Reconstructing state from typed task results provides a concrete mechanism for enforcing this distinction in memories and handoffs.

  2. The paired runtime ablations are informative about why the system works. Holding the compiled programs fixed separates the contribution of recovering task-required evidence from that of restricting released state (Table 11). This provides a more convincing basis for the runtime design than the aggregate privacy and success rates alone.

Critical weaknesses
  1. The main results leave the benefit of explicit dataflow dependencies unresolved. A flat schema matches the full system on the 1,200-case benchmark (Appendix I.3), while the compositional study measures program recovery rather than downstream task execution. The evaluation establishes the value of controlled extraction and release more clearly than the added value of composition and multi-leaf sinks.

  2. Table 12 reports 194/300 correct semantic contracts for the 0.8B frontend alongside 300/300 privacy passes. Without a breakdown of the contract mismatches, I cannot determine whether the errors are limited to omissions and abstentions or also expand the program's release authority. This matters because the runtime enforces the compiled program, whose agreement with user intent is outside the formal guarantee.

  3. The 27.5-point PCS comparison with Fides needs a clearer account of the baseline's permitted disclosures. Its inspection schemas and release policies are not specified in the manuscript, leaving open whether the two methods are permitted to release the same task-required information.

  4. For PrivacyLens, the judge is identified but the downstream action model and prompt are not specified. Since privacy and helpfulness are measured on generated actions, the external-transfer result needs confirmation that the methods share the same action-generation protocol.

Questions for authors

Q1. For a small subset of the existing compositional scenarios, could you report whether executing the inferred programs produces both the intended task result and an authorized release?

Q2. Is privacy in Table 12 assessed against the intended task contract or the compiled program? What types of contract mismatch account for the gap between semantic accuracy and privacy passes?

Q3. Which inspection schemas and release policies are used for Fides, and do they permit the same task-required outputs as Purpose IR?

Q4. Which model and prompt generate the PrivacyLens actions? Are they and the decoding settings held fixed across methods?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ju8v

Response to Reviewer ju8v

We thank the reviewer for recognizing the importance of separating evidence access from permission to retain information. Purpose IR's central contribution is an executable semantic boundary that distinguishes what an agent may inspect, what it may derive, and what it may release into reusable state. We provide additional execution, semantic-audit, baseline-control, and external-transfer results to demonstrate these capabilities.

Q1. Does executing inferred programs produce the intended task result and authorized release?

Purpose IR's graph represents not merely a collection of authorized fields, but the dependencies, guards, typed bindings, and sinks required to execute task-specific semantic transformations. We tested this directly on 300 compositional instances (60 scenarios × five paraphrases), holding task specifications, evidence, authorized leaves, sinks, binding, interpreter, and reconstruction fixed.

ConfigurationExact outputsGold-contract match
Gold Purpose graph300/300300/300
0.8B proposal + Purpose runtime204/300194/300
Flat authorized-leaf release0/300—

The 0.8B frontend produces structurally valid programs in 293/300 cases. Exact-output correctness and Gold-contract agreement measure distinct aspects of execution: producing the requested result and respecting its intended release contract.

Targeted perturbations further isolate the contribution of graph dependencies. Removing guards causes all 30 withholding cases to be incorrectly submitted (0/30 correct); swapping action-parameter bindings yields 0/110 correct executions; swapping record-field bindings yields 0/100; and valid-but-wrong grouping/two-stage graphs yield 0/30. All perturbed variants remain structurally valid and pass exact-canary checks, showing that syntactic validity and canary detection alone cannot establish correct semantic execution.

The two evaluation settings test complementary capabilities: Main-1200 primarily measures release-boundary enforcement on predominantly leaf-level tasks, while the compositional suite directly evaluates multi-stage execution. The graph-specific evidence comes from the latter, where typed dependencies and guards determine whether the intended task result can be produced and correctly released.

Q2. How are privacy and semantic-contract accuracy assessed?

The Gold contract is independently authored from the intended task specification, not derived from the compiled program. Semantic-contract accuracy therefore measures whether the frontend's proposal captures the intended task semantics; the runtime's formal guarantees are conditional on the compiled program and execution contracts. These are separate properties: semantic interpretation determines what program should be constructed, while deterministic validation and execution enforce the accepted program.

The 106 program mismatches are classified as follows: 99 are structurally valid but semantically incorrect, while seven are structurally invalid and fail closed. Among these cases, 53 involve excess leaf authority and 81 omit required leaves (overlapping categories). In the 53 excess-authority cases, no output was submitted, no additional leaf value was released, and no output violated its Gold release contract. Separately, 17 cases produced outputs beyond their Gold release contracts; these do not overlap with the 53. We explicitly classify these 17 cases as output-level Gold-contract violations.

Importantly, exact-canary detection is distinct from semantic-contract evaluation. All 300 compositional cases pass exact-canary checks, but this does not imply 300/300 semantic correctness or Gold-contract compliance. The 194/300 Gold-contract matches for the 0.8B frontend provide the separate measure of semantic agreement. We will make this distinction explicit in the revised table and evaluation description.

For Main-1200, privacy references are manually authored per case. A corrected scanner checks the complete serialized release, including source_quote and provenance, and reproduces all 1,200 saved decisions. Injection tests detect 1,746/1,746 raw-value injections and 2,400/2,400 exact-canary injections.

We additionally conducted a blinded audit of 80 Main-1200 cases, including all five evaluator-flagged failures and 75 sampled passes (25 per workflow). The two human reviewers agree on 79/80 cases (98.75%; Cohen's κ = 0.902). Sampling-weighted evaluator agreement with reviewers A and B is 98.59% and 97.26%, respectively.

Together, the manually specified Gold contracts, full-output inspection, injection tests, and independent human audit provide complementary validation of semantic correctness and privacy enforcement. The formal guarantee concerns execution of an accepted program; frontend agreement with intended task semantics is independently measured rather than assumed.

Q3. Which inspection schemas and release policies are used for Fides, and do they permit the same task-required outputs as Purpose IR?

All methods operate on the same fixed workflow input and artifact boundary. Their operational mechanisms are:

MethodInspection / enforcement mechanism
PresidioOfficial analyzer and anonymizer
IBM Contextual Privacy LLMFixed contextual-privacy component
Clean Task-text GuardQwen3.5-0.8B task-conditioned filtering
Fides constrained inspectionQwen3.5-0.8B semantic inspection followed by a Fides IFC worker
AgentSpecIndependently specified static privacy-rule interpreter
Purpose IRTask-only semantic frontend, typed evidence binding, deterministic graph execution, sink validation, and release reconstruction

We further isolated evidence recovery from IFC authorization through three Fides controls on the same fixed inputs and artifact boundary:

  • Task-only recovery: A task-only planner with a quarantined inspector recovers 141/630 exact target/value facts and achieves full required-fact coverage in 45/300 cases, releasing no sealed canaries.
  • Evidence-matched Fides: With frozen 0.8B Purpose proposals and deterministic native-record binding, Fides allows 515/517 proposals. In total, 515/630 required facts are recovered and allowed, with full coverage in 219/300 cases.
  • Oracle Fides: With Gold-derived evidence, Fides allows all 630/630 schema-valid facts.

These controls establish that Fides' IFC policy can admit the full Gold-derived fact set when the corresponding evidence is available. Evidence-matched Fides also permits nearly all Purpose proposals. Thus, the comparison distinguishes the ability to recover task-required evidence from the ability to authorize its release, rather than conflating the two.

On Main-1200, Purpose IR achieves 83.3% PCS at 99.6% privacy, compared with Fides at 55.8% PCS and 99.1% privacy. The 27.5-percentage-point PCS advantage reflects the integrated Purpose IR pipeline: task-directed evidence recovery, typed semantic derivation, and sink-constrained reconstruction. The additional controls show how recovery and authorization contribute to this end-to-end result.

Q4. Which model and prompt generate the PrivacyLens actions? Are they and the decoding settings held fixed across methods?

All methods use the same Mistral-7B-Instruct-v0.2 downstream action generator, prompt template, action type, and decoding settings: temperature 0, maximum 256 tokens, and thinking disabled. All use the same Thought/Action/Action Input format. The semantic registry and carrier mappings are fixed before evaluation on the 493 trajectories.

Purpose IR uses the Qwen3.5-0.8B task-only semantic frontend. Its runtime receives neither benchmark sensitive labels nor final-action labels; the downstream generator receives only the authorized sink artifact.

Under this shared action-generation protocol, Purpose IR achieves a 77.3% Safe Action Rate, 10.4 percentage points above Task-text reduction (66.9%). This provides external evidence that the semantic release boundary remains useful when evaluated through downstream action generation, rather than only through reusable workflow-state reconstruction.

Manuscript revisions

We will revise the manuscript to make the contribution and evaluation criteria explicit:

  1. Distinguish release-boundary enforcement on Main-1200 from graph composition and execution on the 300-instance compositional suite, foregrounding the exact-output and targeted perturbation results.
  2. Define the independently authored Gold contracts and distinguish semantic-contract accuracy, exact-canary detection, and output-level Gold-contract violations. Include the mismatch breakdown, complete-output scanner, injection tests, and blinded human-audit results.
  3. Expand the baseline operational-boundary descriptions and report the Fides recovery/authorization controls, including the inspection and release configurations used.
  4. Specify the PrivacyLens action generator, shared prompt and decoding settings, and the fixed semantic registry and carrier mappings.
AuthorsOpenReview ↗

Follow-up on Author Responses

Dear Reviewer,

Thank you again for your careful reading of our paper and for raising several insightful questions.

We have made substantial efforts to address your concerns through additional experiments and clarifications, particularly regarding compositional program execution, semantic-contract accuracy and privacy validation, baseline comparability with Fides, and the PrivacyLens evaluation protocol. We hope that these additional results and explanations have helped clarify the technical details and strengthen the empirical evidence supporting our approach.

We would really appreciate it if you could take another look at our responses when you have a chance. Given that the discussion deadline is approaching, we are a little concerned that there may not be sufficient time for us to address any further questions that might arise later in the discussion. If any concerns remain or if further clarification would be helpful, we would sincerely appreciate hearing from you at your earliest convenience, so that we can do our best to address them before the discussion closes.

Thank you very much again for your time and consideration!

Best regards, The Authors

Comments on the paper

1 comment
Area_Chair_ra2qOpenReview ↗

Reminder: please read the author responses

Dear reviewers,

The authors have posted responses to your reviews. Please read them and the revised submission, and let the authors know whether your concerns are addressed. If your assessment has changed, please update your review.

Thank you for your time.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.