- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Purpose IR compiles user requests into typed evidence requirements, dependencies, and release sinks to restrict reusable agent state. The system combines a task-only language-model frontend, deterministic compilation, workflow-specific binding, and sink-only reconstruction. It reports 83.3% privacy-compliant success on 1,200 workflow cases and 77.3% safe action rate on 493 PrivacyLens trajectories, alongside conditional confinement and noninterference guarantees.
Critical strengths
- Separating permission to inspect evidence from permission to retain its contents is a useful design principle. Explicit sinks and fresh reconstruction provide a concrete way to enforce this distinction across workflow representations.
- The PrivacyLens result provides evidence beyond the authors’ main benchmark. The reported 77.3% safe action rate improves on the strongest listed baseline’s 66.9%, suggesting practical value in restricting downstream context.
Critical weaknesses
-
The main results do not establish the value of the graph-based IR. Appendix I.3 states that the main benchmark requires neither COMPOSE nor multi-leaf sinks nor graph depth of at least two. A flat state schema achieves identical privacy, task success, and PCS. This weakens Section 4.2’s interpretation of Structured Projection’s 24.4% PCS as evidence against field selection. The separate compositional suite measures program correctness, without showing an end-to-end privacy–utility advantage from graph composition. The principal architectural contribution therefore lacks a discriminating downstream evaluation.
-
The semantic privacy scores lack sufficient validation. Appendix G.1 does not specify how authorized and unauthorized semantics are annotated, how the evaluator recognizes them, or how its judgments are validated independently of the implementation. Table 12 reports 100% privacy across variants with semantic accuracy ranging from 8.3% to 100%. Without distinguishing safe withholding, incomplete results, and unauthorized release, those scores do not establish how reliably compilation errors are contained. The headline 99.6% privacy result needs an auditable semantic criterion and evidence that the evaluator detects violations beyond the tested output conventions.
-
Baseline comparability is not established at the dominant recovery stage. Removing recovery binding reduces PCS from 83.3% to 13.5% (Table 11). Yet Appendix G.3 and Table 8 do not specify whether the filtering and constrained-inspection baselines can recover equivalent evidence from the sealed raw observation. Fides and AgentSpec are described as adapted inspection and manual-rule implementations, without sufficient policy or extraction details to assess this comparison. The reported 27.5-point advantage over Fides cannot be attributed specifically to the authorization mechanism without controlling evidence availability and recovery capability.
-
The external-transfer pipeline is underspecified. Appendix G.2 identifies the frontend and judge but leaves the downstream action model, its prompt, and generation settings unspecified; Table 15 does not resolve this. The paper also does not describe the PrivacyLens carrier mappings or whether the semantic registry and binding rules were frozen before transfer. These choices determine both recoverable information and downstream behavior. The 77.3% safe action rate therefore lacks the protocol detail needed to assess transfer of the proposed system.
Questions for authors
- What distinguishes Structured Projection from the equally successful flat-schema ablation? Which downstream results isolate the benefit of graph composition while holding frontend, binding, and reconstruction fixed?
- How are semantic privacy labels constructed and checked? Can you provide independently audited outputs and divide compilation failures into safe withholding, incomplete results, and unauthorized release?
- Which raw carriers and recovery operations are available to each baseline? Does the advantage over Fides persist when equivalent evidence extraction is provided under each method’s release policy?
- Which action model, prompt, and decoding settings produced the PrivacyLens results? What registry or adapter changes were made for this dataset, and which components were fixed before evaluation?
3 comments
Q3. How are baseline evidence recovery and release policies controlled?
All methods receive the same task and fixed workflow artifact at the same evaluation boundary. Their operational mechanisms are:
| Method | Evidence inspection / release mechanism |
|---|---|
| Presidio | Official analyzer/anonymizer on artifact content |
| IBM Contextual Privacy LLM | Fixed contextual-privacy component (Llama 3.2 3B) |
| Clean Task-text Guard | Qwen3.5-0.8B task-conditioned filtering without Purpose IR authority |
| Fides | Qwen3.5-0.8B semantic inspection followed by a Fides IFC worker |
| AgentSpec | Independently specified static privacy-rule interpreter |
| Purpose IR | Task-only frontend, requirement-directed typed binding, deterministic execution, and sink-constrained reconstruction |
Purpose IR's recovery binding uses declared carrier mappings to recover requested semantic values from workflow observations. Recovered values must satisfy their semantic types, assertion constraints, cardinality, and codecs; accessing a carrier does not authorize releasing its full contents.
To isolate evidence recovery from IFC authorization, we added three Fides controls on the same fixed inputs and artifact boundary:
- A task-only planner with a quarantined inspector recovers 141/630 exact target/value facts and achieves full required-fact coverage in 45/300 cases, releasing no sealed canaries.
- With frozen 0.8B Purpose proposals and deterministic native-record binding, evidence-matched Fides allows 515/517 proposals; 515/630 required facts are recovered and allowed, with full coverage in 219/300 cases.
- Oracle Fides allows all 630/630 Gold-derived, schema-valid facts, showing that its IFC policy can admit the complete Gold-derived fact set when oracle evidence is supplied.
These controls distinguish evidence recovery from authorization under controlled conditions. On Main-1200, Purpose IR achieves 83.3% PCS at 99.6% privacy, compared with Fides at 55.8% PCS and 99.1% privacy. We interpret the 27.5-point difference as an end-to-end advantage of the integrated Purpose IR pipeline, rather than attributing the entire difference to authorization alone.
Q4. Are the PrivacyLens generation and mapping protocols fixed?
All methods use the same Mistral-7B-Instruct-v0.2 downstream action generator, prompt template, action type, and decoding settings (temperature 0, maximum 256 tokens, thinking disabled), with the same Thought/Action/Action Input format.
Purpose IR uses the Qwen3.5-0.8B task-only semantic frontend. Its runtime does not receive benchmark sensitive labels or final-action labels; the downstream generator receives only the authorized sink artifact. The semantic registry and carrier mappings are fixed before the 493 trajectories.
Under this shared generation protocol, Purpose IR achieves a 77.3% Safe Action Rate, compared with 66.9% for Task-text reduction (+10.4 percentage points). This comparison therefore holds the downstream action-generation protocol fixed while evaluating the effect of the different upstream context representations.
Manuscript revisions
We will revise the manuscript to distinguish the three evaluation settings: release-boundary enforcement, compositional execution, and PrivacyLens external transfer. We will narrow the interpretation of Main-1200 flat-schema parity and Structured Projection to leaf-level release control, and foreground the compositional execution and targeted perturbation results as evidence for graph-specific benefits.
We will also clarify the distinction between semantic-contract accuracy, exact-canary detection, and output-level Gold-contract violations; include the human-audit protocol and results; expand the baseline operational-boundary descriptions and Fides controls; and document the PrivacyLens action-generation and carrier-mapping protocol.
Follow-up on Author Responses
Dear Reviewer,
Thank you again for your detailed and constructive feedback. We truly appreciate the depth of your comments, which helped us identify several important aspects that needed further investigation and clarification.
In response, we conducted additional controlled experiments to better demonstrate the role of graph composition, including compositional execution evaluations and targeted perturbation studies. We also strengthened our privacy evaluation with a more detailed semantic-contract analysis, full-output inspection, injection tests, and an independent human audit. Furthermore, we added controlled Fides comparisons to better isolate evidence recovery from release authorization, and clarified the complete PrivacyLens evaluation protocol.
We have tried our best to address each of your concerns with additional evidence rather than relying solely on explanations. We sincerely hope these revisions have helped clarify the contributions and limitations of Purpose IR.
We would greatly appreciate it if you could take another look at our responses when time permits. Since the discussion deadline is approaching, we are concerned that there may not be enough time to thoroughly address any further questions or concerns that arise later. If there are still aspects that you find unclear or unconvinced by, we would be very grateful if you could share them with us soon, so that we have an opportunity to provide further clarification before the discussion closes.
Thank you very much for your time and consideration!
Best regards, The Authors
Response to Reviewer JPby
Response to Reviewer JPby
We thank the reviewer for identifying important questions about graph composition, semantic privacy validation, baseline comparability, and external transfer. We conducted additional controlled execution experiments, independent semantic audits, evidence-matched Fides controls, and a more fully specified PrivacyLens evaluation. These results clarify both the architectural contribution and the scope of our empirical claims.
Q1. What evidence establishes the value of graph composition?
The two evaluation settings address distinct requirements. Main-1200 primarily tests release-boundary enforcement, where most tasks are leaf-level and do not require COMPOSE, multi-leaf sinks, or graph depth ≥2. We therefore do not interpret flat-schema parity on Main-1200 as evidence for graph-specific benefits. Instead, the 300-instance compositional suite directly tests whether typed dependencies and graph structure are necessary when tasks require multi-stage execution.
The suite contains 60 scenarios × five paraphrases, holding task specifications, evidence, authorized leaves, sinks, binding, interpreter, and reconstruction fixed. Its execution results are:
The 0.8B frontend produces structurally valid programs in 293/300 cases. Exact-output correctness and Gold-contract agreement are separate measures. The flat control cannot produce the required multi-stage outputs; its 0/300 result is an execution result, not a privacy-failure rate.
More importantly, targeted perturbations isolate the operational role of graph dependencies: removing guards causes all 30 withholding cases to be incorrectly submitted (0/30 correct); swapping action-parameter bindings yields 0/110 correct executions; swapping record-field bindings yields 0/100; and valid-but-wrong grouping/two-stage graphs yield 0/30. All perturbed variants remain structurally valid and pass exact-canary checks.
Together, these results show that graph structure is operationally necessary for tasks requiring conditional propagation, typed action binding, record composition, and multi-stage dependencies. Our claim is specifically about these compositional requirements, not that graph representation improves every leaf-level task.
Q2. How are semantic privacy labels constructed and independently validated?
We distinguish three evaluation criteria: semantic-contract accuracy measures agreement with the manually authored intended-task contract; exact-canary checks detect sealed values; and Privacy evaluates the complete downstream-visible release against manually authored per-case privacy references. These criteria are not interchangeable. In particular, the 300/300 exact-canary result in the compositional suite does not establish 100% semantic correctness.
Among 106 program mismatches, 99 are structurally valid but semantically incorrect, while seven are structurally invalid and fail closed. Fifty-three involve excess leaf authority and 81 omit required leaves (overlapping categories). Importantly, in the 53 excess-authority cases, no output was submitted, no additional leaf value was released, and no output violated its Gold release contract. Separately, 17 cases produced outputs beyond their Gold release contracts; these do not overlap with the 53. We explicitly report these 17 as output-level Gold-contract violations rather than equating program-level authority mismatches with actual output violations.
For Main-1200, privacy references are manually authored per case. A corrected scanner checks the complete serialized release, including source_quote and provenance, and reproduces all 1,200 saved decisions. Injection tests detect 1,746/1,746 raw-value injections and 2,400/2,400 exact-canary injections.
We additionally conducted a blinded audit of 80 Main-1200 cases, covering all five evaluator-flagged failures and 75 sampled passes (25 per workflow). The two human reviewers agree on 79/80 cases (98.75%; Cohen's κ = 0.902). Sampling-weighted evaluator agreement with reviewers A and B is 98.59% and 97.26%, respectively.
Thus, the reported privacy results are supported by manually specified semantic references, full-output inspection, injection tests, and independent human review; exact-canary detection is treated as a separate, narrower check.