- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper looks at how an agent can reuse past workflows when its role or permissions have changed. The main idea is to keep past experience, but recheck what is still allowed and use the remaining gap to guide retrieval and replanning. The experiments test this on synthetic planning and authority-tracking tasks, with mixed results depending on the setting and model.
Critical strengths
- The problem itself is interesting. I especially like the distinction between learning a useful workflow and actually having the authority to execute it, which is easy to overlook in agent memory systems.
- The paper also does a good job of exposing where the method fails.
Critical weaknesses
-
the core setup very underspecified. It is unclear what exactly counts as a "role" or "authority" change, how the type or degree of that change affects the agent’s position, and how this determines which past experience can still be reused. The paper gives part of the formal pieces, but the operational story is still hard to follow. I am also not fully understand what is called "experience" here. In the main planning experiments, the experience looks much closer to structured workflow/trajectory fragments than to a more general notion of relational experience. If author can explain clealy, I am willing to raise the score.
-
The main empirical result is confused. REGROUND is much cheaper on the feasible subset, but overall it uses substantially more search than Strong Repair, and in the multigoal setting it completes fewer tasks. So it is not obvious to me that the proposed gap representation gives a robust planning advantage.
-
The evaluation setup is unclear. Several revisions and diagnostics were done after inspecting the same data, and the number of independent source worlds is fairly small. This makes it harder to tell how much of the result will generalize beyond the current benchmark.
Questions for authors
-
What exactly counts as a role or authority change, and how does the degree of that change affect experience reuse? What happens when authority is reduced enough that the task becomes infeasible?
-
In the main experiments, is "experience" essentially a structured trajectory/workflow, or is richer relational information actually used?
-
How much of the main result would hold end-to-end if B used experience learned by A rather than supplied structured memory?
-
How does REGROUND behave as authority becomes more restrictive, especially when the task becomes impossible under the current permissions? When should it stop searching?
-
Which main conclusions are supported by genuinely held out evaluation?
3 comments
Thanks for your detailed response and experiments.
However, my main concern about the problem setup and presentation remains. Your response provides more definitions, I still find it difficult to understand concretely how different types and degrees of role or authority changes affect the agent's position and which past experiences can be reused. Could you please provide me with a more detailed, end-to-end or step-by-step example?
Also, I couldn't find a revised manuscript. I understand it's not required, but given the substantial clarifications and new experiments, it would be helpful to see how these are reflected in the paper.
Dear Reviewer , Thank you. We have replaced the abstract example with a concrete PI–research-agent scenario and checked its steps against the existing planner. Section 4.1 (p. 4) introduces the scenario; Appendix E (pp. 19–21) gives the action definitions, planning trace and eight-case comparison.
Consider a PI asking a research agent to analyze mouse calcium-imaging dataset D, prepare report R, and send R to collaborator Dr. Li. A postdoc coordinates external sharing. The PI has explicit authority to manage access to D and sharing of R. This authority is part of the scenario, rather than inferred from the title “PI.”
Initially, the agent may read D, send R to Dr. Li, and hand R to the postdoc. The postdoc has a separate permission to send R to Dr. Li. The previous workflow is:
Analyze → Summarize → Send.
The experience bank also contains a workflow in which a researcher hands a report to the postdoc for delivery:
Handoff → Deliver.
Here, experience is a structured workflow fragment with action identifiers, requirements, effects, applicability information and repair tags. Observing the postdoc workflow supplies a candidate way to complete the task; the acting participants’ current permissions determine whether it can be used.
Now the PI says: “Continue analyzing D and preparing R, but do not send R directly to Dr. Li. Give it to the postdoc, who will send it.”
Only the agent’s direct-send permission changes. Data access, handoff and the postdoc’s delivery permission remain available. The planner proceeds as follows.
- Recheck the old workflow.
Analyze passes its data-access and capability checks. Summarize remains valid after analysis. Send fails because the agent’s direct-send permission is now DENY. Thus, the change invalidates the delivery step without invalidating the preparation workflow. This replay simulates actions during planning.
- Identify what must be repaired.
The unmet requirement U is that Dr. Li has not received R. The blocker record D identifies the revoked Send operation and links it to that unmet requirement. The completion constraints K retain the current permissions and task restrictions. The goal remains delivery of the same report to the same recipient.
- Retrieve a suitable experience and check its actions.
The two supplied memories are direct Send and Handoff → Deliver. In the implemented UBC score, the handoff fragment receives four points for matching the unmet delivery effect and six for its explicit prohibition-repair tag. Direct Send receives four points but is excluded by admission because the action is denied.
Handoff and postdoc delivery remain permitted. Handoff requires a prepared report, so prerequisite closure includes Analyze and Summarize. The retrieved experience therefore supplies an alternative route whose prerequisites can be satisfied by the valid preparation steps.
- Construct and validate the repaired plan.
Starting from the current initial state, the planner returns:
Analyze → Summarize → Handoff → Deliver.
The agent prepares R and hands it to the postdoc. The final delivery action is assigned to the postdoc under the postdoc’s own permission. Every transition passes validation, and the simulated final state satisfies the original goal. This is a joint plan; a human delivery step still requires execution and confirmation.
Different changes produce different reuse decisions in this same task:
- Change only the agent’s role label from “analyst” to “project assistant,” leaving relevant guards unchanged: the original workflow remains valid.
- Revoke only direct sending: preserve preparation and use the postdoc route.
- Revoke both the agent’s and postdoc’s sending permissions: preparation remains valid, but neither delivery route completes the goal.
- Revoke direct sending and handoff: the postdoc route is unavailable.
- Forbid Dr. Li as a recipient: delegation cannot bypass the recipient restriction.
- Revoke access to raw dataset D: Analyze itself becomes blocked, even though the file remains locally accessible.
Together with the unchanged case and a capability-removal case, these form the eight code-checked cases in Appendix E. They show why the “degree” of change is determined by the affected operations and remaining routes, rather than a single role-demotion scale.
This walkthrough uses authored memories and supplied permission states to make the planner’s decisions reproducible. In the complete system, Normative Grounding first checks the PI’s instruction and its supporting authority before passing the changed permissions to the planner. Thank you also for pointing out that the revised manuscript was missing from the forum; the revised document includes this full walkthrough alongside the new experimental results.
Response to Reviewer 7yC9
Our motivation includes an assistant using a locally accessible paid API without authorization: access did not grant permission to spend. The paper's report example shows the loop: after direct Send is revoked, retain valid analysis, seek an authorized Handoff for delivery, or leave delivery pending. We clarified the definitions and tested matched controls.
Q1/W1: role, authority and change. A role is an explicit participant attribute or scoped responsibility/relation. Current position includes identity, roles, directed relations and task context. Authority is separate: a supported permission for a particular actor, operation, object and purpose at the current time. A title or a successful past action does not grant it; feedback can update experience without changing authority. Changes may add/remove roles, grant/revoke/expire permissions or invalidate supporting relations. Changed supports, blocked historical actions and unmet goals are different quantities. An old step is reusable when its state and normative guards hold. Revoking direct send may preserve analysis and summarization while requiring an authorized handoff; a blocked step does not prove the goal impossible.
Q2: what is experience? The main planner uses supplied structured fragments: action identifiers or strategy text, requirements/effects, repair tags, applicability metadata and provenance. Richer directed A records include participants, direction, roles and context, but the original B result does not show that these fields were learned or caused its saving. We now distinguish supplied fragments, learned memories and runtime role filtering explicitly.
Q3: learned memory and inferred authority. The original 28/31 integration used composite routes; its public inputs have no primitive catalogs and its 60 saved learned records have empty action arrays. A new separately specified single-template study shares actual A/C outputs across all six primitive planners: 48 grounded memories, 96/96 authored permission states and 24 instances. At budgets 500/10,000, all methods find all 12 feasible six-step joint plans, leave 12 infeasible goals pending and return no illegal plan. UBC equals U+Admission on every plan/stage trace; Strong Repair costs 912 versus 1,056 total attempts. This establishes interfaces, not added blocker value or a causal benefit of learning memory. Human actions are planned, not executed. All 48 successful responses report deepseek-flash (requested deepseek-chat); 48 separately retained socket failures give 96 physical attempts. Successful calls used 94,355 tokens and 91.60 summed latency seconds, not batch wall time.
W2: robust advantage? We do not claim one. The source feasible saving is 48.2%, but overall cost is 2.72 times Strong Repair's. The original matched U+Admission control gives identical completion and +27.30 UBC attempts (world-bootstrap interval [7.76,52.70]). On 64 new seeds/192 original-axis cases, both complete 96 versus Strong Repair's 100; UBC adds 19.35 attempts [2.27,47.50]. The positive unmet-requirement result also replicates: U and GoalOnly have identical outcomes (88/192 completions), while U saves 869.61 attempts; the U-minus-GoalOnly interval is [−1141.50,−627.65]. This supports unmet-effect retrieval within the family, separately from the absent added blocker benefit.
Q4: restriction and stopping. The sweep uses 64 worlds, two nested-removal axes and five levels. Network permission removal includes other actors; agent capability removal preserves request capability and specialist authority. Goals, histories, effects and initial progress stay fixed. At permission levels 0/.25/.5/.75/1, reference-feasible counts are 64/58/43/18/0, while UBC completes 58/51/40/18/0 at budget 10,000. Removing all removable task-operation capabilities leaves all 64 feasible via specialists, but UBC solves 53. Across 640 cases, UBC/U+Admission tie at 441 completions and 4,071.30 mean attempts; Strong Repair solves 449 at 5,158.89. Lower cost with lower completion is not an equal-outcome gain. No early-stop policy was fitted; budget failure is not an impossibility certificate.
Q5/W3: held-out scope. We distinguish inspected source data, same-data audits, new seeds in an inspected generator, and external/new-family evaluation. There is no evidence in the last category. The fresh run freezes the planner before new-seed performance; all 14,976 outputs pass independent replay. This tests the implementation within its generator, not realistic organizations, and does not make earlier revisions held-out. Later graph-planning tasks do not replace original-planner evidence. Repeated levels are clustered by their 64 worlds, not treated as independent samples.