- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PLANFENCE addresses a failure mode in distributed LLM-agent systems where an executor has current state but inherits a plan derived from older inputs. It stores exact parent links with derived artifacts, reconstructs an action-specific dependency frontier, validates recorded versions against owner heads, and replans once on mismatch. The paper evaluates this through evidence ablations, live five-agent workflows, and controlled replay under different update rates and network conditions.
Critical strengths
1. The distinction between observation currency and derivation currency is crisp and practically important: current local state does not guarantee that an inherited plan was derived from current inputs.
2. The evidence ablation is particularly strong. Holding the validator fixed while supplying inherited versions changes handoff-conflict detection from 0/30 to 30/30, cleanly isolating the missing-evidence problem. The paper is also unusually transparent about negative and regime-dependent results.
Critical weaknesses
1. The strongest baseline is missing.
Table C.2 shows that refreshing the executor's inputs before it generates yields the correct primary action in 28/30 tasks. The workflow still fails (0/30) because the downstream auditor reads stale state. The authors note in Appendix C.5 that this control does not test refreshing every role before generation. That refresh-all baseline is the natural competitor and should be run, with both task success and coordination cost reported. I recognize that two of the 30 executors kept the stale choice despite fresh inputs, so refresh alone depends on the model while PLANFENCE's check is deterministic. The open question is how much the deterministic check adds over refresh-all in practice. Without this comparison, it is unclear whether PLANFENCE provides a distinct correctness benefit or mainly a more selective way to enforce current inputs.
2. The contribution relative to supplied read sets is narrower than the framing suggests.
In Table 1(a), the S-Bus validator given inherited versions detects all 30 handoff conflicts and completes 30/30; PLANFENCE reaches 29/30. The paper itself says the two are equally safe in this setting and differ in what must cross the handoff and where validation runs. In both designs the application must record the inputs: S-Bus needs a declared read set, and PLANFENCE needs parent links and a declared scope D(a). Appendix C.4 shows that model-generated plans did not carry such parent links on their own. Proposition 1 follows from the immutability of records. The contribution is therefore artifact-carried version validation without a shared registry, which is useful but should be stated at that scope.
Questions for authors
Q1 (experiment, limited scope): Please add the refresh-all-roles baseline to the live study and report task success, invalid primary actions, and coordination cost next to PLANFENCE and centralized lineage.
Q2 (clarification): Compared with S-Bus using declared read sets, what must the application supply in each design, and what does PLANFENCE enable beyond removing the shared registry?
Suggestion (does not affect my rating): The paper could state explicitly that owner heads are taken as authoritative and that their currency relative to the external world is out of scope. Related arXiv work on evidence freshness at commitment time, e.g., TRW (arXiv:2607.21910), may be a useful pointer.
2 comments
weakness 2 / question 2
The contribution relative to supplied read sets is narrower than the framing suggests. [...] Compared with S-Bus using declared read sets, what must the application supply in each design, and what does PLANFENCE enable beyond removing the shared registry?
Beyond removing the registry, PlanFence lets any later holder validate a plan derived through several hops. The plan writer's read set alone does not. We measured this in a new experiment on the real systems, with plans derived in one to three hops on one or two branches (30 templates per cell). When S-Bus receives the read set of the plan's writer, it catches every stale plan that reads the requirement directly, and 0 of 30 once one intermediate record lies between them. Forwarding the full read set restores detection. However, every intermediate writer must then forward it, and the stored evidence grows from 43 to 199 bytes with hops and branches. PlanFence catches every stale plan in these cells by following parent links, and it blocks all 60 plans that mix two versions of one input. In a second experiment, we read the scope from the plan's recorded parents, with no separate declaration, and none of 240 actions was stale. Both experiments are detailed in our response to Reviewer tJ8m.
The two designs ask the application for the following.
| S-Bus with a declared read set | PlanFence | |
|---|---|---|
| At each write | nothing | the IDs of the records the writer was given (known locally) |
| At each handoff | the creator's read set, and for a multi-hop plan the union of all upstream read sets | the plan record |
| At the protected action | the read set | the scope , declared or read from the plan's parents |
| To follow the evidence | nothing beyond the read set | the records behind the plan, fetched if not held (3.1 to 13.2 KB per handoff in our experiment) |
| Validation target | one logical registry | the owners of the declared keys, queried concurrently |
Neither design relies on the model for this evidence. The examined model-generated plans did not attach the memory record as a parent on their own (Appendix C.4), so our adapter records the exact record supplied to each role call. The designs differ after the write. A PlanFence writer stores only IDs it already knows, while a read set must be assembled and carried across every handoff. We will state the contribution at the reviewer's scope, together with the multi-hop result that shows what it adds (Section 2 and the first contribution), and present Proposition 1 as a lemma.
Our contribution is artifact-carried version validation. Each derived record keeps exact parent links. Any later
holder of a plan can follow them back to the plan's recorded inputs and check those versions against their owners,
without the creator's session or a shared registry. A session-based validator matches this safety only when the full
read set behind the plan reaches it. For a multi-hop plan, every intermediate writer must then forward its read set
(Appendix G.2).
suggestion
State explicitly that owner heads are taken as authoritative and that their currency relative to the external world is out of scope. [...]
Thank you. We will add to Section 3 and Appendix E.3: "Owner heads are authoritative by assumption. Whether an owner's head reflects the external world is out of scope." We will also cite TRW (Chang, arXiv:2607.21910) as related work on keeping evidence current at commitment time.
Please let us know if this response sufficiently addresses your concerns or if further clarification would be helpful.
We thank the reviewer for the careful reading, for finding the observation/derivation distinction practically important, and for recognizing the evidence ablation and our reporting of negative results.
We ran the refresh-all-roles baseline. Before each role generates, it fetches the owner heads of the role's declared inputs and installs any changed record. It makes no derivation check. Its tool schema allows both the inherited action and the fresh alternatives, as in Table C.2. We compare it with PlanFence, centralized lineage, owner-head freshness and the executor-only refresh of Table C.2, using the same workflows, model, decoding and seeds. Each arm runs 30 tasks per condition, after a smoke stage on a separate seed. We test two revision timings. In the first, the revision lands between planning and execution, as in the paper; we run it over loopback and over the emulated AT&T link. In the second, it lands while the executor is generating, after any refresh before generation and before the action.
Each cell gives successful tasks / stale actions executed, out of 30 tasks. Success is graded after the workflow's repair step; a stale action counts even when it is repaired later. In 17 of the 450 tasks (0 to 4 per cell, under every method), a call between agent processes timed out, and these tasks count as failures.
†Our live centralized-lineage executor compares the plan with the requirement version it read before generation, so it misses an update during generation. A registry check at the action would reject these actions, as in replay. This gap lies in our harness, not in centralized lineage.
Medians over the tasks that finished; model calls are means. Messages and KiB count what crosses the link: owner checks, record fetches and replication for the replicated designs, and every read and write for centralized lineage.
The deterministic check adds what refresh-all-roles cannot guarantee. When the revision lands before execution, refresh-all-roles depends on the model adapting to the refreshed input. It did so in 53 of the 59 tasks that finished, but in the other 6 the executor acted against the revised requirement it had been given. When the revision lands during generation, no refresh before generation can see it, and refresh-all-roles executed the stale action in all 27 tasks that reached the action. PlanFence executed no stale action in either timing, and it replanned in all 27 tasks that finished during generation. Final task success is close (58 of 60 each before execution, 26 against 27 during generation) because the workflow's repair step fixes stale actions afterwards. The difference is whether a stale action reaches the outside world. Refresh-all-roles executed 33 across the three conditions, and PlanFence none. A shipment or a deployment is not undone by a later repair, so this is a distinct correctness benefit, not a more selective refresh.
PlanFence is also lighter on the link. Refresh-all-roles queries the owners before every role call (7 head checks and 2 record fetches per task). PlanFence makes 2 head checks, one at the action and one after the replan, and 3 record fetches. Per task, it sends 13 messages and 18.2 KiB against 17 messages and 21.1 KiB, with a tie on coordination time (0.61 s against 0.65 s). Despite one extra model call, the replan that produces the correct plan, its median task time ties refresh-all-roles by the paper's 10% rule (6.9 s against 6.6 s).