- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how to ensure whether saved state produced by an agent framework still yields the same result after the framework is upgraded. The authors call this setting "authenticated late-bound evaluation" which aims to certify that the old and new object produce the same result without running the old framework. The key idea is to freeze the old meaning when saving by signing the full snapshot, using a Merkle set of facts, or a hash-chained event log, and later comparing the result on the old and new objects. The authors implement their system and evaluate its correctness and cost on several benchmarks and systems.
Critical strengths
(S1) The paper presents a valid concern with respect to how to upgrade agent frameworks safely.
Critical weaknesses
(W1) The paper writing is very hard to follow. Sentences are compressed to the limit (e.g., "Our contribution is not that primitive, but the late-authorized, same-consumer replay protocol, closed IR boundary, and measured choice against the complete snapshot."). The text is overly hedged and the abstract spends as much space saying what the results do not show than what they show. The whole text reads like a research log. Results are spraed everywhere, making it hard to identify the main claims and the evidence for it. In the current form of the paper, I am afraid I cannot assess the technical claims even with reasonable effort. The rest of the points below are based on my best effort to udnerstand the paper and its claims.
(W2) The novelty of the solution is low. It consists of an assembly of known techniques and results (consumer-driven contract testing, differential upgrade testing, etc.) with little justification for why agent frameworks need a new mechanism rather than traditional upgrade testing.
(W3) The threat model appears quite far-fetched. Although SDK upgrades can change the meaning of save state, the setting that the authors consider where the source runtime is not available and the receiver is limited to seeing consumer-read fields, does not appear to be motivated by any realistic deployment. It seems there are simpler alternatives, such as keeping the old SDK in a container running the consumer under both versions and signing the verdict, which would disclose less to the auditor than PBCE.
(W4) Beyond being hard to parse, the evaluation coverage appears to be quite limited and does not show significant benefits to the proposed approach or its actual correctness. The GitOps result seems to be guaranteed by construction since the baselines omit fields they necessarily lose against the full snapshot and PBCE. I could not make sense of the AppWorld results which seem to present a number of different configurations that were not clearly explained or motivated. The AutoGen results seem to indicate that the proposed solution does not certify in most real cases.
(Minor) This sentence does not make sense and appears to be a leftover from editing or experiment iteration: "A concrete defect signed the tree digest but not |As|; Equation 7 now binds both."
Questions for authors
Please see critical weaknesses for more details.
(Q1) Why does this problem need a new mechanism?
(Q2) Would alternatives such as the ones discussed in W3 work?
(Q3) Please explain concisely the evaluation results and benefits of your approach.
2 comments
Thank you for the response. You argue the novelty is in evaluating saved state without the old runtime, but I would like to see a real deployment where that setting arises. The response also does not address how few real cases the approach certifies. I will keep my score.
Author response: lifecycle alternatives and measured benefits
Thank you. Your questions identify the key comparison: what does this protocol provide beyond keeping an old runtime and signing a verdict? It provides receiver-checkable evaluation of a later-selected admitted consumer against an authenticated historical view, without retaining the old runtime. The evidence for selective access is measured against complete snapshots, which provide the same result fidelity.
W2/W3, Q1/Q2: alternatives and the contribution.
The alternatives correspond to different lifecycle choices:
Thus, keeping an old runtime is a valid alternative, but it does not make the offline historical-evaluation setting equivalent to ordinary online differential testing. The paper makes these lifecycle and trust choices explicit and compares complete and selective evidence under the same consumer-relative result criterion. Established authenticated-data primitives implement integrity; the protocol additionally specifies consumer binding, semantic admission, evidence authorization, and abstention. Source materialization and current-side execution remain trusted. The cross-tenant example motivates selective disclosure; it is not presented as a measured deployment.
W4, Q3: what the experiments establish.
The strongest benefits are not an accuracy advantage over complete snapshots. They are a task-relevant compatibility distinction and selective first-handoff savings:
Coverage is reported separately. In the 22-workload AutoGen inventory, 12 workloads exceed the expression profile and five exceed the checked bridge domain, leaving five certified workloads. In the second AppWorld cohort, frozen replay covers 18/36; post-outcome repairs cover 36/36; Agent task success is 15/36. These are different endpoints, and repaired replay remains development evidence. Unsupported cases are retained rather than silently removed from the population.
The workload-dependent backend results also serve a practical purpose: full snapshots are preferable for small dense objects, and keyed events can be cheaper than PBCE. A system can therefore use the complete snapshot as its default and selective evidence where the measured access pattern justifies it.
W1 and the editing issue.
We will consolidate these results and the alternatives comparison in the main text. The Equation 7 sentence should state the bound commitment fields directly; its repair-history wording is an editing error. This correction does not alter the binding definition.
We ask you to reassess the novelty and practical-benefit concerns in terms of this specific capability and its evidence: late-bound, receiver-checkable historical evaluation with explicit semantic admission and workload-dependent selective access. The case for the paper rests on that capability, the continuation result, and the comparison against complete archives, rather than on claiming a new cryptographic primitive or a replacement for all upgrade testing.