acceptodds
Under review as a conference paper at ICLR 2027

Authenticated Late-Bound Evaluation of Persistent Agent Artifacts

Abstract

Persistent Agent artifacts can parse after an SDK upgrade yet change a later consumer's result. We study authenticated late-bound evaluation: an origin authenticates a complete historical view; a receiver binds a later consumer, context and interpreter, then compares historical and current results. Omitting consumer-observable distinctions precludes exact evaluation. We implement signed snapshots, proof-backed consumer execution (PBCE), and authenticated keyed events. Closed-IR simulation and authenticated openings yield conditional, IR-relative soundness; materialization and current execution remain trusted. Unsupported semantics abstain. Source checkers cover five AutoGen schemas and a corrected LangGraph reducer under explicit projection/conversion premises. An unfiltered AutoGen inventory certifies only 5/22 workloads. An empirical AppWorld confirmation covers 18/36 tasks with a frozen replay profile; retrospective repairs reach 36/36, but the Agent solves only 15/36. These empirical extensions remain outside the closed-IR theorem and do not establish independent transfer. Full snapshots favor small dense objects, and keyed events can be cheaper than PBCE. On 35 restored development checkpoints, all 2100 local receiver queries agree. Including selective sender work but granting prepacked full archives, task-median first-handoff time is 0.157 a compact full snapshot and 0.114 a root-authenticated lazy full ZIP; warm-query time is nearly equal. These are local handoff costs, not whole-Agent speedup or superiority over all lazy stores. Our contribution is a scoped evaluation lifecycle and measured coverage–cost frontier, not arbitrary-code certification, generic-store superiority or ecosystem-wide safety.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, persistent artifacts, authenticated evaluation, compatibility, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RtRkOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to ensure whether saved state produced by an agent framework still yields the same result after the framework is upgraded. The authors call this setting "authenticated late-bound evaluation" which aims to certify that the old and new object produce the same result without running the old framework. The key idea is to freeze the old meaning when saving by signing the full snapshot, using a Merkle set of facts, or a hash-chained event log, and later comparing the result on the old and new objects. The authors implement their system and evaluate its correctness and cost on several benchmarks and systems.

Critical strengths

(S1) The paper presents a valid concern with respect to how to upgrade agent frameworks safely.

Critical weaknesses

(W1) The paper writing is very hard to follow. Sentences are compressed to the limit (e.g., "Our contribution is not that primitive, but the late-authorized, same-consumer replay protocol, closed IR boundary, and measured choice against the complete snapshot."). The text is overly hedged and the abstract spends as much space saying what the results do not show than what they show. The whole text reads like a research log. Results are spraed everywhere, making it hard to identify the main claims and the evidence for it. In the current form of the paper, I am afraid I cannot assess the technical claims even with reasonable effort. The rest of the points below are based on my best effort to udnerstand the paper and its claims.

(W2) The novelty of the solution is low. It consists of an assembly of known techniques and results (consumer-driven contract testing, differential upgrade testing, etc.) with little justification for why agent frameworks need a new mechanism rather than traditional upgrade testing.

(W3) The threat model appears quite far-fetched. Although SDK upgrades can change the meaning of save state, the setting that the authors consider where the source runtime is not available and the receiver is limited to seeing consumer-read fields, does not appear to be motivated by any realistic deployment. It seems there are simpler alternatives, such as keeping the old SDK in a container running the consumer under both versions and signing the verdict, which would disclose less to the auditor than PBCE.

(W4) Beyond being hard to parse, the evaluation coverage appears to be quite limited and does not show significant benefits to the proposed approach or its actual correctness. The GitOps result seems to be guaranteed by construction since the baselines omit fields they necessarily lose against the full snapshot and PBCE. I could not make sense of the AppWorld results which seem to present a number of different configurations that were not clearly explained or motivated. The AutoGen results seem to indicate that the proposed solution does not certify in most real cases.

(Minor) This sentence does not make sense and appears to be a leftover from editing or experiment iteration: "A concrete defect signed the tree digest but not |As|; Equation 7 now binds both."

Questions for authors

Please see critical weaknesses for more details.

(Q1) Why does this problem need a new mechanism?

(Q2) Would alternatives such as the ones discussed in W3 work?

(Q3) Please explain concisely the evaluation results and benefits of your approach.

2 comments
AuthorsOpenReview ↗

Author response: lifecycle alternatives and measured benefits

Thank you. Your questions identify the key comparison: what does this protocol provide beyond keeping an old runtime and signing a verdict? It provides receiver-checkable evaluation of a later-selected admitted consumer against an authenticated historical view, without retaining the old runtime. The evidence for selective access is measured against complete snapshots, which provide the same result fidelity.

W2/W3, Q1/Q2: alternatives and the contribution.

The alternatives correspond to different lifecycle choices:

  • A verdict signed at state creation answers the query specified at creation.
  • A trusted retained-runtime service can answer later queries, provided that runtime and service remain available. A signed verdict from such a service can disclose less than PBCE.
  • A complete signed snapshot supports later admitted evaluation without the old runtime, while transferring the complete authenticated representation.
  • PBCE supports that later evaluation by opening and checking the facts the consumer reads, subject to evidence authorization and budget.

Thus, keeping an old runtime is a valid alternative, but it does not make the offline historical-evaluation setting equivalent to ordinary online differential testing. The paper makes these lifecycle and trust choices explicit and compares complete and selective evidence under the same consumer-relative result criterion. Established authenticated-data primitives implement integrity; the protocol additionally specifies consumer binding, semantic admission, evidence authorization, and abstention. Source materialization and current-side execution remain trusted. The cross-tenant example motivates selective disclosure; it is not presented as a measured deployment.

W4, Q3: what the experiments establish.

The strongest benefits are not an accuracy advantage over complete snapshots. They are a task-relevant compatibility distinction and selective first-handoff savings:

  • In the 42-pair AutoGen saved-state continuation study, the sparse consumer matches all pair-level task outcomes, all 22 compatible pairs resume exactly, and complete-state equality rejects two task-compatible pairs. This demonstrates a consequence of consumer-relative observation in that workflow.
  • On 35 restored development checkpoints, first-handoff time has task-median ratios of 0.157 to compact full snapshots and 0.114 to root-authenticated lazy full ZIP. All 2,100 queries agree. Selective sender work is included, while complete packages are prebuilt. Warm queries are nearly tied, and this is local handoff rather than whole-task acceleration.
  • The 320-case omission challenge tests result fidelity: both full snapshots and PBCE achieve 320/320. Its role is to expose incomplete representations, not to estimate natural upgrade prevalence or establish a unique PBCE accuracy advantage.

Coverage is reported separately. In the 22-workload AutoGen inventory, 12 workloads exceed the expression profile and five exceed the checked bridge domain, leaving five certified workloads. In the second AppWorld cohort, frozen replay covers 18/36; post-outcome repairs cover 36/36; Agent task success is 15/36. These are different endpoints, and repaired replay remains development evidence. Unsupported cases are retained rather than silently removed from the population.

The workload-dependent backend results also serve a practical purpose: full snapshots are preferable for small dense objects, and keyed events can be cheaper than PBCE. A system can therefore use the complete snapshot as its default and selective evidence where the measured access pattern justifies it.

W1 and the editing issue.

We will consolidate these results and the alternatives comparison in the main text. The Equation 7 sentence should state the bound commitment fields directly; its repair-history wording is an editing error. This correction does not alter the binding definition.

We ask you to reassess the novelty and practical-benefit concerns in terms of this specific capability and its evidence: late-bound, receiver-checkable historical evaluation with explicit semantic admission and workload-dependent selective access. The case for the paper rests on that capability, the continuation result, and the comparison against complete archives, rather than on claiming a new cryptographic primitive or a replacement for all upgrade testing.

Thank you for the response. You argue the novelty is in evaluating saved state without the old runtime, but I would like to see a real deployment where that setting arises. The response also does not address how few real cases the approach certifies. I will keep my score.

Reviewer_3hQFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • Since agent states can still be parsed after a software upgrade while producing different behavior, this work looks at how to assess compatibility for later consumer.
  • This work sets up an evaluation given historical state and executes a restricted consumer on historical and current results.
Critical strengths
  • The paper studies how software upgrades can change the behavior of saved agent state, a potentially important issue for real-world agent deployment.
Critical weaknesses
  • The clarity and organization is poor. The paper introduces many terms and versions of the method without clearly explaining how they fit together.
  • The paper lacks engagement with prior work, making it difficult to assess how to situate the work in the broader literature and understand if they are making a sufficiently interesting / novel contribution.
  • Limited results are provided and only on a limited set of workloads: only 1 results table in the main text. From it, one cannot easily understand what is the practical benefit of the method.
  • It is not clear how this work benefits the ICRL audience, as the central contribution concerns authenticated data management and systems verification.
Questions for authors

Please address weaknesses above.

1 comment
AuthorsOpenReview ↗

Author response: contribution, evidence, and relevance to Agent evaluation

Thank you. The paper's central contribution is authenticated evaluation of persistent Agent state by consumers selected after state creation. Its three backends implement one procedure: authenticate the historical view, bind the later consumer and input, evaluate historical and current views, and compare their results or abstain. Full snapshots, proof-backed facts, and keyed events provide different cost and disclosure trade-offs for that same procedure.

Contribution and relationship to prior work.

Signatures establish provenance; authenticated data structures establish the integrity of opened facts; translation checking connects supported source consumers to the evaluator. Our contribution is the protocol and semantic contract that connect these guarantees for late-bound historical evaluation. In particular, the system separates authenticated facts from fidelity to the native consumer, and separates permission to issue a logical query from permission to expose the evidence needed to verify it. These distinctions determine whether a compatibility certificate can actually be issued. They are part of the method, rather than additional names for storage backends.

The relevant comparison is therefore not whether we invented signatures or authenticated storage. It is what historical evaluation is supported after the source runtime disappears, under which consumer, evidence, and trust assumptions, and at what cost. The complete signed snapshot is the reference implementation of the same guarantee. PBCE's role is selective access, not additional accuracy over that reference.

The empirical findings supporting this contribution are:

  • Fidelity: full snapshots and PBCE both match 320/320 constructed challenge decisions. This validates the complete-evidence design against omission errors.
  • Task consequence: in the 42-pair AutoGen continuation study, all 22 compatible pairs resume exactly, while complete-state equality rejects two task-compatible pairs.
  • Cost: on 35 restored development checkpoints, first-handoff time is 0.157 times compact full snapshots and 0.114 times root-authenticated lazy full ZIP, with all 2,100 queries agreeing. This measures local handoff, not Agent task-solving speed.
  • Coverage: the unfiltered AutoGen inventory certifies 5/22 workloads. In the second AppWorld cohort, frozen replay covers 18/36; post-outcome repairs cover 36/36, while Agent task success is 15/36. The repaired result is development evidence, not independent transfer.

This separates three questions that the current presentation makes unnecessarily difficult to track: whether an admitted evaluation is faithful, how often a consumer can be admitted, and when selective evidence saves resources. We will organize the main results around these questions and consolidate the existing comparisons, rather than narrate successive implementations.

Relevance to ICLR.

Persistent Agent evaluation requires more than loading a checkpoint: a later evaluator must know whether the restored state preserves the observation relevant to its task. Our AutoGen continuation and AppWorld replay studies instantiate this problem in Agent frameworks and evaluation workloads. The contribution is a mechanism for checking such observations across state lifecycles, with measured fidelity, coverage, and cost. It is an Agent evaluation infrastructure contribution, rather than a learning algorithm.

We ask you to reconsider the contribution and informativeness assessments on this clarified basis: the paper provides an explicit historical-evaluation contract, implementations with different resource trade-offs, and task-level and cost evidence beyond the constructed challenge. The presentation and prior-work comparison need consolidation; the results above are already present in the manuscript, not proposed future evidence.

Reviewer_CNkYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper highlights that software updates to an agent SDK may cause previously saved artifacts to produce different results when executed with an upgraded agent version. To address this issue, the authors propose a novel framework backed by cryptographic guarantees to verify the consistency of consumer-facing outputs across different versions of an agent system. The framework is evaluated across multiple agents and workloads, yielding mixed results.

Critical strengths
  1. The paper tackles a relevant problem: saved agent states may lead to different outcomes after an SDK update, even when they can still be loaded.

  2. Using cryptographic proofs to access only the required parts of historical agent states is an interesting approach.

  3. The paper provides a formal analysis of the proposed framework and clearly states the assumptions under which its guarantees hold.

  4. The authors evaluate the approach on several agent frameworks and workloads. I appreciate that they also report cases where the method does not work well.

Critical weaknesses
  1. Unclear practical significance. How serious and prevalent is this problem in practice? While the paper provides some examples, it remains unclear how frequently such inconsistencies occur in real-world agent applications and how significantly they affect system behavior.

  2. Lack of simpler baselines. The paper could benefit from comparisons with simpler alternatives, such as migrating saved states when updating the SDK or retaining older SDK versions for compatibility. It is unclear why these approaches would be insufficient and when the proposed framework would be preferable.

  3. Limited certification coverage. For AutoGen, only 5 out of 22 workloads can be certified, raising concerns about the framework's applicability to more complex real-world workflows. The paper should explain the reasons for this low success rate and whether it reflects limitations of the framework or the relevance of the underlying problem.

  4. Unclear efficiency benefits. Full snapshots outperform the proposed approach in some cases, and it remains unclear whether the reported savings translate into meaningful improvements in end-to-end agent execution.

Questions for authors

Please see the critical weaknesses. I would like to see the authors' rebuttal to all 4 points above.

1 comment
AuthorsOpenReview ↗

Author response: practical utility, alternatives, coverage, and cost

Thank you for identifying practical value, alternatives, coverage, and efficiency as the deciding issues. The paper provides evidence of both task-relevant compatibility distinctions and substantial first-handoff savings. We clarify below which results establish these benefits and how they relate to the alternatives you suggest.

  1. Task relevance: consumer choice changes a real continuation decision.

The 42-pair AutoGen release study exercises actual saved-state continuation. The sparse consumer matches all pair-level task outcomes, all 22 compatible pairs resume exactly, and complete-state equality rejects two task-compatible pairs. Thus, consumer-relative compatibility can preserve a valid resumption that whole-state equality would block. This is a concrete behavioral consequence, separate from the 320 constructed omission tests. The continuation evidence concerns a fixed deterministic workflow; production failure frequency remains unmeasured.

  1. Alternatives: the distinction is when the consumer becomes known and which environment must survive.

Retaining both SDKs is a valid solution when both runtimes remain available. Our setting instead authenticates the historical view while the source environment is available and evaluates an admitted consumer selected later, after that environment is unavailable. A verdict signed at creation cannot answer a consumer query that has not yet been specified. A retained online evaluator can, but requires continued operation and trust in that service. Successful migration likewise needs a consumer-level check to establish preservation of the later observation.

The paper includes the strongest simple offline baseline: a complete signed snapshot. It is equally correct on all 320 challenge cases. PBCE adds selective evidence access, with its value measured against that complete baseline rather than inferred from comparisons with lossy projections. This makes the contribution an explicit lifecycle capability and a measured cost trade-off, not a claim that conventional upgrade testing is inadequate everywhere.

  1. Coverage: 5/22 is the certified implementation's admission rate, not a 17/22 error rate.

Section 5.2 includes all 11 pinned termination classes at one- and two-call horizons. All 44 native executions complete. Twelve workloads need semantics outside the implemented expression profile; another five fall outside the checked fresh-instance bridge domain. The remaining five certify correctly. These exclusions constrain current applicability and remain in the denominator. They distinguish unsupported consumers from incorrect compatibility verdicts, which is essential for evaluating a selective verifier. The separate AppWorld results test broader empirical replay correspondence, without extending the closed-IR theorem to that adapter.

  1. Efficiency: a measured first-handoff benefit against complete baselines.

On 35 restored development checkpoints, task-median first-handoff time is 0.157 times compact full-snapshot time and 0.114 times root-authenticated lazy full-ZIP time, with agreement on all 2,100 queries (Section 5.3). Selective sender work is charged while full packages are prebuilt. These comparisons establish a local first-handoff benefit even against prepared complete archives. Warm queries are nearly tied; small dense objects favor snapshots (1.83 KB versus 21.05 KB). They identify the workload regime for selective evidence rather than implying whole-Agent acceleration.

We will place the continuation result, admission breakdown, and first-handoff comparison together in the main presentation. In light of the concrete resumption distinction and the measured benefit against complete snapshots, we ask you to reconsider the assessment of practical utility. These results support a selective historical-evaluation system with explicit applicability conditions, even though they do not estimate production prevalence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.