acceptodds
Under review as a conference paper at ICLR 2027

What Can We Infer from Agent Traces? Observation–Claim Identifiability for Action-Routing Evaluation

Abstract

Agent evaluation requires checking not only recorded outputs, but also whether their interpretation preserves the intended question and whether the evidence determines its answer. We introduce an executable source-to-claim contract that distinguishes a changed question, insufficient evidence, and a conclusion requiring revalidation after its supporting inputs change. The workflow checks proposed representation corrections against the original question, acquires evidence from a declared catalogue, and revalidates the resulting answer. We evaluate it on source-linked queries using retained navigation, text-interaction, and hosted tool-use traces, with recorder-conformance checks on 10,800 execution records. A controlled asynchronous execution case shows that identical client-timeout projections can conceal different later transaction outcomes; acquiring transaction evidence resolves the ambiguity. On the tested contracts, direct SMT and finite enumeration reach the same conditional conclusions as the integrated workflow. In a controlled three-layout study of 36 retained records, selective access reads fewer, equal, or more bytes than direct access to the required evidence; a minimal directly available field can be cheaper than a selectively acquired ledger. The contribution is a verifiable path from an evaluation question to evidence and a qualified answer. Guarantees remain conditional on the supplied semantics, admissible execution space, and evidence catalogue.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, trace auditing, claim identifiability, formal verification, evidence acquisition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ko8TOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper identifies the issue of agent traces and whether their projection is information preserving given a question, and whether questions (predicates) over the native space have an equivalent answer in the abstract space (the projection of the trace). The paper presents a workflow that can examine whether a proposed trace representation, and changes to that representation, give the same answer in the projection space as the native space. A tool built using Z3 implements the workflow, alongside an SMT and finite enumeration baseline. The integrated workflows is neither more correct or faster: "Same answers; no measured detection or speed advantage".

This paper was incredibly difficult to read. It is full of esoteric language like 'source to claim interface' that is not given a clear definition. Can this not be defined by typed signatures of the interface, well explicit semantics?

A complete worked example would help communicate what is being proposed. The example in Table 1 is too brief and does not demonstrate what the user inputs and what the system outputs.

Critical strengths

None

Critical weaknesses
  • This paper is not sufficiently relevant to the venue. At its core it is about meta-theoretic properties about the mapping from a concrete execution trace to an abstraction representation (the telemetry logs). Agent interactions and AI are only relevant in so far as they use traces/telemetry, but this applies equally to any software domain that uses logging. For this to be more relevant I would expect to see an explanation about how it addresses unique challenges to agents.

  • The demonstrated novelty is limited, hindered by an opaque presentation. Using formal methods to make statements about telemetry implementations is not new. The proposed workflow (Algorithm 1) is not specified clearly enough to make it possible to reimplement from the paper and therefore gauge its novelty. Other details of the implementation are scattered through the paper in dense language. For example: "A certificate must have a concrete acquisition path". Where are certificates or acquisition paths precisely defined? How would a reader know that their system satisfies this property? Furthermore, the workflow itself is not shown to be better than baselines on key metrics such as speed and correctness (see "Same answers; no measured detection or speed advantage".)

  • The system relies on the user writing the predicates in formal logic, providing a bounded model of the real system in a Z3-compatible language, and a mapping of the model into the abstract domain. This is a significant barrier to entry and the paper does not present evidence that such a tool could be practically adopted. There is no evaluation that demonstrates that the approach can be used to find unknown defects in existing systems, the effort required to model real agentic systems, or an evaluation of how expressive the model is of real systems. Therefore, there is little evidence that the proposed workflow offers practical benefit to users.

Questions for authors

None

1 comment

Thank you for your clear suggestions. We've brought the definitions together so the workflow is easier to follow. Section 2 gives the typed interface and checks admissibility, target preservation and decoder consistency. Section 3 and Algorithm 1 separate solver queries, receipts linking decoded fields to source bytes, and receipts recording the conditions for result reuse.

Users supply finite domains, predicates, an adapter and an evidence catalogue in JSON; expressions are compiled internally to Z3. Outputs distinguish a changed question, insufficient evidence, a conditional answer and refusal. Users must still justify the model: the tool checks consequences of declared semantics and does not automatically discover the behavior of an entire agent runtime.

The agent-evaluation motivation is concrete: proposing a tool call, dispatching it, entering its body and producing a side effect are different events. Sections 5.1-5.2 and Appendix I follow one deliberately selected retained call that was dispatched but excluded from a successful-call view. They show why absence from this view does not establish non-dispatch, what evidence resolves the question, and how an actual file change requires revalidation. This connects the general formal method to a specific agent-evaluation problem.

The contribution is this explicit, executable connection between an evaluation question and the evidence supporting its answer. The comparison shows no detection or speed advantage over managed direct SMT; human modeling costs and adoption remain unmeasured.

Reviewer_mr1MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the problem of whether existing observations can support the original research conclusions when conducting assessments based on Agent execution trajectories. The authors point out that simplification, filtering, or format conversion of execution logs may cause the evaluation process to deviate from the original research objectives, and even if the original objectives are preserved, the existing evidence may still be insufficient to determine the answer. To this end, this paper proposes an executable Source-to-Claim Contract, which integrates semantic preservation checking, evidence sufficiency judgment, evidence acquisition, and conclusion revalidation into a unified workflow. The authors leverage SMT solving and finite enumeration for formal verification, and conduct experiments in scenarios including navigation, text interaction, tool invocation, and asynchronous transactions. The experiments cover verification against 212 semantic contracts, recorder compliance checking on 10800 execution records, as well as experiments on evidence acquisition cost and revalidation after validation result invalidation. The results show that the proposed workflow can distinguish three cases of target semantic mismatch, insufficient evidence, and identifiable target under given formal premises. However, direct SMT solving and finite enumeration can also draw the same conditional conclusions, and the integrated method does not show a universal efficiency advantage.

Critical strengths

This paper focuses on a practically meaningful problem in Agent evaluation, that is, whether the information recorded in execution trajectories is sufficient to support the conclusions that researchers intend to draw. In particular, the authors explicitly distinguish the semantic preservation of research objectives from evidence sufficiency, and further consider the validity of existing conclusions after the change of evidence sources. This provides a useful analytical perspective for understanding the potential semantic deviations in Agent evaluation.

In addition, the experimental design of this paper is relatively rigorous, covering different solving backends, real execution records, controlled asynchronous execution cases, and multiple sensitivity analyses. The authors adopt a relatively fair comparison setup and proactively report the cases where simple baselines can draw the same conclusions or achieve higher efficiency, while explicitly limiting the applicable scope of experimental conclusions. These efforts enhance the technical credibility of the paper under its declared conditions.

Critical weaknesses
  1. The substantive innovation of the method is relatively limited, and the fundamental distinction relative to existing studies has not been sufficiently demonstrated. The identifiability criterion, semantic preservation verification, SMT solving, and counterexample-based evidence selection adopted in this paper are all built on existing theories and methods. The authors also acknowledge in Section 8 and Appendix F that representative prior works, including Beyond Local Accuracy and ClaimReceipt, together with related formal verification research, have covered the core components in this work. The main new contribution of this paper lies in integrating semantic checking between original objectives and abstract representations, evidence acquisition, and revalidation into a unified interface. Although this integration has certain engineering value, the paper has not sufficiently illustrated what new capabilities this framework brings at the conceptual or methodological level that cannot be obtained by reasonably combining existing technologies. Therefore, I consider that its technical contribution is more reflected in the systematic organization of existing methods, rather than in sufficiently remarkable methodological innovation.

  2. The experiments mainly verify the expected functions of the framework, which are still insufficient to prove its practical incremental value. The 212 semantic contracts in Section 4 contain a large number of controlled constructs or cases with pre-injected mismatches, and the recorder compliance experiment in Section 5 also mainly checks whether the declared transformation rules are correctly implemented. These experiments can support the conditional correctness of the framework, but the evidence regarding its practical effectiveness in real, unknown Agent evaluation problems is still limited. Meanwhile, Table 2 shows that the three backends reach consistent conclusions on all 212 contracts, and the enhanced property checker can also discover all 59 semantic mismatch cases. Section 6 further demonstrates that direct SMT solving outperforms the integrated method in all test categories, and the cost advantage of selective evidence acquisition depends on the specific storage layout. Considering that the most prominent distinction of this paper is the integrated auditing workflow, the paper has not sufficiently proved that such integration can bring significant gains in error discovery, auditing efficiency, or maintenance cost through independent real-world scenarios or direct workflow comparisons, which means its practical importance still lacks sufficiently strong supporting evidence.

Questions for authors
  1. The core contribution of this paper appears to mainly stem from the integration of existing semantic verification, identifiability analysis, evidence acquisition, and revalidation techniques. Could the authors further clarify what substantive new capabilities the unified contract proposed in this paper provides, compared with an implementation that simply combines existing SMT verification, evidence selection, and versioned evidence management? In particular, are there any representative problems that are difficult for a reasonable combination of existing methods to handle, but can be effectively solved by the approach in this paper?

  2. The current experiments mainly use the semantic contracts pre-defined by the authors and controlled execution cases designed in advance. Could the authors provide real-world Agent evaluation cases that are independent of the framework development process, to demonstrate that the proposed method can discover semantic issues that are actually missed by the original evaluation pipeline? Alternatively, could the authors conduct more direct comparative experiments to quantify the incremental value of the integrated workflow over simple scripts and the scheme that combines SMT and evidence management in practical auditing tasks? This type of evidence will directly affect my judgment on the practical significance of the paper.

2 comments

Thank you. We've clarified the contribution in Sections 2-3: an executable contract that keeps representation checks, evidence acquisition and result reuse tied to the same original question. The key is what must stay consistent across these steps. For example, updating a file hash does not establish that the old decoder still works, and fixing a representation does not supply missing evidence. The contract makes these dependencies explicit and checks them under the declared model. We do not claim a new solver or a capability that a careful combination of existing tools cannot provide.

Appendix I adds the direct workflow comparison you suggested: a sample script, direct SMT with evidence management, and our workflow. All use the same source and access rules; the formal paths share source and receipt management. Eight controlled conditions cover a wrong proxy, missing evidence, file changes, repair and restoration. We use one deliberately selected retained case; repeats are not independent applications. The script is assessed only on its sample-level claim.

Both formal workflows agree throughout; ours is slower in the ordinary-acquisition condition. This demonstrates execution of the specified workflow, including revalidation, without establishing an advantage over the strong composed baseline. Independent discovery of unknown defects, adoption and human modeling or maintenance costs remain unmeasured.

Thanks for the authors' response. The authors didn't address all my concerns, so I will retain my original score.

Reviewer_rEdmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a source-to-claim contract for agent-trace evaluation. It checks whether a representation preserves the native question, whether available observations determine its answer, and whether supporting evidence remains valid after change. The workflow uses established finite-space and SMT techniques for diagnosis and evidence acquisition, separating global, case and saved-sample conclusions. Validation includes 212 contracts, conformance on 10,800 retained executions, recorder transformations, a controlled late-commit example and access-layout and source-maintenance studies. Guarantees remain conditional on supplied semantics, domains and catalogues.

Critical strengths
  1. The interface checks preservation of the evaluation question before evidence sufficiency, separates global/case/sample conclusions and handles empty fibers explicitly. The temporal example and premise sensitivity show why these distinctions matter (pp. 2-4, 6; Appendices A, E and G).
  2. The validation covers source conformance, solver agreement, acquisition and maintenance, while retaining negative comparisons: bounded checking finds the injected mismatches, direct solvers are faster, and selective access may cost more bytes (pp. 4-8; Appendices C and H). This supports useful applicability guidance without a superiority claim.
Critical weaknesses

Practical value beyond the prepared applications is underdemonstrated. The contracts, misuse cases, schedules and repairs are author-supplied; repeated structures and shared premises limit what backend agreement says about reliable contract construction or use by another evaluator. Supplying and maintaining the trusted semantics remains unmeasured (pp. 4-6, 9; Appendices B, E and H). This limits significance and the strength of acceptance, rather than falsifying the conditional guarantees.

Questions for authors

Could you show one retained nonidentity recorder application end to end: its native domain, adapter and target expressions, decoder/source anchors, acquisition catalogue and changed-source revalidation? Please distinguish manually supplied inputs and repair decisions from extracted or automatically checked components, and report any existing preparation-effort record. No new environment or user study is requested.

1 comment

Thank you for the specific suggestion. We've added a complete example in Sections 5.1-5.2 and Appendix I, following one retained call from its source record through evidence acquisition and revalidation.

The call was dispatched but had a recorded error, so it was excluded from the successful-call view. Treating that exclusion as non-dispatch changes the question. Adding a dispatch variable repairs the representation, but the filtered view alone still cannot answer whether dispatch occurred; reading the invocation ledger resolves it. Appendix I gives the domain, assumptions, adapter, target, decoder, source anchors and acquisition catalogue. This example was deliberately selected to illustrate the distinction, not its prevalence.

We also tested an actual field rename in a copied file. A stale hash causes refusal; updating the hash alone still fails with the old decoder. An explicit decoder repair restores the answer to the same question. The original records are unchanged.

The question, modeling assumptions, field meanings and repairs are supplied inputs. The code checks the formal obligations, source identity, permissions and decoding under those assumptions. We have no contemporaneous record of human preparation hours and do not claim reduced human effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.