acceptodds
Under review as a conference paper at ICLR 2027

Hearsay: Can an Auditor Trust the Record a Deployed Agent Harness Writes?

Abstract

An agent harness, the code that turns a model into an agent, writes its own record of each run, and that record is all a later reader gets when a run is disputed, investigated or audited. We call a record *evidentiary* when a reader who was not there can check it without trusting the writer. Across sixteen deployed frameworks, none writes one in full. Hearsay examines the record, not the task: five harnesses run fourteen tasks, three blinded LLM examiners and a human panel read the records, and every excerpt an examiner quotes is checked mechanically for who wrote it. Two findings follow. First, the record lets a reader name the fault but not prove how the run went. Examiners name the right fault in 74 to 91% of 140 runs by a majority of three model graders, and the fault can be proved, but only from two files the benchmark adds, the failing test's output and the final diff; for what happened in between, fewer than one citation in ten lands on anything the harness did not write, and when we delete, rewrite or fabricate entries in copies of the records, the examiner with the fewest false alarms catches half of them. Second, the remedy is a second author, not a stronger seal on the first. An append-only log of what passes between harness and model, kept outside the harness, is read against the record in both directions: a query finds what the log saw and the record lacks, a reverse check what the record holds and the log never saw. Together they report all 28 omissions and fabrications we made a harness commit as it ran, where a hash chain over the harness's own record passes all 28, since it hashes whatever the harness wrote. On 115 records we falsified by editing copies they report 98, each scored against a clean copy of its run with rules written on those pairs; held out on another vendor's records, the reverse check catches 29 of 34 fabrications. Handed the log, examiners keep their fault verdicts but rest more of their citations on what the harness did not write. What makes a record evidence is who writes it, not what is captured.

Primary area:
datasets and benchmarks
Keywords:
agent harnesses, auditability, evidentiary records, LLM agents, benchmark, provenance, tamper evidence, failure attribution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3E3MOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the record an LLM agent harness writes about its own run can serve as evidence for a reader who was not present. It calls a record evidentiary if it can be checked without trusting its writer. The paper makes three contributions.

Static audit. It audits 16 open-source harnesses on 10 source-level properties with an LLM instrument. None of the 32 “absent-reader” cells is present. Hearsay benchmark. Five harnesses run 14 tasks twice each (140 runs, claude-sonnet-5) in a sandbox that independently captures model traffic and process/file events. Three LLM examiners from three vendors read each record under seven material conditions and answer three questions: reconstruction (Q1), fault attribution (Q2), and alteration detection (Q3). Every quoted citation is mechanically labeled by who wrote the text it lands on: narration, corroborated, or independently witnessed. Second writer. This is an append-only, hash-chained ledger fed from the model and environment boundaries. It is reconciled against the native record by a forward query (omissions) and a per-format reverse check (fabrications).

(omissions) and a per-format reverse check (fabrications).

Main results:

Examiners attribute the fault in 74–91% of runs. Only 16–22% of their native-condition citations land on text the harness did not write, and all of that is the two benchmark anchors. The best low-false-positive examiner detects 48.7% of injected alterations at 5.2% FP. On write-time falsifications in aider, the second writer catches 28/28 while a self-computed hash chain catches none. Under paired scoring it reports 98/115 edited copies. Held out on another vendor’s runs, it catches 29/34 fabrications.

The study is preregistered with dated deviations and releases code, records, and alteration pairs.

Critical strengths
  1. The citation-provenance instrument is the paper’s most reusable idea, and it yields a non-obvious result.

Grading an LLM judge by who authored the text each citation lands on, rather than by its verdict, is a clean, mechanical, model-agnostic measurement. It transfers well beyond agent audit to any LLM-as-judge or LLM-as-auditor setting. It produces a result I would not have predicted: all three examiners overstate how much of their own evidence is corroborated. E1 claims 86.7% where the rule finds 68.8%; E2 and E3 overstate by 55 and 46 points. Examiners with similar fault accuracy rest on very different material: a +27.7-point E1–E2 gap in corroborated share. Combined with 82/100 human agreement on landing labels, this is solid evidence that “the judge was right” and “the judge’s answer was supported” must be measured separately. The underlying thesis, that a self-authored record is testimony and sealing it does not change that, is a structural claim that will survive model scaling.

  1. Process transparency is exceptional.

The preregistration appendix sets every registered item against what was delivered. It reports the failed predictions: reconstruction below band, alteration above chance, held-out fabrication recall below 90%, and the dissociation criterion failing cross-vendor. Every post-hoc measure is marked, and Table 30 gives each headline a second check. Multi-vendor examiners and graders, a 37-person human panel, and full artifact release make the paper auditable in exactly the sense it argues for. Most of the weaknesses below come from the authors’ own disclosures, which deserves credit regardless of the decision.

Critical weaknesses

W1. The headline dissociation is largely true by construction. The parts that are not rest on post-hoc, in-sample, or paired measures that overstate deployment performance.

(a) By-construction results. Under the native condition, the only text the harness did not write is the two anchors: the failing-test output and the final diff. By definition these say nothing about intermediate steps. So the abstract’s claim that fewer than one in ten citations about “what happened in between” land on harness-independent text (2–9% on Q1) is fixed by the design, as §5.2 itself concedes. The write-time contrast is similar. A hash chain over the harness’s own record passing 28/28 self-authored falsehoods is a definitional property of hash chains. A verbatim diff against a proxy log is close to guaranteed to catch injected fabrications that never reached the model. An informed expert would not bet against either result: they demonstrate the concept rather than test it.

(b) Paired, in-sample scoring. The empirical content that is not by construction is where evidence is thinnest.

The 98/115 is scored by paired difference against a clean twin of the same run. A real auditor never has that twin, and the reverse rules were written on those same pairs. Read stand-alone, the reverse check flags 52 of 95 clean copies, including 26/28 for mini-SWE-agent and 26/28 for cline. On held-out runs it still flags 21/22 and 21/21 clean copies for those two formats. Held-out fabrication recall (29/34, 85.3%) misses the registered 90%. Removals are invisible to the reverse check by design (0/35 held out). The deployment-relevant number is never a headline: unpaired detection vs. unpaired false positives, on held-out records, with rules frozen in advance.

(c) Post-hoc measures carry the headlines. Several were added after results were seen:

the independently witnessed share, which underlies both the “one in ten” claim and the ledger-shift claim; the majority-of-three grading rule, which superseded the registered grader 1; the 10% false-positive threshold; the dissociation criterion, which fails on the other-vendor runs (E1 witnessed share 31.1%, E3 accuracy 69%).

Meanwhile the registered primary outcome for the intervention, the corroborated share, barely moves. It rises only for E1 (+5.3), and only when ledger rows of the agent’s own words count; without them it is −1.4, and E2/E3 move −0.5/−1.6. The abstract’s statement that examiners “rest more of their citations on what the harness did not write” is therefore a post-hoc measure presented as a finding. Human graders were also stricter on native answers (12/25 correct vs. 18/25), so the 74–91% accuracies may run high.

(d) Little evidence of false records in the wild. There is roughly one natural case: a SWE-agent run that filed “Exit due to multiple consecutive command timeouts” as an assistant line the model never produced. The other “wrong clean record” came from the authors’ own run bound failing, which no harness could have seen. Beyond that, 7/140 records lack a last unit after crashes or timeouts. This is the finding that would make the problem pressing, and it is the least developed.

Overall, the paper claims at the “the intervention works” rung, but that evidence is in-sample, paired, from one trust domain, on 14 short tasks capped at 7 minutes, and the write-time test uses a single-turn harness.

W2. The “second author” is a second component, not a second party, so the evidentiary/hearsay framing overclaims.

Both writers, the database, and the proxy are run by the operator, who is explicitly trusted (§4.2, §7, Table 2). In the settings the paper invokes (disputes, investigations, regulatory audits under the EU AI Act’s logging duties), the party whose account is in question is usually the operator, not the harness code. Against a buggy harness, the ledger is a useful consistency check. Against an interested operator, it adds little beyond an operator-signed log. Appendix G notes the operator’s role can drop the append-only trigger or truncate the table. On the main grid, the harness was not confined to the proxy and shared a database role with the witness. So the independence assumption was unenforced for all 140 runs; it was enforced only in the 28-run re-run. “What makes a record evidence is who writes it” is right as a principle, but the delivered system changes which process writes, not which party. The one witness the paper itself names as outside the operator’s control, the model provider’s request log, is never tested.

I would want either a scope correction (from “evidentiary record” to “harness-independent consistency check”) or an experiment with a witness the operator does not control.

Presentation (secondary). The prose is extremely compressed and term-dense: located, corroborated, independently witnessed, narration, event vs. narration reading, clean twin, gate, floor/ceiling. Nearly every main-text sentence carries a qualifier resolved only in a 38-page appendix. The abstract is hard to parse on first reading, and Figure 1 and Table 1 pack in more numbers than a reader can absorb. Restructuring the main text around the registered predictions, which ones held, and the deployment-relevant numbers would help more than additional checks.

Questions for authors

I would raise my rating if Q1 shows stand-alone performance near the paired numbers, and the abstract is restated around the registered outcomes and the citation-provenance instrument.

(Could change soundness and rating.) Please report the reconciliation as a deployed auditor would use it. Freeze per-format rules and allowlists before seeing the data, score unpaired (no clean twin) on held-out records, and report detection by alteration type and false positives per format. If fabrication recall stays at or above 85% with stand-alone false positives at or below 10% on all four formats, my main concern about §6 is resolved. (Could change my significance assessment.) How often are deployed records wrong in the wild? You have 140 + 140 + 31 natural runs with a ledger. Please give a full census of query and reverse-check findings on unaltered records, classified as benign bookkeeping (e.g., the synthetic “action was not executed” observation), role mislabeling (the SWE-agent case), or true omissions after crashes. Can this be run at longer horizons, given that the 7-minute bound is short relative to production agent sessions? (Clarification, threat model.) In which concrete audit or dispute scenario is the harness the adversary while the operator is trusted? If the target is bugs and misfiling, would you reframe the contribution as a consistency check? If the target is an interested operator, what does the ledger add over an operator-signed log, and have you tried reconciling against provider-side request logs? (Clarification.) Detection in §5.3 depends on how subtle the injector was made. It was revised five times until E1’s model family could not pick the edited copy above chance, while another vendor’s judge still could (60.9%, p = 0.012). Can you report detection across injector revisions, so that “catches about half” reads as a curve over edit subtlety rather than a property of the record? (Clarification, §3.) Domain provenance (as-of time, entitlement, and cost as first-class fields) is still written by the harness. By your own definition, it cannot make a record checkable without trusting the writer, so why is it one of the two absent-reader properties? Relatedly, the registered instrument (claude-fable-5) was replaced by claude-sonnet-5, and human agreement on the 33 re-judged cells is 26/33, below your registered 80%. How should readers weight the activity correlation (ρ = 0.55, n = 16)? (Clarification.) Please state recruitment, compensation, and IRB approval or exemption for the 37 annotators; the ethics statement does not cover them.

Suggestion (does not affect my score): the strongest contribution is the citation-provenance instrument plus the self-report overstatement finding, not the 14-task grid. Leading with that, and treating the benchmark as its testbed, would position the paper outside an already crowded agent-benchmark space.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 3E3M (1/2): your two conditions

Thank you for the most exact review we received, and for stating what would change your assessment. Your protocol, rules frozen first, one record read alone, held-out records, results by type and by format, is now the paper's main evaluation: §6 and Fig. 7 are built on it, and the abstract reports its numbers and no longer the paired one. We answer your two conditions here, in your terms, and the remaining points in a second comment. "Items" refer to our first detailed comment to all reviewers; the table by format is in the second.

Condition 1: stand-alone performance, rules frozen first, held out, by type and per format.

Your bar: rules and exemptions frozen before the data, one record read alone, fabrication recall at or above 85%, false positives at or below 10%, per format. The test that meets the freezing without qualification is the one whose rules were committed to the repository before any record existed. It covered three formats; because your condition is per format, we ran the other two during the review, rules file untouched, predictions registered before each.

formatclean records flagged (bar: ≤10%)fabrications (bar: ≥85%)rewritesomissions
aider0/28none eligible1/110/10
mini-swe-agent0/2810/105/610/11
SWE-agent0/289/910/109/9
cline‡0/288/105/513/13
opencode‡0/2813/135/78/8†
all five0/14040/4226/2950/51

‡ Run during the review (Appendix G, Table 28). † Read with the database export decoded; see Appendix G.

Where this falls short of your bar, and why.

  • Cline, fabrications 8 of 10. One is a genuine tool reply repeated with a file name exchanged, the re-use gap §6 names; one copies an output the harness had stored truncated.
  • Opencode, rewrites 5 of 7. Both misses change a display copy of a file read, a field no rule reads; the text sent to the model is intact.
  • Aider records held no unit eligible for a fabrication.
  • We did not repair the rules for these: a rule changed after the result is what your condition excludes.

The held-out records (115 pairs, five formats) meet it in part. Stand-alone and paired readings agree pair for pair on every fabrication and rewrite (Table 25). The unit rules were never written on those records, but only three of the seven exemption classes were named beforehand: with those three, 37 of 115 clean records are flagged; with all seven, 1. Fabrications are 38 of 43 pooled, with mini-swe-agent 7 of 9 and cline 6 of 8 under your bar. §6 says this.

If there is a further element you would want fixed in advance, we would be glad to register it.

Condition 2: the abstract restated around the registered outcomes and the citation-provenance instrument.

  • The abstract's first result is the instrument's: examiners claim corroboration for 18 to 55 points more of their evidence than the labels support. We say "then predicted": the gap was first seen on the primary runs, then registered as a prediction for the third vendor's runs (at least 10 points), where it came out at 30.
  • "Rules committed before the records existed" is said of the third vendor's runs only.
  • The registered primary outcome of the intervention is reported in the body, as you asked: the corroborated share "barely moves" when a reader is handed the ledger (+5.3 for E1 on the primary runs, −1.4 on the third vendor's).
  • §1 lists the measurement first among the contributions. Thank you for that suggestion; it is the better framing and we adopted it.
  • Appendix H gives the score of the predictions (45 of 59) and lists every one that did not hold.
AuthorsOpenReview ↗

Reply to Reviewer 3E3M (2/2): remaining points

W1(a). By-construction results. Agreed on all three. The "one in ten" claim is gone; §4 says the witnessed share "is set by the design". The hash chain passing 28 of 28 is presented in §2 as a demonstration that the one protection in use does not bind the writer, and the write-time test opens §6 as "the easiest test".

W1(b). Condition 1, in our first reply. Omissions remain invisible to the reverse check by design; the query reports them, 43 of 44 held out and 29 of 30 on the third vendor's runs.

W1(c). Post-hoc measures. The dissociation criterion and the 10% threshold are out of the body; the criterion's failures on both other sets of runs are in Appendix F. The majority rule and the witnessed share remain post hoc, and Appendix H says so. For the third vendor's runs we registered five predictions on these measures: four held and the dissociation criterion missed (Tables 21 to 23, 36). §4 says the accuracies "may run high" because human graders were stricter.

W1(d), Q2. Records wrong in the wild. The census you asked for is in Table 32: every finding of both checks on 395 unaltered records, classified by rules fixed in a script. In the three main sets, 33 of 365 would mislead a reader, 31 of them SWE-agent's. We state the base rate as it is: such records are a minority, and nothing in a record shows which they are. The SWE-agent fault is real and unprovoked: on its third-vendor runs, 37 of 412 tool calls the model sent are absent from the saved trajectory. Longer horizons were not run: the bound is reached on few runs and longer tasks need a new task set; §7 states the scope.

W2. A second component, not a second party. We did both things you asked. The experiment: the provider's stored copy of all 708 calls of the third vendor's runs is byte-equal to the ledger, and both checks reach the same findings from either on 84 of 84 runs. This is one provider; §7 says so. The enforcement: separate database roles and a network namespace on those 84 runs. Inside the namespace no outside connection succeeded. The scope correction: §7 says that where no outside copy exists, "the ledger is a consistency check under the operator's control and our claim is the narrower one". Fig. 8 shows the operator's rewriting as a row that only the provider's copy exposes.

Q4. Threat model. A harness that is untrustworthy while its operator is honest is the ordinary case of a faulty harness: it drops, misfiles or truncates what it records, with no intent anywhere. The census documents it, and "consistency check" is the right name for what the ledger does there. Against an interested operator the ledger adds nothing over an operator-signed log unless a copy sits outside the operator.

Q5. Detection against subtlety. We can give two things and not the third. Appendix E has the gate's accuracy at each of five rounds (83.7, 70.9, 64.5, 70.3, 57.5%) and under a second vendor's judge (60.9%). Appendix F has each examiner's detection by size of the altered unit (Tables 15, 16): E1 stays between 44% and 53%. Examiners read only the final revision, so we cannot give their detection at earlier rounds; a sentence claiming cruder changes are caught more often has been removed. §3 now states the five rounds in the body.

Q6. The audit. Provenance is no longer an outsider's property; the count is 0 of 16. The instrument swap is disclosed (Appendix B). A second instrument from another vendor agrees on fifteen of sixteen verdicts and gives partial credit on the sixteenth; neither finds one. Human agreement is 26 of 33, one short of the bar we registered, and the reported property has no human-checked cell; §2 says both. The activity correlation should be given no weight.

Q7. Annotators. The ethics statement now covers them: volunteers known to the authors, unpaid, consenting to anonymised release, no ethics-board review, our assessment of minimal risk reported in the statement.

Presentation. We followed your advice that restructuring would help more than additional checks, and did both. The terms you list are gone from the body or defined once where they first appear; Fig. 1 and Table 1 of the submission are replaced by eight figures and a table of record sets.

Most of the weaknesses you raised came, as you noted, from our own disclosures. We have tried to answer in the same spirit, and we would value your reading of whether the revised §6 now claims only what it shows.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer 3E3M,

Thank you again for the most exact review we received, and for stating what would change your assessment. Our reply and the revised manuscript were posted on 11 November.

You named two conditions. On the first, the evaluation now follows your protocol: rules committed before the records existed, one record read alone, results by type and by format. Across five formats, no clean record is flagged (0 of 140; your bar was at most 10%), and 40 of 42 fabrications are reported (your bar was at least 85%). One format, cline, is below the bar at 8 of 10, and we did not repair its rule after seeing the result. On the second, the abstract now leads with the citation-provenance result and reports the registered outcomes.

The discussion period closes in less than 24 hours. Could you let us know whether these results meet your conditions, or what remains open? If they do, we would be grateful if you would update your review to reflect this.

Thank you again for your time!

The Authors

Thank you for the detailed reply. I am raising my rating from 2 to 3.

The frozen-rule test resolves my main concern: 0 of 28 clean records flagged in each of the five formats, fabrication recall at my bar in three of four (cline 8 of 10), and no rule repaired after the result. The abstract now leads with the provenance measurement, as I asked.

For the final version, not conditions for my score:

  • The abstract's "flags none of 84 clean records" holds for the reverse check only; the query reports real omissions on 15.
  • State in the abstract that re-used genuine text passes the committed check (35 of 35) and that the operator is trusted.
  • §6 should give opencode omissions as registered (3 of 8) beside the decoded 8 of 8.
Reviewer_uvnyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether an investigator can trust an agent system’s own log. The paper audits 16 agent frameworks and finds that none writes a record an outsider could verify. The authors run five agent harnesses on 14 tasks and ask examiners to identify faults, reconstruct events, and detect altered records. They also check who wrote the material the examiners cite. Finally, the paper adds a second, independent log outside the harness and compares it with the harness’s own log (the agent log). This catches omissions and fabrications that the harness’s own hash chain does not.

Critical strengths
  1. The question is relevant and clearly motivated. Agent records are increasingly used for debugging, audits and regulation, and the point that a record written by the system under question cannot prove its own correctness is simple and convincing.

  2. The outside log is a clear demonstration. When the authors made the harness write false entries, the harness’s own hash chain accepted all of them, while the outside log caught them. The paper is also transparent about what the outside log cannot catch.

Critical weaknesses
  1. Some results are expected from the setup. The examiners mostly get the agent log, so they mostly quote the agent log. The authors revised their edits five times until one examiner model could no longer pick the edited copy significantly above chance. So that examiner’s trouble finding the edits is partly expected. The experiments are still valid. But the detection results may not hold for other kinds of edits.

  2. The outside log works as a proof of concept, but it is unclear how it becomes a practical solution. When the automatic check compares the agent log with the outside log, it also flags about half of the unedited agent logs as containing false entries. The authors remove these false alarms by comparing with an unedited copy of each run, which a real investigator would not have. The paper assumes the operator runs the outside log and is trusted, and does not discuss the case where the operator itself is the party being checked.

Questions for authors
  1. The paper assumes the operator runs the outside log and is trusted. In hosted products such as ChatGPT agent, the provider runs the model, the agent program and the sandbox, so it would build and run both logs. Who should implement the outside log in practice, and how would an auditor check the provider itself?

  2. Did you test giving the examiners the flags from comparing the two logs? An examiner could then judge each flag, which might reduce the false alarms.

  3. The outside log keeps a copy of all messages to the model. How do you protect that copy, and how do you stop someone from bypassing or breaking the outside log?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer uvny

Thank you. Your presentation score was the lowest of the three, and we took it as the main instruction for the revision. The two detailed comments to all reviewers list the new experiments (items 1 to 9) and the results.

Presentation. The paper was rewritten for a first reading. Fig. 1 states the question and the three parties that might check a record, and each section answers for one of them. Terms are defined where they first appear, each page carries one figure or table, and internal labels left the body. We would be grateful to know whether §1 and §2 can now be followed without the appendix.

W1. Results expected from the setup. You are right on both counts, and the paper now says so. Examiners mostly quote the harness's record. §4 states that this share "is set by the design". It then reports the result that is not: each examiner also labels its own citations, and every one claims more corroboration than the mechanical rule finds, by 18 to 55 points. We then predicted this gap for a third vendor's runs and found it again. The alterations were refined until a model could not spot them. §3 now says that passing the gate "took five rounds of repair to our alteration script, so what we measure is whether a record reveals a careful change". Appendix E gives the gate's accuracy at each round (from 84% down to 58%) and under a second vendor's judge (61%). Within the final set, detection does not depend on the size of the change: the best examiner stays between 44% and 53% from the smallest quarter of changes to the largest. We also tested a different kind of falsehood, a record that was false from the start (item 9).

W2. Half of the unedited logs flagged; the fix needs an unedited copy; the operator. No unedited copy. The checks are now scored as an investigator would use them, one record at a time, never seen before. On 115 held-out records in five formats they report 109 alterations, and reach the same finding as with the unedited copy on every fabrication and rewrite. The false alarms. They were text a harness legitimately writes itself, such as a final tool output that is never sent back to the model. Seven exemption classes, defined by the structure of a record and never by its content, now cover them. The clean test of this is on 84 runs under a third vendor's model, with every rule and exemption committed before the runs: no clean record is flagged. On the held-out records some exemptions were written after the records were read, and the paper gives the figures both ways (§6).

Q1. Who runs the outside log when the provider runs everything; how is the provider checked? The auditor needs a copy that the party under audit cannot edit afterwards. Where the harness and the model are run by different parties, the model provider's stored copy of each call is such a copy, and we tested it: all 708 calls of the third vendor's runs, retrieved from the provider, are byte-equal to the ledger. That is one provider, whose API lets a customer retrieve stored calls by their identifiers; we make no claim about others. Where one company runs the model, the harness and the sandbox, our mechanism does not make that company's record evidence against itself. §7 says so: the ledger is then a consistency check under the operator's control, useful against a faulty harness and not against an interested operator. We do not claim more.

Q2. Giving the examiners the flags. We did not run this. Two results bear on a reader as the judge of a flag. Handed the whole ledger, examiners name the fault no more often and their corroborated share barely moves (§4). And when the harness lies to the model as well, an examiner handed the ledger was misled by the false line in it, while the mechanical check on exit codes was not (§6). We would keep the flag mechanical and let a reader interpret it.

Q3. Protecting the copy; bypassing or breaking the outside log. Protecting it: the ledger is append-only, each row is chained to the one before by a hash, and the latest row is timestamped by a public service (§5). Writing to it or going around it: on the third vendor's runs the two writers use separate database roles and each run is confined to a network namespace whose only way out is the logging proxy; no outside connection succeeded on any of 84 runs. Breaking it from the operator's side: this is what the provider's copy is for (Q1). What the ledger cannot see is listed in §7 and Fig. 8. The ledger stores the model traffic, prompts included, so whoever holds it holds that content; we do not address its confidentiality.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer uvny,

Thank you for the time and effort you put into reviewing our paper. Your comments led us to state which results follow from the study design, to test the false-alarm concern without relying on a clean copy, and to say more precisely what the ledger can and cannot establish when the operator controls the whole system. Our reply and the revised manuscript were posted on 11 November.

On your main concern, that the check flagged about half of the unedited logs and needed an unedited copy: the checks are now scored on one record read alone. On 84 runs with every rule committed before the runs existed, no clean record is flagged. On 115 held-out records in five formats, 109 alterations are reported.

We also took your presentation score of 1 as the main instruction for the revision, and the paper was rewritten for a first reading. Would you be willing to look at the revised Sections 1 and 2? If any paragraph is still hard to follow, we would be grateful to know where.

The discussion period closes in less than 24 hours. Could you let us know whether the revision addresses your concerns, or which questions remain? If it does address them, we would be grateful if you would update your review to reflect this. Even a brief reply now would let us answer before the deadline.

Thank you again for your time.

The Authors

The new experiments and clarifications address my main concerns, so I’m raising my score.

Reviewer_2VpcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the record an LLM agent harness writes about its own run can serve an auditor who was not there. It calls a record "evidentiary" if a reader can check it without trusting the writer. It audits sixteen open-source frameworks on ten properties (Sec 3, Fig. 2). On the two properties an outsider could check (a tamper-evident record and domain provenance), none of the 32 cells is present: 3 partial, 29 absent. It then builds Hearsay. Five harnesses run fourteen tasks twice (140 runs) in a sandbox that separately logs model traffic, processes and files. Three LLM examiners from three vendors, and a human panel, read the records under seven material conditions. Each citation is labelled mechanically by who wrote the text it lands on: agent narration, harness-recorded output, or independently witnessed. Under the native record, independently witnessed material is only the failing test output and the final diff. Main findings: examiners name the fault in 74-91% of runs (majority of three graders), but only 16-22% of their citations rest on material the harness did not write (Sec 5.1-5.2). On 115 edited copies, the examiner with the fewest false alarms catches about half at a 5% false-positive rate (Sec 5.3). The proposed remedy is a second, append-only writer outside the harness, read against the record in both directions: a query for omissions and a reverse check for fabrications. It reports 98 of the 115 edits (paired against a clean copy, with rules written on those pairs), all 28 write-time faults the authors made aider commit, and 29 of 34 fabrications on held-out records from another vendor (Sec 6).

Critical strengths
  1. Transparency. Missed registered predictions are reported: reconstruction below the predicted band, alteration detection above chance, and the held-out reverse check below 90% (Appendix H, Table 30). Post-hoc additions are dated and marked. Alternative readings are collected in one place (Tables 28-29), including readings that weaken the headlines (the narration reading, the other vendor's runs).
  2. The citation-level provenance label is a useful new measurement for LLM-as-judge and agent-trace work. Grading what a verdict rests on, not only the verdict, exposes a large self-report gap (E1 claims 86.7% corroborated against 68.8% by rule; Sec 5.4).
  3. Careful controls: the record without anchors (91%, so the record carries diagnostic signal), a self-preference control on runs driven by another model, other-vendor runs where E3 also drops (so the drop is not examiner familiarity), and a re-examination for test-retest noise.
  4. The remedy is cheap (about 230 lines, 1.06x storage, +14 ms per call), and the write-time test was repeated under enforced isolation. The query also finds natural omissions on clean runs: 8 units on 7 main-grid runs, and 46 units on 23 runs from the other vendor, including tool calls SWE-agent drops.
Critical weaknesses
  1. Some headline results are largely determined by the design, but the abstract presents them as findings. (a) Reconstruction citations are independently witnessed under 10% of the time (9.2/2.1/6.9%, Table 19). Under the native record, an intermediate event exists only in the harness's record, as Sec 5.2 and Appendix F themselves acknowledge ("set by the design"). The abstract should say so. (b) "0 of 32 present" follows closely from a definition that requires a second writer or external anchor (Appendix A, P6). The human re-judging of 33 cells included no gap cell; only aider P6 was human-checked, in the disagreement round. The records also mostly look accurate (median reconstruction precision 1.0; two wrong clean records), so the paper shows that records are unverifiable more than that they are untrue. The title's "trust" blurs the two.
  2. The remedy's headline numbers are in-sample and depend on a clean copy. (a) The reverse rules were written on the 115 pairs ("none held out"), and SWE-agent has no rule, which accounts for 15 of the 17 misses. (b) Read without its clean copy, the reverse check flags 52 of 95 clean copies in-sample, and 21/22 (mini-SWE-agent) and 21/21 (cline) held out. A deployment would not have a clean copy, and the paper reports no detection rate for a single record read alone. (c) The held-out test is the strongest evidence: 47 of 52 rewrites and fabrications, with fabrications at 29/34, below the registered 90%. The abstract should give both figures. (d) The write-time result covers two fault types the authors injected into one single-turn harness with one fixed fabricated paragraph. Detection there is close to guaranteed by construction. A comparison with self-computed chains (as deepseek-harness and Dapr ship) is reasonable, but the result says little about faults the authors did not design.
  3. Trust boundary. The operator who runs both writers is trusted (Sec 7), and on the main grid the writers shared a database role and the harness was not confined to the proxy (Sec 4.2). A truly absent auditor must still trust the operator. Tool outputs were not witnessed on the main grid, so a harness that reports a failed tool as passing is not caught (Table 2).
  4. Vendor and grader dependence, and post-hoc measures. E1 (the first evaluated model) drove the runs, served as the audit instrument, and shares a vendor with grader 1. On the other vendor's runs, accuracy falls (83% vs 91%) and the post-hoc dissociation criterion fails. Several headline measures were added after results were seen: the independently witnessed share, the majority grading rule, the 10% false-positive threshold, and the dissociation criterion. Human graders are stricter than the model graders (12/25 vs 18/25), and the registered corroborated-share shift (+5.3 for E1) is about the size of the re-reading noise (+-5.1).

Notes on the Presentation, Informativeness and Soundness scores

Presentation: The ideas are clear, and the framing (hearsay; a second author, not a stronger seal) is memorable. The main text is very dense, though. The abstract runs about 360 words with many numbers, and key caveats live only in the appendices. For example, understanding "98 of 115" (in-sample rules, a paired clean copy, no rule for SWE-agent) requires Appendix G. Three examiners, three graders, seven conditions, post-registration daggers and two corroboration measures are a lot to track. A short table separating design-determined results from empirical ones, and an abstract that states the main caveats, would help a lot.

Informativeness: I learned something real: the examiners are good at naming the fault from a record, almost none of what they cite is independent, and examiners overstate how corroborated their own citations are (§5.4: E1 says 86.7%, the rule finds 68.8%). The negative and partial results (vendor dependence, the post-hoc dissociation test failing on the other vendor's runs, the held-out reverse check below the registered 90% bar) are in the paper, which I count in its favour. But two of the three headline findings are largely determined by design, as Sec. 5.2 says, so the net amount learned is less than the length suggests.

Soundness: Minor-to-moderate fixable concerns. The measurements are careful, and the reported numbers are internally consistent where they can be recomputed from the tables. The problems are in what the numbers are claimed to show, not in the arithmetic.

Questions for authors
  1. Please report, for all five formats, detection and false-alarm rates when a single altered record is read alone (no clean copy), for the query, the reverse check, and both together. If a per-format allowlist is needed, how many lines does a new harness take?
  2. If the reverse rules were frozen on a development split and tested on the remaining pairs, would the four-format figure (93/95 together with the query) hold? Can SWE-agent get a rule?
  3. Can you test an adversary that knows the check? For example, a harness that fabricates by cloning a unit it genuinely sent (the cline path swap in Appendix G), or one that alters a tool output before sending it to the model (Table 2, "not tested").
  4. Your mechanism is a logging proxy plus process capture plus a diff. Beyond the measurement, what does it add over existing gateway or proxy logging practice? Is there a version in which the auditor need not trust the operator, for example using the model provider's request log, as Appendix I suggests?
  5. Does the activity correlation (rho = 0.55, n = 16) survive dropping the four projects with 0-5 commits in the trailing year (auto-code-rover, Agentless, DARS-Agent, moatless-tools)?
  6. Can you add write-time faults on at least one multi-turn harness, or replay real harness bugs (for example SWE-agent dropping multi-tool-call responses on the other vendor's runs)?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer 2Vpc (1/2): weaknesses

Thank you for a review that recomputed our tables; it made us recompute them too. The two detailed comments to all reviewers give the new experiments (items 1 to 9) and the results by format; here we answer your points in your order.

W1(a). Design-determined results presented as findings. Agreed. The abstract no longer carries the "one in ten" figure. §4 says the witnessed share "is set by the design" and that the next result is not: each examiner calls its evidence corroborated more often than the labelling rule finds, by 18, 55 and 46 points. We registered this gap as a prediction for a third vendor's runs (at least 10 points for E1) and it came out at 30.

W1(b). "0 of 32" follows from the definition; unverifiable is not untrue. Three changes. Provenance is no longer counted on the outsider's side: §2 reports one property, 0 of 16. Because that count is close to definitional, §2 no longer rests on it alone: it adds a test of the one protection in use (a hash chain verifies on all 28 records a harness wrote falsely) and the records that go wrong with no intervention. And on your reading that records are mostly accurate, we agree and give the base rate: 33 of 365 unaltered records would mislead a reader, "but nothing in a record shows whether it is one of the 33". The title no longer uses "trust". The reported property still has no human-checked cell; §2 says so.

W2(a)-(c). In-sample rules, paired scoring, no stand-alone rate. §6 is rebuilt around this (Fig. 7). The paired figure, 98 of 115, is now the second of five tests and is labelled as favouring the checks. The third is one record read alone on 115 held-out pairs in five formats: 109 of 115, the same findings as the paired reading on every fabrication and rewrite. SWE-agent has a rule (9/9, 10/10, 8/9). The abstract gives these figures, not the paired one.

On false alarms: the counts you cite (21 of 22, 21 of 21) are text a harness legitimately writes itself, now covered by seven exemption classes defined by record structure. The clean measurement is the fourth test, a third vendor's runs with every rule and exemption committed first: 0 of 84. On the held-out records only three of the seven classes had been named beforehand, and the table in our second detailed comment gives the figures both ways.

W2(d). The write-time test is near-guaranteed by construction. Agreed; §6 introduces it as "the easiest test, because we know in advance what is false". Faults we did not design: SWE-agent's dropped tool calls, found by the same query (item 7). A different write-time fault on a multi-turn harness: item 6.

W3. Trust boundary. Each part is now tested on the third vendor's 84 runs (items 3 and 4): separate roles, confinement to the proxy, and the provider's own copy of all 708 calls, byte-equal to the ledger. §7 states the limit: we verified one provider, and where no such copy exists the ledger is a consistency check under the operator's control and "our claim is the narrower one". A failed tool reported as passing: item 6.

W4. Vendor and grader dependence; post-hoc measures. E1 now reads runs from three drivers and names the fault in 91%, 83% and 88%: the level moves with the driver, the pattern does not (§7). Every grader shares a vendor with an examiner or a driver, so the majority is reported, and each grader alone in Appendix F. The dissociation criterion and the 10% threshold are out of the body; the criterion's failures are in Appendix F. The measures we keep, the majority rule and the witnessed share, remain post hoc and Appendix H says so; we registered them as predictions for the third vendor's runs, and four of five held (item 8). The registered outcome's small shift is now in the body (§4).

AuthorsOpenReview ↗

Reply to Reviewer 2Vpc (2/2): questions

Q1. Single record read alone, all five formats; lines for a new harness. Our second detailed comment has the table by format; Table 25 gives the query, the reverse check and both together. A new format needs one unit rule of 15 to 26 lines (SWE-agent's is 26) and whatever structural exemptions it needs; the exemption classes are 25 lines, with 16 more for the closing-output condition and 15 for the aider amendment. The remaining mechanism, schema, query and writers, is about 230 lines.

Q2. A frozen split; a rule for SWE-agent. The held-out records are a stricter split for the unit rules, which were never written on them: on the four formats you ask about the checks report 82 of 87 (fabrications 29 of 34, rewrites 18 of 18, omissions 35 of 35), and 109 of 115 with SWE-agent. The cleanest split is the third vendor's runs, rules committed before any record existed: 64 of 66.

Q3. An adversary that knows the check. Both cases were run. Cloning a genuine unit. A model holding the rules, the exemption lists and the ledger rows never got new text past the checks, in 65 attempts. Re-using a genuine sentence passes the committed check on all 35 attempts, by design: that check asks only whether a unit was witnessed. A stricter variant in which a witnessed unit can be claimed once was repaired by one line after round one and frozen. In round two the attacker's prompt stated the repaired rule; it failed on all 50 attempts. The variant flags 2 of 21 clean cline records where the committed check flags none. The committed figures stay primary (Table 30). A tool output altered before the model sees it. On 14 runs a shim turned a failing test's output into a passing one before model or record saw it. The ledger's model side carries the false line, and an examiner handed the ledger was misled by it. Its environment side had logged the exit code of the test process and reports the lie on all 11 runs where the shim had a failure to rewrite, and on none of 14 controls.

Q4. What this adds over gateway logging; a version that does not trust the operator. A gateway log is one more record nobody reads against the first. What we add is the comparison in both directions, rules that make it runnable on a single unseen record, and a measurement of its hits and false alarms as an auditor would get them. The version that does not trust the operator is the provider's copy, now tested (W3).

Q5. Does the activity correlation survive? No: 0.57 on sixteen frameworks, 0.04 (p = 0.90) on the twelve that are not dormant. The body makes no activity claim.

Q6. A multi-turn write-time fault; a real harness bug. The lie experiment of Q3 is a write-time fault on a multi-turn harness, of a different kind from the first. The real bug is characterised in §2 and Fig. 2: SWE-agent discards a reply that carries more than one tool call, or a call with an argument it rejects, tells the model so, and saves neither. Of the 37 dropped calls on the third vendor's runs, 21 are of the first kind, 14 of the second and 2 unexplained.

A correction to a strength you credited. The storage figure of 1.06 times is superseded. Measured online, what the ledger must hold is 1.08 times the record; as we store it, it is 2.78 times, because two harnesses rewrite their whole file at every step and we keep each rewrite (§6).

Presentation. The abstract carries fewer numbers. The dense first figure and table are replaced by eight figures and a table of record sets, so that each denominator in the text has a home. The caveats on "98 of 115" are in the sentence that reports it.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer 2Vpc,

Thank you for the effort you put into reviewing our paper. We are encouraged that you found the citation-provenance measure useful and valued the paper's transparent reporting.

Your suggestions improved the manuscript. Our replies and the revision were posted on 11 November. The paper now separates results that follow from the design from empirical findings. It reports detection and false alarms for one record read alone, by format: 109 of 115 alterations on held-out records in five formats. New runs with every rule committed in advance report 64 of 66 alterations and flag none of 84 clean records. Our replies also cover your questions on an adversary that knows the check, a write-time fault on a multi-turn harness, and the trust boundary, where the provider's own copy of all 708 calls matched the ledger.

The discussion period closes in less than 24 hours. Could you let us know whether the revision and our replies address your concerns? If they do, we would be grateful if you would update your review to reflect this. If anything remains unclear, we would appreciate hearing from you while we can still respond.

Thank you again for your time and effort.

The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of our response

We thank the three reviewers for careful and constructive reviews. The paper has been reorganised in response and its claims narrowed; the revised PDF is uploaded. This comment summarises the response; two comments with the details follow, and a reply under each review.

The two concerns the reviews share. (1) The remedy was evaluated with a clean copy of each record and with rules written on the same records, which an auditor does not have. (2) The paper was hard to read.

What we did.

  • Evaluated the remedy as an auditor would use it, as Reviewer 3E3M specified: one record read alone, held-out records, by type and by format. On held-out records in five formats the checks report 109 of 115 alterations. On 84 further runs under a third vendor's model, with every rule committed to the repository before any record existed, they report 64 of 66 and flag no clean record. With the two remaining formats added under the same frozen rules, the reverse check flags no clean record on any of the five (0 of 140).
  • Tested the trust boundary. Those runs are confined so that the harness can reach the model only through the logging proxy, and the model provider's own stored copy of all 708 calls is byte-equal to our ledger.
  • Faced an adversary that holds the rules. It never got new text past the checks (65 attempts). It found one gap, which we report, and failed on all 50 attempts against the repaired check.
  • Showed the problem occurs with no intervention. A widely used harness, run as released, saves records that lack tool calls the model made, on 19 of 28 runs.
  • Rewrote the paper around one question and three parties (Fig. 1), with every term defined where it first appears. Results that follow from the design are named as such; the abstract leads with the result that does not: every examiner overrates its own evidence, a gap we then predicted for a third vendor's runs and found again.
  • Registered every new experiment before running it. Of 59 predictions, 45 held as written; every one that did not is listed in Appendix H.

Where each reviewer's main requests stand.

reviewerrequeststatus
3E3Mstand-alone scoring, rules frozen first, held out, per formatdone; rules committed first now cover all five formats: no clean record flagged by the reverse check on any; his fabrication bar met on every format with fabrications except cline (8 of 10)
3E3Mabstract restated around the instrument and the registered outcomesdone
3E3Ma witness the operator does not control, or a scope correctiondone: one provider's copies verified; scope stated
2VpcQ1 to Q6each answered
uvnypresentation; false alarms without an unedited copy; who checks the operatorrewritten; measured; answered within a stated scope

Scope of the claims.

  • We left the frozen rules untouched after seeing the results on the two added formats: cline fabrications are 8 of 10 and opencode rewrites 5 of 7, and we report them as they are.
  • Tasks last at most seven minutes; longer ones need a new task set. Stated as scope in §7.
  • Where one company runs model, harness and sandbox, the mechanism cannot make that company's record evidence against itself. Stated as scope in §7.
AuthorsOpenReview ↗

Response to all reviewers, details (1/2): what changed and what we ran

This comment and the next give the details behind the summary above. References such as "item 3" in our replies to individual reviews point here.

The paper. The body is organised by one question, who can check a record, and the three parties that might (Fig. 1): the harness's own seal (§2), a reader (§3, §4), a second writer (§5, §6). One figure or table per page; every term defined where it first appears; internal labels moved to the appendix. A result that follows from the design is named as such in the sentence that reports it.

What we ran. Each experiment was registered with its rules and predictions before it ran. Of the 59 predictions registered for them, 45 held as written, one in part, eleven missed and two could not be read; Appendix H lists every one that did not hold (Tables 34 to 37).

  1. A third vendor's runs, every rule committed first (§6, Table 28). 84 runs of three harnesses under a third vendor's model. The rules and their exemption lists were committed to the repository before the first run. No clean record is flagged; 19 of 19 fabrications, 16 of 17 rewrites and 29 of 30 omissions are reported.
  2. One record read alone, held-out records (§6, Table 25). On 115 held-out pairs in five formats, with no clean copy, the checks report 109 alterations: 38 of 43 fabrications, 28 of 28 rewrites, 43 of 44 omissions. Read alone, the reverse check reaches the same finding as its paired reading on every fabrication and rewrite.
  3. A witness the operator does not hold (§6). We retrieved the model provider's stored copy of all 708 calls of the third vendor's runs. Each is byte-equal to the ledger, and both checks reach the same findings from it on all 84 runs.
  4. The trust boundary enforced (§6). On those runs the two writers use separate database roles and each run is confined to a network namespace whose only way out is the logging proxy. No outside connection succeeded.
  5. An adversary holding the rules (§6, Table 30). A model was given the rules, the exemption lists and the ledger rows and asked for one fabrication per record that the checks would miss, on 100 records in two rounds. With new text it never succeeded (65 attempts). Re-using a genuine sentence passes the committed check, on all 35 attempts. A stricter variant, repaired by one line and frozen, then faced the same attacker, told of the repair: it failed on all 50 attempts. We report the committed figures as primary.
  6. A harness that lies to the model as well (§6, Table 31). On 14 runs of a multi-turn harness a shim reported a failed test as passed, to model and record alike. The ledger's model side carries the lie; its environment side, which logs exit codes, reports it on all 11 runs where the lie fired and on none of 14 controls.
  7. A census of unaltered records (§2, Table 32). Of 365 records in the three main sets, the checks find 33 that would mislead a reader, 31 of them SWE-agent's. Run as released, SWE-agent discards a reply that carries more than one tool call, or a call with an argument it rejects, and saves neither the reply nor its own refusal.
  8. Post-hoc measures tested as predictions (§4, Tables 21 to 23). Five predictions were registered for the third vendor's runs. Four held, among them the gap between what an examiner claims as corroborated and what the rule finds (30 points; at least 10 predicted). The fifth, the dissociation criterion, missed, and the body no longer uses it.
  9. Readers on records that were false from the start (§4). Asked outright whether a record is a faithful account, two examiners could not tell a record with a reply left out from an honest one.
AuthorsOpenReview ↗

Response to all reviewers, details (2/2): results by format, corrections, scope

One record read alone

recordsfabricationsrewritesomissionsclean records flagged
third vendor, all rules committed first (3 formats)19/1916/1729/300/84
same design, cline8/105/513/130/28
same design, opencode13/135/78/8†0/28
held out, five formats38/4328/2843/441/115 (37/115)
held out: aider4/44/48/81/16 (4/16)
held out: mini-swe-agent7/94/49/90/22 (3/22)
held out: cline6/83/310/100/21 (3/21)
held out: opencode12/137/78/80/28
held out: SWE-agent9/910/108/90/28 (27/28)

How to read the last column. The checks exempt text a harness legitimately writes itself, in seven classes defined by the structure of a record, never by its content. On the third vendor's runs every class was committed before the first run, so 0 of 84 needs no qualification. On the held-out records three classes had been named beforehand and four were added after records were read; the figure in brackets counts only the three. Most of the 37 are one condition, a run's final tool output that is never sent back to the model, named beforehand for one format and extended to two more. §6 states this and takes the third vendor's runs as the clean measurement. Read against another run's ledger, the same rules flag all 199 clean records, so the low rate is not a rule too loose to fire (Table 29).

Where the reviewers were right, and what changed.

  • Results set by the design. §4 now says that the witnessed share (16 to 22%) is set by the design, since only the two anchors are written outside the harness. The abstract leads with the result that is not: every examiner overrates its own evidence, by 18 to 55 points.
  • The registered outcome. Handing a reader the ledger moves its corroborated share by +5.3 points for E1 on the primary runs and by −1.4 on the third vendor's. §4 reports this under the heading that a reader handed the ledger answers no better.
  • Domain provenance is written by the harness and is no longer counted as an outsider's property. §2 reports one property: none of sixteen frameworks keeps a record an outsider can check.
  • The activity correlation does not survive the removal of four dormant projects (0.57 to 0.04). The body makes no such claim.
  • Human checks. Two annotators agree with the audit instrument on 26 of 33 cells, one short of the 80% we registered; §2 says so, holds to the bar, and rests no claim on the properties they judged.

What we did not do, and why.

  • Repairs to the frozen rules. The rules-committed-first test now covers all five harnesses; cline and opencode were added during the review. Its misses there, two fabrications on cline and two rewrites on opencode, are of kinds §6 names. We did not repair them: a rule changed after the result is what the test excludes. † Read with the database export decoded; see Appendix G.
  • Tasks longer than seven minutes. The bound is reached on few runs, and longer tasks need a new task set; §7 states the scope.
  • Human re-judging of the audit's one reported property. Recruiting annotators again was not possible within the discussion period; §2 says the property was not in their sample.
  • Handing the checks' flags to the examiners (uvny Q2). With every rule committed first there are no false alarms to filter (0 of 84).
AuthorsOpenReview ↗

Thank you to all reviewers

Dear Reviewers 3E3M, uvny and 2Vpc, and Area Chair,

Thank you for your time, for the care in your reviews, and for coming back to read our replies and the revision. Your responses in the last days of the discussion encouraged us a great deal.

The hardest part of the revision was not the new experiments. It was finding a clear line through the material and telling one story a reader can follow the first time. That took far more effort than we expected, and your suggestions gave it its direction. We would like to name them.

Reviewer 3E3M

  • You asked us to evaluate the checks as a deployed auditor would: rules frozen first, one record read alone, held-out records, results by type and by format. This is now the paper's main evaluation (Section 6, Fig. 7). The paper's central claim no longer depends on a clean copy that no auditor has.
  • You suggested leading with the citation-provenance measurement. It is now the first result in the abstract and the first contribution in Section 1, and the paper is clearer about what is new in it.
  • You asked for a scope correction or a witness outside the operator. We tested the provider's own copy of every call, and Section 7 now states where the ledger is only a consistency check. The claim is narrower and we can defend all of it.

Reviewer uvny

  • Your presentation score told us the paper could not be followed, and we took it as the main instruction for the rewrite. The paper is now organised around one question, with every term defined where it first appears and one figure or table per page.
  • You pointed out that the check flagged half of the unedited logs and relied on an unedited copy. This led to the exemption classes and to the test with every rule committed first, where no clean record is flagged. The remedy is now measured the way an investigator would use it.
  • You asked who checks the provider when the provider runs everything. The paper now answers this directly and says plainly what our mechanism does not do.

Reviewer 2Vpc

  • You asked us to separate results that follow from the design from empirical findings. Each such result is now named as set by the design in the sentence that reports it, so a reader can see at once which numbers are findings.
  • You noted that the title's "trust" blurred unverifiable with untrue. The title changed, and the paper now gives the base rate of unaltered records that would mislead a reader.
  • You asked for an adversary that knows the check and for the activity correlation without the dormant projects. The first gave the paper its red-team result and an honest statement of the re-use gap; the second did not survive, and the claim is gone.

The paper is better in ways we would not have reached alone, and we are grateful for it.

Reviewer 3E3M, thank you also for the three notes for the final version. We agree with all three and will make them: the abstract will say that "none of 84 clean records" refers to the reverse check, with the query's findings on 15 of them stated; it will say that re-used genuine text passes the committed check and that the operator is trusted; and Section 6 will give the opencode omissions as registered, 3 of 8, beside the decoded 8 of 8.

We will keep polishing the paper in the same direction.

With sincere thanks, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.