acceptodds
Under review as a conference paper at ICLR 2027

Computational Provenance in Pretrained Language Models

Abstract

Verifying why a language model produced a response requires knowing which internal states contributed to its generation, yet the resulting output alone does not reveal which activations were responsible. We extend computational provenance—evidence linking an output to the causally relevant internal computation that produced it—from purpose-built neural architectures to pretrained language models. Our construction forms a discrete state from frozen activations, makes downstream computation depend on it, authenticates a trusted observation of the state used, and carries that authorised state into constrained generated output from which it can later be recovered. In separate sealed evaluations, Gemma 3 1B achieved 98.34% state-prediction accuracy on 4,096 examples and Qwen3-0.6B achieved 95.70% on 256 examples; all accepted executions passed the registered causal, authentication, and recovery checks, while negative controls produced the required rejection or abstention. Separately, a state identified in unmodified Gemma activations passed a causal test, with 32/32 answer-preserving and 30/32 answer-changing interventions, but did not reach a valid test of recovery from ordinary generated text. In a developmental study, a less constrained text-generation variant recovered the authenticated state in 27/32 cases at top-1 and 32/32 at top-4, but did not meet all reliability requirements. These results establish end-to-end bounded computational provenance in two pretrained language models through an engineered finite-state pathway, and show partial progress toward naturally occurring states and less constrained text.

Primary area:
interpretability and explainable AI
Keywords:
computational provenance, language models, causal interventions, mechanistic interpretability, output watermarking, scalable oversight

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_G1NYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies computational provenance: leaving evidence in a language model's output about the internal state used during generation. The authors augment frozen pretrained models with state-selection, insertion, and observation modules, and combine authenticated state recording with output encoding. They construct a task in which 16 internal states map to eight answers, allowing state identity to be distinguished even when the answer remains unchanged. The experiments verify state recovery through the constructed pathways in Gemma 3 1B and Qwen3-0.6B, and investigate extensions to natural activations and less-constrained text generation.

Critical strengths
  • The observation that identical answers may arise from different internal states is well motivated. Connecting internal computation to evidence carried in the output is a meaningful research objective.
  • The design clearly separates state prediction, causal intervention, execution authentication, and output recovery. Answer-preserving interventions appropriately test whether state identity can be distinguished beyond what the final answer reveals. (Sections 3–4)
Critical weaknesses
  • There is a gap between provenance of the pretrained model's original computation and provenance created by the added pathway. Sections 3.1–3.2 and Appendix A.4 describe inserting the selector's state and training the added pathway with state-observation and answer losses. Showing that this state controls the answer verifies the constructed system, but does not establish that the original model used the same state. The additional benefit over directly recording the observed state as metadata under the same trust assumptions is also insufficiently explained.
  • The guarantees provided by authentication and recovery are limited. As Section 3.4 acknowledges, HMAC protects the record produced by a trusted observer rather than independently verifying the correctness of state measurement. The public decoder recovers a state but does not authenticate the hidden execution. Appendix A.3 also states that the carrier configuration is not separately HMAC-bound in the receipt. The guarantees should be organized more clearly according to the required trust in the executor and observer.
  • The practical utility of the confirmed output channel is not sufficiently validated. The example reports in Appendix A.3 include malformed text and special tokens, while Qwen recovery uses token IDs, a public nonce, and report indices rather than saved text. The less-constrained generation study fails the joint recovery, detection, and prose-quality requirements. The current result is therefore closer to verification of a constrained state-transmission channel than to preserving computational evidence in useful natural-language responses. (Section 5.5, Appendix A.9)
  • The quality of the figures needs improvement. For example, in Figure 2, some text extends outside its text box.
Questions for authors
  • What is the benefit over directly recording the observed state as metadata using the same trusted instrumentation? Please provide a concrete use case requiring output encoding and a corresponding comparison.
  • Can the authors distinguish whether the added pathway preserves the original computation or replaces it with a newly trained state pathway? Are there end-to-end results for naturally occurring, semantically meaningful states?
  • How are the receipt, observed state, output artifact, and carrier configuration bound together? Please clarify which guarantees remain when the executor or observer cannot be trusted.
3 comments

We thank the reviewer for their careful reading, and for the constructive comments.

There is a gap between provenance of the pretrained model's original computation and provenance created by the added pathway.

Showing that this state controls the answer verifies the constructed system, but does not establish that the original model used the same state.

Can the authors distinguish whether the added pathway preserves the original computation or replaces it with a newly trained state pathway? Are there end-to-end results for naturally occurring, semantically meaningful states?

The confirmed construction deliberately creates a new registered causal pathway; it does not establish that the pretrained model originally used the same state. We agree this distinction should be made clearer. The intervention experiments instead test whether this registered state becomes load-bearing rather than merely decodable. Answer-equivalent interventions alter the 16-way state while preserving the visible answer, while wrong-valid interventions require downstream behaviour, the authenticated record, and output recovery to follow the forced state rather than the earlier prediction.

We do not yet have end-to-end provenance for a naturally occurring state. The separate frozen-Gemma study provides bounded evidence of native causal influence, but because the intervention replaces the full residual vector and no stable output trace qualified for held-out recovery, we do not treat it as completed natural-state provenance.

The additional benefit over directly recording the observed state as metadata under the same trust assumptions is also insufficiently explained.

What is the benefit over directly recording the observed state as metadata using the same trusted instrumentation? Please provide a concrete use case requiring output encoding and a corresponding comparison.

The execution record and output carrier address different problems. A trusted record can state that execution (e) used state (z), but this information is external to the generated artifact and can be lost when the artifact is copied, stored, or passed downstream.

Output carriage asks whether that execution-state binding can persist with the artifact itself. The carrier does not replace the execution record: the record establishes the trusted observation, while the carrier propagates that binding beyond the execution. A concrete use case is an oversight system where a white-box monitor is available at inference time, but later auditors receive only the generated artifact rather than its original activations or logs. We will make this motivation clearer.

The guarantees provided by authentication and recovery are limited.

HMAC protects the record produced by a trusted observer rather than independently verifying the correctness of state measurement. The public decoder recovers a state but does not authenticate the hidden execution.

How are the receipt, observed state, output artifact, and carrier configuration bound together? Please clarify which guarantees remain when the executor or observer cannot be trusted.

We agree that the guarantees should be organised more explicitly. The trusted observer measures the downstream state; the HMAC receipt authenticates the execution record and protects it against alteration, swapping, or replay; the authorisation stage permits the observed state to determine the carrier; and the public decoder recovers the carried state but does not independently authenticate the hidden execution. The carrier configuration is fixed by the registered protocol/manifest and is not separately HMAC-bound in the current receipt.

If the executor or observer is malicious, or measures the state incorrectly, HMAC does not repair that failure. The construction assumes trusted instrumentation and secure key control. We will make these trust boundaries clearer.

The practical utility of the confirmed output channel is not sufficiently validated.

The current result is therefore closer to verification of a constrained state-transmission channel than to preserving computational evidence in useful natural-language responses.

We agree that the current carrier is a high-reliability proof of concept rather than a deployment-ready natural-language mechanism. However, we believe this is a limitation of the present implementation, not of the provenance question itself. The key capability is that information about a causally operative internal state can remain bound to the generated artefact after the original execution environment and monitoring infrastructure are unavailable. This is useful when white-box monitoring exists at inference time but later auditors or downstream systems receive only the artefact. Making this carriage low-distortion, edit-robust, and natural is the next engineering challenge.

As well as improving readability, we will revise the paper so the scoping is clearer while making the contributions more explicit.

The rebuttal failed to address the core concerns, and the promised revised version has not been shared. Therefore, I will maintain my original rating.

Thank you for your follow-up. I would like to clarify that the revised manuscript has indeed already been uploaded to OpenReview and is available under the submission's Revisions tab.

The updated version incorporates the changes outlined in our responses, including revisions to the methodological framing, trust assumptions, limitations, and figures.

I apologise if the revised version was not immediately apparent, and would appreciate it if you could consider these changes in your assessment. If points have not been addressed, it would be beneficial to understand which points in particular were not fully addressed.

Reviewer_fiR7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of Computational Provenance in LLMs, aiming to establish a verifiable link between the outputs generated by a model and the causally relevant internal states used during its execution. The authors construct a finite-state tracking framework on frozen Gemma3-1B and Qwen3-0.6B models. First, a linear classifier is used to predict discrete states from intermediate-layer activations. Then, trained state embeddings and residual stream injection are used to make downstream computation depend on the predicted state. Subsequently, a downstream classifier reads out the state, execution records are authenticated using HMAC, and a constrained token-generation mechanism encodes the state into the final output to support later recovery. To validate the framework, the authors design a synthetic task with 16 internal states and 8 possible final answers. Through state interventions that either preserve or change the final answer, they test the causal role of the state and its consistency throughout the entire tracking pipeline.

Critical strengths
  1. The research problem is important. This paper investigates how to establish a verifiable link between causal states inside language models and their final generated outputs. This problem has considerable research value and may potentially improve the interpretability and trustworthiness of language models.
  2. The experimental design is motivated. The paper constructs a controlled task in which different internal states correspond to the same final answer. It then uses answer-preserving and answer-changing interventions to validate the causal role of these states, allowing the experiments to distinguish changes in internal states from changes in the final answer.
Critical weaknesses
  1. Poor academic writing logic and usage of figures and tables. The abstract and introduction lack necessary method descriptions. Specifically, the abstract contains an excessive amount of experimental description, making it difficult for me to quickly understand the actual content of the authors' method and its relationship with the experiments, lacking a logical narrative. The quality of Figure 1 is extremely low, with disproportionate scaling. Figure 2 is of similarly poor quality, and the colors of the first and third legends are too easily confused. The information density provided by Table 1 is too low, making it unsuitable to appear in this section.
  2. The HMAC in the core construction is an unnecessary component and design. This component is included merely to satisfy the protocol proposed by the authors themselves. Since the current research problem and experiments are all conducted in a controlled environment, removing this component does not affect the experimental results and the current research problem.
  3. Regarding the problem of demonstrating a verifiable connection between the internal causal states of language models and the final generated output, models around the 1B size are not sufficient. There is a lack of research on larger-scale models, such as 4B, 7B, or larger sizes, because the internal causal states of different model sizes are highly likely to differ.
Questions for authors

See the Weaknesses above.

2 comments

We thank the reviewer for their careful review.

The HMAC in the core construction is an unnecessary component and design.

removing this component does not affect the experimental results and the current research problem.

We disagree with this assessment. While HMAC is not required to demonstrate causal dependence between an internal state and an answer, removing it would change what the full system establishes. The aim is not simply to recover a state from generated text, but to link that recovered state to an authorised observation of what the model actually used during execution. Without authentication, the output could carry an arbitrary state label, with no means of distinguishing it from one grounded in the observed computation. HMAC provides this authentication boundary; it is a standard cryptographic component, not a claimed novelty. Importantly, carrying this information in the output also allows the provenance information to accompany the generated artefact beyond the original execution environment. We will make this motivation clearer in our revised work as to not imply that the use of this was simply our construction but as a necessary and key component of this research.

Model scale. 

models around the 1B size are not sufficient.

There is a lack of research on larger-scale models, such as 4B, 7B, or larger sizes, because the internal causal states of different model sizes are highly likely to differ.

We agree that the current experiments do not establish scaling to larger models, and we agree that larger models would strengthen the evidence for generality. However, model size alone does not determine whether a causal-state provenance mechanism is valid. Our experiments test distinct requirements: whether a state can be formed from pretrained activations, made causally relevant to downstream computation, independently observed, and subsequently recovered from the output. The present contribution is a bounded feasibility result: the complete registered construction was prospectively validated in two frozen pretrained model families using different model-specific interfaces. The fact that Gemma and Qwen require different activation sites and state interfaces already indicates that the implementation should not be expected to transfer unchanged as model scale or architecture changes. We therefore do not claim scale invariance or direct architectural portability, and instead our supported claim is that the same abstract provenance architecture can be realised in more than one pretrained model family. We however do appreciate this assessment and we will make adjustments clarifying the scope of this research to specifically smaller language models. To add, testing whether the same construction remains practical and reliable at 4B, 7B, and frontier scale is an important next step, but we do not believe that larger models are required to establish the present mechanistic feasibility claim.

Research workflow and figures. 

The abstract and introduction lack necessary method descriptions.

the abstract contains an excessive amount of experimental description

The quality of Figure 1 is extremely low, with disproportionate scaling. Figure 2 is of similarly poor quality, and the colors of the first and third legends are too easily confused.

The information density provided by Table 1 is too low, making it unsuitable to appear in this section.

We acknowledge the concern regarding the appearance of the figures and will replace them with clearer, carefully reviewed versions. However, we would distinguish presentation quality and the use of disclosed AI assistance from the scientific validity of the work. The experimental claims rest on explicit interventions, independently checked execution records, negative controls and prospective evaluations, rather than on the manner in which the manuscript or figures were prepared. We are happy to clarify the research workflow further where useful. We also acknowledge that the paper could communicate its contribution more effectively, particularly in the abstract, introduction and figures. We will revise these sections to explain the method more directly, improve Figures 1–2, and reconsider the placement of Table 1.

We believe these revisions will address the presentation concerns while making the technical contribution and its limitations more precise.

Based on the current rebuttal from the author, I will not change my judgment or my recommendations.

Reviewer_m7n9OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends computational provenance Belay et al from purpose specific networks to frozen LLMs (Gemma3, Qwen3). The main idea is that a generated output should carry recoverable evidence of relevant internal states that produced it.

Main results: Gemma reaches 98.34% state accuracy and Qwen 95.70% on sealed confirmations. Two extension studies are also reported with partial results.

Critical strengths

Exceptional rigor. Disciplined methodology and honest claims, negative results are retained and reported.

Critical weaknesses

I am not sure if this paper will scale, the experiments are limited to 0.6B–1B models, one synthetic task, 4 bits, and in-distribution evaluation only.

The results appear to be be largely true by construction, so then it says little about the main claim on provenance in pre-trained LLMs.

The motivating use case is safety (detecting deception or reward-hacking) but evidence is weak (e.g no solid threat model articulated).

Questions for authors

Report recovery under retokenization for Qwen, or under light truncation/rephrasing?

How does the selector's accuracy compare with a trivial baseline?

What is model's accuracy on the transfer task without inserted pathway?

Under what scenarios does output carriage support that the HMAC signed trusted record alone does not?

Reviewer_bLwwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The proposed work evaluates how generated outputs can carry verifiable evidence of internal states that produced them. The authors propose a method for frozen open weight models by constructing a pathway where discrete 16-way internal states are extracted from residual activations, re-inserted downstream to steer computation and encoded into generated outputs. The proposed approach results in high state prediction accuracy across multiple sealed evaluations for both Gemma 3 1B and Qwen3-0.6B models.

Critical strengths
  • Extension of computational provenance for frozen pretrained language models (Gemma 3 1B and Qwen3-0.6B)
  • Utilizes a branch task structure where 16-way internal states map to 8-way visible color answer. Answer preserving interventions prove that output recovery tracks the causally active internal state rather than the final answer.
  • High state prediction accuracy on Gemma3 31B and Qwen3-0.6B with 100% exact state recovery in sealed evaluation settings.
Critical weaknesses
  • Architectural portability of the method is limited, since Qwen and Gemma require different layer locations, token positions and decoding mechanisms.
  • Evaluation is limited to synthetic 16 subject task with limited possibility of transfer to unseen entities, ontologies or reasoning structures.
  • The token partitioning results in visibly malformed and unnatural text outputs.
  • The framework relies on an added trained pathway rather than proving provenance for the model’s naturally occurring process.
Questions for authors
  • Since the end-to-end framework relies on trained 16-state pathway rather than verifying naturally occurring model states, is it fair to claim the term “provenance in pretrained language models” ?
  • The hard-token partitioning scheme produces unnatural text. If the generated textual outputs are not of high quality, what are some possible realistic use cases ?
2 comments

We thank the reviewer for their reading of the work, and for their comments and suggestions.

Since the end-to-end framework relies on a trained 16-state pathway rather than verifying naturally occurring model states, is it fair to claim the term "provenance in pretrained language models"?

We believe the term remains appropriate, although we agree the scope should be stated more precisely. We agree that the confirmed experiments don’t establish natural provenance of the models' original unmodified computation, but “provenance in pretrained language models” does not require that the provenance variable itself to have arisen naturally during pretraining. Instead, we ask whether the relevant question is if provenance can be engineered for an execution whose underlying language model is pretrained and frozen. Importantly, although the pathway is deliberately introduced, its required properties are not guaranteed by construction; we observed cases where states were successfully transported but did not control the final answer, and where output recovery failed. The contribution is therefore not merely the introduction of a pathway, but the experimental validation of its causal and provenance properties with the models they are intertwined with. Whilst we do explore a bounded extension towards natural provenance, we will make an effort to emphasise “engineered provenance” whilst still highlighting that this is distinct from naturally occurring states.

Architectural portability of the method is limited, since Qwen and Gemma require different layer locations, token positions and decoding mechanisms.

We agree that the current work does not establish direct mechanism portability as Gemma and Qwen require model-specific insertion sites, observation sites, and output procedures. The cross-model result is therefore better described as realisability of the same abstract provenance protocol in two pretrained model families, rather than a single plug-and-play mechanism whose parameters transfer between architectures. We believe we address this fairly, however we will outline this in a stronger manner and make this clearer in the revised version; we will add towards the discussion as to why we believe that we were not successful in portability methods.

Evaluation is limited to synthetic 16 subject task with limited possibility of transfer to unseen entities, ontologies or reasoning structures.

We agree that these results do not yet establish generalisation to unseen entities, more complex reasoning or naturally occurring safety-relevant states. However, the contribution of the present experiment is not broad task generalisation and is instead to establish whether a causally operative internal state can be bound to recoverable evidence in the output, under a setting where that claim can be tested unambiguously. The 16-state/8-answer construction is specifically designed to make this possible: it creates pairs of executions that have the same visible answer but different internal causal states, so successful recovery cannot be explained by answer information alone. This makes the synthetic setting a controlled test of the provenance mechanism, which now enables a deep exploration into growing these capabilities to richer settings, which was out of scope with our research.

The token partitioning results in visibly malformed and unnatural text outputs.

If the generated textual outputs are not of high quality, what are some possible realistic use cases?

We agree that the current token constraints produce unnatural text. The confirmed token-partition carrier is intentionally a high-reliability proof-of-concept channel, prioritising exact state recovery and controlled verification, rather than natural-language quality. We do not present this encoding as a deployment-ready mechanism. Its role in this paper is to test whether an authenticated causal-state identity can survive into an exported output under a fully specified and independently recoverable protocol, hence focussing on exact carriage and recovery over fluency. We also tested less-constrained generation, but did not promote it to a confirmed result because it failed the joint recovery, detection, and prose-quality requirements, in which we retained this negative result rather than relaxing those requirements.

We will make this limitation more prominent and adjust the language to avoid implying that the current carrier is ready for natural open-ended deployment. Low-distortion and edit-robust carriage still remains future work.

As well as the above, we also take into account the readability of the paper and will revise the presentation, and to also distinguish the demonstrated result from its longer-term applications, while retaining the central contribution.

Reviewer_bLwwOpenReview ↗

Response to authors

Thanks to the authors for their detailed response. Based on the rebuttal it is not clear how the language quality of the outputs can be improved. Hence I want to keep my original rating.

Comments on the paper

1 comment

We thank all reviewers for their detailed feedback. As well as individually responding to each of the reviewer's comments, we have revised the manuscript to address the concerns around the scope of the claim, the engineered pathway, cross-model portability, the synthetic evaluation setting, the role of HMAC, the trust boundary, and the constrained output carrier. We have also revised the figures and simplified the surrounding language.

1. Engineered provenance vs. original model computation

Several reviewers noted that the confirmed system uses an added trained pathway rather than establishing provenance for a naturally occurring state in the original pretrained model. We agree this distinction needed to be clearer.

We now state:

“We do not assume that the registered state is a naturally occurring causal variable used by the unmodified model. Instead, it is derived from representations in the frozen pretrained models and made causally relevant through the added pathway.”

The Discussion further clarifies:

“The interventions show that this state controls the registered answer behaviour, rather than showing that the pretrained models originally relied on the same variable.”

The separate frozen-Gemma experiment is therefore presented only as a bounded extension towards natural provenance.

2. The result is not guaranteed by construction

Although the pathway is deliberately engineered, the required provenance properties are tested independently and earlier configurations failed at different stages. We now state:

“Although deliberately introduced, the pathway does not guarantee successful provenance; its components are tested separately, and earlier configurations failed at different stages (Appendix A.8).”

Thus, introducing the pathway does not itself establish causal answer control, authenticated state recording, or successful output recovery.

3. Cross-model scope

Gemma and Qwen use different model-specific interfaces, insertion sites, and output procedures. We now state:

“Rather, they establish realisability of the same abstract provenance protocol in two pretrained model families using model-specific interfaces.”

We therefore do not claim direct mechanism transfer or scale invariance, and larger-model validation remains future work.

4. Controlled synthetic setting

The 16-state/8-answer task is intended as a controlled provenance test rather than a broad generalisation benchmark. Its structure separates state identity from visible answer identity, allowing recovery to be tested when different states produce the same answer. We now clarify that the present result does not establish generalisation to richer tasks, unseen ontologies, naturally occurring safety-relevant states, or complete model reasoning.

5. HMAC and the trust boundary

We now clarify that HMAC is not used to establish causal dependence and is not a claimed novelty:

“HMAC is not needed to establish causal dependence, but it is needed to authenticate the execution record that authorises the output encoding. Without it, a recovered state could not be distinguished from an arbitrary or altered state label unrelated to the observed execution.”

We also retain the limitation that HMAC cannot establish that the state was measured correctly, so trusted observation, authorisation, and secure key control remain assumptions.

6. Output carriage vs. metadata

We clarify that the execution record and output carrier serve different purposes. The record authenticates the execution-time observation, while carriage allows the state association to persist with the exported artefact:

“Unlike an execution record stored separately as metadata, output carriage preserves the state association with the exported artefact after it leaves the original execution environment.”

The public decoder still does not independently authenticate the hidden execution.

7. Practical scope of the carrier

We agree that the confirmed carrier is not a deployment-ready natural-language mechanism. We now state:

“The confirmed carrier deliberately prioritises exact state transmission and independently testable recovery over natural-language quality, and should therefore be understood as a high-reliability proof-of-concept channel rather than a deployment-ready text mechanism.”

We have also redesigned the figures and improved their scaling, visual distinction, and surrounding explanation. We hope that the reviewers can take into account the revisions made.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.