- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies computational provenance: leaving evidence in a language model's output about the internal state used during generation. The authors augment frozen pretrained models with state-selection, insertion, and observation modules, and combine authenticated state recording with output encoding. They construct a task in which 16 internal states map to eight answers, allowing state identity to be distinguished even when the answer remains unchanged. The experiments verify state recovery through the constructed pathways in Gemma 3 1B and Qwen3-0.6B, and investigate extensions to natural activations and less-constrained text generation.
Critical strengths
- The observation that identical answers may arise from different internal states is well motivated. Connecting internal computation to evidence carried in the output is a meaningful research objective.
- The design clearly separates state prediction, causal intervention, execution authentication, and output recovery. Answer-preserving interventions appropriately test whether state identity can be distinguished beyond what the final answer reveals. (Sections 3–4)
Critical weaknesses
- There is a gap between provenance of the pretrained model's original computation and provenance created by the added pathway. Sections 3.1–3.2 and Appendix A.4 describe inserting the selector's state and training the added pathway with state-observation and answer losses. Showing that this state controls the answer verifies the constructed system, but does not establish that the original model used the same state. The additional benefit over directly recording the observed state as metadata under the same trust assumptions is also insufficiently explained.
- The guarantees provided by authentication and recovery are limited. As Section 3.4 acknowledges, HMAC protects the record produced by a trusted observer rather than independently verifying the correctness of state measurement. The public decoder recovers a state but does not authenticate the hidden execution. Appendix A.3 also states that the carrier configuration is not separately HMAC-bound in the receipt. The guarantees should be organized more clearly according to the required trust in the executor and observer.
- The practical utility of the confirmed output channel is not sufficiently validated. The example reports in Appendix A.3 include malformed text and special tokens, while Qwen recovery uses token IDs, a public nonce, and report indices rather than saved text. The less-constrained generation study fails the joint recovery, detection, and prose-quality requirements. The current result is therefore closer to verification of a constrained state-transmission channel than to preserving computational evidence in useful natural-language responses. (Section 5.5, Appendix A.9)
- The quality of the figures needs improvement. For example, in Figure 2, some text extends outside its text box.
Questions for authors
- What is the benefit over directly recording the observed state as metadata using the same trusted instrumentation? Please provide a concrete use case requiring output encoding and a corresponding comparison.
- Can the authors distinguish whether the added pathway preserves the original computation or replaces it with a newly trained state pathway? Are there end-to-end results for naturally occurring, semantically meaningful states?
- How are the receipt, observed state, output artifact, and carrier configuration bound together? Please clarify which guarantees remain when the executor or observer cannot be trusted.
3 comments
The rebuttal failed to address the core concerns, and the promised revised version has not been shared. Therefore, I will maintain my original rating.
Thank you for your follow-up. I would like to clarify that the revised manuscript has indeed already been uploaded to OpenReview and is available under the submission's Revisions tab.
The updated version incorporates the changes outlined in our responses, including revisions to the methodological framing, trust assumptions, limitations, and figures.
I apologise if the revised version was not immediately apparent, and would appreciate it if you could consider these changes in your assessment. If points have not been addressed, it would be beneficial to understand which points in particular were not fully addressed.
We thank the reviewer for their careful reading, and for the constructive comments.
The confirmed construction deliberately creates a new registered causal pathway; it does not establish that the pretrained model originally used the same state. We agree this distinction should be made clearer. The intervention experiments instead test whether this registered state becomes load-bearing rather than merely decodable. Answer-equivalent interventions alter the 16-way state while preserving the visible answer, while wrong-valid interventions require downstream behaviour, the authenticated record, and output recovery to follow the forced state rather than the earlier prediction.
We do not yet have end-to-end provenance for a naturally occurring state. The separate frozen-Gemma study provides bounded evidence of native causal influence, but because the intervention replaces the full residual vector and no stable output trace qualified for held-out recovery, we do not treat it as completed natural-state provenance.
The execution record and output carrier address different problems. A trusted record can state that execution (e) used state (z), but this information is external to the generated artifact and can be lost when the artifact is copied, stored, or passed downstream.
Output carriage asks whether that execution-state binding can persist with the artifact itself. The carrier does not replace the execution record: the record establishes the trusted observation, while the carrier propagates that binding beyond the execution. A concrete use case is an oversight system where a white-box monitor is available at inference time, but later auditors receive only the generated artifact rather than its original activations or logs. We will make this motivation clearer.
We agree that the guarantees should be organised more explicitly. The trusted observer measures the downstream state; the HMAC receipt authenticates the execution record and protects it against alteration, swapping, or replay; the authorisation stage permits the observed state to determine the carrier; and the public decoder recovers the carried state but does not independently authenticate the hidden execution. The carrier configuration is fixed by the registered protocol/manifest and is not separately HMAC-bound in the current receipt.
If the executor or observer is malicious, or measures the state incorrectly, HMAC does not repair that failure. The construction assumes trusted instrumentation and secure key control. We will make these trust boundaries clearer.
We agree that the current carrier is a high-reliability proof of concept rather than a deployment-ready natural-language mechanism. However, we believe this is a limitation of the present implementation, not of the provenance question itself. The key capability is that information about a causally operative internal state can remain bound to the generated artefact after the original execution environment and monitoring infrastructure are unavailable. This is useful when white-box monitoring exists at inference time but later auditors or downstream systems receive only the artefact. Making this carriage low-distortion, edit-robust, and natural is the next engineering challenge.
As well as improving readability, we will revise the paper so the scoping is clearer while making the contributions more explicit.