acceptodds
Under review as a conference paper at ICLR 2027

Bringing Model Internals into Expert Annotation of Legal Reasoning

Abstract

Large language models (LLMs) increasingly perform complex professional work, creating a need to examine the reasoning behind their decisions. In legal tasks, written rationales alone may provide an incomplete basis for expert oversight. We introduce an activation-informed expert annotation framework in which legal experts inspect generated reasoning alongside natural-language explanations derived from model activations. Three experts contributed 34 hours of review, producing 354 span-level legal annotations across 15 conversational and agentic trajectories, together with explanation-helpfulness ratings for eight trajectories. To supply explanations across models, we propose Cross-LLM Activation Surrogacy: an accessible surrogate model reads a target model’s trajectory, and a pretrained Natural Language Autoencoder (NLA; Fraser-Taliente et al., 2026) describes the surrogate’s resulting activations. This approach requires neither access to the target’s activations nor training a dedicated NLA for each target. Experts judged NLA-based explanations helpful in 41.4% of their ratings. In classification of previously identified legal errors, aligned surrogate explanations improve macro-F1 from 0.257 to 0.324 relative to providing no explanations, with gains also observed over text-only and shuffled-explanation controls. Cross-model comparisons show stronger semantic correspondence between explanations within model families than across families, while reconstruction similarity alone does not reliably track this correspondence. Together, these results establish a practical framework for activation-informed expert annotation, with initial evidence that aligned surrogate explanations support classification of expert-identified legal errors. Code and dataset are available at: https://anonymous.4open.science/r/tr3D8F.

Primary area:
interpretability and explainable AI
Keywords:
expert annotation, cross-model explanation transfer, legal reasoning, process supervision, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2TTEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an expert annotation workflow that presents legal trajectories alongside natural-language explanations of surrogate-model activations. Kimi-K2.6 generates trajectories, Gemma3-27B processes them, an existing NLA verbalizes Gemma activations, and Kimi summarizes these explanations into token-cited claims. Three legal experts contributed 354 span annotations across 15 trajectories and explanation ratings for eight trajectories. Experiments examine explanation correspondence among three open models and classification of already identified legal errors. On 280 error spans in 14 task/document groups, aligned explanations improve macro-F1 from 0.2568 to 0.3243 relative to no explanations, with positive comparisons against text-only and shuffled controls.

Critical strengths
  1. A useful exploratory connection between interpretability and professional annotation. The interface combines localized legal-error judgments, written feedback, and separately rated explanation claims with links to their token-level sources. The study covers conversational and agentic tasks and documents expert coverage, compensation, and review effort. This is an interesting application of activation verbalization to a concrete professional workflow.
  2. Controlled initial utility evidence and careful reporting of limitations. The main classification experiment shares grouped splits and training settings across five conditions, matches explanation-field budgets, and uses paired document-bootstrap intervals. The aligned condition improves macro-F1 by 0.0675, with a 95% interval of [0.0195, 0.1009]. The authors distinguish diagnosis of known errors from detection and human improvement, report unsuccessful helpfulness filtering, and explicitly acknowledge that faithfulness to Kimi's internal processing remains unverified.
Critical weaknesses
  1. The specific value of activation-derived information remains confounded with the explanation-generation pipeline. Appendix D.1 generates text-only explanations with Qwen3-4B, whereas Appendix B.1 uses a Gemma3-27B NLA verbalizer followed by Kimi-K2.6 summarization. The text baseline receives local context and limited document excerpts; matching its accessible history to the surrogate pathway is not established. Matching output token budgets does not match model capacity, context, or summarization. Thus C>B supports this particular pipeline comparison, and C>E supports alignment, but neither isolates the activation pathway from a stronger text-analysis pipeline. The correspondence experiment does not settle this: both models process the same trajectory, shuffled explanations already reconstruct Gemma targets at 95–98% cosine similarity, and semantic agreement lacks a same-context text-explanation reference. Its binomial null of 0.5 is not an established chance-agreement baseline, and positions from the same source example may be dependent. I appreciate the explicit faithfulness caveats; the concern is insufficient evidence for the method's distinctive value, rather than an allegation that the authors claim to have established Kimi faithfulness.
  2. The legal-label resource lacks an independent reliability reference. Experts assign error labels while viewing both NLA claims and judge feedback, and these labels later serve as the prediction target. Improved agreement with such labels need not establish improved diagnosis against independently confirmed legal errors. This is not evidence of direct label leakage, but it makes annotation reliability important. The reported 82.26% agreement concerns 62 explanation-helpfulness claims, not legal-error localization or categories. One expert supplies approximately 73% of explanation ratings and 62% of legal-span annotations. Figure 3b also demonstrates that misleading ratings can reflect disagreement with the underlying legal judgment despite accurate paraphrasing. The paper acknowledges these distinctions, but the resulting resource still needs legal-label agreement or a limited independent audit. With only 14 evaluation groups and NLA coverage on 96 of 280 spans, the scope and distribution of the classification gains also remain important uncertainties.
Questions for authors
  1. Can you compare C against text-only explanations generated with matched observable history, comparable model capacity, and the same Kimi summarization procedure on the existing evaluation groups? Please specify the accessible context in each pathway and report results separately for the 96 NLA-covered spans and the other 184, together with per-category and document-level effects.
  2. In the existing open-model evaluation, how well do same-context text-only explanations agree with target NLA explanations? Can semantic comparisons retain source-example clustering, and can reconstruction include a mean-activation reference or centered variance-explained metric? These analyses would help distinguish shared-context correspondence from target-specific information without requiring access to Kimi activations.
  3. For the four trajectories with repeated expert review, what agreement is observed in legal-error localization and labels, and how are disagreements handled? Is any limited independent legal audit available? If not, please keep the resource's interpretation explicitly tied to assisted expert judgments and distinguish helpfulness from correctness and annotation improvement.
5 comments
AuthorsOpenReview ↗

Authors' response to Reviewer 2TTE [1/3]

Thank you for your comprehensive feedback and suggestions! We appreciate that you found our idea of bringing interpretability into annotation promising. We address your comments and concerns below.

A main and shared concern was the size and composition of the expert data, and we understand they matter. We kept collecting expert annotations after submission. The data now covers 24 trajectories (15 in the submission, +60%), 519 error spans (354, +47%), 8,110 claim ratings (2,892, +180%) and 57.3 expert hours (34.3, +67%). Expert B's share of claim ratings fell from 73% to 57%. All new analyses below use this full data.

W1/Q1. Unclear gain from activations and unbalanced setup

We acknowledge the mismatch between the classifier conditions in section 3.2 and revised the setup so that they differ only in the source of the explanations (external text-based vs. internal NLA-based):

  1. Text-only explanations (B): Gemma3-27B-IT, the same model as the NLA surrogate, writes token-level explanations for each span from the trajectory text up to each token.
  2. Aligned NLA (C): Gemma3-27B-IT's NLA explains the surrogate's activations from a forward pass over the same text. We give the classifier these token-level explanations directly instead of Kimi-summarized claims, so neither condition passes through a third model.
  3. Classifier input: in every condition, the task, the span, and its local context, plus that condition's explanations.
  4. Data: all 24 documents collected to date (the paper's 14 groups plus 10 new ones); intervals are paired document-bootstrap 95% CIs. Starred CIs are statistically significant. All spans (511 after merging identical ones) and their explanations are now covered.

Results (macro F1):

A: noneB: text-onlyC: NLAE: NLA, other positions
0.2550.2680.2920.243
  • C − B: +0.023 [−0.020, +0.062].
  • C − A: +0.036 [+0.001, +0.070]*; C − E: +0.049 [+0.012, +0.082]*.
  • Subsets: C − B stays within the interval for every subset: exposure to NLA claims, span type (reasoning, answer, document), inside vs. outside the rubric judge's scope, and benchmark (LAB, PRBench).
  • Per document: C's macro F1 exceeds A's in 16 of 24 documents and B's in 15 of 24.
  • Per category (F1):
CategoryLabelsABC
Incomplete reasoning2580.5520.6070.560
Fact misread920.2720.2660.313
Wrong rule760.2740.2940.306
Misapplied rule660.1470.1760.200
Other590.1670.1590.250
Missed rule280.1210.1090.122
  • By exposure to NLA claims (macro F1; see next comment on W2):
SubsetSpansABCC − A [95% CI]C − B [95% CI]
Exposed2420.2680.2780.290+0.021 [−0.025, +0.073]+0.012 [−0.050, +0.073]
Unexposed630.2610.3350.330+0.069 [−0.040, +0.167]−0.005 [−0.055, +0.090]
Less exposed2060.2380.2370.278+0.040 [−0.004, +0.066]+0.041 [−0.025, +0.087]
Unexposed + less exposed2690.2430.2530.287+0.044 [+0.004, +0.077]*+0.033 [−0.022, +0.075]
  • With equivalent setups, the NLA-based classifier has the best macro F1 overall (0.292) and in most subsets. Its gain over no explanation is statistically significant overall and on spans labeled without exposure to NLA claims, though its gain over text-only explanations is not significant.
  • We also observed that Gemma, as an external explainer (in B), often stopped following the explanation instructions on long trajectories, so we repeated the instructions after the trajectory.
  • The NLA does not face this issue, since it reads one activation vector per token rather than the text.

We see this as a preliminary study of a new setting. Where downstream results are weak or mixed, we think they can be addressed or improved by:

  1. More expert labels: Each classifier trains on about ~490 spans, and rare categories such as missed rules (28 labels) stay hard for every condition. We want to expand the setup to have more legal experts to support more data collection and better cross-expert matching for agreement and reliability. More fine-grained labelling of legal spans and NLA claims would be helpful to see which explanations actually help downstream and/or help experts' judgement on the data.
  2. Explaining the target directly: While our surrogacy approach shows useful features/explanations for their target model, it would be interesting to see if using explanations from the target’s own NLA helps classification and downstream tasks even further.

We really appreciate highlighting the setup mismatch and helping us to show clearer results; the updated setup and data helped us isolate and clarify the gain from activations and NLA much more and clearly define what we can do in our next steps.

AuthorsOpenReview ↗

Authors' response to Reviewer 2TTE [2/3]

W2. Controls for anchoring on claims/judge

To clarify, the rubric judge saw only output documents (LAB) or the final answer (PRBench), so its feedback never covered the reasoning, which holds 63% of all error spans (327 of 519).

For NLA claims, interaction logs from the third collection week onwards record whether an expert opened a span's NLA explanations before saving its label. Based on our annotation metadata, we split the 519 spans into three subsets:

  1. Exposed (244): activity metadata shows the expert opened, viewed, or rated NLA claims before labeling the span.
  2. Unexposed (64): activity metadata shows the expert did not open the NLA claims before labeling the span.
  3. Less exposed (211): in the first 2.5 weeks of annotation, before activity metadata was recorded, we did not require experts to view or rate NLA claims; they recorded only 1 claim rating across these reviews. Their annotation rate was also about double the later rate (~16 vs. ~7 spans/hour).

Thank you for allowing us to elaborate on the data composition and anchoring on NLA/Judge. We will include a flag on each span for their exposure/anchoring to NLA or rubric-judge claims in our final data release.

W1/Q2. Random control for judge agreement

  • Chance baseline: we paired each target explanation with a surrogate explanation from a different example of the same legal task, so the legal task alone cannot drive agreement.
  • Dependence between positions: confidence intervals resample whole source examples (cluster bootstrap over the 260 source examples), so positions from the same example are not treated as independent.

Rates are the share of pairs judged by Kimi to have the same or partially the same meaning, with 95% CIs:

Target → SurrogateReal pairsRandom pairs
Within family
Gemma-27B → Gemma-12B81.0% [77.5, 84.3]7.4% [5.1, 9.9]
Gemma-12B → Gemma-27B80.8% [77.3, 84.2]8.4% [6.0, 11.0]
Cross family
Qwen-7B → Gemma-12B55.6% [51.5, 59.9]3.6% [2.0, 5.5]
Qwen-7B → Gemma-27B53.8% [49.4, 58.2]3.8% [2.2, 5.6]
Gemma-12B → Qwen-7B56.8% [52.5, 61.0]4.2% [2.5, 6.0]
Gemma-27B → Qwen-7B55.2% [51.1, 59.4]4.8% [3.0, 6.7]

Pooled, real pairs agree at 80.9% [78.5, 83.3] within family and 55.4% [52.7, 58.0] across families, against 5.4% [4.3, 6.6] for random pairs.

Thank you for requesting this addition; it helped us directly separate surrogate explanations from a clearly defined random baseline.

W1/Q2. Text-only control for surrogate-target agreement

We followed the same updated setup we used for text-only condition B from §3.2 into the semantic agreement setup, where we also give it the full causal context and ask it to explain the last token. Both judges favor the surrogate agreement with the target's NLA versus the text-only.

Target → SurrogateKimi: surrogate NLAKimi: text-onlyGPT-5.4-mini: surrogate NLAGPT-5.4-mini: text-only
Within family
Gemma-27B → Gemma-12B81.0%63.6%58.2%54.8%
Gemma-12B → Gemma-27B80.8%61.4%55.4%47.8%
Cross family
Qwen-7B → Gemma-12B55.6%37.4%38.2%33.6%
Qwen-7B → Gemma-27B53.8%41.4%31.6%34.2%
Gemma-12B → Qwen-7B56.8%31.4%34.0%23.8%
Gemma-27B → Qwen-7B55.2%33.0%38.4%30.0%
Pooled63.9%44.7%42.6%37.4%

We appreciate your ask for this addition; the surrogate explanations are directly shown to align better with the target than external explainers with full text context.

W1/Q2. High reconstruction for shuffled explanations and metric inflation

Thank you so much for highlighting the mean reference/centering for reconstruction, as cosine was indeed inflated by the mean direction of the activations (0.58-0.985). We updated the reconstruction metric to be Fraction of Variance Explained (FVE), following your comment and the original NLA reference, instead of cosine similarity. Using FVE (where 0 means no better than predicting the mean activation) shows how all shuffled explanations have negative FVE (−0.56 to −0.94), while surrogate explanations score close to the targets' own NLA (0.44–0.63 vs. 0.66–0.69):

Target → SurrogateFVE: target's own NLAFVE: surrogateFVE: shuffled
Within family
Gemma-27B → Gemma-12B0.690.58−0.65
Gemma-12B → Gemma-27B0.660.63−0.67
Cross family
Qwen-7B → Gemma-12B0.680.52−0.94
Qwen-7B → Gemma-27B0.680.55−0.93
Gemma-12B → Qwen-7B0.660.45−0.60
Gemma-27B → Qwen-7B0.690.44−0.56

Thank you again for this helpful comment; this adjustment significantly improved the alignment of our results by removing the previous inflation.

AuthorsOpenReview ↗

Authors' response to Reviewer 2TTE [3/3]

W3,4. Utility to experts and label agreement and handling

We acknowledge not studying the direct utility to annotators in depth. We found the process itself is difficult at our scale in terms of getting the experts' feedback, time, and aligning them to samples in legal tasks they are familiar with while matching them with each other to collect label agreement data. This also limits our ability to do an independent legal audit within the rebuttal stage, so we will describe the labels as assisted expert judgments and distinguish helpfulness ratings from correctness and annotation improvement.

  • From collected feedback from the 3 experts, A and C found that the claims mostly restate the trajectory text, while B found them useful for checking and verifying it. We will include their exact feedback in the paper.

  • We will also include a crossover study on the direct utility of explanations for expert annotators as future work.

  • For data handling, we merge identical spans and use the union of their labels.

  • Claim-rating agreement: our submission reported 82% agreement between B and C on 62 claims (κ\kappa = 0.60). Since then, A and B co-rated 1,568 claims, with 37% agreement (κ\kappa = 0.00), because A rated 99% of claims neutral.

  • Span and label agreement: 7 trajectories have now been reviewed by two experts (4 in the submission). Each expert chooses where a span starts and ends; on these 7 trajectories, 32 of 210 spans overlap a span by the other expert (F1 0.15), and 3 of 21 overlapping span pairs share a label, likely because each expert highlights different reasoning issues given their background. We share two cases below:

  1. Same error, different label (playbook review). Expert B, tagged "other: missed rule": "Model fails to recognize the 72-hour notice rule in the playbook." Expert A, tagged incomplete reasoning: "Does not state that there is nothing in the agreement related to the 72 hour notice requirement as required by the playbook."
  2. Different focus (clinical-trial cost disclosure). Expert B flags incorrect citations, e.g., "should be 21 CFR 50.25(a)(7)". Expert A flags a missing framework: "45 CFR 46.117 is the overarching statute related to informed consent and should be included in the analysis."

Thank you for highlighting this limitation; these cases helped us demonstrate how each expert approaches the legal errors and how their perspectives can differ in some of these cases.

Reviewer_KHhhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework for expert annotation of legal reasoning using natural-language explanations of model activations. To accommodate targets whose activations are unavailable, an accessible surrogate processes the target’s trajectory, and a pretrained Natural Language Autoencoder explains the surrogate’s activations. Three legal experts provide 354 span-level annotations across 15 trajectories, alongside explanation-helpfulness ratings. Experiments examine correspondence between target and surrogate explanations and their utility for classifying expert-identified errors. Aligned explanations improve macro-F1 from 0.257 to 0.324, although their faithfulness to the main target model remains unverified.

Critical strengths

1.The annotation workflow addresses a practical problem: activation explanations are difficult for domain experts to inspect at token level. Summarizing them into claims with links to their underlying explanations makes the material more manageable and gives experts a way to check the summaries.

2.The expert dataset is a useful contribution. Collecting 34 hours of legal review is a substantial effort, and the combination of localized error labels, written feedback, and explanation ratings offers more detail than answer-level correctness judgments alone.

3.The downstream experiment includes meaningful controls. The document-group holdouts, matched explanation budgets, and comparisons against both text-only and shuffled explanations make the classification improvement worth taking seriously. I also appreciate that the paper explicitly reports the limitations of reconstruction similarity and distinguishes error diagnosis from detection.

Critical weaknesses

1.My main concern is what the surrogacy experiments actually establish. Agreement between two NLA explanations does not show that either explanation captures the computation responsible for the target’s answer. Cross-family agreement is only 54.6–58.6%, and shuffled explanations already achieve very high reconstruction similarity for Gemma targets. More importantly, correspondence is not measured for Kimi–Gemma, the pair used in the expert study. The results support useful surrogate features more clearly than they support explanations of the target’s internal reasoning.

2.The text-only baseline does not adequately isolate the contribution of activations. Its explanations are generated by Qwen3-4B, whereas the NLA claims pass through a different pipeline that includes Kimi summarization. Matching token budgets helps, but differences in generator capability and explanation style remain. A text-only control using a comparable generation and summarization pipeline would make the attribution much more convincing. As presented, Table 3 establishes a benefit from the full explanation pipeline, with less clarity about which component supplies it.

3.The benefit to expert annotation is still unclear. There is no comparison measuring whether experts find more errors, make better judgments, or finish faster with these explanations. Helpfulness ratings cannot answer those questions, especially given the examples where ratings reflect disagreement with the underlying legal conclusion. Since experts also saw explanations and judge feedback while producing annotations, the downstream labels are not an independent test of whether the explanations improve expert oversight.

4.The evidence is narrow relative to the proposed framework. The classification experiment has 14 document groups, and only 96 of its 280 spans receive NLA claims. Results on that covered subset and by error category would help show where the gain comes from. The expert study also involves only three reviewers, without a reported agreement measure for legal-error annotations. These limitations leave substantial uncertainty about whether the improvement carries across legal tasks and annotators.

Questions for authors
  1. Clarification on the text-only control. The text-only explanations use Qwen3-4B, while the NLA claims include a Kimi summarization step. Is there a comparison that controls for this difference in the generation pipeline? I would like to understand how much of the gain can be attributed specifically to activation-derived information.
  2. Clarification on annotation independence. Were any legal-error annotations collected before experts saw the NLA claims or judge feedback? If so, does the classification improvement hold on that subset? This would help assess whether the explanations provide useful information beyond what may already have influenced the reference annotations.
  3. Additional analysis of existing results. Could you report the paired performance difference on the 96 spans with NLA coverage, including per-category results and variation across document groups? Since the other 184 spans receive identical inputs across conditions, this breakdown would help establish whether the improvement is reasonably consistent or driven by a small number of tasks or error categories.
4 comments
AuthorsOpenReview ↗

Authors' response to Reviewer KHhh [1/3]

We sincerely thank you for your thoughtful feedback and suggestions. We are glad you found our approach of bringing summarized explanations into annotation, and the dataset, useful. We address your questions and concerns below.

A main concern shared across reviews was the size and composition of the expert data, and we agree it matters. We kept collecting expert annotations after submission. The data now covers 24 trajectories (15 in the submission, +60%), 519 error spans (354, +47%), 8,110 claim ratings (2,892, +180%) and 57.3 expert hours (34.3, +67%). Expert B's share of claim ratings fell from 73% to 57%. All new analyses below use this full data.

W1. What surrogacy establishes, and untested Kimi → Gemma surrogacy

We appreciate your concern about what surrogacy establishes, and thank you for suggesting how to frame it. We agree that our results show the surrogate's explanations correspond to the target's and provide useful surrogate features, not that they explain the target's internal computation. We will revise the framing throughout.

Kimi → Gemma. We are afraid that including Kimi is impractical as there is no public NLA for Kimi (or for any model of similar scale and capability), and training one ourselves means joint training and RL over two copies of a 1-trillion-parameter model. We're sorry that this is not possible under our current resources. We also chose a frontier-scale target model so that the collected reasoning data is useful for real-world legal work, for two reasons:

  1. In initial experiments with smaller models such as Qwen3-4B, our experts found that these models surface few non-trivial legal reasoning errors, even on older, mostly closed-ended benchmarks such as LegalBench.
  2. On recent benchmarks that represent real legal workflows, such as Harvey's LAB, frontier models and agents reach at most 27.08% end-to-end success, so smaller models rarely produce meaningful reasoning or agentic trajectories.

Cross-family agreement. We added a random-pair baseline and a text-only control (following our updated setup in W2, W4/Q1, Q3 in the next comment). Cross-family agreement (53.8–56.8% with Kimi) is far above the random-pair rate (3.6–4.8%) and above the agreement of text-only explanations (31.4–41.4%):

Target → SurrogateKimi: surrogate NLAKimi: text-onlyKimi: random pairs
Within family
Gemma-27B → Gemma-12B81.0%63.6%7.4%
Gemma-12B → Gemma-27B80.8%61.4%8.4%
Cross family
Qwen-7B → Gemma-12B55.6%37.4%3.6%
Qwen-7B → Gemma-27B53.8%41.4%3.8%
Gemma-12B → Qwen-7B56.8%31.4%4.2%
Gemma-27B → Qwen-7B55.2%33.0%4.8%
Pooled63.9%44.7%5.4%

Surrogate explanations agree with the target's more than text-only explanations do (pooled +19.2 points [16.9, 21.4]), so their agreement goes beyond what an explanation of the text alone reaches.

Shuffled explanations. As Reviewer 2TTE noted, cosine similarity can be inflated by the mean direction of the activations: predicting the mean activation alone scores 0.58–0.985 cosine. We thus utilized another reconstruction metric, Fraction of Variance Explained (FVE). With FVE (where 0 means no better than predicting the mean activation), all shuffled explanations score negative (−0.56 to −0.94), while surrogate explanations score close to the targets' own NLA (0.44–0.63 vs. 0.66–0.69):

Target → SurrogateFVE: target's own NLAFVE: surrogateFVE: shuffled
Within family
Gemma-27B → Gemma-12B0.690.58−0.65
Gemma-12B → Gemma-27B0.660.63−0.67
Cross family
Qwen-7B → Gemma-12B0.680.52−0.94
Qwen-7B → Gemma-27B0.680.55−0.93
Gemma-12B → Qwen-7B0.660.45−0.60
Gemma-27B → Qwen-7B0.690.44−0.56

We truly appreciate this helpful comment, as it helped us clarify our main goals from this study and the data we collected, and using FVE helped improve our interpretation of surrogacy features and how informative they are.

AuthorsOpenReview ↗

Authors' response to Reviewer KHhh [2/3]

W3/Q2. Controls for anchoring on claims and judge feedback

To clarify, the rubric judge saw only output documents (LAB) or the final answer (PRBench), so its feedback does not cover the reasoning, which holds 63% of all error spans (327 of 519).

For NLA claims, interaction logs from the third collection week onwards record whether an expert opened a span's NLA claims before saving its label. Based on these logs, we split the 519 spans into three subsets:

  1. Exposed (244): the logs show the expert opened, viewed, or rated the NLA claims on that span before labeling it.
  2. Unexposed (64): the logs show the expert did not open, view, or rate the NLA claims on that span before labeling it.
  3. Less exposed (211): in the first 2.5 weeks of annotation, before logging started, we did not yet require experts to view or rate NLA claims; they recorded only 1 claim rating in these reviews. Their annotation rate was also about double the later rate (~16 vs. ~7 spans/hour).

Thank you for allowing us to elaborate on the data composition. We will include a flag on each span for their exposure/anchoring to NLA or rubric-judge claims in our final data release.

W2, W4/Q1, Q3. Gain from activations, and the requested analysis

We acknowledge the mismatch between the classifier conditions in section 3.2 and have revised the setup so that they differ only in the source of the explanations (external text-based vs. internal NLA-based):

  1. Text-only explanations (B): Gemma3-27B-IT, the same model as the NLA surrogate, writes token-level explanations for each span, given the full causal context up to each token. We use the same explanation format/structure as the NLA's. The classifier receives the span, its local context, and these explanations.
  2. Aligned NLA (C): we keep Gemma3-27B-IT's NLA but give the classifier its token-level explanations directly instead of Kimi-summarized claims.
  3. Data: all 24 documents collected to date (the paper's 14 groups plus 10 new ones). Each document is held out once; intervals are paired document-bootstrap 95% CIs. Starred CIs are statistically significant. All spans (511 after merging identical ones) and their explanations are now covered.

Results (macro F1):

A: noneB: text-onlyC: NLAE: NLA, other positions
0.2550.2680.2920.243
  • C − B (paired): +0.023 [−0.020, +0.062].
  • C − A: +0.036 [+0.001, +0.070]*; C − E: +0.049 [+0.012, +0.082]*.
  • Per document group: C's macro F1 exceeds A's in 16 of 24 documents and B's in 15 of 24.
  • Per error category (F1):
CategoryLabelsABC
Incomplete reasoning2580.5520.6070.560
Fact misread920.2720.2660.313
Wrong rule760.2740.2940.306
Misapplied rule660.1470.1760.200
Other590.1670.1590.250
Missed rule280.1210.1090.122
  • By exposure to NLA claims and judge feedback, and by benchmark (macro F1; W3/Q2):
SubsetSpansDocsABCC − A [95% CI]C − B [95% CI]
Exposed to NLA claims242160.2680.2780.290+0.021 [−0.025, +0.073]+0.012 [−0.050, +0.073]
Unexposed63100.2610.3350.330+0.069 [−0.040, +0.167]−0.005 [−0.055, +0.090]
Less exposed20680.2380.2370.278+0.040 [−0.004, +0.066]+0.041 [−0.025, +0.087]
Unexposed + less exposed269180.2430.2530.287+0.044 [+0.004, +0.077]*+0.033 [−0.022, +0.075]
Inside the rubric judge's scope180150.2340.2540.256+0.022 [−0.010, +0.052]+0.002 [−0.038, +0.055]
Outside the judge's scope (no judge feedback)331230.2600.2680.299+0.039 [−0.008, +0.091]+0.031 [−0.026, +0.083]
LAB22490.2320.2320.253+0.021 [−0.026, +0.073]+0.021 [−0.018, +0.064]
PRBench287150.2390.2500.268+0.030 [−0.009, +0.075]+0.019 [−0.027, +0.068]
  • With equivalent setups, the NLA-based classifier has the best macro F1 overall (0.292) and in most subsets. Its gain over no explanation is statistically significant overall and on spans labeled without exposure to NLA claims, though its gain over text-only explanations is not significant.
  • We also observed that Gemma, as an external explainer (in B), often stopped following the explanation instructions on long trajectories, so we repeated the instructions after the trajectory.
  • The NLA does not face this issue, since it reads one activation vector per token rather than the text.
AuthorsOpenReview ↗

Authors' response to Reviewer KHhh [3/3]

We regard this work as an initial exploration of a novel methodological approach. Where downstream results are weak or mixed, we think they can be addressed or improved by:

  1. More expert labels: Each classifier trains on about ~490 spans, and rare categories such as missed rules (28 labels) stay hard for every condition. We want to expand the setup to have more legal experts to support more data collection and better cross-expert matching for agreement and reliability. More fine-grained labelling of legal spans and NLA claims would be helpful to see which explanations actually help downstream and/or help experts' judgement on the data.
  2. Explaining the target directly: While our surrogacy approach shows useful features/explanations for their target model, it would be interesting to see if using explanations from the target’s own NLA helps classification and downstream tasks even further.

We really appreciate highlighting the setup mismatch and possible anchoring; the updated setup and data helped us isolate and clarify the gain from activations and NLA much more and clearly define what we can do in our next steps.

W3, W4. Utility to experts and label agreement

We agree that an in-depth evaluation of direct annotator utility is currently lacking. In this work, our primary goal is to introduce and validate this new annotation framework first; running a controlled utility study at our scale presents significant logistical hurdles, requiring additional expert hours, alignment between tasks and annotators' specific legal expertise, and overlapping annotations across experts to assess agreement. We will explicitly note this limitation and outline a controlled crossover study as a key direction for future work.

  • In their written feedback, A and C mostly found that the claims restate the trajectory text, while B found them useful for checking and verifying it. We will include their exact feedback in the paper.

  • Claim-rating agreement: our submission reported 82% agreement between B and C on 62 claims (κ\kappa = 0.60). Since then, A and B co-rated 1,568 claims, with 37% agreement (κ\kappa = 0.00), because A rated over 99% of claims neutral.

  • Span and label agreement: each expert chooses where a span starts and ends. On the 7 trajectories reviewed by two experts, span overlap F1 is 0.15 and 3 of 21 overlapping span pairs share a label, likely because each expert highlights different reasoning issues given their background.

We thank you again for highlighting this limitation in our work; we will polish our writing and define this study for our future work!

AuthorsOpenReview ↗

Thank you for your feedback!

Thank you again, Reviewer KHhh, for finding our dataset and annotation pipeline promising. We also thank you for your helpful feedback on how to frame the surrogacy idea and its results.

As the discussion period ends soon, we would be grateful to know whether our responses addressed your concerns about:

  • the surrogacy results,
  • the data composition, and
  • additional results that isolate the gain from activations in downstream experiments.

If you have further suggestions, we would be glad to work on them or include them in our future work.

We hope that our updated results, additional experiments, extended dataset, and the novelty of our annotation pipeline and surrogacy approach encourage you to reconsider your score.

Thank you very much!

Reviewer_fKULOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary
  • Legal experts review LLM trajectories alongside explanations of model activations.
  • Surrogacy: Gemma3-27B reads the Kimi-K2.6 trajectory, an NLA explains Gemma’s activations, and Kimi summarizes these into claims.
  • Data: 3 experts, 34 hours, 15 trajectories, 354 error spans, 2,892 claim ratings (41.4% helpful).
  • Correspondence on Gemma/Qwen pairs: 79.5–81.8% agreement within family, 54.6–58.6% across.
  • Claims raise error-classification macro-F1 from 0.257 to 0.324.
Critical strengths
  • New released resource: practitioners’ error spans paired with ratings of activation-based explanations.
  • Careful controls (text-only, shuffled, leave-group-out, bootstrap) and honest limitations.
Critical weaknesses

Surrogacy is untested for the actual target, and the results weigh against it.

  • Kimi-Gemma is never measured. Cross-family agreement is near chance (54.6–58.6%, counting partials).
  • The judge is Kimi itself, with no random-pair baseline.
  • Shuffled explanations reconstruct at 95–97%.
  • The gain may not come from activations.

Kimi writes the claims while a weaker Qwen3-4B writes the text-only baseline.

  • Experts saw the claims while labeling. All of the gain comes from 96 of 280 spans (CI lower bound 0.0195).

The expert data is thin.

  • One expert gave 73% of the claim ratings.
  • No kappa is reported, and there is no error-label agreement.
  • No test of whether the explanations help experts.

Other Concerns

  • Three different span subsets (354/280/233).
  • Something off with Section 3.3 ?
Questions for authors
  1. Kimi-K2.6 is described as open-source. Are its activations inaccessible or just costly? Can you measure Kimi→Gemma on a small sample?
  2. What agreement does the judge give random pairs? Please add a non-Kimi judge and a human check.
  3. Does C > B hold if Kimi also writes the text-only claims?
  4. How was anchoring from claims on expert labels controlled?
  5. Can you provide Expert backgrounds, kappa, and results without Expert B?
  6. Any with/without-explanation test of expert accuracy or time?
  7. How do the 354/280/233 subsets relate? Please clarify
  8. Please clarify on section 3.3
5 comments
AuthorsOpenReview ↗

Authors' response to Reviewer fKUL [1/3]

We thank the reviewer for their comments and feedback. We are glad you found our released data promising, and we address your questions and concerns below.

A main and shared concern was the size and composition of the expert data. We kept collecting expert annotations after submission. The data now covers 24 trajectories (15 in the submission, +60%), 519 error spans (354, +47%), 8,110 claim ratings (2,892, +180%) and 57.3 expert hours (34.3, +67%). Expert B's share of claim ratings fell from 73% to 57%. All results will be updated with the full dataset.

W1/Q1. Untested Kimi → Gemma surrogacy and high reconstruction for shuffled explanations

Kimi → Gemma. We are afraid that including Kimi is impractical as there is no public NLA for Kimi (or for any model of similar scale and capability), and training one ourselves means joint training and RL over two copies of a 1-trillion-parameter model. We're sorry that this is not possible under our current resources.

We chose a frontier-scale target model so that the collected reasoning data is useful for real-world legal work, for two reasons:

  1. In initial experiments with smaller models such as Qwen3-4B, our experts found that these models surface few non-trivial legal reasoning errors, even on older, mostly closed-ended benchmarks such as LegalBench.
  2. On recent benchmarks that represent real legal workflows, such as Harvey's LAB, frontier models and agents reach at most 27.08% end-to-end success, so smaller models rarely produce meaningful reasoning or agentic trajectories.

Shuffled explanations. Thank you for highlighting this:

  1. As Reviewer 2TTE noted, cosine similarity can be inflated by the mean direction of the activations, and this is the case here: predicting the mean activation alone scores 0.58–0.985 cosine.
  2. We therefore utilized another metric, Fraction of Variance Explained (FVE), as Reviewer 2TTE suggested. FVE = 0 means no better than predicting the mean activation.
  3. All shuffled explanations have negative FVE (−0.56 to −0.94), while surrogate explanations score close to the targets' own NLA (0.44–0.63 vs. 0.66–0.69):
Target → SurrogateFVE: target's own NLAFVE: surrogateFVE: shuffled
Within family
Gemma-27B → Gemma-12B0.690.58−0.65
Gemma-12B → Gemma-27B0.660.63−0.67
Cross family
Qwen-7B → Gemma-12B0.680.52−0.94
Qwen-7B → Gemma-27B0.680.55−0.93
Gemma-12B → Qwen-7B0.660.45−0.60
Gemma-27B → Qwen-7B0.690.44−0.56

This shows that surrogate activations carry useful features for the target, well above shuffled/mean-activation baselines. Thank you for this helpful comment; it helped us clarify our main goals from this study and the data collected, and using FVE gave a clearer measure of how useful surrogate features are.

W1/Q2. Missing controls for semantic agreement

Thank you for pointing out this missing control. To clarify, Kimi is not part of any judged pair (all are Gemma or Qwen), so it cannot favor itself.

  • Random/Chance baseline: we paired each target explanation with a surrogate explanation from a different example of the same legal task, so the legal task alone cannot drive agreement.
  • Second judge: we reran all pairs with GPT-5.4-mini as well.

Rates are the share of pairs judged to have the same or partially the same meaning:

Target → SurrogateKimi: real pairsKimi: random pairsGPT-5.4-mini: real pairsGPT-5.4-mini: random pairs
Within family
Gemma-27B → Gemma-12B81.0%7.4%58.2%4.6%
Gemma-12B → Gemma-27B80.8%8.4%55.4%3.8%
Cross family
Qwen-7B → Gemma-12B55.6%3.6%38.2%1.8%
Qwen-7B → Gemma-27B53.8%3.8%31.6%2.0%
Gemma-12B → Qwen-7B56.8%4.2%34.0%2.2%
Gemma-27B → Qwen-7B55.2%4.8%38.4%2.4%

We also added a text-only control, similar to the updated setup in section 3.2 (please check W2/Q3 in the next comment):

Target → SurrogateKimi: surrogate NLAKimi: text-onlyGPT-5.4-mini: surrogate NLAGPT-5.4-mini: text-only
Within family
Gemma-27B → Gemma-12B81.0%63.6%58.2%54.8%
Gemma-12B → Gemma-27B80.8%61.4%55.4%47.8%
Cross family
Qwen-7B → Gemma-12B55.6%37.4%38.2%33.6%
Qwen-7B → Gemma-27B53.8%41.4%31.6%34.2%
Gemma-12B → Qwen-7B56.8%31.4%34.0%23.8%
Gemma-27B → Qwen-7B55.2%33.0%38.4%30.0%
Pooled63.9%44.7%42.6%37.4%

This shows that surrogate explanations both:

  1. Agree with target explanations semantically better than external explanations.
  2. Are well above random chance.

We thank you for requesting this additional experiment; it improved our interpretation of the surrogacy signal over random and external explanations.

AuthorsOpenReview ↗

Authors' response to Reviewer fKUL [2/3]

W3/Q4. Controls for anchoring on claims

Based on our annotation metadata, we split the 519 spans into three subsets:

  1. Exposed (244): activity metadata shows the expert opened, viewed, or rated NLA claims before labeling the span.
  2. Unexposed (64): activity metadata shows the expert did not open the NLA claims before labeling the span.
  3. Less exposed (211): in the first 2.5 weeks of annotation, before activity metadata was recorded, we did not require experts to view or rate NLA claims; they recorded only 1 claim rating across these reviews. Their annotation rate was also about double the later rate (~16 vs. ~7 spans/hour).

Thank you for allowing us to elaborate on the data composition. We will include a flag on each span for their exposure/anchoring to NLA or rubric-judge claims in our final data release.

W2/Q3. Unequal classifier inputs make the gain from activations unclear

We acknowledge the mismatch between the classifier conditions and have revised the setup so that they differ only in the source of the explanations (external text-based vs. internal NLA-based):

  1. Text-only explanations (B): Gemma3-27B-IT, the same model as the NLA surrogate, writes token-level explanations for each span, given the full causal context up to each token. The classifier receives the span, its local context, and these explanations.
  2. Aligned NLA (C): we keep Gemma3-27B-IT's NLA but give the classifier its token-level explanations directly instead of Kimi-summarized claims.
  3. Data: all 24 documents collected to date (the paper's 14 groups plus 10 new ones). All spans (511 after merging identical ones) and their explanations are now covered.

Results (macro F1):

A: noneB: text-onlyC: NLAE: NLA, other positions
0.2550.2680.2920.243
  • C − B: +0.023 [−0.020, +0.062].
  • C − A: +0.036 [+0.001, +0.070]; C − E: +0.049 [+0.012, +0.082].
  • Per document: C's macro F1 exceeds B's in 15 of 24 documents.

By exposure to NLA claims before labeling:

SubsetSpansABCC − A [95% CI]C − B [95% CI]
Exposed2420.2680.2780.290+0.021 [−0.025, +0.073]+0.012 [−0.050, +0.073]
Unexposed630.2610.3350.330+0.069 [−0.040, +0.167]−0.005 [−0.055, +0.090]
Less exposed2060.2380.2370.278+0.040 [−0.004, +0.066]+0.041 [−0.025, +0.087]
Unexposed + less exposed2690.2430.2530.287+0.044 [+0.004, +0.077]+0.033 [−0.022, +0.075]
  • With equivalent setups, the NLA-based classifier has the best macro F1 overall (0.292) and in most subsets. Its gain over no explanation is statistically significant and on spans labeled without exposure to NLA claims, though its gain over text-only explanations is not significant.
  • We also observed that Gemma, as an external explainer (in B), often stopped following the explanation instructions on long trajectories, so we repeated the instructions after the trajectory.
  • The NLA does not face this issue, since it reads one activation vector per token rather than the text.

We regard this work as an initial exploration of a novel methodological approach. Where downstream results are weak or mixed, we think they can be addressed or improved by:

  1. More expert labels: Each classifier trains on about ~490 spans, and rare categories such as missed rules (28 labels) stay hard for every condition. We want to expand the setup to have more legal experts to support more data collection and better cross-expert matching for agreement and reliability. More fine-grained labelling of legal spans and NLA claims would be helpful to see which explanations actually help downstream and/or help experts' judgement on the data.
  2. Explaining the target directly: While our surrogacy approach shows useful features/explanations for their target model, it would be interesting to see if using explanations from the target’s own NLA helps classification and downstream tasks even further.

We really appreciate highlighting this setup mismatch; the updated setup and data allowed us to isolate the gain from activations and NLA much more and clearly define what we can do in our future works.

AuthorsOpenReview ↗

Authors' response to Reviewer fKUL [3/3]

W3/Q5, Q6. Expert background, agreement, and benefit from claims

The three experts are lawyers with 10–20 years of experience each:

  1. Expert A practices criminal and family law.
  2. Expert B is a data-privacy legal editor and former health-IT lawyer with a computer science and software engineering background.
  3. Expert C is a commercial-law editor and former False Claims Act litigator.
  • Claim-rating agreement: our submission reported 82% agreement between B and C on 62 claims (κ\kappa = 0.60). Since then, A and B co-rated 1,568 claims, with 37% agreement (κ\kappa = 0.00), because A rated 99% of claims neutral.
  • Span and label agreement: each expert chooses where a span starts and ends; span overlap F1 is 0.15, likely because each expert highlights different reasoning issues given their background. Every trajectory reviewed by two experts includes B, so agreement/overlap without B cannot be computed.
  • Benefit from claims: without B, the helpful rate falls from 35.4% to 0.5%. A and C mostly found that the claims restate the trajectory text, while B found them useful for verifying it. We will include their exact feedback in the paper.
  • Direct expert utility from NLA explanations: Our goal from this study is to explore the idea of bringing interpretability into expert annotation on stronger models in an accessible way through surrogacy and claim summaries. Additionally, such a study is difficult to run at our scale, as it needs additional expert time, matching them with legal areas that suit their experience, and cross-matching them with other experts for label agreement.

Thank you for highlighting the missing or additional work we can do. We will state this explicitly as a limitation and propose a controlled crossover study as future work.

W3, W4/Q7, Q8. Data subsets and section 3.3 clarification

The subsets differ due to experiments running while annotation was ongoing at the time of submission, and annotation continued after the submission as well. For example, compared with the 354 spans, the 280-span set:

  • excludes 31 spans annotated after its export, 59 held out or excluded during preprocessing (App. B.2), and 18 labeled only "other";
  • merges 5 identical spans marked by two experts, combining their labels;
  • includes 39 pilot-phase spans that the 354 do not count.

Section 3.3: It reports two exploratory experiments. Both use the expert annotations only as evaluation data, and neither uses NLA explanations. First, we train an error classifier on synthetic labels, alone or with expert or LLM-judge labels added; no condition was found clearly better (all paired intervals include zero). Second, joint models detect and diagnose errors in passages without marked spans; diagnosis improves, but detection remains weak (gated F1 0.03–0.08).

  • We will polish this section's writing to be more direct and presented as an exploratory idea. Thank you for your feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response by authors

We thank all reviewers for their careful reading of our submission and their thoughtful feedback. We are encouraged by their positive comments, including:

  1. A novel annotation pipeline that brings activation-based explanations of model reasoning, produced by Natural Language Autoencoders (NLAs), to domain experts. The pipeline summarizes token-level explanations into claims that cite their sources, so experts can inspect and check them (Reviewers KHhh and 2TTE).
  2. A dataset of expert-annotated legal-error spans with written feedback, paired with helpfulness ratings of activation-based explanations, covering agentic and conversational trajectories (Reviewers fKUL, KHhh, and 2TTE).
  3. Controlled downstream experiments, with matched explanation budgets, text-only and shuffled controls, document-group holdouts, and bootstrap intervals, together with careful and honest reporting of limitations (Reviewers fKUL, KHhh, and 2TTE).

We answer each review individually below. The main changes are:

  1. Data: We extended the dataset to 24 trajectories (15 in the submission) and 57.3 expert hours (34.3). Expert B's share of claim ratings fell from 73% to 57%.
  2. Surrogacy (section 3.1): We added a random-pair baseline, a second LLM judge, a text-only control, and used Fraction of Variance Explained in place of cosine similarity. Pooled across model pairs, surrogate NLA explanations beat every control on both semantic agreement and reconstruction.
  3. Error classification (section 3.2): The conditions now differ only in the explanation source: NLA or text-only, from the same model and text. NLA explanations give the best macro-F1 and beat no explanation and shuffled explanations. Their gain over text-only explanations is not significant.
  4. Reliability: We report expert agreement, a cross-expert check, and results on spans labeled without exposure to explanations.
  5. Framing: We center our contribution on the pipeline, surrogacy, and dataset, and present downstream results as preliminary. Surrogate explanations correspond to the target's and provide useful features, but do not explain its computation. We mark section 3.3 as exploratory and leave a study of expert benefit to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.