- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates ownership bias in LLM verification, where models tend to evaluate identical artifacts more favorably when they are claimed to be their own. The authors construct paired Self and Other conditions and extract an ownership-associated direction from the differences in hidden states. They then apply centered projection to remove this direction during inference. Experiments across multiple LLMs show that the method substantially reduces the ownership gap while largely preserving correctness discrimination. The extracted direction also demonstrates transferability across different benchmarks and prompt formulations.
Critical strengths
- Important research problem. The paper addresses ownership bias in LLM self-verification, which is relevant to improving the reliability of autonomous agents.
- Well-controlled experimental design. The paired evaluation setup keeps the evaluated artifacts unchanged while varying their claimed ownership. The attribution-only experiments further reduce potential confounding factors.
- Comprehensive experiments. The authors evaluate multiple models and tasks, with comparisons against random, correctness, and leniency directions, as well as activation steering. The results show effective bias reduction with largely preserved correctness discrimination and reasonable transferability.
Critical weaknesses
- Insufficient distinction between ownership information and ownership bias. The extracted direction may primarily encode ownership information rather than a distinct bias mechanism, since it is derived from activation differences between Self and Other prompts. The observed bias reduction could therefore result from suppressing ownership information itself. I suggest evaluating whether ownership information remains accessible after projection.
- Limited methodological novelty. The proposed method mainly combines activation difference-based direction extraction with linear projection, both of which have been explored in prior work. I suggest comparing the method with simply removing ownership information from the prompt to better demonstrate its additional value.
Questions for authors
Does the model retain the ability to recognize ownership information after projection? How can the authors distinguish ownership semantics from ownership-induced bias?
2 comments
Follow-up on our response
Dear Reviewer fkvH,
Thank you again for your time.
As the discussion period nears its end, we would like to follow up on your main concerns regarding ownership information retention and the blind verification baseline. We have added the requested experiments, and the new results show that our projection perfectly retains ownership information while eliminating the bias. They also demonstrate the unique advantage of projection over simple prompt deletion in realistic agent pipelines. We hope these additions satisfactorily address your concerns, and we would appreciate it if you could reconsider your rating.
We are happy to clarify anything further.
Response to Reviewer fkvH
We thank the reviewer for the careful reading of our paper and the recognition of our work. Our responses follow.
A. Yes (new Appendix D, Table 21). On the 133 SWE-bench test rows, we replace the correctness question with "According to the text above, who is stated to have written this code change?" and apply the projection of Table 6 unchanged.
On the same rows, Table 6 shows that this projection removes 91% of the judgment gap. Ownership semantics and ownership-induced bias thus come apart. The verifier still knows who is stated to have written the code, while the projection removes most of the bias that this statement induces.
A. We have added this baseline in new Appendix B (Table 11). When authorship is stated in a single sentence, deleting it is effective. The blind prompt keeps correctness AUC within 0.04 of both attributed prompts, while the Other clause lowers judgment scores across all ten verifiers relative to the blind prompt. This confirms our first finding, that the claimed author alone shifts verification.
Prompt deletion must first locate the authorship text and then discard it, which introduces two limitations. First, in agent pipelines, authorship often arrives as a free-text work log narrating who changed which files (new Appendix E). A sentence filter matches none of these logs, whereas the same frozen direction removes 68% of the gap without locating authorship. An LLM rewrite shrinks the gap substantially, but requires an extra call per diff. Moreover, the log makes the verifier more lenient on correct and incorrect diffs alike. This leniency persists after the rewrite, whereas projection reduces it sharply. Second, projection leaves the authorship information in place (W1 above).
Regarding novelty, our contribution lies in showing that the claimed author alone shifts verification, and that this shift operates along a single direction whose removal preserves correctness ranking.