- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks whether an LLM judge that gives a wrong pairwise verdict still holds the information needed for the correct judgment in its internal activations. Across multiple open-weight judges and different preference benchmarks, the authors train small linear probes on the judges’ frozen activations, and compare them to the judges’ emitted verdicts. Probes recover human preferences that the verdict misses in the adversarial scenario, and the paper uses cf, a length-only classifier’s accuracy, to predict when this helps.
Critical strengths
S1. The experiments span many judges, model families and benchmarks, with appropriate controls and replication, so the main observation is unlikely to be an artifact of a few models.
S2. The paper distinguishes what its evidence supports from what it does not, and reports negative results openly.
Critical weaknesses
W1. The paper recommends reading internals when cf ≳ 0.75. But cf is itself the accuracy of a length-only classifier trained on the same labels, and on the two most confounded benchmarks it matches or beats the probe:
The lens probe used as the deployed reader in §7 loses on both (Table 2a).
The stronger residual-stream probe also loses on RB2, and beats the length classifier on LLMBar by only 0.046 (0.846).
So in exactly the setting the method targets, the simple classifier computed while obtaining cf is already a comparable or better reader. This undermines the practical case for reading internals.
W2. Out of distribution, the probe largely fails (Appendix A.7). The leave-one-benchmark-out probe beats the verdict on only 1 of 8 benchmarks. The read-out therefore needs labels from each new evaluation distribution. This limits its value as a general method and weakens the claim that the judge holds a general preference signal.
W3. The paper is quite hard to follow. The core argument and the presentation are unclear. Implementation and statistical details crowd the main text, while the concepts needed to read the results are deferred to the appendix, such as the probe’s input features and the column definitions of Table 2 ((\Delta r), (\delta_o)). Figure captions are very brief, and Figure 2 in particular is difficult to interpret based on the main text alone. Many implementation details should be moved to the appendix, leaving more space to explain the experimental setup, key takeaways, and how to interpret each figure.
Questions for authors
See weaknesses. I would be happy to improve my score if my questions could be reasonably resolved.
4 comments
Dear Reviewer,
With the discussion deadline approaching, we wanted to gently follow up. We addressed all of your points. We'd be grateful if you could take a look, and if your concerns are resolved, we kindly ask you to consider revising the score. We're glad to answer any remaining questions.
Response by Reviewer 6DSE
Thanks for the detailed response and revision. The presentation is now much clearer. However, my main concern remains. On LLMBar-adversarial, the length-only classifier still outperforms the deployed lens probe and the length-controlled probe, and the gain over the recalibrated verdict is concentrated almost entirely on this benchmark. The practical case for reading internals therefore remains narrow, and I will keep my score.
Thank you, we are glad the presentation is clearer.
Three points on the remaining concern.
- The residual-stream probe beats the length classifier on LLMBar (0.846 vs. 0.799, for 48 of 50 judges).
- The length-controlled probe has, by construction, no access to length, so comparing it with a length classifier asks a different question: does content signal remain? It does (0.686 vs. 0.507 with shuffled labels).
- The length classifier is accurate on LLMBar only because it learns how that benchmark was built. On rb-easy the better answer is longer in 79% of pairs, so LLMBar's "prefer shorter" rule would be right only about 21% of the time. Used as a reward or labeler, it can be satisfied by changing length alone. The length-controlled read-out does not invite this, since length is regressed out of its inputs.
We agree the practical case is conditional, and the paper says so. We believe a conditional result whose condition is measurable in advance and validated on held-out data is itself a contribution.
Thank you for constructive review. Your W1 led us to an issue in our data, and we are grateful for it.
W1. The length-only classifier vs. the probe.
First, the facts. On LLMBar, the length classifier (0.799) beats the lens probe (0.758) and loses to the residual-stream probe (0.846; better for 48 of 50 judges). The revision shows the length classifier as a row in the main table, so readers can see this directly.
Second, RB2. While checking your RB2 numbers, we found a loading error on our side. RB2 stores one correct and three rejected answers per prompt; our loader joined the three rejected answers into one string, so the rejected side was longer in 98% of pairs (53% with proper pairs). RB2's cf of 0.96 was an artifact of this error, not a property of RB2. We removed RB2 (General Response, item a). So of the "two most confounded benchmarks", only LLMBar remains.
Third, why the length classifier is not a competing judge. It is fit to how a benchmark was built, not to answer quality.
What we changed. We now read cf as you suggest. A high cf means the labels can be predicted without reading the answers, which is why the verdict fails there. The practical rule in the revision is: when cf is high, do not trust the verdict; use the internal read-out instead; report the length baseline next to it; and if the read-out will be used as a reward or labeler, use its length-residualized version. We removed any suggestion that the probe is the most accurate label predictor on confounded sets.
W2. Out-of-distribution transfer. We agree. The read-out is specific to a distribution: it needs a small labeled sample from the target distribution (256 labels in our budget experiments). Our appendix already lists cross-benchmark transfer as a claim we do not make. The revision now says this in the abstract and introduction, and removes wording that suggested "a general preference signal". The claim is narrower: for a given evaluation distribution, a judge's layers carry label information that its verdict loses.
Two points of context.
What does transfer is across judges. A read-out fitted on 49 judges and applied to an unseen judge, with none of that judge's labels, improves on its verdict on LLMBar by +0.077 [0.033, 0.118], for 78% of judges.
W3. Presentation. We agree and made these changes: