- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how LLM-as-a-Judge behavior differs when outputs are requested as different formats. The authors use PEAP to investigate 5 instruction-tuned models across 5 judgment tasks. They identify Latent Evaluator as the circuits shared by different output formats, and Task Formatter as the circuits that are format-specific. Validation includes activation restoration, ablation with knowledge-task controls, cross-task steering, cross-format activation transfer, and robustness checks across data splits, prompt paraphrases, and attribution methods. The results indicate shared judgment-related pathways within the models, but their internal organization and cross-format transfer vary considerably. The authors thus argue that format-specific output routing may contribute to judgment inconsistency.
Critical strengths
The research question is meaningful, and the empirical evaluation is comprehensive. Format-dependent inconsistencies in LLM-as-a-Judge behavior are an important and practical research problem. The authors investigate this issue through extensive experiments across five different models and five tasks. This makes the work potentially valuable as a mechanistic investigation.
Critical weaknesses
W1: The proposed explanations for format-induced inconsistency are not sufficiently justified
The "format-induced inconsistency" stated in the title and emphasized throughout the abstract and introduction is never directly measured. Specifically, the authors do not quantify the actual inconsistency or disagreement between the rating and classification systems used. Furthermore, they do not examine whether the LE readouts exhibit inconsistencies when the actual outputs from different formats disagree. Additionally, the paper provides no TF intervention experiments that would directly support the claim that "... so fixes can target the formatter." In fact, TF is never directly intervened upon. Steering, FTI, and ablation experiments in Appendix I are all conducted on LE rather than TF. FTI flip rates are highly heterogeneous and lack low-rated-source, random-component or same-format controls, and the label-free readout is evaluated only on rating prompts.
W2: The second contribution (functional modularity) is not well established by the current experiments
Knowledge and judgment are evaluated using different interventions on different sets of components: knowledge is assessed through zero-ablation of the merged LE, whereas judgment collapse is assessed through resampling ablation of the top-ranked edges across tasks and formats. Moreover, the random control itself is highly destructive (reducing MMLU accuracy to 20–23% on Qwen2.5-7B and Llama). Thus, causing "less damage than random" does not imply that world knowledge is preserved. In fact, LE ablation reduces PopQA pure recall from 44.7 to 4.7 on Qwen2.5-7B and from 62.7 to 5.7 on Llama. The claimed generality is also broader than the evidence supports: while pairwise overlap suggests shared structure, the only multi-task core, the merged LE, covers only 3 structured tasks and excludes RewardBench and Yelp.
Questions for authors
- What are the rating-vs-classification disagreement rates per model and task? On disagreeing instances, does injecting the instance's own rating-prompt LE activations into its classification prompt remove the disagreement, and does intervening on the TFs alone reduce it?
- Can you ablate C_{TF,rate} and C_{TF,class} separately and report both rating and classification performance?
- Is the judgment representation shared across formats? Please report the similarity between rating- and classification-derived directions at the same token-aligned LE sites, and apply the fixed BDAS-1D readout to classification prompts
- How large is the core shared by the LEs of the five tasks, including RewardBench and Yelp, relative to a matched null and the split-half ceiling?
- In FTI and steering, how are sites mapped between the rating and classification templates, which components and positions are injected, and is the per-task or the merged LE used?
- Which metric is used to trace C_class, given that EV is undefined for yes/no and three-way labels? What counts as a "component" in Tables 1, 15 and 16?
3 comments
Thank you for the detailed responses, and I appreciate the authors' efforts to conduct additional experiments. That said, my main concern regarding W1 remains unresolved. While the new experiments provide useful evidence of cross-format disagreement and the causal relevance of the identified circuits, they do not isolate the contribution of the TF components from that of the shared components, particularly on instances where the formats actually disagree. This means the central claim that format-induced inconsistency arises from format-specific output routing still appears stronger than what the current causal evidence establishes. Therefore, I will maintain my rating of 2.
Thank you for the follow-up. We agree that the revision does not cleanly isolate the contribution of the format-specific components. We have since run this intervention on the disagreeing instances of three model-dataset cells (Qwen2.5-7B on MNLI and RewardBench, Gemma-3-27B on MNLI) and find that replacing only the format-specific components of a format with their mean makes at most 3.4% of these instances agree (random components of the same type: 0 – 8.0%).
Could you give us one final recommendation on what would make the paper stronger in light of this result?
Thank you for the detailed review.
We agree with the central point of W1: the submission never measured the inconsistency promised in its title, and it tested the shared set with two formats only. Both are now addressed. We answer your two weaknesses through your six questions.
Q1. Formats disagree in every model
We evaluated identical instances under all three formats in all 25 model-dataset cells. Formats disagree in every model: the median rate per cell is 26%, and 16 of 25 cells exceed 20% (
Table 21, App. V.4).As a first intervention, we inject the shared set from a same-format donor with the opposite label on Qwen2.5-7B MNLI. This is the same-format control you asked for. The classification output moves toward the donor (+1.04 on the score scale, p < 0.001); rating does not respond, which we report.
We did not intervene on the formatter alone. We have therefore removed the sentence that fixes can target the formatter, and the paper now presents FTI as evidence that the injected components can move the output.
Q2. Each format has its own components
With three formats, each format has its own component set next to the shared one in 15 of 17 cells (a median of 4 components per format). These sets are now defined and listed per cell (
App. V.1, Table 19). The paper makes no claim about their individual necessity.Q3. The representation is shared across rating and A–E prompts
A shared set exists in all 17 retraced cells on four models, and 158 of 215 two-format components remain when a third format (A–E letters) is required. Reversing the letter assignment keeps 15 to 19 of 25 components.
We applied the fixed BDAS-1D readout to the other formats on Qwen2.5-7B MNLI: it reads A–E prompts at about 90% accuracy under both letter assignments and does not transfer to classification prompts. The representation is therefore shared across the two ordinal formats, and we state this narrower scope (
App. V.5).Q4. The five-dataset core is small, so we report graded coverage
It is small. With RewardBench and Yelp included, 3 components are shared by all five datasets for Qwen2.5-7B and none for Llama-3.1-8B; 12 and 7 appear in at least three datasets. Against the split-half ceiling, the strict core is also unstable (Jaccard 0.18 and 0.10 between halves), while the three-dataset sets are more stable (0.53 and 0.39).
We now report per-dataset sets with graded coverage and state that the merged LE of the submission covers CoLA, MNLI, and STS-B only (
Table 20, App. V.3). A matched null cannot be built under a strict rule, because it requires a second MLP in the same layer.Q5. FTI uses per-dataset circuits
FTI uses the per-dataset circuits. The merged LE is used only in the knowledge experiments. Forward runs inject the sender components of the rating circuit, and reverse runs those of the classification circuit, so these runs were not LE-only.
We renamed them (
§5, App. K) and added the injection sites, the alignment of positions, and the skipped instructions (App. J). Steering likewise uses the top components of the circuit of the source dataset.Q6. One metric over all valid labels
The classification circuit was traced with 1 + P(second label | two labels), which excluded MNLI Neutral. All new tracing uses the margin of the gold label over all valid labels, including Neutral. A component is an attention head or an MLP that sends at least one circuit edge; this definition is now stated once and used in all tables (
§3.1, App. V.1).W2. Judgment and knowledge now share one intervention
We agree on all three points. Judgment and knowledge were tested with different interventions; the random control was destructive; and "less damage than random" does not mean knowledge is preserved. We removed that wording and now report the absolute drops you cite. We use exactly one intervention, one component set, and one layer-matched random control with up to ten seeds for rating, classification, A–E, and PopQA.
The result is point 3 of our general response (
Table 15, App. U.3): in Qwen2.5-7B and Llama-3.1-8B, the LE is necessary for judgment and for recall alike. In Gemma-3-12B, replacing the LE leaves recall unchanged.We would be glad to hear whether these changes address your concerns, and we are happy to run further checks.