acceptodds
Under review as a conference paper at ICLR 2027

Coordinates Without a Compass: Gauge Symmetry and Trust Anchors in LLM Verification

Abstract

Relational evidence can recover every pairwise truth relation while leaving absolute truth unresolved. We study verification systems whose judgments are correct only up to a shared binary orientation. Flipping both latent truth and this orientation preserves every observation, so no aggregation of such evidence can certify an individual claim. We formalize this ambiguity as a gauge symmetry and show that certification is possible exactly when truth is constant across observationally indistinguishable states. A trusted anchor resolves the ambiguity only when it fixes the target mechanism's orientation. Consequently, in noiseless relational systems, trust cost is determined by the number of independently orientable components, not by the amount of evidence. Experiments support these predictions. On MMLU-Pro, DeepSeek recovers the truth partition with 98.5% accuracy, yet unanchored absolute accuracy remains 50%; one held-out anchor raises it to 97.0%. HumanEvalPlus audits show that anchors help aligned mechanisms but can hurt oppositely oriented ones. A label-free router topology predicts the same within-component advantage in two frozen cohorts. Relational coordinates become a compass only when a mechanism-compatible anchor supplies their orientation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Verification, Identifiability, Gauge Symmetry, Trust Anchors, Statistical Decision Theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_si7COpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies why evidence can reveal which claims share the same truth value without identifying which claims are true. It derives the number of trusted anchors needed to resolve this ambiguity, and proposes a theorem about the transfer between verification mechanisms. Experiments on MMLU-Pro, HumanEvalPlus, and studies of frozen model representations, demonstrate when the trusted anchors help, and when their orientation does not transfer.

Critical strengths
  1. The paper proposes a principled criterion for when agreement can and cannot certify correctness. Perfect relational evidence can leave individual claims uncertifiable, and the value of a trusted label depends on how its source relates to the target. This is a meaningful insight, so I lean toward acceptance.
Critical weaknesses
  1. The transfer comparisons in Section 4.2 share the same source and target mechanisms, so their successes are not independent. The main text does not specify the null hypothesis or what was randomized in the significance test. Without these information, it is hard to gauge the statistical evidence.
  2. (Updated November 18) Reading Reviewer Frnq's critical weakness Item 1 raises my severe concern about the novelty and originality of the work. While the authors have partially addressed this in their revised paper, they did not distinguish their contribution from prior work clearly in their initial submission, which they should have done, and the revisions making explicit the much thinner contribution are significant enough to alter a reviewer's judgment.
Questions for authors

For the significance test in Section 4.2, what was the null hypothesis, and what was randomized? How were dependencies between transfer comparisons preserved?

Reviewer_FrnqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors study verification systems. They check whether the evidence of how claims relate to each other can certify individual claims. The authors propose an anchor that will guide the direction and solve the ambiguity. They evaluate these systems using LLMs, which will provide the answers (deployed answers) and can be judged based on the ground truth from the benchmark; or as judges and verifiers, based on their own confidence. The authors make a comparison between an oracle, the theoretical baseline and the actual evidence of improvement from using an anchor. The labels or anchors needed depend on the group connectivity and not on the amount of evidence.

Critical strengths
  • The assumptions stated for the theory are are explicitly stated.
  • The use of LLMs and the evaluation of how anchors can stir them is a promising direction of research, but I dont undersand yet how can we use these systems to verify/certify LLMs claims.
Critical weaknesses
  • The authors do not cite previous work for the equations, theorems and definitions. There is just a paragraph in the introduction about previous work. Like the orbit-stabilizer theorem. For the paper, the authors should clearly say what is new and what is known in the literature. It would make the paper stronger in its claims. It is hard to parse what they are proposing and it is the novel contribution, and what are they taking from literature.
  • The writing of the paper is hard to follow and very hard to parse and understand. It is not that the paper is badly written, in the sense that the grammar etc is good. The key problem is that it is so convoluted and it is phrased in a way that a statement tries to convey a deep idea, but the real idea is often much simpler. The work would substantially benefit from making the paper humand understandable and to the point by stating the problem, contributions and conclusions in a clear way. It would also help to use terminology and concepts that are standard in the literature.
  • The work clarifies in the experiments that they establish a mechanism, but there is no evidence of practical environments or comparing against previously known work, or baselines. As with the contribution of the work, it is hard to parse which experiments are showing the theory and which ones are evaluating it, or testing its limits.

Due to the limitations of the experiments (e.g., providing enough evidence for their claims and practical implications), unclear contribution, and not understandable writing, I suggest clear rejection. That work needs to clarify what is previous work, cite that work where it is due, state clearly the own contributions.

Questions for authors
  • It seems that Figure 2 and Table 5 can be recomputed from Table 3. Therefore, is there more evidence and results beyond the raw accuracy?
  • Could you add confidence intervals to the results in section 4? The plots are missing them too.
  • Is there anything specific to the chosen models? Why test those three?
  • How many deployment labels were used for the global and raw orientations? In section 4.4 you are missing a baseline, or it is hard to find the evidence against a global baseline?
  • Do you have results for other benchmarks? You could consider work like [1] on true/false benchmarks or by [2] on discovering latent knowledge in language models without supervision.

References

[1] Marks & Tegmark (2024), https://arxiv.org/pdf/2310.06824

[2] Burns et al (2023), Eliciting Strong Capabilities With Weak Supervision

Reviewer_MS9hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies verifiers whose judgments are right only up to a shared sign flip. It shows that pairwise “same/different” can recover the truth partition but can't certify a single claim. Paper claims at least one trust anchor per connected component are needed, and such anchor helps only if its mechanism has the same orientation as the target, and it can hurt if the orientation is opposite. Experiments use MMLU-Pro, HumanEvalPlus, BBH and router logits and hidden states from Granite and Qwen.

Critical strengths
  1. The paper has clear framing, it separates recovering relations from certifying truth, and the point that anchors can hurt oppositely oriented mechanisms are useful and clearly shown in Fig.3.
  2. The protocal is carefully designed, anchors are held out, splits are fixed.
Critical weaknesses
  1. Sign ambiguity under XOR relations with one label needed per component, is basic linear algebra over GF(2) and the same idea as label switching in mixture models. Theorems 1–3 and Eq. 12 are mostly definitions plus simple algebra, the novelty and technical contributions are limited.
  2. The evidence from natural settings is weak. In the prospective audit, the inferred domain topology gets 76.2%, and a single global orientation also gets 76.2%. In the Granite experiment, “the stronger prespecified decision rule was not met” (p. 9).
  3. The samples are small: 70 answers for MMLU and 158 programs per model for HumanEval.
Questions for authors
  1. Is there a real verification setting where only relative judgments are available? If not, why does this matter in practice?
  2. What is new in the theory beyond classic sign and label-switching identifiability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.