acceptodds
Under review as a conference paper at ICLR 2027

Vernier: Diagnosing and Mitigating Lexical Sensitivity in Causal Reasoning

Abstract

Lexical renaming exposes a gap between specifying a causal problem and eliciting a stable answer from a language model. We study this gap through Vernier, a paired-view LoRA update that combines supervision on original and placeholder prompts with answer-token consistency. The analysis separates three outcomes: improved task performance, agreement across lexical views, and the representation measurements used to explain them. Recorded multi-model evaluations show substantial gains in both views for several Qwen and Llama models, with stronger transfer on CRASS than on e-CARE. An independently traceable 900-question evaluation confirms large paired-view accuracy gains, while also showing that such gains need not reduce the absolute lexical gap. Probes measure the transfer of lexical-content decoders, and donor-controlled patching demonstrates strong control of answer identity at an intermediate layer. Together, the results locate a useful target for adaptation at the answer interface and establish why task improvement, lexical consistency, and mechanistic explanation must be evaluated separately.

Primary area:
causal reasoning
Keywords:
Causal Reasoning, Lexical Robustness, Consistency Regularization, Parameter-Efficient Fine-Tuning, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_98MGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the lexical renaming problem. The authors employ the Vernier to analyze the problem. The multi-model evaluations demonstrate performance gains in open-source models like Qwen and Llama. The authors provide a useful analysis for understanding the lexical problem.

Critical strengths
  1. The experiments are comprehensive. The authors test multiple open-source models across different model families.
Critical weaknesses
  1. The paper's presentation can be improved. For example, since the Vernier is an important technique in this study, the authors can draw a better figure to illustrate this method. The current version of Figure 1 is not very appealing, with elements too densely packed and appearing as if generated directly by an AI model. Additionally, Figure 1 could be moved earlier to page 2.

  2. The contribution may need to be specified. It seems that the authors try to emphasize two aspects of the contribution: understanding and performance improvement. For example, the first two paragraphs of the introduction suggest that the authors try to understand the effects of weight updates on lexical failure. The last two paragraphs, however, show that the authors try to solve the lexical failure problem through a method. Such a writing style appears somewhat fragmented, leaving the readers confused about which contribution the author actually intends to emphasize.

  3. The results are not well presented. For example, the text in Figure 2 is too small to be viewed. It is hard for readers to directly perceive the superiority of the method.

Questions for authors

Was the Vernier method originally proposed by the authors? If so, why are there some citations in lines 044 to 046 when introducing this method? If not, the authors should clearly state this.

Reviewer_Uz4iOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Summary: The paper proposes a method to train in a way that reduced instability in logical reasoning when nouns change and do not change the logic.

Critical strengths

The paper has multiple experiments, method, and work behind it.

Critical weaknesses

I don't understand the motivation of the paper. The paper proposes a sophisticated method to solve a problem that seems neither practical nor of deep intellectual importance and fails to motivate it in a convincing manner as to why this question should be asked or be of general interest to the scientific community.

The writing flow is pompous, staccato, and brief. The smell of not edited enough AI writing carries with it and makes the reading a bit hard. Pangram also detects the introduction as 100% AI generated (it has mistakes sure). At the current state of writing, I doubt people will read it, because it gives such unhuman feeling, regardless of the content. Content is important, but a paper is about delivering this finding to humans that will absorb this, and at that, the paper fails.

The introduction contains a lot of jargon and terms that are a bit hard to follow or are not always clear how the logic flows, some of it seems like maybe it comes from another paper (the 2026 one mentioned bellow?). The related work section stops at 2024, so either this paper is not relevant to current study, or it revives something exciting (I do not hold this opinion) or it fails to report related work in an adequate manner.

Out of all citations, only 2 of them are from the last 2 years, this is supposedly fine, but raises eyebrows as there is a lot of work not mentioned, and I wonder why. (The one paper from 2026 that appears in the first paragraph gets a lot of credit, but seems a bit suspicious.

The illustrations are often full of words, or not self explanatory, or have fonts that are too small to read, or otherwise not accommodating, while ok without a one-line in the caption telling me what am I looking for would also be nice. Similarly, the tables have different rows and bold and design that doesn't seem to focus the reader on the point. Why do the tables have different fonts in different numbers and the + and - signs change?

Questions for authors

Questions:

Citations, how much of this bias comes just from how strong those models are? See for example this paper showing how inadequacy is often mistakenly thought of as instability (https://arxiv.org/abs/2602.03344)

What is SCM? It appears in the related work and then used as if it was explained. (Causal masking I guess)

Why is the gap signed and how come the gap is always positive, it means the replacement breaks something, as one expects the responses to change not to always be worse in the replacement, this seems like a hidden bug somewhere. IF the gap is not signed, you should change the writing and also not write + in every line there.

Minor:

What is "Vernier" is that something known? Or is it a metaphor (and then it also should be better introduced).

There is a related work section, but I was surprised not to see there papers about multiprompts or prompt sensitivity (state of what art, or DoVE comes to mind in those regards).

Reviewer_2NQROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Prior work showed that replacing variable names with typed placeholders changes LLM answers in causal question answering even when the structural causal model, query, and gold answer are preserved. This paper asks which aspect of such a lexical failure a weight update repairs: reduced accessibility of task-relevant content under the placeholder view, or a change in how available content is read out as an answer. To study this question, the authors use Vernier — a LoRA-based paired-view fine-tuning intervention with full-vocabulary answer supervision on both views and a symmetric KL term over candidate answer tokens — as a controlled training-level perturbation, and examine it through a traceable 900-question comparison, multi-model results, zero-shot transfer to CRASS and e-CARE, layer-wise diagnostics, and donor-controlled activation patching. They report substantial accuracy gains on both views and stronger transfer on CRASS than on e-CARE, while explicitly delimiting the interpretation: these gains do not establish a smaller absolute lexical gap, the learned invariance is not fully independent of the placeholder scheme, and an injected decision-token state controls answer identity without requiring an item-matched donor. The paper concludes that task correctness, cross-view agreement, and intervention specificity must be evaluated separately.

Critical strengths
  1. The paper's interpretive discipline is a notable strength: it reports signed rather than absolute lexical gaps alongside both component accuracies, treats a negative gap as an inversion rather than a larger success, and proactively reports counterexamples (the Phi-3 regression, the gap-closing-but-accuracy-collapsing MLP trajectory, and the 900-question comparison in which large accuracy gains coincide with a widened absolute gap). The abstract and Section 5 explicitly refrain from claiming that task improvement establishes lexical repair.
  2. The donor-controlled activation patching is carefully designed and conservatively interpreted. Comparing matched, answer-matched, random, and different-prediction donors separates control over answer identity from item-specific information transfer, and Section 4.4 explicitly frames a successful patch as an intervention-site effect rather than a localisation of the computation changed by training. This is a useful negative result for interpretability.
  3. The evaluation is broad: nine models across five families, three benchmarks spanning in-distribution and zero-shot transfer, five training configurations, and behavioural, representational, and interventional diagnostics, supplemented by paired bootstrap tests, larger-n checks, capability screens, and a lexical-overlap contamination audit.
Critical weaknesses

All experiments and diagnostics are conducted on open-weight models released around 2024 (Qwen2.5, Llama-3.1-8B, Phi-3-mini, Mistral-7B-v0.3, Gemma-2-2B). Given the pace of model iteration and the substantial growth in pretraining data since then, it is unclear whether the lexical gap studied here remains pronounced in more recent models. This bears directly on the significance of the research question: if the phenomenon has already been substantially mitigated by newer training recipes, both the motivation and the practical relevance of the analysis are weakened. The paper's own Table A15 reinforces this concern: even within this cohort, the in-distribution base gap on larger models is already small and non-significant (+1.0–2.2 pp, with confidence intervals crossing zero), and Section 4.2 accordingly rests the behavioural case on out-of-distribution transfer. Whether the large OOD gaps persist in newer models is therefore the load-bearing question. Evidence on one or two contemporary models would help establish that the problem remains worth studying.

Questions for authors
  1. On the equivalence assumption behind the gap. Section 3.1 argues that because the perturbation preserves the SCM, the query, and the gold answer, an SCM-grounded reasoner should produce the same response under both views. This appears to treat formal task equivalence as equivalence of task difficulty. Anonymized, symbol-only instantiations are typically harder for human reasoners than semantically concrete ones, and the paper reports neither a human two-view control nor a check of whether the placeholder view changes input length, predictive uncertainty, or answer entropy. Could the authors clarify what evidence supports treating the two views as equally difficult? Would a human baseline or an uncertainty comparison help separate a genuine reliance on lexical anchors from a general difficulty effect?
  2. A presentation suggestion. The main ablation (Table A1) currently appears only in the appendix while Section 4.1 is comparatively long; moving the ablation into the main text and compressing part of the setup would improve readability, and a similar issue occurs elsewhere. The four panels of Figure 2 are also rather small and would be easier to read if split or enlarged.
  3. The anonymous code link (https://anonymous.4open.science/r/Vernier) does not open, so I could not access the source code. Could the authors restore the link or provide an alternative?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.