acceptodds
Under review as a conference paper at ICLR 2027

Verb-Centric Counterfactual Logic Attacks on Large Language Model Reasoning

Abstract

Recent research indicates that large language models (LLMs) can achieve efficient reasoning by leveraging logical shortcuts, potentially reducing reliance on explicit chain-of-thought processes. However, such shortcuts may introduce vulnerabilities, particularly when reasoning is verb-centric and lacks rigorous logical grounding. In this work, we study counterfactual logic attacks that perturb relation verbs in shortcut triplets. We first show that LLMs exhibit boundary-sensitive preferences over factual, counterfactual, and unrelated verb clusters. To support this analysis, we construct Verb-LK, a curated benchmark of verb-driven logical implications for evaluating model stability under counterfactual logic attacks. We further propose Adaptive Logic Attack (ALA), which uses semantic uncertainty to select logically aligned or contradictory verbs within concept clusters and introduces a contrastive adaptation loss to reconfigure verb-cluster decision boundaries while accounting for potential concept distortion. Experiments show that ALA produces a strong targeted shift in relation-level preferences, generalizes to unseen implications, shows corresponding effects in additional Qwen3-8B experiments, obtains encouraging results in an exploratory anti-stereotype evaluation, and causes only a modest change on MixEval as a broad capability check.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Counterfactual logic attack, Verb-centric reasoning, Logical shortcuts, Adaptive Logic Attack, Contrastive adaptation, Semantic uncertainty, LLM robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Stg9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether language models can be shifted from preferring factual relational verbs to preferring a set of counterfactual verbs through targeted modification. The authors treat verbs such as impairs as logical relations linking two concepts and partition candidate verbs into factual, counterfactual, and irrelevant clusters. They build Verb-LK with 1,122 verb entailments and first examine cluster-level preferences in unmodified models and whether in-context examples at inference time can flip those preferences.

They then propose Adaptive Logic Attack (ALA). It uses the target model’s semantic uncertainty to select centroid and support words in the factual and counterfactual clusters, then fine-tunes the model with a contrastive adaptation loss to push the decision boundary toward the counterfactual cluster. Experiments focus mainly on Llama-3.1-8B-Instruct, with Qwen3-8B as a supplement. The authors report that the attack changes verb preferences on seen and unseen entailments, exploratory anti-stereotype effects on StereoSet, and relatively small overall changes on MixEval.

Critical strengths
  1. The paper moves the attack from replacing one relational word to shifting a set of semantically similar verbs, and shows that adding counterfactual examples in context alone usually cannot flip the model’s strong prior preferences. Parameter updates can noticeably change open-source model outputs on such cloze items.
Critical weaknesses
  1. The paper’s “logic attack” is largely a change in commonsense verb preferences. Typical Verb-LK sentences link two concepts with one relational verb; whether the model prefers impairs or supports is not the same as its entailment reasoning being broken. The main metric CLR counts only whether one generated word counts as counterfactual; unmodified Llama is already about 0.33 and can rise to near 0.98 after attack. This does not show that the entire counterfactual verb cluster has overtaken the factual cluster.

  2. Gains of ALA over simpler fine-tuning are unstable, and side effects may be hidden by aggregate scores. Plain contrastive loss already approaches the full method on many Llama settings; counterfactual rate on Qwen3-8B peaks at only about 0.52, with optimal configuration shifting with support-word count. MixEval overall drops by less than one point, but BBH and MATH each drop by about seven points. Data and candidate verbs also come largely from GPT-4o; how much of the “logical categories” the attack relies on are artifacts of the labeling pipeline is not separated in the paper.

Questions for authors
  1. For CLR success, must a generated verb fall in a pre-labeled counterfactual set, or is any non-factual verb enough? Please give the decision rules and report whether mean uncertainty of factual vs. counterfactual clusters swaps on the same samples.

  2. With the same candidate words and training steps, what does ALA change beyond supervised learning on counterfactual verbs alone? Please fix one configuration rather than picking the best cell per method from a large table.

2 comments

Thank you for the review. We focus on the two questions, which also cover the main weaknesses.

Q1 (CLR decision rule). The answer differs by model, and we will state both rules in Appendix E.4. For Llama-3.1-8B-Instruct (Tables 3 and 12), the verb does not have to be in the pre-labeled set. The predicted verb is filled into the sentence, and an LLM judge (GPT-4o-mini) counts it as counterfactual if the resulting sentence changes the meaning or logical content of the factual sentence, or contradicts common sense. The factual verb, its inflections and meaning-preserving synonyms are not counted. For Qwen3-8B (Tables 13 and 14), the verb must be in the set: it has to match, up to inflection, a verb in the pre-labeled counterfactual cluster of that item, and any other output, including irrelevant verbs, is not counted. Under this strict rule unmodified Qwen3-8B is at 0.159, compared with about 0.33 for Llama under the broader rule.

On whether the clusters swap, Table 15 compares the mean uncertainty of the factual and counterfactual clusters on the same items. For unmodified Qwen3-8B the factual cluster is preferred (gap -7.92); after training the sign reverses for every method, with the largest gap for CA (+4.05). The Deviation Ratio (Fig. 5) is a per-sample check over the full clusters: it is positive only when every counterfactual verb has lower uncertainty than every factual verb, and CA has the most samples with a positive ratio.

Q2 (fixed configuration). All methods use the same candidates, data, LoRA setting and epochs. With one shared configuration (beta = 0.05, alpha = 0.5, nsn_s = 3):

  • Llama test (Table 12): CA 0.966, contrastive-only 0.943, adaptive-only 0.795, SFT 0.778.
  • Qwen train (Table 13): CA 0.481, contrastive-only 0.219, adaptive-only 0.252, SFT 0.294. The order is the same when this configuration is averaged over all nsn_s. Contrastive-only is close to CA on Llama but falls below SFT on Qwen3-8B, so CA is the only variant that clearly improves on SFT for both models. Beyond SFT, ALA adds uncertainty-weighted support verbs with soft targets (Eq. 10-11) and the contrastive term (Eq. 12); removing the uncertainty weighting lowers CLR (Fig. 6).

On the other points: the BBH/MATH drop on Qwen3-8B mostly comes from fine-tuning on counterfactual relations, since SFT alone loses 6.7 and 4.5 points (Table 17). GPT-4o only proposes candidates, which are checked by three annotators; with Qwen3-8B's own candidates, CA still reaches 0.402 against 0.294 for SFT (Table 14).

Dear Reviewer,

Thank you again for your review. As the discussion period ends soon, we wanted to check whether our response has addressed your concerns. If anything remains unclear, we would be glad to clarify further. If the response resolves your main concerns, we would appreciate it if you would consider updating your assessment.

Best regards, The Authors

Reviewer_ArNrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how changing relation verbs can affect the logical preferences of large language models. The authors introduce Verb-LK, a dataset of 1,122 logical implications, and Adaptive Logic Attack (ALA), which uses semantic uncertainty and contrastive adaptation to shift model preferences toward counterfactual verb clusters. Experiments on Llama-3.1-8B and Qwen3-8B show that the method can change verb-level predictions and generalize to unseen implications, although its effects on more complex reasoning tasks are less consistent.

Critical strengths

Interesting relation-level attack formulation: The paper studies counterfactual manipulation through relation verbs rather than the more commonly explored subject or object changes. Treating related verbs as semantic clusters provides a useful perspective for studying how model predictions change under targeted interventions.

Useful experimental analysis: The authors introduce a new dataset and compare multiple training objectives, support-verb selection strategies, and model configurations. The additional Qwen3-8B experiments, uncertainty analyses, and downstream capability checks provide useful evidence about the effectiveness and limitations of the method.

Critical weaknesses

Limited evidence of broader reasoning disruption: Most of the evaluation focuses on whether fine-tuned models prefer counterfactual verbs in cloze-style predictions, which does not necessarily demonstrate changes in logical reasoning. Although the paper includes a two-hop evaluation, it uses only 45 selected examples and shows inconsistent results across models and relation types, limiting the strength of the broader reasoning claims.

Dataset and generalization concerns: Verb-LK is constructed using GPT-4o-generated examples and curated verb clusters, and the experiments rely on relatively narrow relation patterns. It is unclear whether the reported improvements reflect generalizable logical manipulation or mainly learned preferences for particular verbs and semantic categories.

Downstream capability trade-offs: Although the overall MixEval changes are small, Qwen3-8B shows drops of around seven percentage points on both BBH and MATH after ALA. These results raise concerns about whether the method preserves reasoning capabilities as well as the aggregate scores suggest.

Questions for authors

Could the authors provide a larger evaluation on multi-step reasoning tasks to demonstrate that ALA affects logical reasoning beyond individual verb predictions? It would be particularly helpful to understand why the two-hop results differ substantially between Llama-3.1-8B and Qwen3-8B.

How much of ALA's performance comes from learning specific verb preferences rather than generalizing the underlying logical relationships? Could the authors evaluate on entirely unseen verbs or concept combinations to better separate these effects?

I would also appreciate testing and results on more modern LLMs including reasoning models

2 comments

Thank you for the review and for recognizing the relation-level formulation and the experimental analysis. Our responses are below.

W1/Q1 (reasoning beyond verb prediction). The paper studies the logical shortcut itself. By Eq. 1, a shortcut triplet (s,r,o)(s,r,o) summarizes a multi-hop path PP, and a model can use it in place of explicit reasoning. Changing rr therefore changes the compressed reasoning step the model relies on, so relation preference on these triplets is the quantity this attack targets. The two-hop set is a controlled check of whether this shift carries over to composition, and we do not present it as a general multi-hop benchmark.

On the difference between the two models, a plausible explanation is how strongly CA's relation shift overrides the context. On Llama-3.1-8B-Instruct, CA also changes the predictions on the factual chains: its factual accuracy drops to 0.756, while all other Llama methods stay at 1.000 (Table 10), and it fails on the R07 factual chains (Table 11). The learned relation preference therefore dominates the direction given by the first hop. On Qwen3-8B, CA keeps factual accuracy at 0.978, the model still follows the first hop, and CA gets the highest Full Flip rate (0.778 vs. 0.644 for SFT). We will add this analysis to Appendix F.7.

W2/Q2 (dataset and generalization). GPT-4o is used only to propose sentences and the candidate verb pool (Eq. 5). The items were validated by three annotators (Cohen's kappa 0.60-0.88, Table 6), and all uncertainty scores, center and support selections, and training signals come from the target model itself (Fig. 2). Three results suggest that ALA does not simply memorize particular verbs. First, every test-set result (Table 12, Fig. 3) is on implications not seen in training, i.e., new combinations of concepts, and CA stays at 0.966-0.989 CLR in its best settings. Second, when Qwen3-8B generates its own candidate pool, CA still reaches 0.402 against 0.294 for SFT on the same self-generated pool (Table 14), so the effect is not tied to one fixed verb set. Third, the same objective transfers to StereoSet, where the clusters are adjectives instead of verbs, and lowers the Stereo Ratio from 0.203 to 0.122 (Table 4).

W3 (BBH and MATH). Most of these drops come from fine-tuning on counterfactual relations, not from the ALA objective specifically. SFT alone already reduces BBH by 6.7 points and MATH by 4.5 points on Qwen3-8B; CA adds 0.4 and 2.5 points on top of that (Table 17). Benchmarks that rely less on multi-step inference, such as BoolQ, CommonsenseQA, PIQA, OpenBookQA and GSM8K, stay at or near the base scores. For an attack on reasoning shortcuts, a concentrated effect on reasoning-heavy tasks with a small aggregate change (-0.7 points) is consistent with the intended target. On Llama-3.1-8B-Instruct, CA loses less on MixEval than SFT (-0.70 vs. -1.45 points, Table 19).

Q3 (modern and reasoning models). Qwen3-8B is a recent model with built-in reasoning support. We use its non-thinking mode on purpose. The attack targets shortcut triplets, which by definition are used in place of explicit reasoning, so the non-thinking mode matches the setting we study: the model answers directly from its relation preference. It also allows consistent scoring of forced-choice verb predictions. How explicit reasoning traces interact with relation-level shifts is a natural next step, which we leave for future work.

In the revised version we will (1) add the explanation of the Llama/Qwen two-hop difference to Appendix F.7, (2) revise Sections 4.6 and 5 so that two-hop transfer is described as depending on the model and the schema, and (3) state in the experimental setup that Qwen3-8B is evaluated in non-thinking mode and why.

Dear Reviewer,

Thank you again for your review. As the discussion period ends soon, we wanted to check whether our response has addressed your concerns. If anything remains unclear, we would be glad to clarify further. If the response resolves your main concerns, we would appreciate it if you would consider updating your assessment.

Best regards, The Authors

Reviewer_XHbzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies counterfactual logic attacks on LLMs by modifying relation verbs in logical statements. The authors construct a benchmark called Verb-LK, which contains logical implications with different relation verbs. They also propose Adaptive Logic Attack (ALA), a method that uses semantic uncertainty to select related verbs and applies contrastive adaptation to change the model's preference from factual to counterfactual relations. The authors evaluate the method on Llama-3.1-8B and Qwen3-8B, and conduct additional experiments on multi-hop reasoning and other benchmarks to examine the attack effectiveness and its influence on model performance.

Critical strengths

none

Critical weaknesses
  1. Unclear threat model and practical motivation. The proposed ALA method requires fine-tuning the target model using LoRA. However, the paper does not clearly explain in what real-world situations an attacker can have access to modify the model parameters. Also, standard SFT already increases the counterfactual logic ratio significantly. Therefore, it is unclear why this specific attack method is necessary and what additional advantage it provides in practical settings.

  2. Limited evaluation of actual logical reasoning. Although the paper focuses on attacking LLM reasoning, most experiments only evaluate whether the model prefers counterfactual verbs in single sentences. Changing the predicted verb does not necessarily mean the model's logical reasoning ability is affected. The paper includes a two-hop reasoning experiment, but it only contains 45 selected questions. Moreover, on Llama-3.1-8B, ALA achieves a lower Full Flip rate than standard SFT. Therefore, the current experiments are not enough to show how the proposed attack affects more complex logical reasoning tasks.

Questions for authors
  1. Since the proposed method requires fine-tuning the target model, can the authors explain what attacker capabilities are assumed in this work? How is ALA different from existing knowledge editing methods, such as ROME or MEMIT, and why these methods are not included in the baseline comparisons?

  2. Have the authors considered evaluating ALA on larger multi-hop reasoning benchmarks, instead of mainly using single-sentence verb prediction? Would the attack still be effective when the model needs to perform several reasoning steps or when the relation verbs are not seen during training?

3 comments

Thank you for the review. Our responses are below.

W1/Q1 (attacker capabilities). We assume white-box access: the attacker can read the target model's logits (Eq. 6-8) and fine-tune it with LoRA on a small set of implications. No pretraining data is needed, and no control over the input at inference time is needed either, since after adaptation the shift shows up on clean queries. We do not model a specific real-world adversary. The parameter-level intervention is used to study whether verb-cluster boundaries can be reconfigured, because in Section 4.3 inference-time context rarely crosses them (crossing rate at most 0.042 on Verb-LK).

W1 (advantage over SFT). SFT on counterfactual targets is the simplest form of the attack, and we report it as a baseline throughout. On unseen test implications with Llama-3.1-8B-Instruct (ns=3n_s=3), CA reaches a CLR of 0.966 compared with 0.778 for SFT (one-tailed paired t-test, p=8.9×10−21p=8.9\times10^{-21}, Table 16). On Qwen3-8B, CA is the best method for every nsn_s, up to 0.520 against 0.294 for SFT (Table 13). The two methods also differ in what they change. SFT moves the single target verb, while ALA shifts the counterfactual cluster: the POS-NEG uncertainty gap on Qwen3-8B is 4.05 for CA and 1.60 for SFT (Table 15), and CLR drops when the uncertainty-weighted support selection is removed (Fig. 6). This is the concept distortion problem that motivates the method (Fig. 1). On Llama, CA also costs less general capability than SFT on MixEval (-0.70 vs. -1.45 points, Table 19).

Q1 (ROME/MEMIT). ROME and MEMIT edit the object of a fact, (s,r)→(s,r,o~)(s,r)\rightarrow(s,r,\tilde{o}), by locating a subject key in the MLP layers and writing one new target string. This is the general counterfactual attack of Eq. 3 and Fig. 1. In our setting ss and oo stay fixed and the relation verb is reversed (Eq. 2), so the edited element encodes an implication rather than a recalled object. Our objective also works on the verb cluster. A single-string edit creates one deleted-inserted pair and leaves the related counterfactual verbs untouched, which is the concept distortion we analyze. Under the same cloze format, our SFT baseline, trained only on the counterfactual center verb, plays the role of such a minimal-pair edit. Finally, knowledge-editing benchmarks measure fact recall (efficacy, paraphrase, specificity), whereas we measure cluster-level preference shifts and their generalization to unseen implications. For these reasons, knowledge-editing methods are not directly comparable baselines for relation-level attacks, so we do not include them in the experiments. We will add a short discussion of this distinction to Section 2.

W2/Q2 (logical reasoning). The paper studies the logical shortcut itself. By Eq. 1, a shortcut triplet (s,r,o)(s,r,o) summarizes a multi-hop path PP, and a model can use it in place of explicit reasoning. Changing rr therefore changes the compressed reasoning step the model relies on, so relation preference on these triplets is the quantity this attack targets. All test-set results (Table 12, Fig. 3) are on implications not seen during training, and CA stays at 0.966-0.989 CLR in its best settings. The MixEval breakdown on Qwen3-8B also shows that the effect falls mostly on reasoning-heavy tasks: BBH drops 7.1 points and MATH 7.0 points under CA, while the overall score changes by 0.7 points (Table 17).

W2 (two-hop results on Llama). We agree that on Llama-3.1-8B-Instruct CA has a lower Full Flip rate than SFT (0.378 vs. 0.622), as reported in Appendix F.7. The result depends on the schema. CA raises R06 from 0.056 to 0.556 but fails on R07. On Qwen3-8B, CA has the highest Full Flip rate (0.778 vs. 0.644 for SFT), mainly from R06 (0.944). The 45-item set is meant as a controlled check of compositional transfer and not as a general multi-hop benchmark.

In the revised version we will (1) state the white-box assumptions and the scope of the study in the introduction, (2) add a short discussion of how our setting differs from ROME/MEMIT to Section 2, and (3) revise Sections 4.6 and 5 so that two-hop transfer is described as depending on the model and the schema, in line with Appendix F.7.

Dear Reviewer,

Thank you again for your review. As the discussion period ends soon, we wanted to check whether our response has addressed your concerns. If anything remains unclear, we would be glad to clarify further. If the response resolves your main concerns, we would appreciate it if you would consider updating your assessment.

Best regards, The Authors

Thank you for the rebuttal. After reviewing the authors' response,I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.