- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether language models can be shifted from preferring factual relational verbs to preferring a set of counterfactual verbs through targeted modification. The authors treat verbs such as impairs as logical relations linking two concepts and partition candidate verbs into factual, counterfactual, and irrelevant clusters. They build Verb-LK with 1,122 verb entailments and first examine cluster-level preferences in unmodified models and whether in-context examples at inference time can flip those preferences.
They then propose Adaptive Logic Attack (ALA). It uses the target model’s semantic uncertainty to select centroid and support words in the factual and counterfactual clusters, then fine-tunes the model with a contrastive adaptation loss to push the decision boundary toward the counterfactual cluster. Experiments focus mainly on Llama-3.1-8B-Instruct, with Qwen3-8B as a supplement. The authors report that the attack changes verb preferences on seen and unseen entailments, exploratory anti-stereotype effects on StereoSet, and relatively small overall changes on MixEval.
Critical strengths
- The paper moves the attack from replacing one relational word to shifting a set of semantically similar verbs, and shows that adding counterfactual examples in context alone usually cannot flip the model’s strong prior preferences. Parameter updates can noticeably change open-source model outputs on such cloze items.
Critical weaknesses
-
The paper’s “logic attack” is largely a change in commonsense verb preferences. Typical Verb-LK sentences link two concepts with one relational verb; whether the model prefers
impairsorsupportsis not the same as its entailment reasoning being broken. The main metric CLR counts only whether one generated word counts as counterfactual; unmodified Llama is already about 0.33 and can rise to near 0.98 after attack. This does not show that the entire counterfactual verb cluster has overtaken the factual cluster. -
Gains of ALA over simpler fine-tuning are unstable, and side effects may be hidden by aggregate scores. Plain contrastive loss already approaches the full method on many Llama settings; counterfactual rate on Qwen3-8B peaks at only about 0.52, with optimal configuration shifting with support-word count. MixEval overall drops by less than one point, but BBH and MATH each drop by about seven points. Data and candidate verbs also come largely from GPT-4o; how much of the “logical categories” the attack relies on are artifacts of the labeling pipeline is not separated in the paper.
Questions for authors
-
For CLR success, must a generated verb fall in a pre-labeled counterfactual set, or is any non-factual verb enough? Please give the decision rules and report whether mean uncertainty of factual vs. counterfactual clusters swaps on the same samples.
-
With the same candidate words and training steps, what does ALA change beyond supervised learning on counterfactual verbs alone? Please fix one configuration rather than picking the best cell per method from a large table.
2 comments
Dear Reviewer,
Thank you again for your review. As the discussion period ends soon, we wanted to check whether our response has addressed your concerns. If anything remains unclear, we would be glad to clarify further. If the response resolves your main concerns, we would appreciate it if you would consider updating your assessment.
Best regards, The Authors
Thank you for the review. We focus on the two questions, which also cover the main weaknesses.
Q1 (CLR decision rule). The answer differs by model, and we will state both rules in Appendix E.4. For Llama-3.1-8B-Instruct (Tables 3 and 12), the verb does not have to be in the pre-labeled set. The predicted verb is filled into the sentence, and an LLM judge (GPT-4o-mini) counts it as counterfactual if the resulting sentence changes the meaning or logical content of the factual sentence, or contradicts common sense. The factual verb, its inflections and meaning-preserving synonyms are not counted. For Qwen3-8B (Tables 13 and 14), the verb must be in the set: it has to match, up to inflection, a verb in the pre-labeled counterfactual cluster of that item, and any other output, including irrelevant verbs, is not counted. Under this strict rule unmodified Qwen3-8B is at 0.159, compared with about 0.33 for Llama under the broader rule.
On whether the clusters swap, Table 15 compares the mean uncertainty of the factual and counterfactual clusters on the same items. For unmodified Qwen3-8B the factual cluster is preferred (gap -7.92); after training the sign reverses for every method, with the largest gap for CA (+4.05). The Deviation Ratio (Fig. 5) is a per-sample check over the full clusters: it is positive only when every counterfactual verb has lower uncertainty than every factual verb, and CA has the most samples with a positive ratio.
Q2 (fixed configuration). All methods use the same candidates, data, LoRA setting and epochs. With one shared configuration (beta = 0.05, alpha = 0.5, ns = 3):
On the other points: the BBH/MATH drop on Qwen3-8B mostly comes from fine-tuning on counterfactual relations, since SFT alone loses 6.7 and 4.5 points (Table 17). GPT-4o only proposes candidates, which are checked by three annotators; with Qwen3-8B's own candidates, CA still reaches 0.402 against 0.294 for SFT (Table 14).