- The connection between the learned intervention effects and the intended logical property remains insufficiently established.
The central claim relies on interpreting the selective intervention effects as evidence of sensitivity to the nonempty-term assumption. However, as acknowledged in the paper, the assumption-dependent and assumption-independent groups also differ in correlated formal features, particularly their quantifier patterns. Although the random-grouping controls demonstrate that the semantic grouping is more easily separated than arbitrary groupings, they do not fully establish which features the learned prefixes actually exploit.
Furthermore, the primary same-answer prompts do not explicitly specify the nonempty-term convention (Appendix A.3). While the convention-switching experiments provide additional evidence, the convention interactions are statistically reliable primarily when assumption-independent forms are targeted, rather than in the assumption-dependent targeting direction that produces the strongest flip gaps (Table 3). Consequently, the evidence does not yet clearly distinguish sensitivity to the intended logical convention from sensitivity to correlated structural patterns or prompt-dependent heuristics. This ambiguity weakens the interpretation of the main result.
- The generality of the proposed behavioral probing framework is limited by the narrow evaluation setting and lack of transfer across equivalent task formulations.
The main semantic intervention experiments rely on a small number of independent logical pairs (eight eligible pairs for each Qwen model), all drawn from the same syllogistic reasoning benchmark. Although the authors provide statistical sensitivity analyses, the limited diversity of logical structures makes it difficult to assess how broadly the findings generalize.
More importantly, the strong separation observed in the validity task largely disappears when the same logical relationship is expressed through an equivalent inconsistency task (Section 5.7 and Appendix D.4). Even prefixes trained directly on the dual formulation do not exhibit statistically reliable separation. While the paper acknowledges this limitation, it raises a fundamental question about whether the learned prefixes reveal generalizable properties of reasoning or primarily task-specific response patterns. This significantly limits the broader implications of the proposed framework.
Thank you for recognizing our experiments and for your thoughtful questions. We ran follow-up tests to address them.
Q1. Is the prefix responding to the nonempty-term assumption, or just simple patterns?
We agree random groupings cannot settle this. In our original pairs, dependent syllogisms have only "All"/"No" premises, while independent partners contain "Some". The prefix could potentially exploit this.
First, we tested syllogisms with exactly the same premises but different conclusions. For example, consider the premises "All X are Y" and "All Y are Z". Under our convention, both conclusions are valid: "Some X are Z", which requires X to exist, and "All X are Z", which remains valid even if X is empty. Across five held-out comparisons, dependent-targeting prefixes still changed that group's answers much more often, with flip gaps of 91.6 percentage points in Qwen3.6 and 87.2 in Qwen3-8B. This cannot rely only on "Some" in the premises.
Second, we changed the training data to make that shortcut insufficient. We added independent examples that also have only "All"/"No" premises. Checking for "Some" in the premises could no longer distinguish the groups. We trained fresh prefixes and found that they could still distinguish unseen problems with identical premises.
This does not eliminate all structural explanations. The original pairs shared conclusions while the new ones share premises. A combined premise/conclusion pattern could still work. Our aim is to rule out simpler explanations and characterize behavior. Inevitably there will always be patterns to logic.
Q2. Why do Sections 5.2 and 5.4 show different results?
The two experiments deliberately test different things. In Section 5.2, we change the syllogisms while keeping the logical convention fixed. In Section 5.4, we keep the syllogisms and prefixes fixed but change the convention.
As you noted, these changes affect the prefixes in different ways. We find this the interesting part. How easily a prefix distinguishes logical problems and how it responds to changes in logical rules are separate aspects of the model's behavior. Understanding these differences is precisely our aim.
The groups remain distinguishable when nonemptiness is stated explicitly. Changing the convention affects the prefixes most clearly for independent targeting (average interaction 0.72 [0.10, 1.37] in score), though the effect varies by wording.
Q3. Why doesn't the prefix work on the equivalent task? Is it simply harder to learn?
We tested two questions: Does a prefix trained on one task work on the other? And can we train a new prefix directly on the other task?
First, we checked whether the result was simply caused by comparing different problems. In our original experiments, we only tested problems that the model answered correctly before adding a prefix. However, the model sometimes answered a problem correctly in one version of the task but not the other. This meant we were potentially comparing different sets of problems.
To eliminate this possibility, we repeated the experiment using exactly the same problems that the model answered correctly in both versions. We also used the same learned prefixes. This way, the only thing that changed was how we asked the logical question. The flip gaps are:
Strong separation still does not carry over to the equivalent task, even after longer training on validity and evaluation on identical examples. Some changes in the answer scores do carry over.
But this does not mean the equivalent task cannot be learned. We separately trained prefixes directly on that task, extending all runs to 1,000 steps. They now distinguish unseen problems in both directions: 53.4 and 68.0 points in Qwen3.6, and 21.9 and 29.7 points in Qwen3-8B.
So the conclusion is simple: both tasks can support selective interventions, but a prefix trained on one need not work on the other. We see this task dependence as part of the behavioral profile.
We acknowledge the small number of logical forms and limited generality. We see this as a promising proof of concept for studying more complex logical problems and developing richer behavioral profiles across models. We will include the new results and limitations in the revision, and welcome further discussion.