- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies semantic omission, where a model must complete a writing task while keeping a specified semantic dimension out of the output. Paraphrase, implication, contrast and similar rhetorical routes all count as leakage. The authors build a benchmark of 100 LLM-generated scenarios across six domains, each instantiated in four formats (Direct, Open-ended, Rewrite, Multi-turn), for 400 tasks. Outputs are scored by an LLM judge on omission and task fulfillment.
The diagnosis covers five open-weight models in the 4B to 9B range, plus API models in the appendix. Its main findings are three. Failures are common and concentrate in Rewrite Removal. Models that correctly articulate the boundary in a separate probe still leak. Clearer prompts, including an Oracle boundary, reduce but do not remove leakage.
The proposed method, SelectOmit, is a LoRA post-training objective with four terms. These are token-level unlikelihood on annotated leakage spans, cross-entropy on clean responses, a DPO-style preference of clean over utility-degraded responses, and span-level cross-entropy on counterfactual inclusion responses. On Qwen3-4B, valid success rises from 80.0% to 97.5%. Paired include/exclude validity reaches 81.5%, against 62.75% for standard DPO, and four-way compositional success reaches 74%. Results are supported by human validation, ablations and a 50-document authentic-source evaluation.
Critical strengths
- The paper reframes negative instruction following as instruction-conditioned semantic control. Prior work asks whether a model understands a negated request. This paper asks whether the same model can make the same content absent or present depending only on the current instruction, which is a stricter and more practically meaningful target. Defining success over paired include/exclude conditions makes over-suppression measurable for the first time in this setting. Standard DPO looks strong on exclusion (95.75%) yet fails on inclusion (64.5%).
- The experiments are comprehensively designed. Each loss term targets one failure mode the formulation exposes. Span-matched Conditional SFT and Positive-matched SFT receive the same inclusion data as SelectOmit, which rules out the simplest explanation that the gain comes from extra counterfactual data. Moreover, blinded human evaluation, authentic-source tests and a stratified 600-response judge validation further strengthen the empirical case.
Critical weaknesses
- The methodological novelty appears somewhat modest. The conditional-control framing is conceptually close to the bidirectional objective of RePS (line 464), and the paper could explain more clearly why applying it through ordinary instructions calls for a new method. SelectOmit combines four established components with fixed weights, with little analysis of why this combination yields conditional control, and the utility-preference term shows no significant effect (+1.25, CI [−1.00, +3.75]).
- The evidence for the two central claims could be strengthened. The identification probe is a separate call, and its rates (60.75% to 78.75%) are lower than valid success (80.42% to 95.25%). Without comparing leakage between identified and non-identified items, the knowing/doing dissociation is hard to separate from probe noise. On repair, the exclusion gain over DPO is +1.0 pp with a CI including zero. DPO saw no inclusion pairs, so its weaker inclusion is somewhat expected, and a bidirectional DPO baseline would be more convincing. SelectOmit also lowers inclusion validity from 92.25% (Base) to 83.75%.
Questions for authors
- How does SelectOmit compare with DPO trained on both exclusion pairs (clean over leaking) and inclusion pairs (target-present over target-absent)?
- What is the leakage rate among items where the probe failed to identify the boundary? If it is close to the rate among identified items, how do you distinguish a knowing/doing dissociation from a probe that is only weakly coupled to generation?
- Every trained method lowers inclusion validity below Base. Can you characterize what SelectOmit loses on inclusion, and whether it suppresses neighboring semantics as well?
If the authors can address the weaknesses and questions raised above, I would raise my score.