acceptodds
Under review as a conference paper at ICLR 2027

SemaSync: Robust Semantic-State Watermarking for Structured LLM Generation

Abstract

Large language models (LLMs) increasingly generate structured outputs such as mathematical solutions and software code, creating a growing need for reliable provenance verification. Watermarking these outputs faces two challenges: rewriting can disrupt token-dependent watermark states and weaken detection, while indiscriminate sampling biases can compromise correctness. We introduce SemaSync, a watermarking framework that uses semantic and structural anchors to support synchronization after rewriting. Multi-anchor voting provides redundancy when individual anchors change, while distribution-aware selective embedding balances watermark detectability with output quality. We establish a margin-based partition stability guarantee and use ablations to clarify the roles of state construction, selective embedding, and multi-anchor voting. Across GSM8K and MBPP with Qwen3.5-9B and Llama-3.1-8B-Instruct, SemaSync achieves higher AUROC on unmodified outputs than all evaluated baselines in all settings. On Qwen GSM8K, SemaSync improves TPR at 1% FPR by 20.1% on unmodified outputs and 44.2% after Mistral rewriting relative to the strongest baseline in each condition with moderate accuracy loss.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Watermarking, Robust Watermark Detection, Structured Text Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_veMaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores digital watermarking techniques for content generated by structured Large Language Models (LLMs), focusing specifically on mathematical reasoning and code generation—scenarios where indiscriminate biasing of token selection could compromise answer correctness or program functionality. The authors propose SemaSync, a semantic-state-based watermarking framework. This framework derives watermarking states from normative anchors inherent in mathematics and code (such as numbers, operations, literals, and control-flow events), constructs dynamic "green-list token" partitions via a multi-anchor voting mechanism, and employs distribution-aware carrier selection rules to apply watermarking biases only at suitable generation points.

The researchers evaluated the method using Qwen3.5-9B and Llama-3.1-8B-Instruct models on the GSM8K and MBPP datasets, comparing it against existing watermarking baselines in both "raw generation" and "LLM-based paraphrasing" scenarios. Results demonstrate that SemaSync achieves excellent detection performance while maintaining superior task utility; the paper also includes ablation studies regarding semantic state construction, multi-anchor voting, and carrier selection mechanisms. Overall, this work presents a well-motivated and practically valuable watermarking scheme for structured LLM outputs that offers performance improvements over baseline methods.

Critical strengths
  1. Problem formulation driven by clear motivation. This paper identifies the practical limitations of existing token-level watermarking methods in mathematical reasoning and code generation tasks: indiscriminate watermark-induced bias can compromise the correctness of the generated content, whereas mechanisms relying on token history are vulnerable to failure upon text rewriting. Constructing watermark states using semantic and structural anchors offers a sound, task-aligned solution to this challenge.

  2. This paper encompasses a wide range of evaluation dimensions, including two types of structured generation tasks, two Large Language Model (LLM) backbones, multiple baseline watermarking methods, original versus rewritten outputs, task utility, runtime overhead, and targeted ablation studies. These extensive evaluations provide compelling evidence that the proposed method achieves excellent detection performance without significantly degrading generation quality.

Critical weaknesses
  1. Limited conceptual innovation. Although the problem formulation is well-motivated, the proposed approach primarily operates within the existing "green-list" watermarking framework, combining task-specific semantic anchor extraction, multi-anchor voting, and selective watermark embedding. While these design choices are sound, they represent engineering-level improvements rather than the substantial conceptual breakthroughs required to meet ICLR standards.

  2. Performance gains are not robust. Across the various tasks, models, and evaluation metrics involving text rewriting, SemaSync does not consistently outperform baseline methods. For instance, when using the Mistral model for rewriting, its AUROC on the Qwen model's MBPP task falls short of the Unigram and SWEET methods; similarly, on the Llama model's GSM8K task, it underperforms compared to the Unigram and SIR methods. The method also fails to demonstrate a significant advantage when using the more powerful Luna model for rewriting. Furthermore, compared to the baseline method using a simple Token + Gate approach, the AUROC for the rewritten MBPP task improved by only 0.012. These results indicate a lack of sufficient evidence to support the broad claim that the method achieves robust semantic-state watermarking.

Questions for authors

Clarification Requests

  1. The paper’s main novelty appears to be constructing dynamic green lists from task-specific semantic/structural anchors, rather than token history, and skipping unsuitable carrier positions through a green-mass gate. Can the authors more clearly explain what conceptual capability this provides beyond existing semantic watermarking and selective-embedding methods, such as SemStamp and SIR, rather than viewing it as a task-specific engineering adaptation?

  2. I view the green-mass gate primarily as a threshold-based heuristic for avoiding large deviations from the base distribution, rather than a core conceptual contribution. More importantly, qtq_t measures only the total base probability mass of the green set and does not explicitly model semantic correctness. Why should this mechanism adequately protect correctness-critical tokens and structures, such as numerical values, operators, function signatures, and control-flow decisions? In difficult mathematical reasoning, both the correct answer and plausible incorrect alternatives may have non-negligible probability.

  3. The direct Token + Gate ablation shows only a modest rewrite-time gain on MBPP (AUROC 0.682 to 0.694). To establish whether this improvement supports the broader claim of robust semantic-state watermarking, could the authors provide three targeted experiments: (i) multi-seed results with confidence intervals for the direct SemaSync versus Token + Gate comparison, to assess statistical reliability; (ii) the same comparison under a stronger or qualitatively different rewriter, to test whether the gain generalizes beyond the current rewrite protocol; and (iii) results under alternative reasonable anchor ontologies, to quantify how strongly the claimed gain depends on task-specific anchor design?

1 comment

Thank you for these questions.

Q1: Capability beyond existing semantic watermarking.

SemStamp uses an auxiliary sentence encoder and repeatedly samples sentences until their embeddings fall in accepted semantic regions. SIR uses an auxiliary encoder and a trained watermark network to map prefix semantics to token-level logit biases. SemaSync instead normalizes explicit mathematical or code events into anchors: when rewriting preserves the active anchors, the detector can reconstruct the same token partitions despite changes in wording. It needs neither sentence rejection sampling nor auxiliary neural networks, but requires domain-specific extraction rules and generator replay for detection.

Under the shared Llama-MBPP protocol in Table 27 (one H100, batch size 1, 50 prompts), generation takes 0.535 s/sample for SemaSync versus 6.905 for SemStamp and 1.405 for SIR: 12.9× and 2.6× speedups, respectively.

Q2: Does green mass protect correctness?

Section 3.3 gives the following relation under fixed sampling support:

qt(δ)=eδqt1−qt+eδqt.q_t^{(\delta)}=\frac{e^\delta q_t}{1-q_t+e^\delta q_t}.

At low qtq_t, the bias promotes choices that the base model considers unlikely: for example, with δ=3\delta=3, green probability rises from 5% to 51.4%. The lower bound avoids such interventions, which can harm task correctness. At high qtq_t, clean text already favors green tokens, and qt(δ)−qtq_t^{(\delta)}-q_t approaches zero as qt→1q_t\to1. These positions add little distinction between clean and watermarked text and can contribute positive clean scores under the nominal-γ\gamma statistic, motivating the upper bound.

Table 4 supports these distinct roles. Removing the lower bound reduces MBPP Pass@1 from .538 to .464 and GSM8K accuracy from .928 to .904. Removing the upper bound leaves MBPP Pass@1 similar (.542 versus .538), but lowers raw AUROC from .884 to .588.

Q3: Contribution of semantic states.

We chose Mistral as a qualitatively different rewriter for outputs generated by Qwen and Llama because it comes from a model family distinct from both. To assess the contribution of semantic states while controlling for the gate and voting configuration, we added a comparison between SemaSync K3 and Token-vote3+Gate. Both use the same gate rule and are configured for three-voter majority voting; their votes come from semantic anchors and token history, respectively. The table reports AUROC after Mistral rewriting and paired 95% confidence intervals for the difference.

DatasetSemaSync K3Token-vote3+GateΔAUROC [95% CI]
GSM8K (500 tasks, 1 seed)0.6920.631+0.061 [0.016, 0.107]
MBPP (500 tasks, 3 seeds)0.7190.717+0.002 [-0.019, 0.023]

After Mistral rewriting, SemaSync K3 significantly outperformed Token-vote3+Gate on GSM8K (+0.061 AUROC), while the three-seed gain on MBPP was small (+0.002) and not statistically significant. Our anchor rules are heuristic, and the code extractor can miss structures during generation. For example, it can miss an anchor on the final line when no newline follows. Even with these simple rules, semantic anchors improved detection on GSM8K. We also transferred the original GSM8K rules to MATH and the original MBPP rules to HumanEval+ without re-tuning. After Mistral rewriting, SemaSync achieved 0.814 AUROC on MATH, significantly outperforming all eight external baselines. On HumanEval+, its AUROC was close to SWEET’s (0.620 vs. 0.625), and it achieved the highest pass@1 among the evaluated watermarking methods (0.750).

Reviewer_zFctOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SemaSync, a watermarking framework for structured LLM generation, particularly mathematical reasoning and Python code. SemaSync extracts deterministic semantic or structural anchors from the generated prefix, uses keyed majority voting over recent anchors to construct a green-token partition, and applies a fixed logit bias only when the base model's green probability mass lies within a prescribed interval. At detection time, it reconstructs anchors and carrier decisions from the observed text using replay with the original model and tokenizer.

Critical strengths

The paper addresses a meaningful problem for structured LLM outputs, for example, mathematical derivations and code. The idea of using canonical content events as synchronization signals is a sensible and potentially useful design direction for such domains. The method is clearly decomposed into state construction, multi-anchor voting, and selective carrier selection, which makes the design easier to inspect than a monolithic watermarking procedure.

Critical weaknesses
  1. The state-source ablation in Table 3 does not sufficiently establish that semantic synchronization, rather than the carrier gate or other task-specific components, produces the reported rewrite-resilience gain. Relative to Token + Gate, SemaSync improves rewritten AUROC from 0.7510.751 to 0.7860.786 on GSM8K and from 0.6820.682 to 0.6940.694 on MBPP. The latter difference is particularly small, and the corresponding MBPP TPR@1% increases only from 0.1020.102 to 0.1100.110. Appendix D.12 (Table 28) explicitly states that these state-source comparisons are point estimates without paired confidence intervals. Thus, it is unclear whether the key semantic-state gains particularly on MBPP are statistically reliable. Could the authors provide paired confidence intervals or paired tests for these differences?

    The MBPP setting is further complicated by the previous-token pseudo-anchor initializer. Table 11 shows that the first valid K=3 semantic state appears only at 76.78% of the completion on average among K3-covered MBPP samples. Before the first structural anchor completes, the full method uses a previous-token pseudo-anchor. Table 29 shows that this initializer contributes 26.1% of scored positions on raw outputs and 29.6% after Mistral rewriting; moreover, its initializer-only channel has higher AUROC than the semantic-only channel in both conditions. Therefore, the MBPP results cannot be directly interpreted as evidence that anchor-derived semantic states are the main driver of performance. The ``without initializer'' comparison in Table 28 does not isolate this contribution, because removing the initializer also removes early carrier positions and changes the total amount of evidence, while the authors note that this comparison is not utility-matched. Could the authors provide an evidence-/carrier-count-matched control that separates the effect of semantic anchors from that of the initializer?

  2. Theorem 1 provides only a local majority-vote property and does not establish robustness to realistic rewriting or improved end-to-end detection. It states that a token's partition label remains invariant to at most rr voter replacements iff its majority-vote margin exceeds 2r2r. This deterministic statement is correct, but it is essentially a basic fact about majority voting. The independent-mask calculation in Appendix B further shows that more odd-numbered voters increase the expected fraction of labels satisfying this local condition under an idealized independent-mask model.

    However, the paper does not explain how a realistic meaning-preserving rewrite corresponds to bounded voter replacement. A rewrite can delete anchors, change their canonical payloads, reorder events, or alter the active anchor window. Nor does the theorem relate partition-label stability to retained watermark score, AUROC, or TPR. In fact, the controlled corruption results in Figure 2 and Table 25 show that the local stability guarantee does not automatically yield a global robustness advantage: at sufficiently high corruption, K=1K=1 outperforms K=3K=3 on both GSM8K and MBPP. The theorem should be framed as a limited characterization of the voting mechanism, rather than as theoretical support for end-to-end rewrite resilience.

  3. The GSM8K rewrite evaluation mixes genuinely rewritten outputs with unchanged source outputs. As described in Appendix D.4 and quantified in Table 16, if a rewrite fails the answer-preservation checks, the pipeline falls back to the original source text, and these rows remain in the 500-sample detection population. The fallback rates are nonzero and differ across methods. Although SemaSync is not the most extreme case, its reported post-rewrite metrics in Table 1 partly reflect samples that were never actually rewritten. This makes it difficult to interpret the reported numbers purely as robustness to rewriting.

  4. Tables 1 and 2 show that SemaSync is a competitive complete system in the tested structured-generation settings, and Table 4 provides useful evidence for the two-sided carrier gate. However, semantic-anchor synchronization does not consistently outperform strong alternatives after rewriting. In Table 2, Unigram and SIR outperform SemaSync on Llama GSM8K after Mistral rewriting. In the open-domain extension (Table 12), SemaSync has the lowest raw AUROC among the compared methods and only mid-range rewritten AUROC. Under the strongly compressive Luna rewrite, Table 20 shows that SemaSync's GSM8K AUROC falls from 0.7860.786 to 0.5420.542, and Table 21 shows that it is not significantly distinguishable from several other remaining above-chance methods. These results seem to show that the contribution should be a promising approach for structured domains with sufficiently stable and sufficiently early anchors, rather than a broadly established solution to semantic synchronization under rewriting.

Questions for authors

See questions in weaknesses.

1 comment

Thank you for the thoughtful review.

W1: Contribution of semantic anchors.

In SemaSync, the state determines which tokens are green, and the gate determines when to apply the watermark bias. Before the first code anchor appears, BOOT uses the previous token to define the green list, and the gate selects positions for embedding. Without BOOT, embedding starts only after the first anchor. The AUROC drop therefore shows that early watermark evidence is useful, but it does not tell us which state source works better at comparable positions.

To test the contribution of the state source more directly, we compared SemaSync K3 with Token-vote3+Gate. Both use the same gate rule. On MBPP, SemaSync K3 starts with the previous token. After the first code anchor is recognized, it uses the latest anchor when the window contains one or two anchors, and a majority vote over the latest three when it contains at least three. Token-vote3+Gate votes over the three most recent ordinary tokens throughout, starting with the last three ordinary tokens in the prompt. The table reports AUROC after Mistral rewriting and paired 95% confidence intervals for the difference.

DatasetSemaSync K3Token-vote3+GateΔAUROC [95% CI]
GSM8K (500 tasks)0.6920.631+0.061 [0.016, 0.107]
MBPP (500 tasks)0.7190.717+0.002 [-0.019, 0.023]

After Mistral rewriting, SemaSync K3 significantly outperformed Token-vote3+Gate on GSM8K (+0.061 AUROC), while the three-seed gain on MBPP was small (+0.002) and not statistically significant. Our anchor rules are heuristic, and the code extractor can miss structures during generation. For example, it can miss an anchor on the final line when no newline follows. Even with these simple rules, semantic anchors improved detection on GSM8K. We also transferred the original GSM8K rules to MATH and the original MBPP rules to HumanEval+ without re-tuning. After Mistral rewriting, SemaSync achieved 0.814 AUROC on MATH, significantly outperforming all eight external baselines. On HumanEval+, its AUROC was close to SWEET’s (0.620 vs. 0.625), and it achieved the highest pass@1 among the evaluated watermarking methods (0.750).

W2: Scope of Theorem 1.

We agree that Theorem 1 characterizes local partition stability under bounded voter replacement, rather than guaranteeing end-to-end robustness to rewriting. The submitted paper already states in Section 4.7 that this local protection does not ensure an overall detection advantage, and in Section 5 that the guarantee does not cover arbitrary deletion, compression, or restructuring. We will make this distinction more explicit alongside Theorem 1, separating the voting property from empirically evaluated rewrite robustness.

W3: Fallbacks versus actual rewriting.

We agree that including unchanged source fallbacks makes the reported results harder to interpret as robustness to rewriting. We therefore added an analysis excluding these fallbacks. On GSM8K under Mistral rewriting, SemaSync's AUROC decreases from 0.7861 to 0.7775. Nevertheless, on pairwise matched subsets where SemaSync, the baseline, and Clean outputs were all genuinely rewritten, SemaSync retains statistically significant AUROC advantages over all six baselines; for example, the difference versus SWEET is +0.0549 [0.0141, 0.0964] on 460 shared tasks. We will report these conditional-subset results separately from the full-pipeline results.

W4: Scope of the contribution.

We agree that SemaSync's effectiveness depends on sufficiently stable and early anchors, and we will clarify this scope in our claims. The new MATH transfer results use the original GSM8K ontology without re-tuning. Below we compare with SWEET, the strongest external baseline by AUROC in both conditions.

MATH

MethodRaw AUROCRaw AccuracyMistral rewrite AUROCMistral rewrite Accuracy
SWEET0.9760.860*0.7700.904
SemaSync0.9920.860*0.8140.920*

* indicates no statistically significant advantage over the other method. SemaSync's AUROC gains are significant in both conditions: Raw +0.016 [0.008, 0.025] and Mistral +0.044 [0.009, 0.079]. These results support the transferability of SemaSync to a new structured reasoning dataset without ontology re-tuning.

Reviewer_HfV9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents SemaSync, a watermarking method for structured LLM outputs such as math solutions and code. Token-history states break under rewriting, so SemaSync instead anchors the watermark state to canonical content events, including numbers, operations, and control-flow constructs. Keyed masks from up to three recent anchors determine the green list by majority vote, and the paper provides a margin-based theorem on label stability. For code, a token-based initializer covers the prefix before the first anchor. A two-sided gate limits correctness loss from uniform biasing by applying a fixed bias only when the base model’s green probability mass lies in [0.10,0.90][0.10, 0.90]. Detection replays the model on the observed text to reconstruct both the watermark states and the gate decisions. The evaluation covers GSM8K and MBPP with two generators and two rewriters.

Critical strengths
  1. The paper builds watermark states from canonical content events and multi-anchor voting instead of previous tokens. Because these anchors tend to survive paraphrasing, the design improves detection after length-preserving rewriting.
Critical weaknesses
  1. The main results rest on a mild rewriter and a single dataset per domain. The paper does not explain why Mistral-Small-3.2-24B is chosen as the primary rewriter. Under the more compressive GPT-5.6 Luna rewriter, KGW achieves the highest AUROC and TPR@1% on GSM8K in Table 21. However, this comparison appears only in the appendix, covers only Qwen, and omits Unigram, SIR, and STONE. The anchor ontology is also a set of task-specific rules, yet each domain is evaluated on one dataset, so the rules can be tuned to its format. The reported robustness is thus untested under stronger rewriting and on other datasets within the same domain.
  2. On code, the anchor-based state depends on the token-based initializer, which is a KGW-style previous-token state. In Table 29, 172 of 500 watermarked MBPP outputs have no semantic-channel evidence. The initializer supplies only 26.1% of scored positions, yet its channel alone achieves an AUROC of 0.813, higher than the 0.766 of the semantic channel. Removing the initializer lowers MBPP raw AUROC from 0.884 to 0.741 in Table 28, below the 0.860 of SWEET at a comparable Pass@1. The MBPP gains therefore rely largely on a token-history mechanism that the paper aims to replace.
Questions for authors
  1. The gate depends on the distribution replayed on the observed prefix. After rewriting, the detector can therefore select different carriers than the generator did. How much do the carrier sets overlap before and after rewriting?
  2. Table 29 shows that 172 of 500 watermarked MBPP outputs have no semantic-channel evidence. How many of these completions contain no anchor at all? For the others, is the missing evidence caused by anchors appearing only near the end, by an empty anchor window, or by the gate?
  3. Why was Mistral chosen as the primary rewriter rather than Luna? In addition, why do the Luna experiments in Table 21 omit Unigram, SIR, and STONE? Unigram in particular achieves the highest rewritten AUROC on Qwen MBPP under Mistral.
  4. The deduplication key κt=(Ht,yt)\kappa_t = (H_t, y_t) retains anchor order, while the majority-vote green list does not. States with the same anchors in a different order therefore yield the same green list but count as different keys, so the same decision is counted more than once. This case is likely frequent in code, where a small set of structural anchors recurs in varying orders. How often does it occur on each dataset, and does an order-invariant key change the reported scores?
  5. The voting ablation examines only K=1 and K=3. Appendix B shows that the protected fraction of labels grows with the voter count. What is the justification for fixing K at 3?
  6. The evaluation uses one dataset per domain, with a task-specific anchor ontology. Does the ontology generalize to other datasets within the same domain, such as MATH [1] or HumanEval [2], without re-tuning the rules?

[1] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt, “Measuring Mathematical Problem Solving With the MATH Dataset”, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.

[2] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al., “Evaluating Large Language Models Trained on Code”, arXiv, 2021.

2 comments

Thank you for the detailed questions.

Q1. Carrier overlap. In an additional audit after Mistral rewriting, 27.6% of all original GSM8K carriers matched a rewritten token selected again by the gate (500 outputs). On MBPP (500 tasks, three seeds), 39.4% of all original scored events were retained. Of these retained events, 93.4% were green.

Q2. Zero semantic evidence. Of the 172/500 MBPP outputs with no semantic evidence, 94 had no anchor recognized during generation. For example, the tracker can miss a structure on the final line if no newline follows. In the other 78, the gate excluded every candidate position after a recognized anchor. None had an empty anchor window or an anchor too late to leave a position to score. Improving how the tracker handles the final line is one direction for further work.

Q3. Luna baselines. Mistral was chosen as a paraphraser from a family different from both Qwen and Llama. We have now added the omitted Luna baselines in Table 21.

MethodGSM8K AUROCGSM8K TPR@1%MBPP AUROCMBPP TPR@1%
Unigram0.566*0.0600.6660.068
SIR0.5370.0400.6350.024
STONE––0.5650.040
SemaSync0.5420.068*0.690*0.072*

* indicates no statistically significant advantage over the runner-up. STONE is designed exclusively for code tasks. Under Luna rewriting, none of the compared methods consistently achieves a statistically significant lead across both datasets. SemaSync nevertheless achieves the highest MBPP AUROC and TPR@1% among the listed methods, with statistically significant AUROC gains over SIR and STONE.

Q4. Deduplication. The order-sensitive key counted 87/89 extra watermarked positions on GSM8K and 54/31 on MBPP (Raw/Mistral). We rescored with a key that ignores anchor order and preserves multiplicity. Every absolute AUROC change was below 0.001 (MBPP Mistral: 0.694080 to 0.693208).

Q5. Why K=3? Table 5 suggests that multiple anchors can improve robustness to paraphrasing. We choose K=3K=3 as a moderate setting that enables majority voting while requiring fewer anchors than larger values of KK. This matters for code, where short outputs may contain too few anchors: in a simple anchor coverage check with K=5K=5, only 70 of 100 MBPP outputs reached five voters.

Q6. Transfer without re-tuning. To evaluate whether our existing ontologies transfer to new datasets without re-tuning, we applied the original GSM8K and MBPP anchor rules to 500 MATH test problems and all 164 HumanEval+ tasks, respectively. All runs use Qwen3.5-9B, a 4096-token budget, and one seed. Thresholds use 500 separate Clean outputs. In this evaluation, each rewriter is given one rewriting attempt, with prompts explicitly requiring preservation of mathematical answers, the meaning of the reasoning, and code behavior.

MATH

MethodRaw AUROCRaw TPRRaw UtilityMistral AUROCMistral TPRMistral Utility
Clean––0.888––0.924
KGW0.9330.5500.8360.7330.1540.896
SWEET0.9760.7800.8600.7700.2600.904
MorphMark0.6510.0720.892*0.5670.0260.922*
MorphMark+EWD0.8320.3200.892*0.6840.1000.922*
SemStamp0.8700.3920.8640.6940.1160.912
TextSeal0.8160.3020.8820.6400.1080.922*
Unigram0.9100.5000.8100.7110.1200.874
SIR0.7530.0540.8080.6190.0260.888
SemaSync0.9920.9340.8600.8140.322*0.920

HumanEval+

MethodRaw AUROCRaw TPRRaw UtilityMistral AUROCMistral TPRMistral Utility
Clean––0.634––0.665
KGW0.7670.2070.4450.5550.0300.512
SWEET0.8470.335*0.6950.625*0.0670.726
MorphMark0.5760.0370.6950.5530.0300.701
MorphMark+EWD0.7570.0910.6950.5630.0430.701
SemStamp0.6660.0240.6460.5510.0180.652
TextSeal0.6350.0550.6220.5490.0000.652
Unigram0.7180.1160.4450.5940.073*0.470
SIR0.6820.1340.5790.5470.0000.659
STONE0.6030.0180.6280.5370.0120.671
SemaSync0.854*0.3230.738*0.6200.0670.750*

Bold indicates the highest mean performance; * indicates no statistically significant advantage over the runner-up. With unchanged ontologies, SemaSync achieves the highest MATH Raw/Mistral AUROC (0.992/0.814), with positive paired 95% difference CIs against every external baseline; Mistral accuracy is 0.920 versus Clean 0.924. On HumanEval+, it has the highest Raw AUROC and utility in both reported conditions. After Mistral rewriting, SWEET has slightly higher AUROC (0.625 versus 0.620), while SemaSync has higher pass@1 (0.750 versus 0.726). These results support a useful detection--utility trade-off when transferring the original ontologies without re-tuning.

Thank you for the detailed response and the additional experiments. However, there are several issues that seem to require more efforts. After Mistral rewriting, the gate finds only 27.6% of the original GSM8K carriers. With the stronger Luna rewriter, the loss is likely larger (Q1). Code detection also still depends on the KGW-style initializer, as 172 of 500 MBPP outputs have no anchor position and HumanEval+ shows no detection advantage (Q2, Q6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.