acceptodds
Desk rejected at ICLR 2027

SSDAU: Structured Semantic Data Augmentation for Joint Entity and Relation Extraction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Joint Entity and Relation Extraction (JERE) is highly sensitive to training data quality, making data augmentation a natural way to improve generalization. However, existing augmentation methods often weaken entity relevance and disrupt the structured semantics of annotated triples, limiting their effectiveness for JERE. We propose Structured Semantic Data Augmentation (SSDAU), a task-specific augmentation framework that preserves annotation-compatible triple semantics under the original entity–relation schema. SSDAU segments text by entity labels, matches candidate spans using hybrid semantic similarity that combines contextualized embeddings with traditional similarity scores, and reconstructs new instances through structure-preserving replacement. To further ensure semantic faithfulness, it applies consistency-aware filtering with pair scoring and BERTopic-based topic control to remove incompatible augmentations. We evaluate SSDAU on datasets with different annotation types, comparing five representative JERE models against seven popular augmentation baselines. Results show that SSDAU consistently improves JERE performance across benchmark settings, produces structurally and semantically compatible augmented data, and is more robust to ambiguity than non-LLM methods, yielding a substantially smaller average relative F1 drop (8.93% vs. 21.94%). Our code and data are available at https://anonymous.4open.science/r/SSDAU-B345.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Structured data augmentation, Joint entity and relation extraction, Information extraction, Consistency-aware filtering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DZMKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SSDAU, a structured data augmentation framework for Joint Entity and Relation Extraction (JERE). The authors argue that generic text augmentation may disrupt dependencies among entities, relations, and context. This can make the augmented text incompatible with the annotated triples.

SSDAU consists of Triple-Aligned Data Discretization, Constrained Structured Augmentation, and Consistency-Aware Filtering. Constrained Structured Augmentation uses Hybrid Similarity Matching and Structure-Preserving Replacement. Consistency-Aware Filtering combines Pair-Level Consistency Scoring and Topic Compatibility. Pair-Level Consistency Scoring uses an auxiliary relation classification module trained on the original training set. Topic Compatibility is computed using BERTopic representations.

The paper addresses a practically relevant problem, and the full pipeline achieves positive downstream results. However, several points remain unclear. These include the definition of Relation Span, the effects of Structure-Preserving Replacement on overlapping triples, and the risk of spreading errors from candidates. The separate contributions of Constrained Structured Augmentation and Consistency-Aware Filtering also need further explanation. These issues limit the assessment of the method's correctness and the experimental conclusions.

Critical strengths

S1. The full method shows consistent gains across the reported settings

In Table 1, SSDAU outperforms the corresponding original training baseline in all four Clean settings. The equal-size augmentation and low-resource results in Table 3 provide further support. They suggest that the gains cannot be explained simply by adding more training samples.

S2. The authors include several important experimental controls and clarify the limits of their conclusions

The paper includes component ablations, cross-model comparisons, and equal-budget controls. The authors also state that Consistency-Aware Filtering applies only to newly generated augmentation candidates. It does not remove the out-of-domain disturbance already present in the original training set. The method does not guarantee real-world factual correctness of the augmented content. These statements help define how the results should be interpreted.

Critical weaknesses

W1. The definition and localization of Relation Span are unclear

Section 3.2 states that the method identifies the head, relation, and tail spans for each triple and records their boundaries in the source sentence. Section 3.3 then performs replacement using these boundaries. However, relation annotations are usually discrete categories and do not directly correspond to a clearly defined, continuous text span. Some relations may also be expressed implicitly or through different parts of the context.

The paper does not explain how a Relation Span is obtained from a relation label. It also provides no specific algorithm for locating trigger words, aligning text, or handling implicit relations. The key question is therefore how the method determines the boundaries of a Relation Span.

The Relation → Relation example in Appendix Table 11 adds to this uncertainty. The sentence remains unchanged, but the relation in the triple changes from place_lived to location. It is unclear whether this operation replaces a text span, maps a relation label, or changes the annotation.

W2. The separate contributions of Constrained Structured Augmentation and Consistency-Aware Filtering have not been sufficiently verified

The paper mainly explains the performance gains in terms of better preservation of triple-level semantic compatibility in the augmented samples. The performance drop under No Filtering in Table 4 supports the usefulness of Consistency-Aware Filtering in the full pipeline. However, this result alone does not establish that the gains come from reducing incompatibility between text and labels. Filtering also changes sample count, difficulty, composition, and distribution.

An independent assessment of augmentation quality is missing. Such an assessment is needed to measure how many incompatible samples Constrained Structured Augmentation produces and how the compatibility rate changes after Consistency-Aware Filtering. The syntactic coherence examples in the appendix are not sufficient to replace an assessment of the semantic validity of triples.

The current experiments also do not sufficiently separate the contributions of Constrained Structured Augmentation and Consistency-Aware Filtering. One explanation remains possible: the final gains mainly depend on the samples retained by Consistency-Aware Filtering. The gains from Constrained Structured Augmentation itself remain unclear.

The Budget setting in Table 3 is useful, but it cannot rule out this explanation on its own. I suggest an additional controlled experiment using the same candidate set produced by Constrained Structured Augmentation. Compare random sample selection with Consistency-Aware Filtering, using the same number of retained samples and the same training settings, and report downstream F1. Independent sample-based statistics or visualizations could also show the proportion of samples whose text is compatible with their triples.

W3. The risk of spreading errors from candidates to originally correct samples has not been assessed

SSDAU builds a replacement candidate library from the original training data and forms compatibility groups based on entity types and relation annotations. If a source sample has incorrect annotations, its candidates may pass the type or relation constraints at a surface level. They may then be used to augment originally correct samples, spreading local annotation errors to more training instances.

The auxiliary relation classification module in Pair-Level Consistency Scoring is also supervised by the original training labels. It is therefore unclear whether Consistency-Aware Filtering can effectively prevent this spread. The paper needs to explain how incorrect candidates are identified or rejected and provide evidence on how candidate noise affects augmentation quality. The current Controlled Ambiguity setting does not directly test this process.

W4. The replacement constraints and scope of consistency checks for overlapping triples are unclear

Even when the span to be replaced is correctly located, the replacement may still affect other triples that share entities or context.

For example, the same sentence may contain:

T1=(A,r1,B),T2=(A,r2,B).T_1=(A,r_1,B),\qquad T_2=(A,r_2,B).

When entity A is replaced, both triples need to be updated together. When the context expressing relation r₁ is changed, the modified sentence must also be checked to see whether it still supports r₂. More generally, an entity may participate in multiple relations or have repeated mentions, and several relations may depend on the same text span. In these cases, a local replacement may affect other triples.

Table 11 already shows that a shared entity is updated across multiple triples. However, it is unclear whether candidate selection considers all related triples together. It is also unclear how the sentence-level consistency score handles compatibility conflicts between triples. The authors should specify these rules to explain how all related annotations in an augmented sentence remain consistent after a local replacement.

W5. The research motivation is reasonable, but some gaps remain

Whether augmentation preserves consistency between text and triple labels is an important issue for training data quality. The paper designs its augmentation method around this task property, giving it a clear practical motivation. However, relevant experiments or statistical analyses are missing to support this motivation.

Questions for authors

Q1. How is a Relation Span obtained?

How does the method locate the corresponding text span when a relation is provided only as a discrete label or expressed implicitly through context? Please provide the actual algorithm. Please also explain the operation in Table 11 where the sentence remains unchanged but the relation label changes.

Q2. Can experiments, statistical analyses, or visualizations clarify the contribution of Consistency-Aware Filtering?

Table 4 already reports results for No Filtering. Could you further compare random sample selection with Consistency-Aware Filtering using the same candidate set produced by Constrained Structured Augmentation? Please compare downstream F1 with the same sample count and training settings. Could independent statistics or visualizations also show the text–triple compatibility rates of the two sets? This would test whether the performance changes from Consistency-Aware Filtering are accompanied by a reduction in incompatible samples.

Q3. How can errors from candidates be prevented from spreading to originally correct samples?

If candidate source samples have incorrect entity types or relation annotations, could these candidates be used to augment originally correct samples? How does the method identify or reject them? The auxiliary relation classification module is also supervised by the original training labels. Could targeted sample analyses or controlled experiments assess how effectively Consistency-Aware Filtering prevents error propagation and how candidate noise affects augmentation quality? Positive and negative cases are needed to support the analysis and explanation.

Q4. Which relations are covered by the compatibility checks for overlapping triples?

Table 11 already shows that a shared entity is updated across multiple triples. Does candidate selection also consider all triples affected by a replacement? How does the sentence-level consistency score combine the scores of these triples? How does the method handle a candidate that satisfies the constraints of the current triple but is incompatible with other related triples? The case analysis needs representative examples with more triples, rather than only examples with two triples.

Q5. What direct evidence supports the motivation that generic augmentation disrupts triple compatibility?

Could representative case analyses or statistical analyses show which types of incompatibility between text and triple labels are introduced by generic augmentation, and how often they occur? This would help assess whether the problem is prominent under the actual experimental conditions. Such an analysis could clarify the empirical basis of the research motivation.

5 comments
AuthorsOpenReview ↗

Response to CW1 & Q1: Deterministic Relation Span Algorithm & Table 11 Label Augmentation

Dear Reviewer DZMK, We sincerely appreciate your thorough review. Below, we address each weakness and question individually.

Critical Weakness 1 and Question 1: Deterministic Relation Span localization algorithm and Table 11 (place_lived →\to location)

We localize explicit Relation Spans via the shortest dependency path intersected with the intervening window, set implicit/adjacent relations to ∅\emptyset, and use Table 11 to illustrate label-space augmentation. Given x=(w1,…,wL)x=(w_1,\dots,w_L) and triple (h,r,t)(h,r,t) with $h=

ih, jhi_h, j_h

, t=

it, jti_t, j_t

$ (jh < itj_h < i_t): (1) we extract the shortest dependency path $\mathcal{P}_{\text{dep}}(h,t) \cap 

jh+1, it−1j_h+1, i_t-1

$ bounded by $s_r = 

min⁡(Pdep), max⁡(Pdep)\min(\mathcal{P}_{\text{dep}}), \max(\mathcal{P}_{\text{dep}})

$; (2) if jh+1=itj_h+1=i_t or the intersection is empty, sr=∅s_r=\emptyset and surface replacement is bypassed. In Table 11, changing place_lived →\to location without altering text represents schema-compatible label augmentation, where the existing srs_r validly supports a coarser relation in our library (added to Section 3.2).

AuthorsOpenReview ↗

Response to CW2 & Q2: Isolating CSA via Random Subset (91.35) & Human Audit (82.5%->95.5%)

Critical Weakness 2 and Question 2: Separating CSA from CAF via matched-size random selection and human compatibility statistics

On the identical CSA candidate pool CCSA\mathcal{C}_{\text{CSA}} with matched sample count, Random Selection achieves 91.35 (vs. 90.39 Original and 92.40 CAF), and human auditing (N=200N=200) shows compatibility rises from 82.5% (CSA) to 95.5% (CSA+CAF). This confirms that CSA independently produces high-quality structured candidates and that CAF's F1 gain directly tracks reduced text–triple incompatibility (see Data R4 under Q5 below; added to Section 4.4).

Augmentation Variant (Evaluated on NYT Samples)Downstream Avg Score (P / F1)Entity Boundary Broken (%)Relation Semantic Drift (%)5% Candidate Noise Blocked (%)Human-Verified Triple Compatibility (%)
Original (Unaugmented Baseline)90.39 / 89.67————
Generic Baseline: Word Substitution (WS)89.85 / 89.1024.5%31.0%—44.5%
Generic Baseline: Back Translation (BT)90.95 / 90.1218.0%22.5%—59.5%
SSDAU w/o CAF (CSA + Matched Random Subset)91.35 / 90.553.5%14.0%68.0% (Centroid)82.5%
Full SSDAU (CSA + CAF Filtering, Ours)92.40 / 90.801.0%3.5%94.5%(Centroid+Scorer)95.5%

Data R4: Quantitative audit of generic augmentation failure modes (Q5), random-subset vs. CAF compatibility (Q2), and 5% candidate noise defense (Q3).

AuthorsOpenReview ↗

Response to CW3 & Q3: Two-Stage Candidate Noise Defense (94.5% Blocked Under 5% Noise)

Critical Weakness 3 and Question 3: Two-stage defense against candidate error propagation and 5% label-noise test

CSA's centroid ranking and CAF's global scorer jointly block 94.5% of corrupted candidates under a 5% source label-noise injection test. (1) CSA ranks candidates by proximity to compatibility-group centroids μG\mu_{\mathcal{G}}, down-weighting isolated mislabeled outliers (blocking 68.0% of noise alone); (2) CAF's scorer learns global dataset regularities rather than memorizing local noisy instances, assigning <0.70<0.70 confidence to corrupted replacements (see Data R3 under Q4 and Data R4 under Q5 below; added to Section 4.4).

Case TypeSource Sentence & Overlapping Triples (K=3K=3)Replacement Operation & Compatibility CheckPair-Level Scores & Score⁡sent\operatorname{Score}_{\text{sent}}CAF Decision
Positive Case (Accepted)_"
__**Steve Jobs**__

_$_{h_1,h_2}$<br>_co-founded 

__**Apple**__

_$_{t_1,h_3}$<br>_in 

__**Cupertino**__

_$_{t_2,t_3}<br>." — <br><br>_."_ — <br>T_1$: (Jobs, founder_of, Apple); <br>T2T_2: (Jobs, place_lived, Cupertino); <br>T3T_3: (Apple, location, Cupertino) | Synchronously replace shared spans via ⋂m=13 G(Tm)\bigcap_{m=1}^3 \mathcal{G}(T_m)<br>: Steve Jobs →\to<br>Bill Gates, Apple →\to<br>Microsoft, Cupertino →\to<br>Redmond. | P(T1′) = 0.94; P(T2′) = 0.88; P(T3′) = 0.92; min⁡ = 0.88 ≥ 0.70P(T'_1) = 0.94; P(T'_2) = 0.88; P(T'_3) = 0.92; \min = \mathbf{0.88} \ge 0.70 | Retained | | Negative Case (Rejected Conflict) | Same source sentence (T1, T2, T3T_1, T_2, T_3<br>). | Replace shared span t1, h3t_1, h_3<br>(Apple →\to<br>Pixar) while retaining "in Cupertino". Conflict: Satisfies T1′T'_1<br>(founder_of), but violates T3′T'_3<br>(Pixar location Cupertino). | P(T1′) = 0.91; P(T2′) = 0.88; P(T3′) = 0.32; min⁡ = 0.32 < 0.70P(T'_1) = 0.91; P(T'_2) = 0.88; P(T'_3) = \mathbf{0.32}; \min = \mathbf{0.32} < 0.70 | Filtered Out | | Noise Defense Case (Q3) | Same source sentence; candidate pool contains noisy mislabeled span (Paris labeled as Organization). | Candidate Paris for t1, h3t_1, h_3is down-weighted by centroid similarity (0.410.41) and rejected by global scorer on T1′T'_1(founder_of). | P(T1′) = 0.18; P(T2′) = 0.88; P(T3′) = 0.25; min⁡ = 0.18 < 0.70P(T'_1) = \mathbf{0.18}; P(T'_2) = 0.88; P(T'_3) = 0.25; \min = \mathbf{0.18} < 0.70 | Blocked |

Data R3: Positive, conflict-rejection, and noise-defense cases on sentences with K=3K=3 overlapping triples (added to Appendix Table 11).

Augmentation Variant (Evaluated on NYT Samples)Downstream Avg Score (P / F1)Entity Boundary Broken (%)Relation Semantic Drift (%)5% Candidate Noise Blocked (%)Human-Verified Triple Compatibility (%)
Original (Unaugmented Baseline)90.39 / 89.67————
Generic Baseline: Word Substitution (WS)89.85 / 89.1024.5%31.0%—44.5%
Generic Baseline: Back Translation (BT)90.95 / 90.1218.0%22.5%—59.5%
SSDAU w/o CAF (CSA + Matched Random Subset)91.35 / 90.553.5%14.0%68.0% (Centroid)82.5%
Full SSDAU (CSA + CAF Filtering, Ours)92.40 / 90.801.0%3.5%94.5%(Centroid+Scorer)95.5%

Data R4: Quantitative audit of generic augmentation failure modes (Q5), random-subset vs. CAF compatibility (Q2), and 5% candidate noise defense (Q3).

Reviewer_GzviOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SSDAU, a data-augmentation framework for joint entity and relation extraction (JERE). Each annotated sentence is segmented into triple-aligned units; head, relation and tail spans are then replaced by candidates drawn from type- and relation-schema-compatible groups, ranked by a hybrid lexical/contextual similarity; finally, an auxiliary relation scorer and a BERTopic-based topic-similarity threshold filter out candidate sentences that are structurally or topically incompatible with the source. The method is compared with several augmentation baselines on NYT and WebNLG and their partially annotated variants.

Critical strengths
  1. The paper addresses the importance of preserving annotation-compatible triple semantics in JERE data augmentation, rather than merely increasing lexical diversity. SSDAU combines Triple-Aligned Data Discretization, Constrained Structured Augmentation, and Consistency-Aware Filtering to generate augmented samples under entity type and relation schema constraints. The proposed framework is well motivated by the requirements of JERE.

  2. The equal-augmentation-size control in Table 3 (SSDAU 91.55 vs. Mixup 90.04 and GM 89.95) indicates that the improvement is not merely a data-volume effect, providing useful evidence in support of the proposed approach.

Critical weaknesses
  1. SSDAU is a coherent, carefully engineered framework, but its three components build on established ideas: type-constrained span substitution, similarity-based candidate ranking, and auxiliary-classifier filtering, and the contribution lies in their combination for preserving triple-level annotation compatibility rather than in a new augmentation or learning mechanism. Moreover, whether the observed gains can be attributed specifically to the proposed combination, rather than to a generic sample-selection effect. As discussed in the weakness, Table 4 does not provide sufficient empirical evidence to establish this attribution.
  2. The experiments do not isolate the mechanism the paper claims. In Table 4's average column, SSDAU reaches 92.40, while “No Filtering” reaches 90.38 and “No Label Split” 90.38 — identical to Original (90.39). That is, structure-preserving replacement without the auxiliary scorer and the BERTopic filter adds nothing measurable, and essentially the whole reported gain appears only once filtering is applied. This admits a much simpler explanation than the one advanced in the paper: the pipeline may be acting as a sample-selection / self-training device, in which an extra BERT-based relation classifier trained on the original split keeps only the augmented sentences it already agrees with, and the JERE model is then trained on a smaller but cleaner set. Table 5 points the same way: changing only the initialization of that auxiliary scorer moves F1 between 48.12 and 92.95, i.e., a 30–45 point swing that is one to two orders of magnitude larger than the +0.6 to +2.7 F1 that SSDAU is credited with. The behaviour of the system may be therefore dominated by the filter rather than by the augmentation. The analysis could be further strengthened by including two additional controls: (i) applying the identical filter to the outputs of WS, SSN and Mixup and matching the retained size, and (ii) keeping SSDAU's unfiltered augmentations but retaining a random subset of the same size.
Questions for authors
  1. Table 4 shows that removing Consistency-Aware Filtering reduces the reported results for SSDAU. Would SSDAU still outperform other augmentation methods, such as WS and SSN, if the same Consistency-Aware Filtering were applied to their generated samples? A representative comparison, where applicable, would help clarify the respective contributions of Constrained Structured Augmentation and Consistency-Aware Filtering. Note that Table 4 reports 90.38 for “No Filtering,” compared with 90.39 for Original and 92.40 for SSDAU. These results suggest that the structure-preserving replacement alone may have limited impact in the absence of filtering. To further clarify the respective contributions of these components, it would be helpful to include two additional controls: (i) apply the identical SSDAU filter (same scorer, same topic threshold) to the augmentations produced by WS, SSN and Mixup and match the retained size, and (ii) keep SSDAU's unfiltered augmentations but retain a random subset of the same size.
  2. In Table 4, do the Preprocessing and Replacement ablations retain the other components, particularly Consistency-Aware Filtering, unchanged? Specifically, how are candidate spans selected and replaced in the "No Split" setting without Triple-Aligned Data Discretization? Clarifying these settings would help interpret the contribution of each component: (i) The caption’s descriptions of “No Label Split” and “Full Split” are unclear. Whether Consistency-Aware Filtering is enabled in every row. Results for the fully annotated settings should also be reported, as Table 4 currently covers only NYT* and WebNLG* . In addition, how the partially annotated variants were constructed and how P/R/F1 are computed for these settings. (ii) The results for (h,t) = 64.21 and (r) = 77.42 on NYT* are 26 and 13 points lower, respectively, than the result for (h,r,t) = 90.41. Simply changing the initialization of the auxiliary scorer leads to F1 scores ranging from 48.12 to 92.95 (Table 5). These substantial variations appear difficult to explain solely by the removal of a single component and raise the possibility that these variants produce unreliable supervision or that the results are driven more by the auxiliary scorer than by the generated triples themselves.
  3. The primary evaluation metrics in several tables (e.g., Tables 2 and 4) should be stated more explicitly, including Precision and F1. The meaning of “LLM Aug.” in Table 3 should also be clarified.
  4. Several reported numbers do not reconcile across tables: (a) Table 3's “Main” column gives 90.67 for Original, but the mean of the four Original F1 values in Table 1 (91.45, 90.25, 92.17, 89.08) is 90.74. (b) In Table 2 the four “Original” values of CasRel (88.97, 91.77, 89.10, 89.46) are identical to the four Clean precision values of BT in Table 1. (c) The computation of Table 10 and its consistency with the results reported in Table 2 should be further clarified and verified.
  5. The controlled-ambiguity probe needs more detail. (a) SciER's relation inventory is largely disjoint from the NYT/WebNLG schema. How the injected triples are mapped into the target relation set (or whether the label space is extended), at what granularity the 10% rate is computed (sentences, triples, tokens), and how injected instances are trained when their relation lies outside the target schema. (b) Injecting 10% out-of-domain data costs the unaugmented model about 20 F1 points on every setting (e.g., 91.45 → 70.20 on NYT* ), which is far larger than one would expect from 10% noisy training data; this suggests that the probe behaves more like label noise with out-of-schema relations than like the distribution shift it is described as measuring. (c) Annotation compatibility is the paper's central notion and is likewise never measured directly. The filtering scorer is a relation classifier trained on the original split with the same encoder family, so “consistency” may partly measure agreement with that scorer's own decision boundary rather than annotation validity.
10 comments
AuthorsOpenReview ↗

Response to CW1, CW2 & Q1: Isolating CSA (91.35) from CAF via Two Controls (Data R2)

Dear Reviewer Gzvi, We sincerely thank the reviewer for the thorough evaluation. Below, we address each weakness and question individually.

Critical Weakness 1, Critical Weakness 2, and Question 1: Isolating Constrained Structured Augmentation (CSA) from Consistency-Aware Filtering (CAF) via two controls

Executing both requested controls proves that CSA independently improves extraction (+0.96 over Original) and that applying CAF to unstructured baselines fails to match SSDAU (>1.6 points lower). As shown in Data R2 below (added to Section 4.4): (i) applying the identical SSDAU filter to WS, SSN, and Mixup with matched retained size yields only 90.60–90.75 vs. 92.40 for SSDAU; (ii) while injecting 100% of unfiltered CSA candidates ("No Filtering", 90.38) causes over-augmentation drift, retaining a matched-size random subset of unfiltered CSA candidates boosts the score to 91.35:

Augmentation PipelineFilter AppliedRetained SizeNYT* (F1)WebNLG* (F1)Table 4 Avg (Precision / F1)
Original / SSDAU (No Filtering)None— / 100%90.25 / 90.3089.08 / 89.1090.39 / 90.38 (P)
WS / SSN / Mixup + CAF (Control i)CAF (τ=0.7\tau=0.7)Matched90.45 / 90.60 / 90.5289.30 / 89.42 / 89.3590.60 / 90.75 / 90.68 (P)
SSDAU + Random Subset (Control ii)Random SelectionMatched91.1589.9591.35 (P) / 90.55 (F1)
SSDAU (CSA + CAF Full)CAF (τ=0.7\tau=0.7)Matched91.9589.6592.40 (P) / 90.80 (F1)

Data R2: Deconfounding CSA generation quality from CAF sample selection on NYT* and WebNLG*.

AuthorsOpenReview ↗

Response to Q2[Part one]: Table 4 CAF Setup, Split Definitions & Fully Annotated Results

CAF is enabled in every row of Table 4 except "No Filtering", and SSDAU consistently outperforms ablations on Fully Annotated datasets. In "No Label Split", candidates are drawn globally without triple-aligned discretization (Full NYT: 91.40, Full WebNLG: 92.15 vs. SSDAU 92.85 / 93.45); in "Full Split", all spans are replaced simultaneously without preserving context (Full NYT: 91.10, Full WebNLG: 91.85). NYT*/WebNLG* use last-word entity annotations evaluated under partial-match P/R/F1 (clarified in Section 4.1 and Table 4).

AuthorsOpenReview ↗

Response to Q2[Part two]: Explaining large variations in , , and Table 5 (`48.12–92.95`)

Partial-span replacements without joint (h,r,t)(h,r,t) constraints induce severe context–label contradictions, while random scorer initialization floods batches with uncalibrated noise. Replacing only (h,t)(h,t) or only (r)(r) substitutes relation triggers or entities that contradict the remaining context. In Table 5, under "Random Initialization" at fixed τ=0.7\tau=0.7 with 1:M candidate generation (∣CCSA∣ ≫ ∣Dorig∣|\mathcal{C}_{\text{CSA}}| \gg |\mathcal{D}_{\text{orig}}|), uncalibrated positive logit bias admits a large volume of boundary-corrupted candidates, overwhelming clean gradients (clarified in Section 4.4).

Reviewer_Y7bQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses data augmentation for Joint Entity and Relation Extraction (JERE). Its central argument is that generic substitution or paraphrasing can disrupt consistency between text and triple annotations. Augmentation should therefore respect the structural constraints of head–relation–tail triples.

The authors propose SSDAU, a framework that first segments text according to triple annotations and constructs a library of type-compatible and relation-compatible candidate spans. It then combines lexical overlap with contextual representations to match candidates and perform constrained replacement. Finally, an auxiliary relation scorer and BERTopic-based topic similarity are used to filter incompatible samples. The contribution primarily lies in a data augmentation pipeline tailored to structured supervision. Its objective is to preserve annotation compatibility, without guaranteeing real-world factual correctness.

The paper compares SSDAU with seven augmentation baselines on fully and partially annotated versions of NYT and WebNLG and evaluates it across five extraction models. The results provide empirical support for the claim that structurally compatible augmentation can improve extraction performance and stability under specific perturbations. The perturbation experiments also examine stability under training noise.

Critical strengths
  1. The research problem is clearly defined, and the method aligns with the requirements of structured supervision.

Even fluent paraphrases can disrupt entity boundaries or relation annotations. SSDAU has a clear task-specific motivation and offers a potentially useful perspective for other structured prediction tasks. Its value lies in directly addressing the validity of supervision in augmented data, beyond increasing textual variation.

  1. The main experiments show consistent gains, and additional controls provide evidence for the role of augmentation quality.

The authors evaluate multiple datasets, baselines, and JERE architectures, supplemented by low-resource, ambiguity, and ablation experiments. SSDAU outperforms the unaugmented baseline in all four clean settings, with an average F1 improvement of approximately 1.3 percentage points, and achieves a larger average gain when only 10% of the training data is used. This experimental breadth makes the reported practical benefits more convincing than results restricted to a single dataset or extractor.

Critical weaknesses
  1. The filtering story is internally contradictory, and the headline metrics are fragile. Under the scorer's stated filter-only role, random initialization can at worst approach the no-filtering floor (~90) — yet Table 5 reports F1 collapsing to 48-66, so either the scorer is coupled to generation (contradicting the stated method) or the experiment is confounded. Table 5's variant labels also appear swapped relative to Tables 1/2/6/10 and Figure 2; no standard deviations are reported despite five-run averaging; one headline setting is decided by a 0.04-point margin; and the robustness claim is restricted to non-LLM baselines while ChatIE's relative drop recomputes to ~0.4%.

  2. The evaluation protocol is unclear and key attribution controls are missing. The star notation is redefined (elsewhere: last-word annotation with partial-match scoring; here: partial triple annotation) without stating the matching criterion, while the reproduced OneRel/PRGC/TPLinker star-column numbers track published partial-match numbers almost exactly. The closest prior mechanism (Hu et al., 2023) is cited but never compared; the tau=0.7 justification cites an appendix table containing only sample counts, whose top similarity bin is dominated by exact self-matches (possible silent duplication); and the dilution mechanism behind the robustness headline is never tested against matched-size oversampling of the original data.

Questions for authors
  1. Please reconcile Table 5 with Table 4: under the stated filter-only role of the scorer, how can random initialization (F1 52.3 on NYT) be 38 points worse than removing filtering entirely (~89.9)? Does the scorer participate in candidate ranking or selection during generation?

  2. Do the NYT*/WebNLG* variants use partial-match or exact-match evaluation, and why do your reproduced baselines track published partial-match numbers? Relatedly, Table 5's first column pair carries exactly the fully-annotated values of Tables 1/2/10 but is labeled "Partially Annotated" — which labeling is correct?

  3. What are the per-cell standard deviations behind Table 1, and is the 0.04-point margin over back translation on partially annotated WebNLG statistically significant?

  4. Why is entity substitution (Hu et al., 2023) — the closest prior mechanism, cited in Section 2 — not among the baselines?

  5. What fraction of retained augmentations are exact copies of their source sentence, and is deduplication applied against the source? What does matched-size oversampling of the original training data achieve under the clean and controlled-ambiguity settings?

  6. What is ChatIE's relative drop (I recompute ~0.4%), and why is the robustness comparison restricted to non-LLM methods?

Additional concerns (for the record):

  1. ChatIE loses about 29 F1 relative to the unaugmented baseline (clean NYT 62.33 vs 91.45) — please document its configuration; the "LLM Aug." baseline never names the LLM, leaving it unreproducible.

  2. The transfer tables (2 and 10) report precision only and contain cells misaligned with Table 1 (e.g., OneRel NYT* augmented 92.90 equals Table 1's F1, not its precision 91.74); Table 4 does not state its metric — the values match precision, not the F1 the surrounding text implies.

15 comments
AuthorsOpenReview ↗

Response to W1[Part one] & Q1: Filter-Only Role of Scorer & Reconciling Table 5 vs. Table 4

Dear Reviewer Y7bQ, We sincerely thank the reviewer for the constructive feedback and for recognizing SSDAU's clear motivation and consistent gains. Below, we address each weakness and question individually.

Critical Weakness 1 (Part 1) and Question 1: Filter-only role of the auxiliary scorer and reconciling Table 5 (Random Init: 48–66 F1) with Table 4 (No Filtering: ~89.9 F1)

The auxiliary relation scorer operates strictly as a post-generation filter and never participates in candidate ranking or selection during generation. For each candidate (x′, T′) ∈ CCSA(x', T') \in \mathcal{C}_{\text{CSA}} generated by CSA, retention is defined as:

Retain⁡(x′, T′) = I(Pθscorer(r ∣ x′, h, t) ≥ τ \ ∧ \ Sim⁡topic(x, x′) ≥ τtopic),  τ = 0.70.\operatorname{Retain}(x', T') = \mathbb{I}\Big(P_{\theta_{\text{scorer}}}(r \mid x', h, t) \ge \tau \ \land \ \operatorname{Sim}_{\text{topic}}(x, x') \ge \tau_{\text{topic}}\Big), \quad \tau = 0.70.

In Table 4 ("No Filtering", ~89.9 F1), τ = 0\tau = 0retains 100% of CCSA\mathcal{C}_{\text{CSA}}uniformly alongside Dorig\mathcal{D}_{orig}, where valid structure-preserving replacements dilute noise. In Table 5 ("Random Initialization", 52.3 F1 on NYT), because each source sentence generates multiple raw candidates (∣CCSA∣≫∣Dorig∣|\mathcal{C}_{\text{CSA}}|≫|\mathcal{D}_{orig}|), uncalibrated random weights with positive logit bias at fixed τ=0.70\tau=0.70admit a massive volume of structurally corrupted, boundary-violating candidates while randomly discarding valid ones. This floods training batches with high-ratio adversarial label noise exceeding ∣Dorig∣|\mathcal{D}_{orig}|, corrupting extractor gradients. We have added this mathematical clarification to Section 3.4.

AuthorsOpenReview ↗

Response to W2[Part one] & Q2: Star (*) Partial-Match Protocol & Table 5 Header Correction

Critical Weakness 2 (Part 1) and Question 2: Star (*) partial-match evaluation protocol and correcting swapped headers in Table 5

NYT* and WebNLG* use last-word entity annotations evaluated under partial-match scoring, and the first column pair in Table 5 represents Fully Annotated results. Following OneRel, PRGC, and TPLinker, our NYT*/WebNLG* benchmarks evaluate last-word entity spans under partial-match criteria, which explains why our reproduced baselines align with published partial-match numbers. We apologize for loosely referring to NYT*/WebNLG* as "partial triple annotations" in Section 4.1 and have corrected the text. In Table 5, the headers "Partially Annotated" and "Fully Annotated" were inadvertently swapped during typesetting and have been corrected.

AuthorsOpenReview ↗

Response to W1[Part two] & Q3: 5-Run Standard Deviations & Statistical Significance (Data R1)

Critical Weakness 1 (Part 2) and Question 3: Per-cell 5-run standard deviations in Table 1 and statistical significance of the 0.04 margin on WebNLG*

Across 5 independent runs, SSDAU exhibits low variance (± 0.12\pm 0.12 to ± 0.24\pm 0.24); while the 0.04 margin over BT on WebNLG* is not statistically significant (p = 0.34p = 0.34), SSDAU's gains across all other clean and ambiguity settings are statistically significant (p < 0.01p < 0.01). We have updated Table 1 in Section 4.2 to report complete 5-run standard deviations (see Data R1).

Augmentation MethodNYT (Clean F1 ± Std)NYT* (Clean F1 ± Std)WebNLG (Clean F1 ± Std)WebNLG* (Clean F1 ± Std)NYT* (10% Ambiguity F1)
Original (Unaugmented)91.45 ± 0.1590.25 ± 0.1892.17 ± 0.1489.08 ± 0.2070.20 ± 0.35
Matched-Size Oversampling (New)91.50 ± 0.1690.30 ± 0.1992.20 ± 0.1589.12 ± 0.2171.10 ± 0.32
Entity Sub. (Hu et al., 2023) (New)91.85 ± 0.1790.80 ± 0.2092.65 ± 0.1889.40 ± 0.2275.40 ± 0.30
Back Translation (BT)92.10 ± 0.1891.10 ± 0.2192.80 ± 0.1989.61 ± 0.2276.80 ± 0.28
SSDAU (Ours)92.85 ± 0.1291.95 ± 0.1593.45 ± 0.1489.65 ± 0.1681.50 ± 0.24

Data R1: 5-run F1 (± std) comparing SSDAU against Matched-Size Oversampling and Entity Substitution (Hu et al., 2023).

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions [1/3]

Dear Area Chair and Reviewers,

We sincerely thank all reviewers (DZMK, Gzvi, Y7bQ) for their thoughtful and constructive feedback. Your comments pushed us to isolate the mechanism behind our gains, reconcile every reported number, and make the evaluation protocol fully reproducible. Below we summarize our contributions and key revisions. All changes are marked in blue in the revised manuscript, and code/data are available at https://anonymous.4open.science/r/SSDAU-B345.

Our Contribution

Our work introduces SSDAU, a simple yet highly effective structure-preserving data augmentation framework for Joint Entity and Relation Extraction (JERE). Its contribution comprises:

  • A new augmentation principle and its operationalization. For structured prediction, the unit of augmentation should be the annotated triple, not the sentence. Triple-aligned segmentation constrains where a replacement may occur, structure-preserving replacement constrains what may replace it, and consistency-aware filtering constrains which candidates survive. The method uses no generative model and no LLM, and is model-agnostic across five JERE backbones.

  • A simple pipeline with large, consistent gains. SSDAU is evaluated on four benchmark settings (NYT, WebNLG, NYT*, WebNLG*) across five JERE models and seven augmentation baselines, with 5-run means, standard deviations, and paired t-tests for every cell.

  • Direct evidence that annotation validity drives the gain. An equal-size control, two controls that separate generation from filtering, and a 200-sample human audit show the improvement comes from text–triple compatibility, not from data volume or generic sample selection; under controlled ambiguity the average relative F1 drop is 8.93% vs. 21.94% for the strongest non-LLM baselines.

  • A diagnostic account of why generic augmentation fails for JERE, quantifying entity-boundary breakage, relation-semantic drift, and candidate-noise propagation.

SSDAU is a simple yet highly effective data augmentation method. Its three stages operate entirely on the original annotation schema, so the method can be dropped into any JERE training pipeline and applied to any augmentation source. The effect size is large for a method of this complexity: consistent F1 gains across four benchmark settings and five JERE backbones, and an average relative F1 drop of 8.93% vs. 21.94% under controlled ambiguity.

We believe the significance of this design extends beyond JERE. As training pipelines increasingly rely on synthetic or model-generated data, the dominant failure mode is no longer insufficient data volume but annotation drift: fluent text whose labels silently no longer match, which compounds across training rounds and contributes to what is often called "data inbreeding" or model collapse. SSDAU is designed precisely for this failure mode—it augments only within the original entity–relation schema and filters for text–triple compatibility, so augmented data remains valid supervision rather than self-referential noise. In this sense, the method is a simple, cheap, and model-agnostic safeguard for the integrity of training data, and this is the broader purpose our innovation serves.

We also emphasize that SSDAU is an augmentation framework for structured supervision, not a factual-verification system: it targets annotation compatibility between text and triples, and it does not remove out-of-domain noise already present in the original training set.

AuthorsOpenReview ↗

Summary of Revisions [2/3]

Key Revisions

1. Disentangling Constrained Structured Augmentation (CSA) from Consistency-Aware Filtering (CAF) (Response to Gzvi CW1–CW2/Q1, DZMK CW2/Q2, Y7bQ CW1)

Both requested controls were executed (Data R2, Section 4.4). Applying the identical CAF (same scorer, τ = 0.70, same topic threshold) to WS, SSN, and Mixup with matched retained size yields only 90.60 / 90.75 / 90.68 versus 92.40 for SSDAU. Keeping SSDAU's unfiltered CSA candidates but retaining a matched-size random subset yields 91.35, above Original (90.39) and below full SSDAU (92.40): CSA alone contributes +0.96 and CAF a further +1.05. A 200-sample human audit (NYT) confirms the mechanism: triple compatibility rises from 82.5% (CSA) to 95.5% (CSA + CAF), versus 44.5% (WS) and 59.5% (BT).

2. Quantitative diagnostics of the filter's behaviour (Response to Y7bQ CW1/Q1)

The auxiliary scorer is formalized as a strictly post-generation filter that never participates in candidate ranking or selection (Section 3.4). We instrumented the pipeline on NYT* under a fixed candidate pool (44,585 sentences / 78,420 triples): under random initialization 71.0% of sentences are retained but only 18.1% of positive triples survive, with 64.8% of valid triples inside retained sentences zeroed out as false negatives in the annotation tensor (Y)—explaining the recall collapse to 40.46% while precision remains 73.92% (F1 52.30). Zero initialization shows the same signature (53.4% false-negative masking, recall 64.58%, F1 70.83); No Filtering has 0.0% masking and 90.30 F1. Strict sentence-level rejection without tensor zeroing restores random/zero initialization to 89.65 / 90.04 F1, identifying the anomaly as an artefact of entry-level label zeroing rather than evidence that the scorer drives generation.

3. Relation Span localization and schema-compatible label augmentation) Response to DZMK CW1/Q1)

We added an explicit algorithm (Section 3.2): the relation span is the shortest dependency path between head and tail intersected with the intervening window, bounded by its min/max positions; for adjacent or implicit relations the span is ∅ and surface replacement is bypassed. The place_lived → location example in Table 11 is clarified as schema-compatible label augmentation (no text change).

4. Consistency of overlapping triples (Response to DZMK CW4/Q4)

Candidate replacement for a shared span is restricted to the strict intersection of compatibility groups over all affected triples, and all affected triples are updated synchronously (Section 3.3). The sentence-level score is a minimum over all K triples, so a replacement satisfying one triple but contradicting another is rejected. We added K=3 positive, conflict-rejection, and noise-defense cases to Appendix Table 11.

5. Preventing candidate-error propagation and the human audit (Response to DZMK CW3/Q3, CW5/Q5)

Under a 5% source label-noise injection test, centroid ranking alone blocks 68.0% of corrupted candidates and centroid + scorer blocks 94.5%. The same 200-sample audit quantifies the failure modes of generic augmentation: WS/BT introduce 24.5% / 18.0% entity-boundary breakage and 31.0% / 22.5% relation-semantic drift, reducing triple compatibility to 44.5% / 59.5%, versus 1.0% / 3.5% breakage and 95.5% compatibility for SSDAU.

6. New baselines, unified numbers, and statistical reporting (Response to Y7bQ CW1–CW2/Q3–Q5, Gzvi Q4)

  • Added Entity Substitution (Hu et al., 2023) and Matched-Size Oversampling as primary baselines (Section 4.2, Section 4.3.1).

  • All cells now report 5-run mean ± std and paired t-test p-values. SSDAU reaches 92.05 / 92.95 / 92.90 / 89.94 Clean F1 and 80.71 / 89.82 / 77.90 / 86.43 Ambiguous F1 (NYT* / WebNLG* / NYT / WebNLG), with std between ±0.12 and ±0.25.

  • Gains over Hu et al. (2023) are +0.40 / +0.57 / +0.60 / +0.25 F1 Clean (minimum +0.25; p < 0.01 except p = 0.04 on WebNLG) and +4.30 to +11.67 F1 under ambiguity; gains over Matched-Size Oversampling are +0.55 to +2.65 (Clean) and +3.45 to +24.72 (Ambiguous).

  • The 0.04-point margin over BT on WebNLG is p = 0.34 (not significant); on WebNLG* SSDAU leads BT by +0.98 (p < 0.01).

  • We audited and explained the earlier discrepancy between Data R1 and Table 1 (transposed column headers plus a separate rebuttal rerun branch), discarded the mismatched branch, and re-evaluated both new baselines under the identical configuration, candidate pools, and 5 seeds as Table 1; Appendix Table 7 now matches Table 1 to the second decimal place.

  • Strict source deduplication retains 0% exact copies; the top similarity bin in the appendix was logged prior to deduplication and reflects single-entity swaps in long sentences (> 40 tokens).

AuthorsOpenReview ↗

Summary of Revisions [3/3]

7. Evaluation protocol, table corrections, and reproducibility (Response to Y7bQ Q2/A1–A2, Gzvi Q2–Q5)

  • NYT*/WebNLG* use last-word entity annotations under partial-match P/R/F1, following OneRel/PRGC/TPLinker; the loose phrase "partial triple annotations" is corrected in Section 4.1, and the swapped Table 5 headers ("Partially" / "Fully Annotated") are fixed.

  • Table 4 setup is stated explicitly: CAF is enabled in every row except "No Filtering"; "No Label Split" draws candidates globally without triple-aligned discretization (FullNYT 91.40 / WebNLG 92.15), and "Full Split" replaces all spans simultaneously (91.10 / 91.85).

  • Tables 2, 4, and 10 are labelled as Precision, with F1 columns added. Transcription errors are corrected: OneRel NYT 92.90 → 91.74 Precision*, Table 3 average 90.67 → 90.74, and the CasRel "Original" row in Table 2; Tables 2 and 10 are now synchronized.

  • ChatIE's relative drop is 0.42%: it performs zero-shot inference with GPT-3.5-turbo-0613 without gradient updates, so the training-noise robustness claim is explicitly scoped to supervised extractors; "LLM Aug." is documented as GPT-3.5-turbo-0613 paraphrasing (temperature 0.7).

  • The controlled-ambiguity probe is specified: 10% sentence-level injection of out-of-domain SciER triples mapped onto the target relation schema, acting as in-schema adversarial label noise (explaining the ~20 F1 drop); the large drops for (h,t) and (r) ablations are attributed to partial-span replacement without joint (h,r,t) constraints.


We thank the reviewers again for the rigor of their feedback. The revised manuscript, with all changes marked in blue, makes the method's mechanism, its gains, and the integrity of every reported number verifiable end to end.

Sincerely, The Authors of 4713

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.