- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SSDAU, a structured data augmentation framework for Joint Entity and Relation Extraction (JERE). The authors argue that generic text augmentation may disrupt dependencies among entities, relations, and context. This can make the augmented text incompatible with the annotated triples.
SSDAU consists of Triple-Aligned Data Discretization, Constrained Structured Augmentation, and Consistency-Aware Filtering. Constrained Structured Augmentation uses Hybrid Similarity Matching and Structure-Preserving Replacement. Consistency-Aware Filtering combines Pair-Level Consistency Scoring and Topic Compatibility. Pair-Level Consistency Scoring uses an auxiliary relation classification module trained on the original training set. Topic Compatibility is computed using BERTopic representations.
The paper addresses a practically relevant problem, and the full pipeline achieves positive downstream results. However, several points remain unclear. These include the definition of Relation Span, the effects of Structure-Preserving Replacement on overlapping triples, and the risk of spreading errors from candidates. The separate contributions of Constrained Structured Augmentation and Consistency-Aware Filtering also need further explanation. These issues limit the assessment of the method's correctness and the experimental conclusions.
Critical strengths
S1. The full method shows consistent gains across the reported settings
In Table 1, SSDAU outperforms the corresponding original training baseline in all four Clean settings. The equal-size augmentation and low-resource results in Table 3 provide further support. They suggest that the gains cannot be explained simply by adding more training samples.
S2. The authors include several important experimental controls and clarify the limits of their conclusions
The paper includes component ablations, cross-model comparisons, and equal-budget controls. The authors also state that Consistency-Aware Filtering applies only to newly generated augmentation candidates. It does not remove the out-of-domain disturbance already present in the original training set. The method does not guarantee real-world factual correctness of the augmented content. These statements help define how the results should be interpreted.
Critical weaknesses
W1. The definition and localization of Relation Span are unclear
Section 3.2 states that the method identifies the head, relation, and tail spans for each triple and records their boundaries in the source sentence. Section 3.3 then performs replacement using these boundaries. However, relation annotations are usually discrete categories and do not directly correspond to a clearly defined, continuous text span. Some relations may also be expressed implicitly or through different parts of the context.
The paper does not explain how a Relation Span is obtained from a relation label. It also provides no specific algorithm for locating trigger words, aligning text, or handling implicit relations. The key question is therefore how the method determines the boundaries of a Relation Span.
The Relation → Relation example in Appendix Table 11 adds to this uncertainty. The sentence remains unchanged, but the relation in the triple changes from place_lived to location. It is unclear whether this operation replaces a text span, maps a relation label, or changes the annotation.
W2. The separate contributions of Constrained Structured Augmentation and Consistency-Aware Filtering have not been sufficiently verified
The paper mainly explains the performance gains in terms of better preservation of triple-level semantic compatibility in the augmented samples. The performance drop under No Filtering in Table 4 supports the usefulness of Consistency-Aware Filtering in the full pipeline. However, this result alone does not establish that the gains come from reducing incompatibility between text and labels. Filtering also changes sample count, difficulty, composition, and distribution.
An independent assessment of augmentation quality is missing. Such an assessment is needed to measure how many incompatible samples Constrained Structured Augmentation produces and how the compatibility rate changes after Consistency-Aware Filtering. The syntactic coherence examples in the appendix are not sufficient to replace an assessment of the semantic validity of triples.
The current experiments also do not sufficiently separate the contributions of Constrained Structured Augmentation and Consistency-Aware Filtering. One explanation remains possible: the final gains mainly depend on the samples retained by Consistency-Aware Filtering. The gains from Constrained Structured Augmentation itself remain unclear.
The Budget setting in Table 3 is useful, but it cannot rule out this explanation on its own. I suggest an additional controlled experiment using the same candidate set produced by Constrained Structured Augmentation. Compare random sample selection with Consistency-Aware Filtering, using the same number of retained samples and the same training settings, and report downstream F1. Independent sample-based statistics or visualizations could also show the proportion of samples whose text is compatible with their triples.
W3. The risk of spreading errors from candidates to originally correct samples has not been assessed
SSDAU builds a replacement candidate library from the original training data and forms compatibility groups based on entity types and relation annotations. If a source sample has incorrect annotations, its candidates may pass the type or relation constraints at a surface level. They may then be used to augment originally correct samples, spreading local annotation errors to more training instances.
The auxiliary relation classification module in Pair-Level Consistency Scoring is also supervised by the original training labels. It is therefore unclear whether Consistency-Aware Filtering can effectively prevent this spread. The paper needs to explain how incorrect candidates are identified or rejected and provide evidence on how candidate noise affects augmentation quality. The current Controlled Ambiguity setting does not directly test this process.
W4. The replacement constraints and scope of consistency checks for overlapping triples are unclear
Even when the span to be replaced is correctly located, the replacement may still affect other triples that share entities or context.
For example, the same sentence may contain:
When entity A is replaced, both triples need to be updated together. When the context expressing relation r₁ is changed, the modified sentence must also be checked to see whether it still supports r₂. More generally, an entity may participate in multiple relations or have repeated mentions, and several relations may depend on the same text span. In these cases, a local replacement may affect other triples.
Table 11 already shows that a shared entity is updated across multiple triples. However, it is unclear whether candidate selection considers all related triples together. It is also unclear how the sentence-level consistency score handles compatibility conflicts between triples. The authors should specify these rules to explain how all related annotations in an augmented sentence remain consistent after a local replacement.
W5. The research motivation is reasonable, but some gaps remain
Whether augmentation preserves consistency between text and triple labels is an important issue for training data quality. The paper designs its augmentation method around this task property, giving it a clear practical motivation. However, relevant experiments or statistical analyses are missing to support this motivation.
Questions for authors
Q1. How is a Relation Span obtained?
How does the method locate the corresponding text span when a relation is provided only as a discrete label or expressed implicitly through context? Please provide the actual algorithm. Please also explain the operation in Table 11 where the sentence remains unchanged but the relation label changes.
Q2. Can experiments, statistical analyses, or visualizations clarify the contribution of Consistency-Aware Filtering?
Table 4 already reports results for No Filtering. Could you further compare random sample selection with Consistency-Aware Filtering using the same candidate set produced by Constrained Structured Augmentation? Please compare downstream F1 with the same sample count and training settings. Could independent statistics or visualizations also show the text–triple compatibility rates of the two sets? This would test whether the performance changes from Consistency-Aware Filtering are accompanied by a reduction in incompatible samples.
Q3. How can errors from candidates be prevented from spreading to originally correct samples?
If candidate source samples have incorrect entity types or relation annotations, could these candidates be used to augment originally correct samples? How does the method identify or reject them? The auxiliary relation classification module is also supervised by the original training labels. Could targeted sample analyses or controlled experiments assess how effectively Consistency-Aware Filtering prevents error propagation and how candidate noise affects augmentation quality? Positive and negative cases are needed to support the analysis and explanation.
Q4. Which relations are covered by the compatibility checks for overlapping triples?
Table 11 already shows that a shared entity is updated across multiple triples. Does candidate selection also consider all triples affected by a replacement? How does the sentence-level consistency score combine the scores of these triples? How does the method handle a candidate that satisfies the constraints of the current triple but is incompatible with other related triples? The case analysis needs representative examples with more triples, rather than only examples with two triples.
Q5. What direct evidence supports the motivation that generic augmentation disrupts triple compatibility?
Could representative case analyses or statistical analyses show which types of incompatibility between text and triple labels are introduced by generic augmentation, and how often they occur? This would help assess whether the problem is prominent under the actual experimental conditions. Such an analysis could clarify the empirical basis of the research motivation.
5 comments
Response to CW2 & Q2: Isolating CSA via Random Subset (91.35) & Human Audit (82.5%->95.5%)
Critical Weakness 2 and Question 2: Separating CSA from CAF via matched-size random selection and human compatibility statistics
On the identical CSA candidate pool with matched sample count, Random Selection achieves 91.35 (vs. 90.39 Original and 92.40 CAF), and human auditing () shows compatibility rises from 82.5% (CSA) to 95.5% (CSA+CAF). This confirms that CSA independently produces high-quality structured candidates and that CAF's F1 gain directly tracks reduced text–triple incompatibility (see Data R4 under Q5 below; added to Section 4.4).
| Augmentation Variant (Evaluated on NYT Samples) | Downstream Avg Score (P / F1) | Entity Boundary Broken (%) | Relation Semantic Drift (%) | 5% Candidate Noise Blocked (%) | Human-Verified Triple Compatibility (%) |
|---|---|---|---|---|---|
| Original (Unaugmented Baseline) | 90.39 / 89.67 | — | — | — | — |
| Generic Baseline: Word Substitution (WS) | 89.85 / 89.10 | 24.5% | 31.0% | — | 44.5% |
| Generic Baseline: Back Translation (BT) | 90.95 / 90.12 | 18.0% | 22.5% | — | 59.5% |
| SSDAU w/o CAF (CSA + Matched Random Subset) | 91.35 / 90.55 | 3.5% | 14.0% | 68.0% (Centroid) | 82.5% |
| Full SSDAU (CSA + CAF Filtering, Ours) | 92.40 / 90.80 | 1.0% | 3.5% | 94.5%(Centroid+Scorer) | 95.5% |
Data R4: Quantitative audit of generic augmentation failure modes (Q5), random-subset vs. CAF compatibility (Q2), and 5% candidate noise defense (Q3).
Response to CW3 & Q3: Two-Stage Candidate Noise Defense (94.5% Blocked Under 5% Noise)
Critical Weakness 3 and Question 3: Two-stage defense against candidate error propagation and 5% label-noise test
CSA's centroid ranking and CAF's global scorer jointly block 94.5% of corrupted candidates under a 5% source label-noise injection test. (1) CSA ranks candidates by proximity to compatibility-group centroids , down-weighting isolated mislabeled outliers (blocking 68.0% of noise alone); (2) CAF's scorer learns global dataset regularities rather than memorizing local noisy instances, assigning confidence to corrupted replacements (see Data R3 under Q4 and Data R4 under Q5 below; added to Section 4.4).
| Case Type | Source Sentence & Overlapping Triples () | Replacement Operation & Compatibility Check | Pair-Level Scores & | CAF Decision |
|---|---|---|---|---|
| Positive Case (Accepted) | _" |
_$_{h_1,h_2}$<br>_co-founded
__**Apple**___$_{t_1,h_3}$<br>_in
__**Cupertino**___$_{t_2,t_3}T_1$: (Jobs, founder_of, Apple); <br>: (Jobs, place_lived, Cupertino); <br>: (Apple, location, Cupertino) | Synchronously replace shared spans via <br>: Steve Jobs <br>Bill Gates, Apple <br>Microsoft, Cupertino <br>Redmond. | | Retained |
| Negative Case (Rejected Conflict) | Same source sentence (<br>). | Replace shared span <br>(Apple <br>Pixar) while retaining "in Cupertino". Conflict: Satisfies <br>(founder_of), but violates <br>(Pixar location Cupertino). | | Filtered Out |
| Noise Defense Case (Q3) | Same source sentence; candidate pool contains noisy mislabeled span (Paris labeled as Organization). | Candidate Paris for is down-weighted by centroid similarity () and rejected by global scorer on (founder_of). | | Blocked |
Data R3: Positive, conflict-rejection, and noise-defense cases on sentences with overlapping triples (added to Appendix Table 11).
| Augmentation Variant (Evaluated on NYT Samples) | Downstream Avg Score (P / F1) | Entity Boundary Broken (%) | Relation Semantic Drift (%) | 5% Candidate Noise Blocked (%) | Human-Verified Triple Compatibility (%) |
|---|---|---|---|---|---|
| Original (Unaugmented Baseline) | 90.39 / 89.67 | — | — | — | — |
| Generic Baseline: Word Substitution (WS) | 89.85 / 89.10 | 24.5% | 31.0% | — | 44.5% |
| Generic Baseline: Back Translation (BT) | 90.95 / 90.12 | 18.0% | 22.5% | — | 59.5% |
| SSDAU w/o CAF (CSA + Matched Random Subset) | 91.35 / 90.55 | 3.5% | 14.0% | 68.0% (Centroid) | 82.5% |
| Full SSDAU (CSA + CAF Filtering, Ours) | 92.40 / 90.80 | 1.0% | 3.5% | 94.5%(Centroid+Scorer) | 95.5% |
Data R4: Quantitative audit of generic augmentation failure modes (Q5), random-subset vs. CAF compatibility (Q2), and 5% candidate noise defense (Q3).
Response to CW1 & Q1: Deterministic Relation Span Algorithm & Table 11 Label Augmentation
Dear Reviewer DZMK, We sincerely appreciate your thorough review. Below, we address each weakness and question individually.
Critical Weakness 1 and Question 1: Deterministic Relation Span localization algorithm and Table 11 (
place_lived→location)We localize explicit Relation Spans via the shortest dependency path intersected with the intervening window, set implicit/adjacent relations to ∅, and use Table 11 to illustrate label-space augmentation. Given x=(w1,…,wL) and triple (h,r,t) with $h=
ih, jh, t=
it, jt$ (jh < it): (1) we extract the shortest dependency path $\mathcal{P}_{\text{dep}}(h,t) \cap
jh+1, it−1$ bounded by $s_r =
min(Pdep), max(Pdep)$; (2) if jh+1=it or the intersection is empty, sr=∅ and surface replacement is bypassed. In Table 11, changing
place_lived→locationwithout altering text represents schema-compatible label augmentation, where the existing sr validly supports a coarser relation in our library (added to Section 3.2).