- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper develops a model for authorship attribution that combines the textual representation of LLMs with that of Word Adjacency Networks. Each text sample is patched to retain only function words and then used to train a LLM that produces a feature embedding and to construct a Word Adjacency Network that models transitions between these function words as a Markov Chain. Then, the stationary distributions of these WANs are compared across text samples to obtain pairwise distances between them. The resulting weighted graph over the collection of texts is is optionally calibrated and used as an input to a GNN that performs author attribution, additionally taking the LLM embeddings as text features. The theoretical contributions focus on the properties of this learning algorithm and can be summarised as a perturbation analysis that shows that the learning procedure remains stable to small textual perturbations while preserving useful stylistic information to distinguish the authors. The method is evaluated on a single dataset of Early Modern English playwrights, achieving marginally better accuracy compared to the baselines, and showing substantially increased robustness to small perturbations of the WAN matrix.
Critical strengths
None.
Critical weaknesses
-
Neither in the Introduction nor in the Related Work section in Appendix A does the paper state the research gap it is seeking to address. The literature review merely collects the relevant context without identifying the strengths or weaknesses of any existing methodology or established baselines in the filed. The Introduction and the methodology (sections 2-6) are instead a listing of the paper's sequence of methodological choices and results and do not provide a clear contextual motivation for them. Similarly, the evaluation does not consider any established baselines in the field and instead focuses on baselines specifically created by the authors.
-
The evaluation is insufficient to establish the properties of and empirical claims on the learning algorithm proved in the paper. The empirical findings are narrow and primarily focus on the predictive comparison between the model and selected baselines on authorship attribution and location and time period classification, but testing only on one dataset is insufficient establish that the accuracy improvement of the proposed method generalizes to a sufficient number of other datasets. Robustness to perturbations (Theorems 3.1, 5.1) is tested by perturbing the WAN transition matrix rather than proving the robustness of the whole learning pipeline from direct textual perturbation to GNN output. The claim that distinct syntactic styles remain separated (Theorems 3.2, 5.2) is not empirically verified with a dedicated experiment. That OT calibration produces informative edges is replaced by a descriptive analysis of the graph geometry.
Overall, this paper looks a lot more like a listing of nice theoretical properties of a new learning algorithm than a method whose necessity and usefulness is truly motivated by the existing literature and the experiments. The paper does not engage with the existing literature, does not identify a research gap, and does not provide a comprehensive and logically consistent evaluation procedure. Therefore, in my opinion, it does not meet the minimum standards for acceptance.
Questions for authors
- What is the research gap? This should be clarified in the Introduction.
- What are other established baselines in the field and why should I choose your method instead of them? This should be established both in the Introduction and in the methodology, and evaluated using additional baselines and benchmarks in the Experiments.
- Can you close the logical gaps between the theoretical results and their experimental validation? This should be fixed in the Experiments.
6 comments
Rebuttal Continued
W1 & Q1 Continued.
The research gap, to be stated explicitly in the introduction, has two sides.
Technical gap (geometry-aware graph representations).
- Graph construction from estimated distributions is unanalyzed. GNNs are stable to graph perturbations (Gama et al. 2020) and transferable across graphs (Ruiz et al. 2020a), but these guarantees take the graph as given. When the graph is built from distributions estimated from finite, noisy data, as in graph-based learning on histograms or statistical manifolds (Lafferty & Lebanon 2005; Belkin et al. 2006), no prior work traces how estimation noise propagates through the distribution estimates, the metric, the edge weights, and the neighbor selection to the GNN output.
- The choice of metric is made by convention. Graph construction choices are known to change downstream results substantially (Maier, von Luxburg & Hein 2009; von Luxburg 2007), yet it is not known which divergences keep the chain above well-conditioned. We show that candidate metrics fall into three sensitivity families with distinct behavior under perturbation, which turns the choice of metric into a principled choice of geometry.
- Edge-weight distributions are uncontrolled. Graph-learning methods control the sparsity of a learned graph (Kalofolias 2016), but fixing a metric and a kernel gives no control over the edge-weight distribution of a graph built from a different source. We propose a rank-preserving OT calibration (Peyré & Cuturi 2019) and characterize when it preserves stability and discriminability.
Gap in existing authorship attribution methods.
- Frequency-based stylometry relies on function-word frequencies, from the Federalist Papers study (Mosteller & Wallace 1963) to distance measures such as Burrows' Delta (Burrows 2002), as surveyed by Stamatatos (2009), Juola (2008), and Koppel et al. (2009). These features are topic-independent but discard the order in which function words occur, and attribution uses fixed distance rules rather than learned representations.
- WAN-based methods (Segarra et al. 2015; 2016) capture this order by modeling each text as a Markov chain over function words, but they assign a text to the closest author profile under relative entropy. Relative entropy is asymmetric and violates the triangle inequality, so it does not define a metric graph (O1); it is unbounded when a profile assigns zero probability to an observed transition and weights perturbations of rare transitions by their inverse probability, so it is not stable under small edits (O2). Each text is compared only with author profiles, so relations among individual texts are not used.
- Neural and LLM-based methods, including sentence embeddings (Reimers & Gurevych 2019) and learned authorship or style representations (Rivera-Soto et al. 2021; Wegmann et al. 2022), learn rich representations but embed each text in isolation, so a classifier on these embeddings cannot use relations among texts in a corpus. They also come without guarantees on how small textual perturbations affect predictions, to which text classifiers are known to be sensitive (Pruthi et al. 2019).
- No existing method combines these strengths: a learned representation, cross-text structure from a valid and stable metric, and anonymization to function words, together with guarantees on how perturbations of the text propagate to the prediction.
We will also rewrite the related work in Appendix A so that each line of prior work ends with its specific limitation and the part of our framework that addresses it, rather than only describing it.
Rebuttal Continued
W1 & Q2. Established baselines and why to choose this method. We agree that the baselines were insufficient, and we added the following. All results are on play-disjoint splits, where every chunk of a play falls in a single partition, so that no test chunk has a training chunk from its own play.
Baselines. The original WAN method (Segarra et al. 2015) builds one WAN profile per author and attributes each chunk to the closest profile under relative entropy; it uses no LLM features, no learning and no graph. GNNs over WANs (Ruiz et al. 2020b) treat function words as nodes and each chunk as a signal on an author's WAN. The linear classifier on the WAN stationary distribution takes the 30-dimensional of each chunk's WAN as input, and the LLM-MLP uses the LLM embeddings without a graph.
| Baseline | Test accuracy (%) |
|---|---|
| Linear classifier on WAN stationary distribution | 58.2 ± 5.7 |
| GNN over WANs (Ruiz et al. 2020b) | 59.4 ± 7.1 |
| LLM-MLP (no graph) | 65.7 ± 5.9 |
| Original WAN (Segarra et al. 2015) | 68.9 ± 3.2 |
| Ours | Raw weights (%) | OT weights (%) |
|---|---|---|
| LLM-GCN | 67.5 ± 5.3 | 70.2 ± 4.9 |
| LLM-GCN, inductive | 66.9 ± 5.3 | 70.4 ± 5.2 |
| LLM-GCN with concatenated to the node features | 69.5 ± 4.6 | 72.4 ± 5.2 |
| LLM-GIN | 66.3 ± 6.1 | 70.0 ± 5.9 |
| LLM-GPRGNN | 72.4 ± 5.8 | 69.8 ± 5.0 |
| LLM-SAGE | 71.2 ± 5.1 | 69.6 ± 3.6 |
| LLM-GraphConv | 68.4 ± 4.5 | 67.2 ± 5.4 |
| LLM-GraphGPS | 74.3 ± 3.5 | 72.6 ± 3.8 |
| LLM-GT | 72.1 ± 3.5 | 72.7 ± 3.7 |
| LLM-NAGphormer | 70.3 ± 5.0 | 71.4 ± 3.8 |
| LLM-VCR-Graphormer | 72.6 ± 4.3 | 72.9 ± 6.5 |
Why choose this method. LLM-GCN-OT outperforms GNNs over WANs by 10.8 points and the LLM-MLP by 4.5 points. With the same WAN statistics as input, it exceeds the original WAN method (72.4% vs. 68.9%, higher in 7/10 seeds, ). With identical inputs and parameter count, the OT-calibrated graph adds a significant gain over an MLP on concatenated [LLM ⊕ ] features (72.4% vs. 69.8%, ), so the cross-text structure contributes beyond the features. The gain holds when test texts are unseen during training (inductive setting, 70.4%), across five GNN architectures (GCN, GIN, GPRGNN, SAGE and GraphConv) and four graph transformers (GraphGPS, GT, NAGphormer and VCR-Graphormer), all of which outperform the LLM-MLP and, with OT weights, GNNs over WANs (by 7.8–13.5 points, ), with larger frozen encoders (Reviewer 2jLP, Q4), and on CCAT50 (W2 below). Three further results support choosing this method:
- Robustness to text edits (see our response to Reviewer 5xPf, Q2): with 300 function words replaced per chunk and no retraining, LLM-GCN-OT loses only 0.9 points on unseen plays, against 5.3 for LLM-GCN, so the calibrated model stays reliable when texts are noisy or edited.
- Robustness to graph construction (see our response to Reviewer 5xPf, Q3): as the k-NN graph becomes denser, raw weights collapse for some architectures (down to 29.9%), while with OT every architecture stays between 69.4% and 72.5%, so the method does not depend on a carefully tuned .
- Robustness to WAN perturbations (Figure 2; W2 & Q3 below): with a model trained once on the clean graph, accuracy drops by at most 0.9 points with OT across the five metrics, against 3.7–6.5 points with raw weights, so the stability guarantees of the calibrated pipeline hold in practice.
We will state the intended use case explicitly. Our method targets settings where texts must be anonymized to function words and where cross-text structure is available. It is not meant as a replacement for fine-tuned attribution models on unrestricted text.
We note that fine-tuned attribution models are not competitors to our framework but possible components of it: the LLM embeddings are the node features of our graph, so any encoder, frozen or fine-tuned, can be used in Stage 1, and the WAN graph adds cross-text structure on top of it. Our larger-encoder experiments support this: with OT calibration, the graph improves over the encoder alone for MiniLM and mpnet (see our response to Reviewer 2jLP, Q4). Moreover, a model fine-tuned on unrestricted text sees the topical content that our anonymized setting deliberately removes, so it would not be evaluated under the same conditions. We will discuss these approaches and where each falls short in the introduction (see W1 & Q1), and add these baselines and CCAT50 to the experiments section of the paper.
Rebuttal
We thank the reviewer for the review, for finding the presentation clear, and for recognizing that the paper establishes a listing of nice theoretical properties. We take the concerns about motivation and evaluation seriously and answer each point below, with additional experiments that validate our theoretical findings and a clearer account of the settings in which our method is a competitive choice.
First, a factual clarification: the LLM is not trained in our pipeline. We use a frozen pretrained encoder (all-MiniLM-L6-v2, Appendix D.1.6) to embed the masked texts, and only the GNN/MLP layers are trained. We will make this more visible in §2.
Experimental setting. Unless otherwise stated, all new experiments in this response use the default setting of the paper (Section 7 and Appendix D): Hellinger distance, k=3, α=β=1, frozen all-MiniLM-L6-v2 embeddings, and the same GCN and training protocol. Results are mean ± std over 10 seeds.
W1 & Q1. What is the research gap? We agree, and we believe the root cause is the framing. The current version makes authorship attribution look like the main focus, whereas our main contribution is technical: studying the design of metrics and edge weights for geometry-aware graph representations built from estimated distributions, with guarantees that can be replicated to apply to a much wider range of graph tasks. We will revise the abstract, introduction, and conclusion to reflect this explicitly and to treat authorship attribution as a case study. Authorship attribution is itself a broad field with diverse subproblems, including stylometric representation learning, detection of AI-generated text, plagiarism detection, tracing how literary style evolves over time, and dialect classification; our time-period and location tasks are instances of the last two. Specifically:
The reviewer also notes that Sections 2–6 read as a sequence of methodological choices without contextual motivation. We will state explicitly that each design choice in Sections 2–6 is motivated by the objective it serves: the metric by (O1)–(O2), the kernel by (O2)–(O3), the k-NN margin by (O2), and calibration by (O4), as stated in the contributions (C1)–(C3).