- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The motivation is data scarcity in law and finance: real contracts and filings are useful for training and testing models but contain private information, so the authors want to release synthetic substitutes. Such a substitute should keep the reference structure that downstream tools rely on (clause numbering, section containment, defined terms, internal cross-references) while regenerating the source-specific wording. The research question, as the abstract states it, is whether long documents of this kind can be regenerated without a single generation prompt whose context grows with document length. Privacy is treated as a secondary, empirical audit rather than a guarantee.
BRIEF parses the source into a content graph, sanitizes and canonicalizes it into a target graph, and writes every clause in its own LLM call from a short "brief" (number path, terms to define or use, the exact cross-reference string, plus a topic, an opening steer and a shared register instruction). No call sees source prose or any other clause, so the prompt stays at about 86 tokens at every length of the synthetic sweep. Fidelity is an exact, extractor-based F1 over containment and cross-reference edges keyed by number path, scored against the target graph. The paper also proposes a per-document capacity diagnostic for structural perturbation, and BRIEF-L, which orders clauses so that cited clauses are written first and adds up to k = 4 clipped summaries of the cited clauses to each prompt.
Experiments use a seeded synthetic corpus (70 to 1200 clauses), 70 CUAD contracts and 47 EDGAR EX-10 exhibits, Qwen3-4B and Llama-3.1-8B as generators, Qwen3-14B and Phi-4 as judges, and a BGE retrieval attacker that holds the source corpus. BRIEF keeps cross-reference F1 near 0.98 on the synthetic sweep, where one-pass generation scores 0.0 on the section tree and erratic cross-reference F1, and it lowers BGE top-1 linkage relative to one-pass on CUAD (0.37 vs 0.70) and EDGAR (0.23 vs 0.49). The authors also report that a per-clause baseline with a generic prompt matches or beats BRIEF on topology, that within-release cross-reference coherence under BRIEF is near zero (0.02 CUAD, 0.06 EDGAR), and that BRIEF-L raises it to between 0.44 and 0.70.
Critical strengths
-
The paper measures a clear dissociation between reference topology and reference semantics. BRIEF's internal cross-references mostly resolve (0.83 on CUAD and 0.88 on EDGAR, Table 13), yet almost none of them point to a clause that makes sense as a target (coherence 0.02 and 0.06, Table 22), and clause roles fall below the majority-class rate (Table 21). Given that the generator never sees the cited clause, the direction is expected, but the size of the gap is a useful warning for anyone who scores generated documents with structural checkers alone. BRIEF-L is a simple, bounded fix (write cited clauses first, pass their summaries forward), and the ablation in Table 23 shows that the summaries, not the edge-consistent topics, carry the gain. This is the part of the paper I learned the most from.
-
The reporting is careful and candid. The authors keep the naive per-clause arm even though it matches or beats BRIEF on topology (Table 3), score against the raw source graph as well as the sanitized target (Table 15), report paired tests with document-level bootstrap intervals, replicate with a second generator family where the replication only partly holds, and list what is not preserved (Table 9). Most of my concerns below are built from numbers the paper itself provides.
Critical weaknesses
-
The structural scores do not yet establish useful long-document regeneration. BRIEF prepends number paths in code, so perfect section-tree F1 partly measures construction rather than model capability; a generic per-clause prompt also reaches cross-reference F1 of 0.98-1.00 (Table 6). One-pass section-tree F1 is already zero with a 1,215-token prompt at 70 clauses (Table 7), which does not establish degradation with length and leaves a rendering/extractor mismatch plausible. Cross-reference performance has no monotonic Qwen length trend. A section-level or plan-then-write baseline is missing between one-pass and clause-wise generation. More importantly, clause-role agreement falls below the majority baseline (Table 21), and no downstream or human utility test shows that preserved numbering and reference strings with unrelated clause content serve the motivating applications. The topology/semantics dissociation is informative, but it does not itself validate the release's usefulness.
-
The topology, coherence and privacy claims refer to different operating points. The headline release keeps one canonical cross-reference per clause; BRIEF-L is evaluated with up to four (Appendix S). In that latter setup even summary-free BRIEF has CUAD linkage of about 0.89, versus the headline 0.37, and BRIEF-L remains at 0.88. The low-linkage headline release has raw-source cross-reference F1 of only 0.67 on CUAD and 0.49 on EDGAR (Table 15). Thus the abstract's small BRIEF-L fidelity/linkage changes are relative to the more-linkable k-citation baseline, not the headline arm. Section 6.7 acknowledges the shared k-citation comparison, but does not make the difference from the headline operating point sufficiently clear. No reported configuration jointly demonstrates preserved source topology, coherent references and low linkage.
Questions for authors
-
How many internal cross-reference edges does the headline target graph keep, and how many does the k = 4 target keep, relative to the raw source graph? Is the number of emitted cross-reference strings the only difference between the headline BRIEF arm and the summary-free BRIEF arm of Appendix S (same prompts, topics, defined terms and document window)? A rise from 0.37 to 0.89 under a mean-pooled embedding attacker from a few extra "Clause x.y" strings per clause is larger than I would expect. If the cross-reference count is the only difference, I would read the privacy claim as applying to a heavily reduced structure, which changes my view of the central trade-off. If you already have the k = 4 releases, their raw-source cross-reference F1 next to Figure 4 would help.
-
What exactly does the one-pass arm receive? The Introduction (p. 1) says it "places the whole source in context and rewrites it," while Appendix H and INV-2 (Sec. 3) indicate that its prompt lists the numbered clauses with the per-clause instructions in brackets and contains no source prose. If it does see source prose on CUAD and EDGAR, its higher linkage is partly explained by that input. What was the maximum output length for one-pass, what fraction of one-pass outputs were truncated at each length, and was Llama's 32K budget (Appendix O) binding at 1,200 clauses? The 0.19 in Table 18 could reflect truncation.
-
For Table 7, what does the extractor see in a one-pass output at 70 clauses that gives section-tree F1 of exactly 0.0 even when the two-level format is stated in the prompt? An example of the raw output would show whether this is a generation failure or a parsing convention.
-
Appendix I (p. 19) says candidate-only edges "are not folded into the F1," but Eq. (1) has |C| in the denominator, and Appendix L (p. 21) attributes one-pass's low fidelity to over-generation of references "rather than omission." Which definition produced the reported numbers? If invented edges are excluded from C, the F1 behaves like a recall, and I do not see how over-generation would lower it.
-
Appendix R explains that the copy arm's coherence differs across tables because each table judges a different subsample or citation setup (0.85 on CUAD in Table 22, 0.57 in Table 23 under the same judge, 0.40 under Phi-4). That spread for the same source documents still suggests a noisy probe, and BRIEF-L scores above the source on EDGAR. Was the probe checked against human judgments on any sample? BRIEF-L writes the citing clause from a summary of the cited one, so a judge that rewards topical overlap would score it highly by construction, while real cross-references often connect clauses on different topics.
-
What does "sanitize" do to defined terms and other strings carried by the target graph? Table 26 shows terms such as "NICE" and "UNDER" that look like source strings or extraction errors. Which source strings are emitted verbatim in a release, and how much of the residual linkage do they account for?
Suggestions for a future version, not needed for my assessment: a deterministic template rendering of the target graph as a baseline; a section-level or plan-then-write baseline in the style of AgentWrite; a larger long-context model for the one-pass arm; and an LLM-based rewriting anonymizer, for example adversarial anonymization (Staab et al., arXiv 2402.13846) or DP-Prompt paraphrasing (Utpala et al., arXiv 2310.16111), as a privacy baseline next to NER masking, which masks only about seven percent of the text (Sec. 6.4).
3 comments
Q5: coherence probe.
- Random-target control. The real cited clause scores 0.59–0.71 for BRIEF-L. Replacing it with a random clause from the same release scores 0.08–0.17. So the probe separates real from random targets, and BRIEF-L's gain is more than topical overlap with any clause.
- Template and source. The deterministic template ("This is subject to Clause x") scores 0.55–0.57. The probe rewards non-contradiction more than specific fit, and the revision states this. The source itself scores 0.56 (CUAD) and 0.38 (EDGAR), below BRIEF-L's 0.63: BRIEF-L writes the citing clause from a summary of the cited one, so the probe favours it by construction. We read it as agreement between citing and cited clause, not realism. The copy arm's spread across tables comes from different subsamples and citation setups, and we do not claim the probe is low-noise.
- Judges. Qwen3-14B and Phi-4 agree on arm rankings (Spearman 1.0, 1.0, 0.94), although Phi-4's absolute scores are lower.
- Human labels. We did not run a human study in the rebuttal window. The camera-ready will report agreement between human labels and the judge on blind pairs.
Q6: sanitization and verbatim strings.
- In prose, 1.2–2.4% of 4-grams and at most 0.1% of 8-grams appear in the source.
- Defined terms are the leak: 47–73% are emitted verbatim, and they drive linkage (Q1). Pseudonymization removes them.
- Entity leakage under pseudonymization:
- ALL-CAPS and organisation names fall to 0–0.4% (from up to 28%).
- Dates and amounts are about 0%.
- "NICE" and "UNDER" are term-extraction artefacts. Table 9 now lists defined terms as preserved unless pseudonymized.
W1: usefulness, and the plan-then-write baseline.
- Assembly. We agree that section-tree F1 = 1.0 comes from code-side assembly and now present it as an invariant. The template renderer confirms it: F1 0.97–0.99, same linkage as BRIEF.
- One-pass, numbering equalized. BRIEF still has higher CUAD fidelity (0.91 vs 0.82) and, without pseudonyms, lower linkage (0.89 vs 0.96); EDGAR ties (0.92).
- Rolling window (brief + previous 4 clauses). Target F1 is 0.88 / 0.87, but coherence is 0.21 / 0.22 (near its random control) at about 2× tokens. Graph-conditioned summaries beat an unconditioned window.
- Downstream. Three LLMs answered "which clause does X cite" on sources and on releases. Their ranking is identical in every condition, and accuracy is higher on releases because references are explicit.
- We claim only that releases preserve the ranking of systems on this structure-dependent task, not absolute accuracy or legal meaning. A sub-clause-count probe was not informative.
W2: operating points. With pseudonymization, one configuration (BRIEF-L, k=4, full graph) shows topology, coherent references and chance-level text linkage together; the numbers are under Q1.
Structural linkability.
- A graph-matching attacker on canonicalized sources links about 0.72 of releases for every arm, including the template.
- A numbering-invariant attacker still links about 0.6 after all sections are merged.
- Our privacy claim now covers only the release text, which contains no party names or confidential details; the section tree is a quasi-identifier of any release that keeps it.
Suggestions. We added the template baseline and a rolling-window baseline, the closest we ran to plan-then-write (it is not AgentWrite-style planning). We did not run a larger long-context model for one-pass. An LLM anonymizer baseline (DP-Prompt or adversarial anonymization) is discussed in Related Work. We did not run it, since pseudonymization already reaches chance text linkage and structure is the remaining channel.
Dear Reviewer 7hwY ,
We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.
Response to Reviewer
Thank you for a careful review; several of your suspicions were right. Setup and shared points are in the general response.
Q1: edges kept, and the 0.37→0.89 linkage gap.
Q2: one-pass input and truncation.
Q3: Table 7 zero.
Q4: Eq. (1) vs Appendix I.