- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Semantic Release generates private reports by releasing shared statistics once, compiling answers from them, and passing those answers to an LLM writer. The paper provides a transcript-level zCDP guarantee and evaluates a nine-view Gaussian implementation on LongWeave SR8K. Across 48 paired reports, correctness increases from 20.96% to 24.74% relative to question-wise reacquisition at the same privacy budget.
Critical strengths
The separation of private acquisition, answer compilation, and writing provides an explicit privacy boundary. The exact-view controls are useful for distinguishing missing statistics from perturbation error, while the compiled-answer evaluation separates acquisition quality from writing quality.
Critical weaknesses
-
The incremental contribution is not established. The main baseline independently reacquires each view for every dependent question (Section 6.2), incurring the variance penalty derived in Section 5. This isolates the benefit of reuse. Reusable measurements already underpin the matrix mechanism and HDMM. Theorem 1 applies standard composition and post-processing, while the tested system acquires all views before writing. The paper provides no matched release-once statistics-to-text baseline establishing what the typed ledger and compiler contribute beyond conventional reuse.
-
The report-level gain is not validated against the judging protocol. The main evaluator retains context across reports, but the bootstrap excludes this dependence and evaluator variability (Appendix D). Its interval [0.82, 6.76] therefore leaves order and context effects untested. The local evaluator gives a gain of 1.97 points with interval [-0.62, 4.53], and rates the non-private control at 41.50% correctness versus 26.97% in the main evaluation. Fresh-context evaluation and adjudication of disagreements are needed to establish the robustness of the central report-level result.
-
The chosen release scope lacks a matched utility comparison. Nine of 21 views are released. Restricting exact statistics to these views reduces structured correct coverage from 93.16% to 60.35% (Figure 3). The budget sweep holds scope fixed, and the development configurations change scope, allocation, and granularity together (Appendix B.1). Neither establishes how the selected workload compares with broader releases at the same total budget. Since scope restriction is the largest measured source of lost coverage, this missing comparison leaves a consequential design choice unsupported.
-
The core compiler is insufficiently specified for reproduction. Equation 3 defines answer functions abstractly, and Appendix D states that dependencies come from question text. The manuscript does not provide the complete question-to-view mapping, executable answer rules, or concrete abstention thresholds. These determine coverage and the answers supplied to the writer. The listed Gaussian views and budgets are insufficient to reconstruct the reported acquisition and report results.
Questions for authors
- What measurable benefit does the ledger/compiler design add over a release-once statistics-to-text pipeline at the same scope and budget? Which comparison isolates that benefit?
- How were reports and conditions ordered and grouped in evaluator streams? Does the gain persist with fresh contexts, blinded condition labels, and independent adjudication of disputed ratings?
- What selection objective produced the nine-view workload, and how does it compare with broader workloads at the same total privacy budget under the same evaluation?
- Can you provide the full dependency mapping, answer functions, abstention thresholds, and structured-answer correctness criteria needed to reproduce Tables 2 and 4?
Response to Reviewer DDK2
Thank you for the detailed review. We address your four concerns below:
Q1 / W1 — Answer compilation
Answer compilation improved report correctness by 7.44 percentage points at the same privacy budget. Both methods used the same DP releases, writer settings, and scoring protocol. The only change was whether the writer received the released statistics or the answers computed from them.
Table 1. Answer compilation.
Q2 / W2 — Report scoring
We hid the method labels and sorted reports from both methods together by their anonymous IDs. We assigned consecutive groups of four reports in turn to three evaluator sessions and kept the order within each group.
We then scored each report in a fresh session using the same rubric. We hid the method labels and had an independent adjudicator resolve disputed ratings. Semantic Release still led by 2.40 percentage points.
Table 2. Independent re-evaluation.
Q3 / W3 — Release scope
On the SR8K development set, we searched allocations over 21 candidate views at ρ=4. We chose the allocation with the largest minimum margin across our preset targets for coverage, precision, correct coverage, monetary error, and recall improvement. That allocation gave positive budget to nine views.
We compared it with an equal split across all 21 views, using the same total budget, development questions, and structured-answer scoring rules.
Table 3. Release scope.
The nine-view allocation answered fewer questions and made fewer mistakes among the answers it gave. The share of questions answered correctly was similar.
Q4 / W4 — Compiler and scoring rules
Dependency mapping.
question_view_dependencies.jsonlists the views each question needs. This entry uses product revenue and total revenue.Answer functions. For this question, the compiler finds the top product in
P_REVand divides its revenue by the total inG_REVto compute its share. The answer functions give the calculations for the other question families.Abstention rules. The compiler returns
UNRESOLVEDwhen a required view is missing, when an estimated denominator is zero or negative, or when a comparison fails its stability check. For a qualitative comparison, the absolute difference must exceed 1.645 propagated standard errors. The compiler code gives the checks for each question family.Correctness criteria. For Tables 2 and 4 of the paper, categories and top-three sets must match exactly, and comparisons must match the reference direction. The tolerances are USD 5,000 or 15% of the absolute reference amount for monetary answers, whichever is larger, and 5 pp for percentages. For counts, the tolerance is 3 or 15% of the reference count, whichever is larger. Every required field must pass for a compound answer to count as correct.