- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes HepUKE, a RAG framework for HEP analysis that combines generation uncertainty, structural disagreement, confidence-based precedent selection, gated write-back to a DSL knowledge base, and adaptive retrieval stopping. It is evaluated on BESIII DSL synthesis, BESIII PhysicsQA, and HotpotQA.
Critical strengths
The problem is relevant, and the paper attempts to connect retrieval confidence, knowledge admission, and retrieval cost within a single scientific-agent pipeline.
The early-stopping experiment also demonstrates a useful cost/accuracy trade-off, with substantially fewer retrieval rounds and tokens.
Critical weaknesses
The central claim is not adequately validated. The paper is framed around a “self-evolving” knowledge base, but the experiments evaluate individual admission decisions rather than repeated write-back and reuse over time. The authors explicitly acknowledge that long-term knowledge evolution is not evaluated. For me, this is a central missing experiment rather than a minor limitation.
I am also not convinced by the interpretation of the proposed disagreement measure as HEP “systematic uncertainty.” It appears to be an operational generation/disagreement signal, and the paper does not establish correspondence with physical systematic uncertainty. Moreover, the main physics-sensitive metric, Phys-Acc, is itself assigned by an LLM judge. This is a weak validation basis for a paper centered on scientific reliability.
Questions for authors
Can the authors provide a genuine sequential evolution experiment in which generated entries are repeatedly written back and reused by later tasks, compared with a static-KB baseline? I would want to see downstream accuracy, erroneous-entry accumulation, retrieval drift, and dependence on previously generated entries.
I would also like to see human HEP-expert agreement statistics for Phys-Acc. Without this, it is difficult to assess whether the reported improvements correspond to actual physical correctness.
Response to Reviewer vTbE
Dear Reviewer vTbE
Thank you for a precise and useful report. We respond to the comments and questions point by point below. Where a weakness and a question address the same issue, we combine them into a single response to avoid repetition and keep the discussion focused.
W1. Validation of the "self-evolving" claim
The original experiments did score admission decisions; that is a fair reading of the submitted version. But the testable claim is narrower than convergence and is now measured directly: does what the system wrote earlier change what it does later? In A1 (Sec. 5.4), 80 dev queries stream into a live collection, and 13/80 top-1 selections are written-back entries, all admitted earlier in the same stream. In the family-held-out stream, on the 16 queries where same-family reuse is even possible, a written-back entry is top-5 in 16/16 and top-1 in 9/16. Both are direct measurements of dependence on generated entries.
What cannot be tested in a benchmark is the closed loop---downstream use, validated outcome, write-back, iterate---because outcomes exist only when analyses are published. The seed is not an arbitrary baseline either: BESIII requires expert and physics-coordination review before submission, so the 500 seed programs are validated precedents at the quality the loop converges to. The fixed point is thus already in the KB; what stays open is whether write-back improves on it, which is A1/A2.
Q1. Sequential evolution
The four measurements requested, in Table 3 and Sec. 5.4:
Hence the family-held-out arm. Four families and their same-paper variants are removed (500→480) and the 20 remaining members streamed. Four are family firsts, where no precedent can exist; on the 16 later members a same-family written-back entry is top-5 in 16/16 and top-1 in 9/16, struct rises 0.796→0.809, and 20/20 are admitted with none below 0.5. The App. E case study shows what the aggregate hides: the written-back entry beats the seed entry on
select_trackslot F1 (0.727→0.889) while the composite score moves only +0.0222→+0.0492, because token averaging is unweighted; structurally weighted averaging is future work (Sec. 7). Confidence-gated evolution is safe (no drift, one degraded admission), neutral where the seed covers the query, and positive where it does not.W2. "Systematic uncertainty"
Sec. 4.1 already limits the claim: the analogy is procedural---target held fixed, methodological choices varied, spread taken---applied to analysis structure rather than to a measured observable, with no calibration against an experimental systematic. The issue the reviewer identifies is in naming and emphasis, and we fix it there: the term is renamed "methodological-variation spread (an analogue of the HEP systematic)" at first use, and the abstract no longer suggests a physical systematic. Hsyst,a itself is used as before and used comparatively: it gates admission; it is not a budget on any observable.
Q2. Phys-Acc validation
The single annotator is not the weak point. HEP selection criteria are stated operationally, and the seven-dimension rubric inherits that property: every dimension and every anchor was fixed by a panel of domain experts before any scoring, so the standard sits in the rubric, not in one person's judgment. Against that pre-agreed standard, all 74 Phys-Acc items were re-scored blind by a BESIII expert: ICC(2,1)=0.841, r=0.864, MAD 7.8, weighted κ 0.698–0.913 (General Response B). That is the reproducibility check the reviewer asks for. Residual disagreement concentrates in track and photon selection, exactly where the rubric's least explicit dimension requires reading a cut sequence.
Best regards,
Authors for submission 42629