- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies Scientific Evidence Minimization: limiting private scientific context shared with external LLMs while preserving sufficient evidence for grounded reasoning. It proposes CEDAR, a verification-guided framework that iteratively refines disclosed context and falls back to local execution when needed. Experiments across four scientific domains show improved grounded task success under strict disclosure constraints, although verifier reliability remains a limitation on open-ended tasks.
Critical strengths
-
Well-motivated problem formulation backed by clear empirical evidence: The distinction between answer preservation and evidence sufficiency is compelling and well-substantiated by the controlled study, which shows that a sparse support-complete subset (21.6% of relations) achieves nearly the same GTS as the full context (0.750 vs. 0.764), whereas a budget-matched random subset drops to 0.333 GTS despite having similar raw answer accuracy (0.875 vs. 0.889).
-
Rigorous evaluation and thorough system auditing: The paper separates the operational search-time verifier from the frozen post-hoc evaluator, and transparently audits both intermediate trajectory exposure (search-wide union) and the respective roles of verification gating versus iterative refinement.
Critical weaknesses
-
Search-time verifier errors on open-ended reasoning tasks (Q1/Q3): In Table 2, CEDAR achieves lower GTS on open-ended tasks Q1 (0.425) and Q3 (0.259) compared to the single-pass One-Shot baseline (0.569 and 0.363, respectively). As shown in the Appendix G.1 audit (Table 16), the search-time verifier exhibits high false-acceptance rates on Q1 and Q3 (57.6% and 74.1% GTS disagreement with the frozen post-hoc evaluator, respectively), causing the search to terminate prematurely before sufficient evidence is disclosed.
-
Assumption of pre-constructed structured graphs and single model-pair evaluation: CEDAR assumes that the private scientific workflow is already represented as a structured graph x=(S,R) with pre-defined exact/abstract representations and relation edges, leaving the impact of automated graph construction from raw unstructured scientific text unexplored. Furthermore, the experiments only evaluate a single local/external model configuration (openai/gpt-oss-120b locally and gpt-5.4-mini-2026-03-17 externally), leaving cross-model generalization unverified.
Questions for authors
-
Clarification on improving Q1 and Q3 performance: Regarding CEDAR's lower GTS relative to One-Shot on Q1 and Q3 in Table 2, is the primary bottleneck the search-time verifier being overly permissive on open-ended answers (as indicated in Table 16), or the initial evidence contract selecting overly sparse seed nodes? Could tightening the verifier prompt directly mitigate this issue?
-
Clarification on the local and external model pairing: The experiments deploy a 120B local model (openai/gpt-oss-120b) in the trusted environment while using gpt-5.4-mini-2026-03-17 as the external model. Could the authors briefly clarify the primary rationale behind this pairing (e.g., controlling API costs across iterative external calls)
-
Future work on extending to raw scientific text: Since the state nodes, abstract representations, and evidence relations x=(S,R) are pre-constructed in the benchmark, which module of CEDAR (e.g., contract generation or failure diagnosis) do the authors expect to be most sensitive to extraction noise when applied to unannotated raw experimental records?