- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BKBAgent combines a physical knowledge graph with an LLM-maintained weighted set of complete battery-parameter hypotheses. The agent proposes parameter vectors and possible simulator outcomes, selects simulations using expected KL belief change, and updates its weights from actual feedback. This is computational experiment selection using fixed measurements, rather than acquisition of new physical measurements.
On 108 synthetic tasks derived from 18 independent perturbation parents, it improves parameter recovery relative to BSA and a closer GP-based reference. Additional evaluations cover ablations, three LLM backbones, unseen operating rates, and one measured cell. 57968_BKBAgent_A_Battery_Param
Critical strengths
-
The scientific-agent application is supported by meaningful controlled comparisons. Methods share parameter domains, response-validation criteria, and a budget counting attempted solves, including initialization and failures. Point estimates are selected without access to parameter ground truth, and parent-level bootstrap respects dependence among rates, parameterizations, and seeds. Matched ablations distinguish external knowledge, relational retrieval, and selection policy. Relative to GP-BKB, the smaller improvements also have paired intervals favoring BKBAgent.
-
The paper carefully distinguishes response matching, parameter recovery, and uncertainty representation. Outputs retain complete parameter vectors, and response validation includes timing and multiple response components. Held-out-rate predictions and a measured-cell case study complement synthetic parameter-recovery results. The authors also disclose limitations, including the absence of an exact Bayesian interpretation, increased API workload, and cases where improved point accuracy does not improve complete-vector recovery.
Critical weaknesses
-
The specific value of information-guided selection and the adequacy of the belief representation remain incompletely established.
- The LLM supplies hypotheses, outcome probabilities, and conditional weights without a coherent joint probabilistic model. Equation (9) correctly shows that the selection utility includes belief drift as well as discrimination between possible outcomes. In Table 5, drift contributes about 31.6% of selected-candidate utility, and drift-only selection produces slightly greater reference improvement than the proposed utility, 0.0908 versus 0.0873.
- These small, one-step diagnostics do not establish that drift-only selection is generally better. They do leave the proposed mechanism unresolved: end-to-end comparisons against uniform, discrimination-only, and drift-only selection are needed. Furthermore, a fixed support of 16 hypotheses cannot recover omitted regions, and the two-dimensional conditional diagnostics do not validate uncertainty representation in the full ten-dimensional problem.
-
Test-set independence is not fully verified, and the evaluation has limited independent breadth.
- Appendix J.2 removes two Extreme parents whose complete parameter truths repeat development cases, but explicitly leaves possible endpoint duplication for coordinates 1 and 10 unverified. Complete separation should be checked and any affected results recomputed.
- More broadly, the 108 tasks represent 18 independent parents under two parameterizations of the same cell family. The unseen-rate tests remain within the benchmark’s operating-rate range, while real-data validation covers one cell and subsequent discharges without parameter ground truth. These support a promising controlled result and predictive case study, but provide limited evidence of robustness across cell populations and model mismatch.
Questions for authors
- Can you verify complete development/test truth separation, especially for Extreme coordinates 1 and 10, and report results after excluding any remaining duplicates?
- How do uniform, discrimination-only, and drift-only selectors perform in full identification runs?
- How sensitive are results to the initial hypothesis support and its size? Can the full-dimensional belief or its predictive uncertainty be assessed?
- Does the advantage over GP-BKB persist under matched token budgets, alongside the existing matched simulation-budget comparison?