acceptodds
Under review as a conference paper at ICLR 2027

From Diagnostic Intent to Physical Experiments: Grounded Experimental Design with Language Models

Abstract

Language-model agents can interpret diagnostic context and formulate useful questions, while physical systems answer experiments with continuous measurements governed by dynamics, operating conditions, and sensor uncertainty. This creates an interface problem when changing context determines which registered experiments should be considered before numerical design. We introduce PhysGround-BED, a grounded interface in which an LLM constructs a posterior-dependent semantic experiment space by proposing typed intents from the current belief, available capabilities, and operating context. A deterministic compiler maps these intents to executable acquisitions, and a grounded response model evaluates continuous-response expected information gain, selects an acquisition, and updates the posterior. A history-conditional analysis decomposes one-step utility loss into response-model mismatch, compilation omission, and numerical estimation error. We evaluate the interface on distribution-network topology identification, supercomputer telemetry, source localization, damped-oscillator identification, hyperspectral sensing, and battery impedance diagnosis. The results show that posterior-dependent semantic restriction can retain informative acquisitions with fewer numerical candidate evaluations in large catalogs and can translate textual sensor-quality information into grounded acquisition restrictions.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Bayesian experimental design, large language models, physical systems, diagnostic agents, active diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MXMoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Given a physical system that may be in one of k states Θ={θ1,...,θk}\Theta = \{\theta_1,...,\theta_k\}, the goal is to find the state by sequentially choosing candidates xtx_t to pass to the system and observing response

yt∼pd(yt∣θ∗,xt,Ht−1)y_t \sim p_d(y_t | \theta^*, x_t, \mathcal{H}_{t-1}).

Sequential Bayesian Experimental Design solves this problem by assuming a prior π0\pi_0 over Θ\Theta, and for every possible candidate in the candidate set xt+1∈Xx_{t+1} \in \mathcal{X} using the statistical model pd(..)p_d(..) as a likelihood to build a posterior πt\pi_t. This posterior can then be used to predict expected information gain (EIG) of observing yt+1y_{t+1} for a hypothetically chosen xt+1x_{t+1}, then the input xx with highest EIG is chosen for evaluation, a true response yy is observed, the posterior over Θ\Theta can be updated, and the process repeats. In my understanding, this is a somewhat standard procedure in BED and active learning and EIG is a standard acquisition function to quantify the hypothetical benefit of observing an output for a given input.

This work takes inspiration from BED-LLM that applies the above recipe to generating questions xx to elicit user answers yy in order to learn latent user preference θ\theta, the LLM proposes questions, EIG quantifies the benefit of each question, the best question is then given to the user. The BED-LLM use case applies when candidate xx and response yy are strings.

This work considers a setting much closer to standard BED where xx and yy are standard mathematical variables, explicitly xx is a point from a given discrete candidate set (hence it may be categorical/integer/continuous variable) and yy is a continuous scalar or vector response.

For applications that have a large candidate set X\mathcal{X}, evaluating the EIG for all possible input points may be expensive. Secondly, some applications may also provide text feedback along with a response, such as error messages, logs, constraint violations.

This paper considers how to use an “LLM-in-the-loop” that may alleviate both of these issues. An LLM has text in (prompt) and text out (answer), while the physical system inputs and outputs are numerical. Hence application specific adapters must be constructed, in this case a “serializer” maps from data history and a list of “intents” to text prompt and a “compiler” from text answer to a subset of candidates X\mathcal{X}.

The BED pipeline proceeds as normal, instead of evaluating EIG for all candidates, we use the the LLM to filter to a subset of candidates and EIG is evaluated only for the subset. If I understand correctly, the contribution of PhysGround-BED is as an acquisition function optimizer in an otherwise standard BED pipeline.

Extensive experiments comparing PhysGround-BED to evaluating EIG across the unfiltered set of candidates (Full Physics BED baseline) shows that the filtering does not hinder performance, and in fact the filtering does drastically reduce the candidate set.

Critical strengths
  • integration of LLMs into BED act as an efficient AF optimizer
  • the benefits over exhaustive evaluation and error mesage incorporation are clearly demonstrated in experiments.
Critical weaknesses

Narrative from BED-LLM

  • If I understand correctly, BED-LLM uses the LLM for candidate generation as candidates xx are text.
  • PhysGround-BED uses the LLM for candidate filtering and saving on EIG evaluations,
  • in contrast with the prior work, this is not a string based setting suited to LLMs and hence adapters are required. The paper thus needs to justify why traditional BED methods should include LLMs
  • I understand that the adapters + LLM are serving the purpose of warm start for EIG optimization (or any acquisition function, AF, optimization). Hence other forms are AF optimization are the baselines for experiments.

Two Main Advantages have weak Evidence

1. Reducing EIG evalautions

  • Is running an LLM (typically multi-GPU workload required 10-100GB memory) cheaper than full EIG evaluation over a finite candidate set in these applications? Is it cheaper than running any other more intelligent form of EIG optimizer?
  • If the candidates are points in a continuous space X⊂Rd\mathcal{X}\subset \mathbb{R}^d and naive brute force is slow, one may use multi-start gradient ascent (grads via reparamerization trick) to find argmaxEIG(x)argmax EIG(x) and just evaluate EIG for all points in the neighbourhood of the argmax.
  • if the candidates are categorical and EIG is estimated MC sampling, one may use off-the-shelf multi-armed bandits to allocated MC samples to promising candidates and ignore poor ones.
  • Regarding the first advantage of reducing EIG evaluations, the paper does presents evidence that placing an LLM in the improves over brute force, but does not compare to any other more efficient method nor does it account for the (significant) extra memory/FLOPs of using an LLM.

2. Error message Incorporation

The second advantage LLMs provide is error message feedback that can be incorporated into the filtering of the candidate set. The paper does provide an experiment that shows a nice clear advantage. However the difficulty of the problem and hence the value of this contribution to ICLR is not clear.

The compile component of the pipeline acts as the adapter from text based LLM output to filtered candidate set. I presume this must be domain specific implementation. What do intents look like? How are they mapped to candidates? Can a basic message parser and some quick heuristics achieve the same filtering of invalid candidates? Do I need a full LLM?

(Presumably prior work along these lines would have fallen into the field of "constrained optimization" with unknown constraints and for benchmarks in such communities, error messages are simply turned into binary feedback for the optimizer, i.e. message parsing and incorporation may be a solved problem. If not text based contrains in constrained optimization may be a fruitful research direction)

The internals of Validate$_d(\epsilon, \mathcal{X}$) and Γd(....)\Gamma_d(....) are not clear, does this work solve a difficult problem? Is the problem implementation/engineering or is it algorithmic/scientific? I struggle to judge if this contribution is suitable for ICLR and if suitable whether it is sufficient for acceptance.

Limitations are not listed

  • a user must write an application specific adapters: serializer l2()l_2(), Validate() and Γ()\Gamma()
  • the computational cost of autoregressive token generation for an LLM is very high, only applications where EIG is very expensive really benefit from compute savings. (This is not the case if a cloud provider is hosting the LLM, in which case this is harness engineering and arguably less suitable for ICLR)

Presentation

I felt that the writing was very cluttered, below are a handful of examples that led to this paper feeling unnecessarily hard to read for me

  • Many sentences contains lists often of items that have not been introduced or defined clearly. For example Lines 201-206 list many many components of all the applications which is rather unhelpful for non-experts in these applications. In general there are a lot of lists.

  • Many of the objects and quantities being discussed are prefixed by many adjectives/descriptors.

  • Details about the applications appear to be spread out throughout the paper. A clear definition would be much more helpful: for each setting, what are X\mathcal{X}, Y\mathcal{Y}, Θ\Theta and their dimensions? what is kk? what are typed semantics/intents? Ideally this should be described once directly after the problem formulation where all variables (including intents) are first defined, or at the start of the experiments/benchmarks whereafter the applications will be used. Currently such information seems to be scattered throughout the paper. Ideally method sections focus only on the mathematical formulation specifically avoiding application specifics (all paper sections follow the single responsibility principle, makes it far easier to read).

  • I may be misunderstanding but terminology appears to be from probabilistic programming languages but may not be standard for BED or more general statistics/LLMs (prior works [1,2] do not use such terminology either). As one option, perhaps the following terminology may broaden the audience of the paper increasing reach and adoption (though many other options may suffice too)

    • "Catalogue": candidate set
    • "Acquisitions": candidates
    • "Compiler": filter, subset selector
    • "Compiled acquisitions", "registered candidates": filtered candidate set
    • "Backend model": likelihood, statistical model
    • "Typed X/Y/X" : a prespecified list of X/Y/Z

Minor comments:

  • Equation (1) is never referenced, can it be removed?

  • Definition one seems overly formal in this more engineering-based paper, can it just be any other equation?

  • Perhaps emphasise the goal of the algorithm more (I missed it the first few times) on line 121 can be replaced

121: "we study finite-hypothesis closed-set identification over a horizon T≤∣Xd∣T ≤ |Xd|, with terminal prediction θ = arg maxθ∈Θd πT (θ)."

"The goal is to identify the true current state, θ∗\theta^*, by sequentially choosing $x\in\mathcal{X}, observing yy and using the model $p_d() to learn about θ\theta."

Questions for authors
  • what is the difficulty in incorporating error messages in the LLM?
  • there are far more efficient ways to optimize EIG than exhaustive evaluation, the only baselines so far, they need to be included as baselines to make a convincing case for introducing an LLM as an alternative
    • continuos xx: multi-start gradient ascent with early stopping when diverging too far from the given candidate set
    • categorical xx: simple UCB multi-armed bandit
Reviewer_spiBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PhysGround-BED, a framework connecting language-based diagnostic intentions to executable experimental design. An LLM generates belief-conditioned semantic proposals, a deterministic compiler maps them to feasible experiments, and a numerical response model supports expected information gain scoring and Bayesian updating. Experiments across several diagnostic and sensing tasks show substantial reductions in candidate scoring in some settings while retaining full-search accuracy, alongside performance losses in others. The paper also examines how textual operating context can guide experiment selection.

Critical strengths
  1. A useful interface between language guidance and executable experiments. The framework explicitly separates semantic proposal generation from numerical scoring and belief updating. This makes the decision process inspectable and provides a concrete way to incorporate textual context and domain constraints into experimental design without relying on the LLM to produce calibrated response probabilities.

  2. Substantial reductions in candidate scoring in several settings. On Source Location and IEEE 123, the method matches full-search endpoint accuracy while using approximately 1.2% and 5.7% of the candidate scores, respectively. These results demonstrate that semantic restrictions can preserve useful experiments within much smaller candidate sets, supporting the potential value of the approach.

Critical weaknesses

The paper clearly explains its extension from language-native experimental design to grounded, continuous-response diagnosis. My remaining reservation concerns the significance of this extension. The typed semantic interface and numerical backend form a useful integration, and the candidate-scoring reductions provide meaningful empirical support. However, I am not yet convinced that the work establishes a substantial methodological advance or a deeper, generalizable insight beyond this integration. I therefore regard the contribution as useful but relatively incremental.

Questions for authors

Which aspect of the proposed interface constitutes the principal methodological advance, and how do the existing analyses or experiments establish its significance beyond a useful integration of established components?

Reviewer_X73iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhysGround-BED, a framework combining LLM-based experiment proposals with Bayesian experimental design for sequential physical diagnosis. Given a predefined finite hypothesis set, acquisition catalog, and response model, an LLM generates structured experimental intents conditioned on the current posterior, operational context, and history. A deterministic domain-specific compiler maps these intents to candidate acquisitions, which a probabilistic backend scores using continuous-response expected information gain before selecting a measurement and updating the posterior. The paper also decomposes one-step utility loss into response-model mismatch, compilation omission, and numerical estimation error. Evaluation across six physical and scientific diagnosis tasks shows that the framework can preserve diagnostic performance while reducing numerical candidate evaluations in several settings, and can translate textual sensor-reliability information into acquisition restrictions.

Critical strengths
  • S1: The framework offers a well-defined interface between language-based experiment proposals and grounded Bayesian inference. Typed intents and deterministic compilation allow the LLM to guide acquisition selection while leaving continuous-response information-gain estimation and posterior updating to an explicit probabilistic backend. This separation is consequential because it enables textual context to inform experimental design without requiring the LLM to generate physical responses or likelihoods. The accompanying one-step error decomposition makes the framework’s dependencies explicit by distinguishing response-model mismatch, candidate omission, and numerical estimation error.
Critical weaknesses
  • W1: The incremental value of the LLM proposer is not sufficiently established. The framework depends on predefined hypotheses, acquisitions, semantic fields, compilation rules, and response models. The deterministic compiler also resolves posterior-relative descriptions such as “between modes,” implementing part of the posterior-dependent mapping to executable candidates. The fixed-history Source Location audit provides useful attribution evidence, but the deterministic posterior-aware heuristic remains competitive, and the audit does not establish the end-to-end advantage of independently executed LLM proposal policies. Comparisons across the principal applications that hold the compiler and probabilistic backend fixed while varying the proposer are needed. This is my primary concern: the current evidence does not sufficiently establish whether the central LLM component provides meaningful benefits beyond the engineered interface and simpler proposal strategies.

  • W2: The sensor-fault experiment does not isolate the benefit of language-based acquisition selection. PhysGround-BED receives an operator note identifying unreliable spectral regions, whereas Full Physics BED uses the nominal response model without equivalent fault information. The improvement from 28.3% to 40.0% balanced accuracy therefore cannot be attributed specifically to the LLM’s processing of that information. A baseline that supplies the same fault information as explicit band-exclusion constraints, and potentially a simple text-parsing alternative using the same backend, would provide a more informative comparison. This omission weakens a central piece of evidence for the framework’s ability to exploit textual operating context beyond what straightforward constraint handling could achieve.

Questions for authors
  • Q1: The evaluation leaves important diagnostic reliability questions unresolved. The method assumes a finite, closed hypothesis set and returns the maximum-posterior hypothesis after a fixed acquisition budget. Beyond fallback or termination rules for failed compilation, the paper does not describe diagnostic abstention, detection of states outside the hypothesis set, or stopping based on posterior uncertainty. Under the stated terminal decision rule, a completed run would still select a predefined hypothesis if the true state were absent, while an insufficient budget could force a decision with substantial residual uncertainty. These are scope limitations rather than violations of the paper’s assumptions, but the absence of evaluations of omitted hypotheses and terminal posterior calibration limits confidence in its applicability to realistic diagnosis. This concern reinforces the methodological and empirical weaknesses above, rather than constituting an objection to fixed-budget closed-set evaluation itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.