acceptodds
Withdrawn from ICLR 2027

TraceSpec: Active Evidence Acquisition for Executable Monitor Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Executable monitor synthesis from language contains two distinct uncertainties: semantic uncertainty about how an alert maps to deployment-local signals, and policy uncertainty about stateful parameters that language alone does not identify. We formulate active monitor synthesis, where a model receives alert language, an opaque local metric/tool catalog, and a measurement budget, and must choose which evidence to acquire before emitting an executable stateful monitor. We introduce TraceSpec, which combines a compact language model for semantic binding and value-of-information decisions with bounded symbolic inference for temporal-policy identification and compilation to MonitorIR. Activation evidence constrains onset-side behavior, whereas recovery evidence resolves parameters that remain observationally equivalent after activation alone. On controlled deployments derived from held-out Thanos and Mimir rules, Qwen3.5-4B reaches 0.904/0.929 execution agreement, versus 0.686/0.738 for the strongest compact controls and 0.912/0.942 for a gold-semantics/oracle-tools upper bound; Phi-4 reaches 0.903/0.922. One informative phase raises execution from about 0.48 to 0.80, and a complementary second phase adds another 10.2–13.9 points. The learned acquisition policy also transfers to an unseen cost value and budget. Within the bounded monitor family studied here, these results show that semantic reasoning, active evidence acquisition, and bounded policy identification form complementary parts of executable synthesis.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
neuro-symbolic learning, active information acquisition, monitor synthesis, language models, formal reasoning, executable policies

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_amNvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Summary

The paper introduces a method for turning natural-language alerts into monitors, the authors call it "active monitor synthesis". The method, named TraceSpec, consists of a fine-tuned language model and brute-force search. The model deals with processing the alert, while the brute-force search fits some parameters, and results are then compiled into a state machine.

Evaluation

The abstract does not contain a basic introduction. Already the first sentence uses the terms "deployment-local signal" and "stateful parameters", before the reader is provided with any context to help them with interpretation.

The introduction fails to introduce the basics and help the reader build an intuition. The paper does not do a proper job of stating the preliminaries, introducing concepts and the corresponding notation. Section 3 is titled "Problem Formulation", while it never explicitly formulates a goal in unambiguous terms. The authors should consider starting the paper with a motivating (real-world) example. Also, they should state the problem in a general form using mathematically explicit terms, after introducing said terms.

The presentatoin makes it almost possible to understand the context of the approach or the proposed method. It is not clear what exactly TraceSpec does or how does it work, because Sections 3 and 4 use terms which were not introduced. For example, subsection 3.2 mentions "admissible tool utu_t" and "semantic plan zz", neither of which was ever explained. Subsection 4.2 starts with "real monitoring rules", "augmentation", "deployment-local hidden policies", and it is not possible to reconstruct the meaning of these terms. Some more examples of concepts which have not been properly introduced or defined are "bounded policy family" and "hidden test traces". These sections need a complete rewrite because currently they do not convey sufficient information.

There are also numerous issues with the writing. Throughout the text, demonstrative pronouns and adjectives are overused, such as "this", making some sentences difficult to parse. The text has an unnatural number of sentences telling the reader what some part is NOT doing, such as "[...] the result is not a measured latency or dollar-cost Pareto frontier." The paper simply invents vocabulary on the spot, making it overfilled with abstract technical-sounding jargon, like "An activation-only history terminates before recovery-side transitions occur [...]". The sections are unmotivated; every time a new section begins, it is immediately through a subsection which does not provide the reader with any intuition or idea of what they are about to read. The authors could add some text at the beginning of a section, before the first subsection.

Figure 1 looks nice and helps convey the constructed pipeline, but it contains tiny text in grey at the bottom which should probably be part of the caption.

The section about Experimental Setup and the Results section are also poorly presented. Also, earlier sections do not provide the necessary framing within which the experiments should be understood, further aggravating the ability to judge the empirical results. Unfortuntunately, the lack of proper preliminaries makes it impossible to provide feedback for this part of the presentation.

Considering the substantial issues with clarity, it is difficult to assess the technical merits of the work reliably in its current form. The paper is therefore not yet ready for publication.

The AI use statement is honest about the use of generative AI. However, given that substantial portions of the paper are extremely difficult to follow, the extent to which it was reviewed and revised by the authors ("prose was reviewed and revised by the authors") is not clear.

Critical strengths
  • Related work motivates the problem well; the related work helps the reader better to understand the contributions of the submission.
Critical weaknesses
  • The abstract is extremely hard to read.
  • The introduction fails to introduce many of the concepts it discusses.
  • The paper does not make it clear what problem is being solved or how.
  • The technical details are insufficient for assessing the merit and correctness of the proposed method.

Overall, the quality of writing and presentation is far below the bar for a publication.

Questions for authors

None.

Reviewer_givCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Summary and key contributions:

  1. the problem being tackled: how to construct a stateful operational monitor when natural-language alerts have either semantic and policy uncertainty (the separation matters and seems well conceived)

  2. the approach, which combines a fine-tuned LM planner with bounded symbolic policy inference. On each turn, the planner receives a package that includes alert, metric, etc. and decides whether to request extra measurements or it emits a structured semantic plan to identify relevant info.

  3. the experiment conducted and the findings. Training uses 120 monitoring rules from kube-prometheus, Ceph, and Grafana monitoring mixins. Evaluation uses 24 held-out Thanos rules and 93 held-out Mimir rules, each with four deployment variants, giving 96 and 372 evaluation instances. A deployment-grounded synthesis task and a neuro-symbolic implementation demonstrating the value of combining semantic interpretation with complementary observations to infer a bounded executable policy.

Critical strengths

The novelty of the approach. Different from existing straightforward language-to-specification translation for monitor synthesis, it is also jointly grounded with an alert in a deployment-local interface and identifying its hidden temporal policy through budgeted observations

Critical weaknesses

The main limitation (noted by the authors) is that the experiments do not establish production reliability, intervention efficiency, or an advantage over a simpler same-model fixed two-probe policy. The external experiment is only an onset-side offline replay, and the remaining errors are not attributed quantitatively to individual modules

Questions for authors
  • Section 3.3 appears to suggest that activation and recovery observations jointly identify the relevant temporal parameters. How should this be reconciled with the gold-semantics/oracle-tools configuration achieving only ~0.9 execution agreement?

  • Can the staged inference procedure fail to recover a policy whose behavior is fully determined by those observations?

  • Are all generated evaluation targets representable in the verifier’s search space? If so, out-of-family targets cannot explain errors on this evaluation. If not, what proportion of the evaluation targets are outside the search space, and how does that affect the interpretation of the oracle result?

  • Why not compare against a simpler same-model fixed two-probe policy?

Reviewer_AqDzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents TraceSpec, an approach for generating executable monitoring rules from natural language when the deployment-specific metrics and temporal parameters are unknown. The approach uses a fine-tuned LLM to identify relevant metrics and actively acquire evidence through tool calls. A deterministic verifier then uses the collected activations and recovery traces to identify the monitoring parameters. The resulting monitoring policy is represented as an executable state machine called MonitorIR.

The authors evaluate their approach on monitoring rules from two repositories, Thanos and Grafana Mimir, achieving execution agreement of 0.904 and 0.929, respectively. They also investigate how complementary activation and recovery evidence improves policy identification and evaluate generalization across different fine-tuned models and different settings.

Critical strengths

The paper addresses a practical, interesting problem of generating executable monitoring rules from natural language when the metric mappings and temporal parameters are not known beforehand.

The authors show that collecting complementary activation and recovery evidence substantially improves execution agreement with two informative phases. This demonstrates the importance of gathering additional evidence using tools.

Critical weaknesses

The authors primarily compare TraceSpec against relatively simple baselines, including lexical matching, Naive Bayes, and MiniLM. These baselines aren’t cited in Section 5.2, and details about their implementation/adaptation and an explanation of why TraceSpec does better than them would strengthen the results section. Reporting results across multiple seeds for all the baselines would also improve the reliability of results.

It is unclear whether fine-tuning is necessary for this task, especially given the reasoning and tool-use capabilities of recent LLMs. Comparisons against zero-shot and few-shot prompting along with tool use, and the same LLMs without fine-tuning, would help establish the contribution of the fine-tuning process. The representativeness and diversity of the tasks used in the fine-tuning and evaluation need more detail.

Although the overall approach is understandable, some of the implementation details and experimental settings are difficult to follow. Clarity in writing would help to improve the paper. For example, Section 5.4 uses shorthand such as "costs 1/2 and budgets 2/3" without clearly explaining the values in that section. Furthermore, the paper would benefit from an end-to-end case study using an actual Thanos or Mimir monitoring rule.

Questions for authors

Is fine-tuning necessary for this task given the recent developments in LLMs’ ability? Please compare it with the same LLMs without fine-tuning and with SOTA LLMs for the same tasks.

The paper shows that complementary activation and recovery evidence is useful. However, could a simpler strategy that selects both types of probes beforehand and collects their evidence (when the budget permits), without adaptively deciding which probe to call next, achieve similar results?

Can the authors provide a concrete end-to-end case study using an actual Thanos or Mimir monitoring rule (or maybe a toy rule for simple illustration), showing how the LLM selects tools, uses the returned observations to decide its next action, and how the verifier generates the final executable monitor?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.