acceptodds
Under review as a conference paper at ICLR 2027

Sequence-Aware Reconstruction for Learning Action Models from Noisy Action-Only Traces

Abstract

Learning symbolic action models from action-only traces is challenging when action identities, argument objects, and object types are uncertain and no states are observed. We introduce Plan-HMM, which uses a fixed-emission, type-specific hidden Markov model to reconstruct object-event traces and aligned pairwise evidence from weighted action candidates. These outputs support symbolic induction without requiring a complete reconstructed plan. Across nine planning domains, Plan-HMM improves transition recovery and invalid-plan rejection over selecting the highest-weight action candidate at each step, with slightly lower valid-plan acceptance on jointly completed settings. It also completes more settings under a fixed time limit. Additional experiments show improved performance under missing observations represented by candidate completions from a fixed training vocabulary.

Primary area:
applications to robotics, autonomy, planning
Keywords:
automated planning, action model learning, hidden Markov models, noisy action-only traces

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dZGgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This work addresses the problem of learning symbolic (i.e., STRIPS) action models from noisy action-only traces. To achieve this, the authors propose plan-HMM, which reconstructs object-event traces from noisy inputs. These reconstructed traces are then given as input to LOCM2, which extracts a STRIPS action model from them. Experiment results show that plan-HMM achieves better trace reconstruction accuracy than related methods and, when coupled with LOCM2, learns more accurate STRIPS models.

The problem addressed in this paper, i.e., learning STRIPS action models from noisy action-only traces, is meaningful. However, I do not believe the paper should be accepted in its current state. The main reason is that the paper is almost entirely unintelligible. I consider myself fairly familiarized with the action model learning literature, including the case of STRIPS action model learning and, yet, I had a very hard time understanding the paper. This problem is exacerbated given the target audience of the current venue (ICLR), who may not be familiar with Automated Planning and STRIPS models. As such, one would expect the authors to carefully lay out the required definitions and terms needed to understand the paper, providing intuitions and examples. This paper, however, followed almost the entirely opposite approach. Section 2 only dedicates half a page to describing the problem setting. Here, several terms are introduced with minor explanations or none at all: "object-event traces" (what are object events?), "weighted action candidates", "steps", "alternative completions from a supplied vocabulary" (what is the "supplied vocabulary"?), "object event a.k" which "identifies action a and its one-indexed argument position k (what is meant by "one-indexed argument position"?), "projection of a deterministic trace onto an object" (what is a "projection" in this setting?), "object sorts", and many more. In fact, despite the final goal being that of learning STRIPS action models, the authors do not provide any background on the STRIPS formalism, with not even a single reference. Later in the text, Section 4 presents the proposed method (plan-HMM), building on the incomplete background of Section 2 and, once again, introducing more confusing terminology: "set of action-position labels EcE_c", "HMM" (the paper never defines what a hidden markov model is), "event-state set", "identity emissions", etc. Thankfully, this section introduces a running example (lines 158-161), but it is not enough to understand everything (also, I would move (part of) this example to Section 2).

In summary, the authors seem to assume that the reader is not only familiarized with STRIPS action model learning, but also with the particular family of LOCM methods. If that is not the case, then the paper becomes extremely dense and hard to understand. As a result, there are several key points that are somewhat lost in the text. The first is that the proposed method, plan-HMM, does not learn a STRIPS action model by itself but, rather, its goal is to remove the noise from the input action traces which are then provided to LOCM2, which will finally output the STRIPS action model. This is mentioned in several parts of the paper (such as in the last paragrahs of the introduction) but not explicitly enough. A second key point concerns the novelty of the proposed method. In lines 117-126, the paper covers the related method LC_M which "extends LOCM to missing and noisy action-only data without underlying states". From this premise, it seems that it addresses exactly the same setting as plan-HMM. Then, the next lines in the paragraph compare LC_M and plan-HMM, without clearly stating what are the advantages of plan-HMM over LC_M. Only at the end, lines 125-126 state that "Its evaluation [of LC_M] under a shared LOCM2 backend does not establish superiority over the complete published system". Such an statement is incredibly ambiguous: does "does not establish superiority" mean plan-HMM performs better or not than LC_M? Does "the complete published system" refer to plan-HMM, LOCM2 or what else? Given the extreme importance of this claim (since the novelty of plan-HMM depends on it), one would expect the authors to be more explicit. In contrast, the reader needs to wait until the experiment results of Table 2, where it seems that plan-HMM clearly outperforms LC_M. To further add to my clarity concerns, Section 5 introduces several metrics that are hard to understand. Since the whole system (including LOCM2) outputs a STRIPS action model, I would like the authors to additionally analyze the errors (e.g., missing/extra preconditions and effects) in the action models learned.

Critical strengths
  • The paper addresses a problem (STRIPS action model learning from noisy action-only traces) that is both hard and meaningful.
  • The proposed method (plan-HMM+LOCM2) seems to learn more accurate STRIPS action models than related approaches (such as LC_M). However, the word seems is key here: due to the overall lack of clarity in the paper, it is very hard to accurately assess the results obtained.
Critical weaknesses
  • By far, the largest weakness is the extreme lack of clarity throughout the entire paper. The reader needs to be familiarized not only with existing literature on Automated Planning and STRIPS action models, but also with the LOCM family of approaches. Otherwise, the paper is extremely confusing and hard to understand.
  • The advantages of the proposed method, plan-HMM, should be more explicitly stated when compared to the existing method LC_M, which addresses the same setting. Experimental results (e.g., Table 2) seem to show that plan-HMM outperforms LC_M, but this should be made more explicit.
Questions for authors

As explained above, my biggest complain concerns the lack of clarity of the paper. As such, I believe the paper should be completely rewritten and made suitable for an audience that may not be familiarized with Automated Planning, STRIPS and, even more so, LOCM. I provide below a non-exhaustive list of suggestions for clarity improvement:

  • Section 2 (Problem Setting) should be greatly extended, carefully explaining the LOCM formalism and terminology.
  • Before Section 2, the paper should provide background on STRIPS action models.
  • I would ensure that Section 3 (Related Work) does not build on terms later introduced in the text. A simple fix is to place this section just above the conclusions (Section 6).
  • As with Section 2, Section 4 (Plan-HMM) should be carefully written, so that it is easy to follow (for this, the running example and Fig.1 may help, but are not enough in their current forms) and every term is carefully defined and explained.
  • Section 5 (experiments) introduces several metrics, which should be motivated better. As I understand it, you evaluate both the trace reconstruction capabilities of plan-HMM alone and, then, also the accuracy of the resulting STRIPS model when coupled with LOCM2. Also, since STRIPS action models are extracted, it would be illustrative to visualize them and analyze their errors (e.g., missing/extra preconditions and effects). In contrast, Figures 2, 5 do not add much (for the latter, what is the difference between "pure missing" and "missing + ambiguous"?).
5 comments

There are clearly too many clarity issues for us to be able to address all of them in a response, but let us discuss one, which is the results of the experiments - specifically, what it is that we measure, and why (we think) this demonstrates that the plan-HMM approach learns better models.

The F1 score (or, if we were to break it down, precision and recall) of the transition matrix reconstruction is arguably only relevant because the approach separates sequence reconstruction and model learning using a LOCM-style algorithm, which uses the consecutive transition matrix as input. This is a somewhat "relaxed" measure of how well the estimated action sequences match the true action sequences. It is "relaxed" because it does not compare the full sequences; it only compares the extent to which pairs of object transitions that are consecutive in the true action sequences are also consecutive (or not) in the reconstruction.

As noted in the response to the first review above, we cannot compare the learned domains directly to the original (true) domains, because the only vocabulary that they have in common are the action signatures. But, if we view a domain as essentially a classifier of action traces - for any action trace, the domain will either consider it valid or not valid - then we can measure the extent to which the learned and true domains agree on this. Note that a "valid" action trace here means one that is executable from some initial state (since we don't have a way to map initial states or goals between the two domains). The VAR and IRR measure are essentially the recall and precision of this comparison: a perfectly learned domain should (1) accept all action traces that are accepted by the true domain, and (2) reject all action traces that are rejected by the true domain. VAR measures the degree to which the learned domain achieves (1) and IRR the degree to which it achieves (2). It is trivial to "learn" a domain that rejects everything (and thus scores high on IRR, but zero on VAR) or that accepts everything (perfect VAR, zero IRR); there is a trade-off between the two measures. The results in table 2 (in particular, consider the "completed only" results, which are not skewed by the number of instances solved) show that plan-HMM achieves a trade-off that we argue is better than all of the baselines. Although the baselines achieve a higher VAR (in particular, all-candidates which is around 10% higher), they do so at the cost of a much bigger drop in IRR (around 30% for all-candidates and top-1, 20% for LC_M). In other words, the baseline approaches (all-candidates in particular) learn more permissive domains, which accept more action traces than they should. Plan-HMM reduces this over-acceptance, without sacrificing too much of correct acceptance (VAR).

The LC_M results reported in the paper are not the results of the LC_M system that Gregory, Lindsay and Porteous built; it is the results of our re-implementation of their system, based only on the description in their workshop paper. As mentioned above, our re-implementation at the time of submission was not complete (it implemented only argument reconstruction, and because LC_M can not use probabilistic observations, it uses the top-1 sequence as input). It has since been extended to a more or less complete re-implementation of the approach, as we understand it from the paper. But we cannot say that the results of running the original LC_M system, if that was available to us, would be exactly the same. That is what the statement in the paper is trying to say (it is, admittedly, expressed in a somewhat clumsy way).

Thank you very much for the clarifications. I understand the approach better now.

If I understood correctly, the action model learning pipeline (sequence reconstruction + model extraction with LOCM) results in a model encoded in a set of predicates different from the ground-truth model (akin to the SIFT algorithm), so it is hard to compare it to the ground-truth model (e.g., missing/extra effects or preconditions, planning accuracy, etc.) Therefore, metrics must evaluate sequence reconstruction/prediction accuracy, which is what you do in the paper. I agree with this, and I believe your approach is still valuable even if the resulting model cannot be used for planning. Still, for future work, my advice would be to consider how to translate problems from the ground-truth encoding to that of the learned model, so that you can perform planning with it. Maybe LLMs can be used for this, although I am unsure.

Another important point is the comparison with LC_M. Given the hypothetical overlap with your system, I strongly believe you should explicitly explain the differences between your method and LC_M, both on the theoretical side and on the empirical side (by showing your method outperforms it, as you currently do). My advice would be to obtain an implementation of LC_M as close as possible to the original and, if that is not possible, to carefully explain the differences and assumptions you made when running LC_M (e.g., LC_M not being compatible with probabilistic observations, unlike your method). This could be placed in an Appendix.

With these revisions, along with improved clarity and extended background, I believe the paper would be a strong contendant for acceptance at a major conference. Unfortunately, I cannot recommend acceptance in its current state, so I will keep my score.

Thank you. We will take all this feedback into account for any future revision/submission of the paper.

Just a comment on how to translate problems from the ground truth model to the learned domain: This is possible, in certain cases, by using action sequences that define the initial state and goal. I would also say it is possible to use the learned domain for planning, but we have to think of the problem specification differently: Given a prefix, \pi_pref = a_p1,...,a_pn and a suffix \pi_suff = a_s1,...a_sm, we can ask a planner to find a "middle" action sequence, \pi_mid, such that the concatenation of \pi_pref, \pi_mid and \pi_suff is "valid" (in the sense defined above, i.e., executable in some state).

For example, let's take blocksworld: If the prefix sequence is putdown(A), pickup(C), stack(C,A), pickup(B), putdown(B), then the end state of executing that sequence is unique (assuming no other blocks than A,B,C, and that the starting state satisfies the domain invariants). Likewise, the sequence unstack(A,B), putdown(A), unstack(B,C), putdown(B), pickup(C) is only executable in a state where A is on B is on C is on the table; in other words, requiring that suffix sequence is equivalent to specifying a goal state with an A-B-C tower.

However, this example works in blocksworld only because that domain (and many others, but not all) have a particular property: roughly, that all aspects of the state of an object are fully determined by the last action that touched it. In domains that do not have that property, finding the prefix and suffix that capture precisely a given initial state and goal does not seem so simple. (I'm not even sure if finding those prefix/suffix sequences, in a domain-independent way, is actually simple even for domains in which we know they must exist.)

Reviewer_n9heOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a methods to learn action model from incomplete and possible noisy action traces. The noise includes changes in the action name, its parameter types and assigned objects. The input include candidate assignments of objects to actions and their parameters, each associated with some weight that reflects the confidence about this assignment. The authors propose to map this input to an HMM, which allows identifying the most likely assignment, which is then passed to LOCM2 for learning. The paper also reports a nice experimental evaluation on standard planning benchmarks, and comparison with reasonable baselines.

Critical strengths
  • The topic is of interest
  • Innovative and effective approach of mapping to HMM
  • Good experimental results
Critical weaknesses

I have been working in this exact area, so I was looking forward to reading the paper. Unfortunately, I had a hard time understanding the algorithm, to the point where I recommend rejecting the paper in its current form. I really hope the authors will work hard on making the paper more accessible and resubmit it, because it has a lot going for it: a nice idea, a good evaluation process, and good results.

Things that would help a reader better understand:

  1. Background on HMM for completeness.
  2. Some pseudo code or at least a step-by-step explanation of how it works. Fig. 1 helps in that, but I didn't see text explaining it and the image itself is not sufficient, at least to me, to understand.

Also, I think there may be a mistake in Equation 2. \alpha t is defined as the sum of candidates not containing the event a.k assigned to type c, so this means it assumes all assignments to all other types. But then the summation is over all types, so it seems to me there is some over-counting: an assignment that is not setting a.k to c but to c', will be counted in \alpha_t and when the summation will get to c'. Am I wrong?

Detailed issues:

Page 2. What is a "one-indexed argument position"? The following sentence "Projecting a deterministic trace ..." is also not clear to me.

Page 3.

  • What is our "paper-guided adapter" or a "Top-1 traces"?
  • What does it mean that "Evidence is retained at every original step"?
  • "An alternative type contributes to that type's event evidence, not to \alpha_t" - not clear to me what does this mean.
  • I think there may be a mistake in Equation 2. \alpha t is defined as the sum of candidates not containing the event a.k assigned to type c, so this means it assumes all assignments to all other types. But then the summation is over all types, so it seems to me there is some over-counting: an assignment that is not setting a.k to c but to c', will be counted in \alpha_t and when the summation will get to c'. Am I wrong?

Page 4.

  • What is "first-event distribution" and "next-even probabilities"?
  • "Here no single label is observed..." - what do you mean by "Here"? what is this describing?
  • I also do not understand what is the "latent type" that is mentioned.
  • "emitting e updates s to e" - maybe a mistake?

Page 5.

  • What is a "frozen benchmark"? what makes it frozen?
Questions for authors

I believe the approach boils down to predicting the most probably traces with the HMM and then running LOCM2. Is this true?

2 comments

First, yes, your hypothesis is correct: The approach essentially uses the HMM to recover the most likely object traces, and then applies LOCM2 to those. The use of LOCM2 here is not an essential feature; any learning algorithm that takes object traces as input could be used. Because LOCM2 only uses object (and some object-pair) traces, we do not need to recover the full action traces; that could also be done, if we wanted to apply an algorithm that needs the full action traces, such as SIFT, it's just likely to be harder.

Equation 2 is for one specific object, call it "o", at one particular time step t. \alpha_t is the total weight of those candidate actions at step t that do not include object o at all; the other term sums over all types that o could have, and over all events that could occur on the object. Leaving the object out of the notation probably made it unnecessarily confusing. If we include it, we would write it as

\alpha_{o,t} = \sum_{j : o \not\in args(a_{t,j})

and

\alpha_{o,t} + \sum_c \sum_{e \in E_c} q^c_{o,t}(e) = 1

For the example candidate set Q_t on page 6, for object "a", \alpha_t equals 1/8, the total weight of those candidates that do not have "a" in their argument (in that example, only the third candidate).

Action argument numbering starts at 1. (This is the convention in all of the LOCM algorithms.) For example, the action load(p,u) induces event load.1 on p and load.2 on u. Projection keeps each object's events in order. In the sequence load(p,u), drive(u,x,y), unload(p,u), package p has events at steps 1 and 3, and a "skip" at step 2.

Top-1 and All-candidates are our two baseline methods. Both of them follow the same process of first "guessing" the true object/action trace(s) and then applying LOCM2, but they guess it in a simpler way: Top-1 simply takes the locally most likely (highest weighted) candidate action at each step, while All-candidates constructs all possible traces that have non-zero probability (multiplying out the options for each candidate set).

The first-event and next-event probabilities are the parameters of the markov process. We learn one HMM for each (object) type, but this is instantiated to one markov process for each object. The "state" of an object is the last event that happened to it. The first-event probability is the probability distribution over all events that can happen to an object of that type, representing how likely each of them is to be the first event on such an object; the next-event probability is simply the transition probability of the HMM, i.e., the probability of each possible next event condtioned on the previous one. We do assume that each object has only one type, and keeps that type throughout the plan, but the type is not necessarily observed with certainty; that's why we called it "latent".

(There are more questions in the review. We will come back to answer them later.)

Reviewer_n9heOpenReview ↗

Thanks! this helps, but I still think this paper needs a thorough revision for readability

I appreciate the authors' response, which helped clarify important issues. As I wrote in my original review, I still think this paper deserves a more major editting to be readable and I hope the authors are not discouraged and do submit this paper again after working harder on how it is presented.

Reviewer_5i4kOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Plan-HMM, an HMM-based approach to reconstruct action sequences from noisy observations. This enables learning symbolic planning models without observing the environment's states.

Critical strengths
  • The proposed method is quite simple and intuitive.
  • The formulation is strong.
  • The evaluation covers 9 widely used benchmarks.
Critical weaknesses
  • More evaluation is needed to better show the merits of the proposal.
  • It is unclear how the model would perform under realistic conditions.
Questions for authors
  • Although the authors acknowledge this, it is unclear how their proposal would perform in realistic (less controlled) cases. The paper's assumptions greatly simplify the problem. The assumption that the correct action is always among the candidates is particularly restrictive.
  • The experimental evaluation is insufficient. Again, the authors acknowledge this, but they do not analyse the individual contributions of the system components.
  • Why did you only implement argument-name substitution in LC_M's repair procedure?
  • The method models action sequences independently for each object and subsequently aligns compatible events. How does this representation handle actions that indirectly affect objects not explicitly included in their arguments, or actions that modify shared/global environment state? It is unclear whether the proposed object-centric reconstruction can recover such dependencies.
  • Since ground-truth symbolic planning domains are available, why do the authors not directly evaluate the accuracy of the recovered action schemas, including preconditions and effects?
  • The baseline comparison is unsatisfactory. Only one baseline is based on previously published work.
  • How would the proposed method compare against state-of-the-art LLMs under the same information constraints? Specifically, given the same weighted action candidates and action ordering, without access to ground-truth actions or intermediate states, could an LLM achieve comparable reconstruction quality? Evaluating an LLM-based reconstruction baseline would help establish the advantages of the proposed approach.
1 comment

There are many questions and comments here to respond to, so we will start with some, and come back to the others later.

The assumption that the correct action is among the candidates is not really restrictive. As long as the number of candidate instantiated actions is finite, we can always assigns a small non-zero probability to each one. However, the construction of our test cases uses only a small pool of 4 candidates, so the test case construction is restricted. We have conducted an additional experiment, in which the correct action was removed from the candidate set in a fraction of plan steps, ranging from 5% to 30% of them. At 30% omissions, the reduction in the behavioural F1 score is about 15%, compared to the values in the paper, which are without omissions.

The OCM formalism is (approximately) expressively equivalent to STRIPS. Both of these formalisms have the property that actions can only affect facts about objects that appear in the action arguments. For actions to express "indirect effect" requires a more expressive formalism, including, for example, quantified conditional effects, or object-valued fluents. For the STRIPS, and OCM, formalisms, it can be shown that domains in these formalisms are learnable from observation of action traces only; for domains with conditional effects this does not hold (meaning there can not exist any algorithm that is guaranteed to correctly learn such domains from observation of action traces only; this follows from the non-learnability of regular languages in this setting). Of course, the (positive) theoretical results only apply to the case without noise.

LC_M is, as far as we know, the only previously published approach to learning from action traces only, with uncertain action identities and argument identities. That is why it is the only previously published baseline; there does not exist any other. The LC_M method is fairly complex, and only described in the one paper, which is why we had only a partial implementation of it at the time of submission. It has since been extended to a more or less full re-implementation of LC_M, as we understand it based on only the description in the published paper (since we have nothing else to go on).

We cannot evaluate the accuracy of the learned action schemas by comparing them against the original (true) domain, because the learned domains were learned from action traces only. That is, the learning algorithm has "invented" its own predicate representation of the domain, which does not need to have any discernible relation to the representation used in the original domain. (Just as two different finite automata, with completely different state sets and transitions, can accept the same language, two different planning domains with completely different sets of predicates and action preconditions/effects can accept the same action traces.) In certain cases, it is possible to align the state representation of the learned domain with that of the original domain, but there is no known practical method to do this that functions uniformly across domains. (Some domains that have particular properties, for example that the state of each object is completely determined by the last action that was done on it, which makes this mapping relatively easy. In the general case, we don't think there is any other method than constructing the explicit state state of both domains, and finding isomorphisms between them.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.