- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This work addresses the problem of learning symbolic (i.e., STRIPS) action models from noisy action-only traces. To achieve this, the authors propose plan-HMM, which reconstructs object-event traces from noisy inputs. These reconstructed traces are then given as input to LOCM2, which extracts a STRIPS action model from them. Experiment results show that plan-HMM achieves better trace reconstruction accuracy than related methods and, when coupled with LOCM2, learns more accurate STRIPS models.
The problem addressed in this paper, i.e., learning STRIPS action models from noisy action-only traces, is meaningful. However, I do not believe the paper should be accepted in its current state. The main reason is that the paper is almost entirely unintelligible. I consider myself fairly familiarized with the action model learning literature, including the case of STRIPS action model learning and, yet, I had a very hard time understanding the paper. This problem is exacerbated given the target audience of the current venue (ICLR), who may not be familiar with Automated Planning and STRIPS models. As such, one would expect the authors to carefully lay out the required definitions and terms needed to understand the paper, providing intuitions and examples. This paper, however, followed almost the entirely opposite approach. Section 2 only dedicates half a page to describing the problem setting. Here, several terms are introduced with minor explanations or none at all: "object-event traces" (what are object events?), "weighted action candidates", "steps", "alternative completions from a supplied vocabulary" (what is the "supplied vocabulary"?), "object event a.k" which "identifies action a and its one-indexed argument position k (what is meant by "one-indexed argument position"?), "projection of a deterministic trace onto an object" (what is a "projection" in this setting?), "object sorts", and many more. In fact, despite the final goal being that of learning STRIPS action models, the authors do not provide any background on the STRIPS formalism, with not even a single reference. Later in the text, Section 4 presents the proposed method (plan-HMM), building on the incomplete background of Section 2 and, once again, introducing more confusing terminology: "set of action-position labels ", "HMM" (the paper never defines what a hidden markov model is), "event-state set", "identity emissions", etc. Thankfully, this section introduces a running example (lines 158-161), but it is not enough to understand everything (also, I would move (part of) this example to Section 2).
In summary, the authors seem to assume that the reader is not only familiarized with STRIPS action model learning, but also with the particular family of LOCM methods. If that is not the case, then the paper becomes extremely dense and hard to understand. As a result, there are several key points that are somewhat lost in the text. The first is that the proposed method, plan-HMM, does not learn a STRIPS action model by itself but, rather, its goal is to remove the noise from the input action traces which are then provided to LOCM2, which will finally output the STRIPS action model. This is mentioned in several parts of the paper (such as in the last paragrahs of the introduction) but not explicitly enough. A second key point concerns the novelty of the proposed method. In lines 117-126, the paper covers the related method LC_M which "extends LOCM to missing and noisy action-only data without underlying states". From this premise, it seems that it addresses exactly the same setting as plan-HMM. Then, the next lines in the paragraph compare LC_M and plan-HMM, without clearly stating what are the advantages of plan-HMM over LC_M. Only at the end, lines 125-126 state that "Its evaluation [of LC_M] under a shared LOCM2 backend does not establish superiority over the complete published system". Such an statement is incredibly ambiguous: does "does not establish superiority" mean plan-HMM performs better or not than LC_M? Does "the complete published system" refer to plan-HMM, LOCM2 or what else? Given the extreme importance of this claim (since the novelty of plan-HMM depends on it), one would expect the authors to be more explicit. In contrast, the reader needs to wait until the experiment results of Table 2, where it seems that plan-HMM clearly outperforms LC_M. To further add to my clarity concerns, Section 5 introduces several metrics that are hard to understand. Since the whole system (including LOCM2) outputs a STRIPS action model, I would like the authors to additionally analyze the errors (e.g., missing/extra preconditions and effects) in the action models learned.
Critical strengths
- The paper addresses a problem (STRIPS action model learning from noisy action-only traces) that is both hard and meaningful.
- The proposed method (plan-HMM+LOCM2) seems to learn more accurate STRIPS action models than related approaches (such as LC_M). However, the word seems is key here: due to the overall lack of clarity in the paper, it is very hard to accurately assess the results obtained.
Critical weaknesses
- By far, the largest weakness is the extreme lack of clarity throughout the entire paper. The reader needs to be familiarized not only with existing literature on Automated Planning and STRIPS action models, but also with the LOCM family of approaches. Otherwise, the paper is extremely confusing and hard to understand.
- The advantages of the proposed method, plan-HMM, should be more explicitly stated when compared to the existing method LC_M, which addresses the same setting. Experimental results (e.g., Table 2) seem to show that plan-HMM outperforms LC_M, but this should be made more explicit.
Questions for authors
As explained above, my biggest complain concerns the lack of clarity of the paper. As such, I believe the paper should be completely rewritten and made suitable for an audience that may not be familiarized with Automated Planning, STRIPS and, even more so, LOCM. I provide below a non-exhaustive list of suggestions for clarity improvement:
- Section 2 (Problem Setting) should be greatly extended, carefully explaining the LOCM formalism and terminology.
- Before Section 2, the paper should provide background on STRIPS action models.
- I would ensure that Section 3 (Related Work) does not build on terms later introduced in the text. A simple fix is to place this section just above the conclusions (Section 6).
- As with Section 2, Section 4 (Plan-HMM) should be carefully written, so that it is easy to follow (for this, the running example and Fig.1 may help, but are not enough in their current forms) and every term is carefully defined and explained.
- Section 5 (experiments) introduces several metrics, which should be motivated better. As I understand it, you evaluate both the trace reconstruction capabilities of plan-HMM alone and, then, also the accuracy of the resulting STRIPS model when coupled with LOCM2. Also, since STRIPS action models are extracted, it would be illustrative to visualize them and analyze their errors (e.g., missing/extra preconditions and effects). In contrast, Figures 2, 5 do not add much (for the latter, what is the difference between "pure missing" and "missing + ambiguous"?).
5 comments
Thank you very much for the clarifications. I understand the approach better now.
If I understood correctly, the action model learning pipeline (sequence reconstruction + model extraction with LOCM) results in a model encoded in a set of predicates different from the ground-truth model (akin to the SIFT algorithm), so it is hard to compare it to the ground-truth model (e.g., missing/extra effects or preconditions, planning accuracy, etc.) Therefore, metrics must evaluate sequence reconstruction/prediction accuracy, which is what you do in the paper. I agree with this, and I believe your approach is still valuable even if the resulting model cannot be used for planning. Still, for future work, my advice would be to consider how to translate problems from the ground-truth encoding to that of the learned model, so that you can perform planning with it. Maybe LLMs can be used for this, although I am unsure.
Another important point is the comparison with LC_M. Given the hypothetical overlap with your system, I strongly believe you should explicitly explain the differences between your method and LC_M, both on the theoretical side and on the empirical side (by showing your method outperforms it, as you currently do). My advice would be to obtain an implementation of LC_M as close as possible to the original and, if that is not possible, to carefully explain the differences and assumptions you made when running LC_M (e.g., LC_M not being compatible with probabilistic observations, unlike your method). This could be placed in an Appendix.
With these revisions, along with improved clarity and extended background, I believe the paper would be a strong contendant for acceptance at a major conference. Unfortunately, I cannot recommend acceptance in its current state, so I will keep my score.
Thank you. We will take all this feedback into account for any future revision/submission of the paper.
Just a comment on how to translate problems from the ground truth model to the learned domain: This is possible, in certain cases, by using action sequences that define the initial state and goal. I would also say it is possible to use the learned domain for planning, but we have to think of the problem specification differently: Given a prefix, \pi_pref = a_p1,...,a_pn and a suffix \pi_suff = a_s1,...a_sm, we can ask a planner to find a "middle" action sequence, \pi_mid, such that the concatenation of \pi_pref, \pi_mid and \pi_suff is "valid" (in the sense defined above, i.e., executable in some state).
For example, let's take blocksworld: If the prefix sequence is putdown(A), pickup(C), stack(C,A), pickup(B), putdown(B), then the end state of executing that sequence is unique (assuming no other blocks than A,B,C, and that the starting state satisfies the domain invariants). Likewise, the sequence unstack(A,B), putdown(A), unstack(B,C), putdown(B), pickup(C) is only executable in a state where A is on B is on C is on the table; in other words, requiring that suffix sequence is equivalent to specifying a goal state with an A-B-C tower.
However, this example works in blocksworld only because that domain (and many others, but not all) have a particular property: roughly, that all aspects of the state of an object are fully determined by the last action that touched it. In domains that do not have that property, finding the prefix and suffix that capture precisely a given initial state and goal does not seem so simple. (I'm not even sure if finding those prefix/suffix sequences, in a domain-independent way, is actually simple even for domains in which we know they must exist.)
There are clearly too many clarity issues for us to be able to address all of them in a response, but let us discuss one, which is the results of the experiments - specifically, what it is that we measure, and why (we think) this demonstrates that the plan-HMM approach learns better models.
The F1 score (or, if we were to break it down, precision and recall) of the transition matrix reconstruction is arguably only relevant because the approach separates sequence reconstruction and model learning using a LOCM-style algorithm, which uses the consecutive transition matrix as input. This is a somewhat "relaxed" measure of how well the estimated action sequences match the true action sequences. It is "relaxed" because it does not compare the full sequences; it only compares the extent to which pairs of object transitions that are consecutive in the true action sequences are also consecutive (or not) in the reconstruction.
As noted in the response to the first review above, we cannot compare the learned domains directly to the original (true) domains, because the only vocabulary that they have in common are the action signatures. But, if we view a domain as essentially a classifier of action traces - for any action trace, the domain will either consider it valid or not valid - then we can measure the extent to which the learned and true domains agree on this. Note that a "valid" action trace here means one that is executable from some initial state (since we don't have a way to map initial states or goals between the two domains). The VAR and IRR measure are essentially the recall and precision of this comparison: a perfectly learned domain should (1) accept all action traces that are accepted by the true domain, and (2) reject all action traces that are rejected by the true domain. VAR measures the degree to which the learned domain achieves (1) and IRR the degree to which it achieves (2). It is trivial to "learn" a domain that rejects everything (and thus scores high on IRR, but zero on VAR) or that accepts everything (perfect VAR, zero IRR); there is a trade-off between the two measures. The results in table 2 (in particular, consider the "completed only" results, which are not skewed by the number of instances solved) show that plan-HMM achieves a trade-off that we argue is better than all of the baselines. Although the baselines achieve a higher VAR (in particular, all-candidates which is around 10% higher), they do so at the cost of a much bigger drop in IRR (around 30% for all-candidates and top-1, 20% for LC_M). In other words, the baseline approaches (all-candidates in particular) learn more permissive domains, which accept more action traces than they should. Plan-HMM reduces this over-acceptance, without sacrificing too much of correct acceptance (VAR).
The LC_M results reported in the paper are not the results of the LC_M system that Gregory, Lindsay and Porteous built; it is the results of our re-implementation of their system, based only on the description in their workshop paper. As mentioned above, our re-implementation at the time of submission was not complete (it implemented only argument reconstruction, and because LC_M can not use probabilistic observations, it uses the top-1 sequence as input). It has since been extended to a more or less complete re-implementation of the approach, as we understand it from the paper. But we cannot say that the results of running the original LC_M system, if that was available to us, would be exactly the same. That is what the statement in the paper is trying to say (it is, admittedly, expressed in a somewhat clumsy way).