- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a new dataset: Town-benchmark. They demonstrate on this benchmark that augmenting an LLM agent with predefined rules for an environment can significantly improve the agent's performance and reduce rule violation rates. To efficiently use the rules, they propose "MinFix", which finds the shortest action sequences that fix the missing predicates. Additional experiments are performed on a subset of ScienceWorld.
Critical strengths
- The experiments show that the method achieves a much better result comparing to the baseline.
- The idea is simple and clear.
Critical weaknesses
- The design of the method is not fully supported by the experiments. For example, since it's possible to find the shortest action sequences to fix the missing predicates, why not use a rule-based planner from the beginning? (That is, why do we need an agent?) Another example is, the paper uses a neural network to filter the rules to consider, but it should be very cheap to simply try all the rules to see whether they apply or not.
- I think the impact of this paper will be limited by the fact that the rules need to be predefined instead of learned by the agent.
Questions for authors
- See weakness 1.
- Could the authors clarify the paper's relationship to learned symbolic and neuro-symbolic world models, such as WorldCoder (Tang et al., 2024) and NeSyS (Zhao et al., 2026)?
3 comments
Thanks for the author's response. For W1, I'm still a little confused. If you do not assume the domain file, how can you know what preconditions are missing for a given action? How can you plan for fixing the missing preconditions without these domain knowledge?
Our phrasing caused the confusion, and it is worth correcting: we do assume a model. What we do not assume is a planning domain.
The model has two parts. For each registered action the registry carries a predicate that decides legality and names the conjunct that fails, and these rules are drafted by the LLM from the environment specification and registered only after being checked against the environment's own validity checks. ScienceWorld makes this concrete, since it exposes 61 preconditions as explicit validity checks and 38 of them can be registered as predicates. A transition function supplies the successor state the search needs to evaluate a candidate bridge.
What a planner needs in addition is the task goal written in its language, together with a typed domain that covers every action and object. In our environments the goal arrives as free text, so that input does not exist. NSAG does not need it, because the repair target is not the goal but the preconditions of the action the model has just proposed. This is also why the search stays local: depth 5 and 2,000 expansions close 96.8% of the 312 blocked steps, whereas a planner would have to search the whole task.
Response to mZT9: why an agent, and how NSAG relates to learned world models
We thank the reviewer for the two direct questions and for the pointer to the world-model literature.
W1. Why not a rule-based planner.
A planner needs a complete action model, a goal written in its language and a state estimate, and in the environments we target none of the three is available. The goal arrives as free text and the constraints live in documents written for people, so there is no domain file for the planner to take as input. Turning that instruction into a concrete action is what the LLM does, and judging the resulting action is what the symbolic layer does.
That file is the expensive part, since every action needs its preconditions and effects, every object a type, every fluent a declaration and the goal a translation, and where the rules live in documents it does not exist at all. ScienceWorld exposes 61 preconditions and only 38 of them are registrable, so the rest would have to be invented before a planner could use them.
Because the action is already given, the repair only has to close a precondition gap at that step, and since the search is bounded by the blocked action rather than by the horizon, a depth limit of 5 and 2,000 expansions find a bridge in 96.8% of the 312 blocked steps.
Q. WorldCoder and NeSyS.
The two systems learn or train a world model, whereas NSAG assumes the model and governs the step, so the lines compose rather than compete, because a learned program is the kind of object the repair search takes as input and a model trained against rules proposes fewer violations for the guard to catch.