- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses a clear and practical research problem: converting lessons from experience into constraints at the action execution interface can prevent language agents from repeatedly taking actions they already know should be avoided. The paired experiments and the decomposition of action masking and rule messages provide valuable evidence. However, the results most strongly support the narrower conclusion that execution-time blocking and feedback help under specific rules and task subsets. The evidence and experimental protocol descriptions remain insufficient to establish reliable offline verification, general-purpose automatic rule discovery, or performance gains attributable to self-evolution across all four environments.
Critical strengths
-
The method is well designed. MetaAct uses failed and successful trajectories as contrastive evidence, prompting an LLM to construct typed GEAR specifications from six manually designed primitives and predefined goal extractors. A fixed compiler converts these specifications into deterministic guards, which reject rule-violating actions before execution and return trigger messages to the planner.
-
The framework is comprehensive, and the evaluation covers a diverse set of benchmarks.
Critical weaknesses
W1. Replay verification does not establish effectiveness after intervention. It measures blocking on historical actions, but cannot determine whether the resulting alternative trajectory improves task completion or causes harmful restrictions.
W2. The main results do not sufficiently isolate GEAR’s contribution. Same-scaffold guard-on/off comparisons and simpler rule-selection baselines are needed to attribute gains to the proposed constraints and synthesis procedure.
W3. Online self-evolution is demonstrated only in a limited setting. The system quickly acquires one specification and then remains unchanged, providing little evidence of sustained refinement, accumulation, or retirement.
W4. Repetitive and formulaic writing reduces readability. The experimental discussion often restates table entries without highlighting the core findings, relevant comparisons, and limitations.
Questions for authors
-
How are harmful and catastrophic constraints labeled in Figure 5, and how is the candidate pool constructed? Is the held-out split defined by trajectory or task type? Are the labels independent of the eff/ok proxies, and do these metrics predict actual task-performance degradation after deployment?
-
Can the authors provide guard-on/off comparisons using the same MetaAct scaffold, particularly on ALFWorld and TextCraft, distinguishing existing fixed guards from newly synthesized ones? How do single-round synthesis or simple rule selection perform with the same evidence?
-
Please clarify the configurations, deployed rule specifications, seeds, and evaluation subsets used in Tables 1/2/3/8. Why does the SciWorld 70%→0% violation result use 20 episodes, whereas Figure 4 uses 16? How do the 17 use-thermometer constraints relate to the four shown in Figure 9?
-
What is the full-stream paired score difference between online MetaAct and the no-learning/textual baselines? Do the gains persist under different episode orders? Please clarify whether the current evidence is limited to one specification accepted early in the stream.