- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how to utilize the skill when the task already has gained certain level of progress. The authors find that replaying the historical operations or skipping the already seen steps are not optimal choices. They propose to select the actual operations based on the environment and the current history, so that they could select the best operation. Expeirments are made on ALFWorld to validate their effectivess against naive baselines.
Critical strengths
- The problem is clearly stated, and concrete failure patterns are summarized as well. The paper separates the operations into several well-motivated categories and picks the optimal operation based on the situation at hand.
- The conducted experiments well separated the key factors, and the results are statistically sound.
Critical weaknesses
- The method is designed as a type of skill, which is widely applicable to all mainstream models. Yet the only model in the experiment is a llama-8B model. And larger models may raise a big question mark about the necessity of the proposed method, as they have a higher intelligence level.
- Experiments are conducted only on ALFWorld, at least one other dataset is required to validate the generality of the skill.
- No cited baselines are presented; only some naive baselines. Since the targeted problem is not so well-distinguished or is hard to come up with, comparisons with published baselines are necessary.
Questions for authors
- According to the results, the gains are mainly from the cleaning tasks, yet the explanation is not convincing. Please elaborate on the reasons.
2 comments
I thank the authors for the additional experiments and the trajectory analysis, which clarify that the gains on cleaning tasks come mainly from pausing and resuming when blocked rather than from skipping completed steps. However, the new results largely confirm my concerns. With a stronger receiver model, the method no longer improves task success, and its benefit reduces to lower execution cost. In the second environment, the gains rely on manually curated skills and domain-specific rules, are concentrated in a single task family, and are not compared against a source-order baseline, so the central scheduling claim remains untested there. Against the published baseline, the method is more efficient but shows no significant improvement in success. I will therefore keep my score. I suggest that a future version evaluate stronger receiver models, include the source-order baselines in additional environments, and explain more clearly why the problem remains important as model capability grows.
Dear Reviewer WrC1,
As the discussion period will end very soon, please check the authors' rebuttal and see if your concerns have been addressed. Thank you.
-AC