- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- The author summarizes/defines a problem existing for DLM agents (task-blind corruption).
- The author proposes a solution to help with this problem in agent tasks, and the results are good.
Critical strengths
-
I like the analysis of the problem. The author offers a clear, testable explanation for a known failure. This paper offers a concrete cause: confidence-first decoding copies the failed action from the history before the open “what now?” decision is made.
-
The proposed idea is simple and effective.
Critical weaknesses
-
Simple baseline alternatives are missing (the gain can be due to an overly simplified baseline): random pick after removing duplicates, majority vote / self-consistency, simply banning the action that just failed, etc.
-
The problem setting (background) should be expanded. A lot of content in section 1 should be in section 2. The reading of paragraph 1 is painful. Not natural at all. A broken cross-reference. The Intro says “Section 5 develops the mechanism”, but Section 5 is Related Work, and the mechanism gets only a few sentences there. In 4.2, the sentence “SR improves on ALFWorld, ScienceWorld, and BabyAI…” appears twice in a row.
-
Temperature or diversity effects. Retries depend strongly on temperature, yet only two temperatures are tested. A higher-temperature One Pass might close much of the gap. Also, given the low success rate of the task, the gain can be hugely contributed to diversity caused by randomness.
Questions for authors
-
Dream models? It seems only the LLaDA family is utilized, which is a bit unnatural to me (not critical, though).
-
Inherently, I feel like this makes a natural connection to the nature of decoding for dLMs. Has the author considered different remasking strategies? Although I understand this is a higher-level setting instead of a decoding strategy paper.
-
Refer to weaknesses 1/3.
Thank you for your helpful feedback. I plan to withdraw the submission, but before doing so, could you please clarify what you mean by an “oracle-selection metric”? Your clarification would help me improve the paper for my next submission.