- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper suggest a strategy of learning for reasoning tasks, which has a recursive structure in the sense that its own output is feed several time to the same model sequentially. This technique is compared to MDMs, where the goal is to approximate a continuous time Markov process in discrete state space, where on starts in a masked token and over time each of this masks is unmasked. Their main contribution is a theoretical analysis on what the hidden state representation, that can be changed for each round, can bring in terms of expressivity.
Critical strengths
The theoretical analysis of the role of the hidden representations in their recursive model is interesting and relevant.
Critical weaknesses
The presentation is really not easy to follow, in the main text there is no concise description of the model. E.g. in the main text something like is missing as well as a loss function. Also there is a statements referring to the appendix "Parity, the XOR of a binary sequence, makes this separation exact; the formal result appears in Section ..." where the content is not understandable without checking the appendix.
Further the statement of Theorem 2: "Under the fixed-depth AC0 round model, an answer-faithful MDM that follows the ready-position protocol can compute a task in T rounds only if the task admits an answer derivation chain of length at most T . If no such chain exists, neither nor succeeds in any finite number of rounds." is really hard to understand, because it uses too many vague adjectives. It would be much easier if there was a short paragraph where everything is introduced with the corresponding formulas, especially since the theoretical part is the main contribution of the paper. The positioning of the paper is a bit unclear to me, since it is called Latent-State MDM but it is more related to recursive model than masked diffusion, since it doesn't try to learn any stochastic process or denoising problem.
Questions for authors
- Without a clear description of the method including formulas and losses, the paper is hard do understand.
- Could you please also give a short introduction on what the circuit is, how it is related to the method as well as to the tasks in the experiments?
- For explaining better the bottleneck of the MDM, it might be helpful to give an elementary example with , like composing which one can do with the recursive model in two steps i.e. and , while for a MDM, where one only has one changable position, the computation has to be done directly, needing more expressivity of the network. This is because the sequence length for MDM is 1, only one function evaluation makes sense if there is no remasking. (At least thats how i understood the statement heuristically, please correct it, if its wrong)
- Is the literature cited sufficiently? E.g. for Theorem 1, how is the relation to "Saunshi et al,REASONING WITH LATENT THOUGHTS: ON THE POWER OF LOOPED TRANSFORMERS", Theorem 5.1.
It is a pity that I didn't have a chance to have a deeper communication with the authors. Therefore, given the weaknesses I mentioned above, I decided to lower my rating.