- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focuses on the field of neural algorithmic reasoning (NAR) and investigates the role of a certain hidden state in NAR. Specifically, NAR refers to using a neural network to simulate, step by step, the iterative execution of a classical algorithm. Each iteration of the classical algorithm produces a corresponding intermediate state (for example, the state of the array after each round of insertion sort). These intermediate states are called hints in NAR, and they are also what the neural network is expected to output at each step. This paper studies how the correctness of these hints affects the final network output. The authors design multiple experiments to examine the impact of intervening on hints on algorithmic reasoning capability, for example, altering the hint at a certain step (to the ground truth or to a deliberately incorrect value), and analyze the resulting changes in algorithmic reasoning accuracy to draw conclusions.
Critical strengths
Studying the role of hints in NAR is an interesting topic. It is analogous to research on chain-of-thought reasoning in LLMs. This line of work can help us better understand how the neural network operates in NAR and provide observations that can inform further improvements in NAR training.
Critical weaknesses
-
One weakness is that the paper is very difficult to read, starting from the abstract, and the overall structure is also weird. For example, in the second paragraph of the abstract, some details, such as the value of N, are unnecessary, and the sentences in Lines 21–24 are very difficult to read and are not readable. Many undefined terms appear that should not be in the abstract. The motivation at the beginning of the Introduction is relatively clear, but the later part introducing the authors' own work is too detailed, making it hard to grasp the contributions at once. Moreover, many notations, such as , are used without formal definition. Sections 3–6 correspond to different experiments, but there is a lack of a high-level picture, and there is no concluding discussion at the end.
-
Another weakness is that although the paper conducts many experiments on intervening in hints, it does not seem to yield many unexpected conclusions. Basically, these observations align with normal expectations, which makes the value of the experiments seem less important. The paper also does not further discuss how these experimental phenomena and observations would affect NAR in the future.
Questions for authors
None