- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper treats recurrent neural networks as weighted directed graphs and characterizes multi-hop input-output information propagation through weighted sums of matrix powers. The sum over truncated paths is used to generalize spatial structures, while the number of hops links the propagation path to the time at which the input appears. Experiments covered modular arithmetic tasks, alternating signal-and-noise inputs, target-delay-selection maze tasks, and time-series XOR. The authors further propose R-RNN, which imposes a penalty on the aggregated input-output paths rather than penalizing individual weights alone. The paper reports that multi-hop influence matrices reflect task-relevant structures more clearly than the original recurrent weights. The maze experiments provide further validation of this application. I interpret this method as a strategy for generalizing and regularizing information propagation based on connectivity structures, which is not equivalent to the complete reconstruction of nonlinear dynamics dependent on specific inputs.
Critical strengths
-
An easily interpretable link between graph paths and time calculations was established. Controlled tasks provide a clear information-propagation structure, facilitating comparison with multi-hop metrics. The decomposition by hop count is particularly valuable because it links the network’s structural properties to the temporal relationship between inputs and final outputs. Sections 4.3 and 5.1 also distinguish between information-propagation capacity and sensitivity to specific inputs.
-
The analysis further yields implementable training methods. R-RNN links the proposed interpretations to practical regularization objectives. Appendix A.4 reports model selection based on the validation set as well as results from ten paired runs, supporting modest but consistent improvements across two modular tasks. This extends the contribution beyond mere visualization analysis, although the evidence remains limited to the shallow recurrent networks and controlled tasks studied in this paper.
Critical weaknesses
-
Some statements regarding sparsity go beyond the conclusions supported by the defined metric. Section 3.3 describes the reduction in input-output interactions as ensuring that communication paths are truly pruned. However, since the metric first sums both positive and negative contributions and then takes the absolute value, a decrease in the metric may also result from mutual offsetting. While this does not negate the reported loss improvement, it does affect the interpretation of the regularization effect.
-
The relationship between the analysis metric and the training objective needs to be clarified. Section 3.2 uses normalized weights and a sum truncated by task duration, while Section 3.3 instead uses unnormalized weights for regularization; Appendix A.1.2 further specifies that k = 10 is used for maze regularization. These settings may be reasonable, but providing explicit training formulas and a concise table of per-task settings would facilitate implementation.
Questions for authors
- Existing Implementation (Request for Clarification): Please write the finite sum formula used for the actual training penalty term, specify whether alpha is retained, and clarify whether the maze setting k = 10 refers to the maximum number of jumps or simply the tenth jump.
- Meaning of sparsity (request for clarification of wording): Does sparsity here refer to a small net effect of inputs and outputs after aggregating signed contributions? If so, please acknowledge the possibility of positive and negative offsetting in the discussion of “truly pruned paths” in Section 3.3, and ensure that the conclusions for nonlinear tasks are consistent with the existing explanation of information transmission capacity in Section 5.1.
Dear Reviewers,
Please take a moment to review the authors’ rebuttal, assess whether your concerns have been adequately addressed, and actively participate in the discussion. Please update your assessment if needed. Thank you for your time and effort!