- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper argues that existing methods evaluate whether a communication topology is good based on the final outcome of the entire communication graph, which may overlook the importance of individual communication edges. Therefore, the paper trains a framework that removes communication edges one by one from a counterfactual perspective and evaluates the importance of each edge based on the change in the outcome before and after removal.
Critical strengths
-
The storyline is clear and the problem is well defined. Evaluating a graph solely based on the final outcome can indeed overlook the importance of individual communication edges.
-
The method is intuitive and consistent with how humans might approach the problem. The counterfactual idea is easy to think of and straightforward to understand.
-
The experiments cover a broad range of settings.
Critical weaknesses
-
Removing individually redundant communication edges may reduce system stability or robustness, even if task performance is unaffected.
-
The first-order counterfactual analysis cannot capture dependencies among multiple edges and therefore cannot determine whether individually removable edges are jointly removable.
-
The edge-level credit assignment may be overly fine-grained, as the importance of communication can arise from interactions among groups of edges rather than individual edges.
-
Because LLM outputs are stochastic, the observed performance change after removing an edge may partly reflect sampling variance rather than the causal effect of that edge.
Questions for authors
Thank you for your paper. I have a few questions about it.
-
In information theory, redundant information may help maintain system stability. If redundant communication is removed simply based on whether it improves the final outcome, could this affect the stability or robustness of the system?
-
Suppose you find that removing A, B, or C individually does not affect the result. However, it could be that at least two of A, B, and C are needed, or that just one of them is enough. How do you distinguish between these cases? Do you need to enumerate different combinations? If we move from first-order counterfactuals to higher-order counterfactuals, where multiple edges are removed at the same time, the computational cost will increase significantly. Is the improvement worth such a large cost? The 97.26% result in Figure 1 can only show that these edges are individually removable, but not that they are jointly removable.
-
The paper tries to solve the problem that graph-level credit assignment is too coarse-grained, but does it go too far in the other direction? Sometimes what matters is not whether A or B is important individually, but whether A and B are important together.
-
Since LLM outputs themselves have variance, is it possible that the change observed after removing an edge in the counterfactual setting is simply caused by resampling variation rather than the effect of removing that edge?