- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In this paper, the authors introduce FedArena, which is a testbed or harness for studying LLM-agent-driven poisoning attacks in FL. There are three granularities: L0, which is a researcher-configured experiment; L1, where the agent revises the attack programs after every FL run; and L2, where the agent edits client messages and chooses observation timing within a run. They run experiments on MNIST and CIFAR with various attack and defense configurations. Overall, I like the idea behind the paper, but I have some concerns and questions, which I will list below.
Critical strengths
-
An LLM-driven harness for adaptive evaluation of robust aggregation is a very worthwhile and timely direction, and I appreciate the paper's ambition.
-
The L0/L1/L2 taxonomy is a useful conceptual contribution since it cleanly separates who revises the strategy and when the feedback arrives.
-
Overall, the paper is clear and easy to follow.
Critical weaknesses
- My main objection is that the evaluation in this paper does not meet the bar this kind of contribution requires.
- The paper uses MNIST and CIFAR, which are very old and basic datasets. They are not federated datasets and are synthetically partitioned to be used in this paper. For a testbed whose entire premise is federated realism, some sort of naturally partitioned dataset such as FEMNIST or Shakespeare etc., should have been used. All experiments should be run on at least one genuinely federated dataset.
- The L2 part of the paper is on IID MNIST only. Why is it only on one dataset, and the easiest one, and why is it only in the IID setting?
- It is hard to believe the success of FedArena if L2 is only evaluated under the IID assumption. That claim can only be made if it gives good performance in the non-IID setting.
- 10 clients with full partition are used. With MNIST and CIFAR, that is not a good setting. There should be at least 100 clients to see the effect of federated clients properly. You can try both settings. 100 total and 100 per round, and 100 total with 10 per round.
- 20 rounds of FL is very short to make a claim about cross-round memory and history-aware defenses.
Questions for authors
- How does the L2 experiment fare under highly heterogeneous settings? This is my main concern with this paper.
- Can FedArena support a naturally partitioned Federated dataset such as FEMNIST?
- What would happen with a larger client pool and partial participation per round?