- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents TeleAgent-Logs, a benchmark for assessing whether agents based on large language models (LLMs) can resolve 5G network issues by actively examining network data rather than merely determining the root cause from a fully observable prompt. Its main contribution is the assessment of four key capabilities: localization, fault detection, evidence grounding, and remediation. The author tested tasks using four different language models and five different agent frameworks, discovering that performance varies significantly across model-harness combinations, with evidence grounding and remediation identified as persistent weaknesses. This paper tackles an important research challenge. Nevertheless, the benchmark relies on synthetic logs and fault scenarios within an anonymized real-world topology, making it crucial to question how well these results translate to actual operational 5G networks.
Critical strengths
-
The paper presents TeleAgent-Logs, a benchmark featuring 500 tasks designed to assess AI agents in troubleshooting 5G networks. It evaluates the agents' ability to detect faults, determine their origins, supply evidence, and propose solutions. This approach enhances the evaluation, making it more comprehensive and beneficial for future research.
-
The benchmark accounts for tasks that may have multiple valid faults, leading to a fairer assessment. The authors perform five tests for each task, collect feedback from three telecom engineers, and examine the reasons behind agent failures. These methods enhance the value of the findings and help in understanding and evaluating AI troubleshooting agents.
Critical weaknesses
-
The paper states that interactive troubleshooting is different from simply providing an AI with all the network data, but it never verifies this assertion. For instance, the worked example in Appendix A.2 includes only 12 user-plane rows, 4 configuration rows, 10 signaling events, 4 traffic rows, and 8 measurement reports. The authors do not indicate whether an agent, when given all this information, would do better or worse. Moreover, some tools offer direct indications of the fault, implying that the benchmark might be more about testing tool selection than actual diagnostic reasoning. As the various fault scenarios and the logs are synthetic, it is also not clear whether the agents would perform in a similar way when using real 5G network data. These points create doubt about the paper's primary claim that interactive troubleshooting offers a meaningful evaluation of AI agents.
-
Agents do fail due to timeouts, context limitations, or restrictions on the use of tools, not as a result of inadequate troubleshooting. The paper notes that certain combinations of model and tool lead to high failure rates, and the outcomes from the successful runs show only small differences between the various tools. The evidence score might also give less weight to correct answers that rely on different but valid evidence, while the remediation score depends on the accurate identification of the fault first. Because of these problems, it becomes difficult to judge the agent's real capabilities and to decide whether the paper's primary conclusions are reliable.
Questions for authors
-
Was the performance comparison of agents done using interactive tools compared with that of agents who were given all the network data directly? And if not, how does the paper show that interactive troubleshooting provides a meaningful advantage over standard diagnosis?
-
How can we distinguish between failures due to technical limitations and genuine diagnostic mistakes? Table 13 shows high failure rates for certain model harness combinations, while Table 14 displays considerably smaller performance differences when only the validated runs are taken into account. Might this affect the paper's conclusions regarding which agents perform best?
-
Remediation is zero in the case of an incorrect identification, so what is remediation accuracy based on when identification is correct? This would allow us to tell the difference between a real remediation gap and the identification error, which is then counted again.
-
What happens when the evidence grounding metric is applied to correct but different evidence? In Appendix A.2, the example is given an evidence score of just 0.49 even though the fault has been correctly identified and remedied. Is valid alternative evidence being unfairly penalized?
-
As shown in Table 2, the majority of the responses given in Complete-mode include only one hypothesis even when dealing with tasks that involve multiple faults. Does the prompt state or suggest that more than one fault could be present? Is this due to a limitation of the capability or rather a problem with the instruction and the stopping criterion?