acceptodds
Withdrawn from ICLR 2027

TeleAgent-Logs: Benchmarking Evidence-Grounded Agents for Interactive 5G Troubleshooting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model agents offer a promising way to automate telecommunications network operations, improve efficiency, and deliver better services to end users, yet realistic benchmarks for network troubleshooting remain scarce. In telecommunication networks, effective diagnosis requires more than predicting a root cause: an engineer must identify where and when degradation occurs, analyze heterogeneous network logs, gather supporting evidence, and determine actionable remediation. We introduce TeleAgent-Logs, an agentic benchmark that reformulates network fault diagnosis and remediation as an interactive decision-making problem. Each task is based on a network drive test and exposes five complementary data sources: user-plane data, signaling-plane data, measurement-report data, network-configuration data, and traffic data—through a fixed set of diagnostic tools for both data access and radio and network analysis. Agents must decide which observations to inspect, which tools to invoke, and how to combine their results into structured fault hypotheses covering four capabilities: localization, fault identification, evidence grounding, and remediation. We consider two evaluation settings that require either one or all evidence-supported fault hypotheses, and introduce taxonomy-aware matching together with component-wise evaluation of each troubleshooting stage. By shifting evaluation from static answer selection to sequential investigation, analysis, and corrective actions, the benchmark measures the full troubleshooting loop and provides a challenging testbed for tool-using agents in 5G network operations. We evaluate four models across five harnesses and observe substantial performance variation across model–harness combinations. Overall, agents are better at locating degradations and identifying faults than they ground diagnoses or propose correct remediation. Finally, our comparison across fault categories highlight systematic confusion between related radio faults such as antenna tilt, transmission power, and mobility configuration.

Primary area:
datasets and benchmarks
Keywords:
benchmarks, datasets, agents, telecommunication networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aS4gOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TeleAgent-Logs, a benchmark for assessing whether agents based on large language models (LLMs) can resolve 5G network issues by actively examining network data rather than merely determining the root cause from a fully observable prompt. Its main contribution is the assessment of four key capabilities: localization, fault detection, evidence grounding, and remediation. The author tested tasks using four different language models and five different agent frameworks, discovering that performance varies significantly across model-harness combinations, with evidence grounding and remediation identified as persistent weaknesses. This paper tackles an important research challenge. Nevertheless, the benchmark relies on synthetic logs and fault scenarios within an anonymized real-world topology, making it crucial to question how well these results translate to actual operational 5G networks.

Critical strengths
  • The paper presents TeleAgent-Logs, a benchmark featuring 500 tasks designed to assess AI agents in troubleshooting 5G networks. It evaluates the agents' ability to detect faults, determine their origins, supply evidence, and propose solutions. This approach enhances the evaluation, making it more comprehensive and beneficial for future research.

  • The benchmark accounts for tasks that may have multiple valid faults, leading to a fairer assessment. The authors perform five tests for each task, collect feedback from three telecom engineers, and examine the reasons behind agent failures. These methods enhance the value of the findings and help in understanding and evaluating AI troubleshooting agents.

Critical weaknesses
  • The paper states that interactive troubleshooting is different from simply providing an AI with all the network data, but it never verifies this assertion. For instance, the worked example in Appendix A.2 includes only 12 user-plane rows, 4 configuration rows, 10 signaling events, 4 traffic rows, and 8 measurement reports. The authors do not indicate whether an agent, when given all this information, would do better or worse. Moreover, some tools offer direct indications of the fault, implying that the benchmark might be more about testing tool selection than actual diagnostic reasoning. As the various fault scenarios and the logs are synthetic, it is also not clear whether the agents would perform in a similar way when using real 5G network data. These points create doubt about the paper's primary claim that interactive troubleshooting offers a meaningful evaluation of AI agents.

  • Agents do fail due to timeouts, context limitations, or restrictions on the use of tools, not as a result of inadequate troubleshooting. The paper notes that certain combinations of model and tool lead to high failure rates, and the outcomes from the successful runs show only small differences between the various tools. The evidence score might also give less weight to correct answers that rely on different but valid evidence, while the remediation score depends on the accurate identification of the fault first. Because of these problems, it becomes difficult to judge the agent's real capabilities and to decide whether the paper's primary conclusions are reliable.

Questions for authors
  • Was the performance comparison of agents done using interactive tools compared with that of agents who were given all the network data directly? And if not, how does the paper show that interactive troubleshooting provides a meaningful advantage over standard diagnosis?

  • How can we distinguish between failures due to technical limitations and genuine diagnostic mistakes? Table 13 shows high failure rates for certain model harness combinations, while Table 14 displays considerably smaller performance differences when only the validated runs are taken into account. Might this affect the paper's conclusions regarding which agents perform best?

  • Remediation is zero in the case of an incorrect identification, so what is remediation accuracy based on when identification is correct? This would allow us to tell the difference between a real remediation gap and the identification error, which is then counted again.

  • What happens when the evidence grounding metric is applied to correct but different evidence? In Appendix A.2, the example is given an evidence score of just 0.49 even though the fault has been correctly identified and remedied. Is valid alternative evidence being unfairly penalized?

  • As shown in Table 2, the majority of the responses given in Complete-mode include only one hypothesis even when dealing with tasks that involve multiple faults. Does the prompt state or suggest that more than one fault could be present? Is this due to a limitation of the capability or rather a problem with the instruction and the stopping criterion?

Reviewer_RbKUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a useful benchmark for interactive 5G troubleshooting, but I have concerns about its synthetic data generation, the validity of its evaluation metrics, the limited novelty relative to existing benchmarks, and whether it meaningfully evaluates the ability of agents to perform real network troubleshooting.

Critical strengths
  1. Evaluating agents on interactive network troubleshooting is important. The benchmark goes beyond static fault classification by requiring agents to inspect telemetry, identify root causes, collect evidence, and recommend remediation.

  2. Separately evaluating localization, fault identification, evidence grounding, and remediation helps identify where agents fail.

  3. The paper evaluates four general-purpose models across five agent harnesses, with five attempts per task and mode.

Critical weaknesses
  1. Section 1 explicitly asks how interactive information acquisition affects troubleshooting performance compared with fully observed diagnosis. However, the experiments only compare different model–harness combinations in the interactive environment. There is no fully observed baseline, no controlled comparison with static diagnosis, and no ablation of tool access.

  2. The benchmark does not execute the proposed remediation or verify that a network fault is resolved. Instead, remediation is assessed through similarity to reference correction objects. This leaves open whether a highly scored correction would actually improve network performance or even be safe to deploy. Consequently, the claimed evaluation of the complete troubleshooting loop is stronger than what the experiments demonstrate.

  3. The overall success score is an average of four component scores, with fault identification contributing only 25%. Even within identification, predicting the correct broad category receives 75% credit regardless of the subcategory. Moreover, remediation scoring assigns 50% weight to selecting the correct parameter and only 10% to the target value. Thus, an agent can receive substantial success credit even when the precise root cause or corrective action is wrong. The paper should report a strict end-to-end success rate in addition to partial-credit scores.

  4. The experiments evaluate various general-purpose LLMs and agent harnesses, but do not include rule-based troubleshooting, conventional radio diagnostics, scripted information-gathering policies, or a non-agentic LLM receiving all observations. Without these comparisons, it is unclear whether the benchmark requires sophisticated agentic reasoning or whether a simpler domain-specific pipeline could perform comparably. This also limits the usefulness of the results as a measure of progress in network troubleshooting.

Questions for authors
  1. is there no fully observed diagnosis baseline? given that comparing fully observed and interactive troubleshooting is explicitly identified as a primary research question?

  2. what fundamentally new evaluation capability does TeleAgent-Logs offer relative to CTBench, NIKA and NetArena? can existing benchmarks or their interfaces be extended to evaluate the same capabilities?

  3. have the authors tested whether the reference remediation actions resolve the injected faults in a simulator? how can they establish that matching a reference correction constitutes successful remediation?

  4. are the multiple reference hypotheses actual coexisting faults or alternative explanations of the same symptoms? how were these alternatives established and validated?

Reviewer_cGCqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents TeleAgent-Logs, a benchmark for evaluating LLM agents on interactive 5G network troubleshooting. The benchmark uses synthetic drive-test scenarios grounded in real network topology and fault distributions, requiring agents to investigate heterogeneous network data through diagnostic tools. It evaluates agents across the end-to-end troubleshooting process, including fault localization, identification, evidence grounding, and remediation. Experiments across multiple models and agent harnesses show substantial performance variation, with evidence grounding and remediation remaining challenging for the evaluated open-weight models.

Critical strengths

This paper is a step forward from prior Q/A-style telecom benchmarks. It treats troubleshooting as an interactive procedure and hides the data behind a tool interface, so the agent has to decide what to look at instead of being handed everything (though, as noted below, the interaction is only mocked). Requiring evidence with each diagnosis is a good idea compared with plain answer selection, even if the way it is implemented here has problems. The results for the telecom-specialized model (only in the appendix) are also interesting: it does worse than its base model mainly because it fails to follow the agent protocol, which suggests such models are not yet ready for realistic operational workflows.

Critical weaknesses

I don't think this paper sufficiently establishes the validity and realism of the benchmark. Many design choices lack clear motivation or ablation studies, and the claimed interactivity is somewhat overstated. The experiments also provide limited insight into why agents succeed or fail. These issues make it difficult to judge how meaningful the benchmark results are and how well they reflect real-world troubleshooting capabilities.

Detailed comments are as follows:

  1. The setting is not really interactive in the sense of actual network operations. The data are fixed and the agents' actions do not change the environment. Without the tool interface, this is essentially a log-analysis task. Research Question 1 (interactive vs. fully observed diagnosis) is also never directly tested.

  2. The paper does not sufficiently explain how the synthetic data are generated. What platform or methodology is used to produce the telemetry, and how are faults injected? The network scale and node heterogeneity are also unclear.

  3. The expert review is too vague. I'd like to know the criteria, what was and wasn't checked, how many tasks were changed, and how much the experts agreed. Ideally an example of a declined task or revised task would be helpful. This would also help the community to understand how a telecom benchmark should be designed.

  4. Beyond the expert review, there is no analysis showing whether each injected fault can be reliably identified from the available data and tools. Penalizing "extra" predictions may also be problematic, since some could be valid explanations or side effects of the injected faults. Similarly, the evidence score relies on overlap with reference evidence, but other valid supporting evidence may exist.

  5. The execution setup is unclear. Are these data analysis tools something common in the real-world troubleshooting tasks? How are the tools exposed (MCP?), can agents use anything else? What if the agent were given shell access to build its own tools? And do Claude Code, Codex or Hermes have shell access? If they do, they might be able to read the answers directly from the source code.

  6. Some key design choices are not sufficiently motivated. For example, the scoring weights and the category-subcategory hierarchy (including the 3:1 weighting and categories with only one subcategory) seem somewhat arbitrary. I am also not convinced of the value of the Discovery/Complete distinction. In particular, Discovery mode seems to mainly produce higher scores without providing much additional insight.

  7. The validator is also questionable. I understand its purpose is to make evaluation more convenient, but it checks not only the output format but also whether the cited evidence exists in the data, and allows up to five submissions. In Appendix A.2, the agent corrects an invalid evidence citation based on this feedback. This means the final scores may reflect not only the agent's diagnostic capability but also its ability to use validator feedback. I would like to see the scores on the first submission, before any such feedback.

  8. The result analysis is rather shallow, mostly just repeating the numbers. I'd like to see deeper analysis of why agents succeed or fail. For example, Figure 2 shows the confusion matrix for fault identification, but the paper does not explain why these confusions happen, or why antenna-tilt faults are misclassified most often. I’d also suggest summarizing the common failure modes and why they happen, helping the community and model developers know what they should improve.

  9. The paper leaves many useful details to the appendix, while §3.1 formalizes fairly standard material. The telecom-specialized model results (A.7) are among the more interesting ones but are only presented in the appendix.

Questions for authors
  1. How are the synthetic data generated, and how are injected faults reflected in the telemetry? How was the data validity checked, and what criteria did the experts use to review the tasks?

  2. How are the tools exposed, and do agents have shell or file access? Did you try any baseline without the custom tools, such as providing the data/file paths directly to the agents?

  3. Why these tools, this taxonomy and these scoring weights, and did you try alternatives?

  4. How do the authors ensure that valid supporting evidence is not unfairly penalized simply because it differs from the reference evidence? How much do the results change if only the first submission is considered, without validator feedback?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.