acceptodds
Under review as a conference paper at ICLR 2027

Iditarod: What Makes a Good Distributed Trace Representation?

Abstract

Distributed traces—graph-based records of individual request executions—are critical for understanding and operating production software systems across industries, supporting security, performance optimization, and debugging. However, they have received comparatively little attention in machine learning (ML), despite rapid progress in both AI-driven software engineering and graph representation learning. We address this gap with Iditarod, the first multitask benchmark for distributed trace representation learning on real-world data. Iditarod is based around a new corpus of 3.18 million traces drawn from a commercial observability platform's own telemetry, combined with a subset of the Uber trace corpus. Its four tasks evaluate two complementary aspects of representation quality: (a) preserving key properties of individual traces through two intrinsic tasks and (b) providing efficient and performant embeddings for downstream applications through the Trace Retrieval task and the multimodal Faulty Deployment detection task over traces and time series. Our evaluation of sixteen different trace embedding methods on Iditarod includes models that process traces as either serialized text, call-path collections, or graphs. We find that to match or outperform language encoders on the intrinsic tasks and Trace Retrieval, graph neural networks often need to exploit the directed acyclic structure of traces. On Faulty Deployment, meanwhile, we find that no encoder produces trace embeddings that improve upon using time series alone, while a handcrafted feature extractor does, revealing a gap in current graph representation capabilities. Iditarod's data and evaluation code are available under the Apache 2.0 License.

Primary area:
datasets and benchmarks
Keywords:
graph representation learning, observability, AIOps, distributed tracing, benchmarking, DAG, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yRMqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Iditarod is a multitask benchmark for trace representation learning on real-world data. The data used for the benchmark consists of a prior work (the Uber dataset) as well as 3.18M new traces presented by the authors, and made public. The traces were labeled to facilitate learning across 4 different problem classes (LB, EP, TR, FD). They evaluated 16 different models on this data across the problems to facilitate future research into this task.

Critical strengths
  • The contribution of a new dataset with 3.18M traces is very beneficial to the graph ML community
  • Budgeted query relaxation is a sound and efficient method to generate labels for the dataset, and the technique was validated using experts
  • The evaluation covers several categories of models, and the results show that the type of structural prior matters more than raw model expressivity on several tasks
  • Detailed experimental results, with the main aggregated results easy to read and highlighted in the main body of the work
Critical weaknesses
  • How does anonymization affect the language model (and potentially other models) vs non-anonymized data? Is it possible to run an ablation on the internal non-anonymized data to see if the LLMs’ intrinsic knowledge of service names provides any benefit?
  • It is unexplained how out of vocabulary tokens (or unseen nodes) are handled at test time. This would affect the results within-source tests since splits are disjoint by root service, and even more important for cross-source evaluations. Appendix A.3 says service identifiers are source-specific between corpora, so every service will be unseen.
  • How are the input embeddings for the graph/set encoders learned (parameter table, node2vec, something else)? The trace inputs section (F.1) should contain more detail, and if possible, be in the body of the work
Questions for authors

The above listed weaknesses are also my questions

3 comments
AuthorsOpenReview ↗

Response to the reviewer

We thank the reviewer for the positive assessment of the benchmark and the helpful questions on anonymization and input encoding. We address your comments below.

Effect of anonymization (Question 1)

Access to the original attributes can benefit retrieval but does not close the FD performance gap observed on the anonymized benchmark. We ran additional evaluations on the internal, non-anonymized version of the same Observability data (Uber remains anonymized), keeping the same tasks, splits, 5 seeds and training recipes. We evaluated Qwen3-4B, Gemma 3, and GFT [1], a pretrained graph foundation model that uses a language encoder to embed node attributes serialized as text. Gemma 3 and GFT improve significantly on TR, while all 3 encoders remain below TS-only and Trace-handcrafted on FD, with no significant difference between anonymized and non-anonymized data. Across all 3 models, EP performance decreases significantly and LB shows no significant change.

Scores are shown as anonymized → non-anonymized, averaged over 3 TR taxonomies, 4 FD windows, or 4 EP/LB source pairs. P-values use two-sided paired t-tests with significance level .05.

Task / metricQwen3-4BGemma 3GFT
TR: Avg MAP@10.120 → .126 (p = .326).117 → .140 (p = .004).090 → .112 (p < .001)
FD: Avg AUPRC.265 → .277 (p = .165).276 → .272 (p = .688).267 → .271 (p = .731)
EP: Avg Macro AUPRC.644 → .586 (p < .001).478 → .417 (p < .001).440 → .395 (p < .001)
LB: Avg AUPRC.249 → .247 (p = .137).237 → .241 (p = .278).246 → .249 (p = .170)

Anonymization also provides a useful setting for evaluating representations of trace structure and timing without relying on semantically informative names. This is relevant to environments whose naming conventions may carry less semantic information than those in our internal data.

Handling unseen values at test time (Question 2)

Thank you for noting this missing detail; we have added an explicit description of how unseen values are handled in Appendix F.1. Task-trained graph and set encoders fit a vocabulary on the training split for service, operation_name, and resource_name. A value absent from that vocabulary maps to a reserved unknown embedding. One-hot fields use an explicit unknown bin, and call-path features place unseen paths in an extra coordinate. Language encoders use the pretrained tokenizer on the serialized trace. Thus, in all cases, unseen categorical values do not cause spans to be discarded or their other attributes to be removed.

The intrinsic split keeps root services disjoint between train and test. That does not make every service in a test trace unknown: other services in the trace can already occur in the training vocabulary. On cross-source evaluation the vocabulary remains the one fitted on the training source, and the same rule applies to strings absent from it.

Learning graph and set input embeddings (Question 3)

The input embeddings of task-trained graph and set encoders are trainable lookup tables. Each training-split value of service, operation_name, and resource_name has a 32-dimensional embedding, stored as a parameter matrix and trained jointly with the encoder by the task loss. They are not node2vec features and not one vector per span. Status, span_kind, type, and language remain one-hot. We have clarified how these embeddings are learned in Section 4.2 and expanded the implementation details in Appendix F.1.

References

[1] Wang et al. GFT: Graph Foundation Model with Transferable Tree Vocabulary. NeurIPS, 2024

Dear Reviewer, Following your constructive suggestions, we ran the non-anonymized data ablation and integrated the input embedding and out-of-vocabulary details into Section 4.2 and Appendix F.1. Could you let us know if these updates fully address your questions? Thank you for your helpful feedback.

Thank you. I think the new ablation shows a strong result. Your point about this meaning that the approach would work in environments with less informative naming conventions is a good one.

My questions have been addressed.

Reviewer_qfe7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Iditarod, a multitask distributed-trace representation benchmark. Authors released 3.18 million real-world production traces, evaluated two intrinsic and two extrinsic tasks. and compared sixteen text, call-path, and graph encoders. They conclude that DAG-aware models excel at structural tasks. Handcrafted features improve early faulty-deployment detection. Paper validates retrieval relevance through human judgments.

Critical strengths
  • The paper provides substantial real-world data (production-derived trace corpus)
  • It addresses limitations of synthetic microservice benchmarks.
  • Authors provide a comprehensive evaluation with sixteen encoders across four tasks.
  • Paper reveals meaningful architectural and generalization limitations.
Critical weaknesses

-My main concern is validity of retrieval construct. Queries paraphrase structured filters. Relevance labels follow handcrafted query relaxations. These may or may not reflect real investigative needs. Human validation covers only 360 comparisons. Also, missing structured-query execution baseline limits comparisons.

  • The conclusions regarding faulty deployment detection appear somewhat overstated. The dataset contains only 169 faulty deployments, which may provide insufficient supervision for training complex learned representations. Furthermore, the ground-truth labels originate from an existing monitoring pipeline, potentially favoring time-series-based detection methods. It is difficult to determine whether the poor performance of learned trace representations reflects fundamental limitations of these representations or simply the experimental setup and limited training data.
Questions for authors
  1. How would structured-query execution baselines perform?
  2. Can human-authored queries validate relaxation-based relevance?
  3. Were faulty deployment labels independently validated?
  4. Would pretraining or additional supervision improve performance?
5 comments
AuthorsOpenReview ↗

Response to the reviewer (1/2)

We thank the reviewer for the constructive feedback and for recognizing the value of the real-world dataset and comprehensive evaluation. We address your comments below.

Structured-query execution baseline (Question 1 / Weakness 1)

We did not originally include a structured-query execution baseline because our goal is soft matching rather than identifying all exact matches from the original query. In particular, because query execution does not rank partial matches, it does not directly define a baseline measurable with nDCG@10. However, we can still compute the corresponding MAP@10 ceiling on the same B=1 test candidate pools, as reported below. Including the newly added graph foundation models, such as GFT [3] (see Question 4 below), the strongest encoders reach only 26.9% to 39.5% of the oracle, leaving substantial headroom. We added this analysis to Section 5.2 and Appendix C.7.

TaxonomyOracle MAP@10Best encoderEncoder MAP@10% of oracle
Non-structural0.5042Gemma 30.181436.0%
Structural0.5135DAGNN0.138026.9%
Trace-level0.1402GFT0.055439.5%

Retrieval relevance and human validation (Question 2 / Weakness 1)

Our structured queries are human-authored and drawn from routine platform use, reflecting the real investigative needs of software professionals(Section 3.1). LLM-assisted construction of NL evaluation queries from structured requirements also appears in retrieval benchmarks such as STaRK [1, Section 2.4] and CypherBench [2, Section 4.4]. BQR assigns relevance labels from the underlying structured queries, independently of their NL wording. We validate this ordering through human judgments: engineers see only the NL query and 2 traces, blind to our labels. The study covers all 3 query taxonomies and different relevance gaps, including adjacent levels. The observed 94.8% agreement supports BQR’s alignment with human relevance preferences beyond exact matches (Appendix C.5).

FD label provenance and independent validation (Question 3 / Weakness 2)

We do not believe that the labeling procedure favors time-series methods. Faulty deployment labels come from a deployment-specific pipeline, not a generic error-rate threshold monitor. A deployment is labeled faulty when the pipeline combines significant impact relative to the service-resource baseline, temporal correlation with the target deployment, and persistence. Because this evidence must accumulate, the labels are finalized one to three hours after deployment, whereas Iditarod-FD predicts them from only the first seconds or minutes of post-deployment observations. Although the released Iditarod-FD data were not directly human-validated, the same pipeline that produced these labels was validated internally against human annotations on 153 deployments, where 90% of its faulty labels agree with the human annotations.

AuthorsOpenReview ↗

Response to the reviewer (2/2)

Pretraining and additional supervision (Question 4 / Weakness 2)

Limited label availability reflects real deployment scenarios, making performance with little task-specific supervision an important practical goal in observability. Thus, while Iditarod has limited data for the faulty deployment task, it is enough to differentiate between methods (as demonstrated by the variation in our evaluations) while being reflective of practical constraints.

Pretrained foundation models are indeed a promising way to resolve the data scarcity issue, as they aim to reduce the need for task-specific labels. Beyond the pretrained language encoders already evaluated, we additionally tested 4 pretrained graph foundation models: GFT [3], GCC [4], OpenGraph [5], and AnyGraph [6]. We kept the pretrained encoders frozen and trained the task-specific FD heads. As shown below, no foundation model improves mean FD AUPRC over time-series-only or recovers the handcrafted gain across the 4 observation windows.

These results, which we have added to the revision, extend our finding to pretrained graph encoders under the available supervision. Since the GFMs were neither pretrained on distributed traces nor designed specifically for traces or DAGs, adapting their pretraining data and architectures to this domain remains a promising direction. Additional supervision could also help, for example by targeting patterns across sets or sequences of traces for FD. Iditarod provides a common evaluation framework to investigate these directions.

CategoryModelMean FD AUPRC
ReferenceTS-only0.36
ReferenceTrace-handcrafted0.40
LMMiniLM0.27
LMQwen3-4B0.26
LMGemma 30.28
GFMGFT0.27
GFMGCC0.30
GFMOpenGraph0.31
GFMAnyGraph0.30

References

[1] Wu et al. STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases. NeurIPS, Datasets and Benchmarks Track, 2024.

[2] Feng, Papicchio, and Rahman. CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era. ACL, 2025.

[3] Wang et al. GFT: Graph Foundation Model with Transferable Tree Vocabulary. NeurIPS, 2024

[4] Qiu et al. GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. KDD, 2020

[5] Xia et al. OpenGraph: Towards Open Graph Foundation Models. Findings of EMNLP, 2024

[6] Xia and Huang. AnyGraph: Graph Foundation Model in the Wild. Findings of ACL, 2026

Dear Reviewer, As the discussion period nears its end, we wanted to briefly follow up on our response including the structured-query oracle ceiling, the fd label validation, and the 4 Graph Foundation Models. We wanted to check whether these additions resolve your main concerns, or if any points would benefit from further clarification before the discussion closes. Thank you for your time.

Reviewer_4DEqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a new large multitask benchmark dataset based on real world trace data, based on two sources (an anonymous commercial platform and an Uber corpus). It establishes four ML tasks, two intrinsic (latency bottleneck and error pattern) and two extrinsic (trace retrieval and faulty deployment), although the last two are essentially for the anonymous commercial platform (and not for the uber data).

This novel corpus is a contribution by itself, but the authors also provide an initial evaluation of a multitude of methods on the four subtasks, ranging from methods based on text or call paths to methods that rely on the graph itself, showcasing the richness of both the data set and the methodologies that can tackle it.

The evaluation shows that no unique method clearly dominates the others for all subtasks, and a comparison with a more classical time series analysis approach can sill provide better results for faulty deployment, hinting that the traces do contain signal and that the experimented learners could not fully exploit.

Critical strengths
  • An open real large and rich multitask data set released to the community, with clear and well designed construction (e.g. for the trace retrieval, the authors introduce the budgeted query relaxation that was validated against human judgments with over 90% agreement)

  • A nice overview of preliminary results using a multitude of models, including negative results, that can inform possible users of this benchmark on how and what to tackle

Critical weaknesses
  • The experimental part is essentially presented and analyzed as "rankings", with weak statistical support (ex: some differences between encoders are almost on the range of seed variance, and other results are close to "random chance")

  • Some claims seem very interpretative (e.g. the claim that modern graph-encoders "do not seem to successfully extract" signal for the faulty deployment task is not clearly demonstrated to be separated from other possible explanations for bad results, such as low quantity of labeled data or the specific aggregation method used).

Questions for authors
  • Can you better explain how the labels were constructed for the faulty deployment (FD)? You state "come from the platform’s deployment-monitoring pipeline" using aspects such as "impact", "temporal correlation" and "persistence" but it is not clear to me which actual signals are used and which ones are from the trace date or from the APM data (which in turn may explain why an APM time series based data would perform better)

  • Can you provide some insight why the tested encoders seem to fail on FD? Can you try to separate the claim about "do not seem to successfully extract" signal to other possible explanations, as stated before on the weaknesses?

  • Do you consider that Latency Bottleneck results seems to be close to random chance on many encoders? And if so, do you feel this is because the training failed? the hardness of task itself? Can you give some insight on what "good" results should be on this task?

  • Can you state what guided the choice of the 25 handcrafted vector features? where they chosen before the other tests or tuned afterwards, and how did the results inform this tuning.

3 comments
AuthorsOpenReview ↗

Response to the reviewer (1/2)

We thank the reviewer for the positive assessment of the paper and for highlighting the value of the benchmark and multitask evaluation. We address your comments below.

Statistical support for the comparisons (Weaknesses 1 and 2)

While imperfect, average rank is a common way to summarize performance across heterogeneous tasks [1]. Furthermore, in Appendix B of the original submission we also reported an alternative aggregate assessment in the form of performance profiles [2], which aim to mitigate many of the limitations of average rank (we have now moved this into the main part of the text, in Section 5). Lastly, note that we also report absolute scores and standard deviations over 5 seeds in Section 5, with complete results in the appendix.

To clarify which performance differences are statistically significant, we conducted two-sided paired t-tests across the 5 seeds on the fixed test sets, using a significance level of .05. As shown below, these tests demonstrate that the main empirical we make are statistically significant. We have added these pairwise comparisons to the appendix and made a note about the results in the revision.

  • FD: With the default aggregation, Handcrafted’s gain over TS-only is significant at 30 s, 1 min, and 5 min (p < .004), but not at 10 min (p = .075). All 16 encoders evaluated in the original submission significantly underperform both TS-only and Trace-handcrafted on the average across 4 windows (all p < .003). Per window, differences are significant in 61/64 comparisons with TS-only (the 3 exceptions against TS-only are DAGformer at 30 s and 5 min, and Gemma 3 at 1 min) and all remain significantly below Handcrafted.

  • TR. Qwen3-4B significantly outperforms DAGNN on non-structural queries (p=.0145), while DAGNN significantly outperforms Qwen3-4B on trace-level queries (p=.0352). Their difference on structural queries is not significant (p=.856). This supports taxonomy-dependent strengths, but not a statistically distinct winner in every taxonomy.

  • EP. DAGNN significantly outperforms GATv2 on all 4 source pairs (p < .021) and Deep Sets on 3/4 pairs (p < .003).

  • LB. Many encoders score close to the class-prior baseline. Nevertheless, DAGNN significantly exceeds it on both in-source pairs (p = .0245 on Obs and p < .001 on Uber), and Graphormer on Obs -> Uber (p = .0325). Performance is therefore not uniformly at baseline level, although cross-source generalization remains challenging.

FD label construction (Question 1)

The impact, temporal correlation, and persistence criteria are those used in production to decide that a deployment was faulty. They are computed from APM error and request statistics for the deployed version. New error signatures derived from traces contribute as well.

Interpretation of FD results (Question 2 / Weakness 2)

While we agree that these results do not completely separate a failure to represent the useful trace signal from limited supervision, in our submission we did conduct a significant investigation of the aggregation procedure in Section 5.3. There we reported an ablation over Tail-exceedance, Mean + Diff, and Attention + Diff on different representations, showing that learned trace encoders do not improve on time-series-only under any of the three.

AuthorsOpenReview ↗

Response to the reviewer (2/2)

LB performance and task difficulty (Question 3)

While many encoders perform close to chance, some do not. In particular, DAGNN significantly exceeds the Majority classifier on both in-distribution test sets (Obs: p = .024, Uber: p < .001; Appendix E.5).

Regarding training failure versus task difficulty: the fact that such significant gains can be achieved by some models like DAGNN and DAGformer suggests that the task is solvable but possibly not by all methods, i.e. it is difficult but not necessarily due to a training defect. For example, during training, Uber-trained DAGNN’s validation AUPRC increases from .151 to .448, with test AUPRC reaching .246 versus .143 for Majority. LB requires combining graph-level structure with span durations, including critical paths, exclusive durations and global comparisons. Transferring this reasoning across sources adds a further challenge, despite the shared labeling rule.

Regarding what a “good” result would look like: we aim to predict LB as reliably as the strongest encoders predict the error pattern. A concrete objective for methods evaluating on Iditarod is to improve cross-source performance while retaining in-source gains.

Choice of handcrafted features (Question 4)

After our cross-task encoders showed no improvement in faulty deployment performance over a time-series-only approach, we investigated complementary trace signals with observability engineers and explored expert-informed handcrafted designs, including a span-kind variant. The retained baseline’s gain then motivated the 3-method aggregation ablation to investigate the other encoders’ weaker performance.

References

[1] Aksu et al. GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation. arXiv:2410.10393, 2024.

[2] Dolan and Moré. Benchmarking Optimization Software with Performance Profiles. Mathematical Programming, 2002

Dear Reviewer,

With the discussion phase wrapping up, we wanted to follow up on our response including the statistical support for our comparisons (reinforced with significance analyses and performance profiles in Section 5), our breakdown of Latency Bottleneck performance, and the clarifications to your other questions. Please let us know if any concerns remain. We appreciate your time and insights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.