- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Iditarod is a multitask benchmark for trace representation learning on real-world data. The data used for the benchmark consists of a prior work (the Uber dataset) as well as 3.18M new traces presented by the authors, and made public. The traces were labeled to facilitate learning across 4 different problem classes (LB, EP, TR, FD). They evaluated 16 different models on this data across the problems to facilitate future research into this task.
Critical strengths
- The contribution of a new dataset with 3.18M traces is very beneficial to the graph ML community
- Budgeted query relaxation is a sound and efficient method to generate labels for the dataset, and the technique was validated using experts
- The evaluation covers several categories of models, and the results show that the type of structural prior matters more than raw model expressivity on several tasks
- Detailed experimental results, with the main aggregated results easy to read and highlighted in the main body of the work
Critical weaknesses
- How does anonymization affect the language model (and potentially other models) vs non-anonymized data? Is it possible to run an ablation on the internal non-anonymized data to see if the LLMs’ intrinsic knowledge of service names provides any benefit?
- It is unexplained how out of vocabulary tokens (or unseen nodes) are handled at test time. This would affect the results within-source tests since splits are disjoint by root service, and even more important for cross-source evaluations. Appendix A.3 says service identifiers are source-specific between corpora, so every service will be unseen.
- How are the input embeddings for the graph/set encoders learned (parameter table, node2vec, something else)? The trace inputs section (F.1) should contain more detail, and if possible, be in the body of the work
Questions for authors
The above listed weaknesses are also my questions
3 comments
Dear Reviewer, Following your constructive suggestions, we ran the non-anonymized data ablation and integrated the input embedding and out-of-vocabulary details into Section 4.2 and Appendix F.1. Could you let us know if these updates fully address your questions? Thank you for your helpful feedback.
Thank you. I think the new ablation shows a strong result. Your point about this meaning that the approach would work in environments with less informative naming conventions is a good one.
My questions have been addressed.
Response to the reviewer
We thank the reviewer for the positive assessment of the benchmark and the helpful questions on anonymization and input encoding. We address your comments below.
Effect of anonymization (Question 1)
Access to the original attributes can benefit retrieval but does not close the FD performance gap observed on the anonymized benchmark. We ran additional evaluations on the internal, non-anonymized version of the same Observability data (Uber remains anonymized), keeping the same tasks, splits, 5 seeds and training recipes. We evaluated Qwen3-4B, Gemma 3, and GFT [1], a pretrained graph foundation model that uses a language encoder to embed node attributes serialized as text. Gemma 3 and GFT improve significantly on TR, while all 3 encoders remain below TS-only and Trace-handcrafted on FD, with no significant difference between anonymized and non-anonymized data. Across all 3 models, EP performance decreases significantly and LB shows no significant change.
Scores are shown as anonymized → non-anonymized, averaged over 3 TR taxonomies, 4 FD windows, or 4 EP/LB source pairs. P-values use two-sided paired t-tests with significance level .05.
Anonymization also provides a useful setting for evaluating representations of trace structure and timing without relying on semantically informative names. This is relevant to environments whose naming conventions may carry less semantic information than those in our internal data.
Handling unseen values at test time (Question 2)
Thank you for noting this missing detail; we have added an explicit description of how unseen values are handled in Appendix F.1. Task-trained graph and set encoders fit a vocabulary on the training split for service, operation_name, and resource_name. A value absent from that vocabulary maps to a reserved unknown embedding. One-hot fields use an explicit unknown bin, and call-path features place unseen paths in an extra coordinate. Language encoders use the pretrained tokenizer on the serialized trace. Thus, in all cases, unseen categorical values do not cause spans to be discarded or their other attributes to be removed.
The intrinsic split keeps root services disjoint between train and test. That does not make every service in a test trace unknown: other services in the trace can already occur in the training vocabulary. On cross-source evaluation the vocabulary remains the one fitted on the training source, and the same rule applies to strings absent from it.
Learning graph and set input embeddings (Question 3)
The input embeddings of task-trained graph and set encoders are trainable lookup tables. Each training-split value of service, operation_name, and resource_name has a 32-dimensional embedding, stored as a parameter matrix and trained jointly with the encoder by the task loss. They are not node2vec features and not one vector per span. Status, span_kind, type, and language remain one-hot. We have clarified how these embeddings are learned in Section 4.2 and expanded the implementation details in Appendix F.1.
References
[1] Wang et al. GFT: Graph Foundation Model with Transferable Tree Vocabulary. NeurIPS, 2024