- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SimCMTC aims to identify mobile apps from their network traffic. Its CMTE model learns a compact representation of each TCP connection from packet headers and a picture of packet sizes and timing. Pretraining does not use app labels. The paper tests three uses: adding a classifier and training it on five datasets; finding the closest labeled examples without further model training; and using an agent to compare retrieved examples with network details. The reported results are strongest for classification after further training and retrieval on TENAPP. The full agent is tested only on 1,635 TENAPP cases where the closest-example prediction was wrong. All these cases belong to known apps.
Critical strengths
1 Useful traffic representations. After learning without app labels, CMTE can be trained further for app classification. It reports the highest accuracy and F1 among the methods shown on all five datasets, including tests across different network environments. This gives the work practical value, although the model-selection procedure still needs clarification.
2 Useful retrieval without further model training. On TENAPP, simply using the label of the closest stored example reaches accuracy 0.910 and F1 0.898. This supports a useful alternative to retraining a classifier. It still requires a large labeled reference collection, and the weaker results on other datasets limit how broadly this finding can be applied.
Critical weaknesses
1 The test does not establish recognition of new apps. The experiment used to support unknown-app recognition contains no genuinely unknown apps. Appendix F states that all 1,635 cases belong to the ten known apps. Of these, 1,158 receive the correct app label, giving 70.83%; 271 are incorrectly rejected as unknown. The appendix clearly acknowledges this limitation. These results show that the system can correct some earlier errors, but they do not measure detection of new apps. The agent was also tested only on cases already known to be wrong, so its effect on the full test set is not established.
A reasonable explanation is that the system is meant to refuse a decision when a connection lacks enough app-specific information. That can be useful, but it is different from detecting a new app. To keep the broader claim, the paper needs a test with apps excluded from the known reference classes and a practical rule for when to call the agent. Otherwise, the conclusion should be limited to error correction and cautious refusal.
- The final test may have been used to select the model Sections 4.1 and 4.2 say that the best test result during training was reported. If the final test set was used to choose the model, the reported scores may be optimistic. Applying the same rule to all methods does not remove this problem. The wording may instead refer to a separate validation set used to choose the model, but Appendix C does not make that split clear.
Please identify the data used to choose the model and any separate final test. If an independent final test already exists, this concern may be resolved by the split and training records. If not, the main comparison needs an independent evaluation before the reported gains can be relied on.
Questions for authors
1 Data used to choose the model Does “best test result” mean a separate validation set, or the final test set? Please give the split and the rule used to choose the saved model. A separate final test would substantially reduce the second concern above.
2 What an unknown result means Is the target the app that started the connection, the service it contacted, or a refusal to decide? Is there an existing test containing genuinely new app classes? If so, please report both missed known apps and new apps incorrectly accepted as known. If not, the unknown-app claim needs a narrower wording or a direct test.
3 What the agent adds Is there a full-test result using a rule that decides when to call the agent without knowing the true label? Please also compare with a simple method using the same retrieved examples and network information. The agent model, full instructions, decision rules, retries and running cost would help make this comparison checkable.
4 Benefit of contrastive training Table 2 gives slightly better matching results after removing MoCo. Which measured task shows the benefit claimed for keeping it? Also, does the vector analysis use the TENAPP training set or test set? A direct result using the existing models would be more informative than the number of dimensions alone.
Response to Reviewer rc5x
We thank the reviewer for the careful reading. All changes are in the revised paper; we answer each sub-question separately.
1. Data used to choose the model
1.1 Is "best test result" a validation set or the final test set? We acknowledge the earlier wording was ambiguous and could be read as selecting the checkpoint with the best test score, which is optimistic. The revised main results (Table 1) report the final-epoch test result under a fixed schedule: CMTE, ET-Bert, and TrafficFormer are fine-tuned for 10 epochs and YaTC for 200 epochs (§4.1). No test-set feedback selects any model.
1.2 What is the split and the saved-model rule? The rule is deterministic and applied identically to all methods (final-epoch checkpoint). Except CHNAPP — the only same-source 9:1 split (train 2,684 / test 295, App. C) — every test set is collected in a different network environment or time from its training set (CrossNet2021: env. A to B; Blend-60: 5 device/network scenarios; MingledApp: Tehran/Qom to Iraq; TENAPP: different dates). Hence final-epoch results still reflect genuine out-of-domain generalization. The splits and the checkpoint rule are reported in §4.1 / App. C.
2. Meaning of the unknown result and genuinely new apps
2.1 What does "unknown" mean — the initiating app, the service, or a rejection decision? The target label is the hosting app class of the session. "Unknown" is a deliberate rejection decision: when neither the domain/DPI evidence nor the retrieval-fusion can confidently attribute a session, it is reported as one of the open-set categories (SDK / OAS / unknown) rather than forced into a target app (§3.3). This is the cautious, abstaining behavior the reviewer describes.
2.2 Is there a test with genuinely new app classes? Yes. §4.2.2 holds Bilibili, Rednote, and Kuaishou out of the reference store as open-set classes. On the full 18,250-session TENAPP test set, the framework reaches 0.8866/0.8833 vs. the pure-DPI agent's 0.8116/0.7817 (Table 2). App. F resolves these flows into SDK (951), OAS (4,226), and unknown (1,467) sessions — held-out/third-party traffic is safely rejected, not accepted as a target class.
3. What the agent adds
3.1 Is there a full-test result under a label-free "when to call the agent" rule? Yes. The invocation rule is label-free: only L1-unresolved sessions enter the agent/L2 path (§3.3). Running the complete framework end-to-end on the full test set gives 0.8866/0.8833 vs. the pure-DPI baseline 0.8116/0.7817 (Table 2).
3.2 What does the agent add over a simple method using the same retrieved examples and network information? Two things, and one controlled comparison. The agent (i) avoids hand-writing a large complex rule base: it autonomously judges the ownership of each extracted domain (TLS SNI / HTTP Host / nDPI SNI) as a target app, SDK, other app, or unknown, so new/unseen domains need no new rules; and (ii) makes autonomous open-set decisions — attributing unfamiliar domains to SDK/OAS/unknown and falling back to retrieval-fusion when a domain cannot be attributed, instead of forcing a class (§3.3). For the controlled comparison,
w_vec=0in App. G scores the same embedding-retrieved candidates by nDPI similarity only: best 0.726/0.585 vs. best full 0.7758/0.6779 (validation unknown sessions), i.e., +5.0% Acc / +9.3% Macro-F1 from the semantic-similarity term; pure-DPI without retrieval is 0.1707/0.0365. Retrieval itself relies on CMTE embeddings.3.3 Agent model, full instructions, decision rules, retry, and running cost. The complete SimilarityDPI skill (pipeline, mandatory-condition gate, fusion config
w_vec=0.60, fuse_min=0.75) is public athttps://anonymous.4open.science/r/SimCMTC-ED76/skills/SEVENAPPForSessionPcap_en/SKILL.md; the agent model, full instructions, decision rules, retry mechanism, and running cost are described in Appendix F.4. Benefit of contrastive training
4.1 On which measurement is the claimed benefit realized? On the embedding-space geometry that retrieval depends on, not a single-point accuracy gain. §4.4 / Table 4 measures it directly: adding debiased contrastive learning lowers the average inter-sample similarity and increases the effective dimensionality, so the space is well spread and avoids dimensional collapse.
4.2 Is the vector/PCA analysis on the training or test set? On the TENAPP train-set embeddings (and the NUDT-Mobile-mini subset for the threshold analysis), computed from the existing model (§4.4, App. D): inter-sample similarity 0.4641$\to$0.0798 and PCA dims 77$\to$209 on TENAPP train (mini: 0.2628$\to$0.0011, 189$\to$461).