Harnessing Accurate and Automatic Trend Detection in Data Streams via Tbps-Level Inference

Detecting frequency trend patterns, such as items with sustained growth or decline, is an important and challenging task in high-speed data streams. State-of-the-art solutions require manual tuning of many coupled parameters, while their bloom filters suffer from high false positive rates. In this paper, we propose NeuTrend, a framework that provides accurate and automatic trend detection with high-performance in-network inference. Our key idea is that a lightweight BRNN can learn to classify incoming items as likely trending or non-trending. Thus, it eliminates unqualified items more accurately than a bloom filter and removes the need for manual parameter tuning. The BRNN uses only XNOR and popcount operations. Thus, it aligns with the strict resource constraints of data-plane switches. NeuTrend further introduces an adaptive detection module. The module uses the BRNN confidence scores to automatically tune the growth and decay thresholds. Extensive experiments on real-world datasets show that the learned filter reduces false positive rates by 23%-44% and increases effective detector occupancy from 47% to 70%. Hence, NeuTrend improves the overall F1 score by 35%-68% over existing solutions, and achieves Tbps-level line-rate processing.