Railgun: managing large streaming windows under MAD requirements release_otrohxszszay7dwp4cdmcrfqke

by Ana Sofia Gomes, João Oliveirinha, Pedro Cardoso, Pedro Bizarro

Released as a article .

2021  

Abstract

Some mission critical systems, e.g., fraud detection, require accurate, real-time metrics over long time sliding windows on applications that demand high throughput and low latencies. As these applications need to run 'forever' and cope with large, spiky data loads, they further require to be run in a distributed setting. We are unaware of any streaming system that provides all those properties. Instead, existing systems take large simplifications, such as implementing sliding windows as a fixed set of overlapping windows, jeopardizing metric accuracy (violating regulatory rules) or latency (breaching service agreements). In this paper, we propose Railgun, a fault-tolerant, elastic, and distributed streaming system supporting real-time sliding windows for scenarios requiring high loads and millisecond-level latencies. We benchmarked an initial prototype of Railgun using real data, showing significant lower latency than Flink and low memory usage independent of window size. Further, we show that Railgun scales nearly linearly, respecting our msec-level latencies at high percentiles (<250ms @ 99.9%) even under a load of 1 million events per second.
In text/plain format

Archived Files and Locations

application/pdf  1.1 MB
file_gl3a6fe7nfanvafjcbpreggeeq
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2021-06-23
Version   v1
Language   en ?
arXiv  2106.12626v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 1b1b5963-1ef6-4936-9f42-c0cddcb9db5b
API URL: JSON