A Case Study on the Stability of Performance Tests for Serverless Applications
release_btcur6zzcjej5afgtmeyy22fzy
by
Simon Eismann, Diego Elias Costa, Lizhi Liao, Cor-Paul Bezemer, Weiyi Shang, André van Hoorn, Samuel Kounev
2021
Abstract
Context. While in serverless computing, application resource management and
operational concerns are generally delegated to the cloud provider, ensuring
that serverless applications meet their performance requirements is still a
responsibility of the developers. Performance testing is a commonly used
performance assessment practice; however, it traditionally requires visibility
of the resource environment.
Objective. In this study, we investigate whether performance tests of
serverless applications are stable, that is, if their results are reproducible,
and what implications the serverless paradigm has for performance tests.
Method. We conduct a case study where we collect two datasets of performance
test results: (a) repetitions of performance tests for varying memory size and
load intensities and (b) three repetitions of the same performance test every
day for ten months.
Results. We find that performance tests of serverless applications are
comparatively stable if conducted on the same day. However, we also observe
short-term performance variations and frequent long-term performance changes.
Conclusion. Performance tests for serverless applications can be stable;
however, the serverless model impacts the planning, execution, and analysis of
performance tests.
In text/plain
format
Archived Files and Locations
application/pdf 891.1 kB
file_jrieshbjifghfh3wtplo7vdzgi
|
arxiv.org (repository) web.archive.org (webarchive) |
2107.13320v1
access all versions, variants, and formats of this works (eg, pre-prints)