Arguing Practical Significance in Software Engineering Using Bayesian
Data Analysis
release_byxcyvbsb5h5fg6rrlkjfmshsm
by
Richard Torkar and Robert Feldt and Carlo A. Furia
2018
Abstract
This paper provides a case for using Bayesian data analysis (BDA) to make
more grounded claims regarding practical significance of software engineering
research.
We show that using BDA, here combined with cumulative prospect theory (CPT),
is appropriate when a researcher or practitioner wants to make clearer
connections between statistical findings and practical significance in
empirical software engineering research. To illustrate our point we provide an
example case using previously published data. We build a multilevel Bayesian
model for this data, for which we compare the out of sample predictive power.
Finally, we use our model to make out of sample predictions while, ultimately,
connecting this to practical significance using CPT.
Throughout the case that we present, we argue that a Bayesian approach is a
natural, theoretically well-grounded, practical work-flow for data analysis in
empirical software engineering. By including prior beliefs, assuming parameters
are drawn from a probability distribution, assuming the true value is a random
variable for uncertainty intervals, using counter-factual plots for sanity
checks, conducting predictive posterior checks, and out of sample predictions,
we will better understand the phenomenon being studied, while at the same time
avoid the obsession with p-values.
In text/plain
format
Archived Files and Locations
application/pdf 4.2 MB
file_vhm3t6sb3rhu5ckmpfaugqqmhy
|
arxiv.org (repository) web.archive.org (webarchive) |
1809.09849v1
access all versions, variants, and formats of this works (eg, pre-prints)