Arguing Practical Significance in Software Engineering Using Bayesian Data Analysis release_byxcyvbsb5h5fg6rrlkjfmshsm

by Richard Torkar and Robert Feldt and Carlo A. Furia

Released as a article .

2018  

Abstract

This paper provides a case for using Bayesian data analysis (BDA) to make more grounded claims regarding practical significance of software engineering research. We show that using BDA, here combined with cumulative prospect theory (CPT), is appropriate when a researcher or practitioner wants to make clearer connections between statistical findings and practical significance in empirical software engineering research. To illustrate our point we provide an example case using previously published data. We build a multilevel Bayesian model for this data, for which we compare the out of sample predictive power. Finally, we use our model to make out of sample predictions while, ultimately, connecting this to practical significance using CPT. Throughout the case that we present, we argue that a Bayesian approach is a natural, theoretically well-grounded, practical work-flow for data analysis in empirical software engineering. By including prior beliefs, assuming parameters are drawn from a probability distribution, assuming the true value is a random variable for uncertainty intervals, using counter-factual plots for sanity checks, conducting predictive posterior checks, and out of sample predictions, we will better understand the phenomenon being studied, while at the same time avoid the obsession with p-values.
In text/plain format

Archived Files and Locations

application/pdf  4.2 MB
file_vhm3t6sb3rhu5ckmpfaugqqmhy
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2018-09-26
Version   v1
Language   en ?
arXiv  1809.09849v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: f1ad94ea-fa72-4d9d-9429-2eec6a5f943c
API URL: JSON