A Theory of Statistical Inference for Ensuring the Robustness of
Scientific Results
release_2kxi527kqjdprdln6h7tdox3nq
by
Beau Coker, Cynthia Rudin, Gary King
(2018)
Abstract
Inference is the process of using facts we know to learn about facts we do
not know. A theory of inference gives assumptions necessary to get from the
former to the latter, along with a definition for and summary of the resulting
uncertainty. Any one theory of inference is neither right nor wrong, but merely
an axiom that may or may not be useful. Each of the many diverse theories of
inference can be valuable for certain applications. However, no existing theory
of inference addresses the tendency to choose, from the range of plausible data
analysis specifications consistent with prior evidence, those that
inadvertently favor one's own hypotheses. Since the biases from these choices
are a growing concern across scientific fields, and in a sense the reason the
scientific community was invented in the first place, we introduce a new theory
of inference designed to address this critical problem. We derive "hacking
intervals," which are the range of a summary statistic one may obtain given a
class of possible endogenous manipulations of the data. Hacking intervals
require no appeal to hypothetical data sets drawn from imaginary
superpopulations. A scientific result with a small hacking interval is more
robust to researcher manipulation than one with a larger interval, and is often
easier to interpret than a classical confidence interval. Some versions of
hacking intervals turn out to be equivalent to classical confidence intervals,
which means they may also provide a more intuitive and potentially more useful
interpretation of classical confidence intervals
In text/plain
format
Archived Files and Locations
application/pdf 2.1 MB
file_x2hzylxm6fcldisefkhsoizpxm

web.archive.org (webarchive) arxiv.org (repository) 
article
Stage
submitted
Date 20180423
Version
v1
Language
en
^{?}
1804.08646v1
access all versions, variants, and formats of this works (eg, preprints)