Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets release_dv23qjgj65d4nmjbjcjxxm5q6m

by Alexander Gruen, Karl Mattingly, Ellen Morwitch, Frederik Bossaerts, Manning Clifford, Chad Nash, John Ioannidis, Anne-Louise Ponsonby

Released as a post by Cold Spring Harbor Laboratory.

2023  

Abstract

The recent COVID-19 crisis highlighted the inadequacy of human forecasting. We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n=1822) and Next Generation Social Science (NGS2) platform (n=103) were utilised. A 43-feature model predicted top quintile relative Brier accuracy scores in two out-of-sample datasets (pboth<1x10-9). Trades graded as high machine accuracy quality vs. other trades had a greater AUC temporal gain from before to after trade. Hybrid human-machine forecasts had higher accuracy than human forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial AUC gains of 13.2%, p=1.35x10-14 and 13.8%, p=0.003 in the out-of-sample Almanis B and NGS2 datasets respectively. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for human-only models, p=0.007. This net classification benefit was replicated in the separate Almanis B dataset, p=2.4x10-7. Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. Implementation may allow improved anticipation of and response to emerging risks and improved human collective efforts generally.
In application/xml+jats format

Archived Files and Locations

application/pdf  1.9 MB
file_dhzckj56ovculhsflitwzz2bdq
www.medrxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  post
Stage   unknown
Date   2023-01-20
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 34f3335a-295f-48de-b3c6-552b08be7964
API URL: JSON