Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets release_dv23qjgj65d4nmjbjcjxxm5q6m

Abstract

The recent COVID-19 crisis highlighted the inadequacy of human forecasting. We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n=1822) and Next Generation Social Science (NGS2) platform (n=103) were utilised. A 43-feature model predicted top quintile relative Brier accuracy scores in two out-of-sample datasets (pboth<1x10-9). Trades graded as high machine accuracy quality vs. other trades had a greater AUC temporal gain from before to after trade. Hybrid human-machine forecasts had higher accuracy than human forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial AUC gains of 13.2%, p=1.35x10-14 and 13.8%, p=0.003 in the out-of-sample Almanis B and NGS2 datasets respectively. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for human-only models, p=0.007. This net classification benefit was replicated in the separate Almanis B dataset, p=2.4x10-7. Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. Implementation may allow improved anticipation of and response to emerging risks and improved human collective efforts generally.
In application/xml+jats format

Archived Files and Locations