The Limitations of Standardized Science Tests as Benchmarks for Artificial Intelligence Research: Position Paper release_yqgncd7cvngu3fyojgrg2yniq4

by Ernest Davis


In this position paper, I argue that standardized tests for elementary science such as SAT or Regents tests are not very good benchmarks for measuring the progress of artificial intelligence systems in understanding basic science. The primary problem is that these tests are designed to test aspects of knowledge and ability that are challenging for people; the aspects that are challenging for AI systems are very different. In particular, standardized tests do not test knowledge that is obvious for people; none of this knowledge can be assumed in AI systems. Individual standardized tests also have specific features that are not necessarily appropriate for an AI benchmark. I analyze the Physics subject SAT in some detail and the New York State Regents Science test more briefly. I also argue that the apparent advantages offered by using standardized tests are mostly either minor or illusory. The one major real advantage is that the significance is easily explained to the public; but I argue that even this is a somewhat mixed blessing. I conclude by arguing that, first, more appropriate collections of exam style problems could be assembled, and second, that there are better kinds of benchmarks than exam-style problems. In an appendix I present a collection of sample exam-style problems that test kinds of knowledge missing from the standardized tests.
In text/plain format

Released as a article
Version v2
Release Date 2015-10-16
Primary Language en (lookup)

Known Files and URLs

application/pdf  205.3 kB
sha1:00589586b5992bf6e89c... (webarchive) (repository)
Read Full Text
Type  article
Stage   submitted
Date   2015-10-16
Version   v2
arXiv  1411.1629v2
Work Entity
grouping other versions (eg, pre-print) and variants of this release
Cite This Release
Fatcat Bits

State is "active". Revision:
As JSON object via API